Sensor data segmentation

Abstract

To provide a method, an apparatus and a system for segmenting sensor data.SOLUTION: A method includes: receiving a plurality of images representing an environment; transmitting image data to an image segmentation network configured to segment the images; receiving sensor data associated with the environment including sensor data generated by a sensor of a type different from an image capture device; associating the sensor data with segmented images to create a training dataset; transmitting the training dataset to a machine learning network configured to run a sensor data segmentation model; and training the sensor data segmentation model using the training dataset such that the sensor data segmentation model is configured to segment sensor data.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to sensor data segmentation, and more particularly to sensor data segmentation directed to methods, apparatus, and systems for segmenting sensor data. [Background technology]

[0002] Sensors may be used to generate sensor data indicative of objects in an environment. However, the raw form of the sensor data generated by the sensors can make the data difficult to use or analyze. Therefore, the sensor data may be segmented to transform it into a more useful form (e.g., identifying individual objects in the data, areas in the sensor data that are drivable (i.e., that can be driven over), etc.). Generally, segmenting data involves dividing or organizing the sensor data into a more meaningful or organized form, for example, so that areas of data captured in the sensor data can be identified or classified. Data segmentation may be performed manually by a human. However, manual segmentation can be extremely time-consuming and expensive, and is often unsuitable for many applications. Image data may be automatically segmented using a computer that divides the image into distinct segments to provide a more meaningful or usable representation of the image. For example, an image may be segmented to uniquely identify objects within the image, which may be useful in some applications, such as, for example, autonomous vehicle operation. However, it can be difficult to automatically segment sensor data obtained from other types of sensors. In the above example, manual segmentation by a human may be required, making the sensor data obtained from the above sensor types prohibitively expensive or difficult to use for some applications.

[0003] The detailed description will be set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The same reference number in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a pictorial flow diagram of an exemplary process for capturing image data and sensor data, segmenting the captured image, projecting the sensor data onto the segmented image, and generating a training data set. [Figure 2] 1 illustrates an exemplary environment in which an exemplary vehicle is capturing image data and sensor data as it travels. [Figure 3] 1 illustrates an exemplary architecture for generating a segmented image, training an exemplary sensor data segmentation model, and segmenting sensor data. [Figure 4A] 1 illustrates a side view of an exemplary vehicle having multiple image capture devices and sensors. [Figure 4B] 4B illustrates a top view of the exemplary vehicle shown in FIG. 4A. [Figure 5A] 1 depicts an exemplary image including image data and a segmented version of the exemplary image segmented for objects. [Figure 5B] 5B illustrates the exemplary image shown in FIG. 5A and a segmented version of the exemplary image segmented relative to the drivable surface. [Figure 6] FIG. 1 is a pictorial flow diagram of an exemplary process for capturing image data and sensor data. [Figure 7] FIG. 7 is a pictorial flow diagram of an exemplary process for segmenting the image shown in FIG. 6 and projecting the sensor data captured in FIG. 6 onto the segmented image to generate a training data set. [Figure 8] FIG. 10 is a pictorial flow diagram of an exemplary process for generating segmented sensor data for use by a trajectory module that ingests sensor data and uses an exemplary sensor data segmentation network to segment the sensor data to generate a vehicle trajectory. [Figure 9] FIG. 10 is a flow diagram of an example process for training an example sensor data segmentation model using an example training dataset including segmented images and sensor data. [Figure 10] FIG. 1 is a flow diagram of an example process for segmenting sensor data using an example sensor data segmentation model. [Figure 11] 1 depicts a block diagram of an exemplary computer architecture for implementing the processes described herein. DETAILED DESCRIPTION OF THE INVENTION

[0005] In general, the present disclosure is directed to methods, apparatus, and systems for segmenting sensor data. The above-described segmentation of sensor data may be aided by segmentation of other types of sensor data that are more easily segmented. For example, a system may simultaneously or substantially simultaneously capture image data using one or more image capture devices and sensor data using one or more sensors of a type different from the image capture devices. Because segmentation of image data can produce more accurate results than segmentation of sensor data (other than image data), the image data may be used to aid or train the segmentation of the sensor data. The system may segment an image and associate the sensor data with the segmented image to create a training data set that can be used to train a sensor data segmentation model. For example, sensor data (e.g., sensor data captured by a LIDAR sensor as described herein) can be projected onto the segmented image, and segmentation information associated with the segmented image may be transformed or applied to the sensor data to create the training data set. The trained sensor data segmentation model can be used to automatically segment sensor data obtained from other types of sensors, thereby making the sensor data more useful for many applications, such as, for example, operating an autonomous vehicle. Other applications are possible. Similarly, in some examples, any one type of sensor data can be used to assist and / or train the segmentation of another type of sensor data.

[0006] In general, the present disclosure is directed to a system configured to receive multiple images representing an environment. The images may include image data generated by an image capture device. The system may be configured to send the image data to an image segmentation network configured to segment the images and generate segmented images. Additionally, the system may receive sensor data associated with the environment. The sensor data may include sensor data generated by a sensor type other than the image capture device. For example, the image capture device and the sensor may capture data associated with the environment simultaneously or substantially simultaneously (within technical capabilities). For example, the image capture device may include one or more cameras (e.g., an RGB camera, a monochrome camera, an intensity (grayscale) camera, an infrared camera, an ultraviolet camera, a depth camera, a stereo camera, etc.). For example, the sensors may include one or more light detection and ranging (LIDAR) sensors, one or more radio detection and ranging (RADAR) sensors, one or more sound navigation and ranging (SONAR) sensors, and other sensors. Additionally, the system may be configured to associate the sensor data with the segmented images to create a training dataset. The training data set can be sent to a machine learning network configured as a sensor data segmentation model. Further, the system can be configured to train the sensor data segmentation model using the training data set, such that the sensor data segmentation model is configured to segment sensor data obtained from a sensor type.

[0007] In some examples, the system may be configured to project the sensor data onto the segmented image and associate the sensor data with a class (or classification) of the segmented image. Each of the images captured by the image capture devices may be captured at a respective image time, and different portions of the sensor data may be captured at different sensor times. In some examples, the system may be configured to select an image captured at an image time that minimizes the difference between the image time and the sensor time associated with a portion of the sensor data, and associate the portion of the sensor data with the selected image or portion of the image. In the exemplary manner described above, the difference in timing between the capture of image data and the capture of sensor data may be minimized, thereby reducing the likelihood of potential errors in the segmented sensor data. In some examples, multiple image capture devices may capture image data, and the image data may be aligned and fused before the sensor data is associated with the segmented image.

[0008] In some examples, the system may be configured to align the sensor data with the segmented image. The image data may be captured from a different angle and / or may be associated with a different view of the environment compared to the view of the other sensor. For example, the image data may encompass a view having a different width, height, and / or aspect ratio than the view encompassed by the sensor data. In some examples, the system may be configured to temporally and / or spatially align the segmented image with the sensor data. Additionally, the field of view of the image capture device and the field of view of the other sensor may at least partially overlap, and the image data and the sensor data include data associated with portions of the environment that are common to both the field of view of the image capture device and the field of view of the other sensor.

[0009] In some examples, the system may be configured to identify image data associated with partial representations of objects in an image and to ignore or mask the segmented image data associated with the partial representations when associating sensor data with segmented images to create a training dataset. Partial representations can be a source of error in the training data; therefore, some examples of the system may be configured to ignore such data to reduce the likelihood of errors being created in the training dataset. In some examples, the system may be configured to remove sensor data associated with discontinuities in the sensor data. For example, a discontinuity in the sensor data can be an indication that the sensor detected more than a single object, such as a person and an object behind the person, such as a wall. Removing, masking, or ignoring data associated with the discontinuity can reduce the likelihood of errors being created in the training dataset. Other techniques for reducing the likelihood of errors are possible. For example, identifying a discontinuity can include determining that LIDAR data points associated with a single object represent an unusually wide range of distances. For example, a person may be standing 10 feet in front of a solid wall. When segmenting the associated image, the background may be captured by the "pedestrian object." However, when projecting LIDAR data onto the "pedestrian object," some of the LIDAR data may be 10 feet farther from the sensor than the data actually associated with the person. In the above example, if the distance exceeds a threshold, this would indicate a discontinuity, and in some instances, the data may be masked, ignored, or removed to avoid creating inaccuracies in the training data and in the machine learning network.

[0010] In some examples, rather than using segmented images to generate a training data set for training a machine learning network, previously acquired and segmented data from a sensor may be used to generate a training data set for training a sensor data segmentation model.

[0011] Additionally, the present disclosure is generally directed to systems configured to segment sensor data generated by sensors other than image capture devices. For example, the system may be configured to receive data from such sensors, e.g., one or more LIDAR sensors, one or more RADAR sensors, one or more SONAR sensors, and other sensors. For example, the system may be configured to segment sensor data received from a LIDAR sensor using a sensor data segmentation model to generate segmented LIDAR data. For example, the system may use a machine learning model (e.g., a neural network) configured to run the sensor data segmentation model to generate the segmented sensor data (e.g., for a neural network, to use the trained network in inference mode to segment the sensor data). In this exemplary manner, the system may automatically segment the sensor data. By segmenting the sensor data, the segmented sensor data may have a more useful shape. For example, the segmented sensor data may be used by a perception system of an autonomous vehicle, which may inform one or more systems that generate trajectories for operation of the autonomous vehicle based at least in part on the segmented sensor data. Other uses of the segmented sensor data are possible.

[0012] Additionally, the present disclosure is generally directed to a method for segmenting sensor data. The method may include receiving one or more images representing an environment. The images may include image data generated by one or more image capture devices. Furthermore, the method may include transmitting the image data to an image segmentation network configured to segment the image. Furthermore, the method may include receiving sensor data generated by sensors other than the image capture devices. The sensor data may include data representing the environment. Furthermore, the method may include segmenting the image to generate a segmented image and associating the sensor data with the segmented image to create a training dataset. Furthermore, the method may include transmitting the training dataset to a machine learning network configured to run a sensor data segmentation model (i.e., to use the segmentation model to segment the sensor data), and training the sensor data segmentation model using the training dataset, such that the sensor data segmentation model is configured to segment sensor data acquired from the sensors.

[0013] Additionally, the present disclosure is generally directed to a method for generating a trajectory for an autonomous vehicle. The method may include receiving sensor data from a sensor other than an image capture device and segmenting the sensor data received from the sensor using a sensor data segmentation model. Further, the method may include generating one or more trajectories for the autonomous vehicle based at least in part on the segmented sensor data.

[0014] The generation of training data and training of machine learning models described herein can improve the functionality of computing devices by generating training data in a more accurate and reliable manner that improves the performance of segmentation and / or classification of objects represented in sensor data. In some examples, the improved training data generation and associated systems may provide more accurate and / or faster segmentation by training machine learning networks and / or algorithms for improved automatic segmentation and / or classification of sensor data. Automatically determining segmentation information based on image segmentation using the training data generation and training operations described herein leads to more accurate and / or faster classification of objects by exposing additional data to the machine learning algorithm. Furthermore, the training data and training techniques described herein can take into account deep learning techniques that provide improved processing. In some examples, faster and / or more accurate segmentation and / or classification may be utilized when generating trajectories for autonomous vehicles, which may improve safety for occupants of the autonomous vehicle. In some examples, the training data generation and training operations described herein may reduce memory requirements or reduce the amount of processing by applying machine learning operations (e.g., artificial neural networks) to simplified (e.g., segmented) data. In further examples, such methods may be easily extended to incorporate multiple sensor types and subtypes, which could otherwise require creating multiple training sets. These and other improvements to computer functionality are discussed herein.

[0015] The techniques and systems described herein may be implemented in multiple forms, example implementations of which are provided below with reference to the drawings.

[0016] FIG. 1 is a pictorial flow diagram 100 of an example process for capturing images and sensor data, segmenting the captured images, projecting the sensor data onto the segmented images, and generating a training dataset for training a sensor data segmentation model. In the illustrated example, an example vehicle 102 is traveling through an environment 104. For example, the vehicle 102 may be an autonomous vehicle, such as the example vehicle described herein with respect to FIG. 2. The vehicle 102 may include one or more image capture devices 106 and one or more sensors 108 (e.g., one or more LIDAR sensors, radar sensors, ultrasonic transducers, etc.) configured to capture image data 110 and sensor data 112 associated with the environment 104 provided by the one or more image capture devices 106 and the one or more sensors 108, respectively. While FIG. 1 identifies a single image capture device 106 and a single sensor 108, more image capture devices 106 and / or more sensors 108 are contemplated, for example, as described herein with respect to FIGS. 4A and 4B . Image capture device 106 may be any type of image capture device configured to capture images representative of an environment, such as, for example, one or more cameras (e.g., an RGB camera, a monochrome camera, an intensity (grayscale) camera, an infrared camera, an ultraviolet camera, a depth camera, a stereo camera, etc.). Sensor 108 may be any type of sensor, other than an image capture device, configured to generate data representative of an environment. For example, sensor 108 may be one or more LIDAR sensors, one or more RADAR sensors, one or more SONAR sensors, and other sensors.

[0017] 1 , image capture device 106 is shown capturing image data 110 in the form of images 114A-114C corresponding to three respective fields of view at three consecutive times t1, t2, and t3. For example, as shown schematically, image capture device 106 captures image 114A associated with a field of view at time t1, image 114B associated with a field of view at time t2, and image 114C associated with a field of view at time t3. Each of images 114A-114C may have a slightly different field of view as image capture device 106 progresses through environment 104. In the above example, each image 114A-114C has a different respective perspective encompassing a distinct portion of environment 104. In some examples, the image data 110 captured by each of images 114A-114C may be fused to generate a more complete or accurate representation of environment 104.

[0018] For illustrative purposes, the exemplary sensor 108 shown in FIG. 1 is a LIDAR sensor configured to capture sensor data 112 in the form of LIDAR sensor data. Other types of sensors are contemplated. A LIDAR sensor may have a light emitting element and a light sensor, where the light emitting element includes one or more lasers that direct highly focused light toward an object or surface, which reflects the light back to the light sensor. LIDAR sensor measurements may be represented as three-dimensional LIDAR data having coordinates (e.g., Cartesian coordinates, polar coordinates, etc.) corresponding to the position or distance captured by the LIDAR sensor. In some examples, a transformation operation may be used to convert the three-dimensional LIDAR data into multi-channel, two-dimensional data, where azimuth and elevation are mapped to XY coordinates, one channel includes range, and the other channel includes intensity. In some examples, the LIDAR data may be automatically segmented as described herein; for example, the segmented LIDAR data may be used as input for determining a trajectory for an autonomous vehicle.

[0019] In some examples, a LIDAR sensor may be configured such that one or more lasers are mounted to rotate (e.g., about a substantially vertical axis), thereby causing the lasers to sweep 360 degrees, for example, to capture LIDAR data associated with an environment. For example, as shown in FIG. 1 , the one or more lasers may sweep an angle α such that at time t, one or more lasers are pointed at the environment 104 along a vertical position p. Similarly, at time t, one or more lasers are pointed at the environment along a vertical position p, at time t, one or more lasers are pointed at the environment along a vertical position p, and at time t, one or more lasers are pointed at the environment along a vertical position p.

[0020] In some examples, the exemplary sensor 108 is configured to capture a field of view 116 that may differ from the field of view associated with each of the respective images 114A-114C captured by the image capture device 106. While shown in FIG. 1 for illustrative purposes, such field of view may be less than or equal to, for example, 360 degrees. In the example shown, the exemplary field of view 116 of the sensor 108 also has a different aspect ratio than the field of view associated with each of the images 114A-114C captured by the image capture device 106. For example, the field of view 116 of the LIDAR sensor may be wider and narrower (e.g., shorter in height) than the respective field of view associated with the images 114A-114C. It is contemplated that in some examples, the field of view 116 may be the same as one or more of the fields of view associated with the images captured by the image capture device 106. In examples where the field of view 116 of the sensor 104 differs from one or more of the fields of view associated with the images captured by the image capture device 106, as described in further detail herein, the systems described herein may be configured to align the image with the sensor data 112 to account for the differences. Additionally, the sensor 104 (e.g., a LIDAR sensor) may be configured to generate sensor data in multiple channels and / or as a function of beam spread, and the system may be configured to align the image with the sensor data 112 to account for the differences between the image data 110 and the sensor data 112.

[0021] As shown in FIG. 1 , an exemplary process 100 may include sending image data 110, including images 114A-114C, to an image segmentation network 118 configured to automatically segment images 114A-114C to generate segmented images 120A-120C, for example, via machine learning and / or algorithms that may predict segmented images from input images. Machine learning generally refers to a broad class of algorithms in which outputs are generated based on learned parameters, and will be described in more detail below. In some embodiments, an exemplary machine learning algorithm that may be used to generate segmented images 120A-120C is an artificial neural network (ANN). ANNs are biologically inspired algorithms that pass input data through a series of connected layers to generate outputs. Each layer in an ANN may include any number of layers and may further include other ANNs. Generally, the manner in which the various layers of an ANN are connected to each other is referred to as the architecture of the ANN. Other types of image segmentation networks for segmenting images are contemplated.

[0022] Image segmentation can be used to partition an image into distinct segments, or superpixels, to provide a more meaningful representation of the image. For example, an image can be segmented to uniquely identify objects within the image, with each object having a unique identifier. Image segmentation can be used in a number of different applications. For example, a segmented image associated with a scene in an environment can include a representation of one or more objects located in the environment. The presence and / or location of objects included in the segmented image can then be used for obstacle avoidance, object detection and tracking, etc., in systems that employ aspects of machine vision. For example, as described herein, the segmented image and / or segmented sensor data can be used by an autonomous vehicle's perception system to generate a vehicle trajectory for moving the autonomous vehicle through an environment. In some examples, an image can be segmented to provide a drivable surface. In the above example, pixels can be broadly classified as belonging to a drivable surface or not. The segmented image can be used to inform the autonomous vehicle of which trajectories are feasible.

[0023] 1 , a sensor 108 may generate sensor data 112 representing the environment 104 in which the sensor 108 resides. At 122, the sensor data 112 may be projected (e.g., via deformation and unprojection) onto the segmented images 120A-120C. As described herein, the sensor data 112 may be projected onto the segmented images 120A-120C such that the sensor data 112 and the segmented images 120A-120C are substantially temporally aligned (within technical capabilities). Additionally, as described herein, the sensor data 112 may be projected onto the segmented images 120A-120C such that sensor data 112 associated with objects in the environment 104 are spatially aligned (within technical capabilities) with the objects in the segmented images 120A-120C.

[0024] For example, the sensor 108 may be a LIDAR sensor including, for example, one or more lasers rotating about an axis (e.g., a substantially vertical axis), as described above, capturing LIDAR sensor data as the one or more lasers rotate. As the lasers rotate, at times t0, t1, t2, and t3, they capture sensor data 112 corresponding to vertical lines p0, p1, p2, and p3. In some examples, the LIDAR sensor may provide LIDAR sensor data at a frequency that can be mapped (within technical tolerances) to a single image or group of images. In some examples, the image capture device 106 may be configured to capture images at a rate faster than the laser completes a sweep over the environment. Thus, in some examples, the sensor data 112 may be temporally aligned with the image data 110. For example, sensor data 112 captured at a given time (e.g., sensor time) may be aligned in time with an image captured at approximately the same time (e.g., image time) as the sensor data 112, e.g., by selecting an image captured closest to the time the sensor data 112 was captured (e.g., in sensor time).

[0025] 1 , the LIDAR sensor begins its sweep across the environment 104 at t0 and continues its sweep until t1. At t1, the image capture device 106 captures image 114A. Thus, in the example just described, sensor time t1 is aligned in time with image time t1, at which image 114A was captured. Additionally, at sensor time t1, the LIDAR sensor is only sweeping partially across the sensed environment 104 (e.g., approximately one-third of the way across the sensed environment 104), which in the example shown correlates to approximately the rightmost one-third of the way across image 114A. As the LIDAR sensor continues its sweep from sensor time t1 to t2, the image capture device 106 at image time t2 captures image 114B. Thus, the LIDAR data captured during the sensor time between times t1 and t2 most closely corresponds in time to image time t2. Additionally, between sensor times t1 and t2, the LIDAR sensor sweeps partially across the sensed environment 104 (e.g., from about 1 / 3 of the way across the sensed environment 104 to about 2 / 3 of the way across the sensed environment 104), which in the example shown correlates to about the middle 1 / 3 of the way across image 114B. As the LIDAR sensor continues its sweep from sensor time t2 to t3, image capture device 106 at image time t3 captures image 114C. Thus, the LIDAR data captured during the sensor time between times t2 and t3 most closely corresponds in time to image time t3. Additionally, between sensor times t2 and t3, the LIDAR sensor sweeps from partially across the sensed environment 104 (e.g., from about 2 / 3 of the way across the sensed environment 104 to almost entirely across the sensed environment 104), and in the example shown, what has just been described correlates to about the leftmost 1 / 3 of the way across image 114C.In this exemplary manner, at least some of the images captured by the image capture device 106 may be correlated to at least a portion of the sensor data 112 captured by the sensor 108 (e.g., a LIDAR sensor).

[0026] In some examples, the LIDAR sensor may continue to sweep 360 degrees while some of the captured data does not correspond to the sensed environment. In some examples, the captured data that does not correspond to the sensed environment may be ignored or used for other purposes, such as, for example, calibrating the LIDAR sensor.

[0027] When the sensor data 112 is spatially projected onto the segmented image data 120, the projection may include a transformation determined using the relative extrinsics of the sensor(s) and imaging device(s), as well as dehomogenization using the camera matrix of the imaging device(s).

[0028] 1 , projecting at least a portion of the sensor data 112 onto the segmented images 120A-120C creates a training data set 124 that can be used to train a sensor data segmentation model 126 for automatically segmenting sensor data generated by types of sensors other than image capture devices, as described herein. For example, objects in the environment 104 identified in the segmented images 120A-120C can be correlated with the sensor data 112, and features in the sensor data 112 can thus be used to identify similar objects in the future using types of sensors other than image capture devices. In some examples, the sensor data 112 can be used as additional training data for training the sensor data segmentation model 126.

[0029] 2 illustrates an example environment 200 through which an example vehicle 202 is traveling. The example vehicle 202 may be a driverless vehicle, such as an autonomous vehicle configured to operate according to a Level 5 classification issued by the U.S. Federal Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey, without the driver (or passenger) being expected to control the vehicle at any time. In such an example, the vehicle 202 may be configured to control all functions from the beginning to the completion of the journey, including all parking functions, and thus may not include a driver and / or controls for operating the vehicle 202, such as a steering wheel, accelerator pedal, and / or brake pedal. This is merely an example, and the systems and methods described herein may be incorporated into any land, air, or water vehicle, including vehicles ranging from those that must be manually controlled by a driver at all times to those that are partially or fully autonomously controlled.

[0030] The exemplary vehicle 202 may be a vehicle of any configuration, such as, for example, a van, a sport utility vehicle, a crossover vehicle, a truck, a bus, an agricultural vehicle, or a construction vehicle. The vehicle 202 may be powered by one or more internal combustion engines, one or more electric motors, hydrogen power, any combination thereof, and / or any other suitable power source. While the exemplary vehicle 202 has four wheels 204, the systems and methods described herein may be incorporated into vehicles having a fewer or greater number of wheels, tires, and / or tracks. The exemplary vehicle 202 may have four-wheel steering and may generally operate with equal performance characteristics in all directions, for example, such that a first end 206 of the vehicle 202 is the front end of the vehicle 202 when traveling in a first direction 208, and such that the first end 206 is the rear end of the vehicle 202 when traveling in an opposite second direction 210, as shown in FIG. 2 . Similarly, the second end 212 of the vehicle 202 would be the front end of the vehicle 202 when traveling in the second direction 210, and would be the rear end of the vehicle 202 when traveling in the opposite first direction 208. The exemplary characteristics described thus far may facilitate greater maneuverability in tight spaces or crowded environments, such as, for example, parking lots and urban areas.

[0031] For example, a vehicle, such as exemplary vehicle 202, may travel through environment 200 and be used to collect data to generate training dataset 124 for training sensor data segmentation model 126. For example, the vehicle may include one or more image capture devices 102 and one or more sensors 104 of a different type than the image capture devices 102. For illustrative purposes, the one or more sensors 104 may be one or more LIDAR sensors. Other sensor types are contemplated. In some examples, one or more of the image capture devices 102 and one or more of the sensors 104 may be mounted in close proximity to one another on vehicle 202 to, for example, reduce differences in the viewpoints and / or fields of view of the image capture devices 102 and sensors 104. This may reduce discrepancies between objects detected in environment 200 by image capture devices 102 and sensors 104.

[0032] As the vehicle 202 moves through the environment 200, the image capture devices 102 and sensors 104 simultaneously or substantially simultaneously (within technical capabilities) capture data associated with detected objects 214 (e.g., vehicles, pedestrians, buildings, obstacles, etc.) within the fields of view 216 of one or more of the image capture devices and sensors 104. The captured data may be used to create a training data set 124, for example, as described with respect to FIGS.

[0033] 3 shows an example architecture 300 for implementing one or more of the example processes described herein. For example, the architecture 300 may include one or more computer systems 302 that include various hardware and / or software for implementing aspects of the systems, methods, and apparatus described herein. For example, the computer system(s) 302 may include a sensor module 304 (e.g., a LIDAR module), an image capture module 306, a dimensional transformation module 308, a segmentation module 310, a classification module 312, and a trajectory module 314.

[0034] In some examples, the computer system(s) 302 may be embodied in an autonomous vehicle, such as, for example, the example vehicle 202 shown in FIG. 2. In some examples, the computer system(s) 302 may provide perception and planning functionality for the autonomous vehicle. In general, the computer system(s) 302 may include LIDAR perception, visual (camera) perception, segmentation and classification, tracking and fusion, and prediction / planning.

[0035] The sensor module 304 may include one or more sensors 104, such as, for example, LIDAR sensors, for capturing sensor data 114 for training data, segmentation, and / or classification, as described herein. For example, the sensor module 304 may be configured to combine or synthesize LIDAR data from multiple LIDAR sensors to generate a meta spin of the LIDAR data, which may reference the LIDAR data generated by the multiple LIDAR sensors. In the case of a meta spin of the LIDAR data, the sensor module 304 may be configured to determine a virtual origin of the meta spin data (e.g., by mapping all of the sensor data to a common origin). In some examples, the sensor module 304 may be configured to determine the range between the LIDAR sensor and a point on an object or surface, and in some examples, the sensor module 304 may be configured to determine a surface normal vector for each point captured and / or sensed by the sensor module 304. As a non-limiting example, determining such a surface normal may be performed by calculating the normal to the cross product of vectors pointing in the direction from a point to its two nearest neighbors. As may be understood in the context of the present disclosure, the sensor module 304 may capture data and transmit the data set to the computer system(s) 302 for subsequent processing.

[0036] The image capture module 306 may include one or more image capture devices 102 configured to capture image data 108 for image segmentation and / or classification, for example, as described herein. The image capture module 306 may include any number and type of image capture sensors. For example, the image capture module 306 may include an image capture device, for example, any camera (e.g., an RGB camera, a monochrome camera, an intensity (grayscale) camera, an infrared camera, an ultraviolet camera, a depth camera, a stereo camera, etc.). As can be understood in the context of the present disclosure, the image capture module 306 may capture the image data 108 and may transmit a dataset to the computer system(s) 302 for subsequent processing. For example, the data from the image capture module 306 may be included as one or more channels of a multi-channel image.

[0037] The computer system(s) 302 may include any number and type of other sensors suitable for use in an autonomous vehicle, for example. The various sensors may include, without limitation, one or more LIDAR sensors, one or more RADAR sensors, one or more SONAR sensors, ultrasonic transducers, wheel encoders, microphones, inertial measurement unit(s) (IMU), accelerometers, gyroscopes, magnetometers, temperature sensors, humidity sensors, light sensors, a global positioning system (GPS), an inertial navigation system (INS), etc.

[0038] In some examples, the sensor module 304 and the image capture module 306 may provide one or more datasets to the computer system 302 for combining and / or synthesizing data to provide improved image segmentation and / or classification, generate a training dataset, and / or provide sensor data segmentation and / or classification. In some examples, the computer system(s) 302 may be configured to receive and store sensor datasets as described herein. In some examples, the computer system(s) 302 may be configured to annotate the stored data, which may include detecting, identifying, classifying, segmenting, labeling, etc. the data.

[0039] In some examples, the computer system(s) 302 may further include simulated data generated by the computer simulation algorithm for use in part in testing. In some examples, the simulated data may include any type of simulated data, such as, for example, image data, sensor data (e.g., LIDAR data), GPS data, etc. In some examples, the computer system(s) 302 may be configured to modify, transform, and / or perform transformation operations on the simulated data to validate operations and / or to train models executable by machine learning networks.

[0040] Some examples of the dimensional transformation module 308 may be configured to convert, transform, or map data having a first dimension to data having a second dimension. For example, the dimensional transformation module 308 may be configured to convert one or more three-dimensional datasets into one or more multi-channel two-dimensional images. For example, the dimensional transformation module 308 may be configured to convert and / or map data stored in three dimensions to a different dimensional representation. Such a transformation may include, for example, relating sensor data described in cylindrical or spherical coordinates to planar coordinates. In some examples, such a transformation from cylindrical coordinates to planar coordinates may be envisioned as “unrolling” the cylindrical coordinates. In some examples, the dimensional transformation module 308 may be configured to perform any transformation operation to convert three-dimensional data to two-dimensional data, including, but not limited to, spherical projections (e.g., stereographic and cylindrical projections), Mercator projections, direct polar transformations (e.g., spherical projections or equirectangular projections), etc. The channels of such representations may be associated with range (or distance) and intensity.

[0041] The exemplary segmentation module 310 may be configured to perform segmentation on one or more multi-channel two-dimensional images. For example, the segmentation module 310 may be configured to input one or more multi-channel two-dimensional images to one or more machine learning algorithms. For example, the segmentation module 310 may perform image segmentation to segment objects represented in the data for subsequent image classification. In some examples, hardware and / or software configured to perform segmentation operations on the data may be considered a "segmenter." In some examples, the segmentation module 310 may be configured to operate on any number of channels associated with the two-dimensional image. For example, the segmentation module 310 may receive as input one or more channels, including, but not limited to, a range channel, an x-axis channel, a y-axis channel, a z-axis channel, a surface normal vector channel, a reflectance channel, a time channel, etc. In some examples, the segmentation module 310 may use any of one or more machine learning algorithms to perform the segmentation. For example, the segmentation module 310 may utilize artificial neural networks, such as one or more convolutional neural networks (CNNs), trained to segment multi-channel two-dimensional data. In some examples, the segmentation module 310 may be configured to use an asymmetric kernel that results in pyramiding of the data, as discussed in more detail herein.In some examples, asymmetric strides of the kernel (e.g., having a higher stride in the x direction compared to the y direction) may be applied and selected based, for example, on the number of channels (number of beams), number of points per rotation, etc., so that the data representation at higher abstractions (i.e., higher levels of the pyramid) is more uniform in both the x and y directions.

[0042] An example classification module 312 may be configured to receive the segmented data and identify the type of object represented by the data. For example, the classification module 312 may classify one or more objects including, but not limited to, a car, a building, a pedestrian, a bicycle, a tree, an open space, an occupied space, a road sign, a lane marking, etc. The classification module 312 and / or the segmentation module 310 may include any machine learning algorithm, such as, for example, a neural network, to perform segmentation and classification operations.

[0043] Although described in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning algorithms may include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees ( CART), Iterative Dichotomy 3 (ID3), Chi-Square Automatic Interaction Detection (CHAID), Decision Cutoff, Conditional Decision Tree), Bayesian algorithms (e.g., Naive Bayes, Gaussian Naive Bayes, Multinomial Naive Bayes, Average One Dependence Estimators (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning algorithms These may include algorithms (e.g., perceptron, backpropagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), Adaboost, stacked generalization (blending), gradient boosting machine (GBM), gradient boosted regression tree (GBRT), random forest), support vector machine (SVM)), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0044] In some examples, multiple types of machine learning may be used to provide respective results for each type of machine learning used. In some examples, a confidence score may be associated with each of the results, and the trusted results may be based at least in part on the confidence score associated with the result. For example, the result associated with the highest confidence score may be selected over other results, or results may be combined based on the confidence scores, e.g., based on a statistical method such as a weighted average, etc.

[0045] In some examples, the trajectory module 314 may be configured to receive the segmented and / or classified data and determine a trajectory for the operation of an autonomous vehicle, such as, for example, the exemplary vehicle 202. For example, the trajectory module 314 may be configured to receive segmented image data, segmented sensor data, and / or associated classification information to identify open spaces on a road for the autonomous vehicle to navigate and generate a trajectory for the autonomous vehicle to follow. In some examples, the trajectory module 314 may receive segmented and / or classified objects as discussed herein as input, track the objects, and generate a trajectory based at least in part on the objects.

[0046] 4A is a side view 400 of a vehicle 202 having multiple sensor assemblies mounted on or carried by the exemplary vehicle 202. In some examples, data sets from multiple sensor assemblies may be combined or synthesized to form a metaspin (e.g., LIDAR data representing multiple LIDAR sensors) or may be combined or fused using sensor fusion techniques to improve accuracy or processing for segmentation, classification, prediction, planning, trajectory generation, etc.

[0047] As shown in side view 400, example vehicle 202 may include any number of sensors in any combination or configuration. For example, the example vehicle 202 shown includes sensors 404, 406, and 408. In some examples, sensor 404 may include a RADAR sensor having a vertical field of view illustrated as θ1. Example sensor 406 may include a LIDAR sensor mounted on the roof of vehicle 202 and may have a vertical field of view illustrated as θ2. In some examples, sensor 408 may include an image capture device, such as a camera, having a vertical field of view θ3. Vehicle 202 may include any number and type of sensors and is not limited to the examples shown and described with respect to FIG. 4A .

[0048] Figure 4B is a top view 410 of the example vehicle 202 shown in Figure 4A. As shown in Figure 4B, the example vehicle 202 also includes sensors 412, 414, 416, and 418. For example, sensors 408 and 418 may be co-located or located near each other, but may include distinct sensor types or modalities having different fields of view. In some examples, sensors 412, 414, 416, and 418 may include LIDAR sensors, RADAR sensors, and / or image capture devices.

[0049] 4B, sensor 404 may have a horizontal field of view θ, sensor 406 may have a horizontal field of view θ, sensor 408 may have a horizontal field of view θ, sensor 412 may have a horizontal field of view θ, sensor 414 may have a horizontal field of view θ, sensor 416 may have a horizontal field of view θ, and sensor 418 may have a horizontal field of view θ. The mounting locations and / or fields of view of the sensors may have any number of configurations.

[0050] 5A and 5B show examples of input images and corresponding segmented images. The exemplary input image 500 shown in FIG. 5A may correspond to an exemplary object-segmented image 510. Segmenting the image 500 may determine some meaningful representation of the image. For example, segmenting the input image 500 based on the objects present in the input image 500 may result in the object-segmented image 510 as shown. As shown in FIG. 5A, each different shade of the object-segmented image 510 corresponds to a different object (e.g., a car, a building, a pedestrian, a road marker, etc.). In segmented form as shown, it may be possible to determine the location, classification, and / or presence or absence of a particular object in the input image 500.

[0051] FIG. 5B shows an example of another possible segmentation scheme for segmenting input image 500. In contrast to the segmentation of input image 500 shown in FIG. 5A, input image 500 could be segmented based on “free space,” or potential areas where a vehicle could travel. FIG. 5B shows an example representation of free space segmented image 520. Free space segmented image 520 could provide information (e.g., different shading) about the reliability of a drivable path. As shown in the illustrated example, lighter shading in free space segmented image 520 corresponds to road surfaces that are not blocked by objects, such as other vehicles, buildings, or pedestrians. Such segmentation could be useful for developing trajectories for autonomous vehicles such that the vehicle could avoid collisions with objects in the environment.

[0052] FIG. 6 is a pictorial flow diagram 600 of an exemplary process for capturing image and sensor data. In the illustrated example, three image capture devices 102A-102C and an exemplary LIDAR sensor 602 capture data associated with a scene 604 in which the image capture devices 102A-102C and LIDAR sensor 602 are present. While FIG. 6 depicts three image capture devices 102A-102C and a single LIDAR sensor 602, fewer or more image capture devices are contemplated, and multiple LIDAR sensors are contemplated. The image capture devices 102A-102C can be any type of image capture device configured to capture images representative of an environment, for example, any cameras (e.g., RGB cameras, monochrome cameras, intensity (grayscale) cameras, infrared cameras, ultraviolet cameras, depth cameras, stereo cameras, etc.).

[0053] In the example shown in FIG. 6 , three exemplary image capture devices 102A-102C are shown capturing image data 606 in the form of images 608 corresponding to respective fields of view 610A-610C of image capture devices 102A-102C. For example, as shown schematically, image capture device 102A captures an image associated with field of view 610A, image capture device 102B captures an image associated with field of view 610B, and image capture device 102C captures an image associated with field of view 610C. Although images 608A-608C are shown schematically as being the same in FIG. 6 due to limitations inherent in the drawings, one or more (e.g., each) of images 608A-608C will be slightly different, for example, due to slightly different views and times at which images 608A-608C were captured. In the example shown, each of image capture devices 102A-102C has a slightly different field of view 610 than the other image capture devices 102. In the above example, each image capture device 102A-102C may provide a respective image 608A-608C having a different respective perspective that encompasses a distinct portion of scene 604. For example, image 608A may correspond to three images captured by image capture device 102A at respective image times t1, t2, and t3, image 608B may correspond to three images captured by image capture device 102B at respective image times t1, t2, and t3, and image 608C may correspond to three images captured by image capture device 102C at respective image times t1, t2, and t3. In some examples, the image data 606 captured by each of image capture devices 102A-102C may be fused to generate a more complete representation of scene 604.

[0054] 6, the exemplary LIDAR sensor 602 is configured to capture LIDAR sensor data 612 from a field of view 614 that may be different from the fields of view 610A-610C associated with the image capture devices 102A-102C, for example, as shown in FIG. 6. In the illustrated example, the exemplary field of view 614 of the LIDAR sensor 602 also has a different aspect ratio than the fields of view 610A-610C associated with the image capture devices 102A-102C. For example, the field of view 614 of the LIDAR sensor 602 is wider and narrower (i.e., shorter in height) than the fields of view 610A-610C of the image capture devices 102A-102C, respectively. This may be due to physical parameters of the sensor, such as, for example, beam spread, number of channels, etc. In examples where the field of view 614 of the LIDAR 104 differs from one or more of the fields of view 610A-610C of the image capture devices 102A-102C, the systems described herein may be configured to take into account such differences.

[0055] 7 is a pictorial flow diagram 700 of an exemplary process for segmenting an image and projecting LIDAR sensor data onto the segmented image to create a training data set for training a sensor data segmentation model of a sensor data segmentation network. As shown in FIG. 7, image data 606, including image 608 (see FIG. 6), is sent to an image segmentation network 702 that is configured to automatically segment image 608 to generate a segmented image 704, e.g., via an algorithm and / or machine learning, e.g., as described herein, that may predict a segmented image from an input image.

[0056] As shown in Figure 7, the LIDAR sensor 602 (see Figure 6) generates LIDAR sensor data 612 representing the scene 604 in which the LIDAR sensor 602 is present. At 706, the LIDAR sensor data 612 is projected onto the segmented images 704A-704C. The LIDAR sensor data 612 is projected onto the segmented images 704A-704C such that the LIDAR sensor data 612 and the segmented images 704A-704C are substantially temporally aligned (within the limits of technical capability), for example, as described above with respect to Figure 1. Additionally, the LIDAR sensor data 612 may be projected onto the segmented images 704A-704C such that the LIDAR sensor data 612 associated with objects in the scene 604 is substantially spatially aligned (within the limits of technical capability) with the objects in the segmented images 704A-704C, for example, as described above with respect to Figure 1.

[0057] 7, projecting the LIDAR sensor data 612 onto the segmented images 704A-704C creates a training data set 708 that can be used to train a sensor data segmentation model 710 of a sensor data segmentation network 712 for automatically segmenting the LIDAR sensor data generated by the LIDAR sensor. For example, once trained, such a segmentation model 710 can take raw sensor data (e.g., the LIDAR sensor data 612) as input for future segmentation of objects using the LIDAR sensor. In some examples, the sensor data captured by the LIDAR sensor can be used as additional training data to train the sensor data segmentation model 710 of the sensor data segmentation network 712.

[0058] The sensor data segmentation network 712 may include any type of machine learning model described herein. For example, the sensor data segmentation network 712 may be a CNN. In some examples, the sensor data segmentation network 712 may include multiple machine learning networks. As described herein, multiple types of machine learning may be used to provide respective results for each type of machine learning used. In some examples, a confidence score may be associated with each of the results, and the trusted result may be based at least in part on the confidence score associated with the result. For example, the result associated with the highest confidence score may be selected over other results, or results may be combined based on the confidence scores, e.g., based on a statistical method such as a weighted average, etc.

[0059] In some examples (e.g., examples in which a CNN is used), the sensor data segmentation network 712 may include symmetric and / or asymmetric kernels (e.g., kernels that are squared or non-squared) to segment the sensor data (e.g., the LIDAR sensor data 612) in a manner such that the stride is asymmetric (i.e., the step between kernel applications along the length dimension may be higher than the step in the height dimension). In some such examples, successive applications of kernels using asymmetric strides may result in pyramiding of the data in a manner that increases the receptor field. Such strides may be selected based on physical parameters of the sensor data, such as, for example, without limitation, the number of channels, the width between channels, the number of data returns per rotation, etc. What has just been described is that such pyramiding may facilitate segmenting sensor data (e.g., LIDAR sensor data 612) in a manner that increases the receptive field of the network and may result in more efficient use of data and computing power when making those projections.

[0060] To generate useful outputs, a machine learning network, such as a CNN, must first learn or "train" a set of parameters. Training is accomplished by inputting a dataset into the network, which is associated with expected output values. These expected output values ​​may generally be referred to as "ground truth." For example, the ground truth may include the identification of a particular object in an image, as well as a semantic classification or label associated with the object (e.g., identifying and labeling the object as a car or a building). The accuracy of the network may be based on the amount and / or accuracy of the data provided in the training dataset. As a result, an appropriate dataset for training a network to output segmented sensor data would include segments for which the sensor data is known or previously determined. In some examples, the training dataset may include one or more segmented images representing real-world scenes correlated to one or more sensor datasets representing the real-world scenes, and may be annotated manually or via one or more algorithms configured to segment, detect, classify, and / or label objects in the sensor datasets. As a non-limiting example, raw LIDAR sensor data may be input to the network, and the output may be compared to a corresponding segmented set of LIDAR data (i.e., ground truth) to update internal parameters, as discussed in detail below. By way of example, such ground truth may be LIDAR data with associated segmentation provided by corresponding image data.In some examples, the training data set may include synthetic (e.g., computer-generated) data that includes annotated objects or that has been annotated by a computer algorithm. Training can be performed using offline and / or online data.

[0061] A loss function can be used to adjust the internal parameters of a network during training. The loss function is a function of the expected output (or ground truth) values ​​for a dataset and the values ​​output by the network. The information contained in the loss function can be sent through the network as backpropagation to adjust the internal parameters, thereby tuning the network to provide valid outputs. All else being equal, the more data used to train a network, the more reliable the network can be (e.g., at providing accurate segmentation and / or classification).

[0062] One example of such a loss function that may be used to train a network to segment sensor data such as LIDAR data is a softmax function, but any other function of the input data with expected, or ground truth, segmented data is contemplated. Other exemplary loss functions include, but are not limited to, support vector machine (SVM) loss, hinge loss, etc.

[0063] In some examples, ground truth for the sensor data segmentation network 712 may be provided by the segmented image 704. In some examples, an image may be automatically segmented using the image segmentation network 702 (e.g., a CNN) to obtain the segmented image 704, and sensor data associated with the segmented image 704 may be used as ground truth for training the sensor data segmentation model 710 of the sensor data segmentation network 712 to automatically segment sensor data obtained from a sensor that is not an image capture device. For example, LIDAR sensor data 612 associated with the segmented image 704 may be used as ground truth for LIDAR sensor data obtained from a LIDAR sensor. As shown schematically in FIG. 7 , LIDAR sensor data 612 acquired while image data 606 is captured and segmented may be projected onto a segmented image 704 to obtain a training dataset 708, which may be used to train a sensor data segmentation model 710 run by a sensor data segmentation network 712, which may be a CNN or any type of network described herein. As described above, raw sensor data may be input to the network. The resulting output may then be compared to the sensor data with associated segmentation (e.g., ground truth), such as provided by the segmented image 704. Such comparison may be determined by a loss function (or cost function) to drive variation of the network parameters. In some examples, the loss function may be a cross-entropy softmax loss, focal loss, logistic regression loss, etc.

[0064] In some examples, the sensor data segmentation network 712 may be trained for object segmentation, e.g., by segmenting the input data based on objects contained in the input data. Other segmentation types are also contemplated. For example, the sensor data segmentation network 712 may be trained to identify “free space” or areas in the sensor data that would provide a suitable path for planning a trajectory for an autonomous vehicle, similar to the segmentation of free space shown in FIG. 5B . In such an example, the free space defined in the segmented sensor data may be used to generate a trajectory for maneuvering the autonomous vehicle. Additionally, the sensor data segmentation network 712 may be trained to find occupied space or areas encompassed by sensors that would not provide a suitable path for the trajectory of the autonomous vehicle.

[0065] 8 generally illustrates a pictorial flow diagram 800 of an exemplary process for ingesting sensor data and segmenting the sensor data using an exemplary sensor data segmentation network to generate segmented sensor data for use by a trajectory module configured to generate a vehicle trajectory for an autonomous vehicle. In the illustrated example, an exemplary vehicle 202 autonomously navigates through an environment 802 on a road 804. An exemplary sensor 806 is mounted on the vehicle 202 to detect an object 808 in the environment 802. The sensor 806 could include multiple sensor types, such as, for example, an image capture device and / or other sensor types described herein, including a LIDAR sensor. The sensor 806 senses the object 808 in the environment 802 and generates sensor data 810 in the form of a signal indicative of the object 808.

[0066] In some examples, sensors 806 may include an image capture device and a second sensor type other than an image capture device, such as, for example, a LIDAR sensor. In such examples, one or more LIDAR sensors may be configured to generate LIDAR sensor data 810. The LIDAR sensor data 810 may be communicated to a sensor data segmentation network 812 configured to segment the LIDAR sensor data 810 (e.g., automatically segment the LIDAR sensor data 810) using a sensor data segmentation model 814. In one example, sensor data segmentation model 814 may be configured to identify and / or classify objects 808 and / or open space sensed by the one or more LIDAR sensors to provide segmented sensor data 816 (e.g., segmented LIDAR sensor data). The segmented sensor data 816 may be communicated to a trajectory module 818 configured to generate a vehicle trajectory 820 for vehicle 202 to follow as it navigates through environment 802.

[0067] In various implementations, the parameter values ​​and other data shown herein may be contained in one or more data stores, may be combined with other information not described, or may be partitioned separately into more, fewer, or separate data structures. In some implementations, the data stores may be physically located in one memory or may be distributed among two or more memories.

[0068] Those skilled in the art will understand that the exemplary architecture 300 (see FIG. 3 ) is merely exemplary and is not intended to limit the scope of the present disclosure. In particular, computing systems and devices may include any combination of hardware or software capable of performing the functions shown, including computers, network devices, Internet appliances, tablet computers, PDAs, wireless telephones, pagers, etc. Furthermore, the architecture 300 may be connected to other devices not illustrated, or may instead operate as a stand-alone system. Additionally, the functionality provided by the illustrated components may, in some implementations, be combined in fewer components or distributed among additional components. Similarly, in some implementations, the functionality of some of the illustrated components may not be provided, and / or other additional functionality may be available.

[0069] Additionally, those skilled in the art will understand that while various items are shown as being stored in memory or storage during use, these items, or portions thereof, may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other implementations, some or all of the software components may execute in memory on another device and communicate with the illustrated architecture 300. Furthermore, some or all of the system components or data structures may be stored (e.g., as instructions or structured data) on a non-transitory, computer-accessible medium or portable object readable by an appropriate drive, various examples of which are described above. In some implementations, instructions stored on a computer-accessible medium separate from the architecture 300 may be transmitted to the architecture 300 via a transmission medium or signal, such as an electrical, electromagnetic, or digital signal conveyed over a communications medium such as a wireless link. Various implementations may further include receiving, transmitting, or storing on a computer-accessible medium instructions and / or data implemented in accordance with the foregoing description. Thus, the techniques described herein may be implemented with other control system configurations. Additional information regarding the operation of the modules of the vehicle 202 (see FIG. 2) is discussed below.

[0070] 9 and 10 are flow diagrams of exemplary processes illustrated as a collection of blocks in a logical flow graph, which represent sequences of operations that may be implemented in hardware, software, or a combination thereof. In the software context, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement a process.

[0071] 9 is a flow diagram of an example process 900 for training an example sensor data segmentation model using an example training dataset including segmented images and sensor data. At 902, the example process may include receiving one or more images representing an environment, where the images include image data generated by an image capture device.

[0072] At 904, the example process 900 may further include sending the image data to an image segmentation network configured to segment the image and generate a segmented image. For example, the image may be automatically segmented using a machine learning network, such as a CNN. As described herein, image segmentation by other types of machine learning models or computer vision techniques is contemplated.

[0073] The example process 900 may include, at 906, segmenting the image data to generate a segmented image. For example, the image may be automatically segmented by a machine learning network such as a CNN. For example, the segmentation may divide the image based on detected objects, based on a determination of free space, etc.

[0074] At 908, the example process 900 may further include receiving sensor data generated by a sensor configured to generate data representative of the environment. In some examples, the sensor may be a type of sensor other than an image capture device. For example, the sensor may be a LIDAR sensor, and the sensor data may be LIDAR sensor data. Other sensor types, such as, but not limited to, RADAR(s), ultrasonic transducer(s), etc., and data from the above sensors are contemplated.

[0075] At 910, the example process 900 may further include associating sensor data with the segmented images to create a training dataset (e.g., a set including raw sensor data and an associated set of sensor data with corresponding segmentation information). For example, associating the sensor data with the segmented images may include projecting the sensor data onto the segmented images and associating the segmentation information from the segmented images with the sensor data. In some examples, each of the images is captured at a respective image time, and the sensor data is captured at a sensor time. Associating the sensor data with the segmented images may include selecting image data captured at an image time that minimizes the difference between the image time and the sensor time, and associating the sensor data with the selected image. In some examples, associating the sensor data with the segmented images may include aligning the sensor data with the segmented images. In some examples, aligning the sensor data with the segmented images may include temporally and / or spatially aligning the sensor data with the segmented images. In some examples, associating the sensor data with the segmented image may include identifying image data associated with large depth discontinuities and ignoring image data associated with such discontinuities in the segmented image when associating the sensor data with the segmented image to create the training data set, which may reduce the likelihood of creating errors in the training data set.

[0076] Further, at 912, the example process 900 may include sending the training dataset to a machine learning network configured to run a sensor data segmentation model (i.e., determine a sensor data segmentation using the sensor data segmentation model).

[0077] At 914, the example process 900 may further include training a sensor data segmentation model using the training data set, such that the sensor data segmentation model is configured to segment the sensor data obtained from the sensor. For example, receiving the sensor data generated by the sensor may include receiving sensor data generated by a LIDAR sensor, and training the sensor data segmentation model may include training the sensor data segmentation model using the training data set (e.g., by back-propagating the calculated error in a loss function of the sensor data associated with the segmented image data and the output of the network), such that the sensor data segmentation model is configured to segment the sensor data obtained from the LIDAR.

[0078] 10 is a flow diagram of an example process 1000 for segmenting sensor data using an example sensor data segmentation model. At 1002, the example process may include receiving sensor data from a sensor configured to generate a signal indicative of an environment in which the sensor is located. For example, the sensor may be a type of sensor other than an image capture device. For example, the sensor may be a LIDAR sensor configured to generate LIDAR sensor data representative of the environment in which the sensor is located, although any other sensor type is contemplated.

[0079] At 1004, the example process 1000 may further include segmenting the sensor data received from the sensor using a sensor data segmentation model, such as, for example, a sensor data segmentation model described herein. For example, the sensor data segmentation model may be configured to facilitate identification and / or classification of objects in the environment detected by the sensor. In some examples, the sensor data segmentation model may be configured to facilitate identification of open space in the environment. In some examples, the sensor data segmentation model may be trained using the segmented image data to provide ground truth, for example, as described herein. In some examples, the sensor data segmentation model may be trained with additional data captured by the sensor and / or other sensors of the same and / or different types.

[0080] At 1006, the example process 1000 may further include generating one or more trajectories for the autonomous vehicle based at least in part on the segmented sensor data. For example, a trajectory module of the perception system may be configured to receive the segmented sensor data and generate one or more trajectories based at least in part on the segmented sensor data.

[0081] At 1008, the example process 1000 may further include moving the autonomous vehicle according to one or more of the trajectories.

[0082] 11 is a block diagram of an example computer architecture 1100 for implementing the processes described herein. The example architecture 1100 includes one or more computer systems 1102, which may include storage 1104, one or more processor(s) 1106, and memory 1108 including an operating system 1110. The storage 1104, the processor(s) 1106, the memory 1108, and the operating system 1110 may be communicatively coupled via a communications infrastructure 1112. In some examples, the computer system(s) 1102 may interact with a user or an environment via input / output (I / O) device(s) 1114, as well as one or more other computing devices via one or more networks 1116 using the communications infrastructure 1112. The operating system 1110 may interact with other components to control one or more applications 1118 in the memory 1108.

[0083] In some examples, computer system(s) 1102 may correspond to computer system(s) 302 of Figure 3. Computer system(s) 302 may be implemented with any hardware and / or software to implement modules 304, 306, 308, 310, 312, and 314 that perform the image analysis and / or sensor data analysis discussed herein.

[0084] The systems and methods described herein may be implemented in software or hardware or any combination thereof. The systems and methods described herein may be implemented using one or more computing devices, which may or may not be physically or logically separated from each other. The methods may be performed by components arranged as either on-premise hardware, an on-premise virtual system, or a hosted private instance. In addition, various aspects of the methods described herein may be combined or integrated and merged into other functions.

[0085] In particular, a processor or computer system may be configured to perform some or all of the methods described herein. In some examples, the methods may be partially or fully automated by one or more computers or processors. The systems and methods described herein (or any part(s) or function(ies) thereof) may be implemented using hardware, software, firmware, or a combination thereof, and may be implemented in one or more computer systems or other processing systems. In some examples, the illustrated system elements may be combined into a single hardware device or divided into multiple hardware devices. If multiple hardware devices are used, the hardware devices may be physically located in close proximity or apart from each other. The method examples described and illustrated are illustrative and not intended to be limiting. For example, some or all of the method steps may be combined, rearranged, and / or omitted in separate examples.

[0086] In some examples, the systems and methods described herein may be directed to one or more computer systems capable of performing the functionality described herein. Without limitation, an exemplary computing device may be, for example, a personal computer (PC) system running any operating system, such as, but not limited to, OS X™, iOS™, Linux™, Android™, and Microsoft™ Windows™. However, the systems and methods described herein may not be limited to the platforms mentioned above. Instead, the systems and methods described herein may be implemented on any suitable computer system running any suitable operating system. Additionally, other components of the systems and methods described herein may be implemented using computing devices, such as, but not limited to, computing devices, communication devices, mobile phones, smartphones, telephony devices, telephones, PDAs (personal digital assistants), personal computers (PCs), handheld PCs, interactive televisions (iTVs), digital video recorders (DVDs), client workstations, thin clients, thick clients, proxy servers, network communication servers, remote access devices, client computers, server computers, routers, web servers, data, media, audio, video, telephony or streaming technology servers, etc. Services may be provided on demand, for example, but not limited to, using interactive televisions (iTVs), video-on-demand systems (VODs), and via digital video recorders (DVRs), or other on-demand viewing systems.

[0087] The systems described herein may include one or more processors. The processor(s) may be connected to a communications infrastructure, such as, but not limited to, a communications bus, crossover bar, or network. The process and the processor need not be located in the same physical location. In other words, a process may be executed in one or more geographically remote processors, for example, via a LAN or WAN connection. A computing device may include a display interface that may transfer graphics, text, and other data from the communications infrastructure for display on a display unit.

[0088] In some examples, a computer system may further include, but is not limited to, a main memory, a random access memory (RAM), and a secondary memory. The secondary memory may include, for example, a hard disk drive and / or a removable storage drive, such as a compact disk drive (CD-ROM), etc. The removable storage drive may read from and / or write to the removable storage unit. As will be appreciated, the removable storage unit may include computer-usable storage media storing computer software and / or data. In some examples, a machine-accessible medium may refer to any storage device used to store data accessible by a computer. Examples of machine-accessible media may include, but are not limited to, a magnetic hard disk, a floppy disk, an optical disk similar to at least a compact disk read-only memory (CD-ROM) or a digital versatile disk (DVD), a magnetic tape, and / or a memory chip.

[0089] Further, the processor may include, or be operatively coupled to communicate with, one or more data storage devices for storing data. Such data storage devices may include, by way of non-limiting example, magnetic disks (including internal hard disks and removable disks), magneto-optical disks, optical disks, read-only memory, random access memory, and / or flash storage. Furthermore, storage devices suitable for tangibly embodying computer program instructions and data may include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, application-specific integrated circuits (ASICs).

[0090] The processing system may be in communication with a computerized data storage system. The data storage system may include non-relational or relational data stores, such as MySQL™ or other relational databases. Other physical and logical database types may be used. The data store may be a database server, whether relational or not, such as Microsoft SQL Server™, Oracle™, IBM DB2™, SQLITE™, or any other database software. The data store may store information identifying syntactic tags and any information needed to operate on the syntactic tags. In some examples, the processing system may use object-oriented programming and store data in objects. In the above example, the processing system may use an object-relational mapper (ORM) to store data objects in a relational database. The systems and methods described herein may be implemented using any number of physical data models. In some examples, a relational database management system (RDBMS) may be used. In the above example, a table in the RDBMS may include a column representing a coordinate. The data may be stored in tables in an RDBMS. The tables may have predefined relationships between them. Furthermore, the tables may have attachments associated with the coordinates.

[0091] In some examples, secondary memory may include other similar devices that allow computer programs or other instructions to be loaded into a computer system. For example, such devices may include removable storage units and interfaces. Examples of the above may include program cartridges and cartridge interfaces (such as, for example, but not limited to, those found in video game devices), removable memory chips (such as, for example, but not limited to, erasable programmable read-only memories (EPROMs) or programmable read-only memories (PROMs) and associated sockets), other removable storage units, and interfaces that may allow software and data to be transferred from the removable storage units to the computer system.

[0092] Additionally, a computing system may include input devices such as, but not limited to, a voice input device such as a microphone, a touchscreen, a gesture recognition device such as a camera, other natural user interfaces, a mouse or other pointing device such as a digitizer, and a keyboard or other data entry device. Additionally, a computer system may include output devices such as, but not limited to, a display and a display interface. A computing system may include input / output (I / O) devices such as, but not limited to, a communications interface, cables, and communications paths. The aforementioned devices may include, but are not limited to, a network interface card and a modem. The communications interface(s) may enable software and data to be transferred between the computer system and one or more external devices.

[0093] In some examples, the computer system may be operably coupled to an automotive system. Such automotive systems may be either manually operated, semi-autonomous, or fully autonomous. In the above examples, the input and output devices may include one or more image capture devices, controllers, microcontrollers, and / or other processors for controlling automotive functions such as, but not limited to, acceleration, braking, and steering. Furthermore, the communication infrastructure in the above examples may include a controller area network (CAN) bus.

[0094] In some examples, the computer system may be operatively coupled to any machine vision-based system. For example, such machine vision-based systems may include, but are not limited to, manually operated, semi-autonomous, or fully autonomous industrial or agricultural robots, domestic robots, inspection systems, security systems, etc. For example, the examples described herein are not limited to one particular context and may be applicable to any application that utilizes machine vision.

[0095] Some examples may be implemented in the context of one or more computer networks. The networks may include private networks or public networks (e.g., the Internet), or a combination of both. The networks may include hardware, software, or a combination of both.

[0096] From a telecommunications-oriented perspective, a network may be described as a set of hardware nodes where one or more processes (hardware, software, or a combination thereof) are interconnected by communication facilities operating at each node. The processes may interact and exchange information with each other over communication paths between them, using communication paths between the processes. Appropriate communication protocols may be used on these paths.

[0097] An exemplary computer and / or telecommunications network environment may include nodes, which may include hardware, software, or a combination of hardware and software. The nodes may be interconnected via a communications network. Each node may include one or more processes executable by a processor incorporated in the node. For example, a single process may be run by multiple processors, or multiple processes may be run by a single processor. Additionally, each node may provide an interface point between the network and the outside world and may incorporate a collection of subnetworks.

[0098] In some examples, processes may communicate with each other through inter-process communication paths that support communication through any communication protocol. The paths may function serially or in parallel, continuously or intermittently. The paths may use any of the communication standards, protocols, or technologies described herein with respect to communication networks, in addition to the standard parallel instruction sets used by many computers.

[0099] A node may include any entity capable of performing a processing function. Examples of such nodes that may be used with the examples herein may include computers (e.g., personal computers, workstations, servers, or mainframes), handheld wireless and wired devices (e.g., personal digital assistants (PDAs), modem mobile phones with processing capabilities, wireless email devices including, but not limited to, BlackBerry™ devices), document processing devices (e.g., scanners, printers, facsimile machines, or multifunction document machines), or composite entities (e.g., local or wide area networks) connected to a collection of processors, as described. For example, a node itself may be a wide area network (WAN), a local area network (LAN), a private network (e.g., a virtual private network (VPN), or a collection of networks).

[0100] Communication between the nodes may be enabled by a communication network. The nodes may be continuously or intermittently connected to the communication network. By way of example, the communication network may be a digital communications infrastructure providing adequate bandwidth and information security.

[0101] The communications network may include wired communications capabilities, wireless communications capabilities, or a combination of both, at any frequency, using any type of standard, protocol, or technology. Additionally, in some examples, the communications network may be a private network (e.g., a VPN) or a public network (e.g., the Internet).

[0102] A non-exhaustive list of exemplary wireless protocols and technologies used by the communications network may include Bluetooth™, General Packet Radio Service (GPRS), Cellular Digital Packet Data (CDPD), Mobile Solutions Platform (MSP), Multimedia Messaging (MMS), Wireless Application Protocol (WAP), Code Division Multiple Access (CDMA), Short Message Service (SMS), Wireless Markup Language (WML), Handheld Device Markup Language (HDML), BREW (binary runtime environment for wireless), Radio Access Network (RAN), and Packet-Switched Core Network (PS-CN). Additionally, various generations of wireless technologies are included. An exemplary, non-exhaustive list of primarily wired protocols and technologies used by communications networks includes Asynchronous Transfer Mode (ATM), Enhanced Interior Gateway Routing Protocol (EIGRP), Frame Relay (FR), High-Level Data Link Control (HDLC), Internet Control Message Protocol (ICMP), Interior Gateway Routing Protocol (IGRP), Internetwork Packet Exchange (IPX), ISDN, point-to-point protocol (PPP), transmission control protocol / internet protocol (TCP / IP), Routing Information Protocol (RIP), and User Datagram Protocol (UDP). Any other known or anticipated wireless or wired protocols and technologies could also be used.

[0103] Examples disclosed herein may include apparatus for performing the operations described herein, which may be specially constructed for the desired purposes or may comprise a general-purpose device selectively activated or reconfigured by a program stored on the device.

[0104] Some examples may be embodied in machine-executable instructions. The instructions may be used to cause a processing device, such as a general-purpose or special-purpose processor programmed with the instructions, to perform steps of the processes described herein. Alternatively, the steps described herein may be performed by specific hardware components that contain hardwired logic for performing the steps, or by any combination of programmed computer components and custom hardware components. For example, the systems and processes described herein may be provided as a computer program product, as outlined above. Some examples may include a machine-readable medium having instructions stored thereon. The instructions may be used to program any one or more processors (or other electronic devices) to perform a process or method according to the examples described herein. Additionally, some examples may be downloaded and stored on a computer program product. In the above example, the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communications link (e.g., a modem or network connection) via data signals embodied in a carrier wave or other propagation medium, which may ultimately be stored on a computer system for subsequent execution.

[0105] The process may be implemented in a computer program product accessible from a computer-usable or computer-readable storage medium that provides program code for use by or in connection with a computer or any instruction execution system. The computer-usable or computer-readable storage medium may be any apparatus that can contain or store a program for use by or in connection with a computer or instruction execution system, apparatus, or device.

[0106] A data processing system suitable for storing and / or executing corresponding program code may include at least one processor coupled directly or indirectly to computerized data storage devices, such as memory elements. Input / output (I / O) devices, including but not limited to a keyboard, display, pointing device, etc., may be coupled to the system. Additionally, network adapters may be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. To provide for user interaction, features may be implemented on a computer having a display device, such as an LCD (liquid crystal display) or another type of monitor for displaying information to a user, and input devices, such as a keyboard and a mouse or trackball, through which a user may provide input to the computer.

[0107] A computer program may be a set of instructions that can be used directly or indirectly in a computer. The systems and methods described herein may be implemented using a programming language, such as, for example, CUDA, OpenCL, Flash™, JAVA™, C++, C, C#, Python, Visual Basic™, JavaScript™, PHP, XML, HTML, etc., or a combination of programming languages, including compiled or interpreted languages, and may be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. Software may include, but is not limited to, firmware, resident software, microcode, etc. For example, protocols such as SOAP / HTTP may be used when implementing an interface between programming modules. The components and functionality described herein may be implemented on any desktop operating system running in a virtualized or non-virtualized environment using any programming language suitable for software development, including, but not limited to, various versions of Microsoft Windows™, Apple™ Mac™, iOS™, Unix™ / X-Windows™, Linux™, etc. The system may be implemented using, for example, a web application framework such as Ruby on Rails.

[0108] Processors suitable for executing a program of instructions include, but are not limited to, general-purpose and special-purpose microprocessors, and the sole processor or one of multiple processors or cores of any kind of computer. A processor may receive and store instructions and data from a computerized data storage device, such as, for example, read-only memory, random-access memory, both, or any combination of the data storage devices described herein. A processor may include any processing or control circuitry that operates to control the operation and performance of an electronic device.

[0109] The systems, modules, and methods described herein may be implemented using any combination of software or hardware elements. The systems, modules, and methods described herein may be implemented using one or more virtual machines operating alone or in combination with one another. Any applicable virtualization solution may be used to encapsulate a physical computing machine platform into a virtual machine that executes under the control of virtualization software running on a hardware computing platform or host. A virtual machine may have both virtual system hardware and guest operating system software.

[0110] The systems and methods described herein may be implemented in a computer system that includes back-end components such as, for example, a data server, or includes middleware components such as, for example, an application server or an Internet server, or includes front-end components such as, for example, a client computer having a graphical user interface or an Internet browser, or any combination thereof. The components of the system may be connected by any form or medium of digital data communication, such as, for example, a communications network. Examples of communications networks include, for example, a LAN, a WAN, and the computers and networks forming the Internet.

[0111] One or more embodiments of the present disclosure may be practiced in conjunction with other computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Moreover, the systems and methods described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a network.

[0112] It should be understood that the subject matter presented herein may be implemented as a computer process, a computer-controlled apparatus, a computing system, or as an article of manufacture such as, for example, a computer-readable storage medium. While the subject matter described herein is presented in the general context of program modules executing on one or more computing devices, those skilled in the art will appreciate that other implementations may be made in combination with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types.

[0113] Furthermore, those skilled in the art will understand that aspects of the subject matter described herein may be implemented on or in conjunction with other computer system configurations other than those described herein, including multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, handheld computers, mobile telephone devices, tablet computing devices, special purpose hardware devices, network appliances, and the like.

[0114] Based on the foregoing, it should be understood that technology for operating systems and implementing processes has been presented herein. Furthermore, while the subject matter presented herein has been described in terms specific to structural features of computers, methodological acts, and computer-readable media, it is understood that the inventions defined in the appended claims are not necessarily limited to the specific features, acts, or media described herein. Rather, those specific features, acts, and media are disclosed as exemplary forms of implementing the subject matter recited in the claims.

[0115] The above subject matter is provided by way of example only and should not be construed as limiting. Furthermore, the claimed subject matter is not limited to implementations that solve any and all disadvantages noted in any part of this disclosure. Various modifications and changes may be made to the subject matter described herein without following the examples and applications illustrated and described, without departing from the spirit and scope of the invention, as set forth in the following claims.

[0116] Example clauses A. one or more processors; communicatively coupled to one or more processors, by the one or more processors; receiving a plurality of images representing an environment, the plurality of images including image data generated by one or more image capture devices; sending the image data to an image segmentation network configured to segment the plurality of images and generate segmented images; receiving first sensor data associated with the environment, the first sensor data being received from a light detection and ranging (LIDAR) sensor; Associating the first sensor data with the segmented images to create a training data set; submitting the training dataset to a machine learning network configured to run a sensor data segmentation model; training a sensor data segmentation model using the training dataset, such that the sensor data segmentation model is configured to segment second sensor data received from the LIDAR sensor; one or more computer-readable storage media storing executable instructions; An exemplary system including:

[0117] B. The instructions are executed by one or more processors projecting at least a first portion of the first sensor data onto a first one of the segmented images; Projecting at least a second portion of the first sensor data onto a second one of the segmented images. The system of Example A is further viable as follows.

[0118] C. A first image of the plurality of images is captured at a first image time; a second image of the plurality of images is captured at a second image time; a first portion of the first sensor data associated with a first sensor time; a second portion of the first sensor data associated with a second sensor time; The instructions are executed by one or more processors: determining a first time difference between a first image time and a first sensor time; determining a second time difference between the second image time and the first sensor time; determining that the first time difference is less than the second time difference; Associating the first image with the first portion of the first sensor data based at least in part on the first time difference being less than the second time difference. The system of Example A or Example B is further operable as follows. D. The system of any one of Examples A to C, wherein the instructions are further executable by the one or more processors to receive second sensor data from the first sensor and segment the second sensor data based at least in part on the trained sensor data segmentation model.

[0119] E. The sensor data segmentation model includes one or more kernels, the one or more kernels associated with asymmetric strides; The system of any one of Examples A to D, wherein training the sensor data segmentation model includes calculating a loss function, the loss function based at least in part on an output of the sensor data segmentation model and the training dataset, and the loss function includes one or more of a cross-entropy softmax loss, a focal loss, or a logistic regression loss.

[0120] F. The instructions are executed by one or more processors: Identifying discontinuities in sensor data; Remove sensor data associated with discontinuities The system of any one of Examples A to E may be further implemented as follows:

[0121] G. Instructions are executed by one or more processors Receives sensor data from the LIDAR sensor, Use the sensor data segmentation model to segment the sensor data received from the LIDAR sensor and generate segmented sensor data The system of any one of Examples A to F is further operable as follows:

[0122] H. The system of any one of Examples A through G, wherein the instructions are further executable by the one or more processors to generate a trajectory for the autonomous vehicle based at least in part on the segmented sensor data.

[0123] I. receiving a plurality of images representing an environment, the plurality of images including image data generated by an image capture device; sending the image data to an image segmentation network configured to segment the plurality of images and generate segmented images; receiving first sensor data generated by a first sensor, the first sensor data including data representative of an environment; Segmenting the image data to generate a segmented image; Associating first sensor data with the segmented image to create a training data set; submitting the training dataset to a machine learning network configured to run a sensor data segmentation model; training a sensor data segmentation model using the training dataset, such that the sensor data segmentation model is configured to segment second sensor data acquired from the first sensor; An exemplary method comprising:

[0124] J. The method of Example I, wherein receiving first sensor data generated by a first sensor includes receiving sensor data generated by a light detection and ranging (LIDAR) sensor, and training the sensor data segmentation model includes training the sensor data segmentation model using the training dataset, such that the sensor data segmentation model is configured to segment second sensor data obtained from the LIDAR sensor.

[0125] K. The method of Example I or Example J, further comprising projecting at least a first portion of the first sensor data onto a first segmented image of the segmented images.

[0126] L. a first image of the plurality of images is captured at a first image time; a second image of the plurality of images is captured at a second image time; a first portion of the first sensor data associated with a first sensor time; a second portion of the first sensor data associated with a second sensor time; How to do it determining a first time difference between a first image time and a first sensor time; determining a second time difference between the second image time and the first sensor time; determining that the first time difference is less than the second time difference; associating the first image with the first portion of the first sensor data based at least in part on the first time difference being less than the second time difference; The method of any one of Examples I to K further comprising:

[0127] M. The method of any one of Examples I to L, further including receiving second sensor data from the first sensor and segmenting the second sensor data based at least in part on the trained sensor data segmentation model.

[0128] N. The sensor data segmentation model includes one or more kernels, the one or more kernels associated with asymmetric strides; The method of any one of Examples I to M, wherein training the sensor data segmentation model includes calculating a loss function, the loss function based at least in part on an output of the sensor data segmentation model and the training dataset, and the loss function includes one or more of a cross-entropy softmax loss, a focal loss, or a logistic regression loss.

[0129] O. identifying discontinuities in first sensor data; removing the first sensor data associated with the discontinuity; The method of any one of Examples I to N further comprising:

[0130] P. receiving second sensor data from the sensor; Segmenting second sensor data received from the sensor using the sensor data segmentation model; The method of any one of Examples I to O further comprising:

[0131] Q. The method of any one of Examples I through P, further including generating a trajectory for the autonomous vehicle based at least in part on the segmented second sensor data.

[0132] R. When executed by a computer, the computer receiving a plurality of images representing an environment, the plurality of images including image data generated by an image capture device; sending the image data to an image segmentation network configured to segment the image and generate a segmented image; receiving first sensor data generated by a first sensor, the first sensor data including data representative of an environment; Segmenting the image to generate a segmented image; Associating the first sensor data with the segmented images to create a training data set; submitting the training dataset to a machine learning network configured to run a sensor data segmentation model; training a sensor data segmentation model using the training dataset, such that the sensor data segmentation model is configured to segment second sensor data obtained from the sensor; An exemplary computer-readable medium having stored thereon computer-executable instructions for causing S. The computer-readable storage medium of example R, wherein the first sensor data is generated by a light detection and ranging (LIDAR) sensor, and training the sensor data segmentation model includes training a sensor data segmentation model using the training dataset, such that the sensor data segmentation model is configured to segment second sensor data obtained from the LIDAR sensor.

[0133] T. A first image of the plurality of images is captured at a first image time; a second image of the plurality of images is captured at a second image time; a first portion of the first sensor data associated with a first sensor time; a second portion of the first sensor data associated with a second sensor time; Computer-executable instructions are instructions that a computer can use to: determining a first time difference between a first image time and a first sensor time; determining a second time difference between the second image time and the first sensor time; determining that the first time difference is less than the second time difference; Associating the first image with the first portion of the first sensor data based at least in part on the first time difference being less than the second time difference. A computer-readable recording medium of Example R or Example S. [Explanation of symbols]

[0134] 100 Flow Diagram 102 vehicles 104 Environment 106 devices 108 sensors 110 Image Data 112 Sensor Data 114A Image 114B Image 114C Images 116 LIDAR field of view 118 Image Segmentation Network 120 Segmented Image Data 120A Images 120B Image 120C Images 122 Projection onto segmented image data 124 training datasets 126 Sensor Data Segmentation Model

Claims

1. one or more processors; communicatively connected to the one or more processors; receiving image data including first segmentation information; sending the image data to an image segmentation network configured to automatically segment the image via an algorithm that predicts a segmented image from an image included in the image data to generate the segmented image; receiving sensor data from at least one of a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, or a sound navigation and ranging (SONAR) sensor; associating the first segmentation information with the sensor data as part of a training data set; training a sensor data segmentation model using the training data set to output second segmentation information in response to receiving at least one of additional lidar data, additional radar data, or additional sonar data; one or more non-transitory computer-readable storage media storing instructions executable by the one or more processors to perform operations including: A system comprising:

2. Associating the first segmentation information with the sensor data includes: projecting a portion of the sensor data onto the image data; The system of claim 1 , comprising:

3. The operation is capturing a portion of the image data at a first time; capturing a portion of the sensor data at a second time; determining a time difference between the first time and the second time; determining at least one of temporal alignment information or spatial alignment information associated with the portion of the sensor data and the portion of the image data based at least in part on the time difference; The system of claim 1 further comprising:

4. The operation is receiving second image data including second segmentation information; applying the second segmentation information to the sensor data to generate a training data set; The system of claim 1 further comprising:

5. 2. The system of claim 1, wherein the second segmentation information is output based on a step between the kernels of the sensor data segmentation model along a length dimension that is higher than a step between the kernels of the sensor data segmentation model along a height dimension.

6. The operation is identifying discontinuities associated with the sensor data; Ignoring a portion of the sensor data associated with the discontinuity; and The system of claim 1 further comprising:

7. The sensor data includes LIDAR sensor data received from the LIDAR sensor, and the operation includes: Segmenting the LIDAR sensor data as third segmentation information according to the sensor data segmentation model; storing the third segmentation information; The system of claim 1 further comprising:

8. The operation is transmitting the sensor data segmentation model to an autonomous vehicle, the autonomous vehicle configured to generate a trajectory based at least in part on the sensor data segmentation model; The system of claim 7 further comprising:

9. 1. A method implemented by a computer system, comprising: receiving image data including first segmentation information; sending the image data to an image segmentation network configured to automatically segment the image via an algorithm that predicts a segmented image from an image included in the image data to generate the segmented image; receiving sensor data from at least one of a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, or a sound navigation and ranging (SONAR) sensor; associating the first segmentation information with the sensor data as part of a training data set; training a sensor data segmentation model using the training data set to output second segmentation information in response to receiving at least one of additional lidar data, additional radar data, or additional sonar data; A method comprising:

10. Associating the first segmentation information with the sensor data includes: projecting a portion of the sensor data onto the image data; 10. The method of claim 9, comprising:

11. capturing a portion of the image data at a first time; capturing a portion of the sensor data at a second time; determining a time difference between the first time and the second time; determining at least one of temporal alignment information or spatial alignment information associated with the portion of the sensor data and the portion of the image data based at least in part on the time difference; 10. The method of claim 9, further comprising:

12. receiving second image data including second segmentation information; applying the second segmentation information to the sensor data to generate a training data set; 10. The method of claim 9, further comprising:

13. 10. The method of claim 9, wherein the second segmentation information is output based on a step between the kernels of the sensor data segmentation model along a length dimension that is higher than a step between the kernels of the sensor data segmentation model along a height dimension.

14. identifying discontinuities associated with the sensor data; Ignoring a portion of the sensor data associated with the discontinuity; and 10. The method of claim 9, further comprising:

15. The sensor data includes LIDAR sensor data received from the LIDAR sensor, and the method further comprises: determining third segmentation information based on the LIDAR sensor data by the sensor data segmentation model; and storing the third segmentation information; 10. The method of claim 9, further comprising:

16. 16. The method of claim 15, further comprising generating a trajectory for an autonomous vehicle based at least in part on the third segmentation information.

17. receiving additional sensor data including the at least one of the additional lidar data, the additional radar data, or the additional sonar data, the additional sensor data being described in cylindrical or spherical coordinates; Associating plane coordinates with the additional sensor data; segmenting the additional sensor data as segmented sensor data according to the sensor data segmentation model; outputting the segmented sensor data according to a sensor data segmentation model; 10. The method of claim 9, further comprising:

18. When executed by one or more processors, the one or more processors: receiving image data including first segmentation information; sending the image data to an image segmentation network configured to automatically segment the image via an algorithm that predicts a segmented image from an image included in the image data to generate the segmented image; receiving sensor data from at least one of a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, or a sound navigation and ranging (SONAR) sensor; associating the first segmentation information with the sensor data as part of a training data set; training a sensor data segmentation model using the training data set to output second segmentation information in response to receiving at least one of additional lidar data, additional radar data, or additional sonar data; One or more non-transitory computer-readable storage media having computer-executable instructions stored thereon that cause the device to perform operations including:

19. The sensor data includes LIDAR sensor data received from the LIDAR sensor, and the operation includes: determining third segmentation information based on the LIDAR sensor data by the sensor data segmentation model; and storing the third segmentation information; 20. The one or more non-transitory computer-readable media of claim 18, further comprising:

20. The operation is capturing a portion of the image data at a first time; capturing a portion of the sensor data at a second time; determining a time difference between the first time and the second time; determining at least one of temporal alignment information or spatial alignment information associated with the portion of the sensor data and the portion of the image data based at least in part on the time difference; 20. The one or more non-transitory computer-readable media of claim 18, further comprising:

Images