Detecting errors in sensor data

Abstract

To provide a method and system for detecting errors in sensor data representing an object in an environment.SOLUTION: The method includes: receiving a first signal including data representing an environment, from a first sensor; receiving a second signal including data representing the environment, from a second sensor; determining a group of objects at least partially based on the received signals; and identifying an error associated with data included in the first signal and / or the second signal.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to sensor data, and more particularly to detecting errors in sensor data. [Background technology]

[0002] This patent application claims the benefit of and priority to U.S. Patent Application No. 15 / 944,240, entitled "Detecting Errors in Sensor Data," filed April 3, 2018, which is incorporated herein by reference in its entirety.

[0003] Sensors may be used to generate sensor data indicative of objects in an environment. For example, multiple sensors may provide sensor data to a perception system configured to facilitate autonomous operation of a machine. The perception system may identify a group of objects present in the environment based on the sensor data. However, environmental conditions, damage, miscalibration, and other factors may hinder the effectiveness of one or more of the sensors. Additionally, inappropriate training data for a particular sensor modality may result in the sensor modality's failure to detect objects located in the environment. In particular, the situation just described may be problematic when, for example, an autonomous vehicle operating without a driver relies at least in part on data generated by sensors for proper operation. If one or more sensors provide sensor data that differs from and / or is otherwise inconsistent with the group of objects identified by the perception system, the autonomous vehicle may take action based on inaccurate information related to the environment through which it is traveling. This may hinder operation of the vehicle or machine.

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The same reference number in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0005] [Figure 1] 1 is a pictorial flow diagram illustrating an example process for identifying an object or group of objects in an environment and determining that data received from one or more sensor modalities contains an error associated with the object or group of objects. For example, the process may include determining that an object is not in or is misclassified as being in a group of objects associated with data generated by an individual sensor. [Figure 2] 1 illustrates an exemplary environment through which an exemplary vehicle is traveling and capturing sensor data. [Figure 3] 1 illustrates an exemplary system for identifying objects and initiating a response. [Figure 4A] 1 illustrates a side view of an exemplary vehicle having multiple sensor types. [Figure 4B] 4B illustrates a top view of the exemplary vehicle shown in FIG. 4A. [Figure 5] 10 is a pictorial flow diagram illustrating another example process for identifying an object or group of objects in an environment and determining that data received from one or more sensor modalities contains an error associated with the object or group of objects. The example process may include determining that an object is not in or is misclassified as being in a group of objects associated with data generated by an individual sensor. [Figure 6] 1 is a flowchart illustrating an exemplary process for identifying and initiating a response to detectable objects in an environment. [Figure 7]10 is a flowchart illustrating another exemplary process for identifying and initiating a response to detectable objects in an environment. DETAILED DESCRIPTION OF THE INVENTION

[0006] As discussed above, autonomous vehicles and other machines rely on multiple sensors that provide input to a perception system that detects objects in the environment surrounding the autonomous vehicle or machine. Failure of one or more sensors that detect objects in the environment can disrupt operation of the vehicle or machine and potentially cause an unsafe operating condition. However, it may not be immediately apparent that sensor data from an individual sensor modality is not operating properly or that the data output from the sensor modality is anomalous.

[0007] This application describes techniques for identifying errors associated with individual sensor modalities by using data generated by the individual sensor modalities (e.g., image data, LIDAR (light detection and ranging) sensor data, RADAR (light detection and ranging) data, SONAR (sound navigation and ranging) data, etc.) to identify groups of respective objects and comparing the respective object groups (e.g., processed sensor data) with the output of a perception system. The output of the perception system may include fused data from multiple sensors, which may encompass and / or include groups of objects predicted by the perception system as being present in the environment. The comparison may be used to identify sensors that are malfunctioning, in need of repair, and / or in need of calibration. Additionally, in some examples, the fused sensor data generated by the perception system may be used as ground truth when training the individual sensor modalities, e.g., by a machine learning module, to further improve the performance of the individual sensor modalities.

[0008] In exemplary embodiments, one or more systems of the present disclosure may include multiple sensors disposed in a vehicle, such as an autonomous vehicle, and operably coupled to one or more processors and / or remote computing devices. The one or more processors may receive a first signal from the first sensor disposed in the vehicle. The first sensor may include an image capture device, and the first signal may include image data representing a scene captured by the image capture device (i.e., a portion of the environment visible to the image sensor, as may be determined by a field of view, for example), which may be a first scene. For example, the image data may illustrate and / or include a group of first objects detectable in an environment in which the first sensor and / or vehicle is present. For example, the image data may include multiple images captured by the image capture device and showing the environment. Each of the images may include a respective group of objects detectable by the image capture device in the environment. In the above example, the one or more processors may analyze each signal received from the first sensor to identify a respective group of objects indicated by the image data included in each signal.

[0009] Further, the system may include a second sensor disposed on the vehicle, which may include a LIDAR sensor. The one or more processors may be configured to receive a second signal from the LIDAR sensor, which may include LIDAR sensor data. The LIDAR sensor data may be captured simultaneously with the image data described above. In the above example, the LIDAR sensor data may represent a respective second scene (i.e., a portion of the environment visible to the LIDAR sensor) captured by the LIDAR sensor simultaneously with the image data described above. For example, the LIDAR sensor data may indicate a second scene captured by the LIDAR sensor that includes a group of detectable second objects in the environment. In the above example, the one or more processors may analyze each signal received from the LIDAR sensor to identify a respective group of objects indicated by the LIDAR sensor data included in each signal. Subject to the accuracy and / or fidelity of the LIDAR sensor data, for example, an object included in the second group of objects may be identical to an object included in the first group of objects detectable by the image capture device, and at least a portion of the first group of objects and a portion of the second group of objects may be visible together in the image and the LIDAR sensor at the same time.

[0010] Additionally, the exemplary system may include one or more additional sensors (e.g., a RADAR sensor, a SONAR sensor, a depth sensing camera, a time-of-flight sensor, etc.) disposed on the vehicle and configured to detect objects in the vehicle's environment. The one or more additional sensors may output one or more respective signals to one or more processors. For example, the one or more processors may receive a third signal from at least one of the additional sensors, where the third signal may include sensor data from at least one of the additional sensors. The sensor data may be captured simultaneously with the image data and / or LIDAR sensor data described above. In the above example, the sensor data may represent a respective third scene (i.e., a portion of the environment visible to the additional sensor) captured by the additional sensor simultaneously with the image data and / or LIDAR sensor data described above. The third scene may include, for example, substantially identical objects captured by the image capture device and / or LIDAR sensor. For example, the sensor data may indicate a third scene captured by the additional sensor including a group of detectable third objects in the environment. In the above example, the one or more processors may analyze each signal received from the additional sensor to identify a respective group of objects indicated by the sensor data included in each signal. According to the accuracy and / or fidelity of the sensor data received from the additional sensor, at least some of the objects included in the third group of objects may be identical to at least some of the objects included in the first group of objects detectable by the image capture device and / or at least some of the objects included in the second group of objects detectable by the LIDAR sensor.As described in more detail below, one or more example scenes of the present disclosure, such as the first, second, and / or third scenes described above with respect to the image capture device, LIDAR sensor, and additional sensor, respectively, may include a substantially simultaneous representation of the environment in which the respective sensors are located. Thus, one or more example scenes of the present disclosure may include a representation of the environment as detected, captured, sensed, and / or otherwise observed by the respective sensors.

[0011] In any of the examples described herein, one or more processors and / or one or more remote computing devices may identify and / or determine a group of additional objects based at least in part on one or more signals received from the various sensors described above. For example, through one or more data fusion processes, the perception system of the present disclosure may generate fused sensor data representative of an environment. For example, the fused sensor data may include and / or identify a particular group of one or more objects predicted, determined, and / or otherwise indicated by the perception system as being present in the environment based at least in part on the sensor data received from the individual sensor modalities. The one or more processors and / or remote computing devices of the present disclosure may treat the fused sensor data as ground truth for training object recognition and / or for classification processes of individual sensor modalities, at least because the group of objects is determined using information from multiple sources and, as a result, has a relatively high likelihood of accuracy compared to each individual sensor modality.

[0012] In some examples, one or more processors and / or remote computing devices may compare information included in and / or associated with each of the signals received from each sensor modality with the fused sensor data (e.g., with a specific group of objects included in the fused sensor data) to identify any perceived errors in the group of objects represented by the respective sensor signals. For example, one or more processors and / or remote computing devices may correlate the output of each sensor modality with a specific object and / or a specific location. Using the data association and / or object characterization techniques described above, the output of each sensor can be compared. Through the comparison, the one or more processors and / or remote computing devices may identify one or more objects included in the fused sensor data that are not in or are misclassified as being in at least one of the groups of objects associated with the respective sensor signals. In other examples, additional and / or different errors in the respective sensor signals may be identified, which may include, among other things, differences in posture, differences in posture uncertainty, differences in object size, differences in object location, differences in object extent, etc. In the above examples, upon identifying and / or otherwise determining an error associated with the data contained in one or more of the respective sensor signals (e.g., upon determining that a particular object contained in the fused sensor data is not in or has been misclassified as being in at least one of the groups of objects represented in the respective sensor signals), a response system of the present disclosure may initiate a response intended to correct the just-mentioned error and / or mitigate the effects of said error throughout operation of the vehicle.The above example responses may be initiated in embodiments in which the processing described herein occurs throughout vehicle operation and / or in embodiments in which the processing occurs offline (e.g., during training, programming, and / or machine learning training). Additionally or alternatively, in any of the examples described herein, one or more signals received from the various sensors described above and / or any of the groups of objects described herein may be provided to a machine learning system of the present disclosure. Providing the above information to a machine learning system may help improve the accuracy of training data utilized by the system, thereby generally further enhancing system performance and / or vehicle operation. The above information may be provided to a machine learning system in embodiments in which the processing described herein occurs throughout vehicle operation and / or in embodiments in which the processing occurs offline (e.g., during training, programming, and / or machine learning training).

[0013] The techniques and systems described herein may be implemented in many ways. Example implementations are provided below with reference to the drawings.

[0014] FIG. 1 is a pictorial flow diagram 100 of an example process for comparing individual sensor data with fused sensor data and initiating a response based, at least in part, on determining an error associated with the data included in one or more of the respective sensor signals (e.g., determining that at least one object included in the fused sensor data is not in or is misclassified as part of a group of objects determined, at least in part, based on data received from the individual sensors). As shown in FIG. 1 , the example sensors may include an image capture device 102, a LIDAR sensor 104, and / or one or more other sensors 106, which may be any type of sensor configured to generate a signal indicative of an environment 108. For example, the environment 108 may include one or more objects, and the sensors 106 may include, for example, a RADAR sensor, an ultrasonic transducer such as a SONAR sensor, a time-of-flight sensor, or other sensors as well. The image capture device 102, the LIDAR sensor 104, and the sensors 106 may be connected to a vehicle, and the vehicle may capture sensor data while traveling through the environment 108. For example, the vehicle may be an autonomous vehicle, such as the exemplary vehicle described herein with respect to at least FIG. 2. The image capture device 102 may be any type of image capture device configured to capture images representative of the environment 108, such as, for example, one or more cameras (e.g., an RGB-camera, a monochrome camera, an intensity (grayscale) camera, an infrared camera, an ultraviolet camera, a depth camera, a stereo camera, etc.). The LIDAR sensor 104 may be any type of LIDAR sensor, and some examples may include only a single type of sensor or any combination of different types of sensors configured to generate data representative of the environment. Furthermore, while depicted as a single sensor for illustrative purposes, any number of image capture device(s) 102, LIDAR sensor(s) 104, and / or other sensor(s) 106 are contemplated.

[0015] 1 , image capture device 102 is depicted as capturing image data 110. The image data 110 may be provided by the image capture device in the form of one or more signals to one or more processors and / or other system components of the present disclosure. For example, the signals may include image data 110 representing environment 108. For example, image data 110 may include one or more images that illustrate, show, and / or otherwise represent environment 108. In some examples, image data 110 may include one or more images captured by image capture device 102, and image data 110 may illustrate, show, and / or otherwise represent scene A including a group of corresponding objects with respect to environment 108 detected by image capture device 102 at a particular time. As described below, in some examples, image data 110 may be fused with additional sensor data received from other sensors (e.g., LIDAR sensor 104, additional sensor 106, etc.) to generate a more complete or accurate representation of environment 108.

[0016] The example LIDAR sensor 104 shown in FIG. 1 is depicted as capturing LIDAR sensor data 112. The LIDAR sensor data 112 may be provided by the LIDAR sensor 104 in the form of one or more signals to one or more processors and / or other system components of the present disclosure. For example, the signals may include LIDAR sensor data 112 representative of the environment 108. The LIDAR sensor data 112 may illustrate, show, and / or otherwise represent a scene B including a respective group of objects as detected by the LIDAR sensor 104 at a particular time. In an example where the LIDAR sensor data 112 is captured simultaneously with image data 110, the group of objects included in and / or represented in scene B may be substantially identical to the group of objects included in scene A associated with the image capture device. In the above example, each data acquisition of the LIDAR sensor 104 may result in LIDAR sensor data 112 representative of a respective scene.

[0017] In some examples, the LIDAR sensor data 112 may correspond to multiple data acquisitions of the LIDAR sensor 104 over time, and the LIDAR sensor data 112 may be fused with data from other sensors (e.g., image data 110, sensor data 114, etc.) to generate a more complete or accurate representation of the environment 108. The LIDAR sensor 104 may be configured with one or more lasers mounted to rotate (e.g., about a substantially vertical axis), thereby causing the laser to perform a 360-degree sweep to capture LIDAR sensor data 112 associated with the environment 108. For example, the LIDAR sensor 104 may include one or more lasers having a light emitter and a light sensor, where the light emitter directs highly focused light at an object or surface that reflects the light to the light sensor through any other light emission and light detection (e.g., flash LIDAR, MEMS LIDAR, solid-state LIDAR, etc.) that determines the expected range. The LIDAR sensor 104 measurements may be represented as three-dimensional LIDAR sensor data having coordinates (e.g., Cartesian coordinates, polar coordinates, etc.) corresponding to positions or distances captured by the LIDAR sensor 104. For example, the three-dimensional LIDAR sensor data may include a three-dimensional map or point cloud and may be represented as multiple vectors emanating from a light emitter and terminating at an object or surface. In some examples, a transformation operation may be used to convert the three-dimensional LIDAR sensor data into multi-channel two-dimensional data. In some examples, the LIDAR sensor data 112 may be automatically segmented; for example, the segmented LIDAR sensor data may be used as input for determining a trajectory for an autonomous vehicle.

[0018] The example sensor 106 shown in FIG. 1 may be configured to capture sensor data 114. The sensor data 114 may be provided by the sensor 106 in the form of one or more signals to one or more processors and / or other system components of the present disclosure. For example, the signals may include sensor data 114 representative of the environment 108. The sensor data 114 may illustrate, show, and / or otherwise represent a scene C including a group of respective objects as detected by the sensor 106 at a particular time. In examples where the sensor data 114 is captured simultaneously with the image data 110, the group of objects included in and / or represented in scene C may be substantially identical to the group of objects included in scene A associated with the image capture device. Similarly, in examples where the sensor data 114 is captured simultaneously with LIDAR sensor data 112, the group of objects included in scene C may be substantially identical to the group of objects included in scene B. In some examples, the sensor data 114 may be fused with additional sensor data received from other sensors (e.g., LIDAR sensor 104, image capture device 102, etc.) to generate a more complete or accurate representation of the environment 108.

[0019] In some examples, the image capture device 102, the LIDAR sensor 104, and / or the sensor 106 may capture different fields of view of the environment 108. As a result, each scene A, B, C may include a respective group of objects in the environment 108 sensed by the respective sensor. For example, the signal generated by the image capture device 102 may include image data 110 representing a first scene A, which may illustrate and / or otherwise include a corresponding first group of objects 128 detected by the image capture device 102 at a particular time, as shown in FIG. 1 . Similarly, the signal generated by the LIDAR sensor 104 may include LIDAR sensor data 112 representing a second scene B, which may include a corresponding second group of objects 130 detected by the LIDAR sensor 104 at a particular time. Similarly, the signals generated by the sensors 106 may include sensor data 114 representing a third scene C, which may include a corresponding third group of objects 132 detected by the sensors 106 at a particular time. As mentioned above, in any of the examples described herein, the image data 110, the LIDAR sensor data 112, and / or the sensor data 114 may be captured by the respective sensors 102, 104, 106 substantially simultaneously. In the above example, scenes A, B, and C may include representations of the environment 108 at substantially the same time. Additionally, in the above example, the first group of objects 128 may include substantially identical objects as the second and third groups of objects 130, 132. Similarly, the second group of objects 130 may include substantially identical objects as the first and third groups of objects 128, 132. Similarly, the third group of objects 132 may include objects that are substantially identical to the first and second groups of objects 128, 130.

[0020] 1 , the example process 100 may include communicating image data 110, LIDAR sensor data 112, and / or sensor data 114 to one or more processors configured to identify one or more objects and / or one or more groups of objects present in the environment 108 based at least in part on the data. In some examples, the one or more processors may include and / or be in communication with a perception system 116. The one or more processors and / or perception system 116 may be configured to receive respective signals from the image capture device 102, the LIDAR sensor 104, and / or the sensor 106. Further, the one or more processors and / or perception system 116 may be configured to identify groups of objects 128, 130, and 132, respectively, based at least in part on the image data 110, the LIDAR sensor data 112, and the sensor data 114.

[0021] Further, one or more processors and / or perception system 116 may be configured to generate fused sensor data 134 based at least in part on the image data 110, the LIDAR sensor data 112, and the sensor data 114. The fused sensor data 134 may encompass and / or include a group of objects 120 predicted, determined, and / or indicated by the perception system 116 as being present in the environment 108 at a particular time associated with the image data 110, the LIDAR sensor data 112, and / or the sensor data 114. An example group of objects 120 may be included in the fused sensor data 134 and / or any other output of the perception system 116 and may include and / or identify one or more objects 122 determined by the perception system 116 as being present in the environment 108. In some examples, the perception system 116 may determine an error associated with data included in one or more respective sensor signals. For example, the perception system 116 may determine whether an object 122 included in the fused sensor data 134 (e.g., included in a particular object group 120) is missing from or misclassified in the group of objects 128 associated with the image data 110 (e.g., determined based on), the group of objects 130 associated with the LIDAR sensor data 112 (e.g., determined based on), and / or the group of objects 132 associated with the sensor data 114 (e.g., determined based on). Additionally or alternatively, in any of the examples described herein, the perception system 116 may determine any of the other errors described above. For example, the errors determined by the perception system 116 may further include, among other things, a difference in posture, a difference in posture uncertainty, a difference in object size, a difference in object location, a difference in object range, etc.In still a further example, perception system 116 may compare image data 110 with LIDAR sensor data 112 and / or sensor data 114 to identify and / or otherwise determine errors or other inconsistencies in image data 110. Additionally or alternatively, perception system 116 may compare LIDAR sensor data 112 with image data 110 and / or sensor data 114 to identify and / or otherwise determine errors or other inconsistencies in LIDAR sensor data 112. Furthermore, perception system 116 may compare sensor data 114 with image data 110 and / or LIDAR sensor data 112 to identify and / or otherwise determine errors or other inconsistencies in sensor data 114.

[0022] In some embodiments, the perception system 116 may utilize one or more algorithms, neural networks, and / or other components to identify each object included in the group of objects 120 and / or parameters of each object (e.g., range, size, orientation, etc., and / or uncertainty associated with each parameter). For example, the perception system 116 may use one or more data association engines, object recognition engines, object classification engines, and / or other components to correlate the output of each sensor modality described herein to a particular object, thereby identifying the object or group of objects. Additionally, the object detection system 118 associated with the perception system 116 may detect and / or otherwise determine a particular location for each respective identified object within the environment 108 at a corresponding time. For example, the object detection system 118 may determine a particular location L of an object 122 included in the group of objects 120, as well as a particular time when the object 122 was / is located at location L. The location L may have coordinates (e.g., Cartesian coordinates, polar coordinates, GPS, etc.) that identify the position of the object 122. In some examples, image data 110 indicative of group 128 of objects may be determined by image capture device 102 at a first time. Additionally, LIDAR sensor data 112 indicative of group 130 of objects may be determined by LIDAR sensor 104 contemporaneously (e.g., at the first time) along with image data 110. In the above example, object detection system 118 may identify object 122 and may determine a particular location L of object 122 corresponding to the first time.

[0023] In some examples, the perception system 116 may also be in communication with the response system 124; in some of the above examples, when the perception system 116 generates fused sensor data 134 including the group of objects 120, one or more signals indicative of the group of objects 120 may be communicated by the perception system 116 to the response system 124. The response system 124 may be configured to initiate a response and / or initiate any other action 126 based at least in part on the fused sensor data 134, the group of objects 120, and / or the one or more signals indicative of the identified object 122. For example, as shown in FIG. 1 , the response system 124 may be configured to communicate an action 126 intended to mitigate the effect of the perception system 116 identifying one or more errors associated with one or more determined groups of objects 128, 130, 132 and / or associated with at least one of the image data 110, the LIDAR sensor data 112, or the sensor data 114. For example, the above operations 126 may mitigate the impact of the perception system 116 identifying an object 122 that is included in the fused sensor data 134 but is not in or is misclassified as part of one or more of the object groups 128, 130, 132. Additionally or alternatively, as described below with respect to at least FIG. 5 , the perception system 116 may also be in communication with a machine learning system, and in some of the above examples, when the perception system 116 identifies one or more errors associated with one or more determined object groups 128, 130, 132 and / or associated with at least one of the image data 110, the LIDAR sensor data 112, or the sensor data 114, the perception system 116 may provide one or more signals indicative of the errors to the machine learning system.For example, in the above embodiments, the perception system 116 may provide signals to the machine learning system indicative of one or more determined object groups 128, 130, 132 and / or including at least one of the image data 110, the LIDAR sensor data 112, or the sensor data 114. The machine learning system may use information contained in the signals to assist in training a sensor modality corresponding to the determined error.

[0024] As described above, each of the image capture device 102, the LIDAR sensor 104, and the sensor 106 may generate a respective signal and provide the signal to the perception system 116. In some examples, the signal may be generated substantially continuously by the image capture device 102, the LIDAR sensor 104, and / or the sensor 106. In other examples, the signal may be generated at regular or irregular time intervals by the image capture device 102, the LIDAR sensor 104, and / or the sensor 106.

[0025] As further described above, a first signal generated by the image capture device 102 may include image data 110 representing a scene A indicative of the environment 108. The scene A and / or image data 110 included in the first signal may include and / or be indicative of a first group of objects 128 detected in the environment 108 in which the image capture device 102 is present (e.g., as the perception system 116 may determine upon processing the first signal). Similarly, a second signal generated by the LIDAR sensor 104 may include image data 110 representing a scene B indicative of the environment 108. The scene B and / or LIDAR sensor data 112 included in the second signal may include and / or be indicative of a second group of objects 130 detected in the environment 108 (e.g., as the perception system 116 may determine upon processing the second signal). Similarly, a third signal generated by the additional sensor 106 may include sensor data 114 representing a scene C indicative of the environment 108. The scene C and / or sensor data 114 included in the third signal may include and / or indicate a third group of objects 132 detected in the environment 108 (e.g., the perception system 116 may process the third signal to determine). In the above example, the perception system 116 may further generate fused sensor data 134 based on the first signal, the second signal, and / or the third signal using one or more example fusion techniques described herein. Furthermore, the perception system 116 may determine a group of objects 120 and / or identify one or more specific objects 122 present in the environment 108 based at least in part on the image data 110, the LIDAR sensor data 112, and / or the sensor data 114 included in one or more received signals. Furthermore, the perception system 116 may determine that at least one object 122 included in the fused sensor data 134 is not in or is misclassified as being in at least one of the first group of objects 128, the second group of objects 130, or the third group of objects 132.The determination may be communicated to a response system 124, which may in turn initiate a response and / or any other action 126 to correct the discrepancy. Additionally, the determination may also be communicated to a machine learning system described herein. It is understood that the determination and / or corresponding information may be communicated to the response system 124 in embodiments in which the processing described herein is performed online (e.g., by one or more processors disposed in the vehicle during operation of the vehicle) and / or in embodiments in which the processing is performed offline (e.g., by one or more remote computing devices during training, programming, and / or machine learning exercises associated with the vehicle). Similarly, the determination and / or corresponding information may be provided to a machine learning system in embodiments in which the processing described herein is performed online and / or in embodiments in which the processing is performed offline.

[0026] In any of the examples described herein, the various sensor modalities may each have a corresponding level of confidence associated therewith. For example, the signal and / or image data 110 provided by the image capture device 102 may be characterized by a relatively high first confidence level (or correspondingly low uncertainty). For example, the signal and / or image data 110 may be characterized by a first confidence level of between about 90% and about 98%. Additionally, the signal and / or LIDAR sensor data 112 provided by the LIDAR sensor 104 may have a second confidence level that is lower than the first confidence level associated with the signal and / or image data 110 provided by the image capture device 102. For example, the signal and / or LIDAR sensor data 112 may be characterized by a second confidence level of between about 85% and about 90%. The aforementioned confidence levels are merely examples, and in further embodiments, image data 110, LIDAR sensor data 112, and / or sensor data 114 may be characterized by greater or lesser confidence levels than the above-identified confidence levels. In an example where a particular object is determined by perception system 116 as being included in group 128 of objects associated with image capture device 102 but is determined by perception system 116 as being absent or misclassified in group 130 of objects associated with LIDAR sensor 104, perception system 116 may include the particular object in fused sensor data 134 (e.g., may include the particular object in group 120 of objects) based at least in part on at least one of the first confidence level or the second confidence level. As noted above, in additional examples, perception system 116 may determine one or more additional and / or different errors associated with data included in one or more respective sensor signals.

[0027] In still a further example, statistical information may be stored in memory accessible by the perception system 116 and / or otherwise associated with one or more image capture devices 102, LIDAR sensors 104, or sensors 106. For example, such statistical information may include aggregated measured data associated with each one of the sensors and indicative of the accuracy and / or consistency of the data captured by the sensors. For example, the LIDAR sensor 104 may capture LIDAR sensor data 112 that is approximately 95% accurate when sensing a particular object from a distance of less than approximately 10 meters. In the above example, the perception system 116 may identify the particular object using the sensor data 112 with 95% accuracy. Additionally or alternatively, the LIDAR sensor 104 may capture LIDAR sensor data 112 that is approximately 90% accurate when sensing a particular object from a distance greater than approximately 10 meters and closer than approximately 30 meters. In the above example, the perception system 116 may identify the particular object using the sensor data 112 with 90% accuracy. The above accuracy percentages and distances are merely exemplary. Furthermore, the above accuracy percentages and / or other statistical information may be determined over a period of time based on repeated use of the LIDAR sensor 104 in one or more environments 108. Additionally or alternatively, the above accuracy percentages and / or other statistical information may be determined as actual measurements through testing of the LIDAR sensor 104.

[0028] Further, in the above examples, the perception system 116 and / or other components associated with the vehicle's one or more processors may perform probabilistic or other comparisons between the stored statistics and aggregated statistics collected over time during vehicle use. For example, the perception system 116 and / or other components associated with the vehicle's one or more processors may aggregate accuracy percentages and / or other statistics corresponding to one or more sensors during vehicle use. In the above examples, the perception system 116 and / or other components associated with the vehicle's one or more processors may compare the "in use" statistics to stored statistics associated with the corresponding sensor modalities (e.g., in use statistics for the LIDAR sensor 104 may be probabilistically compared to stored LIDAR sensor statistics). In the above example, if the statistics in use (e.g., percentage accuracy) are outside a predetermined range (e.g., + / - 3%) of the stored statistics throughout the use of the vehicle, the determination may be communicated to a response system 124, which may in turn initiate a response and / or any other action 126 to correct the discrepancy. Additionally or alternatively, the determination may cause the LIDAR sensor data 112 collected using the LIDAR sensor 104 to not be used in training various sensor modalities. For example, the determination may cause the LIDAR sensor data 112 collected using the LIDAR sensor 104 to not be provided to the machine learning system described above.

[0029] 2 illustrates an example environment 200 through which an example vehicle 202 is traveling. The example vehicle 202 may be a driverless vehicle, such as an autonomous vehicle configured to operate according to a Level 5 classification issued by the U.S. Federal Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey, without the driver (or passenger) being expected to control the vehicle at any time. In such an example, the vehicle 202 may be configured to control all functions from the beginning to the completion of the journey, including all parking functions, and therefore may not include a driver and / or controls for operating the vehicle 202, such as a steering wheel, accelerator pedal, and / or brake pedal. The foregoing is merely an example, and the systems and methods described herein may be incorporated into any land, air, or water transportation vehicle, including vehicles ranging from those that must be manually controlled by a driver at all times to those that are partially or fully autonomously controlled.

[0030] The exemplary vehicle 202 may be a vehicle of any configuration, such as, for example, a van, a sport utility vehicle, a crossover vehicle, a truck, a bus, an agricultural vehicle, and a construction vehicle. The vehicle 202 may be powered by one or more internal combustion engines, one or more electric motors, hydrogen power, any combination thereof, and / or any other suitable power source. While the exemplary vehicle 202 has four wheels 204, the systems and methods described herein may be incorporated into vehicles having a fewer or greater number of wheels, tires, and / or tracks. The exemplary vehicle 202 may have four-wheel steering and may generally operate with equal performance characteristics in all directions, for example, such that a first end 206 of the vehicle 202 is the front end of the vehicle 202 when traveling in a first direction 208 and such that the first end 206 is the rear end of the vehicle 202 when traveling in an opposite second direction 210, as shown in FIG. 2 . Similarly, the second end 212 of the vehicle 202 would be the front end of the vehicle 202 when traveling in the second direction 210, and would be the rear end of the vehicle 202 when traveling in the opposite first direction 208. The exemplary characteristics described thus far may facilitate greater maneuverability in tight spaces or crowded environments, such as, for example, parking lots and urban areas.

[0031] The vehicle 202 may proceed through the environment 200 while relying at least in part on sensor data indicative of objects in the environment 200 to determine the trajectory of the vehicle 202. For example, as the vehicle 202 proceeds through the environment 200, one or more image capture devices 102, LIDAR sensors 104, and / or other types of sensors 106 capture data associated with detected objects (e.g., the vehicle 214 shown in FIG. 2 , and / or pedestrians, buildings, barriers, road signs, etc.). For example, the captured data may be used as input for determining the trajectory of the vehicle 202. As depicted schematically in FIG. 2 and as described above with respect to FIG. 1 , the vehicle 202 may include a perception system 116 configured to receive respective signals from one or more sensors (e.g., the image capture device(s) 102, the LIDAR sensor(s) 104, and / or other types of sensor(s) 106). In some examples, the perception system 116 may be configured to identify one or more objects (e.g., one or more vehicles 214 illustrated in FIG. 2 , one or more groups of objects 128, 130, 132 described with respect to FIG. 1 , etc.) based at least in part on one or more of the above signals. Additionally, the perception system 116 may be configured to generate fused sensor data 134 including the group of objects 120. In the above examples, the perception system 116 may also be configured to determine errors associated with the data included in one or more respective sensor signals and / or with one or more groups of objects 128, 130, 132. For example, the object detection system 118 associated with the perception system 116 may be configured to identify one or more objects 122 in the group of objects 120 that are not in or are misclassified as being in one or more of the groups of objects 128, 130, 132. In the above examples, the response system associated with the perception system 116 may initiate a response and / or any other action to correct the discrepancy.

[0032] 3 is a block diagram illustrating an example system 300 for implementing one or more of the example processes described herein. For example, system 300 may be configured to identify objects and initiate a response based at least in part on sensor data described herein. In at least one example, system 300 may include a vehicle 302, which may be the same vehicle as vehicle 202 described above with reference to FIG. 2.

[0033] The vehicle 302 may include a vehicle computing device 304, one or more sensor systems 306, one or more emitters 308, one or more communication connections 310, at least one direct connection 312, and one or more drive modules 314.

[0034] The vehicle computing device 304 may include one or more processors 316 and a memory 318 communicatively coupled to the one or more processors 316. In the illustrated example, the vehicle 302 is an autonomous vehicle; however, the vehicle 302 could be any other type of vehicle. In the illustrated example, the memory 318 of the vehicle computing device 304 stores a localization system 320, a perception system 322 (e.g., the perception system 116 described above with reference to FIGS. 1 and 2 and including the detection system 118), a planning system 324, one or more system controllers 326, a prediction system 328, and a response system 330 (e.g., the response system 124 described above with reference to FIGS. 1 and 2). While depicted in FIG. 3 as being in memory 318 for illustrative purposes, it is anticipated that the perception system 322, the response system 330, and / or other components of the vehicle computing device 304 may additionally or alternatively be accessible to the vehicle 302 (e.g., stored remotely).

[0035] In at least one example, the localization system 320 is capable of determining where the vehicle 302 is with respect to local and / or global maps based at least in part on sensor data received from the sensor system(s) 306, a perception system 322 that performs entity detection, segmentation, and / or classification based at least in part on the sensor data received from the sensor system(s) 306, and a planning system 324 that determines a route and / or trajectory to be used to control the vehicle 302 based at least in part on the sensor data received from the sensor system(s) 306. Additional details of usable localizer, perception, and planning systems can be found in U.S. patent application Ser. No. 14 / 932,963, entitled "Adaptive Mapping to Navigate Autonomous Vehicle Responsive to Physical Environment Changes," filed Nov. 4, 2015, and Ser. No. 15 / 632,208, entitled "Trajectory Generation and Execution Architecture," filed Jun. 23, 2017, both of which are incorporated herein by reference. In instances where vehicle 302 is not an autonomous vehicle, one or more of the above-described components can be omitted from vehicle 302.

[0036] In at least one example, the localization system 320, the perception system 322, and / or the planning system 324 can process sensor data received from the sensor system(s) and send their respective outputs to one or more remote computing devices 334 (e.g., one or more server computers or other computing devices) via one or more network(s) 332. In the above example, the sensor system(s) 306 may include the image capture device 102, the LIDAR sensor 104, and / or one or more additional sensors 106 described above. In at least one example, the localization system 320, the perception system 322, and / or the planning system 324 can send their respective outputs to the one or more remote computing device(s) 334 at a particular frequency, after a predetermined time, in near real time, etc.

[0037] In at least one example, one or more system controller(s) 326 can be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 302. The just-mentioned system controller(s) 326 can communicate with and / or control corresponding systems of the drive module(s) 314 and / or other components of the vehicle 302.

[0038] In at least one example, the prediction system 328 may receive sensor data directly from the sensor system(s) 306 and / or from one of the other systems (e.g., the localization system 320, the perception system 322, etc.). In some examples, if the prediction system 328 receives the sensor data from the sensor system(s) 306, the sensor data may be raw sensor data. In additional and / or alternative examples, if the prediction system 328 receives the sensor data from one of the other systems, the sensor data may be processed sensor data. For example, in examples, the localization system 320 may process the image data 110, the LIDAR sensor data 112, and / or the sensor data 114 to determine where the vehicle 302 is with respect to local and / or global maps and may output processed sensor data (e.g., location data) indicative of the above information. Additionally or alternatively, the perception system 322 may process the sensor data to perform object detection, segmentation, and / or classification. In some examples, the perception system 322 may provide processed sensor data indicating the presence of an object (e.g., the object 122 described above with respect to FIG. 1 ) closest to the vehicle 302 and / or the classification of the object as an object type (e.g., car, pedestrian, cyclist, barrier, road sign, unknown, etc.). In additional and / or alternative examples, the perception system 322 may provide processed sensor data indicating one or more characteristics associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the object 122 identified by the perception system 322 may include, but are not limited to, x-position (global position), y-position (global position), z-position (global position), orientation, object type (e.g., classification), object speed, etc.Features associated with the environment may include, but are not limited to, the presence of other objects in the environment, the state of other objects in the environment, the time of day, the day of the week, the season, weather conditions, darkness / light indications, etc.

[0039] For example, the sensor system(s) 306 may include an image capture device 102, such as any camera (e.g., an RGB-camera, a monochrome camera, an intensity (grayscale) camera, an infrared camera, an ultraviolet camera, a depth camera, a stereo camera, etc.). The image capture device 102 may capture image data 110 indicative of a group of objects 128, and the sensor system(s) 306 may transmit the image data 110 to the perception system 322 and / or other systems of the vehicle computing device 304 for subsequent processing.

[0040] Similarly, the sensor system(s) 306 may include one or more LIDAR sensors 104 configured to capture LIDAR sensor data 112 for use as described herein. For example, the sensor system(s) 306 may be configured to combine or synthesize LIDAR data from multiple LIDAR sensors 104 to generate a meta spin of the LIDAR data, which may be the LIDAR sensor data 112 generated by the multiple LIDAR sensors 104. In the case of a meta spin of the LIDAR data, the sensor system(s) 306 may be configured to determine a virtual origin (e.g., a common origin) of the meta spin data. In some examples, the sensor system(s) 306 may be configured to determine the range between the LIDAR sensors 104 and a point on an object or surface, and in some examples, the sensor system(s) 306 may be configured to determine a surface normal vector for each point captured and / or sensed by each LIDAR sensor 104. As a non-limiting example, the determination of the surface normal may be performed by calculating the normal to the cross product of vectors pointing in the direction from a point to two of its nearest neighbors. As may be understood in the context of the present disclosure, the sensor system(s) 306 may transmit any of the LIDAR sensor data 112 to the perception system 322 and / or to other systems of the vehicle computing device 304 for subsequent processing.

[0041] In some examples, the sensor system(s) 306 may provide image data 110, LIDAR sensor data 112, and / or other sensor data 114 to the vehicle computing device 304 for combining, fusing, segmenting, classifying, labeling, synthesizing, and / or otherwise processing the data. In some examples, the memory 318 of the vehicle computing device 304 may also store simulated data generated by computer simulation algorithms, in part for use in testing. In some examples, the simulated data may include any type of simulated data, such as, for example, image data, sensor data (e.g., LIDAR data, RADAR data, and / or SONAR data), GPS data, etc. In some examples, the vehicle 302 may be configured to modify, transform, and / or perform transform operations on the simulated data to validate operations and / or to train models executable by machine learning systems.

[0042] In some examples, the prediction system 328 may access the heat map and / or other information stored in memory 318 and may use the information to perform a lookup to assist in predicting future behavior of the identified object 122. In at least one example, the heat map and / or other information stored in memory 318 may be associated with an object type (e.g., car, pedestrian, cyclist, barrier, road sign, unknown, etc.). In at least one example, the prediction system 328 may perform a lookup to determine a behavior pattern associated with the detected object type. In at least one example, the prediction system 328 may use the object's location and one or more characteristics to identify a cell in the heat map. That is, a cell may indicate or otherwise be referenced by a unique index that includes the object's location and one or more characteristics associated with the object and / or the environment in which the object resides. In some examples, a cell may be associated with data indicative of behavior patterns for one or more objects (of the same object type) at a location having the same one or more characteristics related to the object and / or the environment. The prediction system 328 can retrieve data indicative of behavioral patterns associated with the cell and can utilize the data indicative of behavioral patterns to determine a predicted behavior of the object.

[0043] Based at least in part on determining the predicted behavior of the object, the prediction system 328 can provide an indication of the predicted behavior to other systems of the vehicle computing device 304. In at least one example, the predicted behavior can be used to inform the perception system 322 to perform object detection, segmentation, and / or classification (e.g., in an example, a heat map and / or other information related to the identified object 122 can be used to perform a reverse lookup to determine the object's type). Additionally or alternatively, the planning system 324 can utilize the predicted behavior to determine a trajectory that the vehicle 302 may follow. The planning system 324 can execute the trajectory (e.g., in an autonomous vehicle example) and send the trajectory to the system controller 326, which can cause the vehicle 302 to drive along the trajectory. Additionally or alternatively, the predicted behavior can be used to weight a trajectory generated by the planning system 324, which can determine a route and / or trajectory to use to control the vehicle 302. Furthermore, the predicted behavior can be used to inform a neural network, which can be used to predict the behavior(s) of the entity(ies).

[0044] In at least one example, the sensor system(s) 306 may include LIDAR sensors, RADAR sensors, ultrasonic transducers, SONAR sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras and / or other image capture devices (e.g., RGB, IR, intensity, depth, etc.), microphones, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor system(s) 306 may include multiple instances of each of the just mentioned or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors located at the corners, front, rear, sides, and / or top of the vehicle 302. In another example, the cameras and / or other image capture devices may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle 302.

[0045] The sensor system(s) 306 may provide signals to the vehicle computing device 304. For example, each sensor included in the sensor system(s) 306 may be configured to send a respective signal to the vehicle computing device 304, which may include data indicative of a group of objects detected by the respective sensor. In some examples, the image capture device 102 of the sensor system(s) 306 may provide a first signal to the vehicle computing device 304, which may include image data 110 indicative of a first group of objects 128 detectable in the environment 108 in which the image capture device 102 is present. Additionally, the LIDAR sensor 104 of the sensor system(s) 306 may provide a second signal to the vehicle computing device 304, which may include LIDAR sensor data 112 indicative of a second group of objects 130 detectable in the environment 108. Additionally, the RADAR sensor or one or more additional sensors 106 associated with the sensor system(s) 306 may provide a third signal to the vehicle computing device 304, where the third signal may include sensor data 114 indicative of a third group of objects 132 detectable in the environment 108. In the above example, the perception system 322 (e.g., the object detection system 118 described above) may confirm and / or determine that a group of objects 120 is present in the environment 108 based at least in part on one or more signals received from the sensor system(s) 306. For example, the perception system 322 (e.g., the object detection system 118) may confirm that at least one object 122 is present in the environment 108 based at least in part on one or more signals received from the sensor system(s) 306.Additionally, the perception system 322 may identify and / or otherwise determine errors associated with the data contained in one or more respective sensor signals (e.g., determine whether an identified object 122 is not in or has been misclassified as one or more groups 128, 130, 132 of objects associated with the received signals).

[0046] In examples where the perception system 322 determines that the object 122 is not in or has been misclassified as being in at least one of the object groups 128, 130, 132, the response system 330 may initiate a response 126 and / or any other action based at least in part on the determination. In some examples, the response and / or other action 126 may include, among other things, at least one of ignoring portions of the image data 110, the LIDAR sensor data 112, and / or the sensor data 114. Furthermore, the response and / or other action 126 may include modifying weights, confidence values, and / or other metrics associated with one or more image capture devices 102, the LIDAR sensor 104, one or more additional sensors 106, other sensors of the sensor system(s) 306, and / or signals received from each sensor of the sensor system(s) 306. Additionally, the above responses and / or other actions 126 may include modifying training data associated with one or more image capture devices 102, LIDAR sensor 104, one or more additional sensors 106, other sensors of sensor system(s) 306.

[0047] In some examples, the response and / or other action 126 may further include generating a verification request and / or sending the verification request to a service center for review by a human operator (e.g., a teleoperator) using communication connection(s) 310. The teleoperator may provide one or more indications that the sensor in question is malfunctioning, may confirm operation of a proposed solution and / or mitigation, or may otherwise communicate control data to the vehicle 302 in response to receiving an indication of a sensor error. Further, the response and / or other action 126 may include controlling the drive module(s) 314 to vary the speed, direction, and / or other operating parameters of the vehicle 302. Additionally or alternatively, the sensor system(s) 306 may send any of the signals and / or sensor data described herein to one or more remote computing device(s) 334 via one or more networks 332, at a particular frequency, after a predetermined time, in near real time, etc. In the above examples, one or more remote computing device(s) 334 (e.g., one or more processor(s) 336) may perform one or more processes described herein.

[0048] Additionally, the vehicle 302 may include one or more emitters 308 for emitting light and / or sound, as described above. The emitters 308 in the present example include interior audio and visual emitters that communicate with passengers of the vehicle 302. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touchscreens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.). Additionally, the emitters 308 in the present example also include exterior emitters. By way of example and not limitation, the exterior emitters in the present example include lights that signal direction of travel or other indications regarding vehicle operation (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) that audibly communicate with pedestrians or other nearby vehicles.

[0049] Additionally, vehicle 302 may also include one or more communication connection(s) 310 that enable communication between vehicle 302 and one or more other local or remote computing device(s). For example, communication connection(s) 310 may facilitate communication with other local computing device(s) on vehicle 302 and / or with drive module(s) 314. Additionally, communication connection(s) 310 may also enable vehicle 302 to communicate with other nearby computer device(s) (e.g., other nearby vehicles, traffic signals, etc.). Additionally, communication connection(s) 310 may also enable vehicle 302 to communicate with remote teleoperated computing devices, remote service centers, or other remote services.

[0050] The communication connection(s) 310 may include physical and / or logical interfaces for connecting the vehicle computing device 304 to another computing device or to a network, such as, for example, network(s) 332. For example, the communication connection(s) 310 may enable Wi-Fi-based communications, such as, for example, via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth™, or any suitable wired or wireless communications protocol that enables each computing device to interface with other computing device(s).

[0051] In at least one example, the vehicle 302 may include one or more drive modules 314. In some examples, the vehicle 302 may have a single drive module 314. In at least one example, if the vehicle 302 has multiple drive modules 314, the individual drive modules 314 may be located at opposite ends of the vehicle 302 (e.g., front and rear, etc.). In at least one example, the drive module(s) 314 may include one or more sensor systems that detect the state of the drive module(s) 314 and / or the state of the vehicle 302's surroundings. By way of example and not limitation, the sensor system(s) may include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the drive module's wheels, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive module, cameras or other imaging sensors, ultrasonic sensors that acoustically detect objects around the drive module, LIDAR sensors, RADAR sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive module(s) 314. In some cases, the sensor system(s) on the drive module(s) 314 may overlap or complement corresponding systems on the vehicle 302 (e.g., sensor system(s) 306).

[0052] The drive module(s) 314 can include many vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which can be electrically powered), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a safety control system for distributing braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as a DC / DC converter, a high-voltage junction, high-voltage cables, a charging system, a charge port, etc.). Additionally, the drive module(s) 314 can receive and preprocess data from the sensor system(s) and can include a drive module controller for controlling the operation of various vehicle systems. In some examples, the drive module controller can include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more modules that perform various functionality of the drive module(s) 314. Furthermore, the drive module(s) 314 may also include one or more communication connection(s) that enable the respective drive module to communicate with one or more other local or remote computing device(s).

[0053] As described above, the vehicle 302 may send signals and / or sensor data over the network(s) 332 to one or more remote computing device(s) 334. In some examples, the vehicle 302 may send raw sensor data to the remote computing device(s) 334. In other examples, the vehicle 302 may send processed sensor data and / or representations of the sensor data to the remote computing device(s) 334. In some examples, the vehicle 302 may send sensor data to the one or more remote computing device(s) 334 at a particular frequency, after a predetermined time, in near real time, etc.

[0054] The remote computing device(s) 334 may receive the signals and / or sensor data (raw or processed) and may perform any of the processing described herein based at least in part on the signals and / or sensor data. In at least one example, the remote computing device(s) 334 may include one or more processors 336 and a memory 338 communicatively coupled to the one or more processors 336. In the illustrated example, the memory 338 of the remote computing device(s) 334 stores a sensor data store 340, a sensor data processing system 342, and a machine learning system 344.

[0055] The sensor data store 340 may store sensor data (raw or processed) received from one or more vehicles, such as, for example, the vehicle 302. The sensor data in the sensor data store 340 may represent sensor data (e.g., previously concatenated sensor data) collected at a previous time(s) by one or more onboard sensor systems (e.g., such as the onboard sensor system(s) 306) or other sensor system(s). In some examples, the sensor data may be stored with location, object type, and / or other types of characteristics. Additionally, in at least one example, behavior determined from the sensor data may be stored in the sensor data store 340. That is, the behavior of an individual object may be associated with the particular sensor data from which the behavior was determined.

[0056] In at least one example, the sensor data processing system 342 can receive sensor data (raw or processed) from one or more vehicles, such as, for example, the vehicle 302. As described above, the vehicle 302 can send signals including the sensor data to one or more remote computing device(s) 334 at a particular frequency, after a predetermined time, in near real time, etc. Additionally, the sensor data processing system 342 can receive sensor data at a particular frequency, after a predetermined time, in near real time, etc. In additional and / or alternative examples, the sensor data processing system 342 can receive data from additional and / or alternative sensor system(s) (e.g., not associated with the vehicle). In some examples, the sensor data processing system 342 can send the sensor data to the sensor data store 340 for storage.

[0057] In at least one example, the sensor data processing system 342 can process the sensor data. In some examples, the sensor data processing system 342 can determine behavior of objects associated with a particular object type based on the sensor data. That is, the sensor data processing system 342 can analyze sensor data associated with a particular time period to determine how an object(s) present in the environment behaves throughout the time period. In at least one example, the sensor data store 340 can store data indicative of behavior of objects associated with the object type, which can be associated in the sensor data store 340 with the sensor data used to determine the behavior. In at least one example, the data indicative of behavior of objects associated with the object type, as determined from the sensor data, can be associated with observations. The observations can be stored in the sensor data store 340.

[0058] As described above, the localization system 320, the perception system 322, and / or other components of the vehicle computing device 304 may be configured to detect and classify outside objects, such as pedestrians, bicyclists, dogs, other vehicles, etc. Based at least in part on the classification of the outside object, the outside object may be labeled as a dynamic object or a static object. For example, the perception system 322 may be configured to label a tree as a static object and a pedestrian as a dynamic object. Further data about the outside object may be generated by tracking the outside object, and the type of object classification may, in some examples, be used by the prediction system 328 to predict or determine the likelihood that the outside object may obstruct the vehicle 302 traveling along the planned path. For example, an outside object classified as a pedestrian may be associated with a maximum speed and / or an average speed. The localization system 320, the perception system 322, the segmentation system of the vehicle computing device 304, the sensor data processing system 342, and / or other components of the remote computing device 334 may use a machine learning system 344 that may run any one or more machine learning algorithms, such as, for example, neural networks, to perform classification operations.

[0059] Neural networks utilized by machine learning system 344 may include biologically inspired algorithms that pass input data through a series of connected layers to generate an output. One example of a neural network is a CNN (convolutional neural network). Furthermore, each layer of a CNN may contain another CNN, or may contain any number of layers. Neural networks may utilize machine learning, a broad class of algorithms described above in which an output is generated based on learning parameters.

[0060] Although described in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning algorithms may include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (C ART), Iterative Dichotomy Method 3 (ID3), Chi-Square Automatic Interaction Detection (CHAID), Decision Cutoff, Conditional Decision Tree), Bayesian algorithms (e.g., Naive Bayes, Gaussian Naive Bayes, Multinomial Naive Bayes, Average One Dependence Estimators (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-median, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning These may include algorithms (e.g., perceptrons, backpropagation, Hopfield networks, radial basis function networks (RBFNs)), deep learning algorithms (e.g., deep Boltzmann machines (DBMs), deep belief networks (DBNs), convolutional neural networks (CNNs), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), Adaboost, stacked generalization (blending), gradient boosting machines (GBMs), gradient boosted regression trees (GBRTs), random forests), support vector machines (SVMs)), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0061] In some examples, multiple types of machine learning systems may be used to provide respective results for each type of machine learning used. In some examples, a confidence score may be associated with each of the results, and the trusted results may be based at least in part on the confidence score associated with the result. For example, the result associated with the highest confidence score may be selected over other results, or results may be combined based on the confidence scores, e.g., based on statistical methods such as weighted averaging, etc. Additionally, while the machine learning system 344 is illustrated as a component of the memory 338, in other examples, the machine learning system 344 and / or at least a portion thereof may comprise a component of the memory 318 of the vehicle computing device 304.

[0062] The processor(s) 316 of the vehicle 302 and the processor(s) 336 of the remote computing device(s) 334 can be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, the processor(s) 316 and 336 can include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into registers and / or other electronic data that can be stored in memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices can also be considered processors as long as they are configured to implement encoded instructions.

[0063] Memory 318 and memory 338 are examples of non-transitory computer-readable media. Memory 318 and memory 338 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods described herein and functionality attributed to the various systems. In various implementations, memory may be implemented using any suitable memory technology, such as SRAM (static RAM), SDRAM (synchronous DRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, and those shown in the accompanying drawings are merely examples relevant to the discussion herein.

[0064] 3 is illustrated as a distributed system, in alternative examples, components of the vehicle 302 may be associated with the remote computing device(s) 334 and / or components of the remote computing device(s) 334 may be associated with the vehicle 302. That is, the vehicle 302 may perform one or more functions associated with the remote computing device(s) 334, and vice versa.

[0065] 4A is a side view 400 of a vehicle 202 having multiple sensor assemblies mounted on or carried by the example vehicle 202 (e.g., the vehicle 302 described with respect to FIG. 3 ). In some examples, data sets from the multiple sensor assemblies may be combined or synthesized to form a metaspin (e.g., LIDAR data representing multiple LIDAR sensors) or may be combined or fused using sensor fusion techniques to improve accuracy or processing for segmentation, classification, prediction, planning, trajectory generation, etc.

[0066] As shown in side view 400, vehicle 202 may include any number, combination, or configuration of sensors. For example, vehicle 202 includes at least sensors 402, 404, and 406. In some cases, sensor 402 may include a RADAR sensor having a vertical field of view exemplified as θ1. Sensor 404 may include a LIDAR sensor mounted on the roof of vehicle 202, with sensor 404 having a vertical field of view exemplified as θ2. In some cases, sensor 406 may include a LIDAR sensor having a vertical field of view exemplified as θ3. θ3 Of course, vehicle 202 may include any number and type of sensors and is not limited to the example provided in FIG.

[0067] FIG. 4B depicts a top plan view 408 of example vehicle 202 having multiple sensor assemblies mounted thereon. For example, sensors 402, 404, and 406 can be seen in FIG. 4B , as can additional sensors 410, 412, 414, and 416. For example, sensors 406 and 416 may be located together or proximate to each other, but may include different sensor types or sensor modalities with different fields of view. In some cases, sensors 410, 412, 414, and 416 may include additional LIDAR sensors, RADAR sensors, and / or image capture devices. As can be understood in the context of the present disclosure, vehicle 202 may include any number and types of sensors. As shown in FIG. 4B , sensor 402 may include a horizontal field of view θ4, sensor 404 may include a horizontal field of view θ5, sensor 406 may include a horizontal field of view θ6, sensor 410 may include a horizontal field of view θ7, sensor 412 may include a horizontal field of view θ8, sensor 414 may include a horizontal field of view θ9, and sensor 416 may include a horizontal field of view θ 10 As can be understood in the context of this disclosure, the mounting location and field of view can include any number of configurations.

[0068] Additionally, one or more sensors described herein may be used to detect one or more objects 418 located within the vehicle 202 and / or the environment in which the sensor resides. For example, in embodiments in which the sensor 404 includes a LIDAR sensor (e.g., the LIDAR sensor 104), the sensor 404 may be configured to generate signals including LIDAR sensor data 112 indicative of a group 130 of objects detectable in an environment in which the sensor 404 resides (e.g., the environment 108 illustrated in FIG. 1 ). In some examples, the signals generated by the LIDAR sensor 404 may include a signal (e.g., a first signal) from a first light sensor of the LIDAR sensor 404 indicative of the detection of light reflected from the object 418, and the object 418 may be included in the group 130 of objects indicated by the sensor data 112. In the above examples, further, the signals generated by the LIDAR sensor 404 may include one or more additional signals (e.g., at least a second signal from a second light sensor of the LIDAR sensor 404) indicating detection of light reflected from the object 418, and the object 418 may be included in the group of objects 130. Any of the above signals may be provided to the perception system 322 and / or other components associated with the vehicle computing device 304. In some examples, the perception system 322 may identify the object 418 based at least in part on the one or more signals generated by the LIDAR sensor 404, and may determine whether the object 122 described above is missing from or misclassified in one or more groups of objects 128, 130, 132 (e.g., missing from or misclassified in the group of objects 130 corresponding to the LIDAR sensor data 112) based at least in part on identifying the object 418 as included in the group of objects 130.

[0069] In further embodiments in which the sensor 406 includes a camera or other image capture device (e.g., the image capture device 102), the sensor 406 may be configured to generate a signal including image data 110 indicative of a group of objects 128 detectable in an environment in which the sensor 406 is present (e.g., the environment 108 illustrated in FIG. 1 ). In some examples, the signal generated by the sensor 406 may include a signal (e.g., a first signal) including one or more images captured by the sensor 406 that illustrate, show, and / or otherwise include an object 418, where the object 418 may be included in the group of objects 128 indicated by the image data 110. In the above example, any signal generated by the sensor 406 may be provided to a perception system 322 and / or other components associated with the vehicle computing device 304. In some examples, the perception system 322 may identify the object 418 based at least in part on the one or more signals generated by the sensor 406. For example, the perception system 322 may identify and / or otherwise determine an error associated with data included in one or more respective sensor signals. In the above example, the perception system 322 may determine whether the object 122 described above is missing from or misclassified in one or more groups of objects 128, 130, 132 (e.g., missing from or misclassified in the group of objects 128 corresponding to the image data 110) based at least in part on identifying the object 418 included in the group of objects 128. Additionally, it will be understood that the perception system 322 may perform similar processing on signals and / or sensor data 114 received from any other sensors described herein (e.g., sensor 106, sensor 402, sensors associated with sensor system(s) 306, etc.).

[0070] 5 is an example pictorial flow diagram 500 of an example process for comparing individual sensor data to fused sensor data and initiating a response based at least in part on determining an error associated with the data included in one or more of the respective sensor signals (e.g., determining that at least one object included in the fused sensor data is not in or is misclassified as part of the group of objects associated with the data received from the individual sensors). Similar to FIG. 1 , sensors, which may include an image capture device 102, a LIDAR sensor 104, and / or another sensor 106, may be connected to a vehicle traveling through an environment 108. The image capture device 102, the LIDAR sensor 104, and the sensor 106 may capture image data 110, LIDAR sensor data 112, and sensor data 114, respectively, and communicate such data to a perception system 116. As described above, the signal from the image capture device 102 may include image data 110, which may represent a scene including and / or otherwise depicting a first group of objects 128 ( FIG. 1 ) detectable in the environment 108. Similarly, signals from LIDAR sensor 104 may include LIDAR sensor data 112, which may represent a scene that includes and / or otherwise indicative of a second group of objects 130 (FIG. 1) detectable in environment 108. Additionally, signals from sensor 106 may include sensor data 114, which may represent a scene that includes and / or otherwise indicative of a third group of objects 132 (FIG. 1) detectable in environment 108.

[0071] 5 , image capture device 102, LIDAR sensor 104, and sensor 106 may capture the above-described image data 110, LIDAR sensor data 112, and sensor data 114 substantially continuously and / or at regular or irregular time intervals. In the above example, image data 110 captured by image capture device 102 may show, illustrate, and / or represent one or more successive scenes D. In the above example, each of the successive scenes D may include respective object groups 502, 504, 506 detected by image capture device 102 at different (e.g., successive) times T1, T2, T3, respectively. While three successive scenes D and three corresponding object groups 502, 504, 506 are illustrated in FIG. 5 , it will be understood that more or fewer scenes and / or object groups may be detected by image capture device 102 at different (e.g., successive) respective times or time intervals throughout operation. Each group of objects 502, 504, 506 may be similar to and / or identical to group of objects 128 described above with respect to Figure 1. In the above example, image capture device 102 may send image data 110 to perception system 116, which may identify one or more groups of objects 502, 504, 506 described herein based at least in part on the received image data 110.

[0072] Similarly, the LIDAR sensor data 112 captured by the LIDAR sensor 104 may show, illustrate, and / or represent one or more successive scenes E. In the above example, each of the successive scenes E may include respective object groups 508, 510, 512 detected by the LIDAR sensor 104 at different (e.g., successive) times T1, T2, T3. Although three successive scenes E and three object groups 508, 510, 512 are illustrated in FIG. 5 , it will be understood that more or fewer scenes and / or object groups may be detected by the LIDAR sensor 104 at different (e.g., successive) respective times or time intervals throughout operation. Each object group 508, 510, 512 may be similar and / or identical to the object group 130 described above with respect to FIG. 1. In the above example, the LIDAR sensor 104 may send LIDAR sensor data 112 to the perception system 116, and the perception system 116 may identify one or more groups 508, 510, 512 of objects described herein based at least in part on the received LIDAR sensor data 112.

[0073] Additionally, the sensor data 114 captured by one or more additional sensors 106 may show, illustrate, and / or represent one or more successive scenes F. In the above example, each of the successive scenes F may include respective object groups 514, 516, 518 detected by the sensor 106 at different (e.g., successive) times T1, T2, T3. Although three scenes F and object groups 514, 516, 518 are illustrated in FIG. 5 , it will be understood that more or fewer scenes and / or object groups may be detected by the sensor 106 at different (e.g., successive) respective times or time intervals throughout operation. Each object group 514, 516, 518 may be similar and / or identical to the object group 132 described above with respect to FIG. 1 . In the above example, the sensor 106 may send sensor data 114 to the perception system 116, and the perception system 116 may identify one or more groups 514, 516, 518 of objects described herein based at least in part on the received sensor data 114.

[0074] 5, perception system 116 may be configured to receive signals from image capture device 102, LIDAR sensor 104, and sensor 106, which may include image data 110, LIDAR sensor data 112, and sensor data 114 described above with respect to FIG. 5. In the example shown, perception system 116 may be configured to determine corresponding groups of objects 522, 524, 526 based at least in part on one or more signals. For example, perception system 116 may be configured to generate, identify, define, and / or otherwise determine group 522 of objects present in environment 108 at time T1 based at least in part on groups of objects 502, 508, 514 corresponding to time T1. Further, the perception system 116 may determine a group 524 of objects present in the environment 108 at time T2 based at least in part on the groups of objects 504, 510, 516 corresponding to time T2, and may determine a group 526 of objects present in the environment 108 at time T3 based at least in part on the groups of objects 506, 512, 518 corresponding to time T3. In the above example, the groups of objects 522, 524, 526 determined by the perception system 116 through use of the image data 110, the LIDAR sensor data 112, and the sensor data 114 may individually or collectively be considered fused sensor data 134. Further, the perception system 116 may use data association, object recognition, data characterization, and / or any of the other techniques described herein to determine which objects in each group of objects 522, 524, 526 are identical.

[0075] In the above example, the perception system 116 may also be configured to identify one or more objects 122 and / or determine one or more groups of objects 522, 524, 526 using sensor data collected at a later time and / or based at least in part on sensor data otherwise collected at a later time. For example, the group of objects 504, 510, 516 associated with time T2 (e.g., observed at a later time than T1 associated with the group of objects 502, 508, 514) and / or the group of objects 506, 512, 518 associated with time T3 (e.g., observed at a later time than T1 associated with the group of objects 502, 508, 514) may be used by the perception system 116 and / or the object detection system 118 to determine the group of objects 522 indicated by the perception system 116 as being present in the environment 108 at time T1. Additionally, the perception system 116 and / or object detection system 118 may identify and / or otherwise determine an error associated with a group of objects observed at a first time (e.g., time T1) based at least in part on sensor data and / or a group of objects observed at a second time (e.g., time T2) that is later than the first time. For example, the perception system 116 and / or object detection system 118 may identify a particular object as being included in the fused sensor data 134 described above but as being absent or misclassified in one or more groups of objects (e.g., group of objects observed at time T1) based at least in part on the groups of objects 504, 510, 516 observed at time T2 and / or the groups of objects 506, 512, 518 observed at time T3.The subsequently observed (e.g., future) sensor data and corresponding groups of objects may provide a high level of confidence by the perception system 116 when generating fused sensor data 134 including the group of one or more objects 522 indicated as present in the environment 108 at time T1 and / or when identifying one or more objects 122 present in the environment 108 at time T1. Additionally, the subsequently observed (e.g., future) sensor data and corresponding groups of objects may also be used, for example, as log data for training various sensor modalities in offline machine learning processes.

[0076] In some examples, the perception system 116 may provide any of the image data 110, the LIDAR sensor data 112, the sensor data 114, the fused sensor data 134 (e.g., one or more groups of objects 522, 524, 526), ​​and / or other outputs of the perception system 116 to a machine learning system 344 (e.g., a convolutional neural network (CNN)), such as the machine learning system 344 described above with respect to FIG. 3 . For example, the image capture device 102, the LIDAR sensor 104, and / or the sensors 106 may communicate one or more signals including data representing respective continuous scenes associated with the environment 108 to the perception system 116. The perception system 116 and / or the object detection system 118 may communicate the one or more signals to the machine learning system 344, and the data may be segmented (e.g., the image data 110 may be segmented) using the machine learning system 344. In the above examples, any of the signals or information provided to the machine learning system 344 may be used as ground truth to train the machine learning system 344. The same or similar algorithms may be used to segment any one or more other sensor modalities. In some examples, the machine learning system 344, either alone or in combination with the object detection system 118 or other components of the perception system 116, may execute a segmentation model or train other components and / or segment objects in the data. In some examples, based at least in part on the segmentation and / or any of the data association, object recognition, data characterization, and / or other processes described herein, the machine learning system 344 may be configured to identify objects 122 included in the fused sensor data 134 (e.g., included in one or more object groups 522, 524, 526) that are not in at least one of the object groups 502-518 or that are misclassified by identifying at least one data segmentation that is inconsistent with other data segmentations.In some examples, the identification of object 122 in the manner just described may be confirmed or discounted, e.g., according to other methods described herein. For example, as described above, subsequently observed (e.g., future) sensor data and corresponding groups of objects may be used by machine learning system 344 to confirm the identification of object 122, to confirm the location of object 122, and / or to confirm any other parameters associated with object 122. Furthermore, as described above, the present disclosure is not limited to identifying one or more objects 122 that are missing or misclassified in one or more of the above scenes. Instead, additional example processing may include determining one or more additional and / or different errors associated with the data included in one or more respective sensor signals.

[0077] The machine learning system 344 may include any type of machine learning system described herein. For example, the machine learning system 344 may be a CNN. In some examples, the machine learning system 344 may include multiple machine learning systems. As described herein, multiple types of machine learning may be used to provide respective results for each type of machine learning used. In some examples, a confidence score may be associated with each of the results, and the trusted result may be based at least in part on the confidence score associated with the result. For example, the result associated with the highest confidence score may be selected over other results, or results may be combined based on the confidence scores, e.g., based on a statistical method such as a weighted average, etc.

[0078] To generate useful output, a machine learning system 344, such as a CNN, must first learn or "train" a set of parameters. Training is accomplished by inputting training data 528 into the machine learning system 344, which is associated with expected output values. In general, the expected output values ​​just mentioned may be referred to as "ground truth." For example, ground truth may include the identification of a particular object in an image, as well as a semantic classification or label associated with the object (e.g., identifying and labeling the object as a car or a building). The accuracy of the machine learning system 344 may be based on the amount and / or accuracy of the data provided in the training data 528. As a result, an appropriate dataset for training the machine learning system 344 to output segmented sensor data would include a known or previously determined segmentation of the sensor data. In some examples, the training data 528 may include one or more segmented images representing a real-world scene correlated to one or more sensor datasets representing the real-world scene, and may be annotated manually or via one or more algorithms configured to segment, detect, classify, and / or label objects in the sensor datasets. In some examples, the training data 528 may include synthetic data including annotated objects or annotated by a computer algorithm (e.g., computer-generated). Training can be performed using offline and / or online data, as well as later-observed (e.g., future) sensor data as described above, and corresponding groups of objects may be used for the training and / or machine learning training.In any of the examples described herein, the fused sensor data 134 may be provided to the machine learning system 344 and / or may be used as training data 528. Additionally, the image data 110, the LIDAR sensor data 112, and / or the sensor data 114 may be provided to the machine learning system 344 and / or may be used as training data 528.

[0079] In examples where training (e.g., machine learning), object classification, and / or other processing is performed online (e.g., using the vehicle computing device 304 and / or other components of the vehicle 302), the capacity of the memory 318, the speed / power of the processor(s) 316, and / or other parameters associated with the vehicle computing device 304 may limit the speed at which such processing is performed and may also limit the sophistication of the neural networks, algorithms, and / or other components used. In examples where training (e.g., machine learning), object classification, and / or other processing is performed offline (e.g., using the processor(s) 336 and / or other components of the remote computing device(s) 334), the capacity of the memory 338 may be greater than the capacity of the memory 318. Similarly, the speed / power of the processor(s) 336 and / or other parameters of the remote computing device(s) 334 may be greater than the speed / power of the corresponding processor(s) 316. As a result, more sophisticated neural networks, algorithms, and / or other components may be used in offline processing. Furthermore, in any of the examples described herein, due to the relative robustness of such offline systems, an offline 3D perception pipeline associated with the remote computing device 334 may be utilized to train, for example, the online sensors of the vehicle 302.

[0080] Furthermore, in some examples, the offline machine learning techniques may include the use of data acquired later in time to make decisions and / or predictions about the past locations of objects and / or about object identification / classification. For example, it is understood that log data and / or other historical data may be stored in the sensor data store 340. For example, the log data may include information indicative of the sensed locations of particular objects at various times, the characterization and / or identification of the objects, etc. In some examples, the historical data may be used in various forward and backward looping processes to help train one or more sensors of the vehicle 302. For example, the historical data may be used to verify predictions made during the data association and / or object tracking processes of the present disclosure. As a result, using historical data to help train one or more sensor pipelines of the vehicle 302 during offline machine learning training may improve the accuracy of online decisions made by the perception system 322 and / or other components of the vehicle computing device 304.

[0081] A loss function may be used to adjust internal parameters of the machine learning system 344 during training. The loss function is a function of the expected output (or ground truth) values ​​for the training data 528 and the values ​​output by the network. The information contained in the loss function may be sent through the machine learning system 344 as backpropagation to adjust the internal parameters, thereby tuning the machine learning system 344 to provide valid outputs. All else being equal, the more training data 528 used to train the machine learning system 344, the more reliable the machine learning system 344 may be (e.g., at providing accurate segmentation and / or classification).

[0082] For example, the loss functions may include support vector machine (SVM) loss, hinge loss, etc. The loss functions described above may be used to train the machine learning system 344 to segment the sensor data, but any other function of the input data with expected, or ground truth, segmented data is contemplated.

[0083] In any of the above-referenced examples, one sensor modality may inform any other sensor modality. As a non-limiting example, LIDAR sensor data 112 may indicate an object (whether static or dynamic) in the environment 108 closest to the vehicle, which may be determined based on, for example, LIDAR feature tracking, LIDAR segmentation, LIDAR classification, etc. In the above example, the object determined in LIDAR sensor data 112 may be used to determine expected sensor returns in the remaining sensor modalities (e.g., in other LIDAR sensors, in image data 110, or in sensor data 114). A discrepancy between the expected object detection and measuring data may indicate a malfunction of the respective sensor, a miscalibration of the respective sensor, etc. In any of the examples described herein, one or more signals and / or corresponding image data 110, LIDAR sensor data 112, and / or sensor data 114 may be input to a machine learning system 344, and a response system 124 may initiate a corresponding response and / or other action 126 based at least in part on the output of the machine learning system 344. It is understood that the machine learning system 344 and / or training data 528 may be configured and / or utilized to train the image capture device 102, the LIDAR sensor 104, the sensor 106, and / or any of the other sensor modalities described herein.

[0084] 5 , in some examples, a confidence level may be associated with the identification of the object 122, and in the above example, the response system 124 may be configured to initiate a response and / or any other action 126 when the confidence level associated with the identification of the object 122 falls below a threshold confidence level. For example, these may be used to account for inaccuracies associated with the image data 110, the LIDAR sensor data 112, and / or the sensor data 114, such as one or more of the responses and / or any other action 126 described herein, and may improve operation of the vehicle 202.

[0085] In some examples, the perception system 116 may be connected to the vehicle (e.g., either physically or via a communication system), and initiating the response and / or any other action 126 may include one or more of initiating communication with a teleoperation system 530 configured to assist in the operation of the vehicle or generating a verification request, e.g., via communication connection(s) 310 ( FIG. 3 ). In some examples, generating the verification request may include initiating notification of a vehicle service center 532 regarding the absence of one or more identified objects 122 as obstacles. For example, the vehicle may be configured to operate according to an assistance mode, wherein the teleoperation system 530, which may be located remotely from the vehicle, may receive one or more signals from the vehicle regarding the operation (e.g., via communication connection(s) 310 and / or an associated communication network). For example, the teleoperation system 530 may be configured to verify the presence or absence of the object 122 in the environment 108 based on one or more signals received from the vehicle, e.g., via a teleoperator and / or one or more methods described herein. The teleoperation system 530 may be configured to send one or more signals to the vehicle that cause the vehicle to initiate a response and / or any other action 126, as described herein, that takes into account inaccuracies associated with the image data 110, the LIDAR sensor data 112, and / or the sensor data 114. By way of non-limiting example, such teleoperator actions may include sending a command to stop the vehicle, adjusting weights assigned to various sensor modalities, sending a command to eliminate and / or mitigate errors, etc.

[0086] As mentioned above, in some examples, initiating the response and / or other action 126 may include reversing the vehicle's direction of travel, for example, by communicating with drive module(s) 314 ( FIG. 3 ). In some examples, the vehicle may be a bi-directional vehicle configured to operate with generally equal performance in either a first direction or a second, opposite direction, for example, as described herein. In the above example, the vehicle may have at least similar sensors at both ends of the vehicle, and the drive module(s) 314 may be configured to cause the vehicle to operate in the opposite direction of travel if the perception system 116 and / or the object detection system 118 identify an object 122 as present in the environment 108 and determine a corresponding error associated with data included in one or more respective sensor signals received by the perception system 116 (e.g., determining that the identified object 122 is not in or has been misclassified as one or more of the aforementioned groups of objects).

[0087] Furthermore, as mentioned above, the responses and / or other actions 126 may include, among other things, at least one of ignoring portions of the image data 110, the LIDAR sensor data 112, and / or the sensor data 114. Additionally, the responses and / or other actions 126 may also include modifying weights, confidence values, and / or other metrics associated with one or more image capture devices 102, the LIDAR sensor 104, one or more additional sensors 106, other sensors of the sensor system(s) 306 ( FIG. 3 ), and / or signals received from each sensor of the sensor system(s) 306. Additionally, the responses and / or other actions 126 may include modifying the training data 528 described above.

[0088] Further, in any of the examples described herein, the perception system 116, the object detection system 118, the machine learning system 344, and / or other systems of the present disclosure may be configured to compare the various groups of objects described above with one another and may initiate responses and / or other actions 126 based at least in part on the comparisons. For example, the perception system 116 may be configured to compare the groups of objects 502, 508, 514 and / or other groups of objects associated with the sensor data collected at time T1 with one another. The object detection system 118 of the perception system 116 may be configured to determine, for example, whether any differences exist between the groups of objects 502, 508, 514 and / or between any parameters associated with the groups of objects 502, 508, 514. For example, the object detection system 118 may be configured to determine that one or more objects included in the group of objects 502 are not in the group of objects 508 and / or the group of objects 514 (e.g., a false negative). In a further example, object detection system 118 may be configured to determine that one or more objects included in group of objects 502 are different (e.g., false positive or misclassified) from corresponding objects included in group of objects 508 and / or group of objects 514. In the above example, response system 124 may initiate a response and / or any other action based at least in part on the determination. It is further understood that perception system 116 may be configured to compare groups of objects 504, 510, 516 and / or other groups of objects collected at time T2 to each other, etc.In any of the examples described herein, comparing groups of objects (e.g., groups of objects 502, 508, 514) to one another and / or comparing any of the respective groups of objects (e.g., groups of objects 502, 508, 514) to the fused sensor data 134 may include associating individual objects included in each group of objects with one another for purposes of comparison. In some examples, the above-described “object association” process may include projecting data or objects from one sensor data set onto another. As a non-limiting example, data indicating group of objects 502 included in image data 110 may be projected (or otherwise associated) to LIDAR sensor data 112 including group of objects 508 and / or to sensor data 114 including group of objects 514 to determine objects in the group that correspond to the same object in the environment. Additionally or alternatively, at least a portion of the fused sensor data 134 (e.g., one or more objects included in the group of objects 522) may be associated with a portion of the image data 110, a portion of the LIDAR sensor data 112, and / or a portion of the sensor data 114 to determine which objects in the group of objects 522 detected in the fused sensor data 134 correspond to objects in the groups of objects 502, 508, and / or 514. For example, as part of the object association process described above, the perception system 116 may project (or otherwise associate) one or more objects included in the group of objects 522 onto scenes D, E, and / or F to determine which objects, if any, in the group of objects 522 correspond to any of the groups of objects 502, 508, 514.Alternatively, one or more objects included in and / or identified in each object group 502, 508, 514 may be projected into the fused sensor data 134 (e.g., object group 522). The above object association processes may be performed by the perception system 116 using one or more data alignment, feature matching, and / or other data mapping techniques. In any of the above object association processes, the perception system 116 may correlate the output of each sensor modality with a particular object and / or with its respective location. As a result, the individual outputs for each sensor modality can be compared for fidelity. In some examples, any of the above-described object association processes may be performed by the perception system 116 as part of one or more of data association, object recognition, object characterization, and / or other processes described herein.

[0089] 6 is a flow diagram of an example process 600 illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that may be implemented in hardware, software, or a combination thereof. In a software context, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order and / or in parallel to implement a process.

[0090] 6 may be employed to compare individual sensor data with fused sensor data and identify and / or otherwise determine errors associated with data included in one or more respective sensor data received by the perception system 116 (e.g., determine a particular object that is present in the fused sensor data but is not present or misclassified in a group of one or more objects associated with the various sensor signals). Furthermore, the process 600 may include initiating a response or other action based, at least in part, on determining that the error exists (e.g., based, at least in part, on identifying the object). Additionally or alternatively, the process 600 may include training a machine learning system based, at least in part, on determining that the error exists. Any of the steps associated with the process 600 may be performed online (e.g., by one or more processors 316 located on the vehicle 302) and / or offline (e.g., by one or more remote computing devices 334). At 602, the example process 600 may include receiving, by one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334 (e.g., one or more server computers or other remote computing devices), a first signal from a first sensor. In the above example, the first sensor may include, among other things, one or more image capture devices 102 and / or one or more other sensors described herein with respect to the sensor system(s) 306. Additionally, the above first sensor may be located on and / or otherwise coupled to the vehicle 302. In some examples, the vehicle 302 may include an autonomous vehicle, a semi-autonomous vehicle, and / or any other vehicle known in the industry.The first signal received at 602 may include first data (e.g., image data 110) and / or other data representative of the environment 108. For example, the image data 110 received at 602 may represent a first scene A detected by the image capture device 102. As described above with respect to at least FIG. 1 , in some examples, the first scene A may include a group of first objects 128 detected in the environment 108 in which a first sensor is present. In particular, the first scene A may include a group of first objects 128 actually detected in the environment 108 by the first sensor (e.g., the image capture device 102). In any of the examples described herein, at 602, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334 may determine the group of first objects 128 using one or more segmentation, classification, and / or other data analysis processes based at least in part on the first data (e.g., the image data 110).

[0091] At 604, the example process 600 may include receiving, by one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334, a second signal from a second sensor (e.g., the LIDAR sensor 104) disposed on the vehicle 302. In the above example, the second signal received at 604 may include second data (e.g., the LIDAR sensor data 112) and / or other data representative of the environment 108. For example, the LIDAR sensor data 112 received at 604 may represent a second scene B detected by the LIDAR sensor 104. In some examples, the second scene B may be detected by the second sensor at the same time (e.g., substantially simultaneously) as the first scene A is detected by the first sensor. Additionally, as described above with respect to FIG. 1 , the second scene B may include a second group of objects 130 detectable in the environment 108. In particular, scene B may include a second group of objects 130 that are actually detected by a second sensor (e.g., LIDAR sensor 104) in the environment 108. In examples where scene B is detected by a second sensor at the same time that scene A is detected by the first sensor, the first group of objects 128 may be substantially identical to the second group of objects 130. In any of the examples described herein, at 604, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine the second group of objects 130 using one or more segmentation, classification, and / or other data analysis processes based at least in part on the second data (e.g., LIDAR sensor data 112).

[0092] In a further example, at 606, the example process 600 may include receiving, by one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334, a third signal from one or more additional sensors 106 (e.g., RADAR sensors) disposed on the vehicle 302. In the above example, the third signal received at 606 may include third data (e.g., sensor data 114) and / or other data representative of the environment 108. For example, the sensor data 114 (e.g., RADAR sensor data) received at 606 may represent a third scene C detected by one or more additional sensors 106 (e.g., by the RADAR sensors). The third scene C may be detected by the one or more additional sensors 106 at the same time (e.g., substantially simultaneously) as a first scene A is detected by the first sensor and / or a second scene B is detected by the second sensor. Additionally, as described with respect to FIG. 1 , the third scene C may include a third group of objects 132 detectable in the environment 108. In particular, the scene C may include the third group of objects 132 that are actually detected in the environment 108 by one or more additional sensors 106. In an example where the scene C is detected by one or more additional sensors 106 at the same time that the scene A is detected by the first sensor, the third group of objects 132 may be substantially identical to the first group of objects 128, or at least have some data in common (e.g., by having overlapping fields of view). In an example where the scene C is detected by one or more additional sensors 106 at the same time that the scene B is detected by the second sensor, the third group of objects 132 may be substantially identical to the second group of objects 130, or at least have some data in common (e.g., by having overlapping fields of view).In any of the examples described herein, at 606, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine the third group of objects 132 using one or more segmentation, classification, and / or other data analysis processes based at least in part on the third data (e.g., sensor data 114).

[0093] At 608, the perception system 116 may determine a fourth group of objects 120 based at least in part on the first data received at 602 (e.g., image data 110), the second data received at 604 (e.g., LIDAR sensor data 112), and / or the third data received at 606 (e.g., sensor data 114). For example, at 608, the perception system 116 may generate and / or otherwise determine fused sensor data 134 predicted, determined, and / or indicated by the perception system 116 as being present in the environment 108. The perception system 116 may generate such fused sensor data 134 at 608 based at least in part on information contained in one or more signals received at 602, 604, and / or 606 using any of the fusion techniques described herein. For example, the perception system 116 may generate the above-described fused sensor data 134 based at least in part on the received image data 110, the received LIDAR sensor data 112, and / or the received sensor data 114, and the fused sensor data 134 generated at 608 may include the above-described fourth group of objects 120. In the above example, the group of objects 120 (e.g., the fused sensor data 134) may include objects predicted, determined, and / or indicated by the perception system 116 as being present in the environment 108. As described herein, the above-described fused sensor data 134 may be used by the perception system 116 and / or by the machine learning system 344 as ground truth information to train individual sensor modalities.

[0094] At 610, the perception system 116 may compare the first data received at 602 (e.g., the image data 110), the second data received at 604 (e.g., the LIDAR sensor data 112), the third data received at 606 (e.g., the sensor data 114), and / or any other data included in the one or more signals received at 602, 604, and / or 606 to the fused sensor data 134 generated at 608 to determine whether a discrepancy exists between the fused sensor data 134 and the data received from the individual sensor modalities. For example, at 610, the perception system 116 may compare the first object group 128, the second object group 130, and the third object group 132 with the fourth object group 120 determined at 608 to determine whether one or more errors exist in the respective object groups 128, 130, 132. In the above comparison, the fourth group of objects 120 may be treated as ground truth because the fourth group of objects 120 was generated using data (e.g., image data 110, LIDAR sensor data 112, sensor data 114, etc.) from multiple sources (e.g., image capture device 102, LIDAR sensor 104, one or more additional sensors 106, etc.).

[0095] At 612, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine whether the first group of objects 128, the second group of objects 130, and / or the third group of objects 132 contain errors (e.g., with respect to the assumed ground truth) with respect to the group of objects 120. For example, at 612, one or more processor(s) 316 and / or other components of the vehicle computing device 304 may determine whether one or more objects 122 included in the fused sensor data 134 (e.g., one or more objects 122 included in the group of objects 120) are not in or are misclassified as being in at least one of the groups of objects 128, 130, 132 corresponding to the signals received at 602, 604, 606, respectively. For example, at 612, the object detection system 118 and / or the perception system 116 may be configured to determine that one or more objects (e.g., object 122) included in the group of objects 120 are not in at least one of the groups of objects 128, 130, 132 (e.g., are false negatives associated with at least one of the groups of objects 128, 130, 132). In a further example, at 610, the object detection system 118 and / or the perception system 116 may be configured to determine that one or more objects (e.g., object 122) included in the group of objects 120 are different from corresponding objects included in at least one of the groups of objects 128, 130, 132 (e.g., are false positives or misclassifications associated with objects in at least one of the groups of objects 128, 130, 132).In still a further example, at 610, the object detection system 118 and / or the perception system 116 may be configured to determine and / or identify any of the other errors described herein. In any of the above examples, at 612, the object detection system 118 and / or the perception system 116 may determine that "yes," an error exists. For example, at 612, the perception system 116 may determine that "yes," an object (e.g., object 122) included in the fused sensor data 134 (e.g., included in object group 120) is not in or has been misclassified as part of at least one of object groups 128, 130, 132, and the system may proceed to 614 and / or 616. Alternatively, at 612, the object detection system 118 and / or the perception system 116 may determine that "no," an error does not exist. For example, at 612, the perception system 116 may determine that no objects included in the fused sensor data 134 (e.g., included in object group 120) are not in object groups 128, 130, 132 or are misclassified (e.g., objects identified in object groups 128, 130, 132 are each included in object group 120). In the above example, the system may proceed to 602.

[0096] At 614, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may use information contained in one or more signals received at 602, 604, and / or 606 as ground truth when training the machine learning system 344 described above. For example, at 614, the perception system 116 may identify which of the signals received at 602, 604, 606 correspond to a determined error and / or which of the signals include information that may be useful for training the machine learning system 344. The above information may be used at 614 as an example of ground truth for training the machine learning system 344. In some examples, the machine learning system 344, either alone or in combination with the object detection system 118 or other components of the perception system 116, may identify, classify, and / or categorize all objects detected in a scene (e.g., one or more of scenes A-G described above), and may identify one or more objects 122 that are missing or misclassified in one or more of the above scenes. In some examples, the processing associated with 614 may occur offline or online.

[0097] At 616, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may initiate a response and / or any other action 126 based at least in part on determining the error at 612. For example, at 616, one or more processor(s) 316 and / or other components of the vehicle computing device 304 may determine that the object 122 identified at 612 is not in at least one of the object groups 128, 130, 132 or is misclassified. For example, at 612, the perception system 116 may determine that the object 122 is not in the object group 130 and that the object 122 is included in the object group 128 and the object group 132. Based at least in part on the above determination, at 616, the response system 124 may initiate any of the responses described above. For example, the response may include at least one of ignoring portions of the image data 110, the LIDAR sensor data 112, and / or the sensor data 114, among others. Additionally, the response and / or other operations 126 initiated in 616 may include modifying weights, confidence values, and / or other metrics associated with one or more image capture devices 102, the LIDAR sensor 104, the one or more additional sensors 106, other sensors of the sensor system(s) 306, and / or signals received from the respective sensors of the sensor system(s) 306. Additionally, the response and / or other operations 126 initiated in 616 may include modifying training data 528 associated with one or more image capture devices 102, the LIDAR sensor 104, the one or more additional sensors 106, other sensors of the sensor system(s) 306.In some examples, the response and / or other actions 126 initiated at 616 may further include generating a verification request and / or sending the verification request to a service center 532 and / or a teleoperation system 530 using the communication connection(s) 310 for review by a human operator and / or a teleoperator. Additionally, the response and / or other actions 126 initiated at 616 may include controlling one or more drive module(s) 314 to vary the speed, direction, and / or other operating parameters of the vehicle 302. In some examples, the processing associated with 616 may occur offline or online.

[0098] As described above, method 600 is illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the software context, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described blocks can be combined in any order and / or in parallel to implement a process. In some embodiments, one or more blocks of processing may be omitted entirely. Furthermore, method 600 may be combined, in whole or in part, with other methods.

[0099] FIG. 7 is a flow diagram of another exemplary process 700 illustrated as a collection of blocks in a logical flow graph, which represent sequences of operations that may be implemented in hardware, software, or a combination thereof. In a software context, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described blocks can be combined in any order and / or in parallel to implement a process. In some respects, process 700 may be substantially similar and / or identical to process 600 described above with respect to FIG. 6. Where possible, like item numerals will be used hereinafter to describe aspects of process 700 that are substantially similar and / or identical to corresponding aspects of process 600.

[0100] In some examples, the process 700 of FIG. 7 may be employed to determine parameters associated with each group of objects associated with sensor data. The process 700 may include comparing various parameters to determine whether one or more differences exist between the parameters. Furthermore, the process 700 may include initiating a response or other action based at least in part on determining that the differences exist. Additionally or alternatively, the process 700 may include training a machine learning system based at least in part on determining that the differences exist. Any of the steps associated with the process 700 may be performed online (e.g., by one or more processors 316 located on the vehicle 302) and / or offline (e.g., by one or more remote computing devices 334).

[0101] At 702, the example process 700 may include receiving, by one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334 (e.g., one or more server computers or other remote computing devices), a first signal from a first sensor. In the above example, the first sensor may include, among other things, one or more image capture devices 102 and / or one or more other sensors described herein with respect to the sensor system(s) 306. Additionally, the first sensor may be located on and / or otherwise coupled to the vehicle 302. In some examples, the vehicle 302 may include an autonomous vehicle, a semi-autonomous vehicle, and / or any other vehicle known in the industry. The first signal received at 702 may include first data (e.g., image data 110) and / or other data representative of the environment 108. For example, the image data 110 received at 702 may represent a first scene A detected by the image capture device 102. In any of the examples described herein, at 702, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine a first group of objects 128 using one or more detection, segmentation, classification, and / or other data analysis processes based at least in part on the first data (e.g., the image data 110).

[0102] Further, at 702, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine one or more parameters (e.g., first parameters) associated with the first group of objects 128 using the first data. In any of the examples described herein, the parameters may include, among other things, a classification of the objects in the environment 108, a determination (i.e., detection) regarding the existence and / or presence of the objects in the environment 108, a location (e.g., location L) of the objects in the environment 108, an orientation of the objects, a number of objects, uncertainty, and / or any other characteristic, metric, or aspect of the objects.

[0103] At 704, the example process 700 may include receiving, by one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334, a second signal from a second sensor (e.g., the LIDAR sensor 104 or any other sensor of the sensor system(s) 306) disposed on the vehicle 302. In the above example, the second signal received at 704 may include second data (e.g., the LIDAR sensor data 112) and / or other data representative of the environment 108. For example, the LIDAR sensor data 112 received at 704 may represent a second scene B detected by the LIDAR sensor 104. As described above with respect to FIG. 1 , the second scene B may include a second group of objects 130 detectable in the environment 108. In particular, scene B may include a second group of objects 130 that are actually detected by a second sensor (e.g., LIDAR sensor 104) in the environment 108. In examples where scene B is detected by a second sensor at the same time that scene A is detected by the first sensor, at least one of the first group of objects 128 may be substantially identical to at least one of the second group of objects 130. In any of the examples described herein, at 704, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine the second group of objects 130 using one or more detection, segmentation, classification, and / or other data analysis processes based at least in part on the second data (e.g., LIDAR sensor data 112).Further, at 704, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334 may determine one or more parameters (e.g., second parameters) associated with the second group of objects 130 using the second data. In any of the examples described herein, the parameters may include any of the parameters described above with respect to 702.

[0104] In a further example, at 706, the example process 700 may include receiving, by one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334, a third signal from one or more additional sensors 106 (e.g., a RADAR sensor, or any other sensor of the sensor system(s) 306) disposed on the vehicle 302. In the above example, the third signal received at 706 may include third data (e.g., sensor data 114) and / or other data representative of the environment 108. For example, the sensor data 114 (e.g., RADAR sensor data) received at 706 may represent a third scene C detected by the one or more additional sensors 106 (e.g., by a RADAR sensor). As described with respect to FIG. 1 , the third scene C may include a third group of objects 132 detectable in the environment 108. In particular, scene C may include a third group of objects 132 that are actually detected in the environment 108 by one or more additional sensors 106. In any of the examples described herein, at 706, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine the third group of objects 132 using one or more detection, segmentation, classification, and / or other data analysis processes based at least in part on the third data (e.g., sensor data 114). Further, at 706, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine one or more parameters (e.g., third parameters) associated with the third group of objects 132. In any of the examples described herein, the parameters may include any of the parameters described above with respect to 702.

[0105] At 708, the perception system 116 may determine a fourth parameter associated with a fourth group of objects based at least in part on the first data received at 702 (e.g., image data 110), the second data received at 704 (e.g., LIDAR sensor data 112), and / or the third data received at 706 (e.g., sensor data 114). For example, at 708, the perception system 116 may generate and / or otherwise determine fused sensor data 134 predicted, determined, and / or indicated by the perception system 116 as being present in the environment 108. The perception system 116 may generate such fused sensor data 134 at 708 based at least in part on information contained in one or more signals received at 702, 704, and / or 706 using any of the fusion techniques described herein. For example, the perception system 116 may generate the above-mentioned fused sensor data 134 based at least in part on the received image data 110, the received LIDAR sensor data 112, and / or the received sensor data 114, and the fused sensor data 134 generated at 708 may include the fourth group of objects 120. In the above example, the group of objects 120 (e.g., the fused sensor data 134) may include objects predicted, determined, and / or indicated by the perception system 116 as being present in the environment 108. Accordingly, at 708, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or the remote computing device 334 may determine one or more parameters (e.g., a fourth parameter) associated with the group of objects 120. In any of the examples described herein, the above-mentioned parameters may include any of the parameters described above with respect to 702.

[0106] At 710, the perception system 116 may determine one or more corresponding objects among the groups of objects 120, 128, 130, 132. The perception system 116 may then compare the fourth parameter determined at 708 with the first parameter determined at 702, the second parameter determined at 704, and / or the third parameter determined at 706 for the corresponding objects (i.e., detected objects that represent the same object in the environment) to determine whether there is a difference in the parameters (and / or whether the difference exceeds a threshold). For example, at 710, the perception system 116 may compare a classification of an object in the environment 108 (including an indication of certainty regarding said classification), a determination regarding the existence and / or presence of an object in the environment 108 (including uncertainty regarding said detection), a location (e.g., location L) of an object in the environment 108 (including uncertainty regarding said location), an orientation of the object (including uncertainty regarding said uncertainty), a number of objects, uncertainty, and / or any other first parameter determined at 702 with the corresponding parameter of the object determined at 708. Further, at 710, the perception system 116 may make a similar comparison between the second and third parameters determined at 704 and 706 with the corresponding parameters determined at 708.

[0107] In a further example, at 710, the perception system may compare the first data received at 702 (e.g., image data 110), the second data received at 704 (e.g., LIDAR sensor data 112), the third data received at 706 (e.g., sensor data 114), and / or any other data included in one or more signals received at 702, 704, and / or 706 with the fused sensor data 134 described above to determine whether a discrepancy exists between the fused sensor data 134 and the data received from the individual sensor modalities. In the above example, at 710, the recognition system 116 may compare the first group of objects 128, the second group of objects 130, and the third group of objects 132 with the fourth group of objects 120 determined at 708 to determine whether differences exist between the groups of objects 128, 130, 132 and the group of objects 120 and / or determine whether one or more errors exist in the respective groups of objects 128, 130, 132.

[0108] At 712, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may determine whether a difference exists (and / or whether the difference exceeds a threshold) between the parameter determined at 708 and any of the parameters determined at 702, 704, and / or 706. For example, in an example where the first parameter determined at 702 includes a first classification of an object 122 in the environment 108, the second parameter determined at 704 includes a second classification of the object 122, the third parameter determined at 706 includes a third classification of the object 122, and the fourth parameter determined at 708 includes a fourth classification of the object 122, the difference identified at 712 may include a difference between the first, second, or third classification and the fourth classification, and / or a difference in the uncertainty regarding the determination of the classification. Similarly, in an example where the first parameter determined at 702 comprises a first decision regarding the presence of object 122 in environment 108, the second parameter determined at 704 comprises a second decision regarding the presence of object 122, the third parameter determined at 706 comprises a third decision regarding the presence of object 122, and the fourth parameter determined at 708 comprises a fourth decision regarding the presence of object 122, the difference identified at 712 may include a difference between the first, second, or third decision and the fourth decision and / or an uncertainty associated with the decision.In any of the examples described herein, the difference in one or more parameters identified at 712 may be determined based at least in part on collected data, e.g., by the image capture device 102, the LIDAR sensor 104, the one or more additional sensors 106, and / or any other sensor of the present disclosure, at a time later than the particular time at which the first data associated with the first signal received at 702, the second data associated with the second signal received at 704, and / or the third data associated with the signal received at 706 are collected. For example, in some embodiments, the first group of objects 128 indicated by the image data 110 included in the first signal may be detected by the image capture device 102 at a first time t1. In the above embodiments, the second group of objects 130 indicated by the LIDAR sensor data 112 included in the second signal may be detected by the LIDAR sensor 104 at a first time T1. In the above example, at 712, the parameter difference may be determined based at least in part on additional image data 110 and / or additional LIDAR sensor data 112 detected at a second time T2 that is later than the first time T1.

[0109] In a still further example, at 712, one or more processor(s) 316 and / or other components of the vehicle computing device 304 may determine whether parameter differences exist by determining whether one or more objects 122 included in the fused sensor data 134 (e.g., one or more objects 122 included in the group of objects 120) are missing from or misclassified in at least one of the groups of objects 128, 130, 132 corresponding to signals received at 702, 704, 706, respectively. In a still further example, it is understood that two or more groups of objects 128, 130, 132 described herein may include at least one common object. In the above example, process 700 may include identifying a common object based at least in part on data collected simultaneously by various separate sensors. Additionally or alternatively, the common object may be identified based at least in part on data collected at separate respective times by one or more sensors. In some examples, the first parameter determined at 702 may include a first location of a common object (e.g., object 122), the second parameter determined at 704 may include a second location of the common object, the third parameter determined at 706 may include a third location of the common object, and the fourth parameter determined at 708 may include a fourth location of the common object. Thus, in the above example, the difference identified at 712 may include a difference between at least two of the locations and / or uncertainties in the determination. For example, the difference identified at 712 may include a difference between the first location and the second location. In a still further example, the difference identified at 712 may include a difference between the first location and the fourth location.

[0110] 7 , in any of the examples described above, at 712, the object detection system 118 and / or the perception system 116 may determine that "yes," a difference exists (and / or is equal to or greater than a threshold) in the various parameters being compared. For example, at 712, the perception system 116 may determine that "yes," a first classification (e.g., first parameter) of an object associated with the first group of objects 128 is different from a fourth classification (e.g., fourth parameter) of an object associated with the fourth group of objects 120, and the system may proceed to 714 and / or 716. Alternatively, at 712, the object detection system 118 and / or the perception system 116 may determine that "no," a difference does not exist in the various parameters being compared. For example, at 712, the perception system 116 may determine that "no," no classification of an object associated with the groups of objects 128, 130, 132 is different from the classification of the corresponding object associated with the group of objects 120. In the above example, the system may proceed to 702.

[0111] At 714, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may use information contained in one or more signals received at 702, 704, 706 as ground truth when training the machine learning system 344 described above. For example, at 714, the perception system 116 may identify which of the one or more signals received at 702, 704, 706 are associated with the difference identified at 712. At 714, the perception system may also identify which of the one or more signals received at 702, 704, 706 include information that may be useful for training the machine learning system 344. The above information may be used at 714 as an example of ground truth for training the machine learning system 344. In some examples, the machine learning system 344, either alone or in combination with the object detection system 118 or other components of the perception system 116, may identify, classify, and / or categorize all objects detected in a scene.

[0112] At 716, one or more processor(s) 316 and / or other components of the vehicle computing device 304 and / or remote computing device 334 may initiate a response and / or any other action 126 based at least in part on determining the existence of a discrepancy at 712. Substantially, the response may be similar and / or identical to one or more responses described above with respect to at least FIG.

[0113] As described above, process 700 is illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the software context, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described blocks can be combined in any order and / or in parallel to implement a process. In some embodiments, one or more blocks of the process may be omitted entirely. Furthermore, process 700 may be combined, in whole or in part, with other processes, such as process 600. Similarly, process 600 may be combined, in whole or in part, with process 700.

[0114] The systems, modules, and methods described herein may be implemented using any combination of software and / or hardware elements. The systems, modules, and methods described herein may be implemented using one or more virtual machines operating alone or in combination with one another. Any applicable virtualization solution may be used to implement a physical computing machine platform as a virtual machine running under the control of virtualization software running on a hardware computing platform or host.

[0115] Example clauses A. An exemplary system includes one or more processors, communicatively coupled to the one or more processors, for receiving, by the one or more processors, a first signal from an image capture device, the first signal including image data representing a first portion of an environment, determining a first group of objects associated with the environment based at least in part on the image data, and transmitting a second signal to an image capture device, communicatively coupled to the one or more processors, the first signal including image data representing a first portion of an environment, determining a first group of objects associated with the environment based at least in part on the image data, and transmitting a second signal to an image capture device, communicatively coupled to the one or more processors, the first signal including image data representing a first portion of an environment, the first signal including image data representing a first portion of an environment, the second ... and one or more computer-readable storage media storing executable instructions to at least one of: receive a second signal from a moving object (e.g., a moving ranging) sensor, the second signal including LIDAR sensor data representative of a second portion of the environment, the second portion at least partially overlapping the first portion; determine a second group of objects associated with the environment based at least in part on the LIDAR sensor data; determine a third group of objects based at least in part on the image data and the LIDAR sensor data; identify a first object included in the third group of objects; determine that the first object is present in the first and second portions of the environment and is not or misclassified in at least one of the first group of objects or the second group of objects; and initiate a response based at least in part on the identifying the first object; or train a machine learning system in communication with the one or more processors based at least in part on the first signal or the second signal.

[0116] B. The system of clause A is further executable by the one or more processors to receive a third signal from an additional sensor, the third signal including additional sensor data representative of a third portion of the environment, and determine a fourth group of objects associated with the environment based at least in part on the additional sensor data, and the first object is present in the third portion of the environment and in the fourth group of objects.

[0117] C. The system of clause A or B, wherein the image sensor and the LIDAR sensor are disposed on the autonomous vehicle, and the response includes ignoring a portion of the image data or LIDAR sensor data, modifying weights associated with the image data or LIDAR sensor data, and modifying training data used by the machine learning system to train the image capture sensor or the LIDAR sensor to generate a validation request or cause the autonomous vehicle to change direction.

[0118] D. The method includes receiving, by one or more processors, a first signal from a first sensor, the first signal including first sensor data representative of an environment; determining, by the one or more processors, a first parameter associated with a first group of objects based at least in part on the first sensor data; receiving, by one or more processors, a second signal from a second sensor, the second signal including second sensor data representative of the environment; determining, by the one or more processors, a second parameter associated with a second group of objects based at least in part on the second sensor data; Thus, the method includes determining a third parameter associated with the third group of objects based at least in part on the first sensor data and the second sensor data; comparing, by one or more processors, the first parameter or the second parameter to the third parameter and identifying a difference between the third parameter and the first parameter or the second parameter; and at least one of initiating, by the one or more processors, a response based at least in part on identifying the difference; or training a machine learning system in communication with the one or more processors based at least in part on the first signal or the second signal.

[0119] E. The method of clause D, wherein the first sensor includes a LIDAR sensor, the first sensor data includes LIDAR sensor data, the first parameter includes a first classification of an object in the environment, the second sensor includes an image capture device, the second sensor data includes image data, the second parameter includes a second classification of the object, the third parameter includes a third classification of the object, and the difference includes a difference between the first classification or the second classification and the third classification.

[0120] F. The method of clause D or E, wherein the first sensor includes a LIDAR sensor, the first sensor data includes LIDAR sensor data, the first parameter includes a first determination regarding the presence of an object in the environment, the second sensor includes an image capture device, the second sensor data includes image data, the second parameter includes a second determination regarding the presence of an object in the environment, the third parameter includes a third determination regarding the presence of an object in the environment, and the difference includes a difference between the first determination or the second determination and the third determination.

[0121] G. The method of clause D, E, or F further includes collecting a plurality of first sensor data over a period of time; determining, by the one or more processors, first statistical information based on the plurality of first sensor data; comparing, by the one or more processors, the first statistical information to the stored statistical information; and initiating a response based at least in part on the comparing.

[0122] H. The method of clauses D, E, F, or G, wherein the first sensor and the second sensor are disposed on the vehicle, and the response includes at least one of ignoring a portion of the first sensor data, ignoring a portion of the second sensor data, modifying a first weight associated with the first signal, modifying a second weight associated with the second signal, modifying training data associated with the first sensor, modifying training data associated with the second sensor, generating a validation request, or causing the vehicle to turn.

[0123] I. The method of clause D, E, F, G, or H, wherein at least one of the first parameter, the second parameter, or the third parameter includes an object location, an object orientation, a number of objects, an uncertainty, a determination of object presence, or an object classification.

[0124] J. The method of clause D, E, F, G, H or I further includes receiving a third signal from a third sensor, the third signal including third sensor data representative of the environment; determining a fourth parameter associated with the group of fourth objects based at least in part on the third sensor data; and identifying a difference between the fourth parameter and the first parameter or the second parameter, wherein the first sensor includes an image capture device, the second sensor includes a LIDAR sensor, and the third sensor includes a RADAR sensor.

[0125] K. The method of clause D, E, F, G, H, I, or J, wherein first sensor data is detected by a first sensor at a first time and second sensor data is detected by a second sensor at the first time, the method further including identifying a difference at a second time, later than the first time, based at least in part on additional data collected by the first sensor or the second sensor.

[0126] L. The method of clause D, E, F, G, H, I, J or K, wherein identifying the difference includes determining that the object is not in or is misclassified as being in at least one of the first group of objects or the second group of objects.

[0127] M. The method of clause D, E, F, G, H, I, J, K, or L, wherein the first group of objects and the second group of objects include a common object, the first parameter includes a first location of the common object, the second parameter includes a second location of the common object, and the difference includes a difference between the first location and the second location.

[0128] N. The method of clause M, wherein determining the first parameter includes determining the first parameter using a first machine-learned model, determining the second parameter includes determining the second parameter using a second machine-learned model, the machine learning system includes the first machine-learned model or the second machine-learned model, and the method further includes training the machine learning system based at least in part on identifying the differences, and training the machine learning system includes training the first machine-learned model or the second machine-learned model using at least one of the first sensor data, the second sensor data, or a group of third objects as ground truth.

[0129] O. The method of clause D, E, F, G, H, I, J, K, L, M, or N includes the first parameter being associated with a first confidence level, the second parameter being associated with a second confidence level, and the response modifying the first confidence level or the second confidence level.

[0130] P. A computer-readable storage medium having stored computer-executable instructions that, when executed by a computer, cause the computer to at least one of receive a first signal from a first sensor, the first signal including first sensor data representative of an environment, determine a first parameter associated with a first group of objects based at least in part on the first sensor data, receive a second signal from a second sensor, the second signal including second sensor data representative of the environment, determine a second parameter associated with a second group of objects based at least in part on the second sensor data, determine a third parameter associated with a third group of objects based at least in part on the first sensor data and the second sensor data, identify a difference between the third parameter and the first parameter or the second parameter, and initiate, by one or more processors, a response based at least in part on identifying the difference; or train a machine learning system in communication with the one or more processors based at least in part on the first signal or the second signal.

[0131] Q. The computer-readable recording medium of clause P further includes the computer-executable instructions, when executed by a computer, causing the computer to receive a third signal from a third sensor disposed on the vehicle, the third sensor including a RADAR sensor, the third signal including third sensor data representative of the environment, and determining a fourth parameter associated with a fourth group of objects based at least in part on the third sensor data, and identifying a difference includes identifying a difference between the third parameter and the first parameter, the second parameter, or the fourth parameter.

[0132] R. The computer-readable storage medium of clause P or Q, wherein the first group of objects and the second group of objects include a common object, the first parameter includes a first classification of the common object, the second parameter includes a second classification of the common object, and the difference includes a difference between the first classification and the second classification.

[0133] S. The computer-readable storage medium of clause P, Q, or R, wherein at least one of the first parameter, the second parameter, or the third parameter includes an object location, an object orientation, a number of objects, an uncertainty, a determination of the presence of an object, or a classification of an object.

[0134] T. The computer-readable medium of clause P, Q, R, or S, wherein a first sensor and a second sensor are disposed on a vehicle, and the response includes at least one of ignoring a portion of the first sensor data, ignoring a portion of the second sensor data, modifying a first weight associated with the first signal, modifying a second weight associated with the second signal, modifying training data associated with the first sensor, modifying training data associated with the second sensor, generating a verification request, or causing the vehicle to turn.

[0135] termination The various techniques described herein may be implemented in the context of computer-executable instructions or software, such as program modules, that are stored in computer-readable storage and executed by a processor(s) of one or more computers or other devices, such as those illustrated in the figures. Generally, program modules include routines, programs, objects, components, data structures, etc., and that define operational logic for performing particular tasks or implement particular abstract data types.

[0136] Other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although a particular allocation of responsibilities is defined above for purposes of discussion, the various functions and responsibilities may be allocated and divided in different manners depending on the circumstances.

[0137] Similarly, software may be stored and distributed in a variety of ways and using different means, and the storage and execution configurations of the particular software described above may vary in many different ways. Thus, software implementing the techniques described above may be distributed on various types of computer-readable media and is not limited to the memory forms specifically described.

[0138] While the above discussion describes example implementations of the described techniques, it is contemplated that other architectures may be used to implement the described functionality and are within the scope of this disclosure. Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims. [Explanation of symbols]

[0139] 100 Processing 102 devices 104 LIDAR sensors 106 Sensors 108 Environment 110 Image Data 112 LIDAR sensor data 112 Image data 112 Sensor Data 114 Sensor Data 116 Cognitive Systems 118 Object Detection System 120 Groups 122 objects 124 Response System 126 Operation 128 Group 130 Groups 132 Groups 134 fused sensor data

Claims

1. one or more processors; When executed by the one or more processors, receiving first sensor data associated with a first sensor modality; receiving second sensor data based at least in part on the first sensor data, the second sensor data being associated with fused sensor data, the fused sensor data being associated with the first sensor modality and a second sensor modality different from the first sensor modality; determining a difference between parameters determined for a first group of one or more objects represented by the first sensor data and parameters determined for a second group of one or more objects represented by the second sensor data, wherein the parameters determined for the first group include a classification of the one or more objects represented by the first sensor data and the parameters determined for the second group include a classification of the one or more objects represented by the second sensor data; Based at least in part on said difference, Initiate a response or Do they output indications for training machine learning systems? Do at least one of the following: one or more non-transitory computer-readable media storing instructions for causing the one or more processors to A system equipped with

2. the response or the machine learning system is configured to ignore a portion of the first sensor data based on at least one object in the first group of one or more objects indicated by the first sensor data being absent from the second group of one or more objects indicated by the second sensor data. The system of claim 1 .

3. The instruction: determining a third object indicated by the fused sensor data that is not in the first group of one or more objects or that is misclassified; further executable by the one or more processors to: The system of claim 1 .

4. The instruction: comparing a first object classification associated with a first object in the first group of one or more objects to a second object classification associated with a second object in the second group of one or more objects; Identifying differences between the first object classification and the second object classification further causing the one or more processors to: the difference between the parameters determined for the first group and the parameters determined for the second group comprises the difference between the first object classification and the second object classification. The system of claim 3 .

5. receiving first sensor data associated with a first sensor modality; receiving second sensor data based at least in part on the first sensor data, the second sensor data being associated with fused sensor data, the fused sensor data being associated with the first sensor modality and a second sensor modality different from the first sensor modality; determining a difference between parameters determined for a first group of one or more objects represented by the first sensor data and parameters determined for a second group of one or more objects represented by the second sensor data, wherein the parameters determined for the first group include a classification of the one or more objects represented by the first sensor data and the parameters determined for the second group include a classification of the one or more objects represented by the second sensor data; Based at least in part on said difference, Initiating a response, or Output of indications for training machine learning systems and performing at least one of A method comprising:

6. the response or the machine learning system is configured to ignore a portion of the first sensor data based on at least one object in the first group of one or more objects indicated by the first sensor data being absent from the second group of one or more objects indicated by the second sensor data. The method of claim 5. determining a third object indicated by the fused sensor data that is not in the first group of one or more objects or that is misclassified; Further provided with The method of claim 5.

8. comparing a first object classification associated with a first object in the first group of one or more objects to a second object classification associated with a second object in the second group of one or more objects; identifying a difference between the first object classification and the second object classification; Furthermore, the difference between the parameters determined for the first group and the parameters determined for the second group comprises the difference between the first object classification and the second object classification. The method of claim 7.

9. comparing a first parameter associated with the first object to a second parameter associated with the second object; Identifying a difference between the first parameter and the second parameter; comparing a third parameter associated with the third object to the second parameter; determining a difference between the third parameter and the second parameter; Further provided with The method of claim 7.

10. determining that the first object is the same as the second object; determining that the third object is the same as the second object; Further provided with The method of claim 8.

11. collecting a plurality of the first sensor data over a period of time; determining first statistical information based on the plurality of first sensor data; comparing the first statistical information with stored statistical information; initiating said response if said first statistical information is outside a predetermined range of said stored statistical information; Further provided with The method of claim 5.

12. the first sensor modality is a first sensor disposed on a vehicle, and the second sensor modality is a second sensor disposed on the vehicle; The response may be: ignoring a portion of the first sensor data; ignoring a portion of the second sensor data; modifying a first weight associated with the first sensor data; modifying a second weight associated with the second sensor data; modifying training data associated with the first sensor; modifying training data associated with the second sensor; Generate a verification request; or causing said vehicle to change direction at least one of: The method of claim 5.

13. the first sensor data is indicative of objects in an environment; At least one of the first sensor data or the second sensor data includes at least one of information regarding the location of the object, the orientation of the object, uncertainty, a determination of the presence of the object, or a classification of the object. The method of claim 5.

14. the first sensor data is detected by a first sensor at a first time; The fused sensor data is obtained by fusing the first sensor data with sensor data detected by a second sensor at the first time, and the method further comprises: determining the difference based at least in part on additional data collected by the first sensor or the second sensor at a second time subsequent to the first time. Further provided with The method of claim 5.

15. When executed by a computer, receiving first sensor data associated with a first sensor modality; receiving second sensor data based at least in part on the first sensor data, the second sensor data being associated with fused sensor data, the fused sensor data being associated with the first sensor modality and a second sensor modality different from the first sensor modality; determining a difference between parameters determined for a first group of one or more objects represented by the first sensor data and parameters determined for a second group of one or more objects represented by the second sensor data, wherein the parameters determined for the first group include a classification of the one or more objects represented by the first sensor data and the parameters determined for the second group include a classification of the one or more objects represented by the second sensor data; Based at least in part on said difference, Initiate a response or Do they output indications for training machine learning systems? Do at least one of the following: A computer-readable medium having computer-executable instructions stored thereon for causing the computer to:

16. the response or the machine learning system is configured to ignore a portion of the first sensor data based on at least one object in the first group of one or more objects indicated by the first sensor data being absent from the second group of one or more objects indicated by the second sensor data.

16. The computer-readable storage medium of claim 15.

17. The computer-executable instructions may cause the computer to: determining a third object indicated by the fused sensor data that is not in the first group of one or more objects or that is misclassified; It is further feasible to 16. The computer-readable storage medium of claim 15.

18. the first sensor modality is a first sensor disposed on a vehicle, and the second sensor modality is a second sensor disposed on the vehicle; The system of claim 1 .

19. the first sensor data is indicative of objects in an environment; At least one of the first sensor data or the second sensor data includes at least one of information regarding the location of the object, the orientation of the object, uncertainty, a determination of the presence of the object, or a classification of the object. The system of claim 1 .

20. the first sensor modality is a first sensor disposed on a vehicle, and the second sensor modality is a second sensor disposed on the vehicle; 16. The computer-readable storage medium of claim 15.

Images