Intermediate input for machine learning models

Abstract

Techniques for determining classification probabilities of objects in an environment are discussed herein. The techniques may include analyzing sensor data associated with the environment from a perspective, such as a top-down perspective, using multi-channel data. From this perspective, the techniques may determine channels of multi-channel input data and additional feature data. Channels corresponding to spatial features are included in the multi-channel input data, and data corresponding to non-spatial features are included in the additional feature data. The multi-channel input data may be input to a first portion of a machine learning (ML) model, and the additional feature data may be concatenated with intermediate output data from the first portion of the ML model and input to a second portion of the ML model for further processing and determination of classification probabilities. The techniques may also be performed on a multi-resolution voxel space representing the environment.

Description

Technical Field

[0001] Cross - References to Related Applications This patent application claims priority to U.S. Utility Patent Application No. 17 / 137,873, filed December 30, 2020, and U.S. Utility Patent Application No. 17 / 137,947, filed December 30, 2020. Application Nos. 17 / 137,873 and 17 / 137,947 are hereby incorporated by reference herein.

Background Art

[0002] Autonomous vehicles can use sensors to capture data representing objects in the environment. Techniques can be used to determine the classification of such objects. Current techniques often involve processing large amounts of input data, which can pose challenges in resource - constrained environments.

Brief Description of the Drawings

[0003] The detailed description is explained with reference to the accompanying drawings. In the drawings, the left - most digit of the reference number identifies the drawing in which that reference number first appears. The use of the same reference number in different drawings indicates similar or identical components or features.

[0004] [Figure 1] FIG. 1 is a flow diagram of an exemplary process that receives sensor data of an environment, inputs input data into a first part of a machine learning (ML) model, inputs intermediate input data into a second part of the ML model, and determines classification probabilities of objects in the environment based on output data from the first and second parts of the ML model. [Figure 2A] FIG. 2A shows an example of a multi - resolution voxel space and multi - channel image data associated with the voxel space. [Figure 2B] FIG. 2B shows an exemplary processing pipeline for processing various ranges of an environment that can be rendered at different resolutions. [Figure 3A]Figure 3A shows an exemplary architecture of a system for processing multichannel image data using an ML model. [Figure 3B] Figure 3B shows an exemplary architecture of a system for processing multichannel image data through a first part of the ML model, processing intermediate input data through a second part of the ML model, and concatenating the results for further data processing. [Figure 4] Figure 4 is a block diagram of an exemplary system for implementing the technology described herein. [Figure 5] Figure 5 illustrates an exemplary process in which input data is fed into the first part of a machine learning (ML) model, intermediate input data is fed into the second part of the ML model, and the classification probability of objects in the environment is determined based on the output data from the first and second parts of the ML model. [Figure 6] Figure 6 shows an illustrative process in which first input data is input to a first ML model, second input data is input to a second ML model, a first classification associated with an object is received from the first ML model, a second classification associated with an object is received from the second ML model, and an autonomous vehicle is controlled based on the first or second classification. [Modes for carrying out the invention]

[0005] Techniques for processing sensor data to determine classifications associated with objects in an environment are described herein. For example, the technique may include generating environmental data representing the environment based on the sensor data. For example, such environmental data may be represented as multi-channel image data that can represent a top-down view of the environment. In some examples, the sensor data may include data associated with an environment in which one or more objects (e.g., pedestrians, vehicles, bicycles, etc.) may exist. The sensor data (or data based on sensor data) may include feature data representing spatial attributes associated with the objects (e.g., map information, occupancy layer, etc.) and / or non-spatial attributes associated with the objects (e.g., speed, acceleration, vehicle lights, etc.). The techniques described herein may include, but are not limited to, receiving sensor data, displaying sensor data (or data based on sensor data) as multi-channel image data from a “top-down view”, determining feature data associated with objects represented in the input data, inputting the input data into a first part of a machine learning (ML) model, determining intermediate input data based on concatenating intermediate output data received from the first part of the ML model with feature data, inputting the intermediate input data into a second part of the ML model, and determining classification probabilities associated with objects based on output data received from the second part of the ML model.

[0006] Additionally or alternatively, the techniques described herein, such as the process for processing multi-channel image data in a machine learning processing pipeline, may be executed several times in parallel, corresponding to the number of renderings of the environment as multi-channel image data, for example, by inputting a first subset of image data layers in a first part of the pipeline and a second subset of image data layers in a second part of the pipeline.

[0007] For example, a first portion of an environment can be rendered as a first multi-channel image data (e.g., top-down multi-channel image data) associated with a first extent (e.g., 50m x 50m). A second portion of the environment (e.g., a 100m x 100m area) can be rendered as a second multi-channel image data associated with a second extent. A third portion of the environment (e.g., 200m x 200m) can be rendered as a third multi-channel image. Any number of areas of the environment can be represented as multi-channel image data. In some examples, multi-channel image data can represent at least partially overlapping portions of the environment. In some examples, different multi-channel image data can be centered around common points. In some examples, each of the input data can be associated with the same "resolution".

[0008] Input data can be fed into different machine learning models in parallel to determine classification information for "pixels" or objects represented by the input data. Outputs associated with overlapping portions of the environment can be aggregated to determine the classification of objects represented by multiple input data. Classification probabilities may be passed to other components and may be used for generating inferences, determining vehicle trajectories, etc.

[0009] This separation of processing pipelines for each of the different ranges of the environment represented by the multichannel image data avoids processing constraints that may arise when processing all different ranges in a single pipeline. In some examples, expanding the range represented by the image data while using the same resolution increases processing and spatial requirements proportionally to the increase in range. For example, a first multichannel image data associated with a first range (e.g., 50m x 50m) at a first resolution (e.g., about 10cm per pixel) may be rendered and processed at substantially similar speeds to a second multichannel image data associated with a second range (e.g., 100m x 100m) at a second resolution (e.g., about 40cm per pixel), since substantially the same number of pixels are processed while the range is increasing. The larger second range has a coarser resolution than the shorter first range, but the first range may provide an acceptable resolution for identifying objects near a vehicle that require finer resolution pixels for processing, resulting in more accurate classification results. In addition, configuring separate processing pipelines to run in parallel and then aggregating the resulting data may further improve rendering and / or processing times.

[0010] As described above, sensor data can be represented as multi-channel image data associated with various parts of the environment. Feature data can be determined about regions of the environment. For example, feature data may include multiple spatial and non-spatial features. In some examples, spatial features may include, but are not limited to, map information of the environment (e.g., lane markings, crosswalks, traffic light locations, building locations, etc.), bounding boxes of objects in the environment, and sizes of objects in the environment. Additionally or alternatively, non-spatial features may include, but are not limited to, the velocity of objects in the environment, the acceleration of objects in the environment, the lighting state associated with lights on objects in the environment, and the behavior class of objects (e.g., aggressive, normal, etc.). In some examples, one or more features associated with the environment may be encoded as layers of multi-channel image data (e.g., multi-channel input images), or otherwise stored. Thus, the data can be efficiently represented for input to machine learning models.

[0011] For example, a machine learning model may accept data in the form of a top-down multi-channel input image. In particular, the model may accept data in the form of an image whose size is equal to the number of X and Y discretizations of the environment, and the image has a number of channels representing different spatial features of the environment.

[0012] In some examples, multichannel input data may include one or more layers of image data. For example, multichannel image data may include a first two-dimensional layer (e.g., the first channel) associated with first feature data and a second two-dimensional layer (e.g., the second channel) associated with second feature data. Additionally or alternatively, multidimensional image data may include a group of two-dimensional layers, such as the first and second channels, associated with feature data representing a single feature.

[0013] In some examples, multidimensional image data can include layers associated with multiple classes of semantic information, while in other examples, data associated with a particular classification can be omitted from multi-channel image data.

[0014] In some cases, various sensor data can be associated with multi-channel image data. For example, in some cases, multi-channel image data may represent LiDAR data, radar data, time-of-flight data, or any other depth data.

[0015] In some cases, the technologies discussed herein may be implemented in the context of vehicles such as autonomous vehicles. Autonomous vehicles may capture sensor data as they traverse their environment and determine multi-channel image data.

[0016] As described above, the technology may include performing an operation based on data associated with the environment, single-resolution multi-channel image data, and / or multi-resolution multi-channel image data. Using a first portion of sensor data represented by multi-channel image data and feature data determined from a second portion of the sensor data (the portions may overlap at least partially), the operation may include inputting the multi-channel image data into a first portion of an ML model to process the data and receive intermediate output data. In some examples, the multi-channel image data may represent a top-down view of the environment. Additionally or alternatively, the multi-channel image data may represent spatial features and / or objects within the environment. In some examples, the intermediate output data may represent dense features associated with the sensor data extracted from the multi-channel image data. Additionally or alternatively, the operation may include determining intermediate input data based on the intermediate output data and feature data, inputting the intermediate input data into a second portion of an ML model to process the intermediate input data, and receiving output data. In some examples, the output data may represent multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for each object in the environment represented by multichannel image data. Additionally or alternatively, the operation may further include determining one or more classification probabilities for the objects based on the output data.

[0017] It should be noted that either the input data or output data discussed herein can be represented by any data structure capable of storing feature data, such as tensors (e.g., numbers, scalars, arrays, vectors, two-dimensional arrays, matrices, etc.).

[0018] In some examples, the first renderer may be configured to generate multichannel input data, which may be configured as multichannel image data containing N channels, where N can be any integer corresponding to several features associated with sensor data. The N channels of the multichannel image data may then be divided into two subsets of channels based on feature data. For example, the N channels may be divided into a first A channel and a second B channel, where A can be any integer corresponding to several primary features, such as spatial features represented by the feature data, and B can be any integer corresponding to several secondary features, such as non-spatial features represented by the feature data, for each entity in the environment. The N channels of the multichannel image data may then be processed by the system at different points and by different parts of the ML model. For example, the input data containing the first A channel may be input to the first part of the ML model, where the data representing the first A channel may be transformed to represent all features of a given object, by aggregating all features associated with pixels corresponding to objects (or entities) in the environment. These A channels, when represented as object-specific features, can then be combined with B channels containing non-spatial features represented by feature data for each entity later in the processing pipeline.

[0019] This separation between the A-channel and the B-channel of the N-channel input image data results in a lower power cost requirement for execution, and only the first A-channel is input into the high-density backbone part of the pipeline with a smaller footprint, leading to a faster execution of the ML classification pipeline. Additionally, when processing all that is contained in the multi-channel input data spatially using a dense neural network, a proportionate increase in the weights corresponding to the number of channels is brought about. Therefore, reducing the number of channels input into the dense network from N channels to the first A-channel (where A is a subset of N) can result in a significant reduction in the inference time in proportion to the decrease in the number of channels.

[0020] In some examples, the first part of the ML model can be configured as the backbone or feature extraction component of a neural network, such as a convolutional neural network (CNN) for example. Exemplary backbones can include, but are not limited to, ResNet101, ResNet52, etc. The first part of the ML model can process the multi-channel image data including the first A-channel and determine intermediate output data corresponding to the multi-channel image data. For example, the intermediate output data can include a high-density representation (or a set of values) of the features associated with the first A-channel. In some examples, the high-density representation of the features can include feature vectors associated with the pixels represented in the multi-channel image data.

[0021] In some examples, a second renderer configured as a sparse renderer can be configured to generate additional feature data based on sensor data corresponding to specific features, such as a second B-channel representing non-spatial features for example. The additional feature data can include multiple features for each object represented by the sensor data. In some examples, the additional feature data can be configured as a feature vector associated with the object.

[0022] In some examples, intermediate input data may be determined based on concatenating intermediate output data and additional feature data object by object. For example, a single-channel image may be used to mask the intermediate output data, detect one or more objects, and / or determine one or more graphic references corresponding to specific locations, such as the center locations of one or more objects in multi-channel image data. In some examples, the first renderer may generate a single-channel image that includes representations of the centers of the objects contained in the image (or the respective centers of each object). The single-channel image may also include representations of occupations associated with objects in the environment. For example, the single-channel image may be aligned with the intermediate output data or otherwise mask the intermediate output data to provide representations of occupations and / or graphic references corresponding to the corresponding locations of objects in the intermediate output data. If there are no representations of occupations in the intermediate output data, the corresponding feature vector may be ignored; if there are representations of occupations in the intermediate output data, the corresponding feature vector may be collected. The collected feature vectors corresponding to a single object may be aggregated to determine a single feature vector associated with the object, which may be represented by a graphic reference corresponding to the corresponding location of the object in multi-channel image data. Furthermore, feature vectors represented by graphic references can be concatenated with additional per-object features to determine the intermediate input data (i.e., concatenate the feature vector associated with an object with the additional feature vector associated with that object).

[0023] The intermediate input data can be input into the second part of the ML model. In some examples, the second part of the ML model can be configured as a high-density neural network, such as a deep neural network (DNN) for example. The intermediate input data can pass through the second part of the ML model for each object in the environment, which can process the intermediate input data and determine output data representing multiple logits (e.g., a function representing probability values from 0 or negative infinity to 1 or infinity) for each object in the environment.

[0024] In some examples, one or more classification probabilities of the objects in the environment can be determined based on the output data. For example, the logits can indicate multiple probabilities associated with multiple classifications related to the object, such as object type classification (e.g., vehicle, pedestrian, bicycle, etc.), object behavior classification (e.g., assertive, normal, evasive, aggressive, etc.), object line-of-sight classification (e.g., distracted pedestrian, warning pedestrian, etc.), object trajectory classification (e.g., on the road, remaining on the sidewalk, jaywalking, etc.), lane change classification (e.g., changing lanes into my lane), and / or emergency vehicle classification, etc., but are not limited thereto. These classification probabilities are merely examples, and the technology is not limited to these specific examples.

[0025] The technologies discussed herein can improve the functionality of computing devices, such as computing devices in autonomous vehicles, in several ways. For example, using multiple levels within multi-resolution, multi-channel image data facilitates high-resolution management of data near the vehicle and low-resolution management of data far away from the vehicle. Such levels can significantly reduce the amount of memory required to store sensor data compared to input data associated with a single high-resolution level. In some examples, complex spatial and semantic data can be visually represented within multi-channel image data, enabling efficient evaluation and processing of the data. The technologies described herein can also reduce the latency of top-down predictions. As can be understood, reducing the latency of top-down predictions over an increased range can improve safety outcomes, particularly in the context of vehicles and autonomous vehicles. The technologies described herein can also improve safety outcomes by reducing the processing time for detecting and classifying objects, and by detecting and classifying objects over an even wider range. As can be understood, in the context of vehicles and autonomous vehicles, situations may arise where object determination and detection must be performed quickly and accurately. Furthermore, situations may arise where object determination and detection over a wider range must also be performed quickly and accurately. Therefore, the techniques described herein can improve the functionality of computing devices as well as improve safety outcomes.

[0026] The techniques described herein can be implemented in multiple ways. Exemplary implementations are provided below with reference to the drawings. Although described in the context of autonomous vehicles, the methods, apparatus, and systems described herein may be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. For example, similar techniques may be used in a driver-controlled vehicle that can provide an indication of whether it is safe for such a system to perform various operations. In another example, the techniques can be used in the context of a manufacturing assembly line, an aerial surveying context, or a navigational context. Furthermore, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0027] Figure 1 is a flowchart illustrating an exemplary process in which sensor data from the environment is received, input data is fed into the first part of a machine learning (ML) model, intermediate input data is fed into the second part of the ML model, and the classification probability of objects in the environment is determined based on the output data from the first and second parts of the ML model.

[0028] In operation 102, the process may include receiving sensor data of the environment and semantic information associated with the sensor data from sensors operating in conjunction with the perception system of a vehicle (e.g., an autonomous vehicle). Although described in the context of sensor data, operation 102 may include receiving any three-dimensional data or data that includes a depth component. Semantic information may include, for example, one or more semantic classifications, labels, or segmentation information. In some examples, operation 102 may include receiving multiple sensor datasets from multiple sensors operating in conjunction with the perception system. In some examples, operation 102 may include combining or merging data from two or more sensors (and / or over a period of time) into a single lidar dataset (also referred to as a "metaspin"). In some cases, operation 102 may include extracting portions of the sensor data for processing, such as over a period of time. In some examples, operation 102 may include receiving radar data (or other sensor data) and relating the radar data to the sensor data to generate a more detailed representation of the environment.

[0029] As described above, sensor data (e.g., sensor datasets) can be associated with semantic information. An example of such techniques for determining sensor data and associating it with semantic information is discussed, for example, in U.S. Patent Application No. 15 / 820,245, filed November 21, 2017, entitled "Sensor Data Segmentation," which is incorporated herein by reference in its entirety.

[0030] Semantic information may be associated with static and / or dynamic objects in the environment for classification and / or trajectory planning. Parts of the environment corresponding to the ground, static objects, and / or dynamic objects can be identified and labeled with such semantic information. In some examples, data may be segmented based at least in part on semantic information. In some examples, a list of dynamic objects can be provided to a tracking system to track and / or predict the trajectory of each dynamic object over time. In some examples, a list of static objects and / or identification of the ground can be provided to a planner system to generate a trajectory for a vehicle that traverses a drivable surface and avoids or otherwise takes into account the static objects identified herein.

[0031] Example 104 shows a vehicle 106 that captures sensor data 108 of the environment. In some examples, the sensor data 108 may include LiDAR data, radar data, sonar data, time-of-flight data, or other depth data. For example, operation 102 may include capturing image data and generating depth data based on the captured image data.

[0032] In operation 110, the process may include determining multi-channel image data 114 (e.g., a two-dimensional input grid) that represents a top-down view of the environment. In some examples, the multi-channel image data 114 may be determined based on sensor data.

[0033] As described above, in some examples, operation 110 may include determining multichannel image data 114 (e.g., a two-dimensional input grid) based on various parts of the environment. Each region or part of the multichannel image data 114 may be determined as a region of a two-dimensional representation associated with a region (or other dimension) of the environment. In some examples, each region of the multichannel image data 114 may be a cell of a two-dimensional representation associated with a region of the environment. Each region of the two-dimensional input grid may be associated with a 50cm × 50cm region and may contain feature data. Each region may be associated with one or more adjacent (e.g., neighbor) regions (e.g., may be placed adjacent to each other).

[0034] An example of multichannel image data 114 is shown in Example 112. For example, Example 112 includes a first channel 116 and a second channel 118. In some examples, the first channel 116 may represent a bounding box, location, extent (e.g., length and width), and / or any other representation of the presence of the object 120 in the environment. In some examples, the second channel 118 may represent feature data including spatial features 122 associated with the object (e.g., map information, occupancy layer, etc.).

[0035] Multichannel image data may contain multiple regions. Each region may be determined as a region in a two-dimensional representation, or it may be associated with a region of the environment. For example, a region may be associated with a region of the environment based on three-dimensional information associated with the environment, which has been converted into two-dimensional information associated with the region.

[0036] Each region may contain feature data. For example, but are not limited to, feature data may include multiple spatial features 122 and non-spatial features (for sensor data associated with a column). In some examples, spatial features 122 may include, but are not limited to, map information of the environment (e.g., lane signs, crosswalks, traffic light locations, building locations, etc.), bounding boxes of objects in the environment, and sizes of objects in the environment. Additionally or alternatively, non-spatial features may include, but are not limited to, the velocity of objects in the environment, the acceleration of objects in the environment, the lighting state associated with lights on objects in the environment, and the behavior class of objects (e.g., aggressive, normal, etc.). In some examples, one or more features associated with sensor data and / or 3D voxel space may be encoded as layers of a multi-channel input image, or otherwise stored. Thus, the data can be efficiently represented for input to a machine learning model.

[0037] In some examples, the first renderer may be configured to generate multichannel input data, which may be configured as multichannel image data 114 containing N channels, where N may be any integer corresponding to several features associated with the sensor data. The N channels of the multichannel image data 114 may be divided into two subsets of channels based on feature data. For example, the N channels may be divided into a first A channel and a second B channel, where A may be any integer corresponding to, for example, the number of spatial features represented by the feature data, and B may be any integer corresponding to the number of non-spatial features represented by the feature data. The N channels of the multichannel image data 114 may be processed by the system at different points and by different parts of the ML model. For example, input data containing the first A channel may be input to the first part of the ML model.

[0038] To ensure understanding, operation 110 may include generating or determining a multichannel image(s), as described herein. An example of generating or determining a multichannel image is described in relation to U.S. Patent Application No. 16 / 151,607, filed 4 October 2018, entitled “Trajectory Prediction in a Top-Down Scene.” U.S. Patent Application No. 16 / 151,607 is incorporated herein by reference in its entirety.

[0039] Operation 124 may include inputting multi-channel image data 114 (e.g., top-down or another view) to a first part 126 of an ML model configured as a backbone or feature extraction component of a neural network, such as a convolutional neural network (CNN). Illustrative backbones may include, but are not limited to, ResNet 101 and ResNet 52. The first part 126 of the ML model may process multi-channel image data, including a first A channel, to determine intermediate output data corresponding to the multi-channel image data 114.

[0040] Operation 128 may include determining the intermediate input data that will be input to the second part of the ML model. In some examples, determining the intermediate input data may be based on concatenating intermediate output data and additional feature data for each object in the environment.

[0041] In some examples, operation 128 may include utilizing a second renderer, such as a sparse renderer, which may be configured to generate additional feature data based on sensor data corresponding to specific features, such as a second B channel representing non-spatial features.

[0042] An example of intermediate input data is shown in Example 130. In some examples, Example 130 may include intermediate output data 134 and intermediate input data 132 determined based on additional feature data 136 (or features).

[0043] In some examples, the intermediate output data 134 may contain a high-density representation (or set of values) of features associated with the first A channel. In some examples, the high-density representation of features may contain feature vectors associated with pixels represented in the multi-channel image data 114.

[0044] In some examples, the additional feature data 136 may be based on sensor data 108 corresponding to specific features, such as non-spatial features. The additional feature data 136 may include multiple features for each object represented by the sensor data 108. In some examples, the additional feature data 136 may be structured as a feature vector associated with object 120.

[0045] In some examples, the intermediate input data 132 may be determined based on concatenating the intermediate output data 134 and additional feature data 136 object by object. For example, a single-channel image may be used to mask the intermediate output data 134 and detect one or more objects. In some examples, the first renderer may generate a single-channel image that includes a representation of the centers of the objects contained in the image (or the respective centers of each object). The single-channel image may also include a representation of the occupancy associated with object 120 in the environment. For example, the single-channel image may be aligned with the intermediate output data 134, or otherwise masked with the intermediate output data 134, to provide a representation of the occupancy relative to the intermediate output data 134. If there is no representation of occupancy in the intermediate output data 134, the corresponding feature vector may be ignored; if there is a representation of occupancy in the intermediate output data 134, the corresponding feature vector may be collected. The collected feature vectors corresponding to a single object may be aggregated to determine a single feature vector associated with object 120, which may be represented by the center of the object.

[0046] Furthermore, feature vectors represented by graphic references, such as the center of object 120, can be concatenated with additional per-object features to determine the intermediate input data 132 (i.e., concatenate the feature vector associated with an object with the additional feature vector associated with the object).

[0047] Operation 138 may include inputting the intermediate input data 132 to a second part 140 of the ML model, which is configured as a high-density neural network, such as a deep neural network (DNN). The intermediate input data 132 may also pass through the second part of the ML model for each object in the environment, which may process the intermediate input data 132 and determine output data representing multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for each object in the environment.

[0048] Operation 142 may include receiving output data from a second part 140 of the ML model. An example of output data is shown in Example 144. In some examples, Example 144 may include output data 146 representing multiple logits 148 (for example, functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for each object in the environment.

[0049] Operation 150 may include determining one or more classification probabilities for objects. In some examples, determining one or more classification probabilities may be at least in part based on object-specific logits 148 represented by output data 146.

[0050] An example of classification probabilities is shown in Example 152. In some examples, Example 152 may include classification probabilities 154 for object 120 in the environment. As previously mentioned, classification probabilities 154 may be based at least in part on object-specific logits 148 represented by output data 146.

[0051] For example, the logit 148 for each object may represent one or more classification probabilities 154 determined as multiple probabilities associated with multiple classifications associated with the object 120, such as, but are not limited to, object type classification (e.g., vehicle, pedestrian, bicycle, etc.), object behavior classification (e.g., assertive, normal, avoidant, aggressive, etc.), object gaze classification (e.g., distracted pedestrian, warning pedestrian, etc.), object trajectory classification (e.g., on the road, remaining on the sidewalk, jaywalking, etc.), lane change classification (e.g., changing lanes into my lane), and / or emergency vehicle classification.

[0052] Additionally, or alternatively, in operation 150, the process may include controlling the vehicle 106 based at least in part on processing classification probabilities 154. In some examples, operation 150 may be performed by the vehicle 106. In some examples, operation 150 may include generating routes, trajectories, and / or control signals for one or more systems of the vehicle 106 in order to navigate the vehicle 106 within the environment.

[0053] Figure 2 shows an example of a multi-resolution voxel space 202 and multi-channel image data associated with the voxel space 200. In some examples, the multi-resolution voxel space 202 may correspond to the exemplary voxel space described above with respect to Figure 1.

[0054] The multi-resolution voxel space 202 is represented in three dimensions (e.g., x, y, z) and includes a first level 204, a second level 206, and a third level 208.

[0055] A two-dimensional representation (for example, showing the xy-mode of the first level 204) is shown as the first level 210.

[0056] A two-dimensional representation (for example, showing the xy-mode of the second level 206) is shown as the second level 212.

[0057] A two-dimensional representation (for example, showing the xy-mode of the third level 208) is shown as the third level 214.

[0058] Example 216 shows levels 210, 212, and 214 (or levels 204, 206, and 208) juxtaposed with respect to vehicle 218. That is, voxel levels 210, 212, and 214 can correspond to a portion of the environment adjacent to vehicle 218.

[0059] As can be seen in Example 216, some parts of a multi-resolution voxel space can be associated with one level, two levels, three levels, or any number of levels. For example, point 220 can be associated with the first level 210. Point 222 can be associated with the first level 210 and the second level 212. That is, point 222 can be represented by the first voxel associated with the first level 210 and the second voxel associated with the second level 212, with the first and second voxels overlapping at least partially. Point 224 can be associated with the first level 210, the second level 212, and the third level 214. That is, point 224 can be represented by a first voxel associated with a first level 210, a second voxel associated with a second level 212, and a third voxel associated with a third level 214, where the first, second, and third voxels overlap at least partially.

[0060] Additionally or alternatively, each of levels 210, 212, and 214 may be used to determine corresponding multichannel image data, such as multichannel image data representing the top-down views of each of levels 210, 212, and 214, for input to a first part 126 of an ML model to be processed according to process 100 described above with respect to Figure 1, configured for the resolution of the corresponding level. In some examples, each of the multichannel input images corresponding to levels 210, 212, and 214 may be processed in parallel or at any number of levels corresponding to the number of voxel spaces contained in the multi-resolution voxel space 202, and the output data may be aggregated to determine the classification probability of an object. For example, per-entity logits corresponding to multichannel image data representing the top-down view of the coarsest resolution level 210 may be stored magnified to the first and furthest ranges. Furthermore, the entity-specific logits corresponding to multichannel image data representing a top-down view at intermediate resolution level 212 may be stored in an expanded second and intermediate range, and some of the entity-specific logits corresponding to multichannel image data representing a top-down view at the coarsest resolution level 210 may be overwritten in the second and intermediate range. Furthermore, the entity-specific logits corresponding to multichannel image data representing a top-down view at the highest resolution level 214 may be stored in an expanded third and shortest range, and some of the entity-specific logits corresponding to multichannel image data representing a top-down view at intermediate resolution level 212 and / or multichannel image data representing a top-down view at the coarsest resolution level 210 may be overwritten down to the third shortest range.

[0061] In some examples, the operation may include accumulating data within each voxel independently of other voxels and / or voxel levels. That is, sensor data may be represented in voxel space as raw sensor data (e.g., individual values ​​associated with data points) or as a statistical accumulation of data. For example, sensor data may be accumulated in voxel space using individual voxels containing processed data, the number of such data points, average intensity, average x-values ​​of sensor data associated with each voxel, average y-values ​​of sensor data associated with each voxel, average z-values ​​of sensor data associated with each voxel, and / or a covariance matrix based on the sensor data associated with the voxel. Sensor data can be accumulated independently for each voxel, even if voxels at one level overlap with voxels at another level, at least partially.

[0062] Additionally or alternatively, sensor data may be represented as multi-channel image data associated with different parts of the environment. Feature data may be determined for regions of the environment.

[0063] As described above, sensor data may be stored in a voxel space and / or associated with various parts of the environment, and individual voxels and / or parts may contain feature data representing feature data 226, such as multiple spatial features 228 and non-spatial features 230 (for sensor data associated with a column). In some examples, spatial features 228 may include, but are not limited to, map information of the environment (e.g., lane signs, crosswalks, traffic light locations, building locations), bounding boxes of objects in the environment, and sizes of objects in the environment. Additionally or alternatively, non-spatial features 230 may include, but are not limited to, the velocity of objects in the environment, acceleration of objects in the environment, lighting states associated with lights on objects in the environment, and behavior classes of objects (e.g., aggressive, normal). In some examples, one or more features 226 associated with the sensor data and / or 3D voxel space may be encoded as layers of a multi-channel input image, or otherwise stored. Thus, the data can be efficiently represented for input to a machine learning model, such as the first part 126 of an ML model as described with respect to Figure 1.

[0064] In some examples, the renderer is configured to generate multichannel input data containing N channels, where N can be any integer corresponding to the number of features 226 associated with the sensor data. The N channels of the multichannel image data may be divided into two subsets of channels based on the feature data 226. For example, the N channels may be divided into a first A channel and a second B channel, where, for each entity in the environment, A may be any integer corresponding to the number of spatial features 228 represented by the feature data 226, and B may be any integer corresponding to the number of non-spatial features 230 represented by the feature data 226. The N channels of the multichannel image data may be processed by the system at different points and by different parts of the ML model. For example, input data containing the first A channel may be input to the first part of the ML model, where the data representing the first A channel may be transformed to represent all features 226 about a given object, by aggregating all features associated with pixels corresponding to objects (or entities) in the environment. These A channels, when represented as object-specific features, can be combined with B channels containing non-spatial features 130, which are represented by feature data for each entity later in the processing pipeline.

[0065] This separation between A and B channels in N-channel input image data results in faster execution of the ML classification pipeline, as only the first A channel is input to the high-density backbone portion of the pipeline with a smaller footprint, as a result of lower power cost requirements for execution. In addition, spatially processing everything contained in multi-channel input data using a high-density neural network results in a proportional increase in weights corresponding to the number of channels. Therefore, reducing the number of channels input to the high-density network from N channels to a first A channel (where A is a subset of N) can result in a significant reduction in inference time, proportional to the decrease in the number of channels.

[0066] Figure 2B shows exemplary processing pipelines 240, 250, and 260 for handling a variety of environments that may be rendered at different resolutions. In some examples, input blobs 244, 254, and 264 may correspond to the multi-channel image data 114 previously described with respect to Figure 1. In some examples, processing pipelines 240, 250, and 260 may implement techniques corresponding to process 100 described above with respect to Figure 1.

[0067] As described above, the techniques described herein can be executed multiple times in parallel, corresponding to the number of environment renderers, as multi-channel image data. For example, a first-range blob renderer 242 may render a first multi-channel input blob 244 representing a first portion of the environment as multi-channel image data (e.g., top-down multi-channel image data) associated with a first range (e.g., 50m × 50m). Additionally or alternatively, a second-range blob renderer 252 may render a second multi-channel input blob 254 representing a second portion of the environment as multi-channel image data (e.g., top-down multi-channel image data) associated with a second range (e.g., 100m × 100m). Additionally or alternatively, a third-range blob renderer 262 may render a third multi-channel input blob 264 representing a third portion of the environment as multi-channel image data (e.g., top-down multi-channel image data) associated with a third range (e.g., 200m × 200m).

[0068] In some examples, multi-channel input blobs 244, 254, and 264 can represent at least partially overlapping portions of the environment. In some examples, different multi-channel input blobs 244, 254, and 264 can center around commonalities. Input blobs 244, 254, and 264 are input in parallel to different machine learning models 246, 256, and 266 configured to various ranges, which can determine classification information for the "pixels" or objects represented by input blobs 244, 254, and 264. Outputs associated with overlapping portions of the environment can be aggregated to determine the classification of objects represented by multiple input blobs 244, 254, and 264.

[0069] In the first exemplary processing pipeline 240, the first multichannel input blob 244 may be input to a first range ML model 246 configured to process the multichannel input blob 244 representing a portion of the environment associated with a first shortest range. The first range ML model 246 may determine output data representing multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for entities within the first range 248 of the environment.

[0070] In a second exemplary processing pipeline 250, a second multichannel input blob 254 may be input to a second-range ML model 256 configured to process the multichannel input blob 254 representing a portion of the environment associated with a second intermediate range. The second-range ML model 256 may determine output data representing multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for entities within the second range 258 of the environment. Additionally or alternatively, the second-range ML model 256 may determine output data representing logits 258 for entities within the second range and outside the first range.

[0071] In a third exemplary processing pipeline 260, a third multichannel input blob 264 may be input to a third range ML model 266 configured to process the multichannel input blob 264 representing a portion of the environment associated with a third longest range. The third range ML model 266 may determine output data representing multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for entities within a third range 268 of the environment. Additionally or alternatively, the third range ML model 266 may determine output data representing logits for entities within the third range and outside the second range 268.

[0072] In some examples, output data representing the logits of entities within various ranges 248, 258, and 268 determined by ML models 246, 256, and 266, respectively, can be aggregated to determine the classification probability of objects in the environment. For example, the logits of entities within a third range 268 may be collected first for aggregation. The logits of entities within a second range 258 may be collected second, and may overwrite some of the logits of entities within the third range 268 from the center point to the second range. The logits of entities within a first range 248 may be collected third, and may overwrite some of the logits of entities within the third range 268 and / or some of the logits of entities within the second range 258 from the center point to the first range.

[0073] Therefore, the output data may represent logit 248 for entities within a first range, logit 258 for entities within a second range and outside the first range, and logit 268 for entities within a third range and outside the second range. In some examples, the classification probabilities of one or more objects in various ranges of an environment that can be rendered at different resolutions may be determined based on the output data. For example, the logits may represent multiple probabilities associated with multiple classifications associated with an object, such as, but are not limited to, object type classification (e.g., vehicle, pedestrian, bicycle, etc.), object behavior classification (e.g., assertive, normal, evasive, aggressive, etc.), object gaze classification (e.g., distracted pedestrian, warning pedestrian, etc.), object trajectory classification (e.g., on the road, remaining on the sidewalk, jaywalking, etc.), lane change classification (e.g., changing lanes into my lane), and / or emergency vehicle classification. These classification probabilities are merely examples, and the technique is not limited to these specific examples.

[0074] This separation of processing pipelines for each of the different ranges of the environment represented by the multichannel image data avoids processing constraints that may arise when processing all different ranges in a single pipeline. In some examples, expanding the range represented by the image data while using the same resolution increases processing and spatial requirements proportionally to the increase in range. For example, a first multichannel image data associated with a first range (e.g., 50m x 50m) at a first resolution (e.g., about 10cm per pixel) may be rendered and processed at substantially similar speeds to a second multichannel image data associated with a second range (e.g., 100m x 100m) at a second resolution (e.g., about 40cm per pixel), since substantially the same number of pixels are processed while the range is increasing. The larger second range has a coarser resolution than the shorter first range, but the first range may provide an acceptable resolution for identifying objects near a vehicle that require finer resolution pixels for processing, resulting in more accurate classification results. In addition, configuring separate processing pipelines to run in parallel and aggregating the resulting data may further improve rendering and / or processing times.

[0075] Figure 3A shows an exemplary architecture 300 of a system that processes multichannel image data using an ML model.

[0076] As shown, backbone input features 302, such as multi-channel image data, can be input to the neural network's backbone 304 or feature extraction component. Depending on the configuration of the backbone 304, the backbone 304 can be configured in various ways so that the backbone input features 302 can be transformed into high-density backbone features 306. In some examples, the high-density backbone features 306 may contain high-density representations (or sets of values) of features associated with layers of multi-channel image data. In some examples, the high-density representations of features may contain feature vectors associated with pixels represented in the backbone input features 302.

[0077] The high-density backbone feature 306, which includes the feature vectors associated with each pixel represented in the backbone input feature 302, may pass through an aggregation 308 component that can aggregate all the feature vectors of pixels corresponding to a single entity to determine a single feature vector associated with that entity. The feature vector associated with the entity may be further processed to determine multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for each entity in the environment.

[0078] Figure 3B shows an exemplary architecture 320 of a system for processing multichannel image data through a first part of an ML model, processing intermediate input data through a second part of an ML model, and concatenating the results for further processing of the data.

[0079] As shown, backbone input features 302, such as multichannel image data, can be input to the neural network backbone 304 or feature extraction component. In some examples, the multichannel image data may contain N channels, where N can be any integer corresponding to the number of features associated with the sensor data. The N channels of the multichannel image data can then be divided into two subsets of channels based on the feature data. For example, the N channels may be divided into a first A channel and a second B channel, where A can be any integer corresponding to, for example, the number of spatial features represented by the feature data, and B can be any integer corresponding to the number of non-spatial features represented by the feature data.

[0080] As described above, the backbone 304 can be configured in various ways, depending on the configuration of the backbone 304, so that the backbone input features 302 can be converted into high-density backbone features 306. In some examples, the high-density backbone features 306 may include a high-density representation (or set of values) of features associated with the first A channel of multichannel image data. In some examples, the high-density representation of features may include feature vectors associated with pixels represented in the multichannel image data.

[0081] The high-density backbone feature 306 may undergo a masking operation 322 to determine a feature map 324 for each entity (object). In some examples, a single-channel image may be used to mask the high-density backbone feature and detect one or more entities. In some examples, the renderer may generate a single-channel image that includes a representation of the centers of the entities contained in the image (or the respective centers of each entity). The single-channel image may also include a representation of the occupations associated with objects in the environment. For example, the single-channel image may be aligned with the high-density backbone feature 306, or otherwise masked, to provide a representation of the occupations for the high-density backbone feature 306. If there is no representation of occupation within the high-density backbone feature 306, the corresponding feature vector may be ignored; if there is a representation of occupation within the high-density backbone feature 306, the corresponding feature vector may be collected. The collected feature vectors corresponding to a single entity may be aggregated to determine a single feature vector associated with the entity, which may be represented at the center of the entity. All feature vectors for each entity may be included in the per-entity feature map 324.

[0082] In some examples, the sparse renderer 326 may be configured to generate additional feature data representing additional per-entity features 328 based on sensor data corresponding to specific features, such as a second B-channel representing non-spatial features. The additional per-entity features 328 may include multiple per-entity features represented by the sensor data. In some examples, the additional per-entity features may be configured as feature vectors associated with the entities.

[0083] In some examples, feature-wise concatenation operations can be performed. For example, each feature vector, represented by a graphic reference corresponding to the location of an object in multichannel image data, such as the center of an entity included in an entity-wise feature map 324, can be concatenated with an additional entity-wise feature 328 (i.e., concatenate the feature vector associated with an entity with the additional feature vector associated with the entity) to determine the entity-wise feature-concatenation data 330. The entity-wise feature-concatenation data 330 can be input into a dense neural network, such as an entity-wise deep neural network (DNN). The entity-wise feature-concatenation data 330 can pass through a dense network for each entity 332, and the entity-wise feature-concatenation data 330 can be processed to determine multiple logits for each entity 310 included in the environment.

[0084] Figure 4 is a block diagram of an exemplary system 400 for implementing the technology described herein. In at least one example, the system 400 may include a vehicle 402. In the illustrated exemplary system 400, the vehicle 402 is an autonomous vehicle, but the vehicle 402 can be any other type of vehicle.

[0085] Vehicle 402 may be an unmanned vehicle, such as an autonomous vehicle, configured to operate in accordance with the Level 5 classification issued by the U.S. Department of Transportation's National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout the entire journey without expecting a driver (or occupant) to take control of the vehicle at any given time. In such an example, Vehicle 402 may be configured to control all functions from the start to the end of the journey, including all parking functions, and therefore does not need to include a driver and / or controls for driving Vehicle 402, such as a steering wheel, accelerator pedal, and / or brake pedal. This is merely an example, and the systems and methods described herein may be incorporated into any ground, air, or water vehicle, ranging from vehicles that must always be manually controlled by a driver to vehicles that are partially or fully autonomously controlled.

[0086] The vehicle 402 may include one or more computing devices 404, one or more sensor systems 406, one or more emitters 408, one or more communication connections 410 (also referred to as communication devices and / or modems), at least one direct connection 412 (for example, for physically connecting to the vehicle 402 to exchange data and / or provide power), and one or more drive systems 414. One or more sensor systems 406 may be configured to capture sensor data associated with the environment.

[0087] One or more sensor systems 406 may include time-of-flight sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measuring unit (IMU), accelerometer, magnetometer, gyroscope, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, brightness, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, and the like. One or more sensor systems 406 may include multiple instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors positioned at the corners, front, rear, sides, and / or top of the vehicle 402. As another example, the camera sensors may include multiple cameras positioned at various locations on the exterior and / or interior of the vehicle 402. One or more sensor systems 406 may provide input to the computing device 404.

[0088] Vehicle 402 may also include emitters 408 that emit light and / or sound. One or more emitters 408 in this example include internal audio and visual emitters that communicate with the occupants of vehicle 402. Internal emitters may include speakers, lighting, signs, display screens, touchscreens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.). One or more emitters 408 in this example may also include external emitters. External emitters in this example include light for signaling the direction of travel or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for voice communication with pedestrians or other nearby vehicles, one or more of which may incorporate acoustic beam steering technology.

[0089] The vehicle 402 may also include one or more communication connections 410 that enable communication between the vehicle 402 and one or more other local or remote computing devices (e.g., remotely operated computing devices) or remote services. For example, one or more communication connections 410 may facilitate communication with other local computing devices and / or one or more drive systems 414 on the vehicle 402. Alternatively, one or more communication connections 410 may enable the vehicle 402 to communicate with other neighboring computing devices (e.g., other neighboring vehicles, traffic signals, etc.).

[0090] One or more communication connections 410 may include physical and / or logical interfaces for connecting a computing device 404 to another computing device or one or more external networks 440 (e.g., the Internet). For example, one or more communication connections 410 may enable Wi-Fi®-based communication such as via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth®, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), satellite communication, dedicated narrow-area communication (DSRC), or any suitable wired or wireless communication protocol that enables each computing device to interface with other computing devices.

[0091] In at least one example, the vehicle 402 may include one or more drive systems 414. In one example, the vehicle 402 may have a single drive system 414. In at least one example, if the vehicle 402 has multiple drive systems 414, the individual drive systems 414 may be located at opposite ends of the vehicle 402 (e.g., front and rear). In at least one example, the drive system 414 may include one or more sensor systems 406 for detecting the conditions around the drive system 414 and / or the vehicle 402. In non-limiting examples, the sensor system 406 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the drive system's wheels, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the direction and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects around the drive system. Some sensors, such as wheel encoders, may be specific to the drive system 414. In some cases, the sensor system 406 on the drive system 414 may overlap with or supplement the corresponding system (e.g., sensor system 406) on the vehicle 402.

[0092] The drive system 414 may include many vehicle systems, including a high-voltage battery, a motor to propel the vehicle, an inverter to convert DC from the battery to AC for use in other vehicle systems, a steering system including a steering motor and steering rack (which may be electric), a brake system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to reduce traction loss and maintain control, an HVAC system, lighting (e.g., headlights / taillights illuminating the perimeter of the vehicle), and one or more other systems (e.g., a cooling system, safety systems, an on-board charging system, a DC / DC converter, a high-voltage junction, high-voltage cables, a charging system, a charging port, and other electrical components). Furthermore, the drive system 414 may include a drive system controller that can receive and preprocess data from the sensor system 406 and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and a memory communicatively coupled to one or more processors. The memory may store one or more components for performing various functions of the drive system 414. Furthermore, the drive system 414 also includes one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0093] The computing device 404 may include one or more processors 416 and a memory 418 communicatively coupled to one or more processors 416. In the illustrated example, the memory 418 of the computing device 404 stores a positioning component 420, a perception component 422 including a voxel spatial component 424, an imaging component 426, a prediction component 432, a planning component 434, a map component 436, and one or more system controllers 438. Although shown to reside in the memory 418 for illustrative purposes, the positioning component 420, the perception component 422, the voxel spatial component 424, the imaging component 426, the mask component 428, the feature component 430, the prediction component 432, the planning component 434, the map component 436, and one or more system controllers 438 are intended to be additionally or alternatively accessible to the computing device 404 (e.g., stored in different components of the vehicle 402) and / or accessible to the vehicle 402 (e.g., stored remotely).

[0094] In the memory 418 of the computing device 404, the positioning component 420 may include the ability to receive data from the sensor system 406 to determine the position of the vehicle 402. For example, the positioning component 420 may include and / or request / receive a three-dimensional map of the environment and continuously determine the position of the autonomous vehicle within the map. In some examples, the positioning component 420 may use SLAM (simultaneous localization and mapping) or CLAMS (calibration, localization and mapping, simultaneously) to receive time-of-flight data, image data, LiDAR data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof to accurately determine the position of the autonomous vehicle. In some examples, as described herein, the positioning component 420 may provide data to various components of the vehicle 402 to determine the initial position of the autonomous vehicle in order to generate a trajectory.

[0095] The perceptual component 422 may include functionality for performing object detection, segmentation, and / or classification. In some examples, the perceptual component 422 may provide processed sensor data indicating the presence of entities in proximity to the vehicle 402 and / or the classification of entities as entity types (e.g., automobile, pedestrian, cyclist, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perceptual component 422 may provide processed sensor data indicating one or more features associated with the detected entities and / or the environment in which the entities are located. In some examples, features associated with an entity may include, but are not limited to, x-position (global position), y-position (global position), z-position (global position), orientation, entity type (e.g., classification, etc.), entity velocity, entity range (size), etc. Features associated with the environment may include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, time of day, day of the week, season, weather conditions, darkness / light indication, etc.

[0096] As described above, the perceptual component 422 can use perceptual algorithms to determine perception-based bounding boxes associated with objects in the environment based on sensor data. For example, the perceptual component 422 can receive image data, classify the image data, and determine whether an object is represented in the image data. The perceptual component 422 can then use detection algorithms to generate a two-dimensional bounding box and / or a perception-based three-dimensional bounding box associated with the object. The perceptual component 422 can further generate a three-dimensional bounding box associated with the object. As described above, the three-dimensional bounding box can provide additional information such as the position, orientation, pose, and / or size (e.g., length, width, height, etc.) associated with the object.

[0097] The perceptual component 422 may include the ability to store perceptual data generated by the perceptual component 422. In some examples, the perceptual component 422 may determine tracks corresponding to objects classified as object types. For illustrative purposes only, the perceptual component 422 using the sensor system 406 may capture one or more images of the environment. The sensor system 406 may capture images of the environment that include objects such as pedestrians. A pedestrian may be at a first position at time T and at a second position at time T+t (e.g., movement during the period of time t after time T). In other words, a pedestrian may move from the first position to the second position during this period. Such movement may be logged, for example, as stored perceptual data associated with an object.

[0098] In some examples, the stored perceptual data may include fused perceptual data captured by the vehicle. The fused perceptual data may include fused or other combinations of sensor data from sensor systems 406 such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, internal sensors, and / or any combination thereof. The stored perceptual data may additionally or alternatively include classification data, including semantic classification of objects represented in the sensor data (e.g., pedestrians, vehicles, buildings, road surfaces, etc.). The stored perceptual data may additionally or alternatively include track data (location, orientation, sensor features, etc.) corresponding to the movement of objects classified as dynamic objects through the environment. The track data may include multiple tracks of multiple different objects over time. This track data can be mined to identify images of a particular type of object (e.g., pedestrians, animals, etc.) when the object is stationary (e.g., motionless) or moving (e.g., walking, running, etc.). In this example, the computing device determines the track corresponding to the pedestrian.

[0099] The voxel space component 424 can include the function of receiving sensor data and associating the sensor data with one or more voxel spaces as a multi-resolution voxel space. In some examples, the voxel space includes three dimensions. In some examples, the voxel space component 424 can include the function of statistically accumulating sensor data and processing the sensors when the sensors are associated with individual voxels. In other examples, the voxel space component 424 can include the function of associating sensor data with the voxel space as raw sensor data (e.g., individual <x, y, z, range, time> values associated with data points).

[0100] The imaging component 426 can include the function of determining a top-down representation (or a representation from any view) of the multi-resolution voxel space. Additionally or alternatively, the imaging component 426 can include the function of determining multi-resolution multi-channel image data for input to the first part of the ML model. For example, the imaging component 426 can include the function of rendering the multi-channel input blobs 244, 254, and 264 for input to the various ML models 246, 256, and 266, respectively, as previously described with respect to FIG. 2B. Additionally or alternatively, the imaging component can be configured to render various input data for input at various parts of the ML model.

[0101] The mask component 428 can include the function of generating a single-channel image including the display of the center (or the respective centers of each object) of the objects included in the image of the environment. Additionally or alternatively, the mask component 428 can include the function of masking some input data, such as multi-channel image data including one or more feature vectors, for example, to provide a display of occupancy for the input data.

[0102] The feature component 430 may include a function to aggregate feature vectors corresponding to image pixels representing the environment. For example, one or more pixels may be associated with an object, and the feature vectors corresponding to one or more pixels associated with the object may be aggregated to determine a single feature vector associated with the object. Additionally or alternatively, the feature component 430 may include a function to concatenate the feature vectors associated with objects with additional feature vectors representing one or more additional features for each object to determine intermediate input data to be input to the second part of the ML model.

[0103] The prediction component 432 can generate one or more probability maps representing the predicted probabilities of the possible locations of one or more objects in the environment. For example, the prediction component 432 can generate one or more probability maps for vehicles, pedestrians, animals, etc., within a threshold distance from vehicle 402. In some examples, the prediction component 432 can measure the track of an object and, based on the observed and predicted behavior, generate discretized predicted probability maps, heat maps, probability distributions, discretized probability distributions, and / or trajectories for the object. In some examples, one or more probability maps can represent the intent of one or more objects in the environment.

[0104] The planning component 434 can determine the path that the vehicle 402 will take to traverse the environment. For example, the planning component 434 can determine various routes and paths as well as various levels of detail. In some examples, the planning component 434 can determine a route to travel from a first location (e.g., the current location) to a second location (e.g., a target location). For the purposes of this explanation, the route may be a series of waypoints for traveling between the two locations. In non-limiting examples, waypoints may include roads, intersections, and Global Positioning System (GPS) coordinates. Furthermore, the planning component 434 can generate instructions for guiding the autonomous vehicle along at least part of the route from the first location to the second location. In at least one example, the planning component 434 may determine how to guide the autonomous vehicle from a first waypoint in a series of waypoints to a second waypoint in a series of waypoints. In some examples, the instructions may be a route or part of a route. In some cases, multiple paths can be generated substantially simultaneously (e.g., within technical tolerances) according to the receding horizon technique. A single path from among the multiple paths in the receding horizon data with the highest level of confidence can be selected for vehicle operation.

[0105] In other examples, the planning component 434 may, alternatively or additionally, use data from the perceiving component 422 and / or the predictive component 432 to determine the path that the vehicle 402 will take to traverse the environment. For example, the planning component 434 may receive data from the perceiving component 422 and / or the predictive component 432 regarding objects associated with the environment. Using this data, the planning component 434 may determine a path to move from a first location (e.g., the current location) to a second location (e.g., the target location) in order to avoid objects in the environment. In at least some examples, such a planning component 434 may determine that there is no such collision-free path and provide a path that leads the vehicle 402 to a safe stop that avoids all collisions and / or otherwise mitigates damage.

[0106] Memory 418 may further include one or more maps 436 that can be used by the vehicle 402 to navigate within the environment. For illustrative purposes, the maps may be any number of data structures modeled in two, three, or N dimensions that are capable of providing information about the environment, such as topology (like intersections), streets, mountain ranges, roads, terrain, and the general environment. In one example, the map may include, but is not limited to, covariance data (e.g., represented in multi-resolution voxel space), texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), intensity information (e.g., LiDAR information, radar information), spatial information (e.g., image data projected onto a mesh, individual "surfaces" (e.g., polygons associated with individual colors and / or intensities)), and reflectivity information (e.g., specular information, retroreflective information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, as described herein, maps may be stored in a tiled format such that individual tiles of the map represent distinct parts of the environment, and may be loaded into working memory as needed. In at least one example, one or more maps 436 may include at least one map (e.g., an image and / or a mesh). In some examples, a vehicle 402 may be controlled at least in part based on a map 436. That is, a map 436 can be used in conjunction with a positioning component 420, a perception component 422 (and subcomponents), a prediction component 432, and / or a planning component 434 to determine the position of the vehicle 402, identify objects in the environment, generate predicted probabilities associated with the objects and / or the vehicle 402, and / or generate a path and / or trajectory for navigating through the environment.

[0107] In at least one example, the computing device 404 may include one or more system controllers 438, which may be configured to control the steering, propulsion, brakes, safety, emitter, communication, and other systems of the vehicle 402. The system controllers 438 may communicate with and / or control the corresponding systems of the drive system 414 and / or other components of the vehicle 402, which may be configured to operate according to the paths provided by the planning component 434.

[0108] Vehicle 402 can connect to computing device 442 via network 440 and may include one or more processors 444 and memory 446 communicatively coupled to one or more processors 444. In at least one example, one or more processors 444 may be analogous to processor 416, and memory 446 may be analogous to memory 418. In the illustrated example, memory 446 of computing device 442 stores a remote operation component 448 and / or a model component 450. In at least one example, the model component 450 may generate an ML model used by the perceiving component 422 after empirical testing and / or simulation, as discussed herein. Although shown to reside in memory 446 for illustrative purposes, the remote operation component 448 and the model component 450 may, additionally or alternatively, be accessible to computing device 442 (e.g., stored in different components of computing device 442 and / or be accessible to computing device 442 (e.g., stored remotely).

[0109] The model component 450 may include the function of generating a model for determining segmentation information, classification information, etc., of multichannel image data, as discussed herein.

[0110] The processor 416 of computing device 404 and the processor 444 of computing device 442 may be any suitable processor capable of processing data and executing instructions for performing actions, as described herein. In non-limiting examples, processors 416 and 444 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or part of a device that processes electronic data and converts that electronic data into other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs), gate arrays (e.g., FPGAs), and other hardware devices may also be considered processors, insofar as they are configured to execute encoded instructions.

[0111] Memory 418 of computing device 404 and memory 446 of computing device 442 are examples of non-temporary computer-readable media. Memories 418 and 446 can store the operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions resulting from various systems described herein. In various implementations, memories 418 and 446 may be implemented using appropriate memory technology such as static random-access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, some of which are shown in the accompanying drawings, but are merely illustrative examples relevant to the description herein.

[0112] In some examples, some or all aspects of the components described herein may include any model, algorithm, and / or machine learning algorithm. For example, in some examples, the components in memories 418 and 446 may be implemented as a neural network.

[0113] As described herein, an exemplary neural network is an algorithm that passes input data through a series of connected layers to produce an output. Each layer in the neural network may also contain another neural network, or any number of layers (whether convolutional or not). As understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of algorithms in which an output is produced based on learned parameters.

[0114] Although described in the context of neural networks, any type of machine learning may be used in a manner consistent with this disclosure. For example, machine learning or machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression spline (MARS), locally estimated scatter plot smoothing (LOESS), instance-based algorithms (e.g., ridge regression, least absolute reduction selection operator (LASSO), elastic networks, least angle regression (LARS), decision tree algorithms (e.g., classification regression tree (CART), iterative bisection 3 (ID3), chi-squared automatic interaction detection (CHAID), decision stump, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, polynomial naive Bayes, mean 1 dependency estimator (AODE), Bayesian confidence network (BNN), Bayesian network), clustering algorithms (e.g., k-means algorithm, k-median algorithm, expectation maximization (EM), hierarchical clustering), and association rule learning algorithms (e.g., perceptron). This may include, but is not limited to, methods such as backpropagation, Hopfield networks, radial basis function networks (RBFNs), deep learning algorithms (e.g., deep Boltzmann machines (DBMs), deep trust networks (DBNs), convolutional neural networks (CNNs), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), summon mapping, multidimensional scaling (MDS), projection tracking, linear discriminant analysis (LDA), mixture discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), ADA boost, hierarchical generalization (blending), gradient boosting machines (GBMs), gradient boosted regression trees (GBRTs), random forests), SVMs (support vector machines), supervised learning, unsupervised learning, and semi-supervised learning.

[0115] Examples of additional architectures include neural networks such as ResNet50, ResNet52, ResNet101, VGG, DenseNet, and PointNet.

[0116] Figures 1, 5, and 6 illustrate exemplary processes in the examples of this disclosure. These processes are illustrated as logical flow diagrams, and each operation represents a set of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, an operation represents a computer executable instruction stored in one or more computer-readable storage media that, when executed by one or more processors, performs the enumerated operation. Generally, computer executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations may be excluded or combined in any order and / or in parallel to execute the process.

[0117] Figure 5 shows an exemplary process for inputting input data into a first part of a machine learning (ML) model, inputting intermediate input data into a second part of the ML model, and determining the classification probability of objects in an environment based on output data from the first and second parts of the ML model. For example, some or all of the process 500 may be performed by one or more components in Figure 4, as described herein. For example, some or all of the process 500 may be performed by the voxel space component 424, the imaging component 426, the mask component 428, and / or the feature component 430.

[0118] In operation 502, process 500 may include receiving sensor data of the environment. In some examples, operation 502 may include receiving and / or capturing time-of-flight data, LiDAR data, image data, radar data, etc., of the environment. In some examples, operation 502 may be performed by a vehicle (e.g., an autonomous vehicle) as the vehicle traverses the environment. In some examples, the environment may include objects, and the sensor data may include a representation of objects in the environment.

[0119] In operation 504, process 500 may include determining multichannel image data representing a top-down view of the environment. In some examples, the multichannel image data may be based on sensor data. In some examples, the multichannel image data may correspond to multichannel image data 114. In some examples, the multichannel image data may include image data associated with spatial features corresponding to objects in the environment.

[0120] In operation 506, process 500 may include determining whether a spatial feature corresponding to an object exists in the image data. For example, operation 506 may include determining that a spatial feature exists in the multi-channel image data. Additionally or alternatively, operation 506 may include determining that an object exists in the image data. As another example, operation 506 may include determining that a spatial feature and / or an object does not exist in the image data. The process then may include operation 502 to restart the process based on the determination that the spatial feature and / or object does not exist in the image data.

[0121] In operation 508, process 500 may include inputting multi-channel image data to a first part of a machine learning (ML) model. In some examples, the first part of the ML model may be configured as the backbone or feature extraction component of a neural network, such as a convolutional neural network (CNN). The first part of the ML model may process the multi-channel image data and determine intermediate output data corresponding to the multi-channel image data.

[0122] In operation 510, process 500 may include receiving intermediate output data from a first part of the ML model. In some examples, the intermediate output data may be based at least partially on multichannel image data.

[0123] In operation 512, process 500 may include determining non-spatial feature data representing non-spatial features associated with an object. In some examples, the non-spatial feature data may be based at least partially on sensor data. Additionally or alternatively, the non-spatial feature data may be associated with non-spatial features associated with the object. In some examples, the non-spatial feature data may correspond to feature 136.

[0124] In operation 514, process 500 may include inputting intermediate output data and non-spatial feature data to a second part of the ML model. In some examples, the second part of the ML model may be configured as a high-density neural network, such as a deep neural network (DNN). The intermediate output data and / or spatial feature data are transformed object by object or otherwise concatenated and passed to the second part of the ML model, which may process the intermediate output data and / or non-spatial feature data and determine output data representing multiple logits for each object.

[0125] In operation 516, process 500 may include receiving intermediate output data from a second part of the ML model. In some examples, the output data may be at least partially based on the intermediate output data and / or non-spatial feature data. In some examples, the output data may represent multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for each object in the environment.

[0126] In operation 518, process 500 may include determining the classification probability of an object. In some examples, the classification probability may be based at least in part on output data received from a second part of the ML model. In some examples, the classification probability may be based at least in part on object-specific logits represented by the output data. In some examples, the classification probability may represent at least one of the following: object type classification (e.g., vehicle, pedestrian, bicycle, etc.), object behavior classification (e.g., assertive, normal, avoidant, aggressive, etc.), object gaze classification (e.g., distracted pedestrian, warned pedestrian, etc.), object trajectory classification (e.g., on the road, remaining on the sidewalk, jaywalking, etc.), lane change classification (e.g., changing lanes to mine), and / or emergency vehicle classification.

[0127] Figure 6 shows an exemplary process 600 for inputting first input data into a first ML model, inputting second input data into a second ML model, receiving a first classification associated with an object from the first ML model, receiving a second classification associated with an object from the second ML model, and controlling an autonomous vehicle based on the first or second classification. For example, some or all of the process 600 may be performed by one or more components in Figure 4, as described herein. For example, some or all of the process 600 may be performed by a voxel spatial component 424, an imaging component 426, a mask component 428, and / or a feature component 430.

[0128] In operation 602, process 600 may include receiving sensor data of the environment. In some examples, operation 602 may include receiving and / or capturing time-of-flight data, LiDAR data, image data, radar data, etc. of the environment. In some examples, operation 602 may be performed by a vehicle (e.g., an autonomous vehicle) as the vehicle traverses the environment.

[0129] In operation 604, process 600 may include determining a first multichannel image data representing a first top-down view of a first region of the environment. In some examples, the multichannel image data may be based on sensor data. In some examples, the multichannel image data may correspond to multichannel image data 114.

[0130] In operation 606, process 600 may include determining a second multichannel image data representing a second top-down view of a second region of the environment. In some examples, the second multichannel image data may be based on sensor data. In some examples, the second multichannel image data may correspond to multichannel image data 114. In some examples, the second region of the environment may be larger than the first region. In some examples, the second multichannel input data may have a lower resolution than the first multichannel input data.

[0131] In operation 608, process 600 may include inputting the first multi-channel image data into a machine learning (ML) model. In some examples, the first ML model may correspond to any one of the processing pipelines 240, 250, or 260, as previously described with respect to Figure 2B. In some examples, the first ML model may consist of a first part and a second part. In some examples, the first part of the first ML model may be configured as the backbone or feature extraction component of a neural network, such as a convolutional neural network (CNN). In some examples, the second part of the first ML model may be configured as a high-density neural network, such as a deep neural network (DNN).

[0132] In operation 610, process 600 may include inputting a second multi-channel image data into a second machine learning (ML) model. In some examples, the second ML model may correspond to any one of the processing pipelines 240, 250, or 260, as previously described with respect to Figure 2B. In some examples, the second ML model may consist of a first part and a second part. In some examples, the first part of the second ML model may be configured as the backbone or feature extraction component of a neural network, such as a convolutional neural network (CNN). Additionally or alternatively, the second part of the second ML model may be configured as a high-density neural network, such as a deep neural network (DNN).

[0133] In operation 612, process 600 may include receiving first output data from a first ML model. In some examples, the first output data may include a first classification associated with objects in the environment. In some examples, the first output data may be based on multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for each object in the environment. Additionally or alternatively, the first output data may correspond to at least some of the logits of entities 248, 258, or 268, as previously described with respect to Figure 2B.

[0134] In operation 614, process 600 may include receiving second output data from a second ML model. In some examples, the second output data may include a second classification associated with objects in the environment. In some examples, the second output data may be based on multiple logits (e.g., functions representing probability values ​​from 0 or negative infinity to 1 or infinity) for each object in the environment. Additionally or alternatively, the second output data may correspond to at least some of the logits of entities 248, 258, or 268, as previously described with respect to Figure 2B.

[0135] In operation 616, process 600 may include determining whether a first probability associated with a first classification or a second probability associated with a second classification satisfies a threshold probability. For example, operation 616 may include determining that at least one of the first or second probability satisfies a threshold probability and that the autonomous vehicle can be controlled based on the first or second classification. As another example, operation 616 may include determining that neither the first nor the second probability satisfies a threshold probability. Process 600 may then include operation 602, which restarts the process based on the determination that neither the first nor the second probability satisfies a threshold probability.

[0136] In operation 618, process 600 may include controlling the vehicle based at least in part on objects. In some examples, operation 618 may include generating a trajectory that stops the vehicle, or otherwise controlling the vehicle to safely traverse the environment. In some examples, operation 618 may include modifying candidate trajectories based on detected objects, for example, determining a modified trajectory that the vehicle should follow within the environment.

[0137] (Example clause) A. One or more processors, One or more non-temporary computer-readable media for storing instructions executable by the one or more processors, Equipped with, when an instruction is executed, the system Receiving sensor data from sensors associated with the environment, including objects, Determining multichannel image data representing a top-down view of the environment, based at least partially on sensor data, wherein the multichannel image data includes image data associated with spatial features corresponding to objects, Inputting multi-channel input data into the first part of a machine learning (ML) model, Receiving intermediate output data from a first part of an ML model, wherein the intermediate output data includes spatial features encoded as being associated with the object, Determining non-spatial feature data representing non-spatial features associated with the object, based at least partially on sensor data, Intermediate output data and non-spatial feature data are input into the second part of the ML model, Receiving output data from the second part of the ML model, Determining the classification probability of an object based at least partially on the output data. A system that performs actions including those mentioned above.

[0138] B. The system of paragraph A, wherein the spatial features include at least one of map information associated with the environment, a bounding box associated with the object, and the size of the object, and the non-spatial features include at least one of velocity associated with the object, acceleration associated with the object, and lighting state associated with the object.

[0139] C. The system according to paragraph A or B, wherein the classification probability includes at least one of object type classification, object behavior classification, object line of sight classification, object trajectory classification, lane change classification, and emergency vehicle classification.

[0140] D. The system according to any one of paragraphs A to C, further comprising determining a graphic reference corresponding to a corresponding position of an object in the multichannel image data, wherein the spatial feature is encoded as being associated with the object based at least in part on the graphic reference.

[0141] The system described in any one of paragraphs A through D, wherein the first part of the E.ML model includes a convolutional neural network (CNN), and the second part of the ML model includes a deep neural network (DNN).

[0142] F. Receiving sensor data from sensors associated with the environment containing the object, Determining spatial data representing the environment based at least partially on sensor data, Inputting spatial data into the first part of a machine learning (ML) model, Receiving intermediate data from a first part of an ML model, wherein the intermediate data includes spatial feature data corresponding to spatial features encoded as being associated with the object, Determining secondary feature data corresponding to features associated with the object, based at least partially on sensor data, The spatial feature data and secondary feature data are input into the second part of the ML model, Determining the classification probability of an object based at least partially on the data received from the second part of the ML model. A method for providing this.

[0143] G. The method according to paragraph F, wherein the spatial feature includes at least one of map information associated with the environment, a bounding box associated with the object, and a size associated with the object, and the feature includes at least one of velocity associated with the object, acceleration associated with the object, and lighting state associated with the object.

[0144] H. Spatial data represents a top-down view of the environment, as described in paragraph G.

[0145] I. Determining a mask to identify the corresponding location of an object in spatial data, Based on the mask and spatial data, determine the feature vector associated with the object. The method described in any one of paragraphs F through H, further including the method described in any one of paragraphs F through H.

[0146] The method described in any one of paragraphs F through I, wherein the first part of the ML model includes a convolutional neural network (CNN), and the second part of the ML model includes a deep neural network (DNN).

[0147] K. Spatial data is first spatial data corresponding to a first range up to a starting point in the environment, the ML model includes a first ML model, and the method is The method involves determining a second spatial data based at least partially on sensor data, wherein the second spatial data corresponds to a second range up to the starting point, and the first range is smaller than the second range. Inputting the second spatial data into a second ML model configured to determine the classification probability associated with a second range, It further includes, Determining the classification probability is done according to the method described in any one of paragraphs F to J, which is at least partially based on the output of the second ML model.

[0148] L. Further includes relating sensor data to a three-dimensional voxel space representing the environment, The spatial data is a representation of a reduced-dimensional representation of a three-dimensional voxel space, as described in any one of paragraphs F to K.

[0149] The method according to any one of paragraphs F to L, further comprising receiving sensor data from sensors associated with an autonomous vehicle in an environment.

[0150] N. The method according to any one of paragraphs F to M, further comprising controlling an autonomous vehicle on at least partly the classification probability.

[0151] O. One or more non-temporary computer-readable media that store instructions executable by a processor, wherein, when an instruction is executed, the processor... Receiving sensor data from sensors associated with the environment, including objects, Determining spatial data representing the environment based at least partially on sensor data, Inputting spatial data into the first part of a machine learning (ML) model, Receiving intermediate data from the first part of the ML model, the intermediate data including spatial feature data corresponding to spatial features encoded as associated with objects, Determining secondary feature data corresponding to features associated with an object, based at least partially on sensor data, The spatial feature data and secondary feature data are input into the second part of the ML model, The classification probability of the object is determined at least partially based on the data received from the second part of the ML model. One or more non-temporary computer-readable media that cause an operation including the following to be performed.

[0152] P. A spatial feature includes at least one of map information associated with the environment, a bounding box associated with the object, and a size associated with the object, and the feature includes at least one of velocity associated with the object, acceleration associated with the object, and illumination state associated with the object, as described in one or more non-temporary computer-readable media of paragraph O.

[0153] Q. Spatial data is one or more non-temporary computer-readable media described in paragraph P, representing a top-down view of the environment.

[0154] R. Spatial data is one or more non-temporary computer-readable media as described in paragraph Q, comprising a first channel containing a first spatial feature and a second channel containing a second spatial feature.

[0155] S. The above operation is, Determining a mask that identifies the corresponding location of an object in spatial data, The process involves determining feature vectors associated with an object based on a mask and spatial data, wherein the intermediate data includes feature vectors. The following are further included: one or more non-temporary computer-readable media as described in any one of paragraphs Q through R.

[0156] One or more non-temporary computer-readable media as described in any one of paragraphs Q to S, wherein the first part of the T.ML model includes a convolutional neural network (CNN), and the second part of the ML model includes a deep neural network (DNN).

[0157] U.1 or more processors, One or more non-temporary computer-readable media for storing instructions that can be executed by one or more processors, and The system is equipped with such that, when the instruction is executed, the system will Receiving sensor data from sensors associated with the environment, including autonomous vehicles, Based at least partially on sensor data, determine first multi-channel input data representing a first top-down view of a first region of the environment, Determining, at least partially, sensor data to determine second multi-channel input data representing a second top-down view of a second region of the environment, wherein the second region of the environment is wider than the first region of the environment, and the second multi-channel input data has a lower resolution than the first multi-channel input data. The first multi-channel input data is input into the first machine learning (ML) model, The second multi-channel input data is input to the second ML model, Receiving first output data from the first ML model, wherein the first output data includes a first classification associated with an object in the environment. Receiving second output data from the second ML model, wherein the second output data includes a second classification associated with an object. Controlling autonomous vehicles based at least partially on the first or second classification. A system that performs actions including those mentioned above.

[0158] V. The first region of the environment is the system described in paragraph U, which at least partially overlaps with the second region of the environment.

[0159] W. The system described in paragraph U or V, wherein the operation further includes determining that an object is associated with a first area of ​​the environment and discarding a second classification based at least on determining that the object is within the first area, and controlling the autonomous vehicle is at least partially based on the first classification.

[0160] X. A system as described in any one of paragraphs U to W, wherein the first channel of the first multichannel input data includes first feature data representing a feature associated with an object, and the second channel of the second multichannel input data includes second feature data representing a feature associated with an object.

[0161] Y. The first multi-channel input data is a system described in any one of paragraphs U through X, representing spatial features and objects in the environment.

[0162] Z. Receiving sensor data from sensors associated with the environment, Determining first input data representing a first region of the environment based at least partially on sensor data, The method involves determining a second input data representing a second region of the environment, at least partially based on sensor data, wherein the second region of the environment is wider than the first region, and the second input data represents the environment with a lower resolution than the first input data. Inputting the first input data into the first machine learning (ML) model, The second input data is entered into the second ML model, Controlling an autonomous vehicle based at least partially on a first classification associated with objects in the environment received from a first ML model, or a second classification associated with objects in the environment received from a second ML model. A method for providing this.

[0163] AA. The method described in paragraph Z, wherein the first region of the environment at least partially overlaps with the second region of the environment.

[0164] AB. Determine if the object is associated with the first region of the environment, Discarding the second classification based at least on the determination that the object is within the first area, and controlling the autonomous vehicle is at least partially based on the first classification. The method described in paragraph Z or AA, further including the method described in paragraph Z or AA.

[0165] AC. The method according to any one of paragraphs Z to AB, wherein the first input data includes first multichannel input data, the first channel of the first multichannel input data includes first feature data representing a feature associated with an object, and the second channel of the second multichannel input data includes second feature data representing a feature associated with an object.

[0166] AD. The first input data is the method described in any one of paragraphs Z through AC, representing spatial features and objects within the environment.

[0167] The method according to any one of paragraphs Z to AD, further comprising determining feature data associated with AE sensor data, wherein the feature data represents features associated with an object, and the first input data includes several channels based at least in part on a first subset of the features.

[0168] AF. Input the first input data into the first part of the first ML model, Receiving intermediate output data from the first part of the first ML model, Determining intermediate input data based at least partially on a second subset of features associated with the object and intermediate output data, The intermediate input data is input to the second part of the first ML model, From the first part of the first ML model, receive first output data including a first classification associated with an object. The method described in any one of paragraphs Z through AE, further including the method described in any one of paragraphs Z through AE.

[0169] AG. The method according to any one of paragraphs Z to AF, wherein a first subset of the features includes at least one of map information associated with the environment, a bounding box associated with the object, and a size associated with the object, and a second subset of the features includes at least one of velocity associated with the object, acceleration associated with the object, and illumination state associated with the object.

[0170] AH. The method according to any one of paragraphs Z to AG, further comprising associating sensor data with a three-dimensional voxel space representing the environment, wherein the three-dimensional voxel space includes a first region and a second region, and the first input data includes voxel data corresponding to the first region in the three-dimensional voxel space.

[0171] One or more non-temporary computer-readable media storing instructions that can be executed by an AI processor, wherein, when an instruction is executed, the processor... Receiving sensor data from sensors associated with the environment, Determining first input data representing a first region of the environment based at least partially on sensor data, The method involves determining a second input data representing a second region of the environment based at least partially on sensor data, wherein the second region of the environment is larger than the first region, and the second input data represents the environment with a lower resolution than the first input data. Inputting the first input data into the first machine learning (ML) model, The second input data is input into the second ML model, Controlling an autonomous vehicle based at least partially on a first classification associated with objects in the environment, received from a first ML model, or a second classification associated with objects in the environment, received from a second ML model. One or more non-temporary computer-readable media that cause an operation including the following to be performed.

[0172] AJ. The first area of ​​the environment is one or more non-temporary computer-readable media described in paragraph AI that at least partially overlap with the second area of ​​the environment.

[0173] AK. The operation further includes determining that an object is associated with a first area of ​​the environment and discarding a second classification based at least on determining that the object is within the first area, and controlling an autonomous vehicle is at least partially based on the first classification, one or more non-temporary computer-readable media as described in paragraph AI or AJ.

[0174] AL. One or more non-temporary computer-readable media as described in any one of paragraphs AI to AK, wherein the first input data includes first multichannel input data, the first channel of the first multichannel input data includes first feature data representing features associated with an object, and the second channel of the second multichannel input data includes second feature data representing features associated with an object.

[0175] AM. The first input data is one or more non-temporary computer-readable media described in any one of paragraphs AI to AL, representing spatial features and objects in the environment.

[0176] AN. The operation further comprises determining feature data associated with sensor data, wherein the feature data represents features associated with an object, and the first input data includes several channels based at least in part on a first subset of features, in one or more non-temporary computer-readable media as described in any one of paragraphs AI to AM.

[0177] While the above examples of clauses are described in relation to one specific embodiment, it should be understood that in the context of this specification, the contents of the examples of clauses can also be implemented through methods, devices, systems, computer-readable media, and / or other embodiments. Furthermore, any of Examples A through AN can be implemented alone or in combination with any one or more of the other Examples A through AN.

[0178] (Conclusion) While one or more examples of the techniques described herein have been described, various modifications, additions, substitutions, and equivalents of those examples are included within the scope of the techniques described herein.

[0179] In the illustrative descriptions, references are made to the accompanying drawings forming part of this application, illustrating specific examples of the subject matter of the claimed patent. It should be understood that other examples may be used, and that modifications or alterations, such as structural changes, may be made. Such examples, modifications, or alterations do not necessarily deviate from the scope of the intended subject matter of the invention. While the steps in this specification are provided in a specific order, in some cases, the order may be changed so that specific inputs are provided at different times or in different orders without altering the function of the described system and method. The disclosed procedures may also be performed in different orders. Furthermore, the various computer processes in this specification do not need to be performed in the order disclosed, and other examples using alternative orders of computer processes can be readily implemented. In addition to being reordered, computer processes can also be broken down into partial computer processes that produce the same results.

Claims

1. Receiving sensor data from sensors associated with the environment, including objects, Determining spatial data representing the environment based at least partially on the aforementioned sensor data, The spatial data is input into a first part of a machine learning (ML) model, wherein the spatial data includes map information associated with the environment, and at least one of the location of the object in the environment, the bounding box associated with the object, and the size associated with the object. Receiving intermediate data from a first part of an ML model, wherein the intermediate data includes spatial feature data corresponding to spatial features encoded as being associated with the object, Determining secondary feature data corresponding to features associated with the object, based at least partially on the aforementioned sensor data, The intermediate data and the secondary feature data are concatenated as concatenated data for each object, The linked data is input into the second part of the ML model, The classification probability of the object is determined at least partially based on the data received from the second part of the ML model, A method that includes this.

2. The above-mentioned feature is, The speed associated with the aforementioned object, The acceleration associated with the aforementioned object, and The lighting state associated with the aforementioned object, Including at least one of the following: The method according to claim 1.

3. The method of claim 1 or 2, wherein the spatial data represents a top-down view of the environment.

4. Determining a mask that identifies the corresponding position of the object in the spatial data, The process involves determining a feature vector associated with the object based on the mask and the spatial data, wherein the intermediate data includes the feature vector. The method according to any one of claims 1 to 3, further comprising:

5. The method according to any one of claims 1 to 4, wherein the first part of the ML model includes a convolutional neural network (CNN), and the second part of the ML model includes a deep neural network (DNN).

6. The spatial data is a first spatial data corresponding to a first range up to a starting point in the environment, the ML model includes a first ML model, and the method is The method involves determining a second spatial data based at least partially on the aforementioned sensor data, wherein the second spatial data corresponds to a second range up to the starting point, and the first range is smaller than the second range. Inputting the second spatial data into a second ML model configured to determine the classification probability associated with the second range, It further includes, The method according to any one of claims 1 to 5, wherein determining the classification probability is at least partially based on the output of the second ML model.

7. The method further includes associating the sensor data with a three-dimensional voxel space representing the environment, The method according to any one of claims 1 to 6, wherein the spatial data represents a reduced-dimensional representation of the three-dimensional voxel space.

8. Receiving sensor data from sensors associated with autonomous vehicles in the environment, and Controlling the autonomous vehicle based at least partially on the aforementioned classification probability, The method according to any one of claims 1 to 7, further comprising at least one of the following.

9. The aforementioned classification probability is, Object type classification, Object behavior classification, Object gaze classification, Object trajectory classification, Lane change classification, and Emergency vehicle classification, The method according to any one of claims 1 to 8, comprising at least one of the following:

10. One or more non-temporary computer-readable media that, when executed by one or more processors, store instructions causing one or more computing devices to perform the method according to any one of claims 1 to 9.

11. One or more processors, One or more non-temporary computer-readable media for storing instructions executable by the one or more processors, The instruction, when executed, causes one or more processors to Receiving sensor data from sensors associated with the environment, including objects, Determining multichannel image data representing a top-down view of the environment based at least in part on the sensor data, wherein the multichannel image data includes image data associated with spatial features, the spatial features include map information associated with the environment and at least one of the location of the object in the environment, the bounding box associated with the object, and the size associated with the object. The multi-channel image data is input into the first part of the machine learning (ML) model, Receiving intermediate output data from a first part of an ML model as an intermediate output, wherein the intermediate output data includes spatial feature data corresponding to spatial features encoded as being associated with the object, Based at least partially on the aforementioned sensor data, non-spatial feature data representing non-spatial features associated with the object is determined, The intermediate output data and the non-spatial feature data are concatenated as concatenated data for each object, The linked data is input into the second part of the ML model, Receiving output data from the second part of the aforementioned ML model, The classification probability of the object is determined at least partially based on the output data received from the second part of the ML model, A system that performs actions including those mentioned above.

12. The non-spatial feature is, The speed associated with the aforementioned object, The acceleration associated with the aforementioned object, and The lighting state associated with the aforementioned object, Including at least one of the following: The system according to claim 11.

Images