Machine Learning Architectures for Efficient Object Attribute and / or Intent Classification
The ML architecture addresses the perception gap in conventional computer vision by processing multiple images over time, reducing computational load and enhancing detection accuracy for autonomous vehicles.
Patent Information
- Application Number
- JP2024527058
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-09
- Filing Date
- 2022-11-09
- Publication Date
- 2024-12-03
AI Technical Summary
Conventional computer vision systems lack the near-human level of perception needed to operate complex systems such as autonomous vehicles, and they also impose a significant computational burden.
A machine learning (ML) architecture that processes multiple images or sensor data over time, using a cache to store intermediate outputs and apply them in subsequent layers to reduce computational load while enhancing detection capabilities, allowing for improved object attribute and intent classification.
The ML architecture increases the detection accuracy and efficiency of computer vision systems, enabling autonomous vehicles to navigate safely and efficiently by reducing delays and computational overhead.
Smart Images

Figure 2024544502000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to machine learning architectures for efficient object attribute and / or intent classification. [Background technology]
[0002] Related Applications
[0001] This PCT international application claims the benefit of priority to U.S. patent application Ser. No. 17 / 522,832, filed November 9, 2021, which is incorporated herein by reference.
[0003]
[0002] Computer vision is used in many applications, such as operating autonomous vehicles and identifying individuals for security purposes. Computer vision techniques may involve building software components that identify salient features from an image and representing the salient features of the image to a computer in a form that the computer can use to perform further operations, such as detecting, classifying, and / or tracking objects. However, traditional computer vision techniques, by themselves, do not provide the near-human level of perception required to successfully operate complex systems such as autonomous vehicles. [Brief description of the drawings]
[0004] The detailed description will now be described with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and in which the same reference numbers in different drawings indicate similar or identical items. [Figure 1] FIG. 1 illustrates an example scenario in which an autonomous vehicle detects multiple objects in an environment represented in sensor data. [Diagram 2]
[0005] FIG. 2 illustrates an example architecture for detecting object attributes and / or intent. [Diagram 3]
[0006] FIG. 3 illustrates an example machine learning component architecture for detecting object attributes and / or intent and reducing computational bandwidth for doing so. [Figure 4]
[0007] FIG. 4 is a diagram illustrating example image portions associated with different object attributes detectable by the machine learning components discussed herein. [Diagram 5]
[0008] FIG. 5 is a flow diagram illustrating an example process for detecting object attributes and / or intent and reducing computational bandwidth for doing so. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0005]
[0009] As mentioned above, conventional computer vision systems cannot detect enough information by themselves to operate complex systems such as driving an autonomous vehicle. The techniques discussed herein may increase the detection effectiveness and capabilities of computer vision systems. The techniques may additionally or alternatively increase the efficiency of computer vision systems by reducing the computational burden for operating the computer vision systems. The techniques discussed herein may include machine learning (ML) architectures that receive multiple images, although in additional or alternative examples, other forms of sensor data (e.g., time of flight, radar, lidar) are contemplated. Multiple images (or other sensor data) may be associated with the same object multiple times. A first ML component may detect an object in an image and associate the detection with an image captured in a previous time step. These different image portions may be provided as inputs to an instant ML architecture.
[0006]
[0010] The ML architectures discussed herein may generate outputs using inputs in a time range. In other words, the ML architectures may determine outputs using sensor data received over a time range or a number of individual sensor outputs. For example, the ML architecture may receive an image (or a portion thereof associated with an object) and determine a set of features by processing the image using a first set of ML layers. The first set of ML layers may generate a first output that may be stored in a memory (e.g., a cache). In a next stage of the ML architecture, the ML architecture may retrieve a second output associated with a second image from the memory, where the first image and the second image may be associated with the same object, but the second image may be captured at a time prior to the time the first image was captured. The first output and the second output may be processed separately by a second ML layer to create third and fourth outputs, respectively. The third and fourth outputs may be concatenated and provided as inputs to a third ML layer to create a reduced set of features. The reduced set of features may be provided as input to a third ML layer that may output logits that may be used to determine a confidence score associated with the classification task. For example, the logits, which may be infinite in value, may be converted to confidence scores by a sigmoid layer, which may be bounded between 0 and 1. For mutually exclusive outputs, such as object states such as running, walking, standing, crouching, or object attributes such as idling, intent to enter a road, crossing a road, a softmax may be used in addition to or instead of the sigmoid layer. The sigmoid layer may additionally or alternatively be a hyperbolic tangent layer, a rectified linear unit (ReLU) layer, a Gaussian error linear unit (GELU) layer, etc.
[0007]
[0011] In some examples, the ML architectures discussed herein may be used to classify attributes and / or gestures associated with objects. For example, the confidence scores discussed herein may indicate that an object, such as a pedestrian, is idling on the side of the road (e.g., standing or otherwise distracted without actively attempting to cross the road), standing or otherwise positioned to move (e.g., hands on the wheels of a wheelchair, feet on the pedals of a bicycle), sitting or otherwise positioned to remain in place, looking at an autonomous vehicle, intending to enter the road, actively crossing the road, not associated with a road and indicating that the autonomous vehicle should proceed (e.g., waving at the vehicle, nodding), etc. The ML architectures may additionally or alternatively be used to classify light states, such as traffic signal states, vehicle indicator states, etc. For example, the ML architecture may classify vehicle indicator lights as indicating that the vehicle is stopped, parked, reversing, attempting a left turn, attempting a right turn, etc., and / or the ML architecture may classify traffic signals as indicating permission to proceed, permission to proceed with caution, an unprotected turn, an indication to stop, etc. Output from the ML architectures discussed herein may, in some examples, output a confidence score associated with one or more of these attributes, and may output confidence scores associated with individual attributes.
[0008]
[0012] The ML architectures discussed herein may reduce the computational burden of detecting various object attributes from multiple instances of sensor data received over a period of time. The ML architectures may also increase the detection capabilities of a vision system, thereby enabling an autonomous vehicle to interpret its surroundings with greater accuracy, reducing delays and / or assistance requests generated by the vehicle, and increasing the scenarios in which the autonomous vehicle can navigate safely and efficiently.
[0009] Example scenario
[0013] FIG. 1 illustrates an example scenario 100 including a vehicle 102. In some examples, the example scenario 100 may be a real-world scenario and / or the example scenario 100 may be a representation of a real-world scenario modeled as a simulated scenario. In examples where the example scenario 100 is a simulated scenario, the example scenario 100 may be determined based at least in part on input received at a user interface of a computing device (e.g., a user of the computing device may define an environment, objects therein, and / or its characteristics) and / or the example scenario 100 may be based at least in part on log data received from one or more autonomous vehicles (i.e., data captured and stored during actual driving in a physical environment, including raw sensor data and data derived therefrom, including detections, predictions, control signals, etc.). The log data may be based at least in part on sensor data received at the autonomous vehicle, sensory data generated by a perception component, and / or instructions generated by a planning component. In some examples, the autonomous vehicle may store the log data and / or periodically transmit the log data to a remote computing device.
[0010]
[0014] In some examples, the vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey without the expectation that a driver (or passenger) controls the vehicle at all times. However, in other examples, the vehicle 102 may be a fully or partially autonomous vehicle having any other level or classification. It is contemplated that the techniques discussed herein may be applied to more than just robotic control, such as autonomous vehicles. For example, the techniques discussed herein may be applied to mining, manufacturing, augmented reality, and the like. Additionally, the vehicle 102 is depicted as a land vehicle, but may also be a spacecraft, a seaplane, and / or the like. In some examples, the vehicle 102 may be depicted in a simulation as a simulated vehicle. For the sake of brevity, the description herein does not distinguish between simulated and real-world vehicles. Thus, references to a "vehicle" may refer to a simulated vehicle and / or a real-world vehicle.
[0011]
[0015] In accordance with the techniques described herein and examples in which the scenario 100 is real-world, the vehicle 102 may receive sensor data from sensors 104 of the vehicle 102. For example, the sensors 104 may include position sensors (e.g., Global Positioning System (GPS) sensors), inertial sensors (e.g., acceleration sensors, gyroscope sensors, etc.), magnetic sensors (e.g., compasses), position / speed / acceleration sensors (e.g., speedometers, drive system sensors), depth position sensors (e.g., lidar sensors, radar sensors, sonar sensors, time-of-flight (ToF) cameras, depth cameras, and / or other depth-sensing sensors), image sensors (e.g., cameras), audio sensors (e.g., microphones), and / or environmental sensors (e.g., barometers, hygrometers, etc.). In some examples, the simulated sensors may correspond to at least one of the sensors 104 on the vehicle 102, and in the simulation, one or more of the sensors 104 may be simulated. In some examples, the simulated sensor position may correspond to the relative position of one of the sensors 104 with respect to the vehicle 102 .
[0012]
[0016] The sensors 104 may generate sensor data, which may be received by a computing device 106 associated with the vehicle 102. In other examples, however, some or all of the sensors 104 and / or computing devices 106 may be located separately and / or remotely from the vehicle 102, and data capture, processing, command, and / or control may be communicated to and from the vehicle 102 by one or more remote computing devices via wired and / or wireless networks.
[0013]
[0017] The computing device 106 may include a memory 108 that stores a localization component 110, a perception component 112, a planning component 114, and / or a machine learning (ML) component 116. In some examples, the ML component 116 may be part of the perception component 112, and the ML component 116 may include an ML architecture discussed herein. In general, the localization component 110 may include software and / or hardware systems for determining the attitude (e.g., position and / or orientation) of the vehicle 102 relative to one or more coordinate frames (e.g., relative to the environment, relative to the road, relative to an inertial direction of motion associated with the autonomous vehicle). The localization component 110 may output at least a portion of this data to the perception component 112, which may output at least a portion of the localization data and / or use the localization data as a reference for determining at least some of the perception data.
[0014]
[0018] The perception component 112 may determine what is present in the environment around the vehicle 102 and may include a prediction component that predicts detected objects and / or environmental conditions. The ML component 116 may be at least a part of how the perception component 112 determines what is present in the environment, particularly to detect attributes of objects from sensor data determined over time. The planning component 114 may determine how to operate the vehicle 102 according to information received from the localization component 110 and / or the perception component 112. The localization component 110, the perception component 112, the planning component 114, and / or the ML component 116 may include one or more machine learning (ML) models and / or other computer-executable instructions.
[0015]
[0019] In some examples, the localization component 110, the perception component 112, and / or the ML component 116 may receive sensor data from the sensors 104. In some examples, the perception component 112 may determine data related to objects in the vicinity of the vehicle 102 (e.g., classifications, instance segmentations, trajectories associated with detected objects), route data identifying a destination of the vehicle, global map data identifying road features (e.g., features detectable at different sensor means useful for localizing an autonomous vehicle), local map data identifying features detected in the vicinity of the vehicle (e.g., locations and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and any other features detectable at various sensor means), and the like. In some examples, the objects around the vehicle 102 may be simulated objects of a simulated environment. The data generated by the perception component 112 may be collectively referred to as perception data. Once the perception component 112 generates the perception data, the perception component 112 may provide the perception data to the planning component 114.
[0016]
[0020] In some examples, the predictive data determined by the perception component 112 may additionally or alternatively be based at least in part on map data or other data. In some examples, the predictive data may include a top-down segmentation of the environment, as described in more detail in U.S. patent application Ser. No. 15 / 963,833, which is incorporated by reference in its entirety for all purposes, and / or a top-down prediction associated with the environment, as described in more detail in U.S. patent application Ser. No. 16 / 779,576, which is incorporated by reference in its entirety for all purposes. The map data map is stored by the memory 108. In some examples, the predictive data and / or other sensory data may be stored as part of the logged data.
[0017]
[0021] For example, the perception component 112 and / or the ML component 116 may receive an image 118 (and / or other sensor data) that includes representations of a vehicle 120, cyclists, pedestrians, sign objects, and a variety of different environmental objects (e.g., roads, sidewalks, trees, buildings). The perception component 112 may determine a region of interest (ROI) 122 based at least in part on detecting the vehicle 120 as represented in the image 118. In the illustrated example, the ROI 122 is a bounding box, although other techniques for identifying the ROI are contemplated, such as instance segmentation, semantic segmentation, masks, and / or bounding shapes other than rectangular. FIG. 1 additionally illustrates multiple other ROIs that may be generated by the perception component 112 for different objects that may be detected by the perception component 112, although for simplicity and clarity, they are not all labeled. For example, other ROIs may identify a cyclist, another vehicle, and a traffic sign, although it will be understood that additional or alternative portions of the image (and / or other sensor data) may be detected by the perception component 112.
[0018]
[0022] In some examples, the perception component 112 may additionally or alternatively determine a classification of the object by a first ML model. For example, the first ML model may be trained to output an indication of a classification from among a plurality of classifications that corresponds to the highest probability determined by the ML model. The general classifications may include, for example, classifications such as "cyclist," "vehicle," "pedestrian," "animal," and "environmental object." In the illustrative example, these classifications include classification 124 "cyclist," classification 126 "vehicle," classification 128 "pedestrian," and classification 130 "sign." For clarity, only a sampling of general classifications is shown in FIG. 1, but it will be understood that the first ML model of the perception component 112 may determine a classification for one or more ROIs and / or objects detected from the sensor data.
[0019]
[0023] Additionally or alternatively, the perception component 112 may determine a track, which may indicate historical, current, and / or predicted position, orientation, velocity, acceleration, state, etc., associated with an object. Additionally or alternatively, the perception component 112 may determine portions of the sensor data associated with a trajectory. For example, a trajectory may identify portions of different instances of the sensor data associated with the same object. To further illustrate, a trajectory may identify ROIs from different images associated with different times that are all associated with the same object. In other words, a trajectory identifies different object detections in time as associated with the same object.
[0020]
[0024] In some examples, the perception component 112 may provide sensor data associated with the trajectory to the ML component 116. For example, the perception component 112 may provide the last n images associated with the object (as identified by the trajectory) to the ML component 116, where n is a positive integer. In some examples, the images provided to the ML component 116 may be portions of the images identified by the ROI. For example, a portion of the image 118 identified by the ROI 122 may be provided as an input to the ML component 116. In some examples, the perception component 112 may provide such image portions as input to the ML component 116, as determined by the respective ML models of the perception component 112 and / or in batches of n images. For further details regarding ML models for determining object trajectories and determining ROIs, instance segmentation, and / or semantic segmentation associated with detected objects, please see U.S. patent application Ser. No. 16 / 235,862, filed December 28, 2018, and U.S. patent application Ser. No. 16 / 297,381, filed March 2, 2019.
[0021]
[0025] As discussed further herein, the ML component 116 may include a cache for storing portions of sub-outputs of the ML component 116 that are within the processing of the ML component 116. The cache may be part of the memory 108, or the cache may be part of separate hardware associated with the ML component 116, such as an integrated circuit, a gate array, or part of a graphics processing unit. The ML component 116 may output attributes with additional granularity compared to the classifications output by other portions of the perception component 112. For example, these attributes may include whether a pedestrian or driver is paying attention to or looking at the vehicle 102, whether an object (e.g., pedestrian 132, animal) is attempting to cross the road, whether the object is idling on the side of the road (e.g., whether it does not indicate a current intent to enter the road, but indicates some relationship to the road that may enter the road or otherwise affect traffic), whether the object is entering the road or actively crossing the road, whether the object is not associated with the road (e.g., whether it does not indicate an intent to interact with the road by walking along the sidewalk, playing in a field on the other side from the road, etc.), whether the object is indicating that the vehicle 102 should proceed, etc.
[0022]
[0026] The planning component 114 may determine the trajectory 134 based at least in part on the sensory data and / or other information, such as, for example, attributes determined by the ML component 116. For example, the planning component 114 determines a path for the vehicle 102 from a first location to a second location and, substantially simultaneously, based at least in part on the sensory data and / or simulated sensory data, selects a number of potential trajectories for controlling the movement of the vehicle 102 according to a receding horizon technique (e.g., one microsecond, half a second), and selects one of the potential trajectories as a trajectory 134 that the vehicle 102 may use to generate drive control signals that may be transmitted to drive components of the vehicle 102 to control the vehicle to travel the path (e.g., to avoid any detected objects).
[0023]
[0027] 1 shows an example of such a trajectory 134 represented as arrows indicating direction, speed, and / or acceleration, but the trajectory itself may include commands and / or target (future) states of the vehicle 102, which may then actuate the drive system of the vehicle 102. For example, the trajectory 134 may include instructions for a controller of the autonomous vehicle 102 to actuate drive components of the vehicle 102 to provide a steering angle and / or steering rate, which may provide a vehicle position, vehicle speed, and / or vehicle acceleration (or a simulated version thereof, if the autonomous vehicle is simulated) to track the commands and / or target states. The trajectory 134 may include a target direction, target steering angle, target steering rate, target position, target speed, and / or target acceleration that the controller tracks.
[0024] Exemplary System
[0028] FIG. 2 illustrates a block diagram of an example system 200 that implements the techniques discussed herein. In some examples, the example system 200 may include a vehicle 202 that may represent the vehicle 102 of FIG. 1. In some examples, the vehicle 202 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey without the expectation that a driver (or passenger) controls the vehicle at all times. However, in other examples, the vehicle 202 may be a fully or partially autonomous vehicle having any other level or classification. Additionally, in some examples, the techniques described herein may be usable by non-autonomous vehicles.
[0025]
[0029] Vehicle 202 may include vehicle computing device 204, sensors 206, emitters 208, network interfaces 210, and / or drive components 212. By way of example and not limitation, vehicle computing device 204 may be the same as or represent the same type as computing device 106, and sensor 206 may be the same as or represent the same type as sensor 104. System 200 may additionally or alternatively include computing device 214.
[0026]
[0030] In some examples, the sensors 206 may represent the sensors 104 and may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Positioning System (GPS), compass, etc.), inertial sensors (e.g., Inertial Measurement Units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), image sensors (e.g., Red-Green-Blue (RGB), Infrared (IR), intensity, depth, time-of-flight cameras, etc.), microphones, wheel encoders, environmental sensors (e.g., thermometers, hygrometers, light sensors, pressure sensors, etc.), etc. The sensors 206 may include multiple instances of each of these or other types of sensors. For example, a radar sensor may include individual radar sensors located at the corners, front, back, sides, and / or top of the vehicle 202. As another example, a camera may include multiple cameras positioned at various locations about the exterior and / or interior of the vehicle 202. The sensors 206 may provide input to the vehicle computing device 204 and / or the computing device 214. In some examples, the simulated sensors discussed herein may correspond to a particular type of sensor and / or any sensor. As discussed herein, the location associated with a simulated sensor may correspond to a position and / or an origin (e.g., a focal point) of the sensor's field of view relative to the vehicle 202 and / or a direction of motion of the vehicle 202.
[0027]
[0031] The vehicle 202 may also include emitters 208 that emit light and / or sound, as described above. The emitters 208 in this example may include interior audio and visual emitters for communicating with occupants of the vehicle 202. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 208 in this example may also include exterior emitters. By way of example and not limitation, the exterior emitters in this example may include lights for indicating direction of travel or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.
[0028]
[0032] Vehicle 202 may also include a network interface 210 that enables communication between vehicle 202 and one or more other local or remote computing devices. For example, network interface 210 may facilitate communication with other local computing devices on vehicle 202 and / or drive components 212. Network interface 210 may also additionally or alternatively enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Network interface 210 may additionally or alternatively enable vehicle 202 to communicate with computing device 214. In some examples, computing device 214 may include one or more nodes of a distributed computing system (e.g., a cloud computing architecture).
[0029]
[0033] The network interface 210 may include physical and / or logical interfaces for connecting the vehicle computing device 204 to another computing device or network, such as the network 216. For example, the network interface 210 may enable Wi-Fi-based communications, such as over frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communications protocol that allows each computing device to interface with other computing devices. In some examples, the vehicle computing device 204 and / or the sensors 206 may transmit sensor data to the computing device 214 over the network 216 at a particular frequency, after a predetermined period of time, in near real-time, etc.
[0030]
[0034] In some examples, the vehicle 202 may include one or more drive components 212. In some examples, the vehicle 202 may have a single drive component 212. In some examples, the drive component 212 may include one or more sensors for detecting conditions surrounding the drive component 212 and / or the vehicle 202. By way of example and not limitation, the sensors of the drive component 212 may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive component, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive component, cameras, or other image sensors, ultrasonic sensors for acoustically detecting objects surrounding the drive component, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive component 212. In some cases, the sensors on the drive component 212 may overlap or supplement corresponding systems (e.g., sensors 206) of the vehicle 202.
[0031]
[0035] The drive components 212 may include numerous vehicle systems, including a high-voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high-voltage junctions, high-voltage cables, charging systems, charging ports, etc.). Additionally, the drive components 212 may include a drive components controller that may receive and preprocess data from sensors and may control the operation of various vehicle systems. In some examples, the drive components controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components that perform various functions of the drive components 212. Additionally, the drive components 212 may also include one or more communication connections that enable each drive component to communicate with one or more other local or remote computing devices.
[0032]
[0036] The vehicle computing device 204 may include a processor 218 and a memory 220 communicatively coupled to the one or more processors 218. The memory 220 may represent the memory 108. The computing device 214 may also include a processor 222 and / or a memory 224. The processor 218 and / or 222 may be any suitable processor capable of executing instructions to perform data processing and operations as described herein. By way of example and not limitation, the processor 218 and / or 222 may include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs)), gate arrays (e.g., field programmable gate arrays (FPGAs)), and / or any other device or portion of a device that processes electronic data and converts the electronic data into other electronic data that may be stored in registers and / or memory.
[0033]
[0037] Memory 220 and / or 224 may be examples of non-transitory computer-readable media. Memory 220 and / or 224 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods and functions attributed to the various systems described herein. In various implementations, memory may be implemented using suitable memory technology such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which the ones shown in the accompanying drawings are merely examples relevant to the discussion herein.
[0034]
[0038] In some examples, memory 220 and / or memory 224 may store a localization component 226, a perception component 228, a planning component 230, and / or an ML architecture 232, any of which zero or more portions may be hardware, such as a GPU, a CPU, and / or other processing unit. By way of example and not limitation, localization component 226 may be the same as or represent the same type as localization component 110, perception component 228 may be the same as or represent the same type as perception component 112, planning component 230 may be the same as or represent the same type as planning component 114, and ML architecture 232 may be the same as or represent the same type as ML component 116.
[0035]
[0039] In at least one example, the localization component 226 may include hardware and / or software to receive data from the sensors 206 to determine a position, velocity, and / or orientation (e.g., one or more of an x position, a y position, a z position, a roll, a pitch, or a yaw) of the vehicle 202. For example, the localization component 226 may include and / or request / receive a map 238 of the environment and be able to continually determine the position, velocity, and / or orientation of the autonomous vehicle within the map 238. In some examples, the localization component 226 may receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., using simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneously (CLAMS), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the position, attitude, and / or velocity of the autonomous vehicle. In some examples, the localization component 226 may provide data to various components of the vehicle 202 to determine an initial position for the autonomous vehicle for generating a trajectory and / or for generating map data, as discussed herein. In some examples, the localization component 226 may provide the perception component 228 with the position and / or orientation of the vehicle 202 relative to the environment and / or sensor data associated with the environment.
[0036]
[0040] In some examples, the perception component 228 may include a primary perception system and / or a prediction system implemented in hardware and / or software. The perception component 228 may detect objects in the vehicle's 202 environment (e.g., identify that an object is present), classify the objects (e.g., determine an object type associated with the detected object), segment the sensor data and / or other representation of the environment (e.g., identify a portion of the sensor data and / or representation of the environment as associated with the detected object and / or object type), and determine characteristics associated with the object (e.g., a trajectory identifying a current, predicted, and / or previous position, orientation, velocity, and / or acceleration associated with the object). The perception component 228 may include a prediction component that predicts the actions / states of dynamic components in the environment, such as moving objects. Data determined by the perception component 228 is referred to as perception data.
[0037]
[0041] In some examples, the perception component 228 may include an ML architecture 232. For example, the ML architecture 232 may be part of the pipeline of the perception component 228. In other words, the ML architecture 232 may receive as inputs outputs from one or more other parts of the perception component 228, such as a sensor data pre-processing component, an ML model, etc. For example, the inputs to the ML architecture 232 may include portions of sensor data identified as associated with an object by other parts of the perception component 228. The portions may be identified by instance segmentation, semantic segmentation, ROI, etc. In some examples, the ML architecture 232 may be configured to receive sensor data over multiple time points.
[0038]
[0042] The planning component 230 may receive the position and / or orientation of the vehicle 202 from the localization component 226 and / or the sensory data from the perception component 228, and may determine instructions to control the operation of the vehicle 202 based at least in part on any of this data. In some examples, determining the instructions may include determining the instructions based at least in part on a format associated with a system with which the instructions are associated (e.g., a first instruction for controlling the movement of the autonomous vehicle may be formatted in a first format of messages and / or signals (e.g., analog, digital, aerodynamic, kinematic) that the system controller 236 and / or the drive component 212 may analyze / execute, and a second instruction for the emitter 208 may be formatted according to a second format associated therewith). In some examples, if the planning component 230 may include hardware / software-in-the-loop in a simulation (e.g., to test and / or train the planning component 230), the planning component 230 may generate instructions that may be used to control the simulated vehicle.
[0039]
[0043] Memory 220 and / or memory 224 may additionally or alternatively store a mapping system, a planning system, a ride management system, etc. Although perception component 228, planning component 230, and / or performance component 234 are shown as stored in memory 220 and / or 224, perception component 228, planning component 230, and / or performance component 234 may include processor-executable instructions, machine learning models (e.g., neural networks), and / or hardware.
[0040]
[0044] As described herein, the localization component 226, the perception component 228, the planning component 230, and / or the ML architecture 232 may include one or more ML models. For example, the localization component 226, the perception component 228, the planning component 230, and / or the ML architecture 232 may each include a different ML model pipeline. In some examples, the ML model may include a neural network. An exemplary neural network is a biological algorithm that passes input data through a series of connection layers to produce an output. Individual layers in a neural network may also include another neural network, or may include any number of layers (convolutional or not). As can be understood in the context of the present disclosure, the neural network may utilize machine learning, which can refer to a broad class of such algorithms in which an output is generated based on learned parameters.
[0041]
[0045] Although discussed in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative bisection 3 (ID3), chi-squared automated interaction detection (CHAID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average-one dependent estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backpropagation, etc.), and so on. The algorithms may include, but are not limited to, deep learning algorithms (e.g., deep Boltzmann machines (DBM), deep belief networks (DBN), convolutional neural networks (CNN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generation (blending), gradient boosting machines (GBM), gradient boosting regression trees (GBRT), random forests), support vector machines (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc.Additional examples of architectures include neural networks such as ResNet-50, ResNet-101, VGG, DenseNet, and PointNet. In some examples, the ML models discussed herein may include PointPillars, SECOND, top-down feature layers (see, e.g., U.S. Patent Application No. 15 / 963,833, which is incorporated by reference in its entirety), and / or VoxelNet. Architectures that optimize latency may include MobilenetV2, Shufflenet, Channelnet, Peleenet, and the like. The ML models may include residual blocks such as Pixor in some examples.
[0042]
[0046] Memory 220 may additionally or alternatively store one or more system controllers 240 (which may be part of the drive component), which may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 202. These system controllers 240 may communicate with and / or control corresponding systems of drive component 212 and / or other components of vehicle 202. For example, planning component 230 may generate instructions based at least in part on sensory data generated by perception component 228 and / or simulated sensory data, and may send the instructions to a system controller, which may control operation of vehicle 202 based at least in part on the instructions.
[0043]
[0047] 2 is shown as a distributed system, it should be noted that in alternative examples, components of the vehicle 202 may be associated with the computing device 214 and / or components of the computing device 214 may be associated with the vehicle 202. That is, the vehicle 202 may perform one or more functions associated with the computing device 214, or vice versa.
[0044] Exemplary ML Component Architecture
[0048] FIG. 3 illustrates an example architecture 300 of the ML component 116 / ML architecture 232. This arrangement of ML layers and caches may result in the efficiencies highlighted herein and may increase the attributes detectable by the autonomous vehicle's perception components. The example architecture discussed herein is discussed as an image for ease of understanding, but may be adapted for additional or alternative sensor data types beyond images. In some examples, the images discussed herein may be received from a single camera or multiple cameras. In some examples, the received sensor data, e.g., image data or point cloud data, may be received from an upstream component within the perception component 228. For example, the example architecture 300 may receive a first image 302, a second image 304, and an nth image 306, which may include portions of the respective images identified by an upstream component (e.g., via an ROI, instance segmentation, and / or semantic segmentation output by the upstream component) and may be associated via a trajectory output by another upstream component. The trajectory may identify the first image 302, the second image 304, and the nth image 306 as being associated with the same object. In other words, the techniques discussed herein may use a batch of n images to determine a final output, which may be a classification, as discussed herein. The n images may correspond to a sliding time window of the most recent n images received from the camera.
[0045]
[0049] In some examples, the exemplary architecture 300 may be configured to process n time steps of sensor data, where n is a positive integer. In other words, the first image 302 through the nth image may be associated with different times going back n time steps in the past. The time interval between the individual images may depend on the sensor output and the upstream sensory component output frequency. For example, if the interval is 500 milliseconds and n is 9, the first image 302 may be associated with a time 4 seconds in the past, the second image 304 may be associated with a time 3.5 seconds in the past, and the nth image may be associated with the current time. Other intervals and options for n are contemplated, such as, for example, 10 milliseconds, 20 milliseconds, 100 milliseconds, 250 milliseconds, 1 second, 2 seconds, and n may be 2, 3, 4, 5, 10, 20, or any other number. In some examples, the time interval between received sensor data may not be constant and may vary within a tolerance range. For example, the system may tolerate variations in when sensor data is received, such as loss of object tracking for several frames (e.g., due to occlusion), loss or corruption for several frames, distortion between sensor data (e.g., due to rain, glare), etc. Note that n may vary proportionally to the time interval between time steps such that the output of architecture 300 is up-to-date. For example, n may decrease as the interval becomes longer.
[0046]
[0050] In some examples, the ML architecture 300 may resize the nth image 306 to a standard size before processing the image. Additionally or alternatively, the ML architecture 300 may obtain a larger portion of the image than indicated by the ROI for use as the nth image 306. For example, an upstream component may detect a pedestrian and indicate a portion of the image occupied by the pedestrian, but the pedestrian may be holding a sign, which may not be correctly detected as a pedestrian by the upstream component. Thus, the ML architecture 300 may obtain an additional portion of the image outside of the ROI for use by the ML architecture 300.
[0047]
[0051] Nonetheless, the most recently received image, the nth image 306, may be provided as an input to the ML backbone 308, which may include one or more ML layers. For example, the ML backbone 308 may be a ResNet or other suitable neural network, such as ResNeXt, DenseNet, visual transformer (ViT). For example, if the ML backbone 308 is a ResNet, the ML backbone may include a convolutional pre-processing layer 310 (e.g., a feed-forward neural network layer) and a residual layer 312(1)-(m), where m is a positive integer. The ML backbone 308 may be used sequentially for images as they are received. In other words, the same ML backbone 308 is used for every new image that is received, and its output may be stored as discussed herein. Using this same ML backbone may save training time and reduce system complexity, while using stored outputs (i.e., cache 324) in the middle of the pipeline may reduce the computational load and processing time of the architecture 300.
[0048]
[0052] The residual layer 312(1) may include a ResNet block, which may include various components including convolutional layers 314(1)-(4), skip connections 316(1) and (2), and a projection shortcut 318 that may be part of the skip connection. These various parts may process input 320 received from the previous layer and generate output 322 that is provided to the next layer. Note that the projection shortcut may be replaced with an identity shortcut. The projection shortcut may perform the convolution to ensure that the quantities at the summation operation (the end of the skip connection and the output of the last convolutional layer before the summation) correspond. The identity shortcut simply passes the input through, which is a standard skip connection. The convolutional layers 314 may include convolution of the input to the convolutional component with trained kernels (e.g., kernels are parts that can be tuned via ML training), batch normalization (see, e.g., U.S. Patent Application Serial No. 16 / 403,476, filed May 3, 2019), and, for intermediate layers, activation layers such as rectified linear units (ReLU) layers, Gaussian error linear units (GELU), Heaviside activation, sigmoidal or hyperbolic functions (although sigmoidal and hyperbolic functions may contribute to gradient reduction by being saturated by the input (i.e., cannot represent values beyond a certain amount). Essentially, this activation layer characterizes the output at a particular point in the feature data as “on” i.e., 1.0 or “off” i.e., 0.0 / active or inactive. Note that the final convolutional component 314(4) may perform convolution and batch normalization, but not ReLU activation.
[0049]
[0053] In other words, the input to one of the residual components 312 may be convolved in a first convolutional layer, batch normalized, and passed through a ReLU activation before repeating the process twice in a second convolutional layer. The output of the second layer may be summed with the input to the residual component (or a projection version of the input if a projection layer is present in the skip connection). This process may be repeated twice in the third and fourth convolutional layers before being output by that particular residual component.
[0050]
[0054] This represented ML backbone contains approximately 11 million trainable parameters, but it is understood that deeper ML backbones, i.e., ML backbones with a greater number of layers, may be used, in which the ML backbone may have 21.2 million parameters (34 layers), 23.5 million parameters (50 layers), 42.5 million parameters (101 layers), 58.2 million parameters (152 layers), etc. that are trainable. The layers may be modified to reduce the loss calculated between the logits or confidence scores output by the ML architecture and the ground truth attributes that identify the attributes of the object (e.g., the values of the kernels of the convolutional and / or projection layers, or other ML layers downstream from the ML backbone). Such ground truth attributes may be labeled by humans or by complex ML systems that may be too computationally burdensome to include in a vehicle. Such ML systems may be executed on distributed computing systems, such as cloud computing systems.
[0051]
[0055] Ultimately, the ML backbone 308 may determine feature data associated with the nth image 306, which is p-dimensional, and the nth feature data 322, where p is a positive integer (e.g., 32, 64, depending on the number of layers in the CNN and the amount of padding within the layers). The nth feature data 322 may be stored in a cache 324. Feature data associated with previously received images (the first image 302 and the second image 304) may have been previously stored in the cache 324. In other words, the ML backbone 308 may have previously determined the first feature data 326 by processing the first image 302 through the ML backbone 308, and may have previously determined the second feature data 328 by processing the second image 304 through the ML backbone 308. The first image 302 and the second image 304 (as well as any other intermediate images) are not reprocessed upon receipt of the nth image 306, saving computational load. Cache 324 may be part of an integrated circuit that includes ML architecture 300, although in additional or alternative examples, cache 324 may be a cache of a graphics processing unit, a central processing unit, a gate array, or the like.
[0052]
[0056] In some examples, the feature data may include feature maps, also referred to as activation maps, which are the result of convolving kernels in the model backbone with each input to each layer. The feature data is typically a high-dimensional data field that is difficult for humans to understand, such as a tensor (e.g., 56×64 dimensions, 28×128 dimensions, etc.). In other words, the feature data is a computational and / or neural network transformation.
[0053]
[0057] Based at least in part on receiving the nth feature data 322, the ML architecture 300 may provide as input to the ML layer 330 the first feature data 326, the second feature data 328, the nth feature data 322, and any intervening feature data associated with any other images between the first image and the nth image. In some examples, a single ML layer 330 is used for each feature data at this stage, but it is contemplated that more ML layers 330 may be used. In other words, the ML layer 330 may process the first feature data 326 through the nth feature data 322 separately to generate the first feature data 332, the second feature data 334 through the nth feature data 336, all of which may be q-dimensional, where q is a positive integer greater than p. For example, if p is 32, then q may be 128.
[0054]
[0058] It should be noted that in the exemplary architecture 300 for classifying light conditions, such as traffic signal conditions or vehicle light conditions, the ML layer 330 may be an identity layer that simply passes input thereto, or may not be present.
[0055]
[0059] The feature data output by the ML layer 330 (i.e., the first feature data 332 through the nth feature data 336) may be concatenated together and provided as input to the ML layer 338, which may be designed to reduce the dimensionality of the concatenated feature data, which may be of size n to q. The ML layer 338 may project the concatenated feature data from n to q dimensions into a q-dimensional space to achieve reduced features 340. For example, if q is 128 and n is 5, the concatenated feature data may have a dimension of 640 and the reduced features 340 may have a dimension of 128.
[0056]
[0060] The reduced features 340 may be provided by the ML architecture as input to a final ML layer 342, which may include output heads equal to the number of attributes being predicted. For example, if the ML architecture 300 is configured to predict 10 attributes, the ML layer 342 may have 10 output heads, although any other number of attributes is contemplated, e.g., 2, 3, 4, 5, 10, 15, 20, etc. In some examples, the output heads of the ML layer 342 may be associated with the particular attribute being predicted and may output logits indicating the probability that the detected object has the attribute associated with the head. The ML layer 342 may output these logits 344 in association with different attributes, and / or the ML layer 342 may include layers such as sigmoid, tanh, ReLU, GELU, etc., after the output heads or some other layers to convert the logits 344 into confidence scores between 0 and 1, with 1 indicating full confidence that the object has the respective attribute and 0 indicating no confidence that the object has the respective attribute. The confidence scores output by the exemplary architecture 300 may be associated with the most recent time, but may also be based on a time window that goes back n time steps (e.g., logit 344 may be associated with the nth time step, but may be generated based on images 302-306).
[0057]
[0061] The ML layers 330, 338, and 342 may each be a fully connected layer with different input / output shapes. Collectively, the ML layers 330, 338, and 342 may make up the model time head 346.
[0058] Example Attributes
[0062] FIG. 4 illustrates three example frames (discrete portions of sensor data in time) 400-404 and the attributes predicted by ML architecture 300 for each of the frames. The illustrated frames are images, but it is understood that the frames may be part of point cloud data or any other sensor data. FIG. 4 also illustrates a timeline 406 and other frames received and processed by ML architecture 300, such as frame 408, which is shown as a rectangle but is not associated with one of the classifications discussed with respect to FIG. 4. FIG. 4 illustrates frames each of which may be classified as having the same attribute. For example, each of the frames under the brackets indicated by 410 may be classified as showing an object having the attribute "idling", with the first frame 400 being an example of a frame that may be so classified. The frames under brackets 412 may be classified as showing an object having the attribute "intent to enter road", with the second frame 402 being an example of a frame that may be so classified. The frames below bracket 144 may be classified as showing an object having the attribute "crossing the road", and third frame 404 is an example of a frame that may be so classified.
[0059]
[0063] The highest confidence score output by the ML architecture 300 for the first frame 400 may be associated with an idling attribute, which indicates that the object is associated with the road but does not express an intent to enter the road. To classify the first frame 400 as such, the ML architecture 300 may receive and process as input frame 416, which is shown using diagonal hatching. In other words, the attribute output by the ML architecture 300 in relation to the first frame 400 depends on the ML architecture 300 processing the first frame 400 and the three frames that precede the first frame 400 in time, in an example where n is 4. It is contemplated that N may be larger or smaller. As described above and below, processing the frames 416 may include processing each of the frames 416 using the ML backbone for each frame as they are received, and storing the output. Thus, for a first frame 400, processing frame 416 may involve processing the first frame 400 using the ML backbone and recalling past outputs for other frames 416 for the remainder of the ML architecture.
[0060]
[0064] In additional or alternative examples, the ML architecture 300 may determine an attribute to output if it meets or exceeds a confidence threshold and / or if the confidence score is greater than other confidence scores associated with different attributes. In some examples, attributes may be grouped into different groups and the ML architecture 300 may determine to output any confidence score that meets or exceeds a confidence threshold and is greater than all other confidence scores in the same group. For the first frame 400, this may result in outputting both an idle attribute, a ready attribute that identifies that the object is standing or ready / available to enter the road (e.g., the pedestrian is standing, the pedestrian is in a wheelchair and is ready to move), and / or that the object is looking at the vehicle. In some examples, all three of the latter examples may be output in examples where the output head includes at least three groups, one group for intent to enter a road, another group for readiness to enter a road and / or object state (e.g., standing, sitting, walking, running, other (catch-all for dancing, abnormal behavior, etc.)), and / or a third group for whether the object is looking at, watching, or paying attention to the vehicle. An additional group may be directed to whether another object anticipates the vehicle's progress or indicates that the vehicle should proceed, such as waving at the vehicle, seeing the vehicle and not moving, showing signs that the vehicle should stop or move, etc. In practice, more or fewer groups may be used.
[0061]
[0065] For the second frame 402, the highest confidence score output by the ML architecture 300 may be associated with an entry intent attribute that indicates that the object is expressing an intent to enter the road. To classify the second frame 402 as such, the ML architecture 300 may receive and process as input frame 418, shown using diagonal hatching, as described above.
[0062]
[0066] For the third frame 404, the highest confidence score output by the ML architecture 300 may be associated with actively entering / crossing the road, indicating that the object is entering / already entering the road. Additional attributes that may be output by the ML architecture 300 may indicate that the object is not looking at the vehicle, is walking, etc. To classify the second frame 402 as such, the ML architecture 300 may receive and process as input frame 420, shown using diagonal hatching, as described above.
[0063] Example Process
[0067] 5 illustrates a flow diagram of an example process 500 for detecting object attributes and / or intent and reducing computational bandwidth for doing so. In some examples, the example process 500 may be performed by the vehicle machine learning architecture 300. It is understood that the techniques discussed herein discuss images but may be adapted to other types of sensor data.
[0064]
[0068] At operation 502, the example process 500 may include receiving a first image associated with a first time. The first image may be a recently received image associated with an object identified by tracking. The first image may be the nth image 306 of FIG. 3. If the object was recently detected, the ML component 300 may wait and accumulate n images before commencing operation. In additional or alternative examples, the ML component 300 may process images as they come in but not determine an output until n images have been processed.
[0065]
[0069] At operation 504, the example process 500 may include receiving a second image associated with a second time prior to the first time. Both the second image and the first image may be associated with the same object and may be part of a larger image. For example, the first image may be a first portion of the first image identified by a first ROI output by an upstream ML model, and the second image may be a second portion of the second image identified by a second ROI output by an upstream ML model. The two images may be associated by a track output by yet another ML model, see U.S. Patent Application No. 16 / 235,862, filed December 28, 2018.
[0066]
[0070] At operation 506, the example process 500 may include determining a first output by a first set of ML layers and based at least in part on the first image. The first set of ML layers may include a layer of the ML backbone 308. The ML backbone 308 may run once per image. In other words, the ML backbone 308 processes recently received images but does not re-run on previously received images, while other ML layers of the ML architecture 300 may process the output of the ML backbone 308, as discussed below.
[0067]
[0071] At operation 508, the example process 500 may include storing the first output in a memory (e.g., a cache). The second output determined by the first set of ML layers based at least in part on the second image may have been previously stored in the memory, for example, when the second image was received and was the most recent image.
[0068]
[0072] At operation 510, the example process 500 may include obtaining a second output from the memory, the second output being associated with the second image and having been previously processed by the first set of ML layers and stored in the memory.
[0069]
[0073] At operation 512, the example process 500 may include determining, by the one or more ML layers and based at least in part on the first output and the second output, a first confidence score associated with the first object attribute (e.g., pedestrian, animal, cyclist, machine operator) and a second confidence score associated with the pedestrian attribute. For example, the first object attribute may include an indication that the object is ready to enter a road, or a general state of the object (e.g., standing, sitting, ready to enter a road, not ready to enter a road, walking, running, etc.), an object indicator state (e.g., indicator of absence, pedestrians waiting sign, pedestrians waiting sign up, pedestrians indicating go, pedestrians indicating no go, cyclists indicating left turn, cyclists indicating right turn, cyclists indicating brake, vehicle lights indicating turn, vehicle lights indicating brake, traffic light indicating go, traffic light indicating stop, traffic light indicating caution), an object indicating intent to enter a road, an object expressing that it is entering / has entered a road, attention of the object (e.g., sees / does not see a vehicle, is / is not paying attention to a vehicle), and the second object attribute may indicate a different attribute. The confidence score may indicate the likelihood (e.g., posterior probability) that the object attribute accurately describes the object.
[0070]
[0074] At operation 514, the example process 500 may include controlling the vehicle based at least in part on at least one of the first confidence score or the second confidence score. Operation 514 may include determining which confidence score is greater and using that confidence score and / or determining whether the confidence score meets or exceeds a confidence score threshold. The confidence score threshold may be used to determine whether to output the attribute at all. For example, if the confidence score does not meet the confidence score threshold, the attribute may not be output, although in additional or alternative examples, the attribute associated with the largest confidence score may be output anyway. In either case, the confidence score may be output in association with the attribute. A planning component of the vehicle may use the attribute (and the confidence score in examples where the confidence score is also output) to determine a trajectory for controlling the vehicle.
[0071] Illustrative clauses
[0075] A: A method including: receiving a first image associated with a first time; receiving a second image associated with a second time prior to the first time, where the first image and the second image are associated with a same object; determining a first output by a first set of machine learning layers and based at least in part on the first image; storing the first output in a memory; obtaining a second output from the memory, where the second output is associated with the second image and has been previously processed by the first set of machine learning layers and stored in the memory; determining a first confidence score associated with a first pedestrian attribute and a second confidence score associated with a second pedestrian attribute by one or more machine learning layers and based at least in part on the first output and the second output; and controlling a vehicle based at least in part on at least one of the first confidence score or the second confidence score.
[0072]
[0076] B: The method of paragraph A, wherein the first pedestrian attribute indicates one of a pedestrian idling, the pedestrian intending to enter a road, or the pedestrian crossing a road, and the second pedestrian attribute indicates a different one of the pedestrian idling, the pedestrian intending to enter a road, or the pedestrian crossing a road.
[0073]
[0077] C: The method of any of paragraphs A or B, wherein the one or more machine learning layers include a second machine learning layer, a third machine learning layer, and a fourth machine learning layer, and determining the first confidence score associated with the first pedestrian attribute and the second confidence score associated with the second pedestrian attribute includes determining a third output by the second machine learning layer and based at least in part on the first output, determining a fourth output by the second machine learning layer and based at least in part on the second output, concatenating the third output and the fourth output as a concatenated output, providing the concatenated output as an input to the third machine learning layer, receiving a fifth output from the third machine learning layer, providing the fifth output as an input to the fourth machine learning layer, and receiving the third confidence score and the fourth confidence score from the fourth machine learning layer.
[0074]
[0078] D: The method of paragraph C, wherein the first set of machine learning layers includes multiple layers of a neural network, and the second machine learning layer, the third machine learning layer, and the fourth machine learning layer include different fully connected layers.
[0075]
[0079] E: The method of any one of paragraphs A-D, wherein the first image is a first portion of a first larger image and the second image is a second portion of a second larger image, and the first portion and the second portion are generated by a first machine learning component based at least in part on the first larger image and the second larger image.
[0076]
[0080] F: A system comprising: one or more processors; and a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations, the operations including: receiving a first image associated with a first time; receiving a second image associated with a second time prior to the first time; determining a first output by a first set of machine learning layers and based at least in part on the first image; storing the first output in a memory; and obtaining a second output from the memory, the second output being associated with the second image; determining attributes associated with objects in an environment by one or more machine learning layers and based at least in part on the first output and the second output; and controlling a vehicle based at least in part on the attributes.
[0077]
[0081] G: The system described in paragraph F, wherein the attributes associated with the object include an indication of a vehicle motion state, an indication of an indicator state of the object, an indication that the object is idling, an indication that the object intends to enter a road, or an indication that the object is crossing the road.
[0078]
[0082] H: The system of any of paragraphs F or G, wherein the memory is a cache and the cache stores n outputs of the first set of machine learning layers, where n is a positive integer associated with n past time steps.
[0079]
[0083] I: The system of any one of paragraphs F-H, wherein the first output and the second output are each processed once by the first set of machine learning layers.
[0080]
[0084] J: The system of any one of paragraphs F-I, wherein the attribute indicates that the pedestrian is idling near a road, that the pedestrian intends to enter the road, that the pedestrian is crossing the road, that the pedestrian is ready to enter the road, or that the pedestrian is not associated with the road.
[0081]
[0085] K: The system of any one of paragraphs F-J, wherein the one or more machine learning layers include a second machine learning layer, a third machine learning layer, and a fourth machine learning layer, and determining the attribute includes determining a third output by the second machine learning layer and based at least in part on the first output, determining a fourth output by the second machine learning layer and based at least in part on the second output, concatenating the third output and the fourth output as a concatenated output, providing the concatenated output as an input to the third machine learning layer, receiving a fifth output from the third machine learning layer, providing the fifth output as an input to the fourth machine learning layer, and receiving a confidence score associated with the attribute from the fourth machine learning layer.
[0082]
[0086] L: The system of paragraph K, wherein the first set of machine learning layers includes multiple layers of a neural network, and the second machine learning layer, the third machine learning layer, and the fourth machine learning layer include different fully connected layers.
[0083]
[0087] M: The system of any one of paragraphs F-L, wherein the first image is a first portion of a first larger image, the second image is a second portion of a second larger image, and the first portion and the second portion are generated by a first machine learning component based at least in part on the first larger image and the second larger image.
[0084]
[0088] N: One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: receiving a first image associated with a first time; receiving a second image associated with a second time prior to the first time; determining a first output by a first set of machine learning layers and based at least in part on the first image; storing the first output in a memory; and obtaining a second output from the memory, the second output being associated with the second image; determining attributes associated with objects in an environment by one or more machine learning layers and based at least in part on the first output and the second output; and controlling a vehicle based at least in part on the attributes.
[0085]
[0089] O: One or more non-transitory computer-readable media as described in paragraph N, wherein the attributes associated with the object include an indication of a vehicle motion state, an indication of an indicator state of the object, an indication that the object is idling, an indication that the object intends to enter a road, or an indication that the object is crossing a road.
[0086]
[0090] P: The one or more non-transitory computer-readable media of any of paragraphs N or O, wherein the memory is a cache and the cache stores n outputs of the first set of machine learning layers, where n is a positive integer associated with n past time steps.
[0087]
[0091] Q: One or more non-transitory computer-readable media described in any one of paragraphs N-P, wherein the first output and the second output are each processed once by the first set of machine learning layers.
[0088]
[0092] R: One or more non-transitory computer-readable media of any one of paragraphs N-Q, wherein the attribute indicates that a pedestrian is idling near a road, that the pedestrian intends to enter the road, that the pedestrian is crossing the road, that the pedestrian is ready to enter the road, or that the pedestrian is not associated with the road.
[0089]
[0093] S: the one or more machine learning layers include a second machine learning layer, a third machine learning layer, and a fourth machine learning layer, and determining the attribute includes determining a third output by the second machine learning layer and based at least in part on the first output, determining a fourth output by the second machine learning layer and based at least in part on the second output, concatenating the third output and the fourth output as a concatenated output, providing the concatenated output as an input to the third machine learning layer, receiving a fifth output from the third machine learning layer, providing the fifth output as an input to the fourth machine learning layer, and receiving a confidence score associated with the attribute from the fourth machine learning layer.
[0090]
[0094] T: One or more non-transitory computer-readable media described in any one of paragraphs N-S, wherein the first set of machine learning layers includes multiple layers of a neural network, and the second machine learning layer, the third machine learning layer, and the fourth machine learning layer include different fully connected layers.
[0091]
[0095] While the example clauses described above are described with respect to one particular implementation, it should be understood in the context of this specification that the content of the example clauses can also be implemented via a method, device, system, computer-readable medium, and / or another implementation. Additionally, any of Examples A-T can be implemented alone or in combination with any other one or more of Examples A-T.
[0092] summary
[0096] Although the subject matter has been described in language specific to structural features and / or method acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.
[0093]
[0097] The components described herein represent instructions that may be stored on any type of computer-readable medium and implemented in software and / or hardware. All of the methods and processes described above may be embodied and fully automated via software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Some or all of the methods may alternatively be embodied in dedicated computer hardware.
[0094]
[0098] At least some of the processes described herein are illustrated as logical flow graphs, with each operation representing a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, cause a computer or autonomous vehicle to perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.
[0095]
[0099] Unless otherwise indicated, conditional terms such as "may," "could," "may," or "might" are understood in the context to, among other things, state that an example includes certain features, elements, and / or steps that other examples do not include. Thus, such conditional terms are generally not intended to imply that a certain feature, element, and / or step is in any way required in one or more examples or that the one or more examples necessarily include logic for determining whether a certain feature, element, and / or step is included in or should be performed in any particular example, with or without user input or prompting.
[0096]
[0100] A phrase such as "at least one of X, Y, Z" should be understood to denote that the item, term, etc., refers to either X, Y, Z, or any combination thereof, including multiples of the individual elements, unless expressly stated otherwise. "a" refers to singular as well as plural.
[0097]
[0100] Any routine description, element, or block described herein and / or illustrated in the flow diagrams shown in the accompanying drawings should be understood as potentially representing a module, segment, or portion of code that includes one or more computer executable instructions for implementing a specific logical function or element in the routine. Alternative implementations include alternatives within the scope of the examples described herein that remove elements or functions, perform them in a different order from the order shown or described, such as substantially synchronized, reversed, additional operations, or omitting operations, depending on the functions involved, as can be understood by those skilled in the art. Note that the term substantially may indicate a range. For example, the term substantially simultaneously may indicate that two operations occur within a time range of each other, and the term substantially the same dimensions may indicate that two elements have dimensions within a range of each other.
[0098]
[0101] Many variations and modifications may be made to the examples described above, and elements thereof should be understood to be included among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.
Claims
1. 1. A system comprising: one or more processors; and a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations, the operations including: Receiving a first image associated with a first time; receiving a second image associated with a second time that is earlier than the first time; determining a first output by a first set of machine learning layers and based at least in part on the first image; storing the first output in a memory; obtaining a second output from the memory, the second output being associated with the second image; and determining attributes associated with objects in an environment by one or more machine learning layers and based at least in part on the first output and the second output; and controlling the vehicle based at least in part on the attribute.
2. The attributes associated with the object include: Display of vehicle motion status, Displaying the indicator state of an object, an indication that the object is idling; an indication that the object intends to enter the road; or The system of claim 1 including an indication that the object is crossing the road.
3. 3. The system of claim 1, wherein the memory is a cache and the cache stores n outputs of the first set of machine learning layers, where n is a positive integer associated with n past time steps.
4. The system of claim 1 , wherein the first output and the second output are each processed once by the first set of machine learning layers.
5. The system of any one of claims 1 to 4, wherein the attribute indicates that the pedestrian is idling near a road, that the pedestrian intends to enter the road, that the pedestrian is crossing the road, that the pedestrian is ready to enter the road, or that the pedestrian is not associated with the road.
6. The one or more machine learning layers include a second machine learning layer, a third machine learning layer, and a fourth machine learning layer, and determining the attribute includes: determining a third output by the second machine learning layer and based at least in part on the first output; and determining a fourth output by the second machine learning layer and based at least in part on the second output; and combining the third output and the fourth output as a combined output; and providing the concatenated output as an input to the third machine learning layer; receiving a fifth output from the third machine learning layer; and providing the fifth output as an input to the fourth machine learning layer; and receiving a confidence score associated with the attribute from the fourth machine learning layer.
7. 7. The system of claim 6, wherein the first set of machine learning layers comprises multiple layers of a neural network, and the second machine learning layer, the third machine learning layer, and the fourth machine learning layer comprise different fully connected layers.
8. the first image is a first portion of a first larger image; the second image is a second portion of a second larger image; and 8. The system of claim 1 , wherein the first and second portions are generated by a first machine learning component based at least in part on the first larger image and the second larger image.
9. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations, including: Receiving a first image associated with a first time; receiving a second image associated with a second time that is earlier than the first time; determining a first output by a first set of machine learning layers and based at least in part on the first image; storing the first output in a memory; obtaining a second output from the memory, the second output being associated with the second image; and determining attributes associated with objects in an environment by one or more machine learning layers and based at least in part on the first output and the second output; and controlling a vehicle based at least in part on the attribute.
10. The attributes associated with the object include: Display of vehicle motion status, Displaying the indicator state of an object, an indication that the object is idling; an indication that the object intends to enter the road; or The one or more non-transitory computer-readable media of claim 9 including an indication that the object is crossing a road.
11. 11. The one or more non-transitory computer-readable media of claim 9 or 10, wherein the memory is a cache and the cache stores n outputs of the first set of machine learning layers, where n is a positive integer associated with n past time steps.
12. 12. The one or more non-transitory computer-readable media of claim 9, wherein the first output and the second output are each processed once by the first set of machine learning layers.
13. 13. The one or more non-transitory computer-readable media of claim 9, wherein the attribute indicates that a pedestrian is idling near a road, that the pedestrian intends to enter the road, that the pedestrian is crossing the road, that the pedestrian is ready to enter the road, or that the pedestrian is not associated with the road.
14. The one or more machine learning layers include a second machine learning layer, a third machine learning layer, and a fourth machine learning layer, and determining the attribute includes: determining a third output by the second machine learning layer and based at least in part on the first output; and determining a fourth output by the second machine learning layer and based at least in part on the second output; and combining the third output and the fourth output as a combined output; and providing the concatenated output as an input to the third machine learning layer; receiving a fifth output from the third machine learning layer; and providing the fifth output as an input to the fourth machine learning layer; and and receiving a confidence score associated with the attribute from the fourth machine learning layer.
15. 15. The one or more non-transitory computer-readable media of claim 14, wherein the first set of machine learning layers comprises multiple layers of a neural network, and the second machine learning layer, the third machine learning layer, and the fourth machine learning layer comprise different fully connected layers.
Citation Information
Cited By
Crossing prediction system and crossing prediction method
WO2026191416A1
