Point anchor object detection

Abstract

An example computer-implemented method includes (a) obtaining sensor data describing an environment of an autonomous vehicle. An example computer-implemented method includes (b) obtaining a plurality of points describing an environment. An example computer-implemented method includes (c) determining an association between one or more of the plurality of points and an object in the environment using a machine-learned object detection model and based on the sensor data. An example computer-implemented method includes (d) generating an offset relative to one or more points of a spatial region of an environment associated with an object using a machine-learned object detection model.

Description

Technical Field

[0001] priority

[0002] This application claims priority and benefit to U.S. Patent Application No. 18 / 354,415, filed July 18, 2023, the entire contents of which are incorporated herein by reference. This application also claims priority and benefit to U.S. Provisional Patent Application No. 63 / 651,213, filed May 23, 2014, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Autonomous platforms can process data to perceive the environment in which they travel. For example, autonomous vehicles can use various sensors to perceive their environment and identify objects around them. Autonomous vehicles can then identify appropriate paths based on their perceived surroundings and navigate along paths with minimal or no human input. Summary of the Invention

[0004] This disclosure provides exemplary embodiments of systems and techniques for anchoring object detection to point data. Autonomous vehicles can process sensor data to detect objects in their environment. Autonomous vehicles can also access point data (e.g., map data, LiDAR data, RADAR data, etc.) that provides rich information about the environment, such as lane boundary information, elevation maps, object locations, etc. The machine learning object detection model of the autonomous vehicle perception system according to this disclosure can process sensor data fused with point data to directly determine the location of detected objects in the mapped environment.

[0005] For example, anchoring detection to point data can simplify the detection task by constraining the solution space of the detection output used for localization around the associated point (e.g., LiDAR echoes, map markers, etc.). For instance, point data from ranging sensors (e.g., LiDAR, RADAR) can provide points corresponding to ray echoes from the environment in three dimensions. The perception system can transform point locations into reference frames of sensor data. For example, an autonomous vehicle can localize itself within the environment using the returned point data mapping, thereby estimating its own position and orientation. By extension, using calibration data from onboard vehicle sensors (e.g., image sensors), the perception system can determine the relative orientation of the sensors relative to the point data. In this way, the perception system can use the estimated pose of the camera to project point locations onto a camera reference frame to detect objects represented within two-dimensional image data. This projection can immediately provide estimated three-dimensional position values ​​for pixels in the image data because the projected points carry their association with rich depth information from the ranging sensor data.

[0006] Similarly, map data can provide lane markings (e.g., markings along the lane centerline) for locating lanes on a road. These lane markings can encode the two-dimensional or three-dimensional position of the lane centerline. A perception system can transform the lane marking positions into reference frames of sensor data. For example, an autonomous vehicle can position itself within map data to estimate its own position and orientation within the mapped environment. By extending this by using calibration data from onboard vehicle sensors (e.g., image sensors), the perception system can determine the relative orientation of the sensors with respect to the map data. In this way, the perception system can use the estimated pose of the camera to project the lane marking positions onto a camera reference frame to detect objects represented within two-dimensional image data. This projection can immediately provide estimated three-dimensional position values ​​for pixels in the image data because the projected markings carry rich information about their association with the map data.

[0007] In the example, the object detection model can generate object detection output at the projection point location by optimizing the local solution space in the region of the projection point. For example, the object detection model can regress the bounding box dimension and the offset of the bounding box relative to the projection point location. In this way, for example, the system can infer the image scene to predict the bounding box while anchoring the prediction to a specific point in ranging data (e.g., LiDAR).

[0008] For example, in this example, the object detection model can generate object detection output at the location of the projected lane marker by optimizing the local solution space in the region of the projected point. For example, the object detection model can regress the bounding box dimension and the offset of the bounding box relative to the location of the projected lane marker. In this way, for example, the system can infer the image context to predict the bounding box while anchoring the prediction to a specific point on the map.

[0009] By simplifying the detection task in this way, perception systems can achieve better detection outputs with limited sensor data by leveraging the respective strengths of different sensors. For example, dense information from camera sensors can facilitate improved semantic understanding, while native depth information from point-based modalities (e.g., LiDAR, maps) can contribute to improved accuracy in detecting object location.

[0010] Similarly, the advantages of map data can be leveraged to improve performance over long distances. For example, in some cases, LiDAR echoes can become increasingly sparse over longer distances. Conversely, map data can be stored and retrieved at any resolution over any distance. Therefore, fusing map data of the environment with sensor data depicting the same environment can create a (relatively) dense grid of 3D reference locations, which can provide the basis for inference in object detection models even over long distances.

[0011] Furthermore, the example object detection model can be range-invariant. For example, the object detection model can operate without explicitly depending on the absolute range. In this way, for example, the object detection model can be applied to various ranges of sensor input and point projection. The object detection model can operate at runtime in a different range domain than that used for training. For example, it is possible to deploy an object detection model trained using close-range camera input at runtime to perform object detection on long-range camera data. In the example, regressing the offset (instead of directly regressing the range value) improves the range invariance.

[0012] Advantageously, the example object detection model according to this disclosure is capable of learning to implicitly (or explicitly) correct projection errors. Projection errors can be caused by, for example, sensor miscalibration, errors in the estimation of the vehicle's pose relative to the map, etc. By jointly processing fused sensor data and point data, the object detection model can refine the detected object location using the complete context of the sensor data, even in the presence of projection errors. For example, even when the projected point (e.g., lane markings, LiDAR echoes from the vehicle surface) may not be accurately aligned with the same point depicted in the sensor data (e.g., drawn lane markings in an image, a rear view of a vehicle in an image), the object detection model can (implicitly) learn to identify features associated with the point (e.g., the pattern of drawn lane markings, the shape and shading of the rear of the vehicle) and other contextual cues to adjust the predicted relationship with the projected point or marking to accommodate the error of the projected point itself. The perception system can also explicitly obtain an estimate of the projection error to help correct future object detection processing cycles.

[0013] Advantageously, the example object detection model according to this disclosure can provide improved localization of detected objects within a lane, even with coarse range estimation. For example, in some scenarios, accurate lane detection (e.g., identifying the lane associated with the detected object) can have a greater impact on planning decisions compared to precise range estimation. For instance, determining that a vehicle is stopped on the shoulder of a road can be valuable, even if the exact range of the vehicle's location has not yet been determined with high confidence. This allows the vehicle to have a longer range detection with greater certainty, thus giving it more time to plan and execute actions in response to the detected object.

[0014] Using image processing alone over long distances can involve high levels of range uncertainty. Given this level of uncertainty, it is challenging for conventional systems to accurately determine, for example, whether an object is located on a road shoulder or in a moving traffic lane. By directly fusing sensor data with long-range point data containing rich range data (e.g., LiDAR data, map data, lane contour data), the example perception system according to this disclosure can more easily associate detected objects with specific lanes of a road.

[0015] The technology disclosed herein offers numerous technical effects and benefits that improve the functionality of autonomous vehicles and their computing systems, and overall advance the field of autonomous driving technology. For example, the perception system according to this disclosure can achieve better detection outputs using limited sensor data, thereby increasing the vehicle's perception range for a given sensor capability configuration. Additionally, the perception system can compute object detection outputs more efficiently. For a given size of a machine learning object detection model, fusing point or map data and sensor data using geometric priors can free up model parameters for tasks that independently predict 3D positions. This allows the expressiveness of model parameters to be focused on a simplified task optimized in local regions of projected points. Furthermore, when fusing with point data, autonomous vehicles can use cheaper, more robust sensors (e.g., camera sensors compared to LiDAR sensors) to increase detection range, thereby reducing the overall cost of the vehicle, improving functionality, and ultimately accelerating the adoption of emerging technologies for autonomous vehicles.

[0016] In an example aspect, this disclosure provides an example computer-implemented method for object detection. In some embodiments, the example computer-implemented method includes (a) acquiring sensor data describing the environment of an autonomous vehicle. In some embodiments, the example computer-implemented method includes (b) acquiring a plurality of points describing the environment. In some embodiments, the example computer-implemented method includes (c) using a machine learning object detection model and determining, based on the sensor data, the association between one or more of the plurality of points and an object in the environment. In some embodiments, the example computer-implemented method includes (d) using a machine learning object detection model to generate offsets of one or more points relative to a spatial region of the environment associated with the object.

[0017] In an example aspect, this disclosure provides an example autonomous vehicle control system for controlling an autonomous vehicle. In some embodiments, the example autonomous vehicle control system includes one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform operations. These operations can include implementations of methods implemented by the example computer.

[0018] In an example aspect, this disclosure provides one or more example non-transitory computer-readable media storing instructions that are executable to cause one or more processors to perform operations. These operations can include implementations of methods implemented in the example computer.

[0019] Other exemplary aspects of this disclosure relate to other systems, methods, vehicles, apparatuses, tangible non-transitory computer-readable media, and devices for performing the functions described herein. These and other features, aspects, and advantages of the various embodiments will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0020] The specification provides a detailed description of embodiments that are suitable for those skilled in the art, with reference to the accompanying drawings, wherein: Figure 1 This is a block diagram of an example operation scenario based on some embodiments of this disclosure; Figure 2 This is a block diagram of an example system according to some embodiments of the present disclosure; Figure 3A This is a representation of an example operating environment according to some embodiments of this disclosure; Figure 3B This is a representation of an example map of the operating environment according to some embodiments of this disclosure; Figure 3C This is a representation of an example operating environment according to some embodiments of this disclosure; Figure 3D This is a representation of an example map of the operating environment according to some embodiments of this disclosure; Figure 4 This is a block diagram of an example system for object detection according to some embodiments of the present disclosure; Figure 5 This is a block diagram showing example inputs based on some embodiments of this disclosure; Figure 6 This is a block diagram of an example system for object detection according to some embodiments of the present disclosure; Figure 7 This is a block diagram illustrating an example of an off-target projection based on some embodiments of this disclosure; Figure 8 These are charts showing test results based on some embodiments of this disclosure; Figure 9 This is a flowchart of an example method for object detection according to some embodiments of this disclosure; Figure 10 This is a flowchart of an example method for object detection according to some embodiments of this disclosure; Figure 11 This is a flowchart illustrating an example method for training an operating system for machine learning for object detection, according to some embodiments of this disclosure; and Figure 12This is a block diagram of an example computing system for object detection according to some embodiments of the present disclosure. Detailed Implementation

[0021] The techniques disclosed herein are described below for illustrative purposes only within the context of autonomous vehicles. As stated herein, the techniques described are not limited to autonomous vehicles and can be implemented for or within other autonomous platforms and computing systems.

[0022] refer to Figures 1 to 12 The exemplary implementations of this disclosure are discussed in more detail. Figure 1 This is a block diagram of an example operating scenario according to some embodiments of the present disclosure. In the example operating scenario, environment 100 includes an autonomous platform 110 and multiple objects, including a first actor 120, a second actor 130, and a third actor 140. In the example operating scenario, the autonomous platform 110 is capable of moving through environment 100 and interacting with objects located within environment 100 (e.g., the first actor 120, the second actor 130, the third actor 140, etc.). The autonomous platform 110 can optionally be configured to communicate with a remote system 160 via network 170.

[0023] Environment 100 may be or include an indoor environment (e.g., within one or more facilities) or an outdoor environment. An indoor environment may be an environment enclosed by a structure such as a building (e.g., a service facility, maintenance location, manufacturing facility, etc.). An outdoor environment may be one or more areas in the external world, such as one or more rural areas (e.g., having one or more rural access roads, etc.), one or more urban areas (e.g., having one or more urban access roads, highways, etc.), one or more suburbs (e.g., having one or more suburban access roads, etc.), or other outdoor environments.

[0024] Autonomous platform 110 can be any type of platform configured to operate within environment 100. For example, autonomous platform 110 can be a vehicle configured to autonomously sense and operate within environment 100. The vehicle can be a ground-based autonomous vehicle, such as, for example, an autonomous car, truck, van, etc. Autonomous platform 110 can be an autonomous vehicle capable of controlling, being connected to, or otherwise associated with implements, attachments, and / or accessories for transporting people or goods. This can include, for example, an autonomous tractor optionally coupled to a cargo trailer. Additionally or alternatively, autonomous platform 110 can be any other type of vehicle, such as one or more aircraft, water-based vehicles, space-based vehicles, other ground-based vehicles, etc.

[0025] Autonomous platform 110 can be configured to communicate with remote system 160. For example, remote system 160 can communicate with autonomous platform 110 for assistance (e.g., navigation assistance, situational response assistance, etc.), control (e.g., fleet management, remote operation, etc.), maintenance (e.g., updates, monitoring, etc.), or other local or remote tasks. In some embodiments, remote system 160 can provide data instructing autonomous platform 110 to perform tasks. For example, as further described herein, remote system 160 can provide data instructing autonomous platform 110 to perform trips / services, such as user transportation trips / services, delivery trips / services (e.g., for goods, cargo, articles, etc.).

[0026] Autonomous platform 110 is capable of communicating with remote system 160 using network 170. Network 170 is capable of facilitating the transmission of signals (e.g., electronic signals, etc.) or data (e.g., data from computing devices, etc.) and can include any combination of various wired (e.g., twisted-pair cable, etc.) or wireless communication mechanisms (e.g., cellular, wireless, satellite, microwave, radio frequency, etc.) or any desired network topology (or multiple topologies). For example, network 170 can include local area networks (e.g., intranets, etc.), wide area networks (e.g., the Internet, etc.), wireless LAN networks (e.g., via Wi-Fi, etc.), cellular networks, SATCOM networks, VHF networks, HF networks, WiMAX-based networks, or any other suitable communication network (or combination thereof) used to transmit data to or from autonomous platform 110.

[0027] For example, Figure 1 As shown, environment 100 can include one or more objects. These objects can be objects that are not in motion or not predicted to be moving (“static objects”) or objects that are in motion or predicted to be in motion (“dynamic objects” or “actors”). In some implementations, environment 100 can include any number of actors, such as, for example, one or more pedestrians, animals, vehicles, etc. Actors can move within the environment according to one or more actor trajectories. For example, a first actor 120 can move along any one of first actor trajectories 122A-C, a second actor 130 can move along any one of second actor trajectories 132, a third actor 140 can move along any one of third actor trajectories 142, and so on.

[0028] As further described herein, autonomous platform 110 is capable of using its autonomous system to detect these actors (and their movement) and plan its motion to navigate through environment 100 based on one or more platform trajectories 112A-C. Autonomous platform 110 may include an onboard computing system 180. Onboard computing system 180 may include one or more processors and one or more memory devices. The one or more memory devices are capable of storing instructions executable by the one or more processors to cause the one or more processors to perform operations or functions associated with autonomous platform 110, including implementing its autonomous system.

[0029] Figure 2 This is a block diagram of an exemplary autonomous system 200 for an autonomous platform according to some embodiments of the present disclosure. In some embodiments, the autonomous system 200 can be implemented by a computing system of the autonomous platform (e.g., the onboard computing system 180 of the autonomous platform 110). The autonomous system 200 is operable to receive input from sensor 202 or other input devices. In some embodiments, the autonomous system 200 can additionally receive platform data 208 (e.g., map data 210) from local or remote storage devices. The autonomous system 200 can generate control outputs for (e.g., via platform control device 212, etc.) controlling the autonomous platform based on sensor data 204, map data 210, or other data. The autonomous system 200 may include different subsystems for performing various autonomous operations. Subsystems may include a positioning system 230, a sensing system 240, a planning system 250, and a control system 260. The positioning system 230 is capable of determining the location of the autonomous platform within its environment; the perception system 240 is capable of detecting, classifying, and tracking objects and actors in the environment; the planning system 250 is capable of determining the trajectory of the autonomous platform; and the control system 260 is capable of translating the trajectory into vehicle control for controlling the autonomous platform. The autonomous system 200 can be implemented by one or more onboard computing systems. Subsystems can include one or more processors and one or more memory devices. The one or more memory devices can store instructions executable by one or more processors to cause one or more processors to perform operations or functions associated with the subsystem. The computing resources of the autonomous system 200 can be shared among its subsystems, or a subsystem can have a dedicated set of computing resources.

[0030] In some implementations, the autonomous system 200 can be implemented for or by an autonomous vehicle (e.g., a ground-based autonomous vehicle). The autonomous system 200 can perform various processing techniques on inputs (e.g., sensor data 204, map data 210) to perceive and understand the vehicle's surroundings, and generate an appropriate set of control outputs to implement vehicle motion planning (e.g., including one or more trajectories) for traversing the vehicle's surroundings (e.g., [missing information]). Figure 1(Environment 100, etc.). In some embodiments, the autonomous vehicle implementing the autonomous system 200 can drive, navigate, operate, etc. with minimal or no interaction from human operators (e.g., drivers, pilots, etc.).

[0031] In some implementations, the autonomous platform can be configured to operate in multiple operating modes. For example, the autonomous platform can be configured to operate in a fully autonomous (e.g., self-driving, etc.) operating mode, where the autonomous platform is controllable without user input (e.g., capable of driving and navigating without input from a human operator in or away from the autonomous vehicle). The autonomous platform can operate in a semi-autonomous operating mode, where the autonomous platform can operate using some input from a human operator present in or away from the autonomous platform. In some implementations, the autonomous platform can enter a manual operating mode, where the autonomous platform is fully controllable by a human operator (e.g., a human driver, etc.) and can be prohibited or disabled (e.g., temporarily, permanently, etc.) from performing autonomous navigation (e.g., autonomous driving, etc.). The autonomous platform can be configured to operate in other modes, such as, for example, parking or sleep modes (e.g., for use between tasks such as waiting for a trip / service, recharging, etc.). In some implementations, the autonomous platform can implement vehicle operation assistance technologies (e.g., collision mitigation systems, power-assisted steering, etc.) to assist the human operator of the autonomous platform (e.g., when in manual mode, etc.).

[0032] The autonomous system 200 can be onboard (e.g., on or within an autonomous platform) and can be configured to operate the autonomous platform in various environments. The environment can be a real-world environment or a simulated environment. In some embodiments, one or more simulation computing devices can simulate one or more of the following: sensor 202, sensor data 204, communication interface 206, platform data 208, or platform control device 212 for simulating the operation of the autonomous system 200.

[0033] In some implementations, the autonomous system 200 is capable of communicating with one or more networks or other systems having a communication interface 206. The communication interface 206 may include features for communicating with one or more networks (e.g., Figure 1 The communication interface 206 may interface with any suitable component (such as network 170, etc.), including, for example, a transmitter, receiver, port, controller, antenna, or other suitable component that can help facilitate communication. In some embodiments, the communication interface 206 may include multiple components (such as antennas, transmitters, or receivers, etc.) that allow it to implement and utilize various communication technologies (such as multiple-input multiple-output (MIMO) technology, etc.).

[0034] In some implementations, the autonomous system 200 can communicate with one or more computing devices located remotely to the autonomous platform (e.g., remote system 160) via one or more networks (e.g., network 170) using the communication interface 206. For example, in some examples, one or more inputs, data, or functions of the autonomous system 200 can be supplemented or replaced by a remote system communicating via the communication interface 206. For example, in some implementations, map data 210 can be downloaded to a remote system via a network using the communication interface 206. In some examples, one or more of the positioning system 230, sensing system 240, planning system 250, or control system 260 can be updated, influenced, nudged, communicated, etc., by the remote system for assistance, maintenance, situational response override, management, etc.

[0035] Sensor 202 can be airborne on an autonomous platform. In some embodiments, sensor 202 can include one or more types of sensors. For example, one or more sensors can include image capture devices (e.g., visible spectrum cameras, infrared cameras, etc.). Additionally or alternatively, sensor 202 can include one or more depth capture devices. For example, sensor 202 can include one or more light detection and ranging (LIDAR) sensors or radio detection and ranging (RADAR) sensors. Sensor 202 can be configured to generate point data describing at least a portion of a 360-degree view of the surrounding environment. This point data can be point cloud data (e.g., 3D LiDAR point cloud data, RADAR point cloud data). In some embodiments, one or more sensors 202 for capturing depth information can be fixed to a rotating device so that sensor 202 rotates about an axis. Sensor 202 can rotate about the axis while capturing data in spaced fan-shaped groups of different portions of the 360-degree view describing the surrounding environment of the autonomous platform. In some embodiments, one or more sensors 202 for capturing depth information can be solid-state.

[0036] Sensor 202 can be configured to capture sensor data 204 that indicates or otherwise associates with at least a portion of the autonomous platform's environment. Sensor data 204 can include image data (e.g., 2D camera data, video data, etc.), RADAR data, LIDAR data (e.g., 3D point cloud data, etc.), audio data, or other types of data. In some embodiments, the autonomous system 200 may receive input from additional types of sensors, such as inertial measurement units (IMUs), altimeters, inclinometers, odometers, position or location devices (e.g., GPS, compasses), wheel encoders, or other types of sensors. In some embodiments, the autonomous system 200 can acquire sensor data 204 associated with a specific component or system of the autonomous platform. This sensor data 204 can indicate, for example, wheel speed, component temperature, steering angle, cargo or passenger status, etc. In some embodiments, the autonomous system 200 can acquire sensor data 204 associated with environmental conditions, such as environmental or weather conditions. In some embodiments, sensor data 204 can include multimodal sensor data. The multimodal sensor data can be acquired by at least two different types of sensors (e.g., sensor 202) and can indicate static objects or actors within the autonomous platform's environment. The multimodal sensor data can include at least two types of sensor data (e.g., camera and LiDAR data). In some embodiments, the autonomous platform can use sensor data 204 for sensors located remotely (e.g., not onboard) to the autonomous platform. This can include, for example, sensor data 204 captured by different autonomous platforms.

[0037] The autonomous system 200 can acquire map data 210 associated with the environment in which the autonomous platform was previously, is currently, or will be located. Map data 210 can provide information about the environment or geographic area. For example, map data 210 can provide information about the identity and location of different roads (e.g., carriageways), road segments (e.g., road sections), buildings, or other objects (e.g., lampposts, crosswalks, curbs, etc.); the location and orientation of boundaries or boundary markers (e.g., traffic lanes, parking lanes, turning lanes, bicycle lanes, other lanes, etc.); traffic control data (e.g., the location and instructions of signs, traffic lights, other traffic control equipment, etc.); obstacle information (e.g., temporary or permanent obstructions, etc.); event data (e.g., road closures / traffic rule changes due to parades, concerts, sporting events, etc.); nominal vehicle path data (e.g., indicating an ideal vehicle path such as along the center of a lane, etc.); or any other map data that assists the autonomous platform in understanding its surrounding environment and its relationship to that environment. In some embodiments, map data 210 can include high-resolution map information. Additionally or alternatively, map data 210 may include sparse map data (e.g., lane maps). In some embodiments, sensor data 204 may be fused with map data 210 or used to update map data 210 in real time.

[0038] Autonomous system 200 may include positioning system 230, which provides the autonomous platform with an understanding of its location and orientation in the environment. In some examples, positioning system 230 may support one or more other subsystems of autonomous system 200, such as by providing a unified local reference frame for performing operations such as sensing, planning, or control.

[0039] In some implementations, positioning system 230 is capable of determining the current location of the autonomous platform. The current location can include a global location (e.g., with respect to georeferenced anchors) or a relative location (e.g., with respect to objects in the environment). Positioning system 230 can typically include or interface with any device or circuitry for analyzing the location or location changes of the autonomous platform (e.g., an autonomous ground-based vehicle). For example, positioning system 230 can determine location using one or more of the following: inertial sensors (e.g., inertial measurement units, etc.), satellite positioning systems, radio receivers, networking devices (e.g., based on IP addresses, etc.), triangulation or proximity measurements of network access points or other network components (e.g., cell towers, Wi-Fi access points, etc.), or other suitable techniques. The location of the autonomous platform can be used by various subsystems of autonomous system 200 or (e.g., using communication interface 206) provided to a remote computing system.

[0040] In some implementations, the positioning system 230 is capable of registering the relative positions of elements in the autonomous platform's surrounding environment with recorded positions in map data 210. For example, the positioning system 230 can process sensor data 204 (e.g., LiDAR data, RADAR data, camera data, etc.) for alignment or otherwise registration to a map of the surrounding environment (e.g., from map data 210) to understand the autonomous platform within that environment. Therefore, in some implementations, the autonomous platform can identify its position within the surrounding environment (e.g., across six axes, etc.) based on a search on map data 210. In some implementations, given an initial position, the positioning system 230 can update the autonomous platform's position incrementally with realignment based on recorded or estimated deviations from the initial position. In some implementations, the position can be registered directly within map data 210.

[0041] In some implementations, map data 210 may include a large amount of data subdivided into geographic blocks, enabling the reconstruction of a desired area of ​​a map stored in map data 210 from one or more blocks. For example, autonomous system 200 may stitch together multiple tiles selected from map data 210 based on a location obtained by positioning system 230 (e.g., multiple tiles selected near that location).

[0042] In some implementations, the positioning system 230 is capable of determining the (e.g., relative or absolute) location of one or more attachments or accessories to the autonomous platform. For example, the autonomous platform may be associated with a cargo platform, and the positioning system 230 may provide the location of one or more points on the cargo platform. For example, the cargo platform may include trailers or other equipment towed or otherwise attached to or operated by the autonomous platform, and the positioning system 230 may provide data describing the (e.g., absolute, relative, etc.) location of the autonomous platform and the cargo platform. Such information can be obtained by other autonomous systems to assist in operating the autonomous platform.

[0043] Autonomous system 200 may include perception system 240, which enables the autonomous platform to detect, classify, and track objects and actors in its environment. The environmental features or objects perceived within the environment may be those within the field of view of sensor 202 or those predicted to be occluded by sensor 202. This may include objects that are not in motion or are not predicted to move (static objects) or objects that are in motion or are predicted to move (dynamic objects / actuators).

[0044] The perception system 240 is capable of determining one or more states (e.g., current or past states, etc.) of one or more objects within the surrounding environment of the autonomous platform. For example, a state can (e.g., for a given time, time period, etc.) describe the estimated or past location of the object (also referred to as position); current or past velocity / rate; current or past acceleration; current or past direction of travel; current or past orientation; size / occupied area (e.g., represented by boundary shape, object highlighting, etc.); classification (e.g., pedestrian category vs. vehicle category vs. bicycle category, etc.); associated uncertainties; or other state information. In some embodiments, the perception system 240 is capable of using one or more algorithms or machine learning models configured to identify / classify objects based on input from sensor 202 to determine the state. The perception system is capable of using different modalities of sensor data 204 to generate a representation of the environment to be processed by one or more algorithms or machine learning models. In some embodiments, as the autonomous platform continues to perceive or interact with objects, the state of one or more identified or unidentified objects can be maintained and updated over time (e.g., maneuvers such as avoidance, yielding, etc.). In this way, the perception system 240 is able to provide an understanding of the current state of the environment (e.g., including objects therein) as informed by records of the environment's previous states (e.g., including the movement history of objects therein). Such information can help the autonomous platform plan its movement through the environment.

[0045] Autonomous system 200 may include planning system 250, which can be configured to determine how the autonomous platform interacts with and moves within its environment. Planning system 250 can determine one or more motion plans for the autonomous platform. Motion plans may include one or more trajectories (e.g., motion trajectories) indicating the path the autonomous platform should follow. Trajectories may have a length or time range. The length or time range may be defined by the calculated planning range of planning system 250. Motion trajectories may be defined by one or more road points (with associated coordinates). Road points may be the future locations of the autonomous platform. Motion plans can be continuously generated, updated, and considered by planning system 250.

[0046] The motion planning system 250 is capable of determining a strategy for the autonomous platform. The strategy can be a discrete set of decisions made by the autonomous platform (e.g., yielding to an actor, yielding in the opposite direction to an actor, merging, lane changing). The strategy can be selected from multiple potential strategies. The selected strategy can be the lowest-cost strategy determined by one or more cost functions. The cost function can, for example, evaluate the probability of a collision with another actor or object.

[0047] Planning system 250 is capable of determining the desired trajectory for executing a strategy. For example, planning system 250 can obtain one or more trajectories for executing one or more strategies. Planning system 250 can evaluate trajectories or strategies and rank them (e.g., using scores, costs, rewards, constraints, etc.). For example, planning system 250 can inform the autonomous platform of the evaluation of candidate trajectories or strategies using forecast outputs indicating interactions between the autonomous platform's trajectory and one or more objects (e.g., proximity, intersection points, etc.). In some implementations, planning system 250 can utilize static costs to evaluate the autonomous platform's trajectory (e.g., "avoiding lane boundaries," "minimizing tremors," etc.). Additionally or alternatively, planning system 250 can utilize dynamic costs to evaluate the autonomous platform's trajectory or strategy based on predictions of the current operational scenario (e.g., predicted trajectories or strategies leading to interactions between actors, predicted trajectories or strategies leading to interactions between actors and the autonomous platform, etc.). Planning system 250 can rank trajectories based on one or more static costs, one or more dynamic costs, or a combination thereof. The planning system 250 can select a motion plan (and corresponding trajectory) based on the ranking of multiple candidate trajectories. In some implementations, the planning system 250 can select the highest-ranked candidate or the highest-ranked feasible candidate.

[0048] Then, the planning system 250 can verify the selected trajectory against one or more constraints before the trajectory is executed by the autonomous platform.

[0049] To aid its motion planning decisions, the planning system 250 can be configured to perform predictive functions. The planning system 250 can predict the future state of the environment. This can include predicting the future states of other actors in the environment. In some implementations, the planning system 250 can predict future states based on current or past states (e.g., as developed or maintained by the perception system 240). In some implementations, future states can be or include predicted trajectories (e.g., positions over time) of objects in the environment (such as other actors). In some implementations, one or more future states can include one or more probabilities associated with them (e.g., marginal probabilities, conditional probabilities). For example, one or more probabilities can include one or more probabilities conditioned on strategies or trajectory options available to the autonomous platform. Additionally or alternatively, probabilities can include probabilities conditioned on trajectory options available to one or more other actors.

[0050] In some implementations, the planning system 250 is capable of performing interactive forecasting. The planning system 250 can determine the motion plan for the autonomous platform by understanding how the predicted future state of the environment might be affected by the execution of one or more candidate motion plans. As an example, refer again... Figure 1Autonomous platform 110 is capable of determining candidate motion plans corresponding to a set of platform trajectories 112A-C, which correspond to the first actor trajector trajector 122A-C of first actor 120, trajectory 132 of second actor 130, and trajectory 142 of third actor 140, respectively (e.g., having corresponding trajectory correspondences indicated by matching line patterns). For example, autonomous platform 110 (e.g., using its autonomous system 200) can predict that platform trajectory 112A for moving autonomous platform 110 more quickly in front of first actor 120 may be associated with first actor 120 reducing its forward speed and more quickly avoiding autonomous platform 110 based on first actor trajectory 122A. Additionally or alternatively, autonomous platform 110 can predict that platform trajectory 112B for moving autonomous platform 110 gently in front of first actor 120 may be associated with first actor 120 slightly reducing its speed and slowly avoiding autonomous platform 110 based on first actor trajectory 122B. Additionally or alternatively, the autonomous platform 110 can predict any distance that the platform trajectory 112 C used to maintain parallel alignment with the first actor 120 might be associated with, according to the first actor trajectory 122 C, not avoiding the autonomous platform 110. Based on a comparison of the predicted scenario with a set of expected outcomes (e.g., through a cost- or reward-based scoring scenario), the planning system 250 can select a motion plan (and its associated trajectory) in light of the interaction between the autonomous platform and the environment 100. In this way, for example, the autonomous platform 110 can interweave its prediction and motion planning capabilities.

[0051] To implement the selected motion plan, the autonomous system 200 may include a control system 260 (e.g., a vehicle control system). Typically, the control system 260 provides an interface between the autonomous system 200 and the platform control device 212 for implementing the strategy and motion plan generated by the planning system 250. For example, the control system 260 can control the autonomous platform through its environment by following a selected trajectory (e.g., road points included therein). The control system 260 can, for example, translate the motion plan into instructions for the appropriate platform control device 212 (e.g., acceleration control, braking control, steering control, etc.). As an example, the control system 260 can translate the selected motion plan into instructions for adjusting steering components (e.g., steering angle) by a specific degree, applying a certain amount of braking force, increasing / decreasing speed, etc. In some embodiments, the control system 260 can communicate with the platform control device 212 via a communication channel, which includes, for example, one or more data buses (e.g., Controller Area Network (CAN), etc.), an on-board diagnostic connector (e.g., OBD-II), or a combination of wired or wireless communication links. The platform control device 212 can send or receive data, messages, signals, etc. to or from the autonomous system 200 via a communication channel (or vice versa).

[0052] The autonomous system 200 can receive auxiliary signals from the remote assistance system 270 via communication interface 206. The remote assistance system 270 can communicate with the autonomous system 200 via a network (e.g., as a remote system 160 on network 170). In some embodiments, the autonomous system 200 can initiate a communication session with the remote assistance system 270. For example, the autonomous system 200 can initiate a session based on or in response to a trigger. In some embodiments, the trigger may be an alarm, error signal, map feature, request, location, traffic conditions, road conditions, etc.

[0053] After initiating a session, the autonomous system 200 can provide contextual data to the remote assistance system 270. This contextual data may include sensor data 204 and status data of the autonomous platform. For example, the contextual data may include real-time camera feeds from the autonomous platform's cameras and the platform's current speed. The operator of the remote assistance system 270 (e.g., a human operator) can use the contextual data to select assistance signals. These assistance signals can provide values ​​or adjustments for various operating parameters or characteristics of the autonomous system 200. For example, assistance signals may include road points (e.g., path around obstacles, lane changes, etc.), speed or acceleration curves (e.g., speed limits, etc.), relative motion commands (e.g., platoon formation, etc.), operating characteristics (e.g., use of assistance systems, reduced energy handling modes, etc.), or other signals used to assist the autonomous system 200.

[0054] Autonomous system 200 can use auxiliary signals as input to one or more autonomous subsystems for performing autonomous functions. For example, planning subsystem 250 can receive auxiliary signals as input for generating motion plans. For example, auxiliary signals can include constraints for generating motion plans. Additionally or alternatively, auxiliary signals can include cost or reward adjustments for influencing the motion plans performed by planning subsystem 250. Additionally or alternatively, in addition to other received data (e.g., sensor inputs, etc.), autonomous system 200 can also consider auxiliary signals as advisory inputs for consideration.

[0055] The autonomous system 200 can be platform-agnostic, and the control system 260 can provide control commands to the platform control device 212 for various platforms used for autonomous mobility (e.g., multiple different autonomous platforms equipped with autonomous control systems). This can include various types of autonomous vehicles (e.g., cars, vans, SUVs, trucks, electric vehicles, internal combustion engine vehicles, etc.) from various manufacturers / developers operating in various environments, and in some embodiments, perform one or more vehicle services.

[0056] For example, refer to Figure 3A The operating environment can include a dense environment 300. The autonomous platform can include an autonomous vehicle 310 controlled by an autonomous system 200. In some embodiments, the autonomous vehicle 310 can be configured for mobility in dense environments, such as those with a configured wheelbase or other specifications. In some embodiments, the autonomous vehicle 310 can be configured for transporting goods or passengers. In some embodiments, the autonomous vehicle 310 can be configured to transport a large number of passengers (e.g., buses, shuttles, public buses, etc.). In some embodiments, the autonomous vehicle 310 can be configured to transport goods, such as large quantities of goods (e.g., trucks, vans, walk-in vans, etc.) or smaller goods (e.g., food, personal parcels, etc.).

[0057] refer to Figure 3BThe diagram illustrates a selected top view 302 of a dense environment 300, covering an example trip / service between a first location 304 and a second location 306. The example trip / service can be assigned to an autonomous vehicle 320, for example, by a remote computing system. The autonomous vehicle 320 can be, for example, a vehicle of the same type as autonomous vehicle 310. The example trip / service can include transporting passengers or goods between the first location 304 and the second location 306. In some embodiments, the example trip / service can include travel to or through one or more intermediate locations, such as for loading or unloading passengers or goods. In some embodiments, the example trip / service can be pre-scheduled (e.g., for regular traversals, such as in transportation scheduling). In some embodiments, the example trip / service can be on demand (e.g., requested by or used to perform taxi, ride-sharing, ride-hailing, courier, delivery services, etc.).

[0058] refer to Figure 3C In another example, the operating environment can include an open road environment 330. The autonomous platform can include an autonomous vehicle 350 controlled by the autonomous system 200. This can include an autonomous tractor for an autonomous truck. In some embodiments, the autonomous vehicle 350 can be configured for high-payload transportation (e.g., large-volume transport of goods or other cargo or passengers), such as long-distance, high-payload transportation. For example, the autonomous vehicle 350 can include one or more cargo platform attachments, such as trailers 352. Although depicted as a towed attachment in Figure 3, in some embodiments, one or more cargo platforms can be integrated into the autonomous vehicle 350 (e.g., attached to the chassis of the autonomous vehicle 350, etc.) (e.g., as in vans, walk-in vans, etc.).

[0059] refer to Figure 3DThe diagram shows a selected top view of an open road environment 330, including roads 332, intersections 334, transport hubs 336 and 338, access road 340, and locations 342 and 344. In some embodiments, autonomous vehicles (e.g., autonomous vehicle 310 or autonomous vehicle 350) can be assigned example trips / services to traverse one or more roads 332 (optionally connected via intersections 334) to transport goods between transport hubs 336 and 338. For example, in some embodiments, the example trips / services include goods delivery / transport services, such as goods delivery / transport services. The example trips / services can be assigned by a remote computing system. In some embodiments, transport hub 336 can be the origin of goods (e.g., a warehouse, storage facility, etc.), and transport hub 338 can be the destination of goods (e.g., a retailer, etc.). However, in some embodiments, transport hub 336 can be an intermediate point along the final journey of goods between its respective origin and its respective destination. For example, goods can be located at position 342 along access road 340. Therefore, goods can be transported (e.g., by human-driven vehicles, autonomous vehicles 310, etc.) to transport hub 336 for phased operations. At transport hub 336, various goods can be grouped or segmented for longer-distance transport along road 332.

[0060] In some implementations of the example trip / service, a set of phased goods / items can be loaded onto an autonomous vehicle (e.g., autonomous vehicle 350) for transport to one or more other transport hubs, such as transport hub 338. For example, although not depicted, it should be understood that the open road environment 330 can include more transport hubs than transport hubs 336 and 338, and can include more roads 332 interconnected by more intersections 334. A simplified map is presented here only for clarity. In some implementations, one or more goods / items transported to transport hub 338 can be (e.g., by human-driven vehicles, by autonomous vehicle 310, etc.) assigned to one or more local destinations, such as along access roads 340 to location 344. In some implementations, the example trip / service can be pre-scheduled (e.g., for regular traversal, such as in transport scheduling). In some implementations, the example trip / service can be on demand (e.g., as requested by or for the performance of a paid passenger transport or freight delivery service).

[0061] In order to improve the performance of autonomous platforms (such as autonomous vehicles controlled at least in part by autonomous system 200 (e.g., autonomous vehicle 310 or 350)), perception system 240 is capable of implementing detection techniques according to exemplary aspects of this disclosure.

[0062] Figure 4 This is a block diagram illustrating an example detection data stream. The perception system 240 can access sensor data 400. Sensor data 400 can include image data 401 describing the environment 402 and ranging data 406. The environment 402 can include a road with lanes 403 and 404 and shoulder areas 405. Map data 410 can include multiple layers or data types, such as a bird's-eye view lane boundary layer 411, a terrain layer 412, a graphics layer 413, or other layers. Map data 410 can include a high-resolution two-dimensional or three-dimensional geometric representation of at least a portion of the environment 402 in at least one layer.

[0063] In the example, perception system 240 can implement object detection model 420 to detect one or more objects in environment 402. Object detection model 420 can generate associations between one or more portions of sensor data 400 (e.g., image data 401, ranging data 406, both, etc.) or map data 410 and objects in the environment. Foreground data 430 can indicate the associations between one or more portions of sensor data 400 or map data 410 and objects in the environment.

[0064] For example, sensor data 400 can include an image of environment 402. Multiple point markers can be projected onto the image (e.g., from the viewpoint of the image onto a range view). The point markers can represent points from ranging data 406, map data 410, or both. The projection can be based on a known calibration between the corresponding image sensor and the source of the point data: for example, the position of the ranging sensor for points from ranging data 406 or the location of the autonomous vehicle in the mapped environment 402 for points from map data 410.

[0065] Foreground data 430, used for one or more portions of the image data, can indicate which of the projected point markers are associated with the portion of the image data representing the object. For example, foreground data 430 can indicate that point marker 431 (unfilled circle) is not associated with the object. Foreground data 430 can indicate that point markers 432 and 433 (filled circles) are associated with the object. Foreground data 430 can indicate that point marker 434 (filled circle) is associated with the object.

[0066] Object detection model 420 can generate spatial region data 440 based on foreground data 430. For example, object detection model 420 can generate bounding boxes or other detection indicators anchored to point markers associated with objects. For example, object detection model 420 can determine that markers 432 and 433 are associated with the same object and generate a bounding box 441 with centroid 442. Object detection model 420 can determine that marker 434 is associated with an object and generate a bounding box 443 with centroid 444. For example, object detection model 420 can regress the offset of the bounding box relative to the projected markers.

[0067] In this way, for example, perception system 240 can anchor bounding boxes to point data, thereby directly associating object detection with rich descriptive content in the point data. For example, detected objects can be directly registered to lanes of the road (e.g., locations in active driving lanes or shoulder areas) based on regression offsets to projected road markers from the road lanes. In another example, detected objects can be directly registered to a specific point in space based on regression offsets to projected LiDAR echo markers representing that specific point in the space.

[0068] This fused perception data can be used to quickly determine high-level information about the environment. For example, perception system 240 can determine the speed of a distant object and whether the object is in a specific lane. For example, perception system 240 can identify the lane in which the object is located. Additionally or alternatively, perception system 240 can determine whether a vehicle is a stationary vehicle (e.g., having a speed below a threshold). Determining which lane a vehicle is in at a distance can be useful, even if finer-grained information may not yet be available. For example, perception system 240 can determine whether a stationary vehicle is in the shoulder lane or in a moving traffic lane, allowing the autonomous vehicle to plan accordingly. By determining this information over a long period, the autonomous vehicle can have additional time for planning and executing appropriate actions.

[0069] Sensor data 400 may include sensor data 204 from sensor 202. Sensor data 400 may include multiple sensor modes. Sensor data 400 may include imaging data 401 (e.g., from an image sensor, such as a camera). Sensor data 400 may include point-based ranging data 406 (e.g., LiDAR data, RADAR data, stereo camera data, etc.).

[0070] Different image sensor configurations can capture sensor data 400. Imaging devices with varying fields of view can contribute data to sensor data 400. Sensor data 400 can include data from long-range cameras (e.g., cameras with telephoto lenses, cameras with sufficient resolution to resolve long-range details even with a wider field of view). Sensor data 400 can include data from short-range cameras (e.g., cameras with wide-angle lenses, low-resolution cameras that resolve sparse details over long distances). Sensor data 400 can include fused sensor data. Sensor data 400 can include upsampled image data. For example, machine learning image processing models can be used to recover details in the image data for noise reduction, deblurring, sharpening, upsampling resolution, etc. In this way, for example, the effective sensing range of the imaging device can be extended.

[0071] Sensor data 400 may include remote sensing data. The remote sensing data may include data describing the environment 402 beyond the range of the ranging sensor. For example, the remote sensing data may include data describing portions of the environment 402 beyond the detection range of a LIDAR unit, RADAR unit, or stereo sensor. For example, the detection range of a LIDAR or RADAR unit may be a range where the confidence level or uncertainty measure exceeds a threshold.

[0072] Map data 410 can include data describing environment 402. Map data 410 can be registered to sensor data 400 by positioning system 230. Positioning system 230 can process sensor data 400 or sensor data 204 to determine the position and orientation of the autonomous vehicle within environment 402, and to determine the spatial relationship between the vehicle and the map-based representation of environment 402 in map data 410.

[0073] For example, map data 410 can include data representing one or more lanes of a road. Map data 410 can represent road lanes using, for example, vector-based curve representations (e.g., with or without road points, including line segments, splines, etc.). Labeling can be obtained by sampling a continuous representation of the road contour to obtain label data at a desired resolution. For example, map layer 411 can include traveling road data. Traveling road data can include data indicating the path of the traveling road. Traveling road data can include any other representation of lane boundaries, lane centerlines, or lane paths. Traveling road data can include a continuous representation of the traveling road contour that can be sampled at any resolution.

[0074] Although various example implementations of map data 410 are described herein, it should be understood that other 3D data can be used in a similar manner (e.g., in addition to or instead of map data). For example, LiDAR data can be used with map data 410 to fuse image data as described herein. For example, LiDAR data can be passed to object detection model 420 in another input channel.

[0075] For example, it is possible to process combined image point data complexes using one or more color channels and at least one point data channel (e.g., RGB-D scheme). The depth data channel "D" can contain rasterized point data. The depth data channel "D" can contain rasterized map data (e.g., projected map data). The depth data channel "D" can contain rasterized ranging data 406.

[0076] It should also be understood that various techniques can be combined at different range scales. For example, within the LiDAR range, LiDAR-based sensor fusion detection can be given greater weight. Outside the LiDAR range, map-based sensor fusion detection can be given greater weight. The transitions in between can be smooth transitions in detection weights (e.g., interpolation) to facilitate switching from one dominant mode to another.

[0077] Sensor data 400 can also depict the road travel described in the road travel data of map data 410. Locating the vehicle within map data 410 establishes a relationship between map data 410 and sensor data 400, enabling registration between the road travel depiction in sensor data 400 and the road travel data in map data 410. For example, the relationship can include kinematic relationships between one or more sensors and the vehicle, the vehicle's direction of travel within the mapped environment, and the sensor's field of view or orientation relative to the vehicle. This relationship can be based on calibration data that can be updated and refined over time to account for alignment offsets.

[0078] In the example, multiple point markers can be projected onto at least a portion of the sensor data 400. For example, the point data can be projected into the camera coordinate system of the camera capturing image data 401. The projection can be based on a camera transformation or a projection matrix. For example, the camera sensor can be calibrated and fixed to the vehicle. (Vehicle frame) The projection of a point can be obtained by the projection matrix. Define , where K is the camera calibration matrix, and , This involves rotation and transformation from the vehicle coordinate system to the camera coordinate system. For projected ranging data 406, the calibration matrix between the ranging sensor and the vehicle frame can be used to project the ranging data point markers into the camera coordinate system. For projected map data 410, once the vehicle frame and map frame are aligned via positioning, the points in the map frame can be projected into the camera coordinate system.

[0079] For example, Figure 5 An input data set 500, comprising image 501 and lane marking projection 502, is described. Lane marking projection 502 can contain road markings that have been projected into the camera space associated with image 501. In this way, for example, pixels of image 501 can be associated with road markings. Lane marking projection 502 can be stored in the input channel associated with image 501.

[0080] In a similar manner, other map data or point data can be projected onto the coordinate frame associated with the sensor. For example, map data 410 can include a high-resolution ground map (e.g., terrain layer 412). Projection markers can include points indicating the ground surface.

[0081] Refer again Figure 4 The object detection model 420 can process sensor data 400 or map data 410 to generate spatial region data 440. The object detection model 420 can include one or more machine learning models. The object detection model 420 can include models configured to process sensor data (single-modal sensor data, multi-modal sensor data, fused sensor data, aggregated sensor data, etc.). The object detection model 420 can include neural networks, such as deep neural networks. The object detection model 420 can use attention mechanisms (e.g., self-attention, such as in a transformer model architecture). The object detection model 420 can include convolutional layers configured to generate spatial feature maps based on the input. For example, an example object detection model can include a ResNet architecture.

[0082] Object detection model 420 can obtain foreground data 430 to guide the generation of spatial region data 440. For example, foreground data 430 can include data indicating the probability of an object's presence at a specific location. Foreground data 430 can include binary flags indicating whether an object exists at a specific location. Foreground data 430 can include a multi-channel data structure, each channel indicating the presence of an object associated with the category of that channel. For example, channels of the data structure can be associated with the vehicle category. Data values ​​in that layer can indicate the presence of a vehicle at a location associated with the index position of the data value (e.g., corresponding to pixels indexed at the same position on different layers). Other categories can correspond to other layers.

[0083] Foreground data 430 can indicate the probability of the presence of an object in a portion of the image associated with projected map data 410. For example, foreground data 430 can indicate a low probability of containing a foreground object in a region associated with projection marker 431 (e.g., one or more pixels falling under the projection marker). Foreground data 430 can indicate a higher probability of containing a foreground object in a region associated with projection markers 432, 433, or 434 (e.g., one or more pixels falling under the projection marker) (e.g., a value of 1, or a value rounded up, rounded, or otherwise assigned to a specified value).

[0084] Therefore, foreground data 430 can provide an indication of the association between one or more point markers and objects in the environment (e.g., vehicles in the foreground). Various metrics can be used to determine the association between one or more point markers (e.g., from ranging data 406, map data 410, etc.) and objects in the environment. Example metrics include distance metrics, such as defining the radius of the region surrounding a marker, within which an object detected will be associated with that marker. Example distance metrics can be range-adaptive, allowing the metric to relax at longer distances to improve the recall of the detection model.

[0085] Foreground data 430 can act as a mask or filter applied to other model layers, enabling object detection model 420 to generate spatial region data 440 based on foreground markers in foreground data 430. For example, one or more components of object detection model 420 can "excite" or sparsely process foreground points of activity so that object detection model 420 regresses spatial region data 440 with respect to those foreground points of activity.

[0086] Spatial region data 440 can include boundary regions regressed from foreground markers. Spatial region data 440 can be regressed relative to foreground markers in sensor coordinate space (e.g., in an image frame). In this way, rich contextual information in, for example, ranging data 406 or map data 410 can be directly indexed using foreground markers and the corresponding spatial region data 440.

[0087] For example, the centroid 442 of bounding box 441 can be located at a distance away from point markers 432 / 433. Object detection model 420 can regress the distance or offset between centroid 442 and point markers 432, 433. Object detection model 420 can process image data based on foreground markers to output the offset.

[0088] Figure 6An example architecture of one or more of the object detection models 420 is illustrated. The backbone model 610 is capable of processing input data 500. For example, the backbone model 610 is capable of processing sensor data (e.g., image data) and point data (e.g., depth values ​​from projected ranging data 406 or projected map data 410) together. In this way, for example, the backbone model 610 is capable of generating feature maps that encode fused information across channels of the input. The backbone model 610 is capable of inference over the entire input image and the entire set of projected labels. The backbone model 610 is capable of implicitly associating projected labels with identifiable features of the input image.

[0089] The input to the backbone model 610 can be z-buffered to mask points that are occluded during transformation / projection into a camera frame. The input to the backbone model 610 can include full image data or downsampled versions thereof. For example, the input to the backbone model 610 can include image data that has not yet been cropped or otherwise masked. In this way, for example, even if the point data is sparse, the input to the backbone model 610 can preserve the scene's image data throughout the fusion process.

[0090] The backbone model 610 can include a stem network to perform initial preprocessing and feature extraction on features from the image data before processing the fused image and point data in the downstream feature extraction network of the backbone model 610. An example backbone network includes one or more convolutional layers. The backbone network can include a convolution stride length or stride that is configured to reduce the resolution of the image data before fusion with the point data layer. The point data layer can also be downsampled for matching.

[0091] The output of the backbone model 610 can be passed to task-specific output headers, such as a category header 612, a 2D header 614, and a 3D header 616. The category header 612 can process the feature maps generated by the backbone model 610 to determine foreground data. For example, the category header 612 can be configured to determine the presence of objects in one or more categories (e.g., vehicles, pedestrians, etc.). This object data can be masked using projected map data 410 to obtain foreground data 430 indicating one or more foreground markers.

[0092] The two-dimensional head 614 can process the feature maps generated by the backbone model 610 to generate two-dimensional boundary features 624, which can be used to obtain bounding boxes in sensor coordinate frames. The three-dimensional head 616 can process the feature maps generated by the backbone model 610 to generate three-dimensional boundary features 626 that can be used to obtain three-dimensional bounding boxes.

[0093] Any one or more of the task-specific output headers can include machine learning model components. Any one or more of the task-specific output headers can include at least one layer of a neural network (e.g., a convolutional layer, such as a 1x1 convolutional layer). The task-specific output headers can process feature maps from various parts of the backbone model 610. For example, the backbone model 610 can include layers at various resolutions and depths. The task-specific headers can process one or more layers from one or more different resolutions and depths.

[0094] It is possible to train all or some components of the object detection model 420 using supervision in the same coordinate space. It is possible to train all or some components of the object detection model 420 using supervision in the image space (e.g., loss computed only in image coordinate frames).

[0095] In the example, supervision of the predictions from the output header can be performed sparsely, so that the loss is applied only to the pixels corresponding to the projected point data.

[0096] An example configuration of the loss can be expressed as follows. Consider... A set of points (e.g., distance measurement data points, map data points, etc.). and A collection of 2D and 3D tags for a link. (For example, a perceptual log of labeled objects in an environment containing 2D images and 3D points). For The given object in as well as The middle belongs to the object point The 2D bounding box parameters can be expressed as follows. The displacement in pixels from the position of the projection point in the image to the center of the object's 2D bounding box is determined by... and Given:

[0097] in, Represents the center coordinates of the 2D bounding box of object k, and It is the projection point The pixel coordinates. For compact expressions, use symbols. The value obtained through projection transformation is used to represent the value. The width of the 2D bounding box can be expressed as... And height is expressed as .

[0098] For objects Each point associated with the 3D label offset and Able to represent the pixel distance from the point's position to the projected centroid of the 3D bounding box:

[0099] in, Represents the pixel coordinates of the centroid of the projected 3D bounding box. From point The distance to the 3D centroid of object k can be expressed as a dot product.

[0100] in, Indicates pointing the camera ray at the object. The unit vector of the centroid, and From point To object The 3D displacement vector of the centroid.

[0101] The direction of the object's movement The orientation relative to the object can be parameterized as , ,in

[0102] in, It is the forward direction in the camera frame, and It is an object The location.

[0103] The extent of an object can be parameterized as its width. ,length and height .

[0104] Example losses used to train all or some components of the object detection model 420 can be expressed using the above expression as follows. Focus loss can be used to supervise the class probability output (e.g., from the class head):

[0105] Where N represents the number of pixels associated with the valid point projection (e.g., projections with values, projections corresponding to sentinel validity flags, etc.). Let represent the predicted probability of the true class at pixel i, and and It is the hyperparameter of focal length loss.

[0106] To train the network to handle the uncertainty in the regressed location and degree parameters of the 2D / 3D bounding boxes, the network is able to predict both the mean and diversity of the Laplace distribution for each parameter. Supervision of these predictions can involve minimizing the negative log-likelihood of the Laplace distribution, resulting in the following loss function for the 2D / 3D center and degree:

[0107] in, This represents the model's prediction. The loss for 3D location and magnitude parameters can be calculated in a similar manner, and it has the advantage of using... Directed estimation of loss supervision:

[0108] The total loss can be expressed as .

[0109] Refer again Figure 4 The object detection model 420 can perform refinement techniques to obtain high-quality spatial region data 440. For example, the object detection model 420 can perform non-maximum suppression on the predicted spatial region data to determine possible bounding boxes. A threshold of 0.5 IoU can be used.

[0110] The example output decoding process can include sampling a classification heatmap output by the category prediction head. Point markers associated with the predicted object location (sampled from the heatmap, such as with a confidence threshold, cluster centroid, etc.) can form foreground data 430. For markers indicated as foreground, 2D detection head 614 can decode 2D targets to obtain 2D bounding boxes in the image frame. For example, 2D detection head 614 can receive feature maps from backbone model 610 and foreground data 430 as input. Based on those inputs, 2D detection head 614 can regress 2D spatial region data relative to foreground markers. One round of nonmaximum suppression can be applied to the 2D regression of spatial region data. 3D detection head 616 can receive any one or more of the following as input: feature maps from backbone model 610, foreground data 430, and decoded 2D spatial region data. Based on those inputs, 3D detection head 616 can regress 3D spatial region data relative to foreground markers. One round of nonmaximum suppression can be applied to the 3D regression of spatial region data.

[0111] One benefit of the technique described in this paper is that the object detection model 420 can access all sensor contexts around the projection point throughout the regression process, while the resulting bounding boxes are directly anchored to and within the high-resolution map data. This enables learned pose correction. For example, the object detection model 420 can regress the offset from map markers to spatial region data 440. In some scenarios, if the projected map data markers contain projection errors, the offsets may be skewed (e.g., more than they should be). For example, if the projected lane centerline is misaligned with the lane centerline in the image, vehicles in the imaged lane can be detected at the imaging location such that the predicted offset includes the distance from the centroid to the lane marker plus the misalignment distance.

[0112] Advantageously, the object detection model 420 is able to automatically compensate for misalignment. Figure 7 The illustration shows an example of misalignment between projected road markings and imaged lanes. Because spatial region data can be regressed from sensor data 400, detected boundaries can be correctly identified in sensor data 400 despite potential misalignment. Furthermore, despite the potential misalignment, registration between sensor data 400 and map data 410 can continue to provide improved coarse-grained detection information over a long range. For example, even when projected lane markings are misaligned, object detection model 420 can learn to register detections with the correct lanes of the road. Similarly, object detection model 420 can compensate for misaligned LiDAR or other ranging data.

[0113] The perception system 240 can explicitly regress values ​​representing misalignment. For example, components of the object detection model 420 can learn to regress transformation or rotation errors or other projection errors in the map data projection. This error can inform future iterations of the perception system 240 to recalibrate the detection. For example, the perception system 240 can estimate and correct projection errors in real time.

[0114] For example, the relative pose between sensor coordinate frames and point data coordinate frames can be adjusted to reduce offset values. For instance, taking the median (or mean, or other statistical measure) of offsets across a given set of detections in a scene can provide a measure of goodness of projection quality. Projection errors can be reduced by adjusting the relative pose (e.g., one or more in a projection transformation, such as a camera matrix), thereby reducing the statistical measure of offset. Statistical measures can be computed over multiple rounds of detection (e.g., moving averages calculated over time) to smooth calibration signals.

[0115] The perception system 240 is capable of performing projection error estimation using a dedicated output head. The perception system 240 is also capable of performing projection error estimation using a separate neural network trained to regress the projection error based on the output of the object detection model 420. In some implementations, the head or network can be trained by adding noise to a real-world dataset, and the head or network can be trained to regress the added noise or regression transformation parameters or adjustments thereof.

[0116] The perception system 240 can also use other input signals to predict the projection error. Examples of other input signals could include sensor data indicating the state of one or more vehicle components. For example, by processing sensor data indicating suspension motion, the perception system 240 can determine that the projection error is related to suspension motion. For example, in some cases, due to the flexibility of sensor mounting configuration, sensor attitude calibration can be subjected to extreme vibrations or inertial loads. In this way, for example, the perception system 240 can use other input signals to predict the projection error value. This predicted projection error value can be used to calibrate and tune the projection transformation to the sensor coordinate system.

[0117] The example implementations of the machine learning object detection model disclosed herein offer numerous performance benefits, including improvements in both accuracy and latency performance relative to prior art. To illustrate these performance benefits and advantages obtainable using the techniques described herein, example tests are performed using a point anchoring detection system according to this disclosure, where LIDAR points are used as point data. Tests are conducted using an example configuration referred to herein as the test example.

[0118] For the Test Example, camera and LiDAR data are fused using the RGB-Depth method by projecting the LiDAR point cloud onto an image at half resolution (160x790 for 2MP images and 320x1580 for 8MP images) to form a sparse depth raster. The Test Example uses a z-buffer to mask points that become occluded when transforming from LiDAR to a camera frame (or from dual echoes). In practice, for the Test Example, the resolution of the image onto which the LiDAR is projected is high enough that only a small fraction of points (<1%) are ultimately suppressed by the z-buffer, especially when running on an 8MP image. For the Test Example, the sparse depth tensor contains two channels: the Euclidean distance from the camera frame to the points, and a binary sentinel channel indicating which pixels contain valid LiDAR echoes. For the Test Example, this is then resized and concatenated to the RGB channels of the image tensor, resulting in a five-channel input tensor. For the Test Example, the LiDAR data is fused and injected into the network at various stages, including just before the final decoding head. For the test example, the depth raster is then resized to the appropriate resolution using nearest-neighbor sampling cascaded with images or feature channels of various resolutions.

[0119] For the test examples, the RGB-D input tensor at the original image resolution (2MP or 8MP) is first fed into a backbone network consisting of two fully convolutional layers with 32 and 64 dimensions, respectively, with kernel sizes of 7×7 and 3×3. For the test examples, the first layer has a stride of 2, thus reducing the feature resolution to half of the original resolution.

[0120] For the test example, the output of the backbone is then concatenated with a depth raster at half resolution and fed into a three-stage VoVNetV2 feature extractor. For the test example, the first three stages each apply 2x downsampling, while the last three stages resample the feature map back to half resolution. For the test example, in each of the last three upsampling stages, the depth raster is resized and concatenated with the feature map before being fed into the next upsampling stage. For the test example, the depth raster is finally concatenated with the output feature map of the last stage of the VoVNetV2 relay and fed into the decoding head. For the test example, each decoding head consists of a 1×1 convolutional layer, the output activations and dimensions of which are detailed below.

[0121] For the test example, the network's feature extraction stage produces an output with half the resolution H / 2×W / 2 of the original input. This output is then concatenated with a depth raster and fed into three distinct decoding branches: a class head, a 2D bounding box head, and a 3D bounding box head. For the test example, the network is trained to predict class probabilities for each pixel in the output grid and parameterize bounding boxes in both 2D image frames and 3D camera frames. For the test example, supervision of these predictions is sparse, applying a loss only to pixels corresponding to valid LiDAR point projections. This is achieved by masking the loss function in the H / 2×W / 2 output grid, thereby leveraging sentinel channels from the depth raster at the corresponding resolution to identify relevant pixels.

[0122] To decode the network output of the test example, the foreground LiDAR points are first found by querying the LiDAR raster using the heatmap output of the category header. For the test example, given a set of LiDAR points classified as foreground... Their corresponding 2D bounding boxes are decoded and processed in a 2D non-maximum suppression step with an IoU threshold of 0.5, resulting in a reduced set. For the test example, the 3D bounding boxes of this reduced set are then decoded and fed into a final bird's-eye view NMS step with an IoU threshold of 0.2, resulting in the output point L2d / 3d NMS f and the final set of its associated 2D and 3D bounding boxes.

[0123] The above provides detailed expressions for the loss and training techniques used in the test examples.

[0124] Example test results are provided in Tables 1 through 5. The test dataset contains 43,500 five-second clips at 10Hz for training, with image, lidar, and pose data, and 4,000 clips for validation. Image data comes from a long-range camera with a 30-degree FOV at 8MP resolution. All lidar data used in these tests originated from lidar systems with a range greater than 400m.

[0125] For the following instrumentation comparisons, all methods were evaluated within an ROI defined by a 30-degree FOV (from 100m to 500m) of a forward-pointing long-range camera. A minimum evaluation range of 100m was used. For the following instrumentation comparisons, all methods were trained using a single image and 100ms-valued LiDAR data as input, where the average point measurement time is centered on the image timestamp. The test example model was trained using the Adam optimizer for 450,000 iterations. The Adam optimizer has… The initial learning rate and the exponential decay of 0.9 per 4000 iterations.

[0126] During training, image data is downsampled to 2MP, but used in testing at either 2MP or 8MP. Therefore, when running on 8MP images, the depth values ​​at a given spatial scale and the density of the projected LIDAR points per pixel region are no longer consistent with how the model was trained on. Example approaches to this include: (1) applying point-by-point dropout with a 50% probability when training on 2MP; (2) removing point-by-point dropout when testing on 8MP images, thus keeping the projected point density constant across image resolutions; (3) rescaling the LIDAR data range values ​​by 0.5, resulting in depth maps with similar density and depth information ranges per unit area at a given scale between 2MP and 8MP images; and (4) restoring the range rescaling for each detection before returning the values.

[0127] Table 1. Inference latency comparison on NVIDIA A10G GPU

[0128] Table 2. Comparison of VRU and vehicle detection performance of different models at a maximum distance of 500 meters when running on an ROI defined by the camera's FOV.

[0129] Table 3. Performance evaluation of the 2.5D vehicle detection model at various resolutions and distances, where the 2.5D metric is defined as a 0.5 IoU match within the 2D bounding box with a maximum range error of 10%.

[0130] Table 4. Comparison of VRU bev AP @ 0.1 in different test example configurations. 2D pseudo-labels obtained by projecting 3D real-world labels 2 MP during training; 8 MP during inference using rescaled LiDAR Sparse VRU labels in these ranges

[0131] Table 5. Comparison of bev AP @ 0.1 for vehicles in different test configurations. 2D pseudo-labels obtained by projecting 3D real-world labels 2 MP during training; 8 MP during inference using rescaled LiDAR Figure 9 This is a flowchart of a method 900 for performing object detection according to various aspects of this disclosure. One or more portions of method 900 can be implemented by a computing system including one or more computing devices, such as the computing systems described with reference to other figures (e.g., autonomous platform 110, vehicle computing system 180, remote system 160, etc.). Figure 12 (Systems, etc.). Each corresponding part of method 900 can be executed by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of method 900 can be implemented on the hardware components of the devices described herein (e.g., as in...). Figure 1 , Figure 2 , Figure 12 (Waiting)

[0132] Figure 9 Elements are depicted in a specific order for illustrative and discussion purposes. Using the disclosure provided herein, those skilled in the art will understand that elements of any method discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Reference is made to elements / terms used for illustrative purposes in describing other systems and figures. Figure 9 ,and Figure 9 This is not intended to be restrictive. One or more parts of method 900 can be additionally or alternatively executed by other systems.

[0133] At 902, example method 900 can include obtaining sensor data describing the environment of the autonomous vehicle. For example, the sensor data can include sensor data 204, sensor data 400, etc.

[0134] At 904, example method 900 may include obtaining multiple points describing the environment (e.g., point markers from ranging data, road markers from map data). Point markers from ranging data may include LiDAR data, RADAR data, etc. Road markers may include lane markings (e.g., centerline markings, lane boundary markings, etc.). In some implementations, example method 900 may include sampling discrete road markers at 904 from continuous road map data (e.g., vector-based map data format).

[0135] At 906, example method 900 may include using a machine learning object detection model and determining the association between one or more points of a plurality of points and objects in the environment based on sensor data. In some implementations, at 906, example method 900 may include inputting point markers and image data into the machine learning object detection model, and obtaining object data from the machine learning object detection model at the projection locations of the point markers in a reference frame of the image data. For example, the object data may indicate that an object may be present at the projection locations of one or more point markers.

[0136] In some implementations, at 906, example method 900 may include inputting traveling road markings and sensor data into a machine learning object detection model, and obtaining object data from the machine learning object detection model at the projection location of the traveling road markings in a reference frame of the sensor data. For example, the object data may indicate that an object may be present at the projection location of one or more traveling road markings.

[0137] In the example, object data can include foreground data 430 (e.g., object data).

[0138] In some implementations, at 906, example method 900 is capable of subsampling a detection map generated by a machine learning object detection model based on point markers. For example, object detection model 420 is capable of generating a detection map of object data (e.g., using a task-specific output header, such as category header 612), indicating where objects in the input frame might be located. In some implementations of example method 900, one or more portions of the machine learning object detection model are configured to sparsely activate the output layer based on location in sensor data corresponding to the projection location. For example, object detection model 420 is capable of processing foreground points (e.g., using different output headers) to regress spatial region data relative to those foreground points.

[0139] At 908, example method 900 can include using a machine learning object detection model to generate offsets of one or more points relative to a spatial region of the environment associated with the object. In some implementations, example method 900 at 908 can include determining the offset of the centroid of the boundary of the spatial region and determining one or more dimensions of the boundary. For example, object detection model 420 can regress the offset of the centroid of the boundary around the object for each of one or more projection points associated with the object. In some implementations, at 908, example method 900 can include determining a first offset of the centroid of a first boundary of a two-dimensional spatial region (e.g., using a first portion of the machine learning object detection model, such as a first regression head) and a second offset of the centroid of a second boundary of a three-dimensional spatial region (e.g., using a second portion of the machine learning object detection model, such as a second regression head). For example, one or more first offsets can be determined in a sensor coordinate frame (e.g., in a frame aligned with the width and height of the image). The second offset can be determined for one dimension within the frame (e.g., in the depth dimension).

[0140] In some implementations, example method 900 may include: outputting an indicator that the object is a static object based on determining that the object's speed is below a threshold. In some implementations, example method 900 may include outputting the characteristic to a motion planning system for the autonomous vehicle. For example, the motion planning system may plan the autonomous vehicle's motion based on the understanding that the detected object is a static object (e.g., a vehicle parked on the shoulder, such as a parked emergency vehicle).

[0141] In some implementations, example method 900 may include outputting the characteristic that the object is a static object (e.g., on the shoulder of a road) based on determining that the object's speed is below a threshold and that a road in the object's vicinity is located. In some implementations, example method 900 may include outputting the characteristic to the motion planning system of the autonomous vehicle.

[0142] In some implementations of Example Method 900, the spatial region of the environment is outside the effective range of the autonomous vehicle's LiDAR sensors. For example, the object detection model is capable of outputting object detection with spatial region data anchored to 3D map data, without relying on real-time LiDAR scans that reliably provide echoes on the objects.

[0143] In some implementations, example method 900 may include identifying the lane in which an object is located. For example, object detection model 420 may regress an offset based on projected road markings. Map data may associate road markings with specific lanes or lane types. Example method 900 may include identifying lanes based on this association.

[0144] In some embodiments of Example Method 900, a machine learning object detection model is trained using training sensor data (e.g., image data) with a training field of view and training points with a training resolution. For example, the training sensor data can be characterized by a first camera configuration (e.g., having a first field of view, a first resolution, etc.). In some embodiments of Example Method 900, the sensor data (e.g., at runtime) is associated with a runtime field of view. The runtime field of view can be the same as or different from the training field of view. Therefore, point markers can be obtained at a runtime resolution selected based on a comparison of the training field of view and the runtime field of view.

[0145] In this way, for example, the range invariance of the exemplary embodiments of this disclosure enables transitive learning. Transitive learning can include training on one sensor configuration and running at inference time using different sensor configurations.

[0146] The distribution of normalized point data relative to the sensor resolution can facilitate transfer learning. For example, matching the distribution of point markers can often help improve transfer learning for objects of similar size across different configurations. For instance, a first camera configuration can represent a given object with a first number of pixels. Map data or ranging data can be sampled at a first resolution such that the first number of points fall on the object. A second camera configuration can represent the same object with a second number of pixels. Therefore, point data can be sampled at a second resolution such that the second number of points fall on the object. For example, a second resolution can be selected such that the second number matches the first number. Matching the distribution of points allows object detection models to operate on different sensor configurations. One way to determine the scaling factor for the extent of points is to determine the ratio of the number of pixels per unit height at a given distance (e.g., the ratio can provide a scaling factor).

[0147] In some implementations, example method 900 may include determining the projection error or attitude error of the projected point. This can be used to recalibrate the projection operation. For example, example method 900 may include projecting the point onto a reference frame of sensor data using a projection transformation. In some implementations, example method 900 may include determining one or more offsets of a spatial region relative to the point. In some implementations, example method 900 may include determining the projection error of the projected point based on the determined one or more offsets. In some implementations, example method 900 may include recalibrating the projection transformation based on the determined projection error.

[0148] In some implementations of Example Method 900, Example Method 900 includes obtaining ground truth points or 3D labels indicating the ground truth association between an object and one or more points, and determining a sparse loss for a machine-learned object detection model based on a comparison of object data and ground truth point labels. For example, the sparse loss can be computed by ignoring portions of sensor data not associated with points (e.g., projected LiDAR points, projected road markings). In some implementations of Example Method 900, Example Method 900 includes training a machine learning object detection model based on the sparse loss.

[0149] Figure 10 More details are provided for obtaining real-world training data. Figure 10 This is a flowchart of a method 1000 for generating real-world training data for training an object detection model, according to various aspects of this disclosure. One or more parts of method 1000 can be implemented by a computing system including one or more computing devices, such as the computing systems described with reference to other figures (e.g., autonomous platform 110, vehicle computing system 180, remote system 160, etc.). Figure 12 (Systems, etc.). Each corresponding part of method 1000 can be executed by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of method 1000 can be implemented on the hardware components of the devices described herein (e.g., such as...). Figure 1 , Figure 2 , Figure 12 wait).

[0150] Figure 10 Elements executed in a specific order are depicted for illustrative and discussion purposes. Using the disclosure provided herein, those skilled in the art will understand that elements of any method discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Reference is made to elements / terms described with respect to other systems and figures for illustrative purposes. Figure 10 ,and Figure 10 This is not intended to be restrictive. One or more parts of method 1000 can be additionally or alternatively executed by other systems.

[0151] At 1002, example method 1000 can include obtaining real-world or labeled sensor data describing the environment, including spatial region data defining spatial regions of sensor data associated with detected objects. For example, labeled sensor data can include labeled image captures. Labeled image captures can include frames from video recordings. Labeled sensor data can include sensor data that has been automatically or manually reviewed and annotated with one or more labels. Labeled sensor data can be obtained from log data from real or simulated driving sessions.

[0152] At 1004, example method 1000 can include obtaining multiple points describing the environment. This can include obtaining ranging data points (e.g., from LIDAR) or obtaining map data describing the environment. Ranging data points can include real or simulated sensor echoes or points in space. Map data can include real or simulated map data (e.g., real scans of the environment, simulated scans of a synthetic environment, synthetic environmental data, etc.). Map data can include one or more layers of data. Map data can include data describing the path of a traveling road, such as lanes. Map data can be continuous. Map data can be sampled to obtain discrete markers, such as reference curves indicating the traveling road (e.g., lane centerlines, lane boundaries, etc.). Map data can include ground surface data.

[0153] At 1006, example method 1000 can include projecting multiple points onto a coordinate frame associated with sensor data to obtain projected point markers. For example, a projection transformation can be used to project three-dimensional point data onto a two-dimensional sensor coordinate frame. In this way, for example, point data can be registered to marked sensor data.

[0154] At 1008, example method 1000 can include associating one or more of the projected point markers defined by spatial region data with the detected object. For example, even before projection, any 3D label can be associated with a corresponding location in the point data coordinate frame. For example, the 3D bounding box of the marker can be located within ranging data or map data, and any point falling within the 3D bounding box can be associated with the detected object. Therefore, the projected markers corresponding to these points can also be associated with the object. Furthermore, additional projected point markers can be projected onto the spatial region of the sensor data defined by the spatial region data. For example, the projected markers can fall within the 2D bounding box of the marker defined in the sensor coordinate frame. These additional projected markers can be associated with the detected object.

[0155] For example, one technique involves finding all data points within the volume formed by the base of the marked 3D bounding box polygons for an associated 2D / 3D pair. These data points can be projected into a sensor data frame. The remaining points / markers can then be found within the height of the 2D bounding box. These points can all be assigned a class associated with the marked bounding box.

[0156] At 1010, example method 1000 can include determining one or more offsets from one or more of the projected point markers to a reference point in the spatial region data. For example, the reference point of the spatial region data can include the centroid of the bounding box (2D, 3D, or both) described by the spatial region data. The offset can include a two-dimensional or three-dimensional distance between each associated marker and the reference point.

[0157] The training dataset can include determined offsets, the association between point markers and objects, and spatial region data of the markers (e.g., the dimensions of the bounding box). The training dataset can include ego vehicle orientation, sensor-captured orientation, etc.

[0158] Figure 11 A flowchart depicts a method 1100 for training one or more machine learning operational models (e.g., object detection models) according to various aspects of this disclosure. One or more portions of method 1100 can be implemented by a computing system comprising one or more computing devices, such as computing systems described, for example, with reference to other figures (e.g., autonomous platform 110, vehicle computing system 180, remote system 160, etc.). Figure 12 (Systems, etc.). Each corresponding part of method 1100 can be executed by any (or any combination of) one or more computing devices. Furthermore, one or more parts of method 1100 can be implemented on the hardware components of the device described herein (e.g., such as...). Figure 1 , Figure 2 , Figure 12 (etc.) to, for example, verify one or more systems or models.

[0159] Figure 11 Elements executed in a specific order are depicted for illustrative and discussion purposes. Using the disclosure provided herein, those skilled in the art will understand that elements of any method discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Reference is made to elements / terms described with respect to other systems and figures for illustrative purposes. Figure 11 ,and Figure 11 This is not intended to be restrictive. One or more parts of method 1100 can be additionally or alternatively executed by other systems.

[0160] At 1102, method 1100 can include obtaining training data for training a machine learning operational model. The training data can include multiple training instances. The training data can include data generated according to example method 1000.

[0161] When an autonomous platform is within its environment, it can use one or more autonomous platforms (e.g., autonomous platform 110) or their sensors to collect training data. As an example, when a vehicle operates along one or more travel roads, it can use one or more autonomous vehicles (e.g., autonomous platform 110, autonomous vehicle 310, autonomous vehicle 350, etc.) or their sensors to collect training data. In some examples, other sensors (such as mobile device-based sensors, ground-based sensors, aerial sensors, satellite-based sensors, or essentially any sensor interface configured to acquire and / or record measurement data) can be used to collect training data.

[0162] Training data can comprise multiple training sequences partitioned across multiple datasets (e.g., training datasets, validation datasets, or test datasets). Each training sequence can include multiple pre-recorded perceptual data points, point clouds, images, etc. In some implementations, each sequence can include (e.g., LiDAR point clouds collected using LiDAR sensors on an autonomous platform), images (e.g., images collected using monochrome or stereo imaging sensors, etc.). For example, in some implementations, multiple images can be scaled for training and evaluation.

[0163] At 1104, method 1100 can include selecting training instances based at least in part on the training data.

[0164] At 1106, method 1100 can include inputting training instances into a machine learning operational model.

[0165] At 1108, method 1100 is capable of generating one or more loss metrics and / or one or more targets of the machine learning operational model based on at least a portion of the output of the machine learning operational model and labels associated with training instances.

[0166] Foreground data (e.g., foreground data 430) can be used to calculate the masking loss. For example, point markers that are not associated with the projected point markers can be excluded from the loss. For example, multiple point markers can be projected onto the image. Pixels that are not associated with the projected point markers (e.g., located beyond a threshold distance from the markers) can be excluded from the loss calculation (e.g., the weights associated with portions of the point data can be set to zero).

[0167] At 1110, method 1100 can include modifying at least one parameter of at least a portion of a machine learning operational model based at least partially on at least one of the loss metrics and / or at least one of the objectives. For example, a computational system can modify at least a portion of a machine learning operational model based at least partially on at least one of the loss metrics and / or at least one of the objectives.

[0168] In some implementations, the machine learning operational model can be trained in an end-to-end manner. For example, in some implementations, the machine learning operational model can be fully differentiable.

[0169] After being updated, the system can provide an operating model or an operating system including an operating model for verification by a verification system. In some implementations, the verification system can evaluate or verify the operating system. The verification system can trigger retraining, de-debugging, etc., of the operating system based on, for example, failure to meet verification thresholds in one or more regions.

[0170] Figure 12 This is a block diagram of an example computing ecosystem 10 according to an exemplary embodiment of the present disclosure. The example computing ecosystem 10 may include a first computing system 20 and a second computing system 40 communicatively coupled via one or more networks 60. In some embodiments, the first computing system 20 or the second computing system 40 may implement one or more of the systems, operations, or functions described herein for verifying one or more systems or operating systems (e.g., remote system 160, onboard computing system 180, autonomous system 200, etc.).

[0171] In some embodiments, the first computing system 20 can be included in an autonomous platform and used to perform the functions of the autonomous platform as described herein. For example, the first computing system 20 can be airborne on an autonomous vehicle and implement an autonomous system for autonomously operating the autonomous vehicle. In some embodiments, the first computing system 20 can represent the entire onboard computing system or a portion thereof (e.g., positioning system 230, perception system 240, planning system 250, control system 260, or combinations thereof). In other embodiments, the first computing system 20 may not be airborne on the autonomous platform. The first computing system 20 can include one or more different physical computing devices 21.

[0172] The first computing system 20 (e.g., its computing device 21) may include one or more processors 22 and memory 23. The one or more processors 22 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one or more processors operatively connected. The memory 23 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, and combinations thereof.

[0173] Memory 23 is capable of storing information accessible by one or more processors 22. For example, memory 23 (e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) is capable of storing data 24 that can be obtained (e.g., received, accessed, written, manipulated, created, generated, stored, retrieved, downloaded, etc.). Data 24 may include, for example, sensor data, map data, data associated with autonomous functions (e.g., data associated with sensing, planning, or control functions), analog data, or any data or information described herein. In some embodiments, the first computing system 20 is capable of obtaining data from one or more memory devices located remotely from the first computing system 20.

[0174] Memory 23 is capable of storing computer-readable instructions 25 that can be executed by one or more processors 22. Instructions 25 can be software written in any suitable programming language or can be implemented in hardware. Additionally or alternatively, instructions 25 can be executed in logically or virtually separate threads on processor 22.

[0175] For example, memory 23 may store instructions 25 that can be executed by one or more processors (e.g., by one or more processors 22, by one or more other processors, etc.) to perform any operation, function, or method / process (or part thereof) described herein using computing device 21, first computing system 20, or other systems with processors having execution instructions. For example, the operation may include implementing system verification (e.g., as described herein).

[0176] In some embodiments, the first computing system 20 is capable of storing or including one or more models 26. In some embodiments, model 26 can be or can otherwise include one or more machine learning models (e.g., a machine learning operating system, etc.). As an example, model 26 can be or can otherwise include various machine learning models, such as, for example, regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear or nonlinear models. Example neural networks include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the first computing system 20 can include one or more models for implementing a subsystem of autonomous system 200, which includes any of the following: localization system 230, perception system 240, planning system 250, or control system 260.

[0177] In some implementations, the first computing system 20 can obtain one or more models 26 using the communication interface 27 to communicate with the second computing system 40 via the network 60. For example, the first computing system 20 can store the models 26 (e.g., one or more machine learning models) in the memory 23. The first computing system 20 can then implement the models 26 using or otherwise (e.g., by the processor 22). As an example, the first computing system 20 can implement the models 26 to locate an autonomous platform in an environment, perceive the environment of the autonomous platform or objects therein, plan one or more future states for the autonomous platform to move through the environment, control the autonomous platform for interacting with the environment, etc.

[0178] The second computing system 40 may include one or more computing devices. The second computing system 40 may include one or more processors 42 and memory 43. The one or more processors 42 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one or more processors operatively connected. The memory 43 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, and combinations thereof.

[0179] Memory 43 is capable of storing information accessible by one or more processors 42. For example, memory 43 (e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) is capable of storing data 44 that can be accessed. Data 44 may include, for example, sensor data, model parameters, map data, simulation data, simulated environmental scenes, simulated sensor data, data associated with vehicle trips / services, or any data or information described herein. In some embodiments, the second computing system 40 is capable of obtaining data from one or more memory devices located remotely from the second computing system 40.

[0180] The memory 43 may also store computer-readable instructions 45 that can be executed by one or more processors 42. The instructions 45 may be software written in any suitable programming language or may be implemented in hardware. Additionally or alternatively, the instructions 45 may be executed in logically or virtually separate threads on the processor 42.

[0181] For example, memory 43 can store executable instructions 45 (e.g., by one or more processors 42, one or more processors 22, one or more other processors, etc.) to perform any of the operations, functions, or methods / processes described herein (e.g., using computing device 41, second computing system 40, or other systems having processors for executing instructions, such as computing device 21 or first computing system 20). This can include, for example, the functions of autonomous system 200 (e.g., localization, perception, planning, control, etc.) or other functions associated with the autonomous platform (e.g., remote assistance, mapping, fleet management, trip / service allocation and matching, etc.). This can also include, for example, an operating system for validating machine learning.

[0182] In some embodiments, the second computing system 40 may include one or more server computing devices. When the second computing system 40 includes multiple server computing devices, such server computing devices can operate according to various computing architectures, including, for example, sequential computing architectures, parallel computing architectures, or some combination thereof.

[0183] The second computing system 40 may additionally or alternatively include one or more models 46 for model 26 at the first computing system 20. As an example, model 46 may be, or may otherwise include, various machine learning models (e.g., machine learning operating systems), such as, for example, regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear or nonlinear models. Example neural networks include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the second computing system 40 may include one or more models of one or more autonomous systems 200.

[0184] In some embodiments, the second computing system 40 or the first computing system 20 can train one or more machine learning models of model 26 or model 46 using one or more model trainers 47 and training data 48. The model trainer 47 can train any one of model 26 or model 46 using one or more training or learning algorithms. An example training technique is error backpropagation. In some embodiments, the model trainer 47 can perform supervised training techniques using labeled training data. In other embodiments, the model trainer 47 can perform unsupervised training techniques using unlabeled training data. In some embodiments, the training data 48 can include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, environments, etc.). In some embodiments, the second computing system 40 can implement simulations for obtaining training data 48 or for implementing model trainers 47 for training or testing model 26 or model 46. As an example, the model trainer 47 can train one or more components of the machine learning model of the autonomous system 200 using unsupervised training techniques with an objective function (e.g., cost, reward, heuristic, constraint, etc.). In some implementations, the model trainer 47 can perform multiple generalization techniques to improve the generalization ability of the trained model. Generalization techniques include weight decay, dropout, or other techniques.

[0185] For example, in some embodiments, the second computing system 40 is capable of generating training data 48 according to example aspects of this disclosure. For example, the second computing system 40 is capable of generating training data 48. For example, the second computing system 40 is capable of implementing the method according to example aspects of this disclosure. The second computing system 40 is capable of using the training data 48 to train model 26. For example, in some embodiments, the first computing system 20 is capable of including an onboard or otherwise associated computing system with a real or simulated autonomous vehicle. In some embodiments, model 26 is capable of including a perception or machine vision model configured for onboard deployment in a real or simulated autonomous vehicle or service. In this way, for example, the second computing system 40 is capable of providing a training pipeline for training model 26.

[0186] The first computing system 20 and the second computing system 40 may each include a communication interface 27 and a communication interface 49, respectively. Communication interfaces 27 and 49 can be used to communicate with each other or with one or more other systems or devices, including systems or devices located remotely from the first computing system 20 or the second computing system 40. Communication interfaces 27 and 49 may include any circuitry, components, software, etc., for communicating with one or more networks (e.g., network 60). In some embodiments, communication interfaces 27 and 49 may include one or more of the following: a communication controller, receiver, transceiver, transmitter, port, conductor, software, or hardware for transmitting data.

[0187] Network 60 can be any type of network or combination of networks that allows communication between devices. In some implementations, the network can include one or more of a local area network (LAN), a wide area network (WAN), the Internet, a secure network, a cellular network, a mesh network, a peer-to-peer communication link, or some combination thereof, and can include any number of wired or wireless links. Communication through network 60 can be achieved, for example, by using a network interface with any type of protocol, protection scheme, encoding, format, encapsulation, etc.

[0188] Figure 12 An example computing ecosystem 10 that can be used to implement this disclosure is illustrated. Other systems can also be used. For example, in some embodiments, the first computing system 20 can include a model trainer 47 and training data 48. In such embodiments, models 26 and 46 can be trained and used locally at the first computing system 20. As another example, in some embodiments, computing system 20 may not be connected to other computing systems. Additionally, components illustrated or discussed as being included in one of computing systems 20 or 40 can alternatively be included in the other of computing systems 20 or 40.

[0189] The computational tasks discussed herein, which are performed at computing devices remote from an autonomous platform (e.g., an autonomous vehicle), can alternatively be performed at the autonomous platform (e.g., via a vehicle computing system of the autonomous vehicle), or vice versa. Such configurations are possible without departing from the scope of this disclosure. The use of computer-based systems allows for a wide variety of possible configurations, combinations, and partitions of tasks and functions across two or more components. Computer-implemented operations can be performed on a single component or across multiple components. Computer-implemented tasks or operations can be performed sequentially or in parallel. Data and instructions can be stored in a single memory device or across multiple memory devices.

[0190] Various aspects of this disclosure have been described with reference to its illustrative embodiments. Upon reading this disclosure, those skilled in the art will be able to conceive of many other embodiments, modifications, or variations within the scope and spirit of the appended claims. Any and all features of the following claims can be combined or rearranged in any possible manner. Therefore, the scope of this disclosure is by way of example and not as a limitation, and this subject matter disclosure does not exclude the inclusion of such modifications, variations, or additions to the subject matter as will be apparent to those skilled in the art. Furthermore, terms are described herein using lists of example elements combined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided only for illustrative purposes. For example, a list combined by a specific conjunction such as “or” can refer to “at least one” or “any combination” of the example elements listed therein, where “or” is understood as “and / or” unless otherwise indicated. Furthermore, terms such as “based on” should be understood as “at least partially based on.”

[0191] Using the disclosure provided herein, those skilled in the art will understand that elements of any claim, operation, or process discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without departing from the scope of this disclosure. For illustrative purposes, some claim elements are described using letter references and are not intended to be restrictive. Letter references do not imply a specific order of operations. For example, letter identifiers such as (a), (b), (c), ..., (i), (ii), (iii), ... can be used to illustrate operations. Such identifiers are provided for the reader's convenience and do not indicate a specific order of steps or operations. Operations illustrated by list identifiers such as (a), (i), etc., can be performed before, after, or concurrently with another operation illustrated by list identifiers such as (b), (ii), etc.

Claims

1. A computer-implemented method, comprising: (a) Obtain sensor data describing the environment of the autonomous vehicle; (b) Obtain multiple points describing the environment; (c) Using a machine learning object detection model and based on the sensor data, determine the association between one or more of the plurality of points and objects in the environment; as well as (d) Using the machine learning object detection model to generate offsets of one or more points relative to a spatial region of the environment associated with the object.

2. The computer-implemented method of any one of the preceding claims, wherein, The point is obtained using the output of an optical detection and ranging (LIDAR) sensor or a radar detection and ranging (RADAR) sensor.

3. The computer-implemented method of any of the preceding claims, wherein, (c) includes: The points and the sensor data are input into the machine learning object detection model; and Object data is obtained from the machine learning object detection model at the projected location of the point in the reference frame of the sensor data, wherein the object data indicates that the object may exist at the projected location of the one or more points.

4. The computer-implemented method of any one of the preceding claims, wherein, Obtaining the object data includes subsampling the detection map generated by the machine learning object detection model based on the points.

5. The computer-implemented method of any of the preceding claims, wherein, One or more parts of the machine learning object detection model are configured to sparsely use portions of the output layer based on the location in the sensor data corresponding to the projection location.

6. The computer-implemented method of any one of the preceding claims, wherein, The machine learning object detection model includes multiple output headers, wherein the multiple output headers include: A classification head that predicts the category associated with a portion of the sensor data; and The regression head regresses the offset using one or more foreground points obtained using the classification head.

7. The computer-implemented method of claim 6, wherein, The regression head regresses one or more dimensions of the boundary of the spatial region.

8. The computer-implemented method of any one of the preceding claims, wherein, The machine learning object detection model includes multiple output headers, wherein the multiple output headers include: A classification head that predicts the category associated with a portion of the sensor data; A first regression head, which regresses the offset in two dimensions for one or more foreground points obtained using the classification head; and A second regression head regresses the offset in three dimensions for one or more foreground points obtained using the classification head.

9. The computer-implemented method according to claim 8, wherein: The first regression head regresses one or more first dimensions of the first boundary of the spatial region in two dimensions; as well as The second regression head regresses one or more second dimensions of the second boundary of the spatial region in three dimensions.

10. The computer-implemented method of any one of the preceding claims, wherein, The machine learning object detection model is trained in the following way: Obtain real-world point labels that indicate the real-world association between the object and one or more of the points; as well as The sparse loss of the machine learning object detection model is determined by comparing the object data with the real-world point labels.

11. The computer-implemented method of any of the preceding claims, wherein, (d) includes: Determine the offset of the centroid of the boundary of the spatial region; and Determine one or more dimensions of the boundary.

12. The computer-implemented method of any one of the preceding claims, wherein, (d) includes: a first offset of a centroid of a first boundary of the spatial region in two dimensions; and a second offset of a centroid of a second boundary of the spatial region in three dimensions.

13. The computer-implemented method of any preceding claim, comprising: based on determining that the speed of the object is below a threshold, outputting a characteristic for the object indicating that the object is a static object; and outputting the characteristic to a motion planning system of the autonomous vehicle.

14. The computer-implemented method of any preceding claim, comprising: based on determining that the speed of an object is below a threshold and that the object is positioned proximate to a travel roadway in the environment, outputting a characteristic for the object indicating that the object is a static object; and outputting the characteristic to a motion planning system of the autonomous vehicle.

15. The computer-implemented method of any preceding claim, wherein: the machine-learned object detection model is trained using training sensor data having a training field of view and training points having a training resolution; the sensor data is associated with a runtime field of view; and the points are obtained in (c) at a runtime resolution selected based on a comparison of the training field of view to the runtime field of view.

16. The computer-implemented method of any preceding claim, comprising: projecting the points into a reference frame of the sensor data using a projection transform; determining one or more offsets of the spatial region relative to the points; based on the determined one or more offsets, determining a projection error of the projected points; and re-calibrating the projection transform based on the determined projection error.

17. The computer-implemented method of any preceding claim, comprising: identifying a lane in which the object is positioned based on the one or more points.

18. The computer-implemented method of any one of the preceding claims, wherein, the input to the machine-learned object detection model comprises one or more input channels, the one or more input channels comprising depth data, the depth data being determined using the plurality of points.

19. The computer-implemented method of any one of the preceding claims, wherein, the input to the machine-learned object detection model comprises: one or more input channels comprising depth data, the depth data being determined using the plurality of points; and one or more input channels comprising other sensor data, wherein the other sensor data optionally comprises image data or image color channel data.

20. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations comprising: the computer-implemented method of any one or more of the preceding claims.

21. One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations comprising: the computer-implemented method of any one or more of the preceding claims. ​

Images