Map-anchored object detection

Abstract

An example method includes: (a) obtaining sensor data describing an environment of an autonomous vehicle; (b) obtaining a plurality of travel lane markings from map data describing the environment; (c) determining, using a machine-learned object detection model and based on the sensor data, an association between one or more of the plurality of travel lane markings and an object in the environment; and (d) generating, using the machine-learned object detection model, an offset of the one or more travel lane markings relative to a spatial region of the environment associated with the object.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Patent Application No. 18 / 354,415 (filed July 18, 2023), which is incorporated herein by reference in its entirety. Background Technology

[0003] Autonomous platforms can process data to perceive the environment in which they are driving. For example, autonomous vehicles can use various sensors to perceive their environment and identify objects around them. The autonomous vehicle can then identify the appropriate path through its perceived surroundings and navigate along that path with minimal or no human input. Summary of the Invention

[0004] Example embodiments of this disclosure relate to systems and techniques for anchoring object detection to map data. Autonomous vehicles can process sensor data to detect objects in their environment. Autonomous vehicles can also access map data that provides rich information about the environment, such as lane boundary information, elevation maps, etc. The machine learning object detection model of the autonomous vehicle perception system according to this disclosure can process sensor data fused with map data to directly determine the location of the detected objects in the mapping environment. In this way, for example, the perception system can leverage existing knowledge of the environment (e.g., information encoded in the map data) to simplify the detection task.

[0005] For example, anchoring detection to map data can simplify the detection task by constraining the solution space of the detection output used to locate the target location around associated map markers. For instance, map data can provide lane markings (e.g., markings along the lane centerline) for locating lanes in a road. Lane markings can encode two-dimensional or three-dimensional locations of the lane centerline. The perception system can transform lane marker locations into a reference frame for the sensor data. For example, an autonomous vehicle can locate itself within map data, thereby estimating its position and orientation within the mapping environment. By extending this by using calibration data from onboard sensors, the perception system can determine the relative orientation of the sensors with respect to the map data. In this way, the perception system can use the estimated pose of the camera to project lane marker locations into a camera reference frame to detect objects represented within two-dimensional image data. This projection can immediately provide estimated three-dimensional location values ​​for pixels in the image data because the projected markers themselves carry rich associations with the map data.

[0006] Object detection models can generate object detection output at projected lane marker locations by optimizing the local solution space in the region of the projected point. For example, an object detection model can regress the bounding box size and the offset of the bounding box relative to the projected lane marker location. In this way, for example, the system can infer the image context to predict the bounding box while anchoring the prediction to a definite point on the map.

[0007] By simplifying the detection task in this way, perception systems can achieve better detection outputs using limited sensor data. For example, LiDAR echoes may become increasingly sparse over a longer range. In contrast, map data can be stored and retrieved at any resolution over any range. Thus, fusing map data of the environment with sensor data depicting the same environment can create a (relatively) dense grid of 3D reference locations, which can lay the foundation for inference of object detection models even over long ranges.

[0008] Furthermore, object detection models can be range-invariant. For example, an object detection model can operate without explicitly depending on an absolute range. In this way, for example, an object detection model can be applied to sensor inputs and map projections of various ranges. An object detection model can operate at runtime in a different range domain than the one used for training. For example, an object detection model trained using near-range camera inputs can be deployed at runtime to perform object detection on long-range camera data.

[0009] Advantageously, the example object detection model according to this disclosure can learn to implicitly (or explicitly) correct for projection errors. Projection errors may be caused by, for example, sensor miscalibration, errors in the estimation of the vehicle's pose relative to the map, etc. By jointly processing fused sensor and map data, the object detection model can refine the detected object locations using the complete context of the sensor data, even in the presence of projection errors. For example, even when projected lane markings may not be precisely aligned with the drawn lane markings depicted in the sensor data, the object detection model can (implicitly) learn to identify drawn lane markings and other contextual cues to adjust the predicted relationship with the projected markings to accommodate errors in the projected markings themselves. The perception system can also explicitly obtain an estimate of the projection error to aid in error correction in future object detection processing loops.

[0010] Advantageously, even with coarse range estimation, the example object detection model according to this disclosure can provide improved lane detection. For example, in some scenarios, accurate lane detection may have a greater impact on planning decisions than precise range estimation. For instance, determining that a vehicle is stopped on the shoulder of a road can be valuable, even if the exact range in which the vehicle is located has not yet been determined with high confidence. This can provide longer range detection with greater certainty, thus giving the vehicle more time to plan and execute actions in response to detected objects.

[0011] Using image processing over long distances alone can involve uncertainties across a large horizontal range. Given this level of uncertainty, accurately determining, for example, whether an object is located on the shoulder of a road or within an active traffic lane can be challenging for such conventional systems. By directly fusing sensor data with long-range map data, which may include lane contour data, the example perception system according to this disclosure can more easily associate detected objects with specific lanes of a road.

[0012] The techniques disclosed herein can provide numerous technical effects and benefits that improve the functionality of autonomous vehicles and their computing systems, and advance the field of autonomous driving as a whole. For example, the perception system according to this disclosure can achieve better detection outputs using limited sensor data, thereby increasing the vehicle's perception range for a given configuration of sensor capabilities. Additionally, the perception system can compute object detection outputs more efficiently. For a given-size machine learning object detection model, fusing map and sensor data using geometric priors can free up model parameters for tasks that independently predict 3D locations. This allows the expressiveness of model parameters to focus on a simplified task optimized in local areas marked on projected maps. Additionally, when fused with map data, autonomous vehicles can use cheaper, more robust sensors (e.g., camera sensors compared to LiDAR sensors) to increase detection range, reduce the overall cost of the vehicle, improve functionality, and ultimately improve the adoption rate of emerging technologies for autonomous vehicles.

[0013] For example, in one aspect, this disclosure provides an example method for object detection. The example method may include obtaining sensor data describing the environment of an autonomous vehicle. The example method may include obtaining multiple road markers from map data describing the environment. The example method may include using a machine learning object detection model and determining, based on the sensor data, the association between one or more of the multiple road markers and objects in the environment. The example method may include using a machine learning object detection model to generate offsets of one or more road markers relative to a spatial region of the environment associated with the object.

[0014] In some implementations of the example method, the driving road markings include lane markings. In some implementations of the example method, the lane markings include centerline markings.

[0015] In some implementations of the example method, (c) includes inputting driving road markings and sensor data into a machine learning object detection model; and obtaining object data from the machine learning object detection model at the projection location of the driving road markings in the reference frame of the sensor data, wherein the object data indicates that an object may exist at the projection location of one or more driving road markings.

[0016] In some implementations of the example method, obtaining object data includes downsampling the detection map generated by the machine learning object detection model based on driving road markings.

[0017] In some implementations of the example method, one or more parts of the machine learning object detection model are configured to sparsely use portions of the output layer based on locations in the sensor data that correspond to the projected locations.

[0018] In some implementations of the example method, the machine learning object detection model is trained by: obtaining ground truth driving road sign labels that indicate the ground truth association between an object and one or more driving road signs; and determining the sparsity loss of the machine learning object detection model based on a comparison of the object data and the ground truth driving road sign labels.

[0019] In some implementations of the example method, (d) includes the offset of the centroid of the boundary of the spatial region and one or more dimensions of the boundary.

[0020] In some implementations of the example method, (d) includes determining a first offset of the centroid of the first boundary of the spatial region in two dimensions; and determining a second offset of the centroid of the second boundary of the spatial region in three dimensions.

[0021] In some implementations of the example method, the example method includes: based on determining that the object's rate is below a threshold and that the object is located near a driving road in the environment, outputting a characteristic indicating that the object is a static object; and outputting the characteristic to the motion planning system of the autonomous vehicle.

[0022] In some implementations of the example method, the example method includes: based on determining that the object's rate is below a threshold and that the object is located near a driving road in the environment, outputting a characteristic indicating that the object is a static object; and outputting the characteristic to the motion planning system of the autonomous vehicle.

[0023] In some implementations of the example method, (b) includes sampling discrete driving road markers from continuous driving road map data.

[0024] In some implementations of the example method, the spatial area of ​​the environment is outside the effective range of the autonomous vehicle's LIDAR sensors.

[0025] In some implementations of the example methods, the machine learning object detection model is trained using training sensor data with a training field of view and training driving road markings with a training resolution. In some implementations of the example methods, the sensor data is associated with a runtime field of view. In some implementations of the example methods, in (c), the driving road markings are obtained at a runtime resolution selected based on a comparison of the training field of view and the runtime field of view.

[0026] In some embodiments of the example method, the example method includes projecting driving road markings onto a reference frame of sensor data using a projection transformation. In some embodiments of the example method, the example method includes determining one or more offsets of a spatial region relative to the driving road markings. In some embodiments of the example method, the example method includes determining a projection error of the projected driving road markings based on the determined one or more offsets. In some embodiments of the example method, the example method includes recalibrating the projection transformation based on the determined projection error.

[0027] In some implementations of the example method, the example method includes identifying the lane in which an object is located based on one or more driving road markings.

[0028] For example, in one aspect, this disclosure provides an example autonomous vehicle control system for controlling an autonomous vehicle. In some embodiments, the example autonomous vehicle control system includes one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform operations. Operations may include acquiring sensor data describing the environment of the autonomous vehicle. Operations may include acquiring a plurality of driving road markings from map data describing the environment. Operations may include using a machine learning object detection model and determining, based on the sensor data, the association between one or more of the plurality of driving road markings and objects in the environment. Operations may include using a machine learning object detection model to generate offsets of one or more driving road markings relative to a spatial region of the environment associated with the object.

[0029] In some implementations of the example autonomous vehicle control system, road markings include lane markings. In some implementations of the example autonomous vehicle control system, lane markings include centerline markings.

[0030] In some implementations of the example autonomous vehicle control system, (c) includes inputting driving road markings and sensor data into a machine learning object detection model; and obtaining object data from the machine learning object detection model at the projection location of the driving road markings in the reference frame of the sensor data, wherein the object data indicates that an object may exist at the projection location of one or more driving road markings.

[0031] In some implementations of the example autonomous vehicle control system, obtaining object data includes downsampling a detection map generated by a machine learning object detection model based on driving road markings.

[0032] In some implementations of the example autonomous vehicle control system, one or more parts of the machine learning object detection model are configured to sparsely use portions of the output layer based on locations in the sensor data that correspond to the projected locations.

[0033] In some implementations of the example autonomous vehicle control system, the machine learning object detection model is trained by: obtaining ground truth driving road sign labels indicating the ground truth association between an object and one or more driving road signs; and determining the sparsity loss of the machine learning object detection model based on a comparison of the object data and the ground truth driving road sign labels.

[0034] In some implementations of the example autonomous vehicle control system, (d) includes the offset of the centroid of the boundary of the spatial region and one or more dimensions of the boundary.

[0035] In some implementations of the example autonomous vehicle control system, (d) includes determining a first offset of the centroid of a first boundary of a spatial region in two dimensions; and determining a second offset of the centroid of a second boundary of a spatial region in three dimensions.

[0036] In some implementations of the example autonomous vehicle control system, example operations include: based on determining that the rate of an object is below a threshold, outputting a characteristic indicating that the object is a static object; and outputting the characteristic to the motion planning system of the autonomous vehicle.

[0037] In some implementations of the example autonomous vehicle control system, example operations include: based on determining that the object's rate is below a threshold and that the object is near a driving road in the environment, outputting a characteristic indicating that the object is a static object; and outputting the characteristic to the autonomous vehicle's motion planning system.

[0038] In some implementations of the example autonomous vehicle control system, (b) includes sampling discrete driving road markers from continuous driving road map data.

[0039] In some implementations of the example autonomous vehicle control system, the spatial area of ​​the environment is outside the effective range of the autonomous vehicle's LIDAR sensors.

[0040] In some embodiments of the example autonomous vehicle control system, the machine learning object detection model is trained using training sensor data with a training field of view and training driving road markings with a training resolution. In some embodiments of the example autonomous vehicle control system, the sensor data is associated with a runtime field of view. In some embodiments of the example autonomous vehicle control system, in (c), the driving road markings are obtained at a runtime resolution selected based on a comparison of the training field of view and the runtime field of view.

[0041] In some embodiments of the example autonomous vehicle control system, example operations include projecting driving road markings onto a reference frame of sensor data using a projection transformation. In some embodiments of the example autonomous vehicle control system, example operations include determining one or more offsets of a spatial region relative to the driving road markings. In some embodiments of the example autonomous vehicle control system, example operations include determining a projection error of the projected driving road markings based on the determined one or more offsets. In some embodiments of the example autonomous vehicle control system, operations include recalibrating the projection transformation based on the determined projection error.

[0042] In some implementations of the example autonomous vehicle control system, example operations include identifying the lane in which an object is located based on one or more driving road markings.

[0043] For example, in one aspect, this disclosure provides one or more example non-transitory computer-readable media storing instructions executable to cause one or more processors to perform operations. Operations may include obtaining sensor data describing the environment of an autonomous vehicle. Operations may include obtaining multiple driving road markers from map data describing the environment. Operations may include using a machine learning object detection model and determining, based on the sensor data, the association between one or more driving road markers and objects in the environment. Operations may include using a machine learning object detection model to generate offsets of one or more driving road markers relative to a spatial region of the environment associated with the object.

[0044] In some embodiments of one or more example non-transitory computer-readable media, road markings include lane markings. In some embodiments of one or more example non-transitory computer-readable media, lane markings include centerline markings.

[0045] In some implementations of one or more example non-transitory computer-readable media, (c) includes inputting driving road markings and sensor data into a machine learning object detection model; and obtaining object data from the machine learning object detection model at the projection location of the driving road markings in the reference frame of the sensor data, wherein the object data indicates that an object may be present at the projection location of one or more driving road markings.

[0046] In some implementations of one or more example non-transitory computer-readable media, obtaining object data includes downsampling a detection map generated by a machine learning object detection model based on driving road markings.

[0047] In some implementations of one or more example non-transitory computer-readable media, one or more portions of a machine learning object detection model are configured to sparsely use portions of the output layer based on locations in sensor data corresponding to the projected locations.

[0048] In some implementations of one or more example non-transitory computer-readable media, a machine learning object detection model is trained by: obtaining ground truth driving road sign labels that indicate a ground truth association between an object and one or more driving road signs; and determining a sparse loss for the machine learning object detection model based on a comparison of object data and ground truth driving road sign labels.

[0049] In some implementations of one or more example non-transitory computer-readable media, (d) includes an offset of the centroid of the boundary of a spatial region and one or more dimensions of the boundary.

[0050] In some implementations of one or more example non-transitory computer-readable media, (d) includes determining a first offset of the centroid of a first boundary of a spatial region in two dimensions; and a second offset of the centroid of a second boundary of a spatial region in three dimensions.

[0051] In some implementations of one or more example non-transitory computer-readable media, example operations include: based on determining that the rate of an object is below a threshold, outputting a characteristic indicating that the object is a static object; and outputting the characteristic to a motion planning system of an autonomous vehicle.

[0052] In some implementations of one or more example non-transitory computer-readable media, example operations include: based on determining that the object's rate is below a threshold and that the object is near a driving road in the environment, outputting a characteristic indicating that the object is a static object; and outputting the characteristic to a motion planning system of an autonomous vehicle.

[0053] In some implementations of one or more example non-transitory computer-readable media, (b) includes sampling discrete driving road markers from continuous driving road map data.

[0054] In some implementations of one or more example non-transitory computer-readable media, the spatial area of ​​the environment is beyond the effective range of the autonomous vehicle's LIDAR sensors.

[0055] In some embodiments of one or more example non-transitory computer-readable media, the machine learning object detection model is trained using training sensor data with a training field of view and training driving road markings with a training resolution. In some embodiments of one or more example non-transitory computer-readable media, the sensor data is associated with a runtime field of view. In some embodiments of one or more example non-transitory computer-readable media, in (c), the driving road markings are obtained at a runtime resolution selected based on a comparison of the training field of view and the runtime field of view.

[0056] In some embodiments of one or more example non-transitory computer-readable media, example operations include projecting driving road markings onto a reference frame of sensor data using a projection transformation. In some embodiments of one or more example non-transitory computer-readable media, example operations include determining one or more offsets of a spatial region relative to the driving road markings. In some embodiments of one or more example non-transitory computer-readable media, example operations include determining a projection error of the projected driving road markings based on the determined one or more offsets. In some embodiments of one or more example non-transitory computer-readable media, operations include recalibrating the projection transformation based on the determined projection error.

[0057] In some implementations of one or more example non-transitory computer-readable media, example operations include identifying the lane in which an object is located based on one or more driving road markings.

[0058] Other exemplary aspects of this disclosure relate to other systems, methods, vehicles, apparatuses, tangible non-transitory computer-readable media, and devices for performing the functions described herein. These and other features, aspects, and advantages of the various embodiments will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0059] A detailed discussion of embodiments for those skilled in the art is set forth in the description with reference to the accompanying drawings, wherein: Figure 1This is a block diagram of an example operation scenario based on some embodiments of this disclosure; Figure 2 This is a block diagram of an example system according to some embodiments of the present disclosure; Figure 3A This is a representation of an example operating environment according to some embodiments of this disclosure; Figure 3B This is a representation of an example map of the operating environment according to some embodiments of this disclosure; Figure 3C This is a representation of an example operating environment according to some embodiments of this disclosure; Figure 3D This is a representation of an example map of the operating environment according to some embodiments of this disclosure; Figure 4 This is a block diagram of an example system for object detection according to some embodiments of the present disclosure; Figure 5 This is a block diagram showing example inputs based on some embodiments of this disclosure; Figure 6 This is a block diagram of an example system for object detection according to some embodiments of the present disclosure; Figure 7 This is a block diagram illustrating an example of an off-target projection based on some embodiments of this disclosure; Figure 8 This is a flowchart of an example method for object detection according to some embodiments of this disclosure; Figure 9 This is a flowchart of an example method for object detection according to some embodiments of this disclosure; Figure 10 This is a flowchart illustrating an example method for training an operating system for machine learning for object detection, according to some embodiments of this disclosure; and Figure 11 This is a block diagram of an example computing system for object detection according to some embodiments of the present disclosure. Detailed Implementation

[0060] The techniques disclosed herein are described below for illustrative purposes only within the context of an autonomous vehicle. As described herein, the techniques are not limited to autonomous vehicles and can be implemented for or within other autonomous platforms and computing systems.

[0061] refer to Figures 1 to 11 The exemplary implementations of this disclosure are discussed in more detail. Figure 1This is a block diagram of an example operating scenario according to some embodiments of the present disclosure. In the example operating scenario, environment 100 includes an autonomous platform 110 and multiple objects including a first participant 120, a second participant 130, and a third participant 140. In the example operating scenario, the autonomous platform 110 can move through environment 100 and interact with objects located within environment 100 (e.g., the first participant 120, the second participant 130, the third participant 140, etc.). The autonomous platform 110 may optionally be configured to communicate with a remote system 160 via network 170.

[0062] Environment 100 may be or include an indoor environment (e.g., within one or more facilities) or an outdoor environment. For example, an indoor environment may be an environment surrounded by a structure such as a building (e.g., a service warehouse, maintenance site, manufacturing facility, etc.). An outdoor environment may be one or more areas in the external world, such as one or more rural areas (e.g., having one or more rural driving roads, etc.), one or more urban areas (e.g., having one or more urban driving roads, highways, etc.), one or more suburban areas (e.g., having one or more suburban driving roads, etc.), or other outdoor environments.

[0063] Autonomous platform 110 can be any type of platform configured to operate within environment 100. For example, autonomous platform 110 can be a vehicle configured to autonomously sense and operate within environment 100. The vehicle can be a ground-based autonomous vehicle, such as, for example, an autonomous car, truck, van, etc. Autonomous platform 110 can be an autonomous vehicle that can be controlled, connected to, or otherwise associated with appliances, attachments, and / or accessories for transporting people or goods. This can include, for example, an autonomous tractor optionally coupled to a cargo trailer. Additionally or alternatively, autonomous platform 110 can be any other type of vehicle, such as one or more aerial vehicles, water-based vehicles, space-based vehicles, other ground-based vehicles, etc.

[0064] Autonomous platform 110 can be configured to communicate with remote system 160. For example, remote system 160 can communicate with autonomous platform 110 for assistance (e.g., navigation assistance, situational response assistance, etc.), control (e.g., fleet management, remote operation, etc.), maintenance (e.g., updates, monitoring, etc.), or other local or remote tasks. In some embodiments, remote system 160 can provide data instructing autonomous platform 110 to perform tasks. For example, as further described herein, remote system 160 can provide data instructing autonomous platform 110 to perform trips / services such as user transportation trips / services, delivery trips / services (e.g., for goods, cargo, articles), etc.

[0065] Autonomous platform 110 can communicate with remote system 160 using network 170. Network 170 can facilitate the transmission of signals (e.g., electronic signals, etc.) or data (e.g., data from computing devices, etc.) and can include any combination of various wired (e.g., twisted-pair cable, etc.) or wireless communication mechanisms (e.g., cellular, wireless, satellite, microwave, radio frequency, etc.) or any desired network topology. For example, network 170 can include a local area network (e.g., intranet, etc.), a wide area network (e.g., the Internet, etc.), a wireless LAN network (e.g., via Wi-Fi, etc.), a cellular network, a SATCOM network, a VHF network, an HF network, a WiMAX-based network, or any other suitable communication network (or combination thereof) used for transmitting data to or from autonomous platform 110.

[0066] For example, Figure 1 As shown, environment 100 may include one or more objects. Objects may be objects that are not in motion or not predicted to move (“static objects”) or objects that are in motion or predicted to move (“dynamic objects” or “participants”). In some implementations, environment 100 may include any number of participants, such as, for example, one or more pedestrians, animals, vehicles, etc. Participants may move within the environment according to one or more participant trajectories. For example, a first participant 120 may move along any one of first participant trajectories 122A-C, a second participant 130 may move along any one of second participant trajectories 132, a third participant 140 may move along any one of third participant trajectories 142, and so on.

[0067] As further described herein, autonomous platform 110 can utilize its autonomous system to detect these participants (and their movement) and plan their motion to navigate through environment 100 according to one or more platform trajectories 112A-C. Autonomous platform 110 may include an onboard computing system 180. Onboard computing system 180 may include one or more processors and one or more memory devices. The one or more memory devices may store instructions executable by the one or more processors to cause the one or more processors to perform operations or functions associated with autonomous platform 110, including implementing its autonomous system.

[0068] Figure 2This is a block diagram of an example autonomous system 200 for an autonomous platform according to some embodiments of the present disclosure. In some embodiments, the autonomous system 200 may be implemented by a computing system of the autonomous platform (e.g., onboard computing system 180 of autonomous platform 110). The autonomous system 200 may be operable to receive input from sensor 202 or other input devices. In some embodiments, the autonomous system 200 may additionally receive platform data 208 (e.g., map data 210) from local or remote storage devices. The autonomous system 200 may generate control outputs for controlling the autonomous platform (e.g., via platform control device 212, etc.) based on sensor data 204, map data 210, or other data. The autonomous system 200 may include different subsystems for performing various autonomous operations. Subsystems may include a positioning system 230, a sensing system 240, a planning system 250, and a control system 260. The positioning system 230 can determine the location of the autonomous platform within its environment; the perception system 240 can detect, classify, and track objects and participants in the environment; the planning system 250 can determine the trajectory of the autonomous platform; and the control system 260 can convert the trajectory into vehicle controls for controlling the autonomous platform. The autonomous system 200 can be implemented by one or more onboard computing systems. Subsystems may include one or more processors and one or more memory devices. One or more memory devices may store instructions executable by one or more processors to cause one or more processors to perform operations or functions associated with the subsystem. The computing resources of the autonomous system 200 may be shared among its subsystems, or the subsystems may have a collection of dedicated computing resources.

[0069] In some implementations, the autonomous system 200 may be implemented for or by an autonomous vehicle (e.g., a ground-based autonomous vehicle). The autonomous system 200 may perform various processing techniques on inputs (e.g., sensor data 204, map data 210) to perceive and understand the vehicle's surroundings and generate a set of appropriate control outputs to enable maneuvering through the vehicle's surroundings (e.g., [missing information]). Figure 1 The vehicle motion planning (e.g., including one or more trajectories) is performed in an environment 100, etc. In some embodiments, the autonomous vehicle implementing the autonomous system 200 can drive, navigate, operate, etc. with minimal or no interaction from a human operator (e.g., driver, pilot, etc.).

[0070] In some implementations, the autonomous platform can be configured to operate in multiple operating modes. For example, the autonomous platform can be configured to operate in a fully autonomous (e.g., self-driving, etc.) mode, in which the platform is controllable without user input (e.g., able to drive and navigate without input from a human operator present in the autonomous vehicle or a human operator remotely). The autonomous platform can operate in a semi-autonomous mode, in which it can utilize some input from a human operator present in the autonomous platform (or a human operator remotely). In some implementations, the autonomous platform can enter a manual operating mode, in which it can be completely controlled by a human operator (e.g., a human driver, etc.) and can be (e.g., temporarily, permanently, etc.) prohibited or disabled from performing autonomous navigation (e.g., autonomous driving, etc.). The autonomous platform can be configured to operate in other modes, such as, for example, parking or sleep modes (e.g., for use between tasks such as waiting for a trip / service, recharging, etc.). In some implementations, the autonomous platform may implement vehicle operation assistance technologies (e.g., collision mitigation systems, power steering, etc.) to help assist the human operator of the autonomous platform (e.g., when in manual mode, etc.).

[0071] The autonomous system 200 can be mounted on an autonomous platform (e.g., on or within the autonomous platform) and can be configured to operate the autonomous platform in various environments. The environment can be a real-world environment or a simulated environment. In some embodiments, one or more simulation computing devices can simulate one or more of the following: sensors 202, sensor data 204, communication interface 206, platform data 208, or platform control devices 212 used to simulate the operation of the autonomous system 200.

[0072] In some implementations, the autonomous system 200 may utilize the communication interface 206 to communicate with one or more networks or other systems. The communication interface 206 may include interfaces for communicating with one or more networks (e.g., Figure 1 The communication interface 206 may include any suitable component of the network 170 (e.g., network 170, etc.), including, for example, a transmitter, receiver, port, controller, antenna, or other suitable component that can help facilitate communication. In some implementations, the communication interface 206 may include multiple components (e.g., antenna, transmitter, or receiver, etc.) that allow it to implement and utilize various communication technologies (e.g., multiple-input multiple-output (MIMO) technology, etc.).

[0073] In some implementations, the autonomous system 200 can communicate with one or more computing devices (e.g., remote system 160) remote from the autonomous platform via one or more networks (e.g., network 170) using communication interface 206. For example, in some examples, one or more inputs, data, or functionalities of the autonomous system 200 can be added to or replaced by the remote system communicating via communication interface 206. For example, in some implementations, map data 210 can be downloaded to the remote system via a network using communication interface 206. In some examples, one or more of the positioning system 230, sensing system 240, planning system 250, or control system 260 can be updated, influenced, pushed, communicated, etc., by the remote system for assistance, maintenance, situational response overwriting, management, etc.

[0074] Sensor 202 can be mounted on an autonomous platform. In some embodiments, sensor 202 may include one or more types of sensors. For example, one or more sensors may include image capture devices (e.g., visible spectrum cameras, infrared cameras, etc.). Additionally or alternatively, sensor 202 may include one or more depth capture devices. For example, sensor 202 may include one or more light detection and ranging (LIDAR) sensors or radio detection and ranging (RADAR) sensors. Sensor 202 can be configured to generate point data describing at least a portion of a 360-degree view of the surrounding environment. The point data may be point cloud data (e.g., 3D LiDAR point cloud data, RADAR point cloud data). In some embodiments, one or more of the sensors 202 used to capture depth information may be fixed to a rotating device so that the sensor 202 rotates about an axis. The sensor 202 can rotate about the axis while capturing data in spaced sector groups of different portions of a 360-degree view describing the surrounding environment of the autonomous platform. In some embodiments, one or more of the sensors 202 used to capture depth information may be solid-state.

[0075] Sensor 202 may be configured to capture sensor data 204 that indicates at least a portion of the autonomous platform's environment or is otherwise associated with at least a portion of the autonomous platform's environment. Sensor data 204 may include image data (e.g., 2D camera data, video data, etc.), RADAR data, LIDAR data (e.g., 3D point cloud data, etc.), audio data, or other types of data. In some embodiments, the autonomous system 200 may obtain input from additional types of sensors such as an inertial measurement unit (IMU), altimeter, inclinometer, odometer device, location or positioning device (e.g., GPS, compass), wheel encoder, or other types of sensors. In some embodiments, the autonomous system 200 may acquire sensor data 204 associated with a specific component or system of the autonomous platform. This sensor data 204 may indicate, for example, wheel speed, component temperature, steering angle, cargo or passenger status, etc. In some embodiments, the autonomous system 200 may acquire sensor data 204 associated with external conditions such as environmental or weather conditions. In some embodiments, sensor data 204 may include multimodal sensor data. Multimodal sensor data can be acquired by at least two different types of sensors (e.g., sensor 202) and can indicate static objects or participants within the autonomous platform's environment. Multimodal sensor data may include at least two types of sensor data (e.g., camera and LiDAR data). In some implementations, the autonomous platform may use sensor data 204 for sensors located remotely from the autonomous platform (e.g., outside the autonomous platform). This may include, for example, sensor data 204 captured by different autonomous platforms.

[0076] Autonomous system 200 can acquire map data 210 associated with the environment in which the autonomous platform was, is, or will be located. Map data 210 can provide information about the environment or geographic area. For example, map data 210 can provide information about: different driving roads (e.g., roads), segments of driving roads (e.g., road sections), buildings or other objects (e.g., lampposts, crosswalks, curbs, etc.); the location and direction of boundaries or boundary markers (e.g., traffic lanes, parking lanes, turning lanes, bicycle lanes, other lanes, etc.); traffic control data (e.g., the location and instructions of signs, traffic lights, other traffic control equipment, etc.); obstacle information (e.g., temporary or permanent congestion, etc.); event data (e.g., road closures / traffic rule changes due to parades, concerts, sporting events, etc.); nominal vehicle path data (e.g., indicating ideal vehicle paths, such as along the center of a lane, etc.); or any other map data that provides information to help the autonomous platform understand its surrounding environment and its relationships. In some embodiments, map data 210 may include high-resolution map information. Additionally or alternatively, map data 210 may include sparse map data (e.g., lane maps, etc.). In some embodiments, sensor data 204 may be fused with map data 210 in real time or used to update map data 210 in real time.

[0077] Autonomous system 200 may include positioning system 230, which can provide the autonomous platform with an understanding of its location and orientation in the environment. In some examples, positioning system 230 may support one or more other subsystems of autonomous system 200, such as by providing a unified local reference frame for performing operations such as sensing, planning, or control.

[0078] In some implementations, positioning system 230 can determine the current location of the autonomous platform. The current location can include a global location (e.g., relative to a georeferenced anchor point, etc.) or a relative location (e.g., relative to objects in the environment, etc.). Positioning system 230 typically includes or interfaces with any device or circuitry used to analyze the location or changes in location of the autonomous platform (e.g., an autonomous ground-based vehicle, etc.). For example, positioning system 230 can determine location using one or more of the following: inertial sensors (e.g., inertial measurement units, etc.), satellite positioning systems, radio receivers, networking devices (e.g., based on IP addresses, etc.), triangulation or proximity measurements to network access points or other network components (e.g., cell towers, Wi-Fi access points, etc.), or other suitable techniques. The location of the autonomous platform can be used by various subsystems of autonomous system 200 or provided to a remote computing system (e.g., using communication interface 206).

[0079] In some implementations, the positioning system 230 can register the relative positions of elements in the autonomous platform's surrounding environment with recorded positions in map data 210. For example, the positioning system 230 can process sensor data 204 (e.g., LiDAR data, RADAR data, camera data, etc.) for alignment or otherwise registration to a map of the surrounding environment (e.g., from map data 210) to understand the autonomous platform's position within that environment. Therefore, in some implementations, the autonomous platform can identify its position within the surrounding environment (e.g., across six axes, etc.) based on a search of map data 210. In some implementations, given an initial location, the positioning system 230 can update the autonomous platform's location using incremental realignment based on recorded or estimated deviations from the initial location. In some implementations, the position can be directly registered within map data 210.

[0080] In some implementations, map data 210 may include a large amount of data subdivided into geographic tiles, enabling the reconstruction of a desired area of ​​a map stored in map data 210 from one or more tiles. For example, autonomous system 200 may stitch together multiple tiles selected from map data 210 based on locations obtained by positioning system 230 (e.g., multiple tiles selected near the location).

[0081] In some implementations, positioning system 230 can determine the (e.g., relative or absolute) location of one or more attachments or accessories to the autonomous platform. For example, the autonomous platform may be associated with a cargo platform, and positioning system 230 can provide the location of one or more points on the cargo platform. For instance, the cargo platform may include a trailer or other equipment towed to or otherwise attached to or operated by the autonomous platform, and positioning system 230 can provide data describing the (e.g., absolute, relative, etc.) location of the autonomous platform and the cargo platform. Such information can be obtained by other autonomous systems to assist in operating the autonomous platform.

[0082] Autonomous system 200 may include perception system 240, which allows the autonomous platform to detect, classify, and track objects and participants in its environment. Environmental features or objects perceived within the environment may be those within the field of view of sensor 202 or those predicted to be occluded by sensor 202. This may include objects that are not in motion or not predicted to move (static objects) or objects that are in motion or predicted to be in motion (dynamic objects / participants).

[0083] The perception system 240 can determine one or more states (e.g., current or past states, etc.) of one or more objects within the surrounding environment of the autonomous platform. For example, a state can describe (e.g., for a given time, time period, etc.) an estimate of the object's current or past location (also called position); current or past rate / velocity; current or past acceleration; current or past heading; current or past orientation; size / occupied area (e.g., represented by boundary shape, object highlighting, etc.); classification (e.g., pedestrians vs. vehicles vs. bicycles, etc.); associated uncertainties; or other state information. In some implementations, the perception system 240 can use one or more algorithms or machine learning models configured to identify / classify objects based on input from sensor 202 to determine the state. The perception system can use different modalities of sensor data 204 to generate a representation of the environment to be processed by one or more algorithms or machine learning models. In some implementations, as the autonomous platform continues to perceive or interact with objects (e.g., maneuvering with or around objects, yielding to objects, etc.), the state of one or more identified or unidentified objects can be maintained and updated over time. In this way, the perception system 240 can provide an understanding of the current state of the environment (e.g., including objects therein) based on records of previous states of the environment (e.g., the movement history of objects within it). Such information can be helpful when the autonomous platform plans its movement through the environment.

[0084] Autonomous system 200 may include planning system 250, which can be configured to determine how an autonomous platform interacts with and moves within its environment. Planning system 250 may determine one or more motion plans for the autonomous platform. Motion plans may include one or more trajectories (e.g., motion trajectories) indicating the path to be followed by the autonomous platform. Trajectories may have a certain length or time range. The length or time range may be defined by computational planning limits of planning system 250. Motion trajectories may be defined by one or more waypoints (with associated coordinates). Waypoints may be future locations of the autonomous platform. Motion plans may be continuously generated, updated, and considered by planning system 250.

[0085] The motion planning system 250 can determine a strategy for the autonomous platform. The strategy can be a set of discrete decisions made by the autonomous platform (e.g., yielding to a participant, yielding in the opposite direction to a participant, merging, lane changing). The strategy can be selected from multiple potential strategies. The selected strategy can be the lowest-cost strategy determined by one or more cost functions. The cost function can, for example, evaluate the probability of a collision with another participant or object.

[0086] Planning system 250 can determine the desired trajectory for executing a strategy. For example, planning system 250 can obtain one or more trajectories for executing one or more strategies. Planning system 250 can evaluate trajectories or strategies (e.g., using scores, costs, rewards, constraints, etc.) and rank them. For example, planning system 250 can use forecast outputs indicating interactions between the autonomous platform's trajectory and one or more objects (e.g., proximity, intersection, etc.) to inform the evaluation of candidate trajectories or strategies for the autonomous platform. In some implementations, planning system 250 can utilize static costs to evaluate the autonomous platform's trajectory (e.g., "avoiding lane boundaries," "minimizing sprints," etc.). Additionally or alternatively, planning system 250 can utilize dynamic costs to evaluate the autonomous platform's trajectory or strategy based on forecast results of the current operational scenario (e.g., forecast trajectories or strategies leading to interactions between participants, forecast trajectories or strategies leading to interactions between participants and the autonomous platform, etc.). Planning system 250 can rank trajectories based on one or more static costs, one or more dynamic costs, or a combination thereof. The planning system 250 can select a motion plan (and corresponding trajectory) based on the ranking of multiple candidate trajectories. In some implementations, the planning system 250 can select the highest-ranked candidate or the highest-ranked feasible candidate.

[0087] The planning system 250 can then verify the selected trajectory against one or more constraints before the trajectory is executed by the autonomous platform.

[0088] To aid its motion planning decisions, the planning system 250 can be configured to perform a forecasting function. The planning system 250 can forecast the future state of the environment. This may include forecasting the future states of other participants in the environment. In some implementations, the planning system 250 can forecast future states based on current or past states (e.g., those developed or maintained by the perception system 240). In some implementations, the future state may be or include predicted trajectories (e.g., positions over time) of objects in the environment such as other participants. In some implementations, one or more future states may include one or more probabilities associated with them (e.g., marginal probabilities, conditional probabilities). For example, one or more probabilities may include one or more probabilities conditioned on strategies or trajectory options available to the autonomous platform. Additionally or alternatively, probabilities may include probabilities conditioned on trajectory options available to one or more other participants.

[0089] In some implementations, the planning system 250 can perform interactive forecasting. The planning system 250 can determine the motion plan for the autonomous platform by utilizing an understanding of how the predicted future state of the environment might be affected by the execution of one or more candidate motion plans. As an example, refer again... Figure 1Autonomous platform 110 can determine candidate motion plans corresponding to platform trajectory sets 112A-C, which correspond to the first participant trajectories 122A-C of first participant 120, trajectory 132 of second participant 130, and trajectory 142 of third participant 140, respectively (e.g., having corresponding trajectory correspondences indicated by matching line patterns). For example, autonomous platform 110 (e.g., using its autonomous system 200) can predict that platform trajectory 112A, which moves autonomous platform 110 faster into the area in front of first participant 120, may be associated with first participant 120 reducing its forward speed and giving way to autonomous platform 110 faster based on first participant trajectory 122A. Additionally or alternatively, autonomous platform 110 can predict that platform trajectory 112B, which moves autonomous platform 110 gently into the area in front of first participant 120, may be associated with first participant 120 slightly reducing its speed and giving way to autonomous platform 110 slowly based on first participant trajectory 122B. Additionally or alternatively, the autonomous platform 110 may predict that a platform trajectory 112C aligned parallel to the first participant 120 may be associated with the first participant 120 not yielding any distance to the autonomous platform 110 based on the first participant trajectory 122C. Based on a comparison of the predicted scenario with the expected set of outcomes (e.g., by scoring the scenario based on cost or reward), the planning system 250 may select a motion plan (and its associated trajectory) in light of the interaction between the autonomous platform and the environment 100. In this way, for example, the autonomous platform 110 may interleave its prediction and motion planning functionalities.

[0090] To implement the selected motion plan, the autonomous system 200 may include a control system 260 (e.g., a vehicle control system). Typically, the control system 260 provides an interface between the autonomous system 200 and the platform control device 212 for implementing the strategies and motion plans generated by the planning system 250. For example, the control system 260 can implement the selected motion plan / trajectory to control the movement of the autonomous platform through its environment by following the selected trajectory (e.g., waypoints included therein). The control system 260 can, for example, translate the motion plan into instructions for the appropriate platform control device 212 (e.g., acceleration control, braking control, steering control, etc.). As an example, the control system 260 can translate the selected motion plan into instructions for adjusting steering components (e.g., steering angle) by a certain degree, applying a certain amount of braking force, increasing / decreasing speed, etc. In some embodiments, the control system 260 can communicate with the platform control device 212 through a communication channel including, for example, one or more data buses (e.g., a controller area network (CAN), etc.), an onboard diagnostic connector (e.g., OBD-II), or a combination of wired or wireless communication links. The platform control device 212 can send data, messages, signals, etc. to the autonomous system 200 or receive data, messages, signals, etc. from the autonomous system 200 (or vice versa) through the communication channel.

[0091] Autonomous system 200 can receive auxiliary signals from remote assistance system 270 via communication interface 206. Remote assistance system 270 can communicate with autonomous system 200 via a network (e.g., via network 170 as remote system 160). In some embodiments, autonomous system 200 can initiate a communication session with remote assistance system 270. For example, autonomous system 200 can initiate a session based on or in response to a trigger. In some embodiments, the trigger can be an alarm, error signal, map feature, request, location, traffic conditions, road conditions, etc.

[0092] After the initial session, autonomous system 200 can provide environment data to remote assistance system 270. Environment data may include sensor data 204 and status data from the autonomous platform. For example, environment data may include real-time camera feeds from the autonomous platform's cameras and its current speed. The operator of remote assistance system 270 (e.g., a human operator) can use the environment data to select assistance signals. Assistance signals can provide values ​​or adjustments to various operating parameters or characteristics of autonomous system 200. For example, assistance signals may include waypoints (e.g., path around obstacles, lane changes, etc.), rate or acceleration profiles (e.g., speed limits, etc.), relative motion commands (e.g., platoon formation, etc.), operating characteristics (e.g., use of backup systems, reduced energy handling modes, etc.), or other signals assisting autonomous system 200.

[0093] Autonomous system 200 can use auxiliary signals as inputs to one or more autonomous subsystems to perform autonomous functions. For example, planning subsystem 250 can receive auxiliary signals as inputs for generating motion plans. For example, auxiliary signals may include constraints for generating motion plans. Additionally or alternatively, auxiliary signals may include cost or reward adjustments for influencing the motion plans of planning subsystem 250. Additionally or alternatively, autonomous system 200 may also consider auxiliary signals as suggestive inputs for consideration, in addition to other received data (e.g., sensor inputs, etc.).

[0094] Autonomous system 200 may be platform-agnostic, and control system 260 may provide control commands to platform control devices 212 for various different platforms used for autonomous mobility (e.g., multiple different autonomous platforms equipped with autonomous control systems). This may include various types of autonomous vehicles (e.g., cars, vans, SUVs, trucks, electric vehicles, combustion-powered vehicles, etc.) from various different manufacturers / developers that operate in various different environments and perform one or more vehicle services in some embodiments.

[0095] For example, refer to Figure 3A The operating environment may include a dense environment 300. The autonomous platform may include an autonomous vehicle 310 controlled by an autonomous system 200. In some embodiments, the autonomous vehicle 310 may be configured for maneuverability in dense environments, such as having a configured wheelbase or other specifications. In some embodiments, the autonomous vehicle 310 may be configured to transport goods or passengers. In some embodiments, the autonomous vehicle 310 may be configured to transport a large number of passengers (e.g., buses, shuttles, public buses, etc.). In some embodiments, the autonomous vehicle 310 may be configured to transport goods, such as large quantities of goods (e.g., trucks, vans, step vans, etc.) or smaller goods (e.g., food, personal parcels, etc.).

[0096] refer to Figure 3BA top view 302 of the selected dense environment 300 is shown covering an example trip / service between a first location 304 and a second location 306. The example trip / service may be assigned to an autonomous vehicle 320, for example, by a remote computing system. The autonomous vehicle 320 may be, for example, a vehicle of the same type as autonomous vehicle 310. The example trip / service may include transporting passengers or goods between the first location 304 and the second location 306. In some embodiments, the example trip / service may include traveling to or through one or more intermediate locations, such as loading or unloading passengers or goods. In some embodiments, the example trip / service may be pre-scheduled (e.g., for regular crossings, such as on a transport schedule). In some embodiments, the example trip / service may be on demand (e.g., requested or used to perform taxi, ride-sharing, ride-hailing, courier, delivery services, etc.).

[0097] refer to Figure 3C In another example, the operating environment may include an open road environment 330. The autonomous platform may include an autonomous vehicle 350 controlled by the autonomous system 200. This may include an autonomous tractor for an autonomous truck. In some embodiments, the autonomous vehicle 350 may be configured for high-payload transportation (e.g., large-volume transport of goods or other cargo or passengers), such as for long-distance, high-payload transportation. For example, the autonomous vehicle 350 may include one or more cargo platform attachments, such as a trailer 352. Although in Figure 3C The cargo platform is described as a towable attachment, but in some implementations, one or more cargo platforms may be integrated into the autonomous vehicle 350 (e.g., attached to the chassis of the autonomous vehicle 350, etc.) (e.g., in a van, step van, etc.).

[0098] refer to Figure 3DThe diagram shows a top view of the selection of an open driving road environment 330, including driving roads 332, interchanges 334, transfer hubs 336 and 338, access driving roads 340, and locations 342 and 344. In some embodiments, example trips / services may be assigned to autonomous vehicles (e.g., autonomous vehicle 310 or autonomous vehicle 350) to traverse one or more driving roads 332 (optionally connected via interchanges 334) to transport goods between transfer hubs 336 and 338. For example, in some embodiments, the example trips / services include goods delivery / transportation services, such as goods delivery / transportation services. The example trips / services may be assigned by a remote computing system. In some embodiments, transfer hub 336 may be the origin of goods (e.g., a warehouse, warehouse, facility, etc.), and transfer hub 338 may be the destination of goods (e.g., a retailer, etc.). However, in some embodiments, transfer hub 336 may be an intermediate point along the final journey of goods between its respective origin and its respective destination. For example, the origin of goods or articles can be located at point 342 along access road 340. Goods or articles can be transported accordingly (e.g., by human-driven vehicles, autonomous vehicles 310, etc.) to transfer hub 336 for segmented transport. At transfer hub 336, various goods or articles can be grouped or segmented for longer-distance transport along road 332.

[0099] In some implementations of the example trip / service, a segmented transport of goods may be loaded onto an autonomous vehicle (e.g., autonomous vehicle 350) for transport to one or more other transfer hubs, such as transfer hub 338. For example, although not depicted, it should be understood that the open road environment 330 may include more transfer hubs than transfer hubs 336 and 338, and may include more roads 332 interconnected by more interchanges 334. Simplified maps are presented here for clarity only. In some implementations, one or more goods transported to transfer hub 338 may be distributed (e.g., by human-driven vehicles, by autonomous vehicle 310, etc.) to one or more local destinations, such as along access roads 340 to location 344. In some implementations, the example trip / service may be pre-scheduled (e.g., for periodic traversals, such as on a transport schedule). In some implementations, the example trip / service may be on-demand (e.g., as requested by or for performing charter passenger transport or freight delivery services).

[0100] In order to improve the performance of autonomous platforms such as autonomous vehicles (e.g., autonomous vehicle 310 or 350) controlled at least in part by autonomous system 200, perception system 240 may implement detection techniques according to exemplary aspects of this disclosure.

[0101] Figure 4 This is a block diagram of an example detection data stream. The perception system 240 can access sensor data 400. Sensor data 400 can describe an environment 402. Environment 402 can include a road with lanes 403 and 404 and a shoulder area 405. The perception system 240 can access map data 410. Map data 410 can include multiple layers or data types, such as a bird's-eye view lane boundary layer 411, a terrain layer 412, a graphics layer 413, or other layers. Map data 410 can include a high-resolution two-dimensional or three-dimensional geometric representation of at least a portion of environment 402 in at least one layer. The perception system 240 can implement an object detection model 420 to detect one or more objects in environment 402. Object detection model 420 can generate associations between one or more portions of sensor data 400 or map data 410 and objects in the environment. Foreground data 430 can indicate the associations between one or more portions of sensor data 400 or map data 410 and objects in the environment.

[0102] For example, sensor data 400 may include an image of environment 402. Multiple road markings may be projected onto the image (e.g., based on known calibrations between corresponding image sensors and the autonomous vehicle's positioning within the mapping environment 402). Foreground data 430 of one or more portions of the image data may indicate which projected road marking is associated with a portion of the image data representing an object. For example, foreground data 430 may indicate that road marking 431 (an unfilled circle) is not associated with an object. Foreground data 430 may indicate that road markings 432 and 433 (filled circles) are associated with an object. Foreground data 430 may indicate that road marking 434 (a filled circle) is associated with an object.

[0103] Object detection model 420 can generate spatial region data 440 based on foreground data 430. For example, object detection model 420 can generate bounding boxes or other detection indicators anchored to driving road markings associated with the object. For example, object detection model 420 can determine that markings 432 and 433 are associated with the same object and generate a bounding box 441 with centroid 442. Object detection model 420 can determine that marking 434 is associated with the object and generate a bounding box 443 with centroid 444. For example, object detection model 420 can regress the offset of the bounding box relative to the projected markings.

[0104] In this way, for example, perception system 240 can anchor bounding boxes to map data, thereby directly associating object detection with rich descriptive content in the map data. For example, detected objects can be directly registered with lanes of a road (e.g., locations in active lanes or shoulder areas). Such perception data can be used to quickly determine high-level information about the environment. For example, perception system 240 can determine the speed of distant objects and whether the objects are located in a specific lane. For example, perception system 240 can identify the lane in which the object is located. Additionally or alternatively, perception system 240 can determine whether a vehicle is a stationary vehicle (e.g., having a speed below a threshold). Even if more granular information may not be available, determining the distance of a vehicle to which lane it is can be useful. For example, perception system 240 can determine whether a stationary vehicle is in a shoulder lane or an active traffic lane, enabling the autonomous vehicle to plan accordingly. By determining this information over a long period, the autonomous vehicle can have additional time to plan and execute appropriate actions.

[0105] Sensor data 400 may include sensor data 204 from sensor 202. Sensor data 400 may include multiple sensor modes. Sensor data 400 may include imaging data (e.g., from an image sensor, such as a camera). Sensor data 400 may include ranging data (e.g., LiDAR data, RADAR data, stereo camera data, etc.).

[0106] Different image sensor configurations can capture sensor data 400. Imaging devices with varying fields of view can contribute data to sensor data 400. Sensor data 400 can include data from long-range cameras (e.g., cameras with telephoto lenses, cameras with sufficient resolution to resolve long-range details even in wider fields of view). Sensor data 400 can include data from near-range cameras (e.g., cameras with wide-angle lenses, lower-resolution cameras resolving sparse details over long ranges). Sensor data 400 can include fused sensor data. Sensor data 400 can include upsampled image data. For example, machine learning image processing models can be used to recover details in the image data to denoise, deblur, sharpen, upsample, etc., to improve resolution. In this way, for example, the effective sensing range of the imaging device can be extended.

[0107] Sensor data 400 may include long-range sensing data. Long-range sensing data may include data describing the environment 402 beyond the range of the ranging sensor. For example, long-range sensing data may include data describing a portion of the environment 402 beyond the detection range of a LIDAR unit, RADAR unit, or stereo unit. The detection range of a LIDAR or RADAR unit may be, for example, a range where the confidence level or uncertainty measure exceeds a threshold.

[0108] Map data 410 may include data describing environment 402. Map data 410 may be registered to sensor data 400 by positioning system 230. Positioning system 230 may process sensor data 400 or sensor data 204 to determine the position and orientation of the autonomous vehicle within environment 402, thereby determining the spatial relationship between the vehicle and the map-based representation of environment 402 in map data 410.

[0109] For example, map data 410 may include data representing one or more lanes of a road. Map data 410 may represent road lanes using, for example, vector-based curve representations (e.g., with or without waypoints, including line segments, splines, etc.). Labeling can be obtained by sampling a continuous representation of the road profile to obtain labeled data at a desired resolution. For example, map layer 411 may include driving road data. Driving road data may include data indicating the path of a driving road. Driving road data may include lane boundaries, lane centerlines, or any other representation of the lane path. Driving road data may include a continuous representation of the driving road profile that can be sampled at any resolution.

[0110] Although this document describes various example implementations relative to map data 410, it should be understood that other 3D data can be used in a similar manner (e.g., as an addition to or replacement of map data). For example, as described herein, LiDAR data can be used with map data 410 to fuse with image data. For example, LiDAR data can be passed to object detection model 420 in another input channel. It should also be understood that various techniques can be used in combination at different range scales. For example, within the LiDAR range, greater weight can be given to LiDAR-based sensor fusion detection. Outside the LiDAR range, greater weight can be given to map-based sensor fusion detection. The transition in between can be a smooth transition of detection weights to facilitate switching from one dominant mode to another.

[0111] Sensor data 400 can also depict the driving road described in the driving road data of map data 410. Positioning the vehicle within map data 410 establishes a relationship between map data 410 and sensor data 400, enabling registration of the driving road depiction in sensor data 400 with the driving road data of map data 410. For example, this relationship may include kinematic relationships between one or more sensors and the vehicle, the vehicle's heading within the mapping environment, and the sensor's field of view or orientation relative to the vehicle. This relationship can be based on calibration data that can be updated and refined over time to account for alignment shifts.

[0112] For example, multiple road markings can be projected onto at least a portion of the sensor data 400. For example, the road data can be projected into the camera coordinate system of a camera that captures images of the road. The projection can be based on a camera transformation or a projection matrix. For example, the camera sensor can be calibrated and fixed to the vehicle. In the vehicle coordinate system... P v = (x v , y v , z v ) T The projection of a point can be obtained by the projection matrix. C=K[R v |t v ] Define , where K is the camera calibration matrix, and R v , t v It involves rotation and translation from the vehicle coordinate system to the camera coordinate system. Once the vehicle coordinate system and the map coordinate system are aligned, points in the map coordinate system can be projected into the camera coordinate system.

[0113] For example, Figure 5 A set of input data 500, including image 501 and lane marking projection 502, is depicted. Lane marking projection 502 may include driving road markings that have been projected into the camera space associated with image 501. In this way, for example, pixels of image 501 may be associated with driving road markings. Lane marking projection 502 may be stored in the input channel associated with image 501.

[0114] In a similar manner, other map data can be projected onto the coordinate system associated with the sensor. For example, map data 410 may include high-resolution ground mapping (e.g., terrain layer 412). Projection markers may include points indicating the ground surface.

[0115] Refer again Figure 4 Object detection model 420 can process sensor data 400 and map data 410 to generate spatial region data 440. Object detection model 420 may include one or more machine learning models. Object detection model 420 may include models configured to process sensor data (single-modal sensor data, multi-modal sensor data, fused sensor data, aggregated sensor data, etc.). Object detection model 420 may include neural networks, such as deep neural networks. Object detection model 420 may use attention mechanisms (e.g., self-attention, such as in a transformer model architecture). Object detection model 420 may include convolutional layers configured to generate spatial feature maps based on the input. For example, an example object detection model may include a ResNet architecture.

[0116] Object detection model 420 can obtain foreground data 430 to guide the generation of spatial region data 440. For example, foreground data 430 may include data indicating the probability of an object being present at a particular location. Foreground data 430 may include binary flags indicating whether an object exists at a particular location. Foreground data 430 may include a multi-channel data structure indicating the presence of objects in each channel associated with the class of that channel. For example, channels of the data structure may be associated with the vehicle class. Data values ​​in that layer may indicate that a vehicle exists at a location associated with the index position of the data value (e.g., corresponding to a pixel indexed at the same location on a different layer). Other classes may correspond to other layers.

[0117] Foreground data 430 can indicate the probability of an object being present in a portion of the image associated with projected map data 410. For example, foreground data 430 can include an indication of a low probability of a foreground object in a region associated with projection marker 431 (e.g., one or more pixels falling below the projection marker). Foreground data 430 can include an indication of a higher probability of a foreground object (e.g., a value of 1, or a value that is rounded up, rounded, or otherwise aligned to a specified value) in a region associated with projection markers 432, 433, or 434 (e.g., one or more pixels falling below the projection marker).

[0118] Foreground data 430 can therefore provide an indication of the association between one or more driving road markers and objects in the environment (e.g., vehicles in the foreground). Various metrics can be used to determine the association between one or more driving road markers and objects in the environment. Example metrics include distance metrics, such as defining the radius of a region around a marker within which an object detected will be associated with that marker. Example distance metrics can be range-adaptive, allowing the metric to relax at longer distances to improve the recall of the detection model.

[0119] Foreground data 430 can act as a mask or filter applied to other model layers to enable object detection model 420 to generate spatial region data 440 based on foreground markers in foreground data 430. For example, one or more components of object detection model 420 can “excite” or sparsely process foreground points of activity to enable object detection model 420 to regress spatial region data 440 relative to those foreground points of activity.

[0120] Spatial region data 440 may include boundary regions regressed from foreground markers. Spatial region data 440 may be regressed relative to foreground markers in sensor coordinate space (e.g., in an image frame). In this way, for example, rich scene information in map data 410 can be directly indexed using foreground markers and corresponding spatial region data 440.

[0121] For example, the location of the centroid 442 of bounding box 441 can be located at a certain distance from the markers 432 / 433. Object detection model 420 can regress the distance or offset between centroid 442 and markers 432, 433. Object detection model 420 can process image data with reference to foreground markers to output the offset.

[0122] Figure 6 The illustration shows an example architecture of one or more object detection models 420. A backbone model 610 can process input data 500. For example, the backbone model 610 can process sensor data (e.g., image data) and map data (e.g., projected map markers) together. In this way, for example, the backbone model 610 can generate feature maps that encode fused information across input channels. The backbone model 610 can infer the entire input image and the entire set of projected markers. The backbone model 610 can implicitly associate projected markers with identifiable features of the input image.

[0123] The output of the backbone model 610 can be passed to task-specific output heads, such as a classification head 612, a 2D head 614, and a 3D head 616. The classification head 612 can process the feature maps generated by the backbone model 610 to determine foreground data. For example, the classification head 612 can be configured to determine the presence of objects in one or more classes (e.g., vehicles, pedestrians, etc.). This objectness data can be masked with projected map data 410 to obtain foreground data 430 indicating one or more foreground markers.

[0124] The 2D head 614 can process the feature map generated by the backbone model 610 to generate 2D boundary features 624 that can be used to obtain a bounding box in the sensor coordinate system. The 3D head 616 can process the feature map generated by the backbone model 610 to generate 3D boundary features 626 that can be used to obtain a 3D bounding box.

[0125] Any one or more of the task-specific output headers may include machine learning model components. Any one or more of the task-specific output headers may include neural networks. The task-specific output headers may process feature maps from various parts of the backbone model 610. For example, the backbone model 610 may include layers of various resolutions and depths. The task-specific headers may process one or more layers of different resolutions and depths.

[0126] Refer again Figure 4 The object detection model 420 can perform refinement techniques to obtain high-quality spatial region data 440. For example, the object detection model 420 can apply non-maximum suppression to the predicted spatial region data to determine possible bounding boxes.

[0127] The example output decoding procedure may include sampling a classification heatmap of the projected road markings (e.g., to obtain foreground data 430). For markings indicated as foreground, the 2D detection head 614 may decode the 2D targets to obtain 2D bounding boxes in the image frame. For example, the 2D detection head 614 may receive feature maps as input from the backbone model 610 and the foreground data 430. Based on these inputs, the 2D detection head 614 may regress 2D spatial region data relative to the foreground markings. One round of non-maximum suppression may be applied to the 2D regression of the spatial region data. The 3D detection head 616 may receive feature maps as input from the backbone model 610 and the foreground data 430. Based on these inputs, the 3D detection head 616 may regress 3D spatial region data relative to the foreground markings. One round of non-maximum suppression may be applied to the 3D regression of the spatial region data.

[0128] One benefit of the technique described in this paper is that the object detection model 420 can be directly anchored to the high-resolution map data and, simultaneously within the high-resolution map data, has access to all sensor contexts around the projected point throughout the regression process. This can provide learned pose correction. For example, the object detection model 420 can regress offsets from map markers to spatial region data 440. In some scenarios, if the projected map data markers contain projection errors, the offsets may be skewed (e.g., larger than they should be). For example, if the projected lane centerline is misaligned with the lane centerline in the image, vehicles in the imaged lane can be detected at the imaging location such that the predicted offset includes the distance from the centroid to the lane marker plus the misalignment distance.

[0129] Advantageously, the object detection model 420 can automatically compensate for misalignment. Figure 7The illustration shows an example of misalignment between projected road markings and imaged lanes. Because spatial region data can be regressed from sensor data 400, detected boundaries can be correctly identified in sensor data 400 despite the potential misalignment. Furthermore, although potential misalignment exists, registration between sensor data 400 and map data 410 can continue to provide improved coarse-grained detection information over a long range. For example, even when the projected lane markings are misaligned, object detection model 420 can learn to register the detection with the correct lane on the road.

[0130] The perception system 240 can explicitly regress values ​​representing misalignment. For example, components of the object detection model 420 can learn to regress translation or rotation errors or other projection errors in the map data projection. This error can inform future iterations of the perception system 240 to recalibrate detection. For example, the perception system 240 can estimate and correct projection errors in real time.

[0131] For example, the relative pose between the sensor coordinate system and the map coordinate system can be adjusted to reduce offset values. For instance, taking the median (or average, or other statistical measure) of offsets across a given set of detections in a scene can provide a measure of projection quality. Projection errors can be reduced by adjusting the relative pose (e.g., a projection transformation such as the camera matrix) to decrease the statistical measurement of offsets.

[0132] The perception system 240 can use a dedicated output head to perform projection error estimation. The perception system 240 can also use a separate neural network trained to regress the projection error based on the output of the object detection model 420.

[0133] The perception system 240 can also use other input signals to predict the projection error. Examples of other input signals may include sensor data indicating the state of one or more vehicle components. For example, by processing sensor data indicating suspension movement, the perception system 240 can determine that the projection error may be associated with suspension movement. For example, in some cases, sensor attitude calibration may be affected by extreme vibrations or inertial loads due to the flexibility of sensor mounting configurations. In this way, other input signals, for example, can be used by the perception system 240 to predict the projection error value. This predicted projection error value can be used to adjust the calibration of the projection transformation used to project map data 410 into the sensor coordinate system.

[0134] Figure 8This is a flowchart of a method 800 for performing object detection according to aspects of this disclosure. One or more portions of method 800 may be implemented by a computing system including one or more computing devices, such as the computing systems described, for example, with reference to other figures (e.g., autonomous platform 110, vehicle computing system 180, remote system 160, etc.). Figure 11 (Systems, etc.). Each corresponding part of method 800 can be executed by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of method 800 can be implemented on hardware components of the device described herein (e.g., such as...). Figure 1 , Figure 2 , Figure 11 wait).

[0135] Figure 8 Elements are depicted in a specific order for illustrative and discussion purposes. Those skilled in the art who use the disclosure provided herein will understand that elements of any method discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. For illustrative purposes, reference is made to elements / terms described relative to other systems and figures. Figure 8 This is not intended to be restrictive. One or more parts of method 800 may be additionally or alternatively executed by other systems.

[0136] At 802, example method 800 may include obtaining sensor data describing the environment of the autonomous vehicle. For example, the sensor data may include sensor data 204, sensor data 400, etc.

[0137] At 804, example method 800 may include obtaining multiple driving road markings from map data describing the environment. For example, driving road markings may include lane markings (e.g., centerline markings, lane boundary markings, etc.). In some implementations, example method 800 may include sampling discrete driving road markings at 804 from continuous driving road map data (e.g., vector-based map data format).

[0138] At 806, example method 800 may include using a machine learning object detection model and determining the association between one or more driving road markers of a plurality of driving road markers and objects in the environment based on sensor data. In some embodiments, example method 800 at 806 may include inputting driving road markers and sensor data into the machine learning object detection model, and obtaining object data from the machine learning object detection model at the projection location of the driving road markers in the reference frame of the sensor data. For example, the object data may indicate that an object may be present at the projection location of one or more driving road markers. For example, the object data may include foreground data 430 (e.g., object state data).

[0139] In some implementations, example method 800 at 806 may include downsampling of a detection map generated by a machine learning object detection model based on driving road markings. For example, object detection model 420 may generate a detection map of object state data indicating where objects might be located in the input frame (e.g., using a task-specific output head such as classification head 612). In some implementations of example method 800, one or more portions of the machine learning object detection model are configured to sparsely activate the output layer based on locations in the sensor data corresponding to the projection locations. For example, object detection model 420 may process foreground driving road markings to regress spatial region data relative to those foreground driving road markings.

[0140] At 808, example method 800 may include using a machine learning object detection model to generate offsets relative to one or more driving road markings of a spatial region of the environment associated with the object. In some implementations, example method 800 at 808 may include offsets for determining the centroid of the boundary of the spatial region and one or more dimensions for determining the boundary. For example, object detection model 420 may regress the offsets to the centroid of the boundary surrounding the object for each of one or more projected driving road markings associated with the object. In some implementations, example method 800 at 808 may include a first offset for determining the centroid of a first boundary of the spatial region in two dimensions and a second offset for determining the centroid of a second boundary of the spatial region in three dimensions. For example, one or more first offsets may be determined in a sensor coordinate system (e.g., in a coordinate system aligned with the width and height of the image). The second offset may be determined for a dimension in the coordinate system (e.g., in the depth dimension).

[0141] In some implementations, example method 800 may include outputting an indicator that the object is a static object based on determining that the object's rate is below a threshold. In some implementations, example method 800 may include outputting the characteristic to a motion planning system for the autonomous vehicle. For example, the motion planning system may plan the autonomous vehicle's motion based on the understanding that the detected object is a static object (e.g., a vehicle parked on the shoulder, such as a parked emergency vehicle).

[0142] In some implementations, example method 800 may include: based on determining that the object's rate is below a threshold and that the object is located near a driving road in the environment, outputting a characteristic indicating that the object is a static object (e.g., on the shoulder of a road). In some implementations, example method 800 may include outputting the characteristic to the motion planning system of the autonomous vehicle.

[0143] In some implementations of Example Method 800, the spatial region of the environment is outside the effective range of the autonomous vehicle's LiDAR sensors. For example, the object detection model can output object detection with spatial region data anchored to 3D map data, without relying on real-time LiDAR scans that reliably provide echoes on the objects.

[0144] In some implementations, example method 800 may include identifying the lane in which an object is located. For example, object detection model 420 may regress an offset based on projected driving road markings. Map data may associate driving road markings with a specific lane or lane type. Example method 800 may include identifying lanes based on this association.

[0145] In some embodiments of Example Method 800, the machine learning object detection model is trained using training sensor data with a training field of view and training driving road markings with a training resolution. For example, the training sensor data may be characterized by a first camera configuration (e.g., utilizing a first field of view, a first resolution, etc.). In some embodiments of Example Method 800, the sensor data (e.g., at runtime) is associated with a runtime field of view. The runtime field of view may be the same as or different from the training field of view. Therefore, driving road markings can be obtained at a runtime resolution selected based on a comparison of the training field of view and the runtime field of view.

[0146] In this way, for example, the scope invariance of the exemplary embodiments of this disclosure can enable transfer learning. Transfer learning can include training on one sensor configuration using different sensor configurations and running it at inference time.

[0147] Normalizing the distribution of map data relative to the sensor resolution can facilitate transfer learning. For example, the distribution of map data markers that typically matches the size of objects across different configurations can help improve transfer learning. For instance, a first camera configuration might use a first number of pixels to represent a given object. Map data can be sampled at a first resolution such that the first number of map markers fall on the object. A second camera configuration might use a second number of pixels to represent the same object. Therefore, map data can be sampled at a second resolution such that the second number of map markers falls on the object. For example, the second resolution can be chosen such that the second number matches the first number. Matching the distribution of map points allows object detection models to operate on different sensor configurations. One method for determining the scaling factor for the extent of map points is to determine the ratio of the number of pixels representing a unit height at a given distance (e.g., this ratio can provide a scaling factor).

[0148] In some embodiments, example method 800 may include determining a projection error or attitude error of the projected road markings. This can be used to recalibrate the projection operation. For example, example method 800 may include projecting the road markings onto a reference frame of sensor data using a projection transformation. In some embodiments, example method 800 may include determining one or more offsets of a spatial region relative to the road markings. In some embodiments, example method 800 may include determining a projection error of the projected road markings based on the determined one or more offsets. In some embodiments, example method 800 may include recalibrating the projection transformation based on the determined projection error.

[0149] In some embodiments of Example Method 800, Example Method 800 includes obtaining ground truth driving road mark labels indicating a ground truth association between an object and one or more driving road marks, and determining a sparsity loss for a machine learning object detection model based on a comparison of object data and ground truth driving road mark labels. For example, the sparsity loss can be computed by ignoring portions of sensor data that are not associated with driving road marks (e.g., projected driving road marks). In some embodiments of Example Method 800, Example Method 800 includes training a machine learning object detection model based on the sparsity loss.

[0150] Figure 9 More details are provided for obtaining training data with real values. Figure 9This is a flowchart of a method 900 for generating ground truth training data for training an object detection model, according to aspects of this disclosure. One or more parts of method 900 may be implemented by a computing system including one or more computing devices, such as, for example, the computing systems described with reference to other figures (e.g., autonomous platform 110, vehicle computing system 180, remote system 160, etc.). Figure 11 (Systems, etc.). Each corresponding part of method 900 can be executed by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of method 900 can be implemented on hardware components of the device described herein (e.g., such as...). Figure 1 , Figure 2 , Figure 11 wait).

[0151] Figure 9 Elements are depicted in a specific order for illustrative and discussion purposes. Those skilled in the art who use the disclosure provided herein will understand that elements of any method discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Reference is made to elements / terms described relative to other systems and figures. Figure 9 This is for illustrative purposes only and is not intended to be limiting. One or more parts of method 900 may be additionally or alternatively performed by other systems.

[0152] At 902, example method 900 may include obtaining ground truth or labeled sensor data describing the environment. Labeled sensor data includes spatial region data that defines a spatial region of sensor data associated with a detected object. For example, labeled sensor data may include labeled image captures. Labeled image captures may include frames from video recordings. Labeled sensor data may include sensor data that has been automatically or manually reviewed and annotated with one or more tags. Labeled sensor data may be obtained from log data from real or simulated driving sessions.

[0153] At 904, example method 900 may include obtaining map data describing the environment. The map data may include real or simulated map data (e.g., a real scan of the environment, a simulated scan of a synthetic environment, synthetic environment data, etc.). The map data may include one or more layers of data. The map data may include data describing the path of a driving road, such as lanes. The map data may be continuous. The map data may be sampled to obtain discrete markers that indicate, for example, reference curves used for driving roads (e.g., lane centerlines, lane boundaries, etc.). The map data may include ground surface data.

[0154] At 906, example method 900 may include projecting map data into a coordinate system associated with the sensor data to obtain projected map labels. For example, a projection transformation can be used to project three-dimensional map data into a two-dimensional sensor coordinate system. In this way, for example, the map data can be registered to labeled sensor data.

[0155] At 908, example method 900 may include associating one or more projected map markers from a projected map of spatial region data with a detected object. For example, even before projection, any 3D label can be associated with a corresponding location in the map data. For instance, a labeled 3D bounding box can be positioned within the map data, and any map point falling within the 3D bounding box can be associated with a detected object. Therefore, the projected markers corresponding to those map points can also be associated with the object. Furthermore, additional projected map markers can be projected onto a spatial region of sensor data defined by the spatial region data. For example, projected markers can fall within a labeled 2D bounding box defined in the sensor coordinate system. These additional projected markers can be associated with the detected object.

[0156] For example, one technique involves finding all map data points within the volume formed by the base of the labeled 3D bounding box polygon for an associated 2D / 3D pair. These map data points can be projected into the sensor data coordinate system. The remaining points / markers can then be found within the height of the 2D bounding box. These points can all be assigned to the class associated with the labeled bounding box.

[0157] At 910, example method 900 may include determining one or more offsets for one or more projected map markers in the projected map data to a reference point in the spatial region data. For example, the reference point in the spatial region data may include the centroid of a bounding box (2D, 3D, or both) described by the spatial region data. The offset may include a two-dimensional or three-dimensional distance between each associated marker and the reference point.

[0158] The training dataset may include determined offsets, map marker-to-object associations, and labeled spatial region data (e.g., bounding box dimensions). The training dataset may also include ego vehicle orientation, sensor capture orientation, etc.

[0159] Figure 10 A flowchart depicts a method 1000 for training one or more operational models of machine learning (e.g., object detection models) according to aspects of this disclosure. One or more portions of method 1000 may be implemented by a computing system including one or more computing devices, such as the computing systems described, for example, with reference to other figures (e.g., autonomous platform 110, vehicle computing system 180, remote system 160, etc.). Figure 11(Systems, etc.). Each corresponding part of method 1000 can be executed by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of method 1000 can be implemented on hardware components of the devices described herein (e.g., such as...). Figure 1 , Figure 2 , Figure 11 (e.g., to validate one or more systems or models).

[0160] Figure 10 Elements are depicted in a specific order for illustrative and discussion purposes. Those skilled in the art who use the disclosure provided herein will understand that elements of any method discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. For illustrative purposes, reference is made to elements / terms described relative to other systems and figures. Figure 10 This does not imply limitation. One or more parts of method 1000 may be additionally or alternatively executed by other systems.

[0161] At 1002, method 1000 may include obtaining training data for training an operational model of machine learning. The training data may include multiple training instances. The training data may include data generated according to example method 900.

[0162] When an autonomous platform is within its environment, one or more autonomous platforms (e.g., autonomous platform 110) or their sensors can be used to collect training data. As an example, when a vehicle operates along one or more driving roads, one or more autonomous vehicles (e.g., autonomous platform 110, autonomous vehicle 310, autonomous vehicle 350, etc.) or their sensors can be used to collect training data. In some examples, other sensors can be used to collect training data, such as mobile device-based sensors, ground-based sensors, airborne sensors, satellite-based sensors, or essentially any sensor interface configured to acquire and / or record measurement data.

[0163] Training data may include multiple training sequences partitioned across multiple datasets (e.g., training datasets, validation datasets, or test datasets). Each training sequence may include multiple pre-recorded perceptual data points, point clouds, images, etc. In some implementations, each sequence may include (e.g., LiDAR point clouds collected using LiDAR sensors on an autonomous platform), images (e.g., images collected using mono or stereo imaging sensors), etc. For example, in some implementations, multiple images may be scaled for training and evaluation.

[0164] At 1004, method 1000 may include selecting training instances based at least in part on the training data.

[0165] At 1006, method 1000 may include inputting training instances into an operational model for machine learning.

[0166] At 1008, method 1000 may include at least a portion of the output of a machine learning-based operational model and labels associated with training instances to generate one or more loss metrics and / or one or more targets for the machine learning operational model.

[0167] Foreground data (e.g., foreground data 430) can be used to calculate the masking loss. For example, map markers that are not associated with the projected map markers can be excluded from the loss. For example, multiple map markers can be projected onto the image. Pixels that are not associated with the projected markers (e.g., located beyond a threshold distance from the markers) can be excluded from the loss calculation (e.g., the weights associated with portions of the sensor data can be set to zero).

[0168] At 1010, method 1000 may include modifying at least one parameter of at least a portion of the machine learning operational model based at least one of the loss metrics and / or at least one of the objectives. For example, a computational system may modify at least a portion of the machine learning operational model based at least one of the loss metrics and / or at least one of the objectives.

[0169] In some implementations, the operational model for machine learning can be trained in an end-to-end manner. For example, in some implementations, the operational model for machine learning can be fully differentiable.

[0170] After the update, the operating model, or the operating system including the operating model, can be provided for verification by a verification system. In some implementations, the verification system can evaluate or verify the operating system. The verification system can trigger retraining, shutdown, etc., of the operating system based on, for example, failure to meet verification thresholds in one or more regions.

[0171] Figure 11 This is a block diagram of an example computing ecosystem 10 according to an exemplary embodiment of the present disclosure. The example computing ecosystem 10 may include a first computing system 20 and a second computing system 40 communicatively coupled via one or more networks 60. In some embodiments, the first computing system 20 or the second computing system 40 may implement one or more of the systems, operations, or functionalities described herein for verifying one or more systems or operating systems (e.g., remote system 160, onboard computing system 180, autonomous system 200, etc.).

[0172] In some embodiments, the first computing system 20 may be included in an autonomous platform and used to perform the functions of the autonomous platform as described herein. For example, the first computing system 20 may be onboard an autonomous vehicle and implement an autonomous system for autonomously operating the autonomous vehicle. In some embodiments, the first computing system 20 may represent the entire onboard computing system or a portion thereof (e.g., positioning system 230, sensing system 240, planning system 250, control system 260, or combinations thereof). In other embodiments, the first computing system 20 may not be onboard an autonomous platform. The first computing system 20 may include one or more distinct physical computing devices 21.

[0173] The first computing system 20 (e.g., its computing device 21) may include one or more processors 22 and memory 23. The one or more processors 22 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 23 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, and combinations thereof.

[0174] Memory 23 may store information accessible by one or more processors 22. For example, memory 23 (e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) may store data 24 that can be obtained (e.g., received, accessed, written, manipulated, created, generated, stored, pulled, downloaded, etc.). Data 24 may include, for example, sensor data, map data, data associated with autonomous functions (e.g., data associated with sensing, planning, or control functions), analog data, or any data or information described herein. In some embodiments, the first computing system 20 may obtain data from one or more memory devices located remotely from the first computing system 20.

[0175] Memory 23 may store computer-readable instructions 25 that can be executed by one or more processors 22. Instructions 25 may be software written in any suitable programming language or may be implemented in hardware. Additionally or alternatively, instructions 25 may be executed in logically or virtually separate threads on processor 22.

[0176] For example, memory 23 may store instructions 25 that can be executed by one or more processors (e.g., by one or more processors 22, by one or more other processors, etc.) to perform any operation, function, or method / process (or part thereof) described herein using computing device 21, first computing system 20, or other systems with processors having execution instructions. For example, operations may include implementing system verification (e.g., as described herein).

[0177] In some implementations, the first computing system 20 may store or include one or more models 26. In some implementations, model 26 may be, or may otherwise include, one or more machine learning models (e.g., a machine learning operating system, etc.). As an example, model 26 may be, or may otherwise include, various machine learning models, such as, for example, regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear or nonlinear models. Example neural networks include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the first computing system 20 may include one or more models for implementing a subsystem of autonomous system 200, including any of the following: positioning system 230, perception system 240, planning system 250, or control system 260.

[0178] In some implementations, the first computing system 20 may use communication interface 27 to communicate with the second computing system 40 via network 60 to obtain one or more models 26. For example, the first computing system 20 may store the models 26 (e.g., one or more machine learning models) in memory 23. The first computing system 20 may then implement the models 26 using or otherwise (e.g., via processor 22). As an example, the first computing system 20 may implement the models 26 to locate an autonomous platform in an environment, perceive the environment of the autonomous platform or objects therein, plan one or more future states for the autonomous platform to move through the environment, control the autonomous platform for interacting with the environment, etc.

[0179] The second computing system 40 may include one or more computing devices 41. The second computing system 40 may include one or more processors 42 and memory 43. The one or more processors 42 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 43 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, and combinations thereof.

[0180] Memory 43 may store information accessible by one or more processors 42. For example, memory 43 (e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) may store accessible data 44. Data 44 may include, for example, sensor data, model parameters, map data, simulation data, simulated environmental scenes, simulated sensor data, data associated with vehicle trips / services, or any data or information described herein. In some embodiments, the second computing system 40 may obtain data from one or more memory devices located remotely from the second computing system 40.

[0181] The memory 43 may also store computer-readable instructions 45 that can be executed by one or more processors 42. The instructions 45 may be software written in any suitable programming language or may be implemented in hardware. Additionally or alternatively, the instructions 45 may be executed in logically or virtually separate threads on the processor 42.

[0182] For example, memory 43 may store executable instructions 45 (e.g., instructions that can be executed by one or more processors 42, one or more processors 22, one or more other processors, etc.) to perform any of the operations, functions, or methods / processes described herein (e.g., using computing device 41, second computing system 40, or other systems having processors for executing instructions, such as computing device 21 or first computing system 20). This may include, for example, the functionality of autonomous system 200 (e.g., localization, perception, planning, control, etc.) or other functionality associated with an autonomous platform (e.g., remote assistance, mapping, fleet management, trip / service assignment and matching, etc.). This may also include, for example, an operating system for validating machine learning.

[0183] In some embodiments, the second computing system 40 may include one or more server computing devices. Where the second computing system 40 includes multiple server computing devices, such server computing devices may operate according to various computing architectures, including, for example, sequential computing architectures, parallel computing architectures, or some combination thereof.

[0184] As an addition to or alternative to model 26 at the first computing system 20, the second computing system 40 may include one or more models 46. As an example, model 46 may be, or may otherwise include, various machine learning models (e.g., machine learning operating systems), such as, for example, regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear or nonlinear models. Example neural networks include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the second computing system 40 may include one or more models of the autonomous system 200.

[0185] In some implementations, the second computing system 40 or the first computing system 20 can train one or more machine learning models of model 26 or model 46 using one or more model trainers 47 and training data 48. The model trainer 47 can use one or more training or learning algorithms to train any one of model 26 or model 46. An example training technique is backpropagation of error. In some implementations, the model trainer 47 can use labeled training data to perform supervised training techniques. In other implementations, the model trainer 47 can use unlabeled training data to perform unsupervised training techniques. In some implementations, the training data 48 can include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, environments, etc.). In some implementations, the second computing system 40 can implement simulations to obtain training data 48 or to implement the model trainer 47 for training or testing model 26 or model 46. For example, the model trainer 47 can use an objective function (e.g., cost, reward, heuristic, constraint, etc.) to train one or more components of the machine learning model of the autonomous system 200 using unsupervised training techniques. In some implementations, the model trainer 47 can perform various generalization techniques to improve the generalization ability of the trained model. Common techniques include weight decay, dropout, or other techniques.

[0186] For example, in some embodiments, the second computing system 40 may generate training data 48 according to example aspects of this disclosure. For example, the second computing system 40 may generate training data 48. For example, the second computing system 40 may implement a method according to example aspects of this disclosure. The second computing system 40 may use the training data 48 to train model 26. For example, in some embodiments, the first computing system 20 may include an onboard or otherwise associated computing system with a real or simulated autonomous vehicle. In some embodiments, model 26 may include a perception or machine vision model configured for deployment onboard or in a service of a real or simulated autonomous vehicle. In this manner, for example, the second computing system 40 may provide a training pipeline for training model 26.

[0187] The first computing system 20 and the second computing system 40 may each include communication interfaces 27 and 49, respectively. Communication interfaces 27 and 49 can be used to communicate with each other or with one or more other systems or devices, including systems or devices located remotely from the first computing system 20 or the second computing system 40. Communication interfaces 27 and 49 may include any circuitry, components, software, etc., for communicating with one or more networks (e.g., network 60). In some embodiments, communication interfaces 27 and 49 may include one or more of the following: a communication controller, receiver, transceiver, transmitter, port, conductor, software, or hardware for transmitting data.

[0188] Network 60 can be any type of network or combination of networks that allows communication between devices. In some implementations, the network may include one or more of a local area network (LAN), wide area network (WAN), Internet, secure network, cellular network, mesh network, peer-to-peer link, or some combination thereof, and may include any number of wired or wireless links. Communication on network 60 can be accomplished, for example, by using a network interface with any type of protocol, protection scheme, encoding, format, encapsulation, etc.

[0189] Figure 11 An example computing ecosystem 10 that can be used to implement this disclosure is illustrated. Other systems may also be used. For example, in some embodiments, the first computing system 20 may include a model trainer 47 and training data 48. In such embodiments, models 26, 46 may be trained and used locally at the first computing system 20. As another example, in some embodiments, computing system 20 may not be connected to other computing systems. Additionally, components illustrated or discussed as being included in one of computing systems 20 or 40 may alternatively be included in the other of computing systems 20 or 40.

[0190] This document discusses how computational tasks performed at a computing device remote from an autonomous platform (e.g., an autonomous vehicle) can alternatively be performed at the autonomous platform (e.g., a vehicle computing system via the autonomous vehicle), and vice versa. Such configurations can be implemented without departing from the scope of this disclosure. The use of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. Computer-implemented operations can be performed on a single component or across multiple components. Computer-implemented tasks or operations can be performed sequentially or in parallel. Data and instructions can be stored in a single memory device or across multiple memory devices.

[0191] Aspects of this disclosure have been described in accordance with illustrative embodiments thereof. Many other embodiments, modifications, or variations within the scope and spirit of the appended claims will be apparent to those skilled in the art upon review of this disclosure. Any and all features of the following claims may be combined or rearranged in any possible manner. Therefore, the scope of this disclosure is by way of example and not limitation, and this disclosure does not exclude the inclusion of such modifications, variations, or additions to the subject matter, which will be apparent to those skilled in the art. Furthermore, terminology herein is used to describe a list of example elements joined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for illustrative purposes only. For example, a list joined by a particular conjunction such as “or” may refer to “at least one” or “any combination thereof” of the example elements listed therein, and “or” is understood to mean “and / or” unless otherwise stated. Furthermore, terms such as “based on” should be understood to mean “at least partially based on.”

[0192] Those skilled in the art who use the disclosure provided herein will understand that, without departing from the scope of this disclosure, elements of any of the claims, operations, or processes discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways. Some of the claims are described using letter references to claim elements, which are for illustrative purposes and are not intended to be limiting. Letter references do not imply a specific order of operations. For example, letter identifiers such as (a), (b), (c), ..., (i), (ii), (iii) can be used to describe operations. Such identifiers are provided for the convenience of the reader and do not indicate a specific order of steps or operations. Operations indicated by list identifiers (a), (i), etc., may be performed before, after, or in parallel with another operation indicated by list identifiers (b), (ii), etc.

Claims

1. A computer-implemented method, comprising: (a) Obtain sensor data describing the environment of the autonomous vehicle; (b) Obtain multiple driving road markings from map data describing the environment; (c) Using a machine learning object detection model and based on the sensor data, determine the association between one or more of the plurality of driving road signs and objects in the environment; as well as (d) Using the machine learning object detection model, generate offsets of the one or more driving road markings relative to a spatial region of the environment associated with the object. (c) includes: The driving road markings and the sensor data are input into the machine learning object detection model; and At the projection location of the driving road mark in the reference frame of the sensor data, object data is obtained from the machine learning object detection model, wherein the object data indicates that the object may exist at the projection location of the one or more driving road marks.

2. The computer-implemented method according to claim 1, wherein, The road markings include lane markings.

3. The computer-implemented method according to claim 2, wherein, The lane markings include centerline markings.

4. The computer-implemented method according to claim 1, wherein, Obtaining the object data includes: The detection map generated by the machine learning object detection model is downsampled based on the driving road markings.

5. The computer-implemented method according to claim 1, wherein, One or more parts of the machine learning object detection model are configured as follows: Based on the locations in the sensor data that correspond to the projected locations, a portion of the output layer is used sparsely.

6. The computer-implemented method according to claim 1, wherein, The machine learning object detection model is trained through the following operations: Obtain real-value driving road marker labels, the real-value driving road marker labels indicating the real-value association between the object and one or more driving road markers; as well as The sparse loss of the machine learning object detection model is determined by comparing the object data with the real-value driving road marking labels.

7. The computer-implemented method according to claim 1, wherein, (d) includes: Determine the offset of the centroid of the boundary of the spatial region; and Determine one or more dimensions of the boundary.

8. The computer-implemented method according to claim 1, wherein, (d) includes: Determine the first offset of the centroid of the first boundary of the spatial region in two dimensions; and Determine the second offset of the centroid of the second boundary of the spatial region in three dimensions.

9. The computer-implemented method according to claim 1, comprising: Based on the determination that the rate of the object is below a threshold, the characteristic of the object is output, which indicates that the object is a static object; as well as The aforementioned characteristics are output to the motion planning system of the autonomous vehicle.

10. The computer-implemented method according to claim 1, comprising: Based on the determination that the object's rate is below a threshold and the object is located near a driving road in the environment, the characteristics of the object are output, indicating that the object is a static object; as well as The aforementioned characteristics are output to the motion planning system of the autonomous vehicle.

11. The computer-implemented method according to claim 1, wherein, (b) includes: Discrete driving road markers are sampled from continuous driving road map data.

12. The computer-implemented method according to claim 1, wherein, The spatial region of the environment is outside the effective range of the autonomous vehicle's LIDAR sensors.

13. The computer-implemented method according to claim 1, wherein: The machine learning object detection model was trained using training sensor data with a training field of view and training driving road markings with a training resolution. The sensor data is associated with the runtime field of view; as well as In (c), the driving road markings are obtained at a runtime resolution selected based on a comparison of the training field of view and the runtime field of view.

14. The computer-implemented method according to claim 1, comprising: Using projection transformation, the road markings are projected onto the reference frame of the sensor data; Determine one or more offsets of the spatial region relative to the driving road markings; The projection error of the projected road markings is determined based on one or more determined offsets. as well as The projection transformation is recalibrated based on the determined projection error.

15. The computer-implemented method as described in claim 1, comprising: The lane in which the object is located is identified based on one or more driving road markings.

16. The computer-implemented method of claim 1, wherein (d) comprises: The offset is regressed using the machine learning object detection model.

17. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising: One or more processors; as well as One or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the autonomous vehicle control system to perform operations, including: (a) Obtain sensor data describing the environment of the autonomous vehicle; (b) Obtain multiple driving road markings from map data describing the environment; (c) Using a machine learning object detection model and based on the sensor data, determine the association between one or more of the plurality of driving road signs and objects in the environment; and (d) Using the machine learning object detection model, generate offsets of the one or more driving road markings relative to a spatial region of the environment associated with the object. (c) includes: The driving road markings and the sensor data are input into the machine learning object detection model; and At the projection location of the driving road mark in the reference frame of the sensor data, object data is obtained from the machine learning object detection model, wherein the object data indicates that the object may exist at the projection location of the one or more driving road marks.

18. The autonomous vehicle control system as described in claim 17, wherein, Obtaining the object data includes: Based on the driving road markings, the detection map generated by the machine learning object detection model is downsampled.

19. The autonomous vehicle control system as described in claim 17, wherein, (d) includes: Determine the first offset of the centroid of the first boundary of the spatial region in two dimensions; and Determine the second offset of the centroid of the second boundary of the spatial region in three dimensions.

20. The autonomous vehicle control system as described in claim 17, wherein, The machine learning object detection model is trained through the following operations: Obtain real-value driving road marker labels, the real-value driving road marker labels indicating the real-value association between the object and one or more driving road markers; as well as The sparse loss of the machine learning object detection model is determined by comparing the object data with the real-value driving road marking labels.

21. One or more non-transitory computer-readable media storing instructions executable by one or more processors to cause an autonomous vehicle control system to perform operations, said operations including: (a) Obtain sensor data describing the environment of the autonomous vehicle; (b) Obtain multiple driving road markings from map data describing the environment; (c) Using a machine learning object detection model and based on the sensor data, determine the association between one or more of the plurality of driving road signs and objects in the environment; as well as (d) Using the machine learning object detection model, generate offsets of the one or more driving road markings relative to a spatial region of the environment associated with the object. (c) includes: The driving road markings and the sensor data are input into the machine learning object detection model; and At the projection location of the driving road mark in the reference frame of the sensor data, object data is obtained from the machine learning object detection model, wherein the object data indicates that the object may exist at the projection location of the one or more driving road marks.

Images