Vehicle, method for a vehicle, and storage medium

Abstract

The application provides a vehicle, a method for the vehicle, and a storage medium. Generally, an innovative aspect of the subject matter described in this specification can be embodied in a method including the following steps: obtaining area information of at least one area of interest of the vehicle through a sensing circuit; identifying occlusion data associated with the at least one area of interest; determining occlusion information associated with the at least one area of interest based on the area information of the at least one area of interest and the occlusion data associated with the at least one area of interest, wherein the occlusion data includes data associated with the occlusion information, and the occlusion information has a data size smaller than that of the occlusion data; providing the occlusion information to a planning circuit through the sensing circuit for planning a route of the vehicle; and operating the vehicle according to the planned route.

Description

Technical Field

[0001] This specification relates to systems and methods for implementing occluded representations of road features. Background Art

[0002] When making driving decisions, typical autonomous vehicle (AV) systems consider objects (such as other vehicles and obstacles, etc.) that the AV system knows are in the AV environment because the sensor system on the AV observes the objects or because the objects are identified by a map or other data sources. To make driving decisions, the AV system can maintain a model that includes the objects known in the AV environment. The challenges for making good driving decisions also come from vehicles and obstacles that the AV cannot observe or sense, and that cannot otherwise be observed or known to exist based on the available data. Summary of the Invention

[0003] This application provides a method for a vehicle, including: obtaining region information of at least one region of interest of the vehicle through a sensing circuit; identifying occlusion data associated with the at least one region of interest through the sensing circuit; determining occlusion information associated with the at least one region of interest based on the region information of the at least one region of interest and the occlusion data associated with the at least one region of interest through the sensing circuit, wherein the occlusion data includes data associated with the occlusion information, and the occlusion information has a data size smaller than that of the occlusion data; providing the occlusion information to a planning circuit through the sensing circuit for planning the route of the vehicle; and operating the vehicle according to the planned route.

[0004] This application provides a vehicle, including: one or more computer processors; and one or more non-transitory storage media that store instructions which, when executed by the one or more computer processors, cause the method according to this application to be performed.

[0005] This application provides one or more non-transitory storage media that store instructions which, when executed by one or more computing devices, cause the method according to this application to be performed. Brief Description of the Drawings

[0006] Figure 1 An example of an autonomous vehicle (AV) with autonomous capabilities is shown.

[0007] Figure 2 An example "cloud" computing environment is shown.

[0008] Figure 3 A computer system is shown.

[0009] Figure 4Shows an example architecture of an AV.

[0010] Figure 5 Shows examples of inputs and outputs that the perception module can use.

[0011] Figure 6 Shows an example of a LiDAR system.

[0012] Figure 7 Shows a LiDAR system in operation.

[0013] Figure 8 Shows additional details of the operation of the LiDAR system.

[0014] Figure 9 Shows a block diagram of the relationship between the inputs and outputs of the planning module.

[0015] Figure 10 Shows a directed graph used in path planning.

[0016] Figure 11 Shows a block diagram of the inputs and outputs of the control module.

[0017] Figure 12 Shows a block diagram of the inputs, outputs, and components of the controller.

[0018] Figure 13 Illustrates a block diagram of architecture 1300 for implementing an occlusion representation of road features.

[0019] Figure 14A - 14C Illustrates an example of generating region information for implementing an occlusion representation of road features.

[0020] Figure 15 Illustrates the process of implementing an occlusion representation of road features. Detailed Description

[0021] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

[0022] In the drawings, for ease of description, a specific arrangement or order of illustrative elements (such as those representing devices, modules, instruction blocks, and data elements) is shown. However, those skilled in the art should understand that the specific order or arrangement of the illustrative elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or a separation of processing procedures. In addition, the inclusion of illustrative elements in the drawings is not intended to mean that such elements are required in all embodiments, nor does it mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0023] In addition, in the drawings, connecting elements, such as solid lines, dashed lines, or arrows, are used to illustrate the connection, relationship, or association between two or more other illustrative elements. The absence of any such connecting element is not intended to mean that a connection, relationship, or association cannot exist. In other words, the connection, relationship, or association between some elements is not shown in the drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of a signal, data, or instruction, those skilled in the art should understand that such an element represents one or more signal paths (e.g., a bus) that may be required to affect the communication.

[0024] Reference will now be made in detail to the embodiments, which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to those of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0025] Several of the features described below may be used independently of each other or in any combination with other features. However, any individual feature may not solve any of the problems discussed above, or may only solve one of the problems discussed above. Some of the problems discussed above may not be fully solved by any one feature described herein. Although headings are provided, information related to a specific heading may also be found elsewhere in this specification but not in the section with that heading. The embodiments are described herein according to the following outline:

[0026] 1. General Overview

[0027] 2. System Overview

[0028] 3. Autonomous Vehicle Architecture

[0029] 4. Autonomous Vehicle Input

[0030] 5. Autonomous Vehicle Planning

[0031] 6. Autonomous Vehicle Control

[0032] 7. Architecture for Implementing Occlusion Representations of Road Features

[0033] 8. Process for Implementing Occlusion Representations of Road Features

[0034] General Overview

[0035] A vehicle (e.g., an autonomous vehicle) can be configured to address occlusion and visibility uncertainties of critical road features (e.g., stop sign areas, intersections, travel / adjacent lanes, or crosswalks) for safe decision-making. In particular, the vehicle's perception module can provide computationally efficient occlusion information in regions of interest (e.g., priority regions near the vehicle, such as an oncoming traffic lane region or a stop region such as a stop sign area) to the vehicle's planning module for fast and safe route planning. For example, instead of providing full occlusion data (e.g., 2D or 3D data) from a dense occlusion map, the perception module can provide an effective representation of occlusion information by assigning corresponding levels of occlusion to a series of coarsely segmented road segments in the region of interest (e.g., segments of the entry lane along the traffic flow direction). Additionally, the perception module can update the occlusion information through temporal filtering to further improve the visibility for the planning module used for route planning. The perception module can also selectively and periodically update the occlusion information for the vehicle within the current region of interest, e.g., based on the current route from the planning module.

[0036] Some advantages of these techniques are as follows. First, the technique can provide only known features and exclude unknown features (e.g., not reporting the type classification of cars and cyclists, or the footprint size of occluded objects), which can increase clarity and prevent the misuse of forged attributes in downstream decision-making. Second, the technique only needs to process rough 1D occlusion information, which is more storage-efficient compared to using fully occluded drawing data (2D or 3D) and results in lower bandwidth and higher transmission rates between different modules. Third, the technique can utilize the planned route of the AV and previous map information to focus on the most important locations, making the perception module more computationally efficient. Fourth, even when the static AV is occluded by a static object, the technique can apply temporal filtering based on, for example, adjacent segments of the previously observed lane to provide valuable information. Fifth, the technique can also evaluate occlusions in adjacent lanes to prepare for lane changes. Sixth, the technique can also add estimated occupancy probabilities and expected speed ranges in occluded grid cells for safety decision-making. Finally, the technique can not only use lane segment subdivisions to report occlusions on lane / lane connection parts relevant to the planner, but also be applied to the lanes ahead for lane keeping, e.g., to cope with reduced visibility due to hills, curved roads, or bad weather.

[0037] System Overview

[0038] Figure 1 An example of an AV 100 with autonomous capabilities is shown.

[0039] As used herein, the term "autonomous capabilities" refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including but not limited to fully autonomous vehicles (AVs), highly autonomous vehicles, and conditionally autonomous vehicles.

[0040] As used herein, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0041] As used herein, "vehicle" includes means of transporting goods or people. For example, cars, buses, trains, airplanes, drones, trucks, ships, vessels, submersibles, airships, etc. A driverless car is an example of a vehicle.

[0042] As used herein, "trajectory" refers to the path or route that navigates an AV from a first spatio-temporal location to a second spatio-temporal location. In an embodiment, the first spatio-temporal location is referred to as the initial location or starting location, and the second spatio-temporal location is referred to as the destination, final location, target, target position, or target location. In some examples, a trajectory consists of one or more segments (e.g., several segments of a road), and each segment consists of one or more blocks (e.g., a lane or a portion of an intersection). In an embodiment, the spatio-temporal location corresponds to a real-world location. For example, the spatio-temporal location is a pick-up or drop-off location for picking up or dropping off a person or cargo.

[0043] As used herein, "(one or more) sensors" include one or more hardware components for detecting information related to the environment surrounding the sensor. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmission and / or reception components (e.g., laser or radio wave transmitters and receivers), electronic components (such as, analog-to-digital converters), data storage devices (such as, RAM and / or non-volatile memory), software or firmware components, and data processing components (such as, application-specific integrated circuits), microprocessors, and / or microcontrollers.

[0044] As used herein, "scene description" is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV vehicle, or one or more classified or labeled objects provided by a source external to the AV.

[0045] As used herein, "road" is a physical area that can be traversed by a vehicle and can correspond to a named passageway (e.g., a city street, an interstate highway, etc.) or can correspond to an unnamed passageway (e.g., a driveway within a house or office building, a section of a parking lot, a section of a vacant parking lot, a dirt road in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pick-up trucks, sport utility vehicles (SUVs), etc.) are capable of traversing various physical areas that are not particularly suitable for vehicle travel, "road" can be any physical area that has not been formally defined as a passageway by any municipality or other government or administrative agency.

[0046] As used herein, "lane" is the portion of a road that can be traversed by a vehicle. Lanes are sometimes identified based on lane markings. For example, a lane may correspond to most or all of the space between lane markings, or only to a portion of the space between lane markings (e.g., less than 50%). For example, a road with widely spaced lane markings may accommodate two or more vehicles such that one vehicle can pass another without crossing a lane marking, and thus may be interpreted as having lanes that are narrower than the space between the lane markings, or having two lanes between the lane markings. Lanes can also be interpreted in the absence of lane markings. For example, lanes can be defined based on the physical characteristics of the environment (e.g., rocks in a rural area and trees along an avenue, or natural obstacles to be avoided in an underdeveloped area, for example). Lanes can also be interpreted independently of lane markings or physical characteristics. For example, a lane can be interpreted based on any obstacle-free path in an area that otherwise lacks features that would be interpreted as lane boundaries. In an example scenario, an AV can interpret a lane through an obstacle-free portion of a field or open space. In another example scenario, an AV can interpret a lane through a wide (e.g., wide enough for two or more lanes) road that does not have lane markings. In this scenario, the AV can communicate lane-related information to other AVs such that the other AVs can use the same lane information to coordinate path planning between the AVs.

[0047] The term "Over-the-Air (OTA) client" includes any AV, or any electronic device (e.g., computer, controller, IoT device, electronic control unit (ECU)) that is embedded in, coupled to, or communicates with an AV.

[0048] The term "Over-the-Air (OTA) update" means any update, change, deletion, or addition to software, firmware, data, or configuration settings, or any combination thereof, that is delivered to an OTA client using proprietary and / or standardized wireless communication technologies, where the proprietary and / or standardized wireless communication technologies include, but are not limited to: cellular mobile communication (e.g., 2G, 3G, 4G, 5G), radio wireless area network (e.g., WiFi), and / or satellite Internet.

[0049] The term "edge node" refers to one or more edge devices coupled to a network that provide a portal for communicating with an AV and can communicate with other edge nodes and cloud-based computing platforms to schedule and deliver OTA updates to OTA clients.

[0050] The term "edge device" refers to a device that implements an edge node and provides a physical wireless access point (AP) to the core network of an enterprise or service provider (such as VERIZON, AT&T). Examples of edge devices include, but are not limited to: computers, controllers, transmitters, routers, routing switches, integrated access devices (IADs), multiplexers, metropolitan area network (MAN) and wide area network (WAN) access devices.

[0051] "One or more" includes functions performed by one element, functions performed by multiple elements, such as in a distributed manner, several functions performed by one element, several functions performed by several elements, or any combination of the above.

[0052] It will also be understood that, although in some cases, the terms "first", "second", etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0053] It will be understood that in various real-world embodiments, due to data modification effects, the system may require a limited amount of processing time to update data or display, and thus the concepts of "real-time" or "in real-time" as used herein are used to indicate a time frame in which two events occur simultaneously together, where it is assumed that a limited lag may occur due to the capabilities of the processor of the electronic system. In some embodiments, the lag can be on the order of less than or equal to about 1 second and can be, for example, 1 second, 0.5 second, or on the order of a few milliseconds to tens of milliseconds. However, it will be understood that these timelines are intended as examples, and in other embodiments, the lag may be greater or less than the described lag.

[0054] The terms used in the specification of the various embodiments described herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in the specification of the various embodiments described and the appended claims, the singular forms are also intended to include the plural forms unless the context clearly dictates otherwise. It will also be understood that as used herein, "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms "comprise", "comprising", "include", and / or "having" are used in this specification, it specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0055] As used herein, depending on the context, the term "if" is optionally understood to mean "when" or "at that time" or "in response to determining that" or "in response to detecting". Similarly, depending on the context, the phrase "if it has been determined" or "if [the stated condition or event] has been detected" is optionally understood to mean "at the time of determination" or "in response to determining that" or "at the time of detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

[0056] As used herein, an AV system refers to an array of AV and the hardware, software, stored data, and real-time generated data that support AV operations. In an embodiment, the AV system is incorporated within the AV. In an embodiment, the AV system is distributed across several locations. For example, some software of the AV system is implemented on a cloud computing environment similar to the cloud computing environment 200 described below with respect to Figure 2 the description.

[0057] Generally, this document describes techniques applicable to any vehicle having one or more autonomous capabilities, including full AV, highly AV, and conditional AV, such as so-called level 5, level 4, and level 3 vehicles, respectively (see SAE International Standard J3016: Taxonomy and Definitions of Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire content of which is incorporated herein by reference for more details on vehicle autonomy levels). The techniques described in this document are also applicable to partial AV and driver assistance vehicles, such as so-called level 2 and level 1 vehicles (see SAE International Standard J3016: Taxonomy and Definitions of Terms Related to On-Road Motor Vehicle Automated Driving Systems). In an embodiment, one or more level 1, level 2, level 3, level 4, and level 5 vehicle systems are capable of automatically performing certain vehicle operations (e.g., steering, braking, and using maps) under certain operating conditions based on the processing of sensor inputs. The techniques described in this document can benefit any level of vehicle within the range from fully AV to human-operated vehicles.

[0058] AVs have advantages compared to vehicles that require a human driver. One advantage is safety. For example, in 2016, the United States experienced 6 million motor vehicle accidents, 2.4 million people injured, 40,000 people killed, and 13 million vehicle collisions, with an estimated societal cost of over $910 billion. From 1965 to 2015, the number of traffic fatalities per 100 million vehicle miles traveled in the United States has decreased from approximately 6 to approximately 1, in part due to additional safety measures deployed in vehicles. For example, an extra half second of warning related to an impending collision is thought to mitigate 60% of front and rear collisions. However, passive safety features (e.g., seat belts, airbags) may have reached their limit in improving this figure. Thus, active safety measures such as the automatic control of vehicles are a possible next step in improving these statistics. Since the human driver is considered the cause of serious pre-collision events in 95% of collisions, an autonomous driving system may be able to achieve better safety results, for example, by: reliably identifying and avoiding emergencies better than a human; making better decisions than a human, complying with traffic laws better than a human, and predicting future events better than a human; and reliably controlling the vehicle better than a human.

[0059] Reference Figure 1 , the AV system 120 causes the AV 100 to operate along a trajectory 198, through an environment 190 to a destination 199 (sometimes referred to as a final location), while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and complying with road rules (e.g., operating rules or driving preferences).

[0060] In an embodiment, the AV system 120 includes means 101 for receiving and operating on an operation command from a computer processor 146. The term "operation command" is used to denote an executable instruction (or set of instructions) that causes the vehicle to perform an action (e.g., a driving maneuver). Operation commands can include, without limitation, instructions to cause the vehicle to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate, decelerate, turn left, and turn right. In an embodiment, the computing processor 146 is similar to the processor 304 described below with reference to Figure 3 Examples of means 101 include a steering controller 102, brakes 103, gear shift, an accelerator pedal or other acceleration control mechanism, windshield wipers, side door locks, window controllers, and turn indicators.

[0061] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the vehicle 100, such as the position, linear velocity and angular velocity, and linear acceleration and angular acceleration of the AV, as well as the heading (e.g., the direction of the front end of the vehicle 100). Examples of sensors 121 are GPS, an inertial measurement unit (IMU) that measures both the linear acceleration and angular rate of the vehicle, a wheel rate sensor for measuring or estimating the wheel slip rate, a wheel brake pressure or brake torque sensor, an engine torque or wheel torque sensor, and a steering angle and angular rate sensor.

[0062] In an embodiment, the sensors 121 also include sensors for sensing or measuring attributes of the AV's environment. For example, monocular or stereo cameras 122 in the visible, infrared, or thermal (or both) spectrum, LiDAR 123, RADAR, ultrasonic sensors, time-of-flight (TOF) depth sensors, rate sensors, temperature sensors, humidity sensors, and precipitation sensors.

[0063] In an embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with a computer processor 146 or data collected by the sensors 121. In an embodiment, the data storage unit 142 is similar to the ROM 308 or storage device 310 described below with respect to Figure 3 In an embodiment, the memory 144 is similar to the main memory 306 described below. In an embodiment, the data storage unit 142 and the memory 144 store historical, real-time, and / or predictive information about the environment 190. In an embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In an embodiment, data related to the environment 190 is transmitted from a remote database 134 to the vehicle 100 via a communication channel.

[0064] In an embodiment, the AV system 120 includes a communication device 140 for transmitting to the vehicle 100 attributes measured or inferred about the status and conditions of other vehicles, such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices and devices for wireless communication via point-to-point or ad hoc networks or both. In an embodiment, the communication device 140 communicates across the electromagnetic spectrum, including radio and optical communication, or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communication (and in some embodiments one or more other types of communication) is sometimes referred to as vehicle-to-everything (V2X) communication. V2X communication generally complies with one or more communication standards for communication with and between AVs.

[0065] In an embodiment, the communication device 140 includes a communication interface. For example, a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near-field, infrared, or radio interface. The communication interface transfers data from the remote database 134 to the AV system 120. In an embodiment, the remote database 134 is embedded in the cloud computing environment 200 as described in Figure 2 The communication device 140 transfers data collected from the sensors 121 or other data related to the operation of the vehicle 100 to the remote database 134. In an embodiment, the communication device 140 transmits information related to teleoperation to the vehicle 100. In some embodiments, the vehicle 100 communicates with other remote (e.g., "cloud") servers 136.

[0066] In an embodiment, the remote database 134 also stores and transmits digital data (e.g., stores data such as road and street locations). This data is stored in the memory 144 on the vehicle 100 or transferred from the remote database 134 to the vehicle 100 via a communication channel.

[0067] In an embodiment, the remote database 134 stores and transmits historical information (e.g., speed and acceleration profiles) related to the driving attributes of vehicles that previously traveled along the trajectory 198 at similar times of day. In one implementation, such data can be stored in the memory 144 on the vehicle 100 or transferred from the remote database 134 to the vehicle 100 via a communication channel.

[0068] The computer processor 146 located on the vehicle 100 algorithmically generates control actions based on both real-time sensor data and prior information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0069] In an embodiment, the AV system 120 includes computer peripherals 132 coupled to a computing processor 146 for providing information and alerts to a user (e.g., an occupant or a remote user) of the vehicle 100 and receiving input from the user. In an embodiment, the peripherals 132 are similar to the display 312, input device 314, and cursor controller 316 discussed below with reference to Figure 3 discussion. The coupling is wireless or wired. Any two or more of the interface devices can be integrated into a single device.

[0070] In an embodiment, the AV system 120 receives and enforces a privacy level of an occupant specified, for example, by the occupant or stored in a profile associated with the occupant. The privacy level of the occupant determines how use of specific information associated with the occupant (e.g., occupant comfort data, biometric data, etc.) stored in the occupant profile and / or stored on the cloud server 136 and associated with the occupant profile is permitted. In an embodiment, the privacy level specifies specific information associated with the occupant that is to be deleted once the ride is completed. In an embodiment, the privacy level specifies specific information associated with the occupant and identifies one or more entities authorized to access the information. Examples of the specified entities authorized to access the information can include other AVs, third-party AV systems, or any entity that can potentially access the information.

[0071] The privacy level of the occupant can be specified at one or more granularity levels. In an embodiment, the privacy level identifies specific information to be stored or shared. In an embodiment, the privacy level applies to all information associated with the occupant such that the occupant can specify not to store or share her personal information. The specification of the entities authorized to access specific information can also be specified at various granularity levels. The various sets of entities authorized to access specific information can include, for example, other AVs, the cloud server 136, specific third-party AV systems, etc.

[0072] In an embodiment, the AV system 120 or the cloud server 136 determines whether the AV 100 or another entity can access certain information associated with the occupant. For example, a third-party AV system attempting to access occupant input related to a specific spatio-temporal location must obtain authorization, for example, from the AV system 120 or the cloud server 136 to access information associated with the occupant. For example, the AV system 120 uses the specified privacy level of the occupant to determine whether occupant input related to a spatio-temporal location can be presented to a third-party AV system, the AV 100, or another AV. This enables the privacy level of the occupant to specify which other entities are permitted to receive data related to the actions of the occupant or other data associated with the occupant.

[0073] Figure 2Shows an example "cloud" computing environment. Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services). In a typical cloud computing system, one or more large cloud data centers house the machines for delivering the services provided by the cloud. Now refer to Figure 2 , the cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by a cloud 202. The data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to the cloud 202.

[0074] The cloud computing environment 200 includes one or more cloud data centers. Generally speaking, a cloud data center (such as Figure 2 the cloud data center 204a shown in Figure 2 is the physical arrangement of the servers that make up the cloud (such as Figure 3 the cloud 202 shown in or a particular part of the cloud). For example, the servers are physically arranged in rooms, groups, rows, and racks in the cloud data center. A cloud data center has one or more zones, which include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, the servers in the zones, rooms, racks, and / or rows are arranged into several groups based on the physical infrastructure requirements of the data center facilities (including power, energy, heat, heat sources, and / or other requirements). In an embodiment, the server nodes are similar to Figure 3 the computer systems described in. The data center 204a has many computing systems distributed across multiple racks.

[0075] The cloud 202 includes cloud data centers 204a, 204b, and 204c and the networks and network resources (such as network devices, nodes, routers, switches, and network cables) for connecting the cloud data centers 204a, 204b, and 204c and facilitating access to cloud computing services by the computing systems 206a - f. In an embodiment, the network represents any combination of one or more local networks, wide area networks, or internets coupled by wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network is transmitted using a variety of network layer protocols (such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, etc.). Additionally, in embodiments where the network represents a combination of multiple subnets, different network layer protocols are used on each underlying subnet. In some embodiments, the network represents one or more internets (such as the public Internet, etc.).

[0076] The computing systems 206a-f or cloud computing service consumers are connected to the cloud 202 via a network link and a network adapter. In an embodiment, the computing systems 206a-f are implemented as various computer processors, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, AVs (including cars, drones, shuttles, trains, buses, etc.), and consumer electronics. In an embodiment, the computing systems 206a-f are implemented in or as part of other systems.

[0077] Figure 3 FIG. 300 shows a computer system. In an implementation, the computer system 300 is a special-purpose computer processor. The special-purpose computer processor is hard-wired to perform these techniques, or includes digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are persistently programmed to perform the above techniques, or can include one or more general-purpose hardware processors that are programmed to perform these techniques according to program instructions in firmware, memory, other memory, or a combination. Such a special-purpose computer processor can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish these techniques. In various embodiments, the special-purpose computer processor is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that includes hard-wired and / or program logic to implement these techniques.

[0078] In an embodiment, the computer system 300 includes a bus 302 or other communication mechanism for conveying information, and a processor 304 coupled to the bus 302 for processing information. The processor 304 is, for example, a general-purpose microprocessor. The computer system 300 also includes a main memory 306, such as random access memory (RAM) or other dynamic storage device, coupled to the bus 302 to store information and instructions that are executed by the processor 304. In one implementation, the main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 304. When these instructions are stored in a non-transitory storage medium accessible to the processor 304, the computer system 300 becomes a special-purpose machine that is customized to perform the operations specified in the instructions.

[0079] In an embodiment, the computer system 300 also includes a read-only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions for the processor 304. A storage device 310, such as a magnetic disk, an optical disk, a solid-state drive, or a three-dimensional cross-point memory, is provided and coupled to the bus 302 to store information and instructions.

[0080] In an embodiment, computer system 300 is coupled via bus 302 to a display 312, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light emitting diode (LED) display, or organic light emitting diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is a cursor control 316, such as a mouse, trackball, touch display, or cursor direction keys, for communicating direction information and command selections to processor 304 and for controlling cursor movement on display 312. Such input devices typically have two degrees of freedom in two axes (a first axis (e.g., the x-axis) and a second axis (e.g., the y-axis)), which allow the device to specify a position on a plane.

[0081] According to one embodiment, the techniques described herein are performed by computer system 300 in response to one or more sequences of one or more instructions contained in main memory 306 being executed by processor 304. These instructions are read into main memory 306 from another storage medium, such as storage device 310. Execution of the instruction sequences contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hardwired circuitry is used in place of, or in combination with, software instructions.

[0082] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific fashion. Such storage medium includes non-volatile and / or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, solid state drives, or three-dimensional cross-point memory such as storage device 310. Volatile media includes dynamic memory, such as main memory 306. Common forms of storage medium include, for example, floppy disk, flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, RAM, PROM, and EPROM, FLASH-EPROM, NV-RAM, or any other memory chip or cartridge.

[0083] Storage media is distinct from, but can be used in conjunction with, transmission media. Transmission media participates in the transfer of information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or light waves, such as acoustic or light waves generated during radio frequency (RF) and in infrared data communications.

[0084] In an embodiment, various forms of media are involved in carrying one or more sequences of one or more instructions to the processor 304 for execution. For example, these instructions are initially executed on a disk or solid state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions using a modem over a telephone line. The local modem of the computer system 300 receives the data on the telephone line and uses an infrared transmitter to convert the data into an infrared signal. The infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on the bus 302. The bus 302 carries the data to the main memory 306, and the processor 304 retrieves and executes the instructions from the main memory 306. The instructions received by the main memory 306 can optionally be stored on the storage device 310 before or after being executed by the processor 304.

[0085] The computer system 300 also includes a communication interface 318 coupled to the bus 302. The communication interface 318 provides two-way data communication coupled to a network link 320 connected to a local network 322. For example, the communication interface 318 is an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a corresponding type of telephone line. As another example, the communication interface 318 is a local area network (LAN) card for providing a data communication connection with a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, the communication interface 318 sends and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0086] The network link 320 typically provides data communication to other data devices through one or more networks. For example, the network link 320 provides a connection to a main computer 324 or to a cloud data center or device operated by an Internet service provider (ISP) 326 through the local network 322. The ISP 326 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the “Internet” 328. Both the local network 322 and the Internet 328 use electrical, electromagnetic, or optical signals carrying digital data streams. The signals through the various networks and the signals on the network link 320 and through the communication interface 318 are example forms of transmission media, where these signals carry digital data into and out of the computer system 300. In an embodiment, the network 320 includes the aforementioned cloud 202 or a portion of the cloud 202.

[0087] Computer system 300 sends messages and receives data including program code via one or more networks, network link 320, and communication interface 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt, and / or stored in storage device 310, or stored in other non-volatile storage devices for later execution.

[0088] Autonomous vehicle architecture

[0089] Figure 4 Illustrates an example architecture 400 for an AV (e.g., Figure 1 the vehicle 100 shown). Architecture 400 includes a perception module 402 (sometimes referred to as perception circuitry), a planning module 404 (sometimes referred to as planning circuitry), a control module 406 (sometimes referred to as control circuitry), a localization module 408 (sometimes referred to as localization circuitry), and a database module 410 (sometimes referred to as database circuitry). Each module plays a role in the operation of vehicle 100. Collectively, modules 402, 404, 406, 408, and 410 can be Figure 1 part of the AV system 120 shown. In some embodiments, any of modules 402, 404, 406, 408, and 410 is a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application-specific integrated circuits [ASICs], hardware memory devices, other types of integrated circuits, other types of computer hardware, or any or all combinations of these hardware components). Modules 402, 404, 406, 408, and 410 are each sometimes referred to as processing circuitry (e.g., computer hardware, computer software, or a combination of the two). Any or all combinations of modules 402, 404, 406, 408, and 410 are also examples of processing circuitry.

[0090] In use, the planning module 404 receives data representing a destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the vehicle 100 can travel to reach (e.g., arrive at) the destination 412. To enable the planning module 404 to determine the data representing the trajectory 414, the planning module 404 receives data from the perception module 402, the localization module 408, and the database module 410.

[0091] The perception module 402 uses one or more sensors 121, also as shown, for example, Figure 1 to identify nearby physical objects. The objects are classified (e.g., grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.), and a scene description including the classified objects 416 is provided to the planning module 404.

[0092] The planning module 404 also receives data representing the AV position 418 from the positioning module 408. The positioning module 408 determines the AV position by using data from the sensor 121 and data from the database module 410 (e.g., geographic data) to calculate the position. For example, the positioning module 408 uses data from a GNSS (Global Navigation Satellite System) sensor and geographic data to calculate the longitude and latitude of the AV. In an embodiment, the data used by the positioning module 408 includes a high-precision map with lane geometric attributes, a map describing the road network connection attributes, a map describing the physical attributes of the lanes (such as traffic rate, traffic volume, the number of vehicles and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination thereof), and a map describing the spatial locations of road features (such as crosswalks, traffic signs, or various other driving signals, etc.). In an embodiment, the high-precision map is constructed by adding data to a low-precision map via automatic or manual annotation.

[0093] The control module 406 receives data representing the trajectory 414 and data representing the AV position 418, and operates the control functions 420a - 420c (e.g., steering, throttle, braking, ignition) of the AV in a manner that will cause the vehicle 100 to follow the trajectory 414 to the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420a - 420c as follows: the steering angle of the steering function will cause the vehicle 100 to turn left, and the throttle and brakes will cause the vehicle 100 to pause and wait for passing pedestrians or vehicles before making the turn.

[0094] Autonomous Vehicle Input

[0095] Figure 5 Shows examples of the inputs 502a - 502d (e.g., Figure 4 ) used by the perception module 402 ( Figure 1 such as the sensor 121 shown in Figure 1 ) and the outputs 504a - 504d (e.g., sensor data). One input 502a is a LiDAR (Light Detection and Ranging) system (e.g.,

[0096] Figure 1 the LiDAR 123 shown in

[0096] ). LiDAR is a technology that uses light (e.g., a beam of light such as infrared light) to obtain data related to physical objects in its line of sight. The LiDAR system produces LiDAR data as the output 504a. For example, the LiDAR data is a collection of 3D or 2D points (also called a point cloud) used to construct a representation of the environment 190.Another input 502b is a RADAR system. RADAR is a technology that uses radio waves to obtain data related to nearby physical objects. RADAR can obtain data related to objects that are not within the line of sight of the LiDAR system. The RADAR system generates RADAR data as output 504b. For example, the RADAR data is one or more radio frequency electromagnetic signals for constructing a representation of the environment 190.

[0097] Another input 502c is a camera system. The camera system uses one or more cameras (e.g., a digital camera using an optical sensor such as a charge-coupled device [CCD]) to obtain information related to nearby physical objects. The camera system generates camera data as output 504c. The camera data typically takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, for example, for the purpose of stereoscopic imaging (stereo vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as "nearby", this is relative to the AV. In some embodiments, the camera system is configured to "see" distant objects (e.g., up to 1 kilometer or more in front of the AV). Thus, in some embodiments, the camera system has features such as sensors and lenses optimized for perceiving distant objects.

[0098] Another input 502d is a traffic light detection (TLD) system. The TLD system uses one or more cameras to obtain information related to traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system generates TLD data as output 504d. The TLD data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). The TLD system differs from a system that includes a camera in that: the TLD system uses a camera with a wide field of view (e.g., using a wide-angle lens or a fish-eye lens) to obtain information related to as many physical objects that provide visual navigation information as possible, so that the vehicle 100 can access all relevant navigation information provided by these objects. For example, the viewing angle of the TLD system can be about 120 degrees or greater.

[0099] In some embodiments, sensor fusion techniques are used to combine the outputs 504a - 504d. Thus, the individual outputs 504a - 504d are provided to other systems of the vehicle 100 (e.g., provided to as Figure 4The planning module 404 shown), or the combined output can be provided to other systems in the form of a single combined output or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs or both) or a single combined output or multiple combined outputs of different types (e.g., using different respective combination techniques or combining different respective outputs or both). In some embodiments, early fusion techniques are used. Early fusion techniques are characterized by combining the outputs before applying one or more data processing steps to the combined output. In some embodiments, late fusion techniques are used. Late fusion techniques are characterized by combining the outputs after applying one or more data processing steps to the individual outputs.

[0100] Figure 6 An example of a LiDAR system 602 is shown (e.g., Figure 5 the input 502a shown). The LiDAR system 602 emits light 604a - 604c from a light emitter 606 (e.g., a laser emitter). The light emitted by the LiDAR system is typically not in the visible spectrum; for example, infrared light is often used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and is reflected back to the LiDAR system 602. (The light emitted from the LiDAR system generally does not penetrate physical objects, e.g., physical objects in solid form.) The LiDAR system 602 also has one or more light detectors 610 for detecting the reflected light. In an embodiment, one or more data processing systems associated with the LiDAR system generate an image 612 representing the field of view 614 of the LiDAR system. The image 612 includes information representing the boundary 616 of the physical object 608. Thus, the image 612 is used to determine the boundary 616 of one or more physical objects near the AV.

[0101] Figure 7 An operating LiDAR system 602 is shown. In the scenario shown in this figure, the vehicle 100 receives both the camera system output 504c in the form of an image 702 and the LiDAR system output 504a in the form of LiDAR data points 704. In use, the data processing system of the vehicle 100 compares the image 702 with the data points 704. In particular, the physical object 706 identified in the image 702 is also identified in the data points 704. Thus, the vehicle 100 perceives the boundary of the physical object based on the contour and density of the data points 704.

[0102] Figure 8 Additional details of the operation of the LiDAR system 602 are shown. As described above, the vehicle 100 detects the boundary of the physical object based on the characteristics of the data points detected by the LiDAR system 602. As Figure 8As shown, a flat object such as the ground 802 will reflect the light 804a - 804d emitted from the LiDAR system 602 in a consistent manner. In other words, since the LiDAR system 602 emits light at a consistent interval, the ground 802 will reflect the light back to the LiDAR system 602 at the same consistent interval. When the vehicle 100 is traveling on the ground 802 and there is nothing blocking the road, the LiDAR system 602 will continue to detect the light reflected by the next valid ground point 806. However, if an object 808 blocks the road, the light 804e - 804f emitted by the LiDAR system 602 will be reflected from points 810a - 810b in a manner inconsistent with the expected consistent manner. Based on this information, the vehicle 100 can determine that there is an object 808.

[0103] Autonomous Vehicle Planning

[0104] Figure 9 Illustrated (e.g., as Figure 4 shown) is a block diagram 900 showing the relationship between the inputs and outputs of the planning module 404. Generally speaking, the output of the planning module 404 is a route 902 from a starting point 904 (e.g., a source location or an initial location) to an ending point 906 (e.g., a destination or a final location). The route 902 is typically defined by one or more segments. For example, a segment refers to a distance to be traveled on at least a portion of a street, road, highway, lane, or other physical area suitable for automotive travel. In some examples, e.g., if the vehicle 100 is an off - road capable vehicle such as a four - wheel drive (4WD) or all - wheel drive (AWD) car, SUV, or pickup truck, the route 902 includes "off - road" segments such as unpaved paths or open fields.

[0105] In addition to the route 902, the planning module also outputs lane - level route planning data 908. The lane - level route planning data 908 is used to travel through the segments of the route 902 based on the conditions of the segments at a particular time. For example, if the route 902 includes a multi - lane highway, the lane - level route planning data 908 includes trajectory planning data 910, where the vehicle 100 can use this trajectory planning data 910 to select a lane from the multiple lanes, e.g., based on whether an exit is approaching, whether there are other vehicles in one or more of the multiple lanes, or other factors that change over a period of minutes or less. Similarly, in some implementations, the lane - level route planning data 908 includes speed constraints 912 specific to a segment of the route 902. For example, if the segment includes pedestrians or unexpected traffic, the speed constraint 912 can limit the vehicle 100 to a slower travel speed than expected, e.g., a speed based on the speed limit data for that segment.

[0106] In an embodiment, the inputs to the planning module 404 include, for example, database data 914 (from the database module 410 as shown in Figure 4 ), current location data 916 (e.g., the AV location 418 as shown in Figure 4 ), destination data 918 (e.g., for the destination 412 as shown in Figure 4 ) and object data 920 (e.g., the classified objects 416 sensed by the sensing module 402 as shown in Figure 4 ). In some embodiments, the database data 914 includes rules used during planning. The rules are specified using a formal language (e.g., using Boolean logic). In any given situation encountered by the vehicle 100, at least some of these rules will apply to that situation. If a rule has a condition that is satisfied based on information available to the vehicle 100 (e.g., information related to the surrounding environment), then the rule applies to the given situation. The rules can have priorities. For example, the rule "if the road is a highway, move to the leftmost lane" may have a lower priority compared to the rule "if an exit is approaching within one mile, move to the rightmost lane".

[0107] Figure 10 FIG. shows a directed graph 1000 used in path planning (e.g., by the planning module 404 ( Figure 4 )). In general, a directed graph 1000 such as the one shown in Figure 10 is used to determine a path between any starting point 1002 and ending point 1004. In the real world, the distance separating the starting point 1002 and the ending point 1004 may be relatively large (e.g., in two different metropolitan areas), or may be relatively small (e.g., two intersections in adjacent city blocks or two lanes on a multi-lane road).

[0108] In an embodiment, a directed graph 1000 has nodes 1006a - 1006d representing different locations that a vehicle 100 might occupy between a starting point 1002 and an ending point 1004. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different urban areas, the nodes 1006a - 1006d represent sections of a road. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different locations on the same road, the nodes 1006a - 1006d represent different positions on that road. Thus, the directed graph 1000 includes information at different levels of granularity. In an embodiment, a directed graph with a high granularity is also a sub - graph of another directed graph with a larger scale. For example, most of the information in a directed graph where the starting point 1002 and the ending point 1004 are far apart (e.g., many miles apart) is at a low granularity, and the directed graph is based on stored data, but the directed graph also includes some high - granularity information for a portion of the physical locations represented in the field of view of the vehicle 100 in that directed graph.

[0109] The nodes 1006a - 1006d are different from objects 1008a - 1008b that cannot overlap with the nodes. In an embodiment, at a low granularity, the objects 1008a - 1008b represent areas that a car cannot pass through, such as areas without streets or roads. At a high granularity, the objects 1008a - 1008b represent physical objects in the field of view of the vehicle 100, such as other cars, pedestrians, or other entities with which the vehicle 100 cannot share physical space. In an embodiment, some or all of the objects 1008a - 1008b are static objects (e.g., objects that do not change position, such as streetlights or utility poles) or dynamic objects (e.g., objects that can change position, such as pedestrians or other cars).

[0110] Nodes 1006a - 1006d are connected by edges 1010a - 1010c. If two nodes 1006a - 1006b are connected by an edge 1010a, the vehicle 100 can travel between one node 1006a and another node 1006b, for example, without having to travel to an intermediate node before reaching the other node 1006b. (When referring to the vehicle 100 traveling between nodes, it means the vehicle 100 travels between two physical locations represented by the corresponding nodes.) The edges 1010a - 1010c are generally two - way, in the sense that the vehicle 100 can travel from the first node to the second node, or from the second node to the first node. In an embodiment, the edges 1010a - 1010c are one - way, in the sense that the vehicle 100 can travel from the first node to the second node, yet the vehicle 100 cannot travel from the second node to the first node. The edges 1010a - 1010c are one - way in cases where the edges 1010a - 1010c represent, for example, one - way streets, individual lanes of a street, road, or highway, or other features that can only be traversed in one direction due to legal or physical constraints.

[0111] In an embodiment, the planning module 404 uses the directed graph 1000 to identify a path 1012 composed of nodes and edges between the starting point 1002 and the ending point 1004.

[0112] The edges 1010a - 1010c have associated costs 1014a - 1014b. The costs 1014a - 1014b are values representing the resources that will be spent if the vehicle 100 selects that edge. A typical resource is time. For example, if the physical distance represented by one edge 1010a is twice the physical distance represented by another edge 1010b, the associated cost 1014a of the first edge 1010a can be twice the associated cost 1014b of the second edge 1010b. Other factors affecting time include expected traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. Two edges 1010a - 1010b can represent the same physical distance, but, for example, due to road conditions, expected weather, etc., one edge 1010a requires more fuel compared to the other edge 1010b.

[0113] When the planning module 404 identifies a path 1012 between the starting point 1002 and the ending point 1004, the planning module 404 generally selects a path optimized for cost, for example, a path with the minimum total cost when adding together the individual costs of the edges.

[0114] Autonomous Vehicle Control

[0115] Figure 11 Shown (e.g., as Figure 4Block diagram 1100 of the inputs and outputs of control module 406 (shown). The control module operates according to controller 1102, which for example includes: one or more processors similar to processor 304 (e.g., one or more computer processors such as a microprocessor or a microcontroller or both); short-term and / or long-term data storage devices similar to main memory 306, ROM 308, and storage device 310 (e.g., memory, random access memory, or flash memory or both); and instructions stored in the memory that, when executed (e.g., by one or more processors), perform the operations of controller 1102.

[0116] In an embodiment, controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes speed, such as rate and heading. The desired output 1104 can for example be based on data received from planning module 404 (shown). According to the desired output 1104, controller 1102 generates data that can be used as throttle input 1106 and steering input 1108. The throttle input 1106 represents, for example, the magnitude of engaging the throttle (e.g., acceleration control) of vehicle 100 by engaging the throttle pedal or engaging another throttle control to achieve the desired output 1104. In some examples, the throttle input 1106 also includes data that can be used to engage the brakes of vehicle 100 (e.g., deceleration control). The steering input 1108 represents the steering angle at which the steering control of the AV (e.g., the steering wheel, the steering angle actuator, or other functions for controlling the steering angle) should be positioned to achieve the desired output 1104. Figure 4 In an embodiment, controller 1102 receives feedback used in adjusting the inputs provided to the throttle and steering. For example, if vehicle 100 encounters a disturbance 1110 such as a hill, the measured rate 1112 of vehicle 100 drops below the desired output rate. In an embodiment, any measured output 1114 is provided to controller 1102 such that, for example, the required adjustment is made based on the difference 1113 between the measured rate and the desired output. The measured output 1114 includes the measured position 1116, the measured speed 1118 (including rate and heading), the measured acceleration 1120, and other outputs measurable by the sensors of vehicle 100.

[0117] In an embodiment, controller 1102 receives feedback used in adjusting the inputs provided to the throttle and steering. For example, if vehicle 100 encounters a disturbance 1110 such as a hill, the measured rate 1112 of vehicle 100 drops below the desired output rate. In an embodiment, any measured output 1114 is provided to controller 1102 such that, for example, the required adjustment is made based on the difference 1113 between the measured rate and the desired output. The measured output 1114 includes the measured position 1116, the measured speed 1118 (including rate and heading), the measured acceleration 1120, and other outputs measurable by the sensors of vehicle 100.

[0118] In an embodiment, information related to interference 1110 is pre-detected by a sensor such as a camera or a LiDAR sensor, for example, and this information is provided to the predictive feedback module 1122. Then, the predictive feedback module 1122 provides information to the controller 1102 that the controller 1102 can use to adjust accordingly. For example, if the sensors of the vehicle 100 detect (“see”) a hill, the controller 1102 can use this information to prepare to engage the throttle at an appropriate time to avoid significant deceleration.

[0119] Figure 12 Block diagram 1200 showing the inputs, outputs, and components of the controller 1102. The controller 1102 has a rate analyzer 1202 that affects the operation of the throttle / brake controller 1204. For example, the rate analyzer 1202 instructs the throttle / brake controller 1204 to accelerate or decelerate using the throttle / brake 1206 based on feedback received by the controller 1102 and processed by the rate analyzer 1202.

[0120] The controller 1102 also has a lateral tracking controller 1208 that affects the operation of the steering wheel controller 1210. For example, the lateral tracking controller 1208 instructs the steering wheel controller 1210 to adjust the position of the steering angle actuator 1212 based on feedback received by the controller 1102 and processed by the lateral tracking controller 1208.

[0121] The controller 1102 receives several inputs for determining how to control the throttle / brake 1206 and the steering angle actuator 1212. The planning module 404 provides information that the controller 1102 uses, for example, to select the heading when the vehicle 100 starts operating and to determine which road segment to cross when the vehicle 100 reaches an intersection. The positioning module 408 provides information describing the current location of the vehicle 100 to the controller 1102, for example, so that the controller 1102 can determine whether the vehicle 100 is at the location expected based on the way the throttle / brake 1206 and the steering angle actuator 1212 are being controlled. In an embodiment, the controller 1102 receives information from other inputs 1214, such as information received from a database, a computer network, etc.

[0122] Architecture for implementing an occluded representation of road features

[0123] Figure 13 Block diagram illustrating an architecture 1300 for implementing an occluded representation of road features during the operation of a vehicle according to one or more embodiments. In an embodiment, the architecture 1300 is implemented in a vehicle system of the vehicle. In some examples, the vehicle is Figure 1 the illustrated embodiment of the AV 100. In some examples, the vehicle system isFigure 1 An embodiment of the AV system 120 shown. The architecture 1300 provides an efficient way to represent surrounding occlusion information in the environment of a vehicle (e.g., Figure 1 the environment 190 shown), enabling the vehicle system to reason about occlusions and uncertainties in the environment for safety decision-making. For example, as illustrated in further details below, the architecture 1300 presents a lightweight computationally efficient occlusion interface, similar to a rough 1D visibility grid subdivision introducing cross-traffic lanes / lane connection sections at intersections surrounded by priority areas and / or an independent report of visibility at stop areas, for multi-way stop intersection priority determination.

[0124] Referring Figure 13 to Figure 4 the perception module 402 shown), and a planning module 1320 (e.g., Figure 4 the planning module 404 shown). The perception module 1310 obtains area information of at least one region of interest of the vehicle based on previous mapping data 1308 and identifies occlusion data 1306 associated with the at least one region of interest from the occlusion map 1304. The perception module 1310 then determines occlusion information 1315 associated with the at least one region of interest based on the area information and the occlusion data 1306. The occlusion information 1315 has a smaller data size than the occlusion data 1306. The perception module 1310 provides the occlusion information 1315 to the planning module 1320. The planning module 1320 plans the route 1325 of the vehicle based on the occlusion information 1315 and other data (e.g., from Figure 4 the positioning module 408 and the database module 410). The vehicle is operated by a control module (e.g., Figure 4 the control module 406 shown) according to the planned route 1325. The planning module 1320 provides the planned route 1325 back to the perception module 1310 for updating the regions of interest of the vehicle.

[0125] The architecture 1300 includes a sensor system 1302 for sensing or measuring properties of the vehicle environment (e.g., Figure 1 or Figure 4 the sensor 121 shown). In one embodiment, the sensor system 1302 includes a LiDAR system 1302a (e.g., Figure 5 the LiDAR system 502a shown), a RADAR system 1302b (e.g., Figure 5 the RADAR system 502b shown), and a camera system 1302c (e.g., Figure 5The camera system 502c) shown. The sensor system 1302 (e.g., at least one sensor in the sensor system 1302) identifies nearby physical objects, e.g., objects associated with (e.g., represented by) the data generated by at least one sensor in the sensor system 1302. In some examples, the objects include visible objects within the line of sight perceived by, for example, the camera system 1302c or the LiDAR system 1302a. In some examples, the objects include invisible objects outside the line of sight perceived by, for example, the RADAR system 1302b. In some examples, the objects are classified into types such as pedestrians, bicycles, cars, traffic signs (e.g., Figure 4 the classified object 416 shown). The outputs of the sensor system 1302 (e.g., Figure 5 the outputs 504a, 504b, 504c shown) are combined into an occlusion map 1304 of the vehicle environment using sensor fusion techniques by the vehicle. In some examples, the occlusion map 1304 is represented by 2D or 3D points, lines, or shapes and scene descriptions.

[0126] In one embodiment, the vehicle system maintains the occlusion map 1304 by accessing a database including road network information or using the outputs from the sensor system 1302 or both. The occlusion map 1304 includes known (visible or invisible) objects sensed or otherwise known by the vehicle's sensor system 1302.

[0127] In one embodiment, the occlusion map 1304 includes hypothesized (or unknown) objects that cannot be sensed by the sensor system 1302. In some examples, the vehicle system generates hypothesized objects by assuming the existence or properties of possible unknown objects in the unperceived environment based on various factors and methods. Hypothesized objects include moving objects or objects using a driving path from which the vehicle is excluded, or both. For example, hypothesized objects include at least one of the following: a second vehicle, a bicycle, a bus, a train, a pedestrian, and an animal.

[0128] In an example, the occlusion map 1304 includes at least one occluded object. In one embodiment, the occluded object in the occlusion map 1304 is an invisible but known object sensed by the sensor system 1302. In one embodiment, the occluded object is a hypothesized object. The vehicle system stores occlusion data of the occluded object in the occlusion map 1304. In some examples, the occlusion data includes a representation element of the occluded object and an attribute of the occluded object. The representation element is determined based on the type of the occluded object. The attribute includes size or motion state or both. The motion state is a stationary condition, a moving condition, a speed, a moving direction, or a combination of two or more of them. For example, a vehicle as an occluded object is in a stationary condition when the vehicle is not started and parked on a road, and is in a moving condition when the vehicle starts and moves on the road at a speed along the moving direction. The speed is set to be less than or equal to a predetermined maximum value. The predetermined maximum value includes a speed limit. In some cases, the predetermined maximum value includes a quantity derived from other objects observed simultaneously or previously in the environment. The predetermined maximum value is a quantity derived based on historical data, road configuration, traffic rules, events, time, weather conditions, or a combination of two or more of them.

[0129] The perception module 1310 includes a map information extractor 1312 that is configured to obtain, for example, region information of at least one region of interest of the vehicle from previous drawing data 1308 in a drawing database. In one embodiment, the drawing database is implemented in the Figure 4 database module 410 shown. The previous drawing data 1308 includes road network information, such as a high-precision map of lane geometric attributes, a map describing road network connection attributes, a map describing lane physical attributes (such as traffic speed, traffic volume, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination thereof), and a map describing the spatial locations of road features (such as crosswalks, traffic signs, or other types of driving signals). In an embodiment, the high-precision map is constructed by adding data to a low-precision map via automatic or manual annotation.

[0130] In one embodiment, the perception module 1310 determines at least one region of interest of the vehicle as at least one stop region associated with a stop line (or stop sign) on the current route of the vehicle. In some examples, such as when stopping at a multi-way stop intersection, the stop region is occluded. The stop region is represented by a shape (e.g., a rectangle) in the mapping data 1308. In some examples, the length of the stop region is 2 to 3 meters. The map information extractor 1312 extracts region information of the stop region from the mapping data 1308. In some examples, the region information of the stop region includes an identifier (ID) of the stop region in the mapping data 1308. In one embodiment, the region of interest includes at least one of the following: a crosswalk, a vehicle lane region immediately behind a steep hill, or a lane region adjacent to the vehicle's travel lane.

[0131] In one embodiment, the perception module 1310 determines at least one region of interest of the vehicle as at least one priority region (e.g., an enclosing region introducing a lane region) associated with an intersection along the current route of the vehicle. For example, the extension of the priority region or the priority regions along the direction of the corresponding lane overlaps with the lanes in the intersection along the current route of the vehicle or any potential route of the vehicle. In some examples, the priority region is occluded or includes an invisible agent that takes precedence over the vehicle when it attempts an unprotected turn through the intersection. In some examples, the priority region is represented by a shape (e.g., a rectangle) in the mapping data 1308. In some examples, the length of the priority region exceeds a predetermined distance, such as 10 meters, 20 meters, 30 meters, 40 meters, 50 meters, or 60 meters, depending on the spacing requirements for complying with road rules. The map information extractor 1312 extracts region information of the priority region from the mapping data 1308. In some examples, the region information of the priority region includes at least one of a region identifier of the priority region in the mapping data 1308, an identifier of the corresponding lane, or an identifier of the intersection.

[0132] In one embodiment, as Figure 14B illustrated by further details in Figure 14CAs illustrated in further detail below, the priority region is represented by a plurality of lane segments (e.g., from the front to the rear of the priority region along the lane direction) sorted sequentially along at least one lane direction. Each lane segment includes at least one discrete reference path point. Each lane segment has a corresponding identifier. Each lane segment is represented by a shape (e.g., a polygon). In one embodiment, the priority region corresponds to a plurality of lanes along the same lane direction. Each lane in the plurality of lanes is divided into a corresponding plurality of lane segments each having a corresponding identifier. In one embodiment, a graph structure or a tree structure is used to store each lane or each lane connection part or both within the priority region.

[0133] In one embodiment, the region information of the priority region includes discrete reference path points or segmented road segments or both. The region information further includes information on discrete reference path points and segmented lane segments, e.g., the length of each lane segment or the distance between adjacent reference path points. In one embodiment, the region information is included in the previous drawing data 1308, e.g., by adding a discrete layer including segmented lanes or discrete reference path points to the priority region in a drawing database. The perception module 1310 directly obtains the region information from the previous drawing data 1308. In one embodiment, e.g., at the beginning of the process of implementing the occlusion representation, the perception module 1310 generates the region information of the priority region based on the drawing data of the priority region from the previous drawing data 1308.

[0134] In one embodiment, the perception module 1310 generates the region information of the priority region by a brute - force algorithm to analyze all lane points within the priority region. In one embodiment, as Figure 14A - 14C illustrated in further detail below, the perception module 1310 generates the region information of the priority region by first determining the intersection point between the current route of the vehicle and the priority region or the extension of the priority region along the lane direction, and then determining, starting from this intersection point, a series of discrete reference path points or segmented road segments or both within the priority region along the direction opposite to the lane direction.

[0135] The map information extractor 1312 also receives the planned route 1325 of the vehicle from the planning module 1320. The perception module 1310 updates at least one region of interest of the vehicle based on the planned route 1325. In one embodiment, the perception module 1310 periodically queries a plurality of regions of interest within a predetermined distance from the vehicle, and based on periodically querying the plurality of regions of interest near the vehicle, periodically provides occlusion information associated with the plurality of regions of interest to the planning module 1320. In one embodiment, the perception module 1310 filters out a first plurality of regions of interest from the plurality of regions of interest. The paths extending along the respective lane directions from each region of interest in the first plurality of regions of interest intersect with paths different from the current route of the vehicle. That is, each region of interest in the first plurality of regions of interest is in a direction away from the current route. The perception module 1310 publishes occlusion information associated with a second plurality of regions of interest to the planning module 1320. The second plurality of regions of interest are regions of interest that are included in the plurality of regions of interest and are different from the first plurality of regions of interest.

[0136] The perception module 1310 includes an occlusion information generator 1314 configured to generate occlusion information. After determining at least one region of interest of the vehicle (e.g., at least one priority region or at least one stop region or both), the perception module 1310 (e.g., the occlusion information generator 1314) identifies occlusion data 1306 associated with the at least one region of interest from the occlusion map 1304. In one embodiment, the perception module 1310 first identifies at least one region of interest in the occlusion map 1304, and then determines the occlusion data 1306 associated with the at least one region of interest in the occlusion map 1304. The occlusion data 1306 includes occlusion data of at least one occluded object (e.g., an invisible object) in the at least one region of interest. The occlusion data of the at least one occluded object includes the attributes of the occluded object. The representation element is associated with the type of the occluded object. The attributes include size or motion state or both. In one embodiment, the attributes include a speed attribute (e.g., an estimated speed range) and a direction attribute (e.g., a direction of motion). In one embodiment, the perception module 1310 queries the representation element associated with at least one region of interest in the occlusion map 1304. In some examples, the representation element is a polygon or a center point of a query region (e.g., a segmented section in at least one region of interest). Then, the perception module 1310 obtains the occlusion data 1306 associated with the representation element from the occlusion map 1304. In some examples, the occlusion data 1306 is represented by 2D or 3D data.

[0137] The perception module 1310 (e.g., the occlusion information generator 1314) determines occlusion information 1315 associated with at least one region of interest based on region information associated with the at least one region of interest and occlusion data 1306 associated with the at least one region of interest. In one embodiment, the perception module 1310 determines the occlusion information 1315 by determining a corresponding occlusion (or non-occlusion) level for each segmented lane segment (e.g., represented by a polygon) within the at least one region of interest. The occlusion level (or non-occlusion) represents the occlusion (or non-occlusion) confidence level. In some examples, the level of occlusion (or non-occlusion) is a value between 0 and 1, where 1 represents 100% confidence. In one example, the occlusion level represents a boolean decision, either 0 or 1. The occlusion information 1315 is represented by 1D data including a series of shapes (e.g., points or polygons) having corresponding occlusion levels (e.g., values between 0 and 1). The occlusion information 1315 has a smaller data size than the occlusion data 1306.

[0138] In one embodiment, the perception module 1310 determines a corresponding occlusion level for each occluded object in the at least one region of interest. The occluded object is represented by a representation element (e.g., a polygon). The occlusion information 1315 includes the representation element of the occluded object and the corresponding occlusion level.

[0139] In one embodiment, the perception module 1310 identifies occupancy data associated with at least one region of interest in the occupancy map 1304. The occupancy data includes occlusion data and non-occlusion data. The occupancy data represents whether a region is occupied by at least one object (e.g., an occluded object or a non-occluded object or both). The perception module 1310 determines a corresponding occupancy (or non-occupancy) level for each segmented segment (e.g., a lane segment) associated with the at least one region of interest. The occupancy (or non-occupancy) level represents the confidence level of the occupancy (or non-occupancy) level. In some examples, the occupancy (or non-occupancy) level is a value between 0 and 1, where 1 represents 100% confidence. The perception module 1310 provides occupancy information including the corresponding occupancy levels of the segmented segments associated with the at least one region of interest to the planning module 1320 for route planning. The occupancy information is represented by 1D data including a series of shapes (e.g., points or polygons) having corresponding occupancy levels (e.g., values between 0 and 1). The occupancy information has a smaller data size than the occupancy data.

[0140] In the operation of a vehicle, the perception module 1310 recursively (periodically at a predetermined frequency or in real time) queries multiple regions of interest within a predetermined distance from the vehicle or along the vehicle's route (current route or potential route) or both, and identifies occlusion data associated with the regions of interest from the occlusion map 1304. The occlusion information generator 1314 obtains the occlusion data from the occlusion map 1304, obtains region information from the regions of interest from the map information extractor 1312, and generates occlusion information for the regions of interest based on the occlusion data and the region information. In one embodiment, the occlusion information generator 1314 updates the occlusion data obtained from the occlusion map 1304 and associated with at least one region of interest of the vehicle, and generates updated occlusion information associated with the at least one region of interest based on the updated occlusion data.

[0141] In one embodiment, the perception module 1310 (or the occlusion information generator 1314) applies a temporal filtering (or smoothing) algorithm to update the occlusion information of the regions of interest of the vehicle based on the occlusion data associated with the regions of interest. Since the occlusion data (e.g., occluded objects) associated with the regions of interest is recursively generated over time, the temporal filtering or smoothing algorithm uses the temporal relationships of the occlusion data generated at different time steps to make the occlusion data change continuously, which alleviates problems including the flickering of the occlusion data caused by noise in sensor measurements or missing or delayed sensor measurements. For example, the occlusion information generator 1314 updates the occlusion information based on the speed attribute (e.g., estimated speed range) and direction attribute (e.g., direction of movement) of at least one occluded object in the region of interest. The occlusion information generator 1314 updates the corresponding at least one occlusion level of at least one segmented section in the region of interest according to the movement of at least one occluded object. The movement is along the direction attribute of at least one occluded object and based on the speed attribute of the at least one occluded object. In one embodiment, the perception module 1310 also applies temporal filtering to update the occupancy information of the regions of interest of the vehicle based on the occupancy data associated with the regions of interest.

[0142] In one embodiment, the perception module 1310 represents at least one region of interest of the vehicle and the corresponding occlusion information in a structure (e.g., a graphical structure, a hierarchical structure, or a tree structure). Each region of interest has a corresponding identifier. Each lane is a node in the structure and maintains its corresponding connectivity. Discrete (or segmented) information and occlusion information are stored in each node or other maps. Traversal operations are required to obtain appropriate information from the structure.

[0143] The perception module 1310 provides the identifier of at least one region of interest and the corresponding occlusion information 1315 associated with the at least one region of interest to the planning module 1320. In one embodiment, the perception module 1310 provides data associated with a graphical structure (or interface) based on the road segments associated with the at least one region of interest and the corresponding occlusion levels. The graphical structure (or interface) is a map that associates the identifier of the region of interest with the corresponding occlusion information. The planning module 1320 displays the graphical structure (or interface) on a display of the vehicle.

[0144] In one embodiment, the data associated with the graphical structure (or interface) has the following structure:

[0145]

[0146] Process for implementing an occlusion representation of road features

[0147] Figure 14A - 14C Illustrate an example of generating region information for implementing an occlusion representation of road features. The generation is performed offline, for example Figure 4 by the database module 410, or the generation is performed online, for example Figure 13 by the perception module 1310.

[0148] Figure 14A Illustrate the environment 1400 in which the vehicle 1410 (e.g., Figure 1 the AV 100) travels on the road 1402. The road 1402 intersects with another road 1404. The road 1402 includes at least one lane. The road 1404 includes at least one lane, such as lane 1406 and lane 1408. Each lane has a lane direction. Lane 1406 has a lane direction 1407, and lane 1408 has a lane direction 1409 opposite to the lane direction 1407. The lane of the road 1402 is connected to lane 1406 of the road 1404 through a lane connection portion 1403. Information about the roads 1402, 1404, lane 1406, 1408, and lane connection portion 1403 is stored in, for example, a road network database implemented in the Figure 4 database module 410. In one embodiment, the information includes the corresponding identifiers of each road, each lane, and each lane connection portion.

[0149] Vehicle 1410 travels along a route. In one example, the route is route 1412 of lane 1406 of vehicle 1410 along which it turns right from road 1402 to road 1404. In one example, the route is route 1414 of vehicle 1410 along which it travels straight through the intersection between road 1402 and road 1404. In one example, the route is route 1416 of vehicle 1410 along which it turns left from road 1402 to lane 1408 of road 1404.

[0150] As vehicle 1410 travels along the route, a perception module (e.g., Figure 13 perception module 1310) of vehicle 1410 determines at least one region of interest of vehicle 1410 based on at least one of the route, a potential route, or a predetermined distance from vehicle 1410. In some examples, the perception module periodically determines the at least one region of interest at a predetermined time period or at a predetermined frequency.

[0151] As Figure 14A illustrated, the at least one region of interest of vehicle 1410 includes a stop region 1420 and priority regions 1430, 1432. The stop region 1420 is located after a stop line on the route. The stop region 1420 is represented by a rectangular shape. The priority region 1430 is located on lane 1406 and is represented by a rectangular shape. The extension of the priority region 1430 intersects the route 1412. The priority region 1432 is located on lane 1408 and is also represented by a rectangular shape. The extension of the priority region 1432 intersects the route 1414. In one embodiment, the perception module determines the characteristics (e.g., location, size, or area) of the stop region 1420 and the priority regions 1430, 1432. In one embodiment, a database module (e.g., Figure 4 database module 410) determines the characteristics of the stop region 1420 and the priority regions 1430, 1432 based on, for example, road network information in a road network database implemented in the database module. The database module stores the characteristics of the stop region 1420 and the priority regions 1430, 1432 in the road network database. The road network database also stores corresponding identifiers of the stop region 1420 and the priority regions 1430, 1432. The perception module identifies the at least one region of interest and obtains information of the at least one region of interest from the road network database.

[0152] Figure 14BIllustrated is an environment 1450 in which a priority area 1430 of a vehicle 1410 along a route 1412 is discretized by a discrete environment 1450. First, an intersection 1452 is determined by extending the priority area 1430 along a lane direction 1407 of a lane 1406 corresponding to the priority area 1430. Then, a series of discrete reference path points 1460 from a first end of the priority area 1430 to a second end of the priority area 1430 are determined along a discrete direction 1454 opposite to the lane direction 1407. In one example, the discrete reference path points 1460 are evenly distributed at a predetermined distance along the discrete direction 1454. In one example, the discrete reference path points are unevenly distributed along the discrete direction 1454, and adjacent reference path points closer to a lane connection portion 1403 in the priority area have a smaller distance than adjacent reference path points farther from the lane connection portion 1403.

[0153] Figure 14C Illustrated is an environment 1470 in which a priority area 1430 of a vehicle 1410 along a route 1412 is divided into a series of divided road segments 1472. The divided road segments 1472 are road segments on a lane 1406. In one embodiment, each of the divided road segments 1472 includes a discrete reference path point 1460 as a center point of the divided road segment 1472. Each of the divided road segments 1472 is represented by a polygon. The area information of the priority area 1430 includes the discrete reference path points 1460 and the divided road segments 1472. In one embodiment, the area information further includes information on the discrete reference path points 1460 and the divided road segments 1472, for example, the length of each divided road segment or the distance between adjacent discrete reference path points 1460.

[0154] Figure 15 Illustrated is a process 1500 for implementing an occlusion representation of road features during operation of a vehicle (e.g., Figure 1 the AV100 shown). The process 1500 is performed by a vehicle system (e.g., Figure 1 the AV system 120 shown). The vehicle system includes a perception module (e.g., Figure 13 the perception module 1310 shown), a planning module (e.g., Figure 13 the planning module 1320 shown), and a control module (e.g., Figure 4 the control module 406 shown). Similarly, embodiments may include different and / or additional steps, or steps may be performed in a different order.

[0155] The perception module obtains 1502 area information of at least one area of interest of the vehicle. The perception module is based on the current route of the vehicle (e.g., Figure 14A the route 1412 shown), the potential route of the vehicle (e.g.,Figure 14A determine at least one region of interest along a route (e.g., route 1414) shown or within a predetermined distance from the vehicle. The at least one region of interest includes at least one stop region (e.g., Figure 14A stop region 1420 shown), or at least one priority region ( Figure 14A priority region 1430 or 1432 shown), or both.

[0156] The region information includes discrete reference path points (e.g., Figure 14B reference path point 1460 shown) or segmented road sections (e.g., Figure 14C segmented roadway section 1472), or both. In one embodiment, the perception module obtains the region information of the at least one region of interest by obtaining discrete reference path points along at least one lane direction corresponding to the at least one region of interest. In one embodiment, the perception module obtains the region information of the at least one region of interest by obtaining a plurality of roadway sections sorted in sequence along at least one lane direction, each roadway section including at least one discrete reference path point.

[0157] In one embodiment, the perception module obtains the region information of the at least one region of interest from a mapping database (e.g., a road network database implemented in a database module such as Figure 4 database module 410 shown). The region information further includes identifiers of each lane (e.g., Figure 14A lane 1406 shown) or lane connection portion (e.g., Figure 14A lane connection portion 1403 shown) corresponding to the at least one region of interest.

[0158] In one embodiment, the perception module obtains the region information of the at least one region of interest by generating the region information of the at least one region of interest based on mapping data from the mapping database. As Figure 14B illustrated, the perception module determines an intersection between the current route of the vehicle (e.g., route 1412) and a region of interest (e.g., priority region 1430) or an extension of the region of interest in at least one region of interest along a lane direction (e.g., lane direction 1407), and determines a series of discrete reference path points within the region of interest in a direction opposite to the lane direction (e.g., discrete direction 1454) from the intersection point. As Figure 14C illustrated, the perception module determines a plurality of roadway sections (e.g., roadway section 1472) sorted in sequence along at least one lane direction, each roadway section including at least one discrete reference path point.

[0159] The perception module identifies 1504 with an occlusion map (e.g., Figure 13Occlusion data associated with at least one region of interest in the shown occlusion map 1304). In some examples, the occlusion map is represented by 2D or 3D points, lines, or shapes, and scene descriptions. The AV system maintains the occlusion map by accessing a database including road network information (e.g., a road network database) or using the output from the sensor system 1302 (e.g., Figure 13 the shown sensor system 1302), or both. In an example, the occlusion map includes at least one occluded object, e.g., a hypothesized object or an invisible but known object. The AV system stores the occlusion data of the occluded object in the occlusion map. In one embodiment, the occlusion data includes a representation element of the occluded object and an attribute of the occluded object. In some examples, the representation element is determined based on the type of the occluded object. The attribute includes size or motion state, or both. The occlusion data includes 2D or 3D data.

[0160] In one embodiment, the perception module identifies the occlusion data associated with at least one region of interest by querying the representation elements within at least one region of interest and obtaining the occlusion data based on the representation elements. In one embodiment, the representation element corresponds to an occluded object in at least one region of interest. In one embodiment, the representation element corresponds to a discrete reference path point (e.g., Figure 14B the shown reference path point 1460) or a segmented road section (e.g., Figure 14C the shown lane section 1472) in at least one region of interest.

[0161] The perception module determines the occlusion information 1506 associated with at least one region of interest based on the region information of at least one region of interest and the occlusion data associated with at least one region of interest. The occlusion data includes data associated with the occlusion information. The occlusion information has a smaller data size than the occlusion data.

[0162] In one embodiment, the perception module determines the occlusion information associated with at least one region of interest by determining the occlusion information associated with at least one region of interest based on discrete reference path points. In one embodiment, the perception module determines the occlusion information by determining the corresponding occlusion level of each segmented road section based on the occlusion data associated with at least one region of interest. In one embodiment, the perception module determines the occlusion information by determining at least one occlusion level of at least one representation shape corresponding to at least one occluded object within at least one region of interest.

[0163] In one embodiment, the perception module applies a temporal filtering algorithm to the occlusion data to update the occlusion information in at least one region of interest. In one embodiment, the perception module applies a temporal filtering algorithm to update the occlusion information based on the velocity attribute and the direction attribute of at least one occluded object in at least one region of interest. In one embodiment, the perception module updates the corresponding at least one occlusion level of at least one of the plurality of segmented road segments according to the movement of at least one occluded object in at least one region of interest, the movement being based on the velocity attribute of at least one occluded object.

[0164] The perception module provides the occlusion information 1508 to the planning module for planning the route of the vehicle. In one embodiment, the perception module provides the corresponding unique identifier of each region of interest in a plurality of regions and the occlusion information corresponding to each region of interest to the planning module. In one embodiment, the perception module provides data associated with the graphical interface based on a plurality of road segments associated with at least one region of interest and the corresponding occlusion levels. In one embodiment, the planning module causes the display to generate a graphical interface as an output based on the data.

[0165] In one embodiment, the perception module periodically queries a plurality of regions of interest within a predetermined distance from the vehicle, and periodically provides the occlusion information associated with the plurality of regions of interest to the planning circuit based on the periodic query of the plurality of regions of interest. In one embodiment, the perception module filters out a first plurality of regions of interest from the plurality of regions of interest. The path extending from each region of interest in the first plurality of regions of interest along the corresponding lane direction intersects a path different from the current route of the vehicle. The perception module publishes the occlusion information associated with a second plurality of regions of interest. The second plurality of regions of interest are regions of interest included in the plurality of regions of interest and different from the first plurality of regions of interest.

[0166] In one embodiment, the perception module identifies occupancy data associated with at least one region of interest, for example, from an occlusion map or an occupancy map. The occupancy data includes occlusion data and / or non-occlusion data. The perception module determines the corresponding occupancy level of each of the plurality of road segments based on the occupancy data associated with at least one region of interest, and provides occupancy information including the corresponding occupancy levels of the plurality of road segments to the planning module.

[0167] The planning module generates a planned route based on the occlusion information from the perception module. The control module operates the vehicle 1510 according to the planned route. In one embodiment, as Figure 13 illustrated, the perception module receives the planned route of the vehicle (e.g., planned route 1325) from the planning module, and updates at least one region of interest of the vehicle on the planned route.

[0168] In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details, which may vary according to implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive in a limiting sense. The sole and exclusive indication of the scope of the present invention, and what the applicant desires to be the scope of the present invention, is the literal and equivalent scope of the claims that issue from this application in the specific form of the issued claims, including any subsequent amendments. Any definition of terms expressly set forth herein for inclusion in such claims shall be construed to have the meaning such terms have as used in the claims. Additionally, when the term "further comprises" is used in the foregoing specification or the appended claims, the text following such phrase may be additional steps or entities, or sub-steps / sub-entities of the previously described steps or entities.

Claims

1. A method for a vehicle, comprising: Obtaining area information of at least one area of interest of the vehicle through a sensing circuit; Identifying occlusion data associated with the at least one area of interest through the sensing circuit; Determining occlusion information associated with the at least one area of interest through the sensing circuit based on the area information of the at least one area of interest and the occlusion data associated with the at least one area of interest, wherein the occlusion data includes data associated with the occlusion information, and the occlusion information has a data size smaller than that of the occlusion data, wherein the occlusion information includes a corresponding occlusion level for each of a plurality of segmented road sections associated with the at least one area of interest, the plurality of segmented road sections including a plurality of vehicle lane sections of one or more lanes associated with the at least one area of interest, and the plurality of vehicle lane sections being sequentially sorted along at least one lane direction of the one or more lanes; Providing, through the sensing circuit, the occlusion information to a planning circuit for planning a route of the vehicle; and Operating the vehicle according to the planned route.

2. The method according to claim 1, wherein, Obtaining the area information of the at least one area of interest includes: obtaining discrete reference path points along at least one lane direction corresponding to the at least one area of interest, and wherein determining the occlusion information associated with the at least one area of interest includes: determining the occlusion information associated with the at least one area of interest based on the discrete reference path points.

3. The method according to claim 2, wherein, Obtaining the area information of the at least one area of interest includes: Obtaining a plurality of vehicle lane sections sequentially sorted along the at least one lane direction, wherein each vehicle lane section includes at least one discrete reference path point.

4. The method according to claim 3, wherein, Determining the occlusion information includes: Determining a corresponding occlusion level for each of the vehicle lane sections in the plurality of vehicle lane sections based on the occlusion data associated with the at least one area of interest.

5. The method according to claim 4, further comprising: Providing data associated with a graphical interface based on the plurality of vehicle lane sections and the corresponding occlusion levels.

6. The method according to any one of claims 3 to 5, further comprising: Determining occupancy data associated with the at least one area of interest; Determining a corresponding occupancy level for each of the vehicle lane sections in the plurality of vehicle lane sections based on the occupancy data associated with the at least one area of interest; and And Providing occupancy information including the corresponding occupancy levels of the plurality of vehicle lane sections to the planning circuit.

7. The method according to any one of claims 2 to 5, wherein Obtaining the area information of the at least one area of interest includes: Obtaining the area information of the at least one area of interest from a mapping database.

8. The method according to any one of claims 2 to 5, wherein, Obtaining the area information of the at least one area of interest includes: Generating the area information of the at least one area of interest based on mapping data from a mapping database.

9. The method according to claim 8, wherein Generating the area information of the at least one area of interest includes: Determining an intersection point between the current route of the vehicle and an area of interest in the at least one area of interest or an extension of the area of interest along the lane direction; and Determine a series of discrete reference path points within the region of interest along a direction opposite to the lane direction starting from the intersection point.

10. The method according to any one of claims 1 to 5, wherein Determining the occlusion information includes: Determining at least one occlusion level representing a shape corresponding to at least one occluded object within the at least one region of interest.

11. The method according to any one of claims 1 to 5, wherein Identifying occlusion data associated with the at least one region of interest includes: Querying representation elements within the at least one region of interest; and Obtaining occlusion data based on the representation elements.

12. The method according to any one of claims 1 to 5, further comprising: Applying a time filtering algorithm to the occlusion data to update the occlusion information in the at least one region of interest.

13. The method according to claim 12, wherein, Applying the time filtering algorithm includes: Updating the occlusion information based on the speed attribute and direction attribute of at least one occluded object in the at least one region of interest.

14. The method according to claim 13, wherein, Determining the occlusion information associated with the at least one region of interest includes: Determining the corresponding occlusion levels of multiple segmented road sections based on the region information of the at least one region of interest, and wherein updating the occlusion information includes: Updating the corresponding at least one occlusion level of at least one segmented road section among the multiple segmented road sections according to the movement of the at least one occluded object in the at least one region of interest, the movement being based on the speed attribute of the at least one occluded object.

15. The method according to any one of claims 1 to 5, further comprising: Receiving the planned route of the vehicle from the planning circuit; and Updating at least one region of interest of the vehicle based on the planned route.

16. The method according to any one of claims 1 to, further comprising: Periodically querying multiple regions of interest within a predetermined distance from the vehicle; and Based on the periodic querying of the multiple regions of interest, periodically providing occlusion information associated with the multiple regions of interest to the planning circuit.

17. The method according to claim 16, further comprising: Filtering out a first plurality of regions of interest from the multiple regions of interest, where the paths extending along the corresponding lane directions from each region of interest in the first plurality of regions of interest intersect with paths different from the current route of the vehicle; and Releasing occlusion information associated with a second plurality of regions of interest to the planning circuit, the second plurality of regions of interest being regions of interest included in the multiple regions of interest and different from the first plurality of regions of interest.

18. The method according to any one of claims 1 to 5, further comprising: Determining the at least one region of interest as at least one priority region associated with an intersection in the current route of the vehicle.

19. The method according to claim 18, wherein Obtaining the region information of the at least one region of interest includes obtaining at least one of the following: The identifier of the lane corresponding to the at least one priority region, and The identifier of the lane connection portion corresponding to the at least one priority region.

20. The method according to any one of claims 1 to 5 further comprises: Determining the at least one region of interest as at least one stop region associated with a stop line in the current route of the vehicle, Wherein, determining the occlusion information includes: Determining an occlusion level of the at least one stop area.

21. The method according to any one of claims 1 to 5, wherein Obtaining area information of at least one region of interest includes: obtaining a corresponding unique identifier of each region of interest among a plurality of regions of interest, and Wherein, the method further includes: Providing the planning circuit with the corresponding unique identifier of each region of interest and occlusion information corresponding to each region of interest.

22. A vehicle, comprising: One or more computer processors; And One or more non-transitory storage media that store instructions that, when executed by the one or more computer processors, cause the method according to any one of claims 1-21 to be performed.

23. One or more non-transitory storage media that store instructions that, when executed by one or more computing devices, cause the method according to any one of claims 1-21 to be performed.

24. A computer program product that includes instructions that, when executed by one or more computing devices, cause the method according to any one of claims 1-21 to be performed.

Images