Method and system for controlling autonomous vehicles

Abstract

To provide encoding of a yielding scenario for an autonomous system.SOLUTION: An autonomous vehicle can pass through a yielding scenario by a safe and predictable method. In response to detection of the yielding scenario, a stand-by element data structure is generated, which encodes an own course, a geometry of a competitor course including at least one competing point with the own course, and a state of competition related to at least the one competing point. A geometry of a yielding scenario context such as an internal land surface of an intersection and an inlet / outlet line may be encoded. The standby element data structure is handed over to a yielding planner of the autonomous vehicle. The yielding planner determines a yielding behavior of the autonomous vehicle based on at least the stand-by element data structure. An autonomous vehicle control system can operate the autonomous vehicle in response to the yielding behavior so as to allow the autonomous vehicle to safely pass through the yielding scenario.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This application relates to U.S. Patent Application No. 17 / 395,318, entitled "Behavior Planning for Autonomous Vehicles in Yield Scenarios", filed on August 5, 2021, which is hereby incorporated by reference in its entirety.

Background Art

[0002] Advances in machine vision methods, neural network architectures, and computing substrates have begun to enable autonomous vehicles—on-road autonomous vehicles (e.g., self-driving cars and trucks) and robots, among others. For public and government regulatory agencies to accept the widespread deployment of self-driving cars and trucks on roadways, self-driving cars and trucks must achieve a safety level that exceeds the current safety level of the average human driver. Safe and effective driving requires that all drivers are confident in appropriately yielding when other vehicles in the area are obligated to do so. If a vehicle fails to yield, the drivers of other nearby vehicles may not be able to proceed in a safe and efficient manner due to the "unpredictability" of other drivers, e.g., drivers who provide behavioral cues that they may fail to yield when obligated. Thus, a necessary condition for the deployment of self-driving cars and trucks includes successfully navigating scenarios where self-driving cars and trucks must yield (e.g., intersections and merging lanes) "safely and carefully".

[0003] Typically, local traffic rules and driving protocols in an area dictate which vehicle operators have the responsibility or obligation to yield to others (and under what conditions). Such rules include traffic laws (e.g., vehicles must yield to pedestrians at crosswalks), situation-specific signs (e.g., road signs indicating which entry into an intersection is responsible for yielding to other entry points), and other real-time cues (e.g., the near-simultaneous arrival of multiple vehicles into a roundabout). However, conventional autonomous vehicles lack the ability to encode and deploy such protocols. On the contrary, conventional systems may aim to avoid collisions while failing to describe yielding protocols, and therefore cannot safely and predictably navigate yielding scenarios. [Overview of the Initiative] [Means for solving the problem]

[0004] Embodiments of this disclosure relate to encoding concession scenarios for autonomous systems (e.g., manned or unmanned vehicles or robots). Systems and methods are disclosed for providing real-time control of an autonomous system when the system encounters a concession scenario.

[0005] In contrast to conventional systems such as those described above, the disclosed embodiments enable autonomous systems to navigate yielding scenarios in a safe and predictable manner. In at least one embodiment, in response to the detection of a yielding scenario, a data structure is generated that encodes the geometry of the autonomous path, the path of a competitor including at least one point of conflict with the autonomous path, and the state of the conflict associated with at least one point of conflict. Geometry of the yielding scenario context, such as the internal area of ​​an intersection (e.g., as a polygon), and geometry defining entry or exit lines, may also be encoded. The data structure is passed to the yielding planner of the autonomous system. The yielding planner determines the yielding action of the autonomous system based at least on the data structure. The control system of the autonomous system may operate the autonomous system in accordance with the yielding action so that the autonomous system safely navigates the yielding scenario.

[0006] In at least one embodiment, yielding scenarios (e.g., crossing or merging yielding scenarios) may be detected based at least on the analysis of sensor data generated by at least one sensor of the autonomous vehicle. Map localization and / or perception may be used to determine various pieces of information related to yielding scenarios. For example, a first path for the autonomous vehicle and a second path for a competitor (e.g., another vehicle or other object) through the yielding scenario may be determined. At least one conflict point may exist between the paths, indicating that a collision may occur if those paths are crossed. To determine the conflict state (defining how the vehicle should move) at at least one conflict point, the system may determine one or more traffic rules to apply to the yielding scenario. A standby element data structure (also referred to as a standby element) may then encode information used by the vehicle to navigate the yielding scenario, such as path geometry, conflict state, and other information. For example, a standby element may be provided to the vehicle's control agent. The control agent may be able to use a waiting element to determine the first vehicle's yielding action.

[0007] The system and method for encoding yield scenarios for autonomous systems are described in detail below with reference to the attached drawings. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating one example of a concession scenario according to some embodiments of the present disclosure. [Figure 2] This figure shows non-limiting examples of a wait element data structure and a non-limiting example of a race state data structure according to some embodiments of the present disclosure. [Figure 3] This figure shows non-limiting examples of standby element engines in various embodiments according to some embodiments of the present disclosure. [Figure 4]This flowchart illustrates a method for encoding concession scenarios for autonomous vehicles (e.g., self-propelled vehicles) according to some embodiments of the present disclosure. [Figure 5] This flowchart illustrates a method for encoding concession scenarios for autonomous vehicles (e.g., self-propelled vehicles) according to some embodiments of the present disclosure. [Figure 6] This flowchart illustrates a method 600 for resolving a conflict between vehicle paths, according to some embodiments of the present disclosure. [Figure 7A] Illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure. [Figure 7B] Figure 7A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 7C] Figure 7A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 7D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 7A, according to some embodiments of the present disclosure. [Figure 8] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 9] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure. [Modes for carrying out the invention]

[0009] Systems and methods for encoding yield scenarios for autonomous vehicles are disclosed. This disclosure may be described in relation to exemplary autonomous vehicles 700 (referred to herein as "vehicle 700" or "autonomous vehicle 700" as otherwise provided herein, examples of which are described with reference to Figures 7A–7D), but this is not intended to be limiting. For example, the systems and methods described herein may be used without limitation by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), piloted or unpiloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles towed to one or more trailers, flying ships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. In addition, while this disclosure may describe the control of autonomous vehicles to navigate concede scenarios, this is not intended to be limiting, and the systems and methods described herein may be used in any other technological space where autonomous control systems may be used, including augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or other technological spaces where autonomous control systems may be used.

[0010] In the normal course of operating an autonomous vehicle, the control agent must avoid both moving and stationary obstacles (e.g., other vehicles, pedestrians, cyclists, lane barriers, and similar objects). In addition to avoiding collisions, the agent has a fundamental responsibility to yield to other road users in certain scenarios (e.g., “yielding conditions”). Such yielding conditions may exist at (controlled and uncontrolled) intersections, crosswalks, merging lanes, highway (or interstate) on / off ramps, roundabouts, and similar objects, such as parking garages and / or parking garage navigation. To enable another user to safely, confidently, and efficiently “clear” the yielding conditions, yielding actions may include slowing down the vehicle or even bringing it to a complete stop. For example, at an unmarked intersection where another vehicle has arrived first, a later arriving vehicle may perform appropriate yielding actions by slowing down to allow the first vehicle to safely clear the intersection. Such yielding ensures that the following vehicle does not enter the intersection until the first vehicle has safely cleared it. Under such yielding conditions, one or more users may have a clearly defined obligation (or responsibility) to yield to other users.

[0011] Yielding behavior offers benefits beyond mere collision avoidance. Appropriate yielding can ensure the “courteous and expected” driving dynamics necessary for safe and efficient transport. For example, even if an agent acts to avoid a potential collision under yielding conditions (e.g., by accelerating through an intersection), failing to yield when the obligation exists creates tense and unsettling driving conditions for all users in the area. Even if someone accelerates to avoid a collision, a vehicle that does not yield can create anxiety, a sense of danger, and anger (e.g., rage while driving) in other drivers, cyclists, and pedestrians. In other words, even if a collision is avoided by taking aggressive action, the collision was not avoided in the “safe and courteous manner” that other users expect. Therefore, in the operation of autonomous vehicles, the agent of an autonomous vehicle may have an obligation (e.g., legally or normatively) to adopt a yielding strategy of one or more actions when approaching a yielding scenario.

[0012] This disclosure partially realizes a “waiting element engine” for an autonomous vehicle (“Self-Vehicle”) that can proactively monitor the arrival of one or more yielding scenarios (e.g., a vehicle is arriving at an intersection, a vehicle is going through an on / off ramp, or a vehicle is preparing to change lanes). The waiting element engine may generate one or more “waiting element” data structures that encode representations of yielding scenarios. Waiting elements may be provided as input to the Self-Vehicle “yield planner”. Examples of yielding scenarios include crossing (e.g., intersections) and merging (e.g., lanes merging on an on / off ramp). Each yielding scenario may involve at least two parties: the Self-Vehicle and at least one competitor (e.g., another vehicle, a pedestrian, a cyclist, and similar). Competitors may include other vehicles (e.g., autonomous, semi-autonomous, and / or conventionally manually operated), as well as individuals (pedestrians and cyclists).

[0013] In various embodiments, a yielding scenario may involve multiple competitors (e.g., one's own vehicle approaching an intersection with multiple other vehicles, pedestrians, and / or cyclists; one's own vehicle merging into a lane with multiple other vehicles; and similar). Each party involved in a yielding scenario may be involved with one or more “potential paths” or lanes. For a party in a yielding scenario, a potential path or lane may include a current or given set of possible spatial paths, e.g., the party’s current coordinates in spatial velocity-phase space. Thus, a party’s potential paths may depend not only on their current spatial and velocity coordinates but also on the vehicle’s (or individual’s or entity’s) limitations regarding acceleration, deceleration (e.g., braking force), and maneuverability (e.g., turning radius, traction control, and similar).

[0014] When yield conditions are detected, the waiting element engine may receive and / or generate environmental data from various sources related to the yielding scenario (e.g., onboard and / or offboard sensors and / or detectors, perception-based data, map-based data, geolocation data, and similar). The waiting element engine can analyze and merge various data, and can check, analyze, merge, and match the data to various yield-related traffic rules, taking the analyzed data into consideration, to generate various “waiting geometries” and “contest states” that can characterize the yielding scenario of its own vehicle. For example, data encoding various aspects of the vehicle’s potential path and scenario geometry may be analyzed and matched against one or more yield or traffic rules to determine the contest state of a yielding scenario (e.g., proceed, stop at entrance, yield from entrance, and similar). Waiting geometries may be grouped into one or more “waiting groups,” where a waiting group may refer to all waiting elements of a yielding scenario. Waiting geometries and contest states may be encoded in a “waiting element” data structure. The standby element data structure can be provided to the autonomous vehicle's "yield planner" for use when controlling the vehicle.

[0015] The yield planner can receive a standby element data structure and determine an appropriate yield action. When the vehicle's control agent adopts the determined yield action (for example, defined by the conflict state), the vehicle can safely fulfill its required and expected yield obligations while avoiding a collision.

[0016] In at least one embodiment, the standby element engine may receive and / or acquire various input data, which may include geometry-related data, signal-related data, and map-related data. Acquiring (or receiving) data may be achieved using sensing, perceiving, and / or detecting techniques that may leverage geometry or visual perception, map perception (which may include localization), and signal perception. In at least one embodiment, perceiving data may include lane graph data. Lane graph data may include one or more lanes that can be assigned as potential lanes of the own vehicle (e.g., own lanes) and one or more lanes that can be assigned as potential lanes of one or more competitors (e.g., competitor lanes). Other input data may include various raw sensor data from the own vehicle or competitors. Geometry input data may include various information about the geometry of the environment as applied to potential lanes related to the yielding scenario and / or background context. Signal input data may include and / or encode traffic signals such as traffic lights, traffic signs, priority road signs such as stop signs, yield signs, and major road signs, speed limit signs, and hand signs or other body postures used to signal traffic.

[0017] In various embodiments, the wait element data structure can be generated for each possible pairing of the potential paths of the host vehicle and the potential paths of the competitors. In a non-limiting example of a yield scenario, the yield scenario is associated with one host vehicle and j competitors, where j is a positive integer. The host vehicle can be associated with i potential paths, and each of the j competitors can be associated with k potential paths, where i and k are also positive integers. In such an example, the wait element engine can generate i × j × k distinct wait elements. Thus, each wait element can be associated with one host vehicle potential path and one competitor potential path. The wait element can encode the "wait geometry" of the host path, the wait geometry of the competitor path, and the wait geometry of the context of the two paths. The wait element can further encode the "conflict state" between the two paths.

[0018] Briefly here, the wait geometry of a claimed path (e.g., a host path or a competitor path) can include a set of field value pairs or other data types or elements of the path that encode various aspects of the claimed path. Such fields of the wait geometry of a path can include, but are not limited to, entry lines, exit lines, entry and exit competitor areas, intersection entry lines and internal ground, conflict points between the host path and the competitor path (an explicit encoding of one option of an intersection or merge point between paths), and the like. The conflict state of the wait element (e.g., the state of conflict) can be, or can define, an instruction to the yield planner regarding how the host vehicle should yield or take the road with respect to this wait element. Such states include, but are not limited to: proceed on the road, stop at the entry, yield from the entry, and the like.

[0019] In at least one embodiment, to generate waiting elements, geometry input data, which may be referred to as waiting geometry data, can be "fused" with lane graph data and map data. The "fused geometry" data can then be classified (e.g., as a left turn, right turn, U-turn, etc.) and associated with one or more routes. Signal data can be fused with map data. Signal states (e.g., green signal, red line, inactive, etc.) can be determined from the fused signal data. Along with map data, signal states, and other data, the fused, classified, and associated geometry data can be provided as an input to the "conflict resolver" of the waiting element engine. The condition resolver can use geometry, signal, map, and other sensor data, along with traffic rules for determining conflict states and decomposing data into waiting elements.

[0020] Referring to FIG. 1, FIG. 1 shows an example of a yielding scenario 100 according to some embodiments of the present disclosure. The non-limiting yielding scenario 100 of FIG. 1 is an example of an intersecting (or intersection) yielding scenario. Other types of yielding scenarios include at least merging yielding scenarios (e.g., at a highway entrance). In this non-limiting example of the intersecting yielding scenario 100, three vehicles are approaching a four-way intersection. The three vehicles include a first vehicle 102 (e.g., the host vehicle), a second vehicle 104 (e.g., the first competitor), and a third vehicle 106 (e.g., the second competitor). A waiting element engine 130 is used to generate one or more waiting element data structures (e.g., waiting element_1 110 and waiting element_2 120) of the yielding scenario 100. In some embodiments, the waiting element engine 130 can be mounted on the host vehicle 102. In other embodiments, the waiting element engine 130 can be at least partially remote from the host vehicle 102. In such embodiments, the host vehicle 102 can access the waiting element engine 130 via one or more communication networks.

[0021] Various embodiments of the standby element engine will be discussed, at least in conjunction with the standby element engine 300 shown in Figure 3. It should be understood that these and other configurations described herein are merely illustrative. Other configurations and elements (e.g., machines, interfaces, functions, sequences, function groupings, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor that executes instructions stored in memory.

[0022] Await element data structures, such as but not limited to wait element_1 110 and wait element_2 120, may be associated with a pair of claim paths, one of which is the claim path of the own vehicle 102, and the other of the pair's claim paths is the claim path of a competitor (e.g., a first competitor 104 or a second competitor 106). Each wait element may encode the wait geometry of the claim path of the own vehicle 102 (e.g., the self-wait geometry 112 of wait element_1 110 or the self-wait geometry 122 of wait element_2 120) and the wait geometry of the competitor's claim path (e.g., the competitor_1 wait geometry 114 of wait element_1 110 or the competitor_1 wait geometry 124 of wait element_120). Therefore, the standby geometry_1 110 may be associated with a single claim path of the own vehicle 102 and a single claim path of the first competitor 104 (in other examples where more paths may be associated with the standby element). Similarly, the standby geometry_2 120 is associated with a single claim path of the own vehicle 102 and a single claim path of the second competitor 106. Note that the claim path of the own vehicle 102 associated with standby element_1 110 may (but is not required) be the same claim path of the own vehicle 102 associated with standby element_2 120. Various embodiments of the standby element geometry will be discussed in conjunction with at least Figure 2.

[0023] In addition to the wait geometry of the claim path pair, each wait element may encode the wait geometry of the context (e.g., wait geometry context 116 for wait element 110 and wait geometry context 126 for wait element 2 120). Furthermore, each wait element may encode the conflict state of the claim path pair (e.g., conflict state 118 for wait element 110 and conflict state 2 for wait element 2 120). Various embodiments of the wait geometry context and conflict state will be discussed in conjunction with at least Figure 2.

[0024] More generally, a wait element may include (or encode) some subset of the wait geometry for one self-directed path (e.g., the self-vehicle's claimed path) and the wait geometry, wait geometry context, and conflict state for one competitor's path (e.g., the competitor's claimed path in yield scenario 100). The self-wait geometry 112, competitor_1 wait geometry 114, wait geometry context 116, and conflict state_1 118 (of wait element_1 110) may also be data objects and / or data structures. Similarly, the self-wait geometry 122, competitor_2 wait geometry 124, wait geometry context 126, and conflict state_2 128 (of wait element_2 120) may also be data objects and / or data structures. In various embodiments, if one or more data values ​​(or elements) of these data objects are not readily available (or are not applicable to a given scenario), the data encoding of these missing elements may be set to "invalid" and / or "not applicable".

[0025] In one or more embodiments, wait elements (e.g., wait element 1 110 and wait element 2 120) constitute an "atom" of how information (or data) can be encoded with respect to wait conditions (e.g., yield scenario 100). Wait elements may be provided as input to a yield planner (not shown in Figure 1) for the vehicle 102. Based at least on the encoding of the wait elements, the yield planner may determine an appropriate yield action to safely navigate yield scenario 100. As discussed herein, wait elements may be determined and / or generated by using at least one of two methodologies (and / or a combination thereof). One methodology involves using a set of yield-related traffic rules (e.g., yield heuristics) for yield scenario 100. The other methodology involves using real-time perception of mapping and geometry-related and / or signal-state data for yield scenario 100. As described above, in various embodiments, the two methodologies can be combined in various ways to utilize mapping data, real-time perception of geometry / signal data, and concession heuristics.

[0026] Focusing on Figure 2, Figure 2 shows non-limiting examples of the standby element data structure 200 and non-limiting examples of the conflict state data structure 210 (also referred to as the conflict state) according to several embodiments of the present disclosure. In various embodiments, the standby geometry 200 and / or the conflict state 210 may be data objects or any other such structured data. Generally, the standby geometry 200 may represent geometry and associated metadata that arise when additional information (e.g., information and / or data related to yield scenarios, e.g., yield scenario 100 in Figure 1) is applied to the lane graph (e.g., own vehicle claim path and / or competitor claim path). That is, the standby geometry 200 may be applied to the own path (e.g., own vehicle claim path), the competitor path (e.g., competitor claim path), or the background context. For example, if the idle geometry 200 is applied to its own path, then the idle geometry 200 may be similar to the self-idle geometry 112 of idle element 1 110 and / or the self-idle geometry 122 of idle element 2 120 in Figure 1. If the idle geometry 200 is applied to a competitor's path, then the idle geometry 200 may be similar to the competitor 1 idle geometry 114 of idle element 1 110 and / or the competitor 2 idle geometry 124 of idle element 2 120. If the idle geometry 200 is applied to a background context, then the idle geometry 200 may be similar to the idle geometry context 116 of idle element 1 110 and / or the idle geometry context 126 of idle element 2 120. The idle geometry in such a context may be related to the boundary and / or interior area of ​​an intersection, or the presence of an intersection entrance line.

[0027] In various non-limiting embodiments, a standby geometry may encode at least some of its data within a field-value pair. Thus, a set of fields (e.g., standby geometry field 202) may be associated with (or encoded within) a standby geometry 200. One or more values ​​may be associated with each field of standby geometry field 202 to encode a set of field-value pairs. Note that values ​​may be data structures or data. In some embodiments, the values ​​of a particular field may be another field, so that the standby geometry 200 may encode one or more data trees. As shown in Figure 2, such a field 202 may include, but is not limited to, entrance lines (e.g., of the corresponding self-path or competitor path), exit lines (e.g., of the corresponding self-path or competitor path), entrance competitor area (e.g., of the corresponding self-path or competitor path), exit competitor area (e.g., of the corresponding self-path or competitor path), and intersection entrance lines (e.g., of the corresponding self-path or competitor path). In some embodiments, field 202 may include coordinates or other information defining the boundaries and / or location of the internal ground area (as part of the general context of the waiting group) and one or more points of contention between the own path and the competitor's path (an explicit encoding of one of the options of intersection or merging points between the paths). Field 202 may additionally include a speed limit applicable in the general context (which should be considered applicable between the entrance line and the exit line). If a field does not apply to a particular yielding scenario (for example, an on-ramp traffic signal has only its own path and entrance line, and no exit line, competitor's path, or internal ground), the values ​​of each of these fields may be encoded as invalid to conform to the encoding waiting conditions. Another example would be the encoding of a new speed limit by a waiting group, which includes only the entrance line and speed limit in the entire context, and all else is set to invalid. The exit line will then be interpreted as infinite or, unless otherwise notified, and as with other attributes.

[0028] The entry line of one's own path may encode a stopping point for some of the yielding actions. The entry line may also signal the start of a general competition area, flanked by an exit line, which may indicate which segments of one's own path need to be cleared in order to satisfy the waiting conditions of this waiting group. The internal ground area may represent the internal ground of an intersection as a polygonal area that may be called coordinate space. The internal area may encompass the segments between the entry line and the corresponding exit line (sometimes the exit line may move outside, such as by crossing a crosswalk, even if the internal ground is not defined). The entry competitor area and internal ground provide a context for analyzing the other parties. This may be done by assigning the parties to paths and areas (in a non-exclusive manner). The geometry of one's own and competitor paths, as well as the competition points, may be used by the yield planner to implement yielding actions, as needed. The geometry may also be used to determine which rules apply.

[0029] In some instances, a conflict point may indicate or represent an explicit geometric point. In other instances, a conflict point may refer to an outline concept that is a specific conflict, in which a waiting element references and / or encodes its state. In such instances, the state of the conflict at a conflict point may also be the payload of the overall conflict state resolution process, which may provide a decision for each conflict point on whether the vehicle should yield or not with respect to that conflict point. In this sense, a conflict point may also indicate the choice of the vehicle's path, access to and through the competitor's path, the actual competitor, and how the vehicle should behave toward them.

[0030] Waiting geometries can be grouped together (for example, logically organized), so the semantic meaning of a waiting group may be that all waiting conditions in this group can be considered together, specifically that they can all be cleared so that the vehicle itself is not on the left between them (for example, the vehicle itself should not get stuck waiting for a pedestrian at the end of a left turn while the vehicle itself is still in the path of approaching traffic, and therefore the approaching traffic conflict can be considered together with the crosswalk conflict within the same waiting group).

[0031] Referring to the race condition 210, a waiting element engine, for example, the waiting element engine 300 in Figure 3, but not limited to this, may include a race condition resolver (for example, the race condition resolver 340 of the waiting element engine 300). Such a race condition resolver may perform a race condition resolution process. The goal of the race condition resolution process may also be to provide the race condition (for example, race condition 210) for each waiting element (for example, waiting element 1 110 and waiting element 2 120 in Figure 1). The race condition 210 for a waiting element may also be a command to yield planner regarding the attitude of whether its own vehicle should yield or proceed with respect to that waiting element, as a matter of rules, expectations, formal or informal customs or norms.

[0032] In some non-limiting embodiments, the conflict state 210 does not have to indicate what is actually happening in the yielding scenario, what is physically possible in the yielding scenario, or whether the own vehicle has priority in the yielding scenario (e.g., an intersecting yielding scenario or a merging yielding scenario) but may be forced to yield. Rather, the conflict state 210 may indicate what should conventionally happen. It may then be the responsibility of the yielding planner to actually implement the yield, in the sense that it may consider whether the own vehicle is actually in a place where it should stop and obeys that command, and whether the other party (e.g., the competitor in the yielding scenario) appears to be fulfilling their expected yielding obligation and taking appropriate action, as should happen (e.g., as encoded in the conflict state 210). For example, the yielding planner may determine that the conflict state is proceeding but the competitor is not yielding (essentially detecting "it is appropriate to honk"), and although it should not have happened, it may decide to yield. A yielding planner can implement yielding behavior by analyzing all conflicts within a waiting group until all waiting elements within the group can be cleared collectively. This means that all conflicts within a waiting group can be followed collectively, and the most restrictive conflict can define the expected yielding behavior of the vehicle itself. For example, if one conflict state in a waiting group is to proceed down the road and stop at another entrance, the vehicle itself may remain on the entrance line.

[0033] As shown in Figure 2, a race condition 210 (for example, as encoded in a wait element) may include one or more of the 17 states listed in race condition 210. Note that this list of possible race conditions is non-exclusive, and in other embodiments, race condition 210 may include additional and / or alternative race conditions. A "Go Ahead" condition may indicate an expectation that a racer will yield. Thus, a "Go Ahead" condition does not have to indicate a formal constraint on the corresponding wait element (other than for the yielding planner to look at the racers associated with this wait element and ensure they yield as expected). For various race conditions, the keyword "transition" may be used to indicate that the corresponding race condition may be updated and / or developed in the very near future. Thus, a "Go Ahead" transition may indicate that the "Go Ahead" condition currently applies but is likely to change soon to a more restrictive condition. A standard example is the "yellow" state of a traffic light, which can be encoded by the transient "Go Ahead". A stop at entry can indicate that the command is to stop at the entry line, wait for further commands, and not proceed further until the conflict state changes. A yield from entry can indicate that the vehicle should remain at the entry line until it is expected that this conflict will be cleared. For such a stop at entry state, a pre-stop may not be defined by the rules, but the vehicle's control agent should ensure that the conflict is cleared before the vehicle passes the entry line, which often results in a pre-stop. It should also be noted that since wait conditions in a wait group can be considered jointly, this usually means in practice that the control agent should be confident that all conflicts within the wait group are cleared before the vehicle passes the entry line. In other words, if one conflict in a wait group has yield from entry, all other conflicts in the wait group may inherit that same conflict state when analyzed by the yield planner, and if one has stop at entry, all wait elements in the group may inherit pre-stop. The transitional state of yielding from the entrance can also be a transitional version of the state of yielding from the entrance.

[0034] The state of yielding a point of contention may indicate that a pre-stop is not required by the rules, and may not indicate that the vehicle may officially have to remain at the entrance line while waiting for the contention to clear (which in principle is not a problem). The control agent may have to ensure that the vehicle yields appropriately to the competitor regarding this contention, that the vehicle does not block the contention, and that the vehicle behaves in a way that is clear to the competitor regarding the contention to which the vehicle is yielding. This may mean that the vehicle moves forward to turn left at the intersection, but slowly and with enough leeway so that approaching traffic understands that the vehicle appears to intend to yield, and does not become obvious to the approaching traffic. The transient state of yielding a point of contention may be a transient version of the state of yielding a point of contention. The state of stopping at the entrance and then yielding from the entrance may be equivalent (or at least similar) to the state of yielding from the entrance, but with the additional condition that a pre-stop is required at the entrance line. The state of stopping at the entrance and then yielding the point of contention may be equivalent to (or at least similar to) the state of yielding the point of contention, but with the additional condition that a prior stop is required on the entrance line.

[0035] The transient state of stopping at an entrance and then yielding the point of contention may also be a transient version of the state of stopping at an entrance and then yielding the point of contention. The state in which the stop initially has priority may also be the case of a standard "US multiway stop." Priority may be determined as a first-in, first-out queuing system, where "enter" is defined as the first party from its competitor's path to approach the intersection (likely to be within the corresponding competitor area on the entrance line pointing to the interior ground) and stop. In other words, this contention state may further imply a "who stopped first" process for actually breaking it down into states of each party proceeding along their respective paths or yielding from the entrance related to this competitor's path.

[0036] A clearing condition may indicate the absence of known criteria for determining priority at a highway merge, such as when there are no cues from traffic rules, map statistics, road geometry, or size (equivalently large highways with merging and similarly straight shapes). A clearing condition after stopping at the entrance may be equivalent to (or at least similar to) a clearing condition, but with the additional condition that a prior stop is required at the entrance line. This condition may be used when there is an entrance line to clear but no established convention to use as a standard. A not-allowed condition may be used to enable encoding that something is not allowed. For example, a waiting element may include a left-turn lane crossing traffic into a parking lot, where there may be a signal indicating that a change of direction (e.g., crossing a double solid yellow line) is prohibited. In this case, the vehicle may proceed in a way that signals (to competitors) that the condition is not only to stop now, but that it never changes and is simply not allowed. This condition may be useful for yielding planners because it considers multiple options for the vehicle's path (e.g., multiple vehicle paths may be considered simultaneously). The stop and request takeover state may indicate that the vehicle has encountered something determined to be outside the operational design domain (for example, a road work or flag signal may have been detected, and the vehicle has not yet implemented the handling of such conditions). In such a state, the control agent (or yielding planner) may request the company to slow down, stop, and take over the request of the vehicle. The unknown state may be used to encode predictions of future conflict states, in which case it may be useful to encode the absence of knowledge or predictions.

[0037] Figure 3 shows non-limiting examples of the standby element engine 300 in some embodiments of the present disclosure. As described throughout this document, the standby element engine 300 may be mounted on an autonomous vehicle (e.g., the autonomous vehicle 102 in Figure 1). In other embodiments, the autonomous vehicle may have access to the remote standby element engine via one or more communication networks. As discussed throughout this document, when the autonomous vehicle approaches an intersecting or merging yielding scenario (e.g., intersecting yielding scenario 100 in Figure 1), the standby element engine 300 may generate one or more standby elements (e.g., standby element 1 110 in Figure 1, standby element 2 120 in Figure 1, and standby element 310 in Figure 3) as outputs. Various inputs to the standby element engine 300 are discussed below.

[0038] The standby element engine 300 may include a standby geometry perception device 302, a mapper 304, a signal perception device 306, a lane grapher 350, and / or another sensor data receiver 308. The standby geometry perception device 302, the mapper 204, the signal perception device 306, the lane grapher 250, and the other sensor data receiver 308 receive various inputs as described later. The standby element engine 300 may further include a standby geometry fuser 322, a geometry classifier 324, and a geometry association device 326. The standby element engine 300 may also include a signal fuser 328, a signal state estimater 330, and a race state resolver 340. The race state resolver 340 may include a condition checker 342, a base rule parser 344, a map rule checker 346, and the standby element fuser 348. The output waiting element 310 may include a self-waiting geometry 312, a competitor waiting geometry 314, a context waiting geometry 316, and a conflict state 318.

[0039] The lane grapher 350 is generally responsible for receiving one or more lane graphs as input to the standby element engine 300. Lane graphs may be received in response to approaching and / or detecting a yielding scenario. A lane graph may include one set of claiming paths from the same lane bundle, one set of self-directed paths 352, and one set of competitor paths 354. Self-directed paths 352 may be received or generated from the output, maps, and / or vehicle traces of one or more neural networks or other machine learning models. For example, and without limitation, the machine learning models described herein may include one or more machine learning models using any type of machine learning model, such as linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (such as autoencoders, convolutions, recurrences, perceptrons, long / short-term memory (LSTMs), Hopfield, Boltzmann, deep belief, deconvolution, adversarial generation, liquid state machines, etc.), and / or other types of machine learning models.

[0040] The self-directed paths 352 can be ensemble-fused into a new lane graph. The lane graph can be structured in an array of lane graphs, allowing for flexibility to easily use any combination of lane graphs. One or more of the competitor paths 354 may overlap with one or more of the self-directed paths 352 at one or more conflict points. A particular conflict point can be classified as one of two basic conflict point types: an intersecting conflict point or a merging conflict point. With respect to intersecting conflicts, the self-directed path may intersect with a competitor path at a single point (or a small neighborhood of a point), not as an example but as an extension. With respect to merging conflict points, the self-directed path may meet and merge with a competitor path for at least a segment of the competitor path (or vice versa). The competitor paths 354 may be received or generated from the output, map, or intersection parsing of one or more neural networks or other machine learning models. Self-direction 352 is the claimed direction of the self-direction vehicle, while competitor directions are claimed directions from vehicles, bicycles, pedestrians, trams and trains, or any other party in a yielding scenario. Like self-direction 352, competitor directions 354 may be merged from their sources (which may be similar to or the same as the sources of self-direction 352) to form an ensemble.

[0041] The standby geometry sensing device 302 and the signal sensing device 306 can generally perform “standby sensing.” The standby geometry sensing device 302 receives various geometry-related inputs (in response to the identification and / or detection that its own vehicle is approaching a scenario in which it will yield) and generates standby geometry for output (e.g., standby geometry 200 in Figure 2). Thus, the standby geometry sensing device 302 can generate standby geometry for each of its own paths in a set of its own paths 352 (e.g., its own standby geometry 312). Similarly, the standby geometry sensing device 302 can generate standby geometry for each of its competitor paths in a set of competitor paths 354 (e.g., competitor standby geometry 314). Similarly, the signal sensing device 306 receives various signal-related inputs (in response to the identification and / or detection that its own vehicle is approaching a scenario in which it will yield) and generates signals for output.

[0042] Waiting geometry (e.g., self-waiting geometry 312 or competitor waiting geometry 314) may include geometry and metadata that arise when additional information regarding waiting conditions is applied to the lane graph. Waiting geometry may be applied to the self-path (e.g., entrance line), competitor path (e.g., competitor area), or background context (e.g., the interior ground of an intersection, or the presence of an intersection entrance line). As discussed in conjunction with waiting geometry 200 in Figure 2 at least, a particular waiting geometry may include filed value pairs relating to: entrance and exit lines (for both the self-path and competitor paths), entrance and exit competitor areas (for both the self-path and competitor paths), intersection entrance line and interior ground (as part of the general context of the waiting group), and the point of conflict between the self-path and the competitor path (an explicit encoding of one option of an intersection or merging point between paths). Waiting geometry may include field value pairs of speed limits applied in the general context (which are intended to apply between the entrance and exit lines). The values ​​of each of these fields may be encoded as invalid to conform to encoding wait conditions, such as when the field does not apply to a scenario in which it yields (for example, an on-ramp traffic signal light has only its own path and entrance line, and no exit line, competitor path, or internal ground). Another example may be the encoding of a new speed limit by a wait group that in the entire context includes only an entrance line and a speed limit, and anything else set to invalid. The exit line may then be interpreted as infinite or until otherwise notified, and so as with other attributes.

[0043] The value of the entry line field for the self-path may encode the requested path stopping point of the self-vehicle for multiple possible yielding actions. Such an entry line may also encode the start of a general competition area, flanked by an exit line, which may indicate which segment of the self-path needs to be cleared to satisfy the waiting conditions of this waiting group. The internal ground area may represent the internal ground of an intersection as a polygonal area. The internal ground area may encompass the path segment between the entry line and the exit line (sometimes the exit line moves outward, such as by crossing a crosswalk, even if the internal ground is not moved). The entry competitor area and internal ground may provide context for analyzing other parties. This may be made possible by obstacle-in-path analysis (OIPA), which assigns parties to paths and areas (in a non-exclusive manner). The geometry of the self and competitor paths, as well as the competition points, may be used by the yield planner to implement yields as needed. The waiting geometry may also be used to determine which rules apply.

[0044] A race point may represent an explicit geometric point (e.g., the intersection of the own path and the competitor's path). In some embodiments, a race point may be considered as a conceptual outline of a specific race that a waiting element is referencing and encoding its state. In this latter sense, the race state at a race point may also be the payload of a race state resolution process (as performed by the race state resolver 340). The race state may provide a determination for each race point of "in what way should the own vehicle yield with respect to this race point or not." In this sense, a race point may also indicate, given the choice of the own path, how the actual competitor and the own vehicle should proceed toward them through access to and via the competitor's path.

[0045] Waiting geometries can be grouped into one or more waiting groups, where the semantic meaning of a waiting group is that all waiting conditions within that group should be considered together, and specifically, can be cleared together, such that the vehicle itself is not on the left side between the two (for example, the vehicle itself does not get stuck waiting for a pedestrian at the end of a left turn while it is still in the path of approaching traffic, and therefore the approaching traffic competition must be considered together with the crosswalk competition in the same waiting group).

[0046] Mapper 304 may receive map data in response to the identification and / or detection that its vehicle is approaching a yielding scenario (for example, based at least on map localization). Map data may include one or more 2D or 3D maps of the environment of the approaching yielding scenario. Signal perception device 306 may receive signal-related data in response to the identification and / or detection that its vehicle is approaching a yielding scenario. Signal-related data may include signal data encoding indications for traffic lights and traffic signs, such as stop signs, yield signs, priority road signs such as major road signs, and speed limit signs. Signal data may also encode outputs generated from one or more neural networks or other machine learning models, such as whether an intersection is a traffic sign intersection, a stop sign intersection, unmarked, a roundabout, a highway on-ramp, a toll booth, or any other type. Signal data may encode indications for traffic police officers, flag bearers, or road workers, road barriers, and road lights at crosswalks. Signal data can encode instructions for traffic cones, merging arrows, and all temporary items placed on the road to redirect traffic. In various embodiments, the signal perception device 306 can distinguish between the presence and state of a signal. The signal perception device 306 can perform live signal perception, providing both the presence and state of a signal (e.g., a traffic light exists in these 2D or 3D coordinates and its current state is yellow). In some embodiments, a map (e.g., a map received by mapper 304) may contain information about the possible presence and / or location of a signal, such as the presence of a traffic light or stop sign, while the state of a signal, such as the state of a traffic light, may be provided by live perception (or infrastructure to vehicle communication) via the signal perception device 306. Signal detection (via the signal perception device 306) may provide the presence, type, and spatial attributes of a signal, such as the 3D location and bounding box of a traffic light or sign or police officer detecting traffic.

[0047] The idle geometry fuser 322 may “fuse” or combine geometry data (e.g., idle elements) from the idle geometry sensing device 302 and map data from the mapper 304 through a process called idle geometry fusion. Idle geometry fusion may optionally be performed to obtain improved geometry information from a combination of live perception (e.g., real-time geometry perception as performed by the idle geometry sensing device 302) and map information (as received and / or provided by the mapper 304). For example, the idle geometry sensing device 302 may detect the presence of intersections by live (e.g., real-time) perception. The map may also have intersections annotated in the map data. In some embodiments, the map and geometry data may be fused to generate instances of intersections if they exist in any one of the sources, and to link detections if they exist in both sources. Similarly, entry lines can be detected in real time (via the standby geometry sensing device 302) and / or provided in the map (via the mapper 304) based at least on a previous map stream that includes live detection or statistics of actual stopping points. Competitor areas, interior ground, route geometry and conflict points are all entities that can be detected in real time and further inserted into the map stream for the benefit of future driving. Actual driving routes ("de facto lane graph") can also be extracted from multiple driving routes. Thus, standby geometry fusion can coordinate and associate multiple sources to provide clear standby geometry for further processing.

[0048] The signal fuser 328 can “fuse” or combine signal data from the signal perception device 306 with map data from the mapper 304 through a process called signal fusion. Signal fusion may provide the option to modulate information from live perception (via real-time perception of the signal perception device 306) with information from the map (received and / or provided by the mapper 304), for example, to assist live perception in demonstrating the presence of traffic lights, signs, or intersection types. Similar to geometry fusion, signal fusion may be optional in some embodiments. Such signal fusion may also provide the possibility of performing state estimations regarding traffic lights that are difficult to detect, and / or results from real-time signal perception.

[0049] The signal state estimation device 330 is responsible for a process generally referred to as signal state estimation. Signal state estimation may determine and / or provide the state of traffic signals, gestures of traffic-directing police officers, flag-waving "stop" or "slow down" signs, the state of streetlights at crosswalks, or the location of barriers blocking traffic. The result is typically a selection from an enumerated class (e.g., blue / yellow / red) or a combination thereof.

[0050] The geometry classifier 324 is generally responsible for the geometry classification process. Geometry classification sorts the waiting geometries into distinct classes. For example, the geometry classifier 324 may classify both self-directed paths (e.g., as encoded in self-directed waiting geometry 312) and competitor paths (e.g., as encoded in competitor waiting geometry 314) into categories such as, but not limited to, the following: “left turn,” “straight,” “right turn,” “U-turn,” and similar. This classification may be performed to standardize the paths so that common language rules, such as “a right turn on a red light is not allowed in Manhattan,” can be applied to the paths in the race state resolution process performed by the race state resolver 340. To apply such rules, paths may be divided into classifications that include common language (e.g., “right turn”). It should be noted that such classifications can benefit from the context of other paths (e.g., a relatively straight shape may be a right turn if it is the far right path, while it may not be a right turn if, in addition, a very sharp right turn exists). If either the user or a competitor is coming from the right, the geometric classification may also be determined for the waiting element.

[0051] Geometry classification can also be applied to pairs of lanes. For example, geometry classifier 324 can determine whether two lanes (e.g., your own lane and a competitor's lane) intersect or merge (if not explicitly given in the lane graph), where the point of conflict is, and which lane is coming from the right (to support the right-hand priority rule, which is generally applied in Europe and in the United States in some cases). This can be done by checking at the point of conflict whether the direction vectors of the two lanes are distinctly different from parallel, and if so, a marker for the 2D vector cross product between them (a marker for the determinant applied to a 2x2 matrix formed by the own lane direction vector stacked as the top layer and the competitor's lane direction vector as the bottom layer). If the marker is positive, the competitor is coming from the right. Note that this definition may imply that, with respect to your own left turn through approaching traffic, the approaching traffic is considered to be coming from the right (as this is the case at the point of conflict). If the direction vectors are nearly parallel at the point of conflict (as they usually merge), the vectors at the corresponding entry lines may be used.

[0052] The geometry classifier 324 may also determine or integrate from a live perception source (e.g., a standby geometry perception device 302) whether a competitor's path originates from a “stub road” (e.g., a private road, gas station, or parking lot), or whether it is “clearly larger” or “clearly smaller” than a “stub road.” That is, roads can be classified by the geometry classification process (e.g., as a “stub road”). The geometry classifier 324 may provide predicate-like classifications that allow logical rules to be applied to a coherent set of input variables. Some of this information may directly originate from the output of one or more neural networks or other machine learning models that are not geometry determinations, via a standby geometry data structure.

[0053] The geometry classifier 324 can also determine whether a route has attributes such as, but not limited to, the following: an explicit determination of whether the route intersects a line (for example, a route may be classified as "intersects a line"), "intersects a line". Such classifications can be used as a queue for several rules when the priority of how to make a ruling is unclear. For example, if two routes are competing or otherwise equivalent, but one intersects a line and the other does not, the route that does not intersect the line may have priority. A route that has a left turn through approaching traffic may have a classification of type attributes such as "intersects a dashed line", "intersects a solid line", or "intersects a double-solid yellow line", so that country-specific rules may be applied to determine whether this is permitted. The geometry classifier 324 can also set variables for waiting groups (and thereby for each waiting element) such as true or false for "intersection". This can be used to queue certain rules (for example, to distinguish how a crosswalk is treated depending on whether it is adjacent to an intersection or not).

[0054] The geometry association device 326 can associate paths and signals through a process called geometry association. Geometry association determines which signals apply to a path. This can answer questions such as, "Is this light close enough to this path to apply it?", "Is this light the closest / most suitable type for this path?", and "Is this sign intended to apply to this path?". Regarding light-versus-path association, for example, it can be difficult to know if a light applies to a left turn, and in such cases it may not be easily separable from the rule, as it is illuminated by a blue arrow that resolves the ambiguity. It should be noted that this analysis often benefits from all signals and paths being considered simultaneously. For example, a light offset to the right may apply to its own path in one case where no other paths exist, but not in another case where there is another path to the right. Similarly, a light pointing to the right may apply to its own path in one case where no other lights exist, but not in another case where there is a light directly above the path. Therefore, geometry association may take into account the entire scene and the state of the signals (or even the initial sensor data) as needed. For the same reason, the architecture may allow geometry classification performed by the geometry classifier 324 and geometry association performed by the geometry association device 326 to be performed in conjunction and to have access to standby geometry, signals and initial sensor data as well. In this sense, the process can assign links between signals, such as traffic lights and signs, and routes. In at least one embodiment, geometry association can first check whether a light or sign is within a distance that allows for a link to a route, and then, if it finds the nearest (in a sense) light or sign of each type to apply, it can determine the priority (for example, a left-turn light has the highest priority for left turns, but the nearest regular light is also applicable, although it has the second priority). Note that if a light changes state and resolves any form of ambiguity, the association may change instantaneously.

[0055] The outputs of the standby geometry fuser 322, mapper 304, signal state estimator 330, geometry classifier 324, geometry association device 326, and other sensor data receivers 308 may be supplied to the race state resolver 340 (as input). The race state resolver 340 may perform a process referred to as a race state resolution process. The goal of the race state resolution process may also be to provide a race state (e.g., race state 318) of the standby element 310 (and / or other standby elements). The race state 318 of the standby element 310 may also be a yielding instruction to the yielding planner regarding how its own vehicle should yield or proceed with respect to this standby element, as a matter of rules, expectations, formal or informal customs or norms. In some non-limiting embodiments, the race condition 318 may not indicate what is actually happening in the yielding scenario, what is physically possible in the yielding scenario, or whether the own vehicle has priority in the yielding scenario (e.g., an intersecting yielding scenario or a merging yielding scenario) but may be forced to yield. Rather, the race condition 318 may indicate what should happen according to convention. It may be the yield planner's responsibility to actually implement the yield, in the sense that the yield planner will consider what should happen (e.g., as encoded in the race condition 318), whether the own vehicle is actually in a place where it should stop and obeys that command, and whether other parties (e.g., competitors in the yielding scenario) appear to be fulfilling their expected yielding obligations and taking appropriate action. In other words, the yield planner may decide that the race condition is to proceed but the competitor is not yielding (essentially detecting "it is appropriate to honk"), and that it should not have happened but it is possible to decide to yield. A yielding planner can implement a yielding action that analyzes all conflicts within a waiting group until all waiting elements within the group can be cleared collectively. This means that all conflicts within a waiting group can be grouped together, and the most restrictive conflict can define the expected yielding action for its own vehicle.For example, if one race condition in a waiting group is to proceed down the road and stop at another entrance, the vehicle itself may remain at the entrance line. Race condition 318 may be analogous to race condition 210 in Figure 2. Various possible state values ​​for race condition 318 are discussed in conjunction with race condition 210.

[0056] The race state resolution process of the race state resolver 340 may be based on at least a base set of “road rules” (or “base rules”) that may vary on a country-by-country, state-by-state, region-by-region and similar basis. The base rules may also be basic logic rules applied to waiting geometries (e.g., self-waiting geometries 312 and competitor-waiting geometries 314) and signals after they have been reduced to the states of basic enumeration variables by a geometry classification process (e.g., via a geometry classifier 324) and a geometry association process (e.g., via a geometry association device 326). After their reduction, each waiting element may have a well-defined set of geometry classes and signal states applied to the waiting geometry. In addition to the base rules, the race state resolution process may use “map rules”. Map rules may include an array of pairs of propositions linked to waiting elements (e.g., waiting elements 310) and the states of the race.

[0057] Each proposition can be conditional on any number of signal states (it is also permissible for no signal states to be conditional). A race state and / or its proposition can be paired with a race state that would be shown if the proposition were to assess true. The semantics of the map rule may also be that each proposition is assessed in turn, with the first one to assess true defining the race state. A state already included in the wait element can be considered a default race state selected when none of the propositions assess true. The condition checker 342 of the race state resolver 340 may generally be responsible for performing such assessments and linking the propositions.

[0058] In at least one embodiment, the condition checker 342 may generally determine which signals are valid or invalid (or active or inactive) based on at least one or more perceived and / or determined conditions of the environment. For example, some signs may or may not be applicable depending on weather conditions (e.g., rain, snow, fog, wind), time of day, day of the week, presence of other signs (e.g., road work signs may take precedence over other signs), vehicle type of autonomous vehicle (e.g., car vs. truck), etc. For example, if there is a map rule with conditions that apply only from time X to time Y, the condition checker 342 may be used to determine and mark whether the map rule is currently applicable.

[0059] The map-rule matcher 346 is generally responsible for performing the map-rule matching process. The map-rule matching process can understand map rules and match their components (or data components) with items that actually exist and are determined to produce wait elements in the process. For example, the map-rule matcher 346 can take signals and other inputs that the condition checker 342 has decided to apply and reduce them to one or more map rules. For example, the map-rule matcher 346 may decide that one or more map rules apply based on the fact that the signal of the relevant traffic light is green, while the map-rule matcher 346 may not decide that any map rules apply if the traffic light signal is red (but may decide that different rules apply). These decisions may be based at least on the map-rule matcher 346 knowing that the condition is actually applied according to the decision of the condition checker 342 (e.g., the traffic light state).

[0060] To enhance the ability to take advantage of mutual exclusion constraints and to make holistic decisions in general, the map-rule matching process performed by the map-rule matcher 346 may begin with a process of establishing correspondences between routes, maps and idle geometries and signals within routes, and idle geometries and signals determined to actually exist by the condition checker 342. In some cases, these entities may originate from a map in a first location (for example, the idle element engine 300 is configured such that a lane graph is received by the map and considers one of the same routes and a map rule and whether it matches), so it should be noted that it may perform matching by "id" already established during lane graph fusion, idle geometry fusion, or signal fusion. However, for greater flexibility and generality, the map-rule matcher 346 may perform matching without using information from the map. For example, the map rule matcher 346 can use a lane graph from live perception (e.g., via a standby geometry perception device 302) to apply map rules from a map that associate traffic lights with their own paths, thereby avoiding the architectural complexity that would arise from having to propagate map identifier numbers from the beginning to the end of the lane graph, standby geometry, and signals. If a path, a standby geometry, or a signal is actually derived from the map, its geometry should be identical (or nearly identical if it has been successfully merged / mixed) so that the match should recover exactly. Matching can also match entities that are not exactly identical. For example, substantially similar left-turn shapes and arrangements may be expected to match (note again here that if two parallel left turns currently exist, the match benefits from considering them jointly). Thus, this process can essentially perform matching corresponding to a "map scene" having a "real scene" determined by any combination of live perception and map localization. The result may be a one-to-one correspondence between one subset of actual entities and one subset of map entities.

[0061] In various embodiments, each map rule may produce an output waiting element (e.g., waiting element 310 or its precursor) by matching all of its entities and resolving all of its signals. A large number of map rules may contain valid self-paths (since a large number of race conditions are conditionally implemented with respect to them and would not make sense without them). If a self-path does not find a match, there may be several reasons. If localization fails in a known way, then the map rule may not be used, and the failure of localization may be addressed in a different way. However, if localization is inaccurate, it may not match the self-path. Another possibility is that the self-path is inaccurate in the map or in the actual scene. Another possibility is that the path is too far away or blocked from live perception. In this case, a conservative approach may be to push the self-path into the scene. For this reason, self-paths from map rules that did not find a match may be added along with their waiting geometry. The same process may be applied to competitor paths. In fact, the entire set of wait elements from the map rules, excluding signals, can be considered individually, but correspondence can be important when considering the fusion of the map rules with the base rules performed by the wait element fuser 348. Signals, on the other hand, may have to be matched in order to resolve their states. Any state of an unmatched signal may be set to an unknown state, and propositions in the map rules can describe this possibility and assign an appropriate race state. This would generally require, for example, setting a race state to stop at entrance when the state of a single traffic light is unknown. This might also require using one of several synchronized traffic lights to resolve the same wait element, which defaults to stop at entrance only if all of them are unknown.In yet another case, this might require backing up to yield a point of contention with respect to an oncoming left turn, even if the blue U-turn arrow traffic signal light is not visible, but the blue circle is visible, and it is known that a U-turn is always permitted in that case (though it is not known whether it is protected). This example is aggressive, but the design achieves high flexibility without high complexity (another less flexible option would be to list the possible states and enumerate all possible combinations and find the most restrictive contention across all possibilities).

[0062] The base rule parser 344 is generally responsible for performing a base rule parsing process that can obtain propositional rules derived in two ways. Firstly, propositional rules can be derived from base rules through basic state variable estimation, geometry classification, and geometry association. Secondly, propositional rules can be derived from map rules. For example, the base rule parser 344 can take signals and other inputs determined to be applied by the condition checker 342 and decompose them into one or more base rules. In at least one embodiment, the base rule parser 344 may operate similarly to the map rule matcher 346 but apply general or universal rules and conditions of driving that are agnostic to the vehicle's history or observed behavior at conceding scenario locations.

[0063] Base rules may vary by country, state, region, and similar manner, but they can be consistently applied to each yield scenario, provided the corresponding conditions are met. In contrast, map rules may apply driving rules and conditions based at least on the vehicle's history or observed behavior at the yield scenario location or similar yield scenario locations. In at least one embodiment, map rules may be encoded in map data and applied at least on the basis of localizing autonomous vehicles to the map. However, base rules can be applied regardless of localization and yield scenario locations. By providing a base rule parser 344, wait elements can be generated even if map data is unavailable or cannot be applied to or localized to the current yield scenario. For example, the map rule matcher 346 may not be able to determine one or more components and / or elements of a wait element, and the base rule parser 344 can fill the gap, or vice versa. Therefore, waiting elements can be generated entirely from map data, entirely from perceptual data, or from a combination of both types of data.

[0064] The wait element fuser 348 is responsible for fusing or combining data corresponding to the analyzed base rule and matched map rule in order to resolve the conflict state 318. For example, in at least one embodiment, the base rule parser 344 and the map rule matcher 346 may each produce their respective wait elements and / or elements and / or components. The wait element fuser 348 may fuse any of these various embodiments to form the wait element 310. In at least one embodiment, one or more embodiments from the base rule parser 344 and the map rule matcher 346 may be inconsistent. For example, the same field or data element may have different values. The wait element fuser 348 may identify and / or detect such inconsistencies in order to determine the resolved value of the wait element 310. In at least one embodiment, the map rule matcher 346's determination may generally take precedence because the map rule matcher 346's determination is based on spatial relationships and perceptual data may not always be certain. For example, this can be useful when the base rule parser 344 is unable to derive the relevant rule that should be applied. For instance, if there is no sign or other visual indicator indicating that a left turn does not exist at an intersection, the base rule parser 344 may be unable to apply the corresponding base rule, even if the rule is followed by custom. However, the map rule matcher 346 may be able to apply the rule, at least based on observations of the rule's application from the history of autonomous vehicle driving through the intersection. However, there may be temporary or new signals that do not exist or are not reliable enough to be included in the map data (e.g., based on inconsistent or insufficient observations, old observations, etc.).The base rule parser 344 may be used to describe such scenarios when decomposing data into standby elements 310 (for example, for construction signs, electronic signs, or other temporary or transient signals, the corresponding decision by the base rule parser 344 may take precedence at least on the basis of being assigned to or associated with these types of signals).

[0065] Although not shown in Figure 3, the waiting element engine may optionally include a waiting obstacle analyzer that performs Obstacle Analysis in the Path (OIPA). OIPA can link parties to the path and waiting geometry using lane graphs, waiting geometry, and semantic motion segmentation (SMS) obstacle perception outputs. OIPA can be performed by rendering the path and area onto an index image, and then projecting the polygonal shapes of the parties onto the image and integrating the amount of overlap.

[0066] In addition, the standby element engine may include a blockage analyzer. Blockage analysis can provide blockage understanding to OIPA results by acquiring lane graphs and obstacle perception outputs such as SMS and depth diagrams and detecting segments in the lane graph that may conceal a blind party. This may allow the planner to consider blind parties as well as visible parties yielding. A competitor path may have expected velocity limits, and the blockage portion of that path may then be used to insert a blind party with its highest velocity in the nearest blocked configuration it could currently exist, with appropriate warnings about expected practices (for example, if the competition condition is that the first stop has priority, a blind vehicle well behind the competition area and entrance line should not be reasonably expected to be entering at maximum speed if it does not allow it to stop at that entrance line, whereas if the competition condition is yielding the competition point, it may be assumed that it could be entering at maximum speed). With this information, the yielding planner can then precisely determine actions such as slowing down, waiting, or moving slowly forward until the unseen vehicle can be identified and the obstructed area becomes clear, or until the traffic light turns green in the case of a right turn on a red light.

[0067] The standby element engine 300 may additionally perform an analysis of who stopped first. The analysis of who stopped first may determine which party stopped first to support a race condition, where the first to stop has priority, using motion analysis and OIPA results. OIPA results may be used to determine if the parties have entered the inner ground and are in their entrance competitor area and on the entrance line. Motion may also be used to determine if the parties are moving, have stopped but recently moved, or are possibly parked vehicles. Another analysis of the standby element engine 300 may include an analysis of who goes first. The analysis of who goes first may also be a machine learning analysis that estimates the race condition corresponding to each competitor. Such an analysis may be trained by a number of examples of future advances, where it can be determined whether a party has gone ahead past the race point before its own vehicle, or vice versa. Knowing the possibility of who goes first may be an indication of the expectation of yielding.

[0068] As described throughout this book, the output of the standby element 310 of the standby element engine 300 may be passed to the yield planner of the vehicle itself. Given the standby element with resolved race conditions, the OIPA result with a blockage, who stopped first, who goes first, and the obstacle perception output, the yield planner may implement the yielding behavior of the vehicle itself. The yield planner may, if necessary and possible, trigger yielding behavior in the vehicle itself and monitor the yielding of other parties as it proceeds down the road. A yield planner analysis may be performed to predict in advance what will happen if the vehicle itself proceeds down the requested path. If the vehicle itself is going to move forward and yield, it may clear the race before the race with priority is affected (e.g., forced to change their behavior). Yielding may include ensuring that the vehicle itself does not influence the race to change their behavior from their preferred or intended behavior.

[0069] Referring here to Figures 4-6, each block of Methods 400-600 and other methods described herein include computational processes that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Methods may also be performed as computer-usable instructions stored in a computer storage medium. Methods may be provided, to name a few, by standalone applications, services, or hosted services (standalone or in combination with another hosted service), or by plugging into another product. In addition, Methods 400-600 are described, as an example, with respect to the standby element engine 300 in Figure 3. However, these methods may be performed by any one system or any combination of systems, including but not limited to those described herein, in addition or by alternative methods.

[0070] Figure 4 is a flowchart illustrating a method 400 for encoding yielding scenarios for an autonomous vehicle (e.g., a self-vehicle) according to some embodiments of the present disclosure. The method may be implemented via a standby element engine, such as, but not limited to, the standby element engine 300 in Figure 3. In block B402, the method 400 includes detecting and / or identifying an approaching yielding scenario. The yielding scenario may correspond to an intersecting (or intersection) yielding scenario or a merging yielding scenario. The yielding scenario may involve an autonomous vehicle (e.g., a self-vehicle) and one or more competitors.

[0071] In block B404, geometry data may be received. In some embodiments, geometry data may be received in response to detecting a yielding scenario. Geometry data may include geometry perception data. For example, the standby geometry perception device 302 of the standby element engine 300 may receive real-time geometry data generated from one or more sensors of the autonomous vehicle.

[0072] In block B406, signal data may be received. In some embodiments, signal data may be received in response to detecting a yielding scenario. The signal data may include signal perception data. For example, the signal perception device 306 of the standby element engine 300 may receive real-time signal data generated from one or more sensors of the autonomous vehicle.

[0073] In block B408, map data may be received. In some embodiments, map data may be received in response to detecting a yield scenario. For example, the mapper 304 of the standby element engine 300 may receive map data.

[0074] In block B410, lane graph data may be received. In some embodiments, lane graph data may be received in response to the detection of a yielding scenario. The lane graph data may include one or more of the vehicle's own paths and one or more of the competitor's paths in one or more yielding scenarios. For example, the lane grapher 350 of the standby element engine 300 may receive the vehicle's own path 352 of its own vehicle (e.g., an autonomous vehicle) and the competitor's path 354 related to a yielding scenario or a competitor's path.

[0075] In block B412, the geometry of one of one or more self-directed paths can be determined. The geometry of a self-directed path can be determined via geometry data, map data, signal data, and / or lane graph data. As such, the standby geometry perception device 302, the mapper 304, the signal perception device 306, the lane grapher 350, or any combination thereof may generally be responsible for determining the geometry of a self-directed path. In some embodiments, the standby geometry fuser 322 of the standby element engine 300, the geometry classifier 324 of the standby element engine 300, the geometry association device 326 of the standby element engine 300, the signal fuser 328 of the standby element engine 300, the signal state estimation device 330 of the standby element engine 300, or any combination thereof may contribute to determining the geometry of a self-directed path.

[0076] Furthermore, in block B412, the geometry of one or more competitor paths of one or more competitors may be determined. Similar to one's own path, the geometry of competitor paths may be determined via geometry data, map data, signal data, and / or lane graph data. As such, the standby geometry perception device 302, mapper 304, signal perception device 306, lane grapher 350, or any combination thereof may generally be responsible for determining the geometry of competitor paths. In some embodiments, the standby geometry fuser 322 of the standby element engine 300, the geometry classifier 324 of the standby element engine 300, the geometry association device 326 of the standby element engine 300, the signal fuser 328 of the standby element engine 300, the signal state estimater 330 of the standby element engine 300, or any combination thereof may contribute to determining the geometry of competitor paths. In some embodiments, in block 414, the geometry of the path and / or yielding scenario context is determined.

[0077] In block B414, the geometry of the self-path and the competitor's path can be encoded. The geometry of the self-path can be encoded in the self-waiting geometry (for example, the self-waiting geometry 312 in Figure 3). The geometry of the competitor's path can be encoded in the competitor's waiting geometry (for example, the competitor's waiting geometry 314 in Figure 3). In at least one embodiment, the geometry of the path context and / or the yielding geometry are encoded in the context waiting geometry (for example, the context waiting geometry 316 in Figure 3).

[0078] In block B416, the state of conflict between the own path and the competitor's path can be determined at least based on the determined geometry. Various embodiments for determining the state of conflict will be discussed in conjunction with at least the standby element engine 300, method 500 in Figure 5, and / or method 600 in Figure 6. However, briefly here, the conflict state resolver 340 of the standby element engine 300 may generally be responsible for determining and / or resolving the state of conflict between the own path and the competitor's path. The state of conflict can be encoded (for example, the conflict state 318 in Figure 3).

[0079] In block B418, a wait element data structure (or data object), for example, wait element 310 in Figure 3, may be generated. The wait element data structure may include at least one of the following: its own path geometry, the competitor's path geometry, and the race state. The wait element may also include contextual wait geometry.

[0080] In B420, the standby element can be provided to the autonomous vehicle's planner.

[0081] Figure 5 is a flowchart illustrating a method 500 for encoding yielding scenarios of an autonomous vehicle (e.g., a self-vehicle) according to some embodiments of the present disclosure. The method may be implemented via a standby element engine, such as, but not limited to, the standby element engine 300 of Figure 3. In block B502, the method 500 includes perceiving standby geometry of the self-vehicle and competitor paths. A geometry perceiving device (e.g., a geometry perceiving device 302 of the standby element engine 300) may perceive the standby geometry. The standby geometry may be perceived in response to detecting and / or identifying yielding scenarios of the self-vehicle (autonomous vehicle). The self-vehicle and competitor paths may be perceived by a lane grapher (e.g., a lane grapher 350 of the standby element engine 300).

[0082] In block B504, one or more signals may be perceived for one's own / competitor's path. A signal perception device (e.g., signal perception device 306) may perceive the signals of the path.

[0083] In block B506, idle geometry can be fused with map data. An idle geometry fuser (for example, the idle geometry fuser 322 of idle element engine 300) can fuse idle geometry with map data.

[0084] In block B508, the fused idle geometry can be classified. A geometry classifier (for example, the geometry classifier 324 of the idle element engine 300) can classify the idle geometry.

[0085] In block B510, fused idle geometries can be associated. Geometry association devices (for example, the geometry association device 326 of the idle element engine 300) can associate idle geometries.

[0086] In block B512, signals can be fused with map data. A signal fuser (for example, the signal fuser 328 of the standby element engine 300) can fuse signals with map data.

[0087] In block B514, the state of the fused signals can be estimated. A signal state estimation device (for example, the signal state estimation device 330 of the standby element engine 300) can estimate the state of the fused signals.

[0088] In block B516, the state of conflict between the own path and the competitor's path can be resolved. Various embodiments of resolving the conflict state will be discussed in conjunction with at least method 600 in Figure 6. However, briefly here, a conflict state resolver (for example, the conflict state resolver 340 of the standby element engine 300) can resolve the state of conflict between the own path and the competitor's path.

[0089] In block B518, a standby element may be generated. For example, the standby element engine 300 may generate a standby element 310.

[0090] In block B520, the standby element may be provided to a system that provides guidance services to an autonomous vehicle (e.g., a self-driving vehicle) (e.g., a yielding planner).

[0091] Figure 6 is a flowchart illustrating a method 600 for resolving a conflict state between vehicle paths, according to some embodiments of the present disclosure. The method may be implemented via a conflict state resolver, such as, but not limited to, the conflict state resolver 340 in Figure 3. Various inputs may be provided to the conflict state resolver to resolve a conflict state between two paths (e.g., the vehicle's own path and the competitor's path). For example, as shown in Figure 3, map data of the two paths, as well as fused, classified, and / or associated geometry (e.g., standby geometry), may be provided to the conflict state resolver 340. In addition, fused signals (including estimated signal states) may be provided to the conflict state resolver. Various other sensor data (from sensors of the autonomous vehicle) may be provided to the conflict state resolver. Various blocks of the method 600 may use any of this input data.

[0092] Method 600 includes checking the conditions for a race condition in block B602. A condition checker (for example, a condition checker 342 of the race condition resolver 340) may be used to check the conditions for a race condition.

[0093] In block B604, one or more base rules can be analyzed. A base rule parser (for example, the base rule parser 344 of the race state resolver 340) can analyze the base rules.

[0094] In block B606, one or more map rules may match a conflict state. A map rule matcher (for example, map rule matcher 346 of the conflict state resolver 340) may match a conflict state to one or more map rules.

[0095] In block B608, various data structures (e.g., await geometry and race state) can be merged into a single await element. An await element fuzer (e.g., the await element fuzer 348 of the race state resolver 340) can merge data structures.

[0096] In block B610, the fused data structures (e.g., await geometry and resolved race conditions) can be packaged into await elements (e.g., await element 310 in Figure 3).

[0097] Exemplary autonomous vehicle Figure 7A shows an exemplary autonomous vehicle 700 according to some embodiments of the present disclosure. The autonomous vehicle 700 (as otherwise referred to herein as "Vehicle 700") may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, vehicles attached to trailers, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 700 may have the capability to perform functions at one or more levels of autonomous driving from Level 3 to Level 5. Vehicle 700 may have the capability to perform functions at one or more levels of autonomous driving from Level 1 to Level 5. For example, depending on the embodiment, the vehicle 700 may have the capability of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). In this specification, the term “autonomous” may include any and / or all types of autonomy of the vehicle 700 or any other machine, such as being fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, having auxiliary autonomy, semi-autonomous, primarily autonomous, or other designations.

[0098] Vehicle 700 may include components such as the vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. Vehicle 700 may include a propulsion system 750, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 750 may be connected to the vehicle 700's drivetrain, which may include a transmission, to enable the vehicle 700 to propel itself. The propulsion system 750 may be controlled in response to receiving signals from a throttle / accelerator 752.

[0099] A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (for example, along a desired course or route) when the propulsion system 750 is operating (for example, when the vehicle is moving). The steering system 754 may receive signals from the steering actuator 756. The steering wheel may also be an option for fully automated (level 5) functionality.

[0100] The brake sensor system 746 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 748 and / or the brake sensor.

[0101] The controller 736, which may include one or more system-on-a-chip (SoC) 704 (Figure 7C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 700. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 748, actuate the steering system 754 via one or more steering actuators 756, and actuate the propulsion system 750 via one or more throttle / accelerators 752. The controller 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 700. The controller 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functions (e.g., computer vision), a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 736 may handle two or more of the aforementioned functions, and two or more controllers 736 may handle a single function, and / or any combination thereof.

[0102] The controller 736 can provide signals for controlling one or more components and / or systems of the vehicle 700 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system sensors 758 (e.g., global positioning system sensors), RADAR sensors 760, ultrasonic sensors 762, LIDAR sensors 764, inertial measurement unit (IMU) sensors 766 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 796, stereo cameras 768, wide-view cameras 770 (e.g., fisheye cameras), infrared cameras 772, surround cameras 774 (e.g., 360-degree cameras), long-range and / or medium-range cameras 798, speed sensors 744 (e.g., for measuring the speed of a vehicle 700), vibration sensors 742, steering sensors 740, brake sensors (e.g., as part of a brake sensor system 746), and / or other sensor types.

[0103] One or more of the controllers 736 may receive inputs (represented, for example, by input data) from the instrument cluster 732 of the vehicle 700 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 734, an audible annunciator, a loudspeaker, and / or other components of the vehicle 700. The outputs may include information such as vehicle velocity, speed, time, map data (e.g., HD map 722 in Figure 7C), location data (e.g., the location of the vehicle 700, such as on a map), direction, the location of other vehicles (e.g., occupied grid), and information about objects and the status of objects as perceived by the controller 736. For example, the HMI display 734 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.).

[0104] Vehicle 700 further includes a network interface 724 that can communicate over one or more networks using one or more wireless antennas 726 and / or a modem. For example, the network interface 724 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antennas 726 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWAN) such as LoRaWAN, SigFox.

[0105] Figure 7B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 700 of Figure 7A, according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different locations on the vehicle 700.

[0106] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the vehicle 700. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0107] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0108] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0109] A camera having a field of view that includes a portion of the environment in front of the vehicle 700 (e.g., a forward-facing camera) may be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 736 and / or control SoCs, to help provide information essential for generating an occupied grid and / or determining a preferred vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as Lane Departure Warning ("LDW"), Autonomous Cruise Control ("ACC"), and / or traffic sign recognition.

[0110] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 770, which can be used to capture objects entering the view from the periphery (e.g., pedestrians, intersecting traffic, or bicycles). Although only one wide-view camera is shown in Figure 7B, any number of wide-view cameras 770 may be present in the vehicle 700. In addition, long-range cameras 798 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 798 may also be used for object detection and classification, as well as basic object tracking.

[0111] One or more stereo cameras 768 may also be included in a forward-facing configuration. The stereo camera 768 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including distance estimates for all points in the image. An alternative stereo camera 768 may include a compact stereo vision sensor comprising two camera lenses (one on the left and one on the right) and an image processing chip capable of measuring the distance from the vehicle to an object and activating autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 768 may be used in addition to or instead of those described herein.

[0112] A camera having a field of view including a portion of the environment on the sides of the vehicle 700 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 774 (e.g., four surround cameras 774 as shown in Figure 7B) may be positioned on the vehicle 700. The surround cameras 774 may include wide-view cameras 770, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the vehicle. In an alternative configuration, the vehicle may use three surround cameras 774 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0113] A camera having a field of view that includes a portion of the environment behind the vehicle 700 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 798, stereo cameras 768), infrared cameras 772, etc., as described herein.

[0114] Figure 7C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 700 of Figure 7A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0115] Each component, feature, and system of the vehicle 700 in Figure 7C is illustrated as being connected via a bus 702. Bus 702 may include a Controller Area Network (CAN) data interface (or, referred to as the "CAN bus"). CAN may also be a network within the vehicle 700 used to help control various features and functions of the vehicle 700, such as the operation of brakes, acceleration, steering, and windshield wipers. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0116] Bus 702 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or as an alternative to, a CAN bus. In addition, a single line is used to represent bus 702, but this is not intended to limit it. There may be any number of buses 702, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 may be used to perform different functions and / or for redundancy. For example, a first bus 702 may be used for collision avoidance functions, and a second bus 702 may be used for operation control. In any example, each bus 702 may communicate with any of the components of vehicle 700, and two or more buses 702 may communicate with the same component. In some examples, each SoC 704, each controller 736, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 700) and may be connected to a common bus such as a CAN bus.

[0117] The vehicle 700 may include one or more controllers 736, such as those described herein with respect to Figure 7A. The controllers 736 may be used for a variety of functions. The controllers 736 may be connected to any of the various other components and systems of the vehicle 700 and may be used for the control of the vehicle 700, the artificial intelligence of the vehicle 700, infotainment for the vehicle 700, and / or the like.

[0118] Vehicle 700 may include a system-on-a-chip (SoC) 704. The SoC 704 may include a CPU 706, a GPU 708, a processor 710, a cache 712, an accelerator 714, a data store 716, and / or other components and features not shown. The SoC 704 may be used to control vehicle 700 in various platforms and systems. For example, the SoC 704 may be coupled in a system (e.g., a system of vehicle 700) that has an HD map 722 that can obtain map refreshes and / or updates via a network interface 724 from one or more servers (e.g., server 778 in Figure 7D).

[0119] The CPU 706 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU 706 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 706 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 706 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 706 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU 706 clusters to be active at any given time.

[0120] The CPU706 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU706 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0121] The GPU708 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU708 may be programmable and efficient for parallel workloads. In some embodiments, the GPU708 may be able to use an enhanced tensor instruction set. The GPU708 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU708 may include at least eight streaming microprocessors. The GPU708 may be able to use a Computation Application Programming Interface (API). In addition, the GPU708 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0122] The GPU708 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU708 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU708 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0123] In some examples, the GPU708 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0124] The GPU708 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support may be used to allow the GPU708 to directly access the CPU706 page table. In such examples, when the GPU708 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU706. In response, the CPU706 can examine its page table for virtual-to-real-address mapping and send the translation back to the GPU708. As such, unified memory technology can enable a single, unified virtual address space for both the CPU706 and GPU708 memory, thereby simplifying GPU708 programming and porting of applications to the GPU708.

[0125] In addition, the GPU708 may include an access counter that can record how often the GPU708 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0126] The SoC704 may include any number of caches 712, including those described herein. For example, cache 712 may include an L3 cache available to both the CPU706 and the GPU708 (e.g., connected to both the CPU706 and the GPU708). Cache 712 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used.

[0127] The SoC704 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 700 (for example, a processing DNN). In addition, the SoC704 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU706 and / or GPU708.

[0128] The SoC704 may include one or more accelerators 714 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC704 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU708 and to offload some of the GPU708's tasks (e.g., to free up more cycles of the GPU708 to perform other tasks). As an example, accelerator 714 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0129] Accelerator 714 (e.g., hardware acceleration cluster) may include a deep learning accelerator (DLA). A DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. A DLA design can provide more performance per millisecond than a general-purpose GPU and significantly exceed CPU performance. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0130] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0131] DLA can perform any function of GPU708, and by using inference accelerators, for example, a designer can target either DLA or GPU708 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU708 and / or other accelerators 714.

[0132] The accelerator 714 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0133] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may contain any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0134] DMA can enable PVA components to access system memory independent of the CPU 706. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0135] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0136] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can run the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0137] The accelerator 714 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 714. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0138] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0139] In some embodiments, the SoC704 may include a real-time ray tracing hardware accelerator, such as the one described in Patent Document 1. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0140] The accelerator 714 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The PVA's capabilities are suitable for areas of algorithms requiring predictable processing at low power and low latency. In other words, the PVA performs well in semi-high density or high density typical computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, because the PVA is efficient in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms.

[0141] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0142] In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for flight depth processing, for example, by processing raw flight data to provide processed flight data.

[0143] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detection should be considered as a trigger for the AEB. DLA can run a neural network that regresses on the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), vehicle orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation of an object acquired from the neural network and / or other sensors (e.g., LIDAR sensor 764 or RADAR sensor 760), and others.

[0144] The SoC704 may include a data store 716 (for example, memory). The data store 716 may also be the on-chip memory of the SoC704 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 716 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 712 may comprise an L2 or L3 cache 712. References to the data store 716 may include references to memory associated with the PVA, DLA, and / or other accelerators 714, as described herein.

[0145] The SoC704 may include one or more processors 710 (e.g., integrated processors). The processors 710 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC704 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, assistance with system low-power state transitions, management of SoC704 thermal and temperature sensors, and / or management of SoC704 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC704 may use the ring oscillators to detect the temperatures of the CPU 706, GPU 708, and / or accelerator 714. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine, placing the SoC704 into a lower power state and / or putting the vehicle 700 into chauffeur safe shutdown mode (for example, safely shutting down the vehicle 700).

[0146] The processor 710 may further include a set of integrated processors that can perform the functions of an audio processing engine. The audio processing engine may also be an audio subsystem that enables full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0147] The processor 710 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0148] The processor 710 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic to detect any differences between their operations.

[0149] The processor 710 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0150] The processor 710 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0151] The processor 710 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 770, the surround camera 774, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0152] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0153] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU708 is not required to continuously render new surfaces. Even when the GPU708 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU708 to improve performance and responsiveness.

[0154] The SoC704 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC704 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0155] The SoC704 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC704 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., Lidar sensor 764, Radar sensor 760, etc., which may be connected via Ethernet®), data from bus 702 (e.g., vehicle speed, steering wheel position, etc.), and data from GNSS sensor 758 (connected, for example, via Ethernet® or CAN bus). The SoC704 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to free up CPU 706 from routine data management tasks.

[0156] The SoC704 may also be an inter-terminal platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and, together with deep learning tools, provides a platform for a flexible, reliable driving software stack. The SoC704 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 714 is coupled with the CPU 706, the GPU 708, and the data store 716 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0157] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0158] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU720) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs, and passing that semantic understanding to a route planning module running on the CPU complex.

[0159] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network that informs the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU708, for example.

[0160] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of vehicle 700. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, SoC704 provides security against theft and / or vehicle hijacking.

[0161] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 796. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 704 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor 758. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, a control program may be used, with the assistance of ultrasonic sensor 762, to perform emergency vehicle safety routines such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle until the emergency vehicle has passed.

[0162] The vehicle may include a CPU 718 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 704 via a high-speed interconnect (e.g., PCIe). The CPU 718 may include, for example, an x86 processor. The CPU 718 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 704, and / or monitoring the status and condition of the controller 736 and / or the infotainment SoC 730.

[0163] Vehicle 700 may include a GPU 720 (e.g., a separate GPU, or dGPU) which can be connected to SoC 704 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 720 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from sensors in Vehicle 700 (e.g., sensor data).

[0164] Vehicle 700 may further include a network interface 724 which may include one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 724 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with a server 778 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct link may be established between two vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 700 information about vehicles in close proximity to vehicle 700 (e.g., vehicles in front of, beside, and / or behind vehicle 700). This functionality may also be part of the vehicle 700's joint adaptive cruise control functionality.

[0165] The network interface 724 may include an SoC that provides modulation and demodulation functions and enables the controller 736 to communicate over a wireless network. The network interface 724 may include a radio frequency front end for baseband-to-radio frequency upconversion and radio frequency-to-baseband downconversion. Frequency conversion can be performed through well-known processes and / or using superheterodyne processes. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0166] The vehicle 700 may further include a data store 728 which may include storage outside the chip (for example, outside the SoC 704). The data store 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0167] The vehicle 700 may further include GNSS sensors 758. The GNSS sensors 758 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 758 may be used, including, but not limited to, GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0168] Vehicle 700 may further include a RADAR sensor 760. The RADAR sensor 760 may be used by vehicle 700 for long-range vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 760 may use CAN and / or bus 702 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 760) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 760 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0169] The RADAR sensor 760 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 760 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record the area around the vehicle 700 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of vehicles entering or leaving the lane of the vehicle 700.

[0170] As an example, a medium-range RADAR system may include a range of up to 760m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 750 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the vehicle.

[0171] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0172] The vehicle 700 may further include ultrasonic sensors 762. Positioned on the front, rear, and / or sides of the vehicle 700, the ultrasonic sensors 762 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 762 may be used, and different ultrasonic sensors 762 may be used for detection at different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 762 may operate at a functional safety level of ASIL B.

[0173] The vehicle 700 may include a LiDAR sensor 764. The LiDAR sensor 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 764 may also have a functional safety level of ASIL B. In some examples, the vehicle 700 may include multiple LiDAR sensors 764 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0174] In some examples, the LIDAR sensor 764 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 764 may have an advertised range of approximately 700m, for example, with an accuracy of 2cm to 3cm and support for 700Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 764 may be used. In such examples, the LIDAR sensor 764 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a vehicle 700. In such examples, the LIDAR sensor 764 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 764 may be configured for a horizontal field of view between 45 and 135 degrees.

[0175] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around the vehicle up to approximately 200m. The flash LiDAR unit includes a receptor that records the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 764 can be made less susceptible to motion blur, vibration, and / or shock.

[0176] The vehicle may further include an IMU sensor 766. In some examples, the IMU sensor 766 may be positioned in the center of the rear axle of the vehicle 700. The IMU sensor 766 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 766 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 766 may include an accelerometer, a gyroscope, and a magnetometer.

[0177] In some embodiments, the IMU sensor 766 may be implemented as a miniature, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 766 may enable the vehicle 700 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 766. In some embodiments, the IMU sensor 766 and the GNSS sensor 758 may be combined in a single integrated unit.

[0178] The vehicle may include a microphone 796 placed inside and / or around the vehicle 700. The microphone 796 may, among other things, be used for emergency vehicle detection and identification.

[0179] The vehicle may further include any number of camera types, including a stereo camera 768, a wide-view camera 770, an infrared camera 772, a surround camera 774, a long-range and / or medium-range camera 798, and / or other camera types. The cameras may be used to capture image data around the entire exterior of the vehicle 700. The type of camera used will depend on the embodiment and requirements of the vehicle 700, and any combination of camera types may be used to achieve the required coverage around the vehicle 700. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 7A and 7B.

[0180] The vehicle 700 may further include a vibration sensor 742. The vibration sensor 742 can measure vibrations of vehicle components, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 742 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0181] Vehicle 700 may include ADAS system 738. In some examples, ADAS system 738 may include SoC. ADAS system 738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0182] The ACC system may use a radar sensor 760, a lithium-ion sensor 764, and / or a camera. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of vehicle 700 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 700 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LCA and CWS.

[0183] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through a network interface 724 and / or a wireless antenna 726, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., a vehicle in the same lane as vehicle 700, immediately in front of vehicle 700), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 700, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0184] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 760, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual, vibration, and / or quick brake pulses.

[0185] An AEB system can detect an imminent forward collision with another vehicle or object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking.

[0186] The LDW system warns the driver when the vehicle 700 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0187] The LKA system is a modified version of the LDW system. The LKA system provides steering input or braking to correct the vehicle 700 if it begins to drift out of its lane.

[0188] The BSW system detects and warns the driver of a vehicle in its blind spots. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system may provide additional warnings when the driver uses the turn signals. The BSW system may use a rear-facing camera and / or radar sensor 760, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0189] The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 700 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0190] Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgments that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 700, however, if the results are contradictory, the vehicle 700 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 736 or the second controller 736). For example, in some embodiments, the ADAS system 738 may also be a backup and / or secondary computer to provide perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 738 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0191] In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0192] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer is providing a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based forward crossing (FCW) system, the neural network in the supervisory MCU can learn when the FCW is identifying metal objects that are not actually dangerous, such as sewer grates or manhole covers that trigger an alarm. Similarly, when the secondary computer is a camera-based lane departure warning (LDW) system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC704.

[0193] In other examples, ADAS system 738 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0194] In some examples, the output of the ADAS system 738 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 738 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0195] Vehicle 700 may further include an infotainment SoC 730 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 730 may include a combination of hardware and software that can be used to provide vehicle 700 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other vehicle-related information). For example, the infotainment SoC 730 may also include wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 734, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 730 may be further used to provide information (for example, visual and / or audible) to the vehicle user, such as information from the ADAS system 738, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0196] The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and / or components of the vehicle 700 via bus 702 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 730 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 736 (e.g., the vehicle 700's primary and / or backup computer) failure. In such examples, the infotainment SoC 730 can put the vehicle 700 into a chauffeur-safe stop mode as described herein.

[0197] Vehicle 700 may further include an instrument cluster 732 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 732 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 732 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 730 and the instrument cluster 732. In other words, the instrument cluster 732 may be included as part of the infotainment SoC 730, and vice versa.

[0198] Figure 7D is a system diagram of communication between the cloud-based server of Figure 7A and an exemplary autonomous vehicle 700, according to some embodiments of the present disclosure. System 776 may include a server 778, a network 790, and a vehicle including the vehicle 700. Server 778 may include a plurality of GPUs 784(A) to 784(H) (collectively referred to herein as GPU 784), PCIe switches 782(A) to 782(H) (collectively referred to herein as PCIe switch 782), and / or CPUs 780(A) to 780(B) (collectively referred to herein as CPU 780). The GPUs 784, CPUs 780, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 788 and / or PCIe connections 786 developed by NVIDIA. In some examples, the GPU784 is connected via NVLink and / or NVSwitch SoCs, and the GPU784 and PCIe switch 782 are connected via PCIe interconnects. Eight GPU784s, two CPU780s, and two PCIe switches are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 778 may contain any number of GPU784s, CPU780s, and / or PCIe switches. For example, server 778 may contain eight, sixteen, thirty-two, and / or more GPU784s, respectively.

[0199] Server 778 can receive image data from vehicles via network 790, representing images showing unexpected or altered road conditions, such as recently started road construction. Server 778 can transmit map information 794, including information about traffic and road conditions, to vehicles via network 790, including information about neural networks 792, updated neural networks 792, and / or map information 794. Updates to map information 794 may include updates to HD maps 722, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural networks 792, updated neural networks 792, and / or map information 794 may have arisen from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (for example, using server 778 and / or other servers).

[0200] Server 778 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a vehicle and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, it may be used by the vehicle (for example, transmitted to the vehicle via network 790), and / or the machine learning model may be used by server 778 to remotely monitor the vehicle.

[0201] In some examples, Server 778 can receive data from vehicles and apply it to state-of-the-art real-time neural networks for real-time intelligent inference. Server 778 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 784, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 778 may include deep learning infrastructure that uses only CPU-powered data centers.

[0202] The deep learning infrastructure of server 778 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware within vehicle 700. For example, the deep learning infrastructure can receive periodic updates from vehicle 700, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by vehicle 700. If the results do not match and the infrastructure concludes that the AI ​​within vehicle 700 is not functioning correctly, server 778 can send a signal to vehicle 700 instructing the vehicle's fail-safe computer to infer control, notify passengers, and complete a safe parking operation.

[0203] For inference, server 778 may include GPU 784 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when performance is not a major requirement, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0204] Exemplary computing devices Figure 8 is a block diagram of an example of a computing device 800 suitable for use in implementing some embodiments of the present disclosure. The computing device 800 may include an interconnection system 802 that indirectly or directly connects the following devices: memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply device 816, one or more presentation components 818 (e.g., a display), and one or more logical units 820. In at least one embodiment, the computing device 800 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 808 may include one or more vGPUs, one or more of the CPUs 806 may include one or more vCPUs, and / or one or more of the logical units 820 may include one or more virtual logical units. As such, the computing device 800 may include individual components (e.g., an entire GPU dedicated to the computing device 800), virtual components (e.g., a portion of a GPU dedicated to the computing device 800), or a combination thereof.

[0205] The various blocks in Figure 8 are shown connected by lines via the interconnection system 802, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 818, such as a display device, could be considered an I / O component 814 (for example, if the display is a touchscreen). As another example, the CPU 806 and / or GPU 808 may include memory (for example, memory 804 may represent a storage device in addition to the memory of the GPU 808, CPU 806, and / or other components). In other words, the computing devices in Figure 8 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 8 and are therefore not distinguished.

[0206] The interconnection system 802 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 802 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 806 may be directly connected to the memory 804. Furthermore, the CPU 806 may be directly connected to the GPU 808. Where direct or point-to-point connections exist between components, the interconnection system 802 may include PCIe links to implement the connections. In these examples, the PCI bus does not need to be included in the computing device 800.

[0207] Memory 804 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 800. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media.

[0208] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 800. In this specification, computer storage media does not include signals themselves.

[0209] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0210] The CPU 806 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to execute one or more of the methods and / or processes described herein. The CPU 806 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads concurrently. The CPU 806 may include any type of processor, and depending on the type of computing device 800 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 800, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 800 may include one or more CPUs 806 in one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0211] In addition to or instead of the CPU 806, the GPU 808 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to execute one or more of the methods and / or processes described herein. One or more of the GPU 808 may be an integrated GPU (for example, with one or more of the CPU 806), and / or one or more of the GPU 808 may be a discrete GPU. In embodiments, one or more of the GPU 808 may be a coprocessor of one or more of the CPU 806. The GPU 808 may be used by the computing device 800 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPU 808 may be used for GPU-based general-purpose computing (GPGPU). It can be used for a GPU. The GPU808 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU808 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU806 received via the host interface). The GPU808 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 804. The GPU808 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU808 can generate pixel data or GPGPU data for different parts of an output or for different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0212] In addition to or instead of the CPU 806 and / or GPU 808, the logic unit 820 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 800 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 806, GPU 808, and / or logic unit 820 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logic units 820 may be part of and / or integrated with one or more of the CPU 806 and / or GPU 808, and / or one or more of the logic units 820 may be discrete components of the CPU 806 and / or GPU 808 or otherwise external to them. In embodiments, one or more of the logic units 820 may be coprocessors of one or more of the CPU 806 and / or one or more of the GPU 808.

[0213] Examples of logical unit 820 include one or more processing cores and / or components thereof, such as a Data Processing Unit (DPU), Tensor Core (TC), Tensor Processing Unit (TPU), Pixel Visual Core (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC), Streaming Multiprocessor (SM), Tree Traversal Unit (TTU), Artificial Intelligence Accelerator (AIA), and Deep Learning Accelerator (DLA). This includes an Accelerator, a Logical Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), input / output (I / O) elements, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or similar.

[0214] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 800 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 810 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to directly transmit data received via the network and / or via the interconnection system 802 to one or more GPUs 808 (e.g., their memory).

[0215] I / O port 812 can enable the computing device 800 to be logically connected to other devices, including I / O components 814, presentation components 818, and / or other components, some of which can be built into (e.g., integrated into) the computing device 800. Exemplary I / O components 814 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 814 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 800 (as described in more detail below). The computing device 800 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 800 may include accelerometers or gyroscopes to enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 800 to render immersive augmented reality or virtual reality.

[0216] The power supply device 816 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 816 can provide power to the computing device 800 to enable the components of the computing device 800 to operate.

[0217] The presentation component 818 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 818 can receive data from other components (e.g., GPU 808, CPU 806, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0218] Exemplary data center Figure 9 shows an exemplary data center 900 that may be used in at least one embodiment of the present disclosure. The data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and / or an application layer 940.

[0219] As shown in Figure 9, the data center infrastructure layer 910 may include a resource orchestrator 912, grouped computing resources 914, and node computing resources ("node CRs") 916(1) to 916(N), where "N" represents any integer or natural number. In at least one embodiment, the node CRs 916(1) to 916(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules. In some embodiments, one or more nodes CR916(1) to 916(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR916(1) to 9161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR916(1) to 916(N) may correspond to a virtual machine (VM).

[0220] In at least one embodiment, the grouped computing resources 914 may include a separate group of nodes CR916 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR916 within the grouped computing resources 914 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR916, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0221] The resource orchestrator 912 can configure or otherwise control one or more nodes CR916(1) to 916(N) and / or grouped computing resources 914. In at least one embodiment, the resource orchestrator 912 may include a software design infrastructure (SDI) management entity of the data center 900. The resource orchestrator 912 may include hardware, software, or any combination thereof.

[0222] In at least one embodiment, as shown in Figure 9, the framework layer 920 may include a job scheduler 933, a configuration manager 934, a resource manager 936, and / or a distributed file system 938. The framework layer 920 may include a framework to support software 932 of the software layer 930 and / or one or more applications 942 of the application layer 940. The software 932 or application 942 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. The framework layer 920 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 938 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 933 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 900. The configuration manager 934 may have the ability to configure different layers, for example, a software layer 930 and a framework layer 920 including Spark and a distributed file system 938 to support large-scale data processing. The resource manager 936 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 938 and the job scheduler 933. In at least one embodiment, the clustered or grouped computing resources may include computing resources 914 grouped in the data center infrastructure layer 910. The resource manager 936 can coordinate with the resource orchestrator 912 to manage these mapped or allocated computing resources.

[0223] In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least a portion of nodes CR916(1) to 916(N), grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0224] In at least one embodiment, the application 942 included in the application layer 940 may include one or more types of applications used by at least a portion of the nodes CR916(1) to 916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0225] In at least one embodiment, any of the configuration manager 934, resource manager 936, and resource orchestrator 912 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 900 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0226] The data center 900 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to the data center 900. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to the data center 900 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0227] In at least one embodiment, the data center 900 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0228] Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 800 in Figure 8, for example, each device may include similar components, features, and / or functionalities of the computing device 800. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 900, examples of which are further detailed herein with respect to Figure 9.

[0229] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0230] Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0231] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In the embodiment, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0232] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0233] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 800 described herein with respect to Figure 8. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0234] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0235] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0236] The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, expressed.

Claims

1. A method that is performed by one or more circuits provided in a processor, A step of identifying a scenario for the first vehicle, at least based on analyzing perceptual data generated by at least one sensor of the first vehicle in the environment, A step of determining, based at least on the perceptual data, the first path of the first vehicle in the scenario and the second path of the second vehicle in the scenario, wherein the second path includes at least one point of conflict with the first path. A step of determining one or more traffic rules from the set of traffic rules applicable to the scenario, based on the perceptual data and the environment corresponding to the set of traffic rules, A step of assigning a conflict state to at least one conflict point based on at least one of the traffic rules from the set of traffic rules applied to the scenario, A step of generating a waiting element related to the scenario, wherein the waiting element encodes a first geometry related to the first path, a second geometry related to the second path, and a representation of the state of the conflict. A step of providing the standby element to the control agent of the first vehicle, wherein the control agent is configured to use the standby element to determine a yielding action for the first vehicle. Includes, The perceptual data includes first signal information relating to one or more detected traffic signals in the environment, and the method further The steps include receiving map data which includes second signal information relating to one or more localized traffic signals from at least one map, A step of generating fused signal information related to at least one traffic signal, based at least on fusing the first signal information with the second signal information. A method comprising the step of determining one or more traffic rules from the set of traffic rules applicable to the scenario, wherein the step is based at least on the fused signal information.

2. The method according to claim 1, further comprising the step of detecting the current state of one or more traffic signals included in the environment using the perceptual data, wherein the step of determining one or more traffic rules from a set of traffic rules applicable to the scenario is further based on the current state of the one or more traffic signals.

3. The perceptual data further includes first geometric information relating to the first path, and the map data further includes second geometric information relating to the first path. The method according to claim 1, further comprising the step of generating a representation of the first geometry of the first path, at least based on fusing the first geometric information with the second geometric information.

4. The method according to claim 1, wherein the perceptual data includes geometric perceptual data, and the method further includes a step of classifying the first path as belonging to at least one of a set of predetermined classes, based at least on the geometric perceptual data, and the step of determining the one or more traffic rules of a set of traffic rules applicable to the scenario is further based on the first path belonging to at least one class.

5. The method according to claim 1, wherein the perceptual data includes signal perception data relating to one or more traffic signals and geometry data relating to the first route, and the method further includes the step of assigning the one or more traffic signals to the first route, at least on the basis of using the signal perception data and the geometry data to determine the distance between the one or more traffic signals and the first route.

6. The method according to claim 1, wherein the set of traffic rules includes at least one of one or more map rules relating to localizing the first vehicle to at least one map relating to the geographical area of ​​the first vehicle or one or more base rules.

7. Identifying a scenario for the first vehicle based at least on analyzing perceptual data generated by at least one sensor of the first vehicle in the environment, The first path of the first vehicle in the scenario, and the second path of the second vehicle in the scenario, which includes at least one point of conflict with the first path, are determined at least based on localizing the first vehicle to one or more maps. One or more of the traffic rules applicable to the above scenario are determined at least based on the localization of the first vehicle, A conflict state is assigned to the at least one conflict point based on at least one of the traffic rules applicable to the scenario described above. A waiting element related to the scenario generates a waiting element that encodes a first geometry related to the first path, a second geometry related to the second path, and a representation of the state of the conflict. The control agent of the first vehicle, which is configured to use the standby element to determine a yielding action of the first vehicle, is provided with the standby element. One or more circuits for A processor comprising, The perceptual data includes first signal information relating to one or more detected traffic signals in the environment, and the one or more circuits further Based at least on the aforementioned localization, map data of one or more maps is received, which includes map data relating to one or more localized traffic signals from the one or more maps, second signal information relating to one or more localized traffic signals. Based at least on fusing the first signal information with the second signal information, fused signal information related to at least one traffic signal is generated. A processor that determines one or more of the traffic rules applicable to the scenario, based on at least the fused signal information.

8. The processor according to claim 7, wherein the one or more circuits further detect the current state of one or more traffic signals included in the one or more maps using the perceptual data, and determining one or more of the traffic rules applicable to the scenario is further based on the current state of the one or more traffic signals.

9. The map data further includes first geometric information relating to the first path, and the perception data further includes second geometric information relating to the first path. The processor according to claim 7, wherein one or more of the circuits further generate a representation of at least the first geometry of the first path, at least based on fusing the first geometric information with the second geometric information.

10. The processor according to claim 7, wherein the perceptual data includes geometric data, and the one or more circuits further classify the first path as belonging to at least one of a set of predetermined classes based at least on the geometric data, and determining one or more of the traffic rules applicable to the scenario is further based on the first path belonging to at least one class.

11. The processor according to claim 7, wherein the perceptual data includes signal data associated with one or more traffic signals, and the one or more circuits further assign the one or more traffic signals to the first route, at least on the basis of assessing the distance between the one or more traffic signals and the first route using the signal data and geometry data associated with the first route.

12. The processor according to claim 7, wherein the traffic rules include one or more map rules relating to localizing the first vehicle to one or more base rules relating to at least one map or the geographical area of ​​the first vehicle.

13. Based at least on analyzing perceptual data generated using at least one sensor of the vehicle in the environment, one or more traffic rules are determined to be applied to a scenario including the generated path of the vehicle in the environment. Assign a state of conflict based on at least one or more traffic rules applicable to the aforementioned scenario, A waiting element related to the scenario generates a waiting element that encodes at least the first geometry related to the generated path and the state of the conflict, A control agent of the vehicle, configured to use the standby element to determine a yielding action of the vehicle, is provided with the standby element. One or more processing units for A system equipped with, The perceptual data includes first signal information relating to one or more detected traffic signals in the environment, and the one or more processing units further, Receive map data containing second signal information relating to one or more localized traffic signals from at least one map, Based at least on fusing the first signal information with the second signal information, fused signal information related to at least one traffic signal is generated. The system is configured to determine one or more traffic rules applicable to the aforementioned scenario, based at least on the fused signal information.

14. The system according to claim 13, wherein the one or more processing units further detect the current state of one or more traffic signals included in the environment, and the determination of one or more traffic rules to be applied to the scenario is further based on the current state of the one or more traffic signals.

15. The perceptual data further includes first geometric information related to the path, and the map data further includes second geometric information related to the path. The system according to claim 13, wherein one or more processing units further generate at least the first geometry of the path based at least on fusing the first geometry information with the second geometry information.

16. The system according to claim 13, wherein the perceptual data includes geometric perceptual data, and the one or more processing units further classify the paths as belonging to at least one of a set of predetermined classes based at least on the geometric perceptual data, and determining the one or more traffic rules to be applied to the scenario is further based on the paths belonging to the at least one class.

17. The one or more processing units described above, Control systems for autonomous or semi-autonomous machines Perceptual systems for autonomous or semi-autonomous machines, A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 13, which is included in at least one of the following.

Images