Method and system for a vehicle

Abstract

Methods and systems for vehicles are provided. Embodiments for operational envelope detection (OED) with situational assessment are disclosed. Embodiments herein relate to an operational envelope detector configured to receive, as input, information related to sensors of a system and information related to operational design domain (ODD) requirements. The OED then compares the information related to sensors of the system and the information related to ODD requirements and identifies whether the system is operating within its ODD or whether a remedial action is appropriate to adjust the ODD requirements based on current sensor information. Other embodiments are described and / or claimed.

Description

TECHNICAL FIELD

[0001] The present invention relates to methods and systems for vehicles. BACKGROUND

[0002] An autonomous vehicle (AV) operational design domain (ODD) is a specific condition in which an AV is designed to operate. The ODD can be based on various conditions (e.g., a location of the AV, a speed at which the AV needs to travel, etc.). Operation of the AV can be limited based on the ODD in which the AV is configured to operate. SUMMARY

[0003] A method for a vehicle, comprising: determining, with at least one processor, a trajectory of the vehicle based on a location of the vehicle and sensor data associated with at least one sensor; determining, with the at least one processor, whether the trajectory is associated with at least one defined functional requirement; and in accordance with a determination that the trajectory is associated with at least one defined functional requirement: determining, with the at least one processor, a first level of risk of non-compliance of the vehicle based on the at least one defined functional requirement and a progress of the trajectory; determining, with the at least one processor, an area of concern based on the location of the vehicle and the trajectory; generating, with the at least one processor, a minimum perception zone based on the trajectory; evaluating, with the at least one processor, a second level of risk of non-compliance of the vehicle associated with a progress of the trajectory by the vehicle based on the minimum perception zone; determining, with the at least one processor, a total level of risk based on a combination of the first level of risk and the second level of risk; determining, with the at least one processor, whether the total level of risk will cause the vehicle to be in a safe state or an unsafe state; in accordance with the total level of risk causing the vehicle to be in a safe state, traversing the trajectory; and in accordance with the total level of risk causing the vehicle to be in an unsafe state, requesting intervention to cause the vehicle to be in a safe state.

[0004] A system for a vehicle, comprising: at least one processor; and at least one non-transitory computer-readable medium comprising instructions that, when executed by the at least one processor, cause the vehicle to perform operations comprising: determining a trajectory of the vehicle based on a location of the vehicle and sensor data associated with at least one sensor; determining whether the trajectory is associated with at least one defined functional requirement; and in accordance with a determination that the trajectory is associated with at least one defined functional requirement: determining a first level of risk of non-compliance of the vehicle based on the at least one defined functional requirement and performance of the trajectory; determining a region of interest based on the location of the vehicle and the trajectory; generating a minimum perception zone based on the trajectory; evaluating a second level of risk of non-compliance of the vehicle associated with performance of the trajectory by the vehicle based on the minimum perception zone; determining a total level of risk based on a combination of the first level of risk and the second level of risk; determining whether the total level of risk is to place the vehicle in a safe state or an unsafe state; in accordance with the total level of risk placing the vehicle in a safe state to traverse the trajectory; and in accordance with the total level of risk placing the vehicle in an unsafe state to request intervention to place the vehicle in a safe state. BRIEF DESCRIPTION OF DRAWINGS

[0005] Figure 1 is an example environment in which a vehicle that can implement one or more components of an autonomous system in accordance with various embodiments;

[0006] Figure 2 is a diagram of one or more systems of a vehicle that includes an autonomous system in accordance with various embodiments;

[0007] Figure 3 is a diagram of one or more apparatus and / or components of one or more systems in accordance with various embodiments; Figure 1 and Figure 2

[0008] Figure 4 is a diagram of certain components of an autonomous system in accordance with various embodiments;

[0009] Figure 5 illustrates example scenarios that a vehicle can encounter in accordance with various embodiments.

[0010] Figure 6 illustrates an example operational envelope detection (OED) framework in accordance with various embodiments.

[0011] Figure 7 ​An example process flow is illustrated that is related to OED with situational assessment according to various embodiments.

[0012] Figure 8 An example block diagram is illustrated that is related to a sensor pipeline according to various embodiments.

[0013] Figure 9 An example is illustrated of a probability of detection (PoD) map according to various embodiments.

[0014] Figure 10 An example is illustrated of an occlusion map according to various embodiments.

[0015] Figure 11 An example process flow is illustrated that is related to a perception system according to various embodiments.

[0016] Figure 12 An alternative example process flow is illustrated that is related to a perception system according to various embodiments.

[0017] Figure 13 A diagram is illustrated of an assessment pipeline according to various embodiments.

[0018] Figure 14 A flow diagram of a process for situational assessment of an OED framework for autonomous vehicles according to various embodiments.

[0019] Figure 15 An example block diagram is illustrated that is related to immobility detection with situational context according to various embodiments.

[0020] Figure 16 A flow diagram of an example process is illustrated that is related to immobility detection with situational context according to various embodiments.

[0021] Figure 17 A flow diagram of an alternative example process is illustrated that is related to immobility detection with situational context according to various embodiments.

[0022] Figure 18 A flow diagram of an alternative example process is illustrated that is related to immobility detection with situational context according to various embodiments. DETAILED DESCRIPTION

[0023] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the embodiments described herein can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present disclosure.

[0024] In the drawings, specific arrangements or orders of illustrative elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements, etc.) are shown for ease of description. However, one skilled in the art will appreciate that the specific order or arrangement of illustrative elements in the drawings is not intended to imply that a particular processing order or sequence, or separation of processing, is implied unless explicitly described as such. Additionally, the inclusion of an illustrative element in a drawing does not imply that such element is required in all embodiments, nor that the features represented by such element cannot be included in or associated with other elements in some embodiments.

[0025] Furthermore, in the drawings, connecting elements, such as lines or arrows or the like, are used to illustrate connections, relationships or associations between or among two or more other illustrative elements, and the absence of such connecting elements is not intended to imply the absence of a connection, relationship or association. In other words, some connections, relationships or associations between elements are not illustrated in the drawings in order not to obscure the disclosure. Additionally, for ease of illustration, a single connecting element can be used to represent multiple connections, relationships or associations between elements. For example, if a connecting element represents the communication of signals, data or instructions (e.g., "software instructions"), one skilled in the art will appreciate that such element can represent one or more signal paths (e.g., buses) that can be needed to affect the communication.

[0026] Although the terms "first," "second," and / or "third," etc. are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are only used to differentiate one element from another. For example, a first contact can be called a second contact, and similarly, a second contact can be called a first contact, without departing from the scope of the described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0027] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” and / or “has a” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0028] As used herein, the terms “communication” and “in communication” mean at least one of receiving, receiving, transmitting, transferring, and / or providing, among others, information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof, among others) to be in communication with another unit means that the one unit is capable of either directly or indirectly receiving information from and / or transmitting (e.g., sending) information to the other unit. This can refer to a direct or indirect connection that is wired and / or wireless in nature. Additionally, two units can be in communication with each other even though information that is transmitted can be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit can be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit can be in communication with a second unit if at least one intermediary unit (e.g., a third unit positioned between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message can refer to a network packet (e.g., a data packet, among others) that includes data.

[0029] As used herein, the term “if’ is, optionally, interpreted as meaning “when,” “while,” “in response to determining,” and / or “in response to detecting,” and / or the like, depending on the context. Similarly, the phrase “if it is determined” or “if [stated condition or event] is detected” is, optionally, interpreted as meaning “upon determining,” “in response to determining,” or “upon detecting [stated condition or event],” and / or “in response to detecting [stated condition or event],” and / or the like, depending on the context. Also, as used herein, the terms “have,” “has,” or “having” and the like are intended to be open-ended terms. Further, the phrase “based on” is intended to be open-ended, allowing for determination based on a single input or multiple inputs. Unless otherwise indicated, the phrase “based on” is not intended to mean “based only on” or “exclusively based on.”

[0030] Some embodiments of the application are described in connection with thresholds. As described herein, satisfying a threshold can refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, and / or equal to the threshold, and / or the like.

[0031] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description of embodiments, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0032] OVERALL SUMMARY

[0033] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement an autonomous system having an ODD for designing operation of the autonomous system. Embodiments herein relate to an OED framework configured to receive, as input, data related to sensors of the autonomous system and data related to ODD requirements. The OED then compares the data related to sensors of the autonomous system and the data related to ODD requirements and identifies whether the autonomous system is operating within its ODD or whether a remedial action is appropriate to adjust the ODD requirements based on current sensor data.

[0034] In another aspect and / or embodiment, the systems, methods, and computer program products described herein include or involve a perception system of an autonomous system that is used to identify visibility-related factors (such as environmental conditions that affect a sensor, detected sensor occlusions (e.g., caused by objects within the sensor’s path), blockages of the sensor, etc.) related to one or more of the sensors. The perception system uses these factors to generate a perception visibility model (PVM) related to the sensor’s detection capabilities. Based on the PVM, the perception system generates one or more maps. One such map is an occlusion map that indicates where objects are that are occluding the sensor. Another such map is a PoD map that is related to the likelihood that a sensor can detect the presence of an object in a given location. In one embodiment, if the perception system does not receive new data related to a sensor, the PVM model or a portion thereof is iterated towards a default model at a pre-identified time interval.

[0035] In another aspect and / or embodiment, the systems, methods, and computer program products described herein include and / or implement an OED framework of an autonomous system, and more specifically include and / or implement a situation assessment using metrics (SAM) component of the OED framework. The SAM component attempts to understand the trajectory that a vehicle with the autonomous system is traversing in a given environment for a particular driving scenario, and verifies whether the behavior requirements of the vehicle are met for that particular driving scenario. In an embodiment, the processing pipeline of the SAM component includes two subsystems: a maneuver assessment subsystem and an anomaly detection subsystem.

[0036] In another aspect and / or embodiment, the maneuver assessment subsystem receives trajectories from the planner / controller system, current and predicted “world” states (e.g., agent data, traffic light conditions), map data, and target assignments (e.g., a particular destination). The maneuver assessment subsystem outputs compliance analysis for well-defined situations (e.g., driving scenarios where specified rules exist), detection results for not well-defined situations (e.g., driving scenarios where specified rules do not exist), and update requirements for the perception system. Some examples of maneuver assessment include a “gap” analysis to check whether the vehicle maintains a safe gap / distance from other agents (e.g., other vehicles) in the environment when making a lane change maneuver. Another example is “attention zone” compliance, where a minimum perception zone required by the vehicle (e.g., a field of view provided by the vehicle’s sensor suite) is needed to make a safe maneuver.

[0037] In another aspect and / or embodiment, the anomaly detection subsystem receives unambiguously defined situational detection and planner / controller system internal states as input. The anomaly detection subsystem outputs context data related to the stop cause to assist in assigning the appropriate intervention (e.g., remote vehicle assistance). Some examples of anomaly detection include unusual road traffic scenarios (such as traffic accidents, construction zones, other road users breaking priority (e.g., jaywalkers, red light runners), etc.). Another example is a "stuck" detection that the vehicle is in or will soon enter an irresolvable immobile state, which will require remote intervention.

[0038] In other aspects and / or embodiments, when a vehicle becomes "stuck" (e.g., immobile for an extended duration), it is advantageous for a remote vehicle assistance (RVA) operator to understand the cause of the immobility so that the RVA operator can make the appropriate intervention. Embodiments are disclosed related to program flows for monitoring various events related to stops (hereinafter also referred to as stop constraints) as they occur based on metadata related to these stop constraints. If the duration of a stop constraint continues beyond a predefined threshold, data related to the stop constraint is provided to an RVA operator for intervention.

[0039] In an embodiment, a method includes: determining, with at least one processor, a trajectory of a vehicle based on a location of the vehicle and sensor data associated with at least one sensor; determining, with the at least one processor, whether the trajectory is associated with at least one defined functional requirement; and in accordance with a determination that the trajectory is associated with at least one defined functional requirement: determining, with the at least one processor, a first level of risk of non-compliance of the vehicle based on the at least one defined functional requirement and a progress of the trajectory; determining, with the at least one processor, an area of concern based on the location of the vehicle and the trajectory; generating, with the at least one processor, a minimum perception zone based on the trajectory; evaluating, with the at least one processor, a second level of risk of non-compliance of the vehicle associated with the progress of the trajectory by the vehicle based on the minimum perception zone; determining, with the at least one processor, a total level of risk based on a combination of the first level of risk and the second level of risk; determining, with the at least one processor, whether the total level of risk will place the vehicle in a safe state or an unsafe state; in accordance with the total level of risk placing the vehicle in the safe state, traversing the trajectory; and in accordance with the total level of risk placing the vehicle in the unsafe state, requesting intervention to place the vehicle in the safe state.

[0040] In embodiments, the method further comprises requesting an intervention to bring the vehicle in a safe state based on determining that the trajectory is not associated with at least one defined functional requirement.

[0041] In embodiments, the first risk level is based on a perceptual visibility model of the region of interest, the perceptual visibility model indicating a probability of at least one sensor of the vehicle being visible to detect an object in the sensor data in the region of interest, and wherein evaluating the first risk level comprises evaluating the first risk level based on the probability of at least one sensor of the vehicle being visible to detect an object in the environment in the region of interest.

[0042] In embodiments, determining the first risk level that the vehicle does not comply with the at least one defined functional requirement in case the vehicle traverses the trajectory further comprises determining whether at least one quantitative metric associated with the defined functional requirement is satisfied.

[0043] In embodiments, the at least one quantitative metric is associated with at least one regulatory rule, safety rule, or passenger comfort rule.

[0044] In embodiments, the perceptual visibility model is based on at least one prior perceptual visibility model.

[0045] In embodiments, the probability is determined based on sensor condition data associated with an operational state of at least one sensor of the vehicle.

[0046] In embodiments, the operational state indicates whether a field of view of the at least one sensor is obstructed.

[0047] In embodiments, the operational state indicates whether a field of view of the at least one sensor is malfunctioning.

[0048] In embodiments, the probability is determined based on at least one environmental condition.

[0049] In embodiments, the at least one environmental condition is a weather condition.

[0050] In embodiments, the at least one environmental condition is an illumination condition.

[0051] In embodiments, the intervention is a minimum risk maneuver action.

[0052] In embodiments, the method further comprises receiving data associated with a control of the vehicle based on requesting the intervention, wherein the data is configured to cause the vehicle to take one or more actions.

[0053] In embodiments, the minimum perception zone is based on the trajectory.

[0054] In embodiments, a system comprising: at least one processor; and at least one non-transitory computer-readable medium comprising instructions that, when executed by the at least one processor, cause a vehicle to perform any of the preceding methods.

[0055] In embodiments, at least one non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause a vehicle to perform any of the preceding methods.

[0056] Some advantages of the above-described techniques include providing an OED framework that efficiently addresses the complexity involved in AV operation in various driving scenarios. In specific examples herein, the complexity involves different driving scenarios in which multiple factors (e.g., lighting, weather conditions, other objects in or near the road, characteristics of the road, etc.) affect the ODD of the autonomous system. The OED framework enables triggering a minimum risk maneuver (MRM) or other maneuver or intervention based on an overall analysis of the driving scenario.

[0057] Another advantage is that embodiments herein provide a framework that models sensor perception capabilities and performance under different conditions. Such conditions include environmental conditions (e.g., fog, rain, glare, etc.), sensor visibility conditions (e.g., a sensor being obstructed by foreign matter such as mud, etc.), occlusion-related conditions (e.g., detection of an object by a sensor), or sensor structure conditions (e.g., type or location of a sensor). By identifying this OED framework, an autonomous system can use the model of sensor perception capabilities to determine the operational capabilities of the autonomous system in different scenarios, such that the autonomous system can operate within these capabilities.

[0058] Another advantage is that embodiments herein provide quantitative measures of risk of non-compliance with safety, regulatory, and comfort rules under well-defined different driving scenarios, while still being able to enable direct intervention allocation in response to ill-defined situations (e.g., abnormal events) or long periods of inactivity (e.g., “stuck” detection). Multiple situations or scenarios are evaluated concurrently (e.g., lane change during intersection navigation). The internal state-agnostic metrics are summarized for both the current state and the predicted future state. The applicability of the situation assessment is independent of the underlying decision algorithm(s). The situation assessment can be used to evaluate multiple concurrent trajectory proposals.

[0059] Another advantage resides in that the embodiments herein provide a lightweight data format for conveying sufficient situational context during the concurrent conveyance of autonomous system's internal state data to an RVA operator. Thus, the RVA operator is able to provide appropriate intervention without full knowledge of the reason for the vehicle's immobilization. For example, with implementation of the techniques described herein, more accurate and relevant data related to the state of the autonomous system is provided to the RVA operator, thereby reducing the amount of time required for the RVA operator to analyze the provided data and implement appropriate intervention.

[0060] System and Architecture Overview

[0061] Reference is now made to Figure 1 , an example environment 100 is illustrated in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, the environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, an area 108, vehicle-to-infrastructure (V2I) devices 110, a network 112, a remote AV system 114, a queue management system 116, and a V2I system 118. The vehicles 102a-102n, the vehicle-to-infrastructure (V2I) devices 110, the network 112, the remote AV system 114, the queue management system 116, and the V2I system 118 are interconnected (e.g., connections for communication are established, etc.) via wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, the objects 104a-104n are interconnected with at least one of the vehicles 102a-102n, the vehicle-to-infrastructure (V2I) devices 110, the network 112, the remote AV system 114, the queue management system 116, and the V2I system 118 via wired connections, wireless connections, or a combination of wired or wireless connections.

[0062] The vehicles 102a-102n, individually referred to as vehicle 102 and collectively as vehicles 102, include at least one device configured to transport goods and / or people. In some embodiments, the vehicles 102 are configured to communicate with the V2I devices 110, the remote AV system 114, the queue management system 116, and / or the V2I system 118 via the network 112. In some embodiments, the vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, the vehicles 102 are similar to the vehicle 200 described herein (see Figure 2) identical or similar. In some embodiments, the vehicles 200 in a group of vehicles 200 are associated with an autonomous fleet manager. In some embodiments, the vehicles 102 travel along respective routes 106a-106n (individually referred to as route 106 and collectively referred to as routes 106) as described herein. In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system identical or similar to autonomous system 202).

[0063] The objects 104a-104n (individually referred to as object 104 and collectively referred to as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one construction (e.g., a building, a sign, a fire hydrant, etc.), among others. Each object 104 is stationary (e.g., located at a fixed location and for a period of time) or moving (e.g., has a velocity and is associated with at least one trajectory). In some embodiments, the objects 104 are associated with respective locations in the region 108.

[0064] The routes 106a-106n (individually referred to as route 106 and collectively referred to as routes 106) are each associated with (e.g., prescribed) a series of actions (also referred to as a trajectory) connecting states along which a vehicle can navigate. Each route 106 begins at an initial state (e.g., a state corresponding to a first spatiotemporal location and / or velocity, among others) and ends at a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target region (e.g., a subspace of acceptable states (e.g., terminal states)). In some embodiments, the first state includes a location at which one or more individuals are to be picked up by the vehicle, and the second state or region includes one or more locations at which the one or more individuals picked up by the vehicle are to be dropped off. In some embodiments, the route 106 includes a plurality of sequences of acceptable states (e.g., a plurality of sequences of spatiotemporal locations), which are associated with (e.g., define) a plurality of trajectories. In an example, the route 106 includes only high-level actions or imprecise state locations, such as a series of connected roads indicating a turn at an intersection, among others. Additionally or alternatively, the route 106 can include more precise actions or states, such as, for example, a particular target lane or precise location within a lane region and a target velocity at these locations, among others. In an example, the route 106 includes a plurality of sequences of precise states along at least one high-level action with a limited look-ahead to an intermediate target, where a combination of successive iterations of the limited look-ahead state sequences cumulatively correspond to a plurality of trajectories that collectively form a high-level route that terminates at a final target state or region.

[0065] The region 108 includes a physical region (e.g., a geographic area) that the vehicle 102 can navigate. In examples, the region 108 includes at least one state (e.g., a country, a province, an individual state included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, the region 108 includes at least one named thoroughfare (referred to herein as a “road”), such as a highway, an interstate, a parkway, a city street, etc. Additionally or alternatively, in some examples, the region 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of an empty lot and / or undeveloped area, a dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can travel). In examples, a road includes at least one lane associated with (e.g., identified based on) at least one lane marking.

[0066] The vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to- everything (V2X) device) includes at least one device configured to communicate with the vehicle 102 and / or a V2I infrastructure system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, the platoon management system 116, and / or the V2I system 118 via the network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or a three-dimensional (3D) camera), a lane marking, a streetlight, a parking meter, etc. In some embodiments, the V2I device 110 is configured to directly communicate with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the platoon management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0067] The network 112 includes one or more wired and / or wireless networks. In examples, the network 112 includes a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, etc., and / or a combination of one or more of the networks.

[0068] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I devices 110, the network 112, the queue management system 116, and / or the V2I infrastructure system 118 via the network 112. In an example, the remote AV system 114 includes a server, a group of servers, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing, etc. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0069] The queue management system 116 includes at least one device configured to communicate with the vehicle 102, the V2I devices 110, the remote AV system 114, and / or the V2I infrastructure system 118. In an example, the queue management system 116 includes a server, a group of servers, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems), etc.).

[0070] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I devices 110, the remote AV system 114, and / or the queue management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I devices 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a group of servers, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipal authority or a private organization (e.g., a private organization for maintaining the V2I devices 110, etc.).

[0071] Provided Figure 1 The number and arrangement of illustrated components are provided as an example. Numbered Figure 1 Additional components, fewer components, different components, and / or differently arranged components can be present in other embodiments. Additionally or alternatively, at least one of the illustrated components can perform one or more functions described as being performed by one or more other illustrated components. Figure 1 Additionally or alternatively, at least one group of illustrated components can perform one or more functions described as being performed by one or more other groups of illustrated components.

[0072] Reference is now made toFigure 2 The vehicle 200 includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 and the vehicle 102 (see...) Figure 1 The vehicle 200 is similar to or the same as the vehicle in question. In some embodiments, the vehicle 200 has autonomous capabilities (e.g., implementing at least one function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention) and / or highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations)). For a detailed description of fully autonomous and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0073] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, (discussed below) Global Positioning System (GPS) receivers, and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, and safety controller 202g.

[0074] Camera 202a includes components configured to communicate with communication device 202e, AV computing 202f, and / or security controller 202g via a bus (e.g., with...). Figure 3The camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) to capture images including physical objects (e.g., a car, a bus, a curb, and / or a person, etc.). In some embodiments, the camera 202a generates camera data as output. In some examples, the camera 202a generates camera data including image data associated with an image. In this example, the image data can specify at least one parameter corresponding to the image (e.g., an image characteristic such as exposure, brightness, etc., and / or an image timestamp, etc.). In such examples, the image can be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a includes multiple independent cameras configured on (e.g., positioned on) the vehicle to capture images for the purpose of stereoscopic imagery (stereo vision). In some examples, the camera 202a includes multiple cameras that generate image data and transmit the image data to the AV computing 202f and / or a queue management system (e.g., the same or similar queue management system as the queue management system 116 of Figure 1 The camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) to capture images including physical objects (e.g., a car, a bus, a curb, and / or a person, etc.). In some embodiments, the camera 202a generates camera data as output. In some examples, the camera 202a generates camera data including image data associated with an image. In this example, the image data can specify at least one parameter corresponding to the image (e.g., an image characteristic such as exposure, brightness, etc., and / or an image timestamp, etc.). In such examples, the image can be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a includes multiple independent cameras configured on (e.g., positioned on) the vehicle to capture images for the purpose of stereoscopic imagery (stereo vision). In some examples, the camera 202a includes multiple cameras that generate image data and transmit the image data to the AV computing 202f and / or a queue management system (e.g., the same or similar queue management system as the queue management system 116 of

[0075] In embodiments, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects that provide visual navigation data. In some embodiments, the camera 202a generates traffic light data associated with one or more images. In some examples, the camera 202a generates TLD data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data differs from other systems containing cameras described herein in that the camera 202a can include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with an angle of view of about 120 degrees or more, etc.) to generate images related to as many physical objects as possible.

[0076] A laser detection and ranging (LiDAR) sensor 202b includes at least one device configured to communicate with the communication device 202e, the AV computing 202f, and / or the safety controller 202g via a bus (e.g., the same or similar bus as the bus 302 of Figure 3 FIG. 1). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside of the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object that the light encounters. The LiDAR sensor 202b also includes at least one light detector that detects the light after the light emitted from the light emitter encounters a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing a boundary of a physical object and / or a surface of a physical object (e.g., a topology of the surface), etc. In such examples, the image is used to determine the boundary of the physical object in the field of view of the LiDAR sensor 202b.

[0077] A radio detection and ranging (radar) sensor 202c includes at least one device configured to communicate with the communication device 202e, the AV computing 202f, and / or the safety controller 202g via a bus (e.g., the same or similar bus as the bus 302 of Figure 3 FIG. 1). The radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the radar sensor 202c include radio waves within a predetermined frequency spectrum. In some embodiments, during operation, the radio waves emitted by the radar sensor 202c encounter a physical object and are reflected back to the radar sensor 202c. In some embodiments, the radio waves emitted by the radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the radar sensor 202c generates a signal representing objects included in the field of view of the radar sensor 202c. For example, at least one data processing system associated with the radar sensor 202c generates an image representing a boundary of a physical object and / or a surface of a physical object (e.g., a topology of the surface), etc. In some examples, the image is used to determine the boundary of the physical object in the field of view of the radar sensor 202c.

[0078] Microphone 202d includes at least one device configured to communicate with communication device 202e, AV computing 202f, and / or safety controller 202g via a bus (e.g., the same or similar bus as bus 302 of FIG. 1). Figure 3 Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) configured to capture audio signals and generate data associated with (e.g., representative of) the audio signals. In some examples, microphone 202d includes transducer devices and / or the like. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine a position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signals associated with the data.

[0079] Communication device 202e includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, radar sensor 202c, microphone 202d, AV computing 202f, safety controller 202g, and / or drive-by-wire (DBW) system 202h. For example, communication device 202e can include the same or similar devices as communication interface 314 of FIG. 1. In some embodiments, communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles). Figure 3

[0080] AV computing 202f includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, radar sensor 202c, microphone 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, AV computing 202f includes a device such as a client device, a mobile device (e.g., a cellular phone and / or a tablet computer, etc.), and / or a server (e.g., a computing device including one or more central processing units and / or graphics processing units, etc.). In some embodiments, AV computing 202f is the same as or similar to autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, AV computing 202f is configured to communicate with a remote AV system (e.g., the same or similar to remote AV system 114 of FIG. 1), a platoon management system (e.g., the same or similar to platoon management system 116 of FIG. 1), a V2I device (e.g., the same or similar to V2I device 110 of FIG. 1), and / or a V2I system (e.g., the same or similar to V2I system 118 of FIG. 1). Figure 1 Figure 1 Figure 1 Figure 1 AV computing 202f includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, radar sensor 202c, microphone 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, AV computing 202f includes a device such as a client device, a mobile device (e.g., a cellular phone and / or a tablet computer, etc.), and / or a server (e.g., a computing device including one or more central processing units and / or graphics processing units, etc.). In some embodiments, AV computing 202f is the same as or similar to autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, AV computing 202f is configured to communicate with a remote AV system (e.g., the same or similar to remote AV system 114 of FIG. 1), a platoon management system (e.g., the same or similar to platoon management system 116 of FIG. 1), a V2I device (e.g., the same or similar to V2I device 110 of FIG. 1), and / or a V2I system (e.g., the same or similar to V2I system 118 of FIG. 1).​​​​

[0081] The safety controller 202g includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the radar sensor 202c, the microphone 202d, the communication device 202e, the AV computing 202f, and / or the DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., the powertrain control system 204, the steering control system 206, and / or the braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) control signals generated and / or transmitted by the AV computing 202f.

[0082] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the AV computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., the powertrain control system 204, the steering control system 206, and / or the braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (e.g., a turn signal light, a headlight, a door lock, and / or a windshield wiper, etc.).

[0083] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a certain direction, decelerate in a certain direction, make a left turn, and / or make a right turn, etc. In examples, the powertrain control system 204 causes energy (e.g., fuel and / or electricity, etc.) provided to a motor of the vehicle to increase, remain the same, or decrease, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0084] The steering control system 206 includes at least one device configured to cause one or more wheels of the vehicle 200 to rotate. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes two front wheels and / or two rear wheels of the vehicle 200 to rotate to the left or right to cause the vehicle 200 to turn left or right.

[0085] The braking system 208 includes at least one device configured to cause one or more brakes to actuate to cause the vehicle 200 to decelerate and / or remain stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to cause one or more calipers associated with one or more wheels of the vehicle 200 to close on a respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0086] In some embodiments, the vehicle 200 includes at least one on-board sensor (not explicitly illustrated) to measure or infer a property of a state or condition of the vehicle 200. In some examples, the vehicle 200 includes an on-board sensor such as a GPS receiver, an inertial measurement unit (IMU), a wheel rate sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor, etc.

[0087] Reference is now made to Figure 3 , a schematic diagram of an apparatus 300. As illustrated, the apparatus 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, the apparatus 300 corresponds to one or more devices of the vehicle 102 (e.g., one or more devices of a system of the vehicle 102), the remote AV system 114, the platoon management system 116, the vehicle-to-infrastructure system 118, the autonomy system 202, the braking system 208, the DBW system 202h, the steering control system 206, the powertrain control system 204, and / or one or more devices of the network 112 (e.g., one or more devices of a system of the network 112). In some embodiments, one or more devices of the vehicle 102 (e.g., one or more devices of a system of the vehicle 102), the remote AV system 114, the platoon management system 116, the vehicle-to-infrastructure system 118, the autonomy system 202, the braking system 208, the DBW system 202h, the steering control system 206, the powertrain control system 204, and / or one or more devices of the network 112 (e.g., one or more devices of a system of the network 112) include at least one apparatus 300 and / or at least one component of the apparatus 300. As illustrated, the bus 302 includes a path that permits the processor 304 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), etc.), the memory 306, the storage component 308, the input interface 310, the output interface 312, and the communication interface 314 to communicate data between one another. The memory 306 and the storage component 308 each include a tangible machine-Figure 3 As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0088] The bus 302 includes a component that permits communication among the components of the device 300. In some embodiments, the processor 304 is implemented in hardware, software, or a combination of hardware and software. In some examples, the processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least a function (e.g., a field-programmable gate array (FPGA) and / or an application- specific integrated circuit (ASIC), etc.). The memory 306 includes a random access memory (RAM), a read only memory (ROM), and / or another type of dynamic and / or static storage (e.g., a flash memory, a magnetic storage, and / or an optical storage, etc.) that stores data and / or instructions for use by the processor 304.

[0089] The storage component 308 stores data and / or software related to the operation and use of the device 300. In some examples, the storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, and / or a solid state disk, etc.), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, a CD-ROM, a RAM, a PROM, an EPROM, a FLASH-EPROM, NV-RAM, and / or another type of computer- readable medium, along with a corresponding drive.

[0090] The input interface 310 includes a component that permits the device 300 to receive data, such as via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, and / or a camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes a sensor (e.g., a GPS receiver, an accelerometer, a gyroscope, and / or an actuator, etc.) to sense data. The output interface 312 includes a component that provides output data from the device 300 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs), etc.).

[0091] In some embodiments, the communication interface 314 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the device 300 to communicate with other devices, via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the device 300 to receive data from another device and / or provide data to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a Bluetooth® interface, or a cellular network interface, to name a few. In some embodiments, the communication interface 314 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the device 300 to communicate with other devices, via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the device 300 to receive data from another device and / or provide data to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a Bluetooth® interface, or a cellular network interface, to name a few.

[0092] In some embodiments, the device 300 performs one or more processes described herein. The device 300 performs these processes based on the processor 304 executing software instructions stored by a computer-readable medium, such as the memory 305 and / or the storage component 308. A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes a memory space within a single physical storage device or a memory space spread across multiple physical storage devices.

[0093] In some embodiments, the software instructions are read from another computer-readable medium or from another device, via the communication interface 314, into the memory 306 and / or the storage component 308. The software instructions stored in the memory 306 and / or the storage component 308, when executed, cause the processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0094] The memory 306 and / or the storage component 308 include a data store or at least one data structure (e.g., a database or the like). The device 300 can receive data from, store data in, communicate data to, or search

[0095] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the other device, which, when executed by the processor of processor 304 and / or the processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more of the processes described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0096] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3 Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a group of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another group of components of device 300.

[0097] Now for reference Figure 4FIG. 4, for example, illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an“AV stack”). As illustrated, the AV computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a localization system 406 (sometimes referred to as a localization module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in and / or implemented in an autonomous navigation system of a vehicle (e.g., the AV computing 202f of the vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in one or more standalone systems (e.g., one or more systems that are the same as or similar to the AV computing 400, etc.). In some examples, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in one or more standalone systems located in a vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the AV computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., by a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), and / or an FPGA, etc.), or a combination of computer software and computer hardware. It will also be appreciated that, in some embodiments, the AV computing 400 is configured to communicate with remote systems (e.g., an autonomous vehicle system that is the same as or similar to the remote AV system 114, a queue management system that is the same as or similar to the queue management system 116, and / or a V2I system that is the same as or similar to the V2I system 118, etc.).

[0098] In some embodiments, the perception system 402 receives data associated with at least one physical object in an environment (e.g., data used by the perception system 402 to detect the at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., the camera 202a) that is associated with (e.g., represents) one or more physical objects within a field of view of the at least one camera. In such examples, the perception system 402 classifies the at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of the physical object by the perception system 402, the perception system 402 transmits data associated with the classification of the physical object to the planning system 404.

[0099] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 receives data (e.g., data associated with classifications of physical objects described above) from the perception system 402 periodically or continuously, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In some embodiments, the planning system 404 receives data associated with updated locations of a vehicle (e.g., vehicle 102) from the localization system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the localization system 406.

[0100] In some embodiments, the localization system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, the localization system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In certain examples, the localization system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the localization system 406 generates a combined point cloud based on the individual point clouds. In these examples, the localization system 406 compares the at least one point cloud or the combined point cloud to a two-dimensional (2D) and / or a three-dimensional (3D) map of the area stored in the database 410. Based on the localization system 406 comparing the at least one point cloud or the combined point cloud to the map, the localization system 406 then determines a location of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of a carriageway, a map describing connectivity properties of a road network, a map describing physical properties of a carriageway such as traffic speed, traffic flow, number of vehicle and bicycle traffic lanes, lane width, traffic direction of lanes, or type and location of lane markings, or a combination thereof, and a map describing spatial locations of road features such as pedestrian crossings, traffic signs, or various types of other travel signals. In some embodiments, the map is generated in real-time based on data received by the perception system.

[0101] In another example, the positioning system 406 receives global navigation satellite system (GNSS) data generated by a GPS receiver. In some examples, the positioning system 406 receives GNSS data associated with a location of the vehicle in an area, and the positioning system 406 determines a latitude and a longitude of the vehicle in the area. In such examples, the positioning system 406 determines a position of the vehicle in the area based on the latitude and the longitude of the vehicle. In some embodiments, the positioning system 406 generates data associated with the position of the vehicle. In some examples, based on the positioning system 406 determining the position of the vehicle, the positioning system 406 generates data associated with the position of the vehicle. In such examples, the data associated with the position of the vehicle includes data associated with one or more semantic properties corresponding to the position of the vehicle.

[0102] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls operation of the vehicle by generating and transmitting control signals to cause a powertrain control system (e.g., the DBW system 202h and / or the powertrain control system 204, etc.), a steering control system (e.g., the steering control system 206), and / or a braking system (e.g., the braking system 208) to operate. In examples, where the trajectory includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust a steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (e.g., headlamps, turn signal lights, door locks, and / or windshield wipers, etc.) to change state.

[0103] In some embodiments, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model individually or in combination with one or more of the systems described above. In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in an environment, etc.).

[0104] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software and for at least one system using AV computing 400 (e.g., with...). Figure 3 (The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0105] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), a remote AV system (e.g., a remote AV system identical or similar to remote AV system 114), a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0106] Operational envelope detection using a context evaluation framework

[0107] As used herein, the term "operational envelope" refers to the envelope in which an AV operates within its capabilities. Typically, an AV operates within its designed operational envelope when its current capabilities (e.g., perception, planning, prediction, control, communication, etc.) meet or exceed the functional requirements imposed by the vehicle's ongoing or anticipated maneuvers in a given context with identified functional constraints (e.g., environmental conditions, road structure, behavior of other road users, etc.).

[0108] Figure 5 Example scenarios 500 that a vehicle 200 with an autonomous system 202 might encounter are depicted. Specifically,Figure 5 The AV 200 is depicted with an intent to turn right at the intersection. The right turn is depicted, showing the intent trajectory 520 of the AV 200. As described above, the term “trajectory” refers to a path that the AV 200 follows through the environment as a function of time. In particular, as used herein, when the AV 200 performs a sequence of actions, the AV 200 will be referred to as “threading a trajectory.” In another embodiment, such a sequence of actions is referred to as “performing (and performing) a maneuver.”

[0109] In this example scenario 500, there are multiple objects such as a parked vehicle 510 and a pedestrian 515. In particular, the parked vehicle 510 is depicted as being parked along a curb line in the lane that the AV 200 intends to turn into. The pedestrian 515 is also at least partially in this lane. In this embodiment, when the AV 200 performs a maneuver along the trajectory 520 by turning right, the parked vehicle 510 can block the visibility of the AV 200 such that the AV 200 cannot see the pedestrian. Thus, in this embodiment, the operational envelope of the AV 200 is affected by the location of the parked vehicle 510 such that the capabilities of the AV 200 (e.g., the perception of the AV 200) affect (in this case, constrain) the ability of the AV 200 to navigate to the trajectory 520.

[0110] Embodiments herein relate to autonomous systems for measuring the functional capabilities and requirements of an AV 200 based on an accurate assessment of the environment in which the AV 200 is located and / or the context in which the AV 200 is involved to assist in ensuring operation of the AV 200 within its ODD.

[0111] Reference is made to Figure 6 , Figure 6 is an example OED framework. In some embodiments, one or more of the elements described with respect to the OED framework 600 are performed (e.g., entirely and / or partially, etc.) by one or more vehicles 102 (e.g., one or more devices of the vehicle 102). Additionally or alternatively, one or more of the elements described with respect to the OED framework 600 can be performed (e.g., entirely and / or partially, etc.) by another device or set of devices separate from or including the vehicle 102, such as Figure 1 , 2 , any one of the figures in 3 and 4, and / or the like.

[0112] In some embodiments, the OED framework 600 includes at least one sensor 605, which includes at least one of the camera 202a, LiDAR sensor 202b, radar sensor 202c, microphone 202d, etc. The sensor 605 is configured to output sensor data 606 to the perception system 402. The sensor data 606 includes, for example, radar data, camera data, LiDAR data, etc.

[0113] In addition to the functionality described above with reference to Figure 4 The perception system 402 provides at least one perception map 607, in addition to the functionality described above with reference to

[0114] The OED framework 600 also includes a planning system 404. The planning system 404 receives data from various systems and subsystems as described with respect to, for example Figure 4 In addition, the planning system 404 receives perception data 619 (e.g., object detections) and / or PVM data 618 from the perception system 402, which the planning system 404 uses to generate or update trajectory data 609 for evaluation by the autonomous vehicle computing 400 and / or for traversal by the vehicle 102. The trajectory data 609 includes one or more parameters of a trajectory, such as the trajectory 520, etc. In embodiments, the PVM data 618 is generated by the perception system 402, as discussed in detail below.

[0115] The OED framework 600 also includes an evaluation system 620. In some embodiments, the evaluation system 620 can be included in the AV computing 400 as a standalone system, or included in the perception system 402. In some embodiments, the evaluation system 620 is referred to as a “SAM” system 620. The evaluation system 620 is configured to provide a perception polygon (along a top view or bird’s eye view of the environment in which the AV is operating) that indicates a minimum perception zone required in a region of interest. As used herein, a “perception zone” is a portion of the operating environment that is within the detection capabilities of the sensor 605, and a “region of interest” refers to an area within the vicinity of the AV, and more specifically, an area of the environment in which the AV is present or will move to based on the trajectory data 609. Specifically, the evaluation system 620 identifies and outputs minimum perception zone data 611 based on the trajectory data 609 provided by the planning system 404. The minimum perception zone data 611 relates to a minimum perception zone required to execute a trajectory described by the trajectory data 609.

[0116] An inability of the evaluation system 620 to identify the minimum perception zone required can indicate that the proposed trajectory is not associated with a functional requirement. In this case, the evaluation system 620 provides an indication of undefined functional requirement 617 to the intervention request system 630. The evaluation system 620 is described in more detail below.

[0117] The OED framework 600 also includes an arbiter system 625. The arbiter system 625 is configured to receive the perception map(s) 607 and minimum perception zone data 611. In embodiments, the arbiter system 625 is also configured to receive trajectory data 609 directly from the planning system 404. The arbiter system 625 then compares the perception map(s) 607 and minimum perception zone data 611 (and, in one embodiment, the trajectory data 609) to identify whether the AV is operating within its ODD. Specifically, the arbiter system 625 is configured to assign a first risk level to the output of the perception system 402. In some embodiments, the arbiter system 625 is configured to assign the first risk level of non-compliance with the functional requirements related to the trajectory based on the sensor data 606.

[0118] The arbiter system 625 also assigns a second risk level of non-compliance with the functional requirements to the output of the evaluation system 620 (e.g., the minimum perception zone data 611). The arbiter system 625 calculates a total risk level based on the assigned first risk level and the second risk level. Based on the total risk level, the arbiter system 625 identifies whether the AV is in a safe state. An example of a safe state is where the AV is able to perform the assigned trajectory / maneuver or navigate the assigned trajectory / maneuver within the ODD of the AV. An example of an unsafe state is where the requirements of the trajectory / maneuver exceed the functional capabilities of the AV. If the arbiter system 625 identifies that the AV is in an unsafe state, the arbiter system 625 outputs an unsafe indicator data 616 to the intervention request system 630, which generates and sends / transmits an intervention request to, for example, the remote AV system 114 for an RVA intervention, or to the planning system 404 and / or the AV control system 408 for an MRM intervention.

[0119] The OED framework 600 also includes a stillness detection system 635. The stillness detection system 635 is configured to identify that the AV is still based on the trajectory received from the planning system 404. As described in more detail below, the stillness detection system 635 is also configured to identify the cause of the stillness and outputs a trigger signal data 614 and a stop cause data 613 to the intervention request system 630.

[0120] The OED framework 600 also includes an intervention request system 630. The intervention request system 630 is configured to generate an intervention request 612. The intervention request 612 can be based on at least one of the perception maps 607, the unsafe metric data 616, the minimum perception zone data 611, the stop reason data 613, the trigger signal data 614, and the indication of undefined functional requirements 617.

[0121] In one embodiment, the intervention request 612 is, for example, a request for data associated with control of the vehicle to an RVA operator or teleoperator. For example, the intervention request 612 can include a request for data from which the control system 408 can act to cause the AV to take one or more actions. In another embodiment, the intervention request 612 is an MRM, which can be a pre-identified maneuver for causing the AV to return to a safe state (such as AV deceleration, stop, pull over, etc.). In another embodiment, the intervention request 612 can include a request for a degraded mode operation (DMO) task (such as reduced speed operation, etc.).

[0122] In embodiments in which the intervention request 612 is based on at least one perception map 607, the perception system 610 and / or the intervention request system 630 detects an error with respect to sensor operation or an emergency situation. In this embodiment, the perception system 610 can directly notify the intervention request system 630 of the existence of such an emergency situation, or provide the at least one perception map 607 so that the intervention request system 630 can generate the intervention request 612.

[0123] In one embodiment, as described in further detail below with respect to Figure 13 The intervention request system 630 is configured to receive data such as the stop reason data 613 and the trigger signal data 614 directly from the immobility detection system 635, in one embodiment. Specifically, when the immobility detection system 635 identifies that the AV is “stuck” (e.g., immobile), the immobility detection system 635 provides contextual data (e.g., data 613, 614) related to the immobility of the AV to the intervention request system 630.

[0124] Figure 7 An example process flow related to operational envelope detection with situational assessment is depicted in accordance with various embodiments. In some embodiments, one or more of the elements described with respect to process 700 are performed (e.g., entirely and / or partially, etc.) by the example OED framework 600 (e.g., a device or set of devices included in the OED framework 600). Additionally or alternatively, in some embodiments, one or more of the elements described with respect to process 700 can be performed by another device or set of devices separate from the example OED framework 600, such as the example perception system 610, the example intervention request system 630, the example control system 408, the example RVA 410, the example teleoperator 412, the example MRM system 620, the example DMO system 640, the example immobility detection system 635, etc. Figure 1 、 2one or more elements described with respect to the OED framework 600 (e.g., fully and / or partially, etc.) by one or more of the devices of any of Figures 1, 2, 3, and 4, etc.

[0125] The process 700 includes determining, at 705, a trajectory for the AV based on the AV's location and sensor data (such as the sensor data 606, etc.). For example, the trajectory is generated at or by the planning system 404 as described above. The trajectory includes at least one command to be implemented by the control system 408 to cause the AV to take an action such as a turn, acceleration, maintaining a speed, braking, etc. In addition, as described in more detail below, 705 can be based on the localization data 801.

[0126] The process 700 also includes determining, at 710, whether the trajectory is associated with at least one defined functional requirement. Functional requirements are rules or constraints that the AV must adhere to when traversing a given trajectory. These requirements include, but are not limited to: 1) basic control of the vehicle such as stopping, maintaining a vehicle speed, and turning, etc.; 2) basic needs for perceiving the environment such as detecting traffic lights, pedestrians, bicycles, other vehicles, and any other stationary or dynamic objects; 3) a need for determining the vehicle's position in a map or lane (i.e., localization); 4) the ability of the vehicle to smoothly pass through intersections, U-turns, lane changes, merges, unprotected turns, roundabouts, and any other maneuvering action. In one embodiment, the determination is made by the evaluation system 620, while in other embodiments, the determination is additionally or alternatively made by the arbiter system 625 or some other system or subsystem of the vehicle.

[0127] If it is determined at 710 that the trajectory is not associated with at least one defined functional requirement (e.g., based on an indication of undefined functional requirements 617), the process 700 proceeds directly to requesting intervention at 715 to place the vehicle in a safe state. The intervention request is made by the intervention request system 630 as described above. Such intervention can include a request for one or more of an RVA, an MRM, a DMO task, and some other intervention. In this case, such intervention can be desirable because the indication that the maneuvering action is not associated with a functional requirement indicates that the maneuvering action is an exceptional maneuvering action. An exceptional maneuvering action can be, for example, a crash, an emergency maneuvering action (e.g., a sharp turn or an emergency stop), or some other type of maneuvering action.

[0128] However, if it is determined at 710 that the trajectory is associated with at least one defined functional requirement, the process 700 includes determining, at 720, a first level of risk that the vehicle will not comply with the at least one defined functional requirement if the vehicle traverses the trajectory. In embodiments, the first level of risk is determined at 720 by one or both of the perception system 402 and the arbiter system 625. For example, in one embodiment, the perception system 402 identifies the first level of risk and provides an indication of the first level of risk to the arbiter system 625. In another embodiment, the first level of risk is determined by the arbiter system 625 based on the perception map(s) 607 provided by the perception system 402.

[0129] For example, the first level of risk of non-compliance is determined based on whether the functional requirement of the maneuver exceeds the functional capabilities of the vehicle, and more specifically based on the sensor(s) 605. In particular, the determination is made based on a PVM. In one embodiment, the generation of the PVM is based on at least one prior PVM. The PVM is described in more detail below.

[0130] In embodiments, the determination of the first level of risk is based on a determination of whether at least one quantitative metric associated with the defined functional requirement is satisfied. Such quantitative metric can be based on at least one rulebook, which is a data structure including, but not limited to, regulatory rules, safety rules, passenger comfort rules, or some other type of rule.

[0131] The process 700 also includes determining, at 725, a region of interest based on the location of the vehicle (e.g., provided by the localization system 406) and the trajectory data (as provided by the planning system 404). Such determination is made, for example, by one or both of the arbiter system 625 and the planning system 404.

[0132] The process 700 also includes generating, at 730, a minimum perception zone required for the region of interest based at least in part on the sensor data. In embodiments, such generation is made by the evaluation system 620. The minimum perception zone required can be based on a minimum amount of perception data (e.g., a minimum amount of data associated with the environment) required for the vehicle to safely traverse the trajectory. In embodiments, the minimum amount of perception data is based on an analysis of the trajectory (e.g., the minimum amount of perception data is dynamic), while in another embodiment, the minimum amount of perception data is pre-identified (e.g., the minimum amount of perception data is static).

[0133] The process 700 also includes evaluating, at 735, a second level of risk associated with the vehicle threading trajectory based on the minimum perception zone. In embodiments, this evaluation is performed by one or both of the evaluation system 620 and the arbiter system 625. For example, in one embodiment, the evaluation system 620 identifies the risk and provides this data to the arbiter system 625. In another embodiment, the evaluation system 620 provides the minimum perception zone data 611 to the arbiter system 625, and the arbiter system 625 identifies the second level of risk. Similar to the evaluation of the first level of risk, the evaluation of the second level of risk is based on determining whether the functional requirements on AV operation exceed the capabilities of the AV or sensors of the AV.

[0134] The process 700 also includes determining, at 740, a total level of risk based on a combination of the first level of risk (as determined at element 720) and the second level of risk (as evaluated at element 735). This determination is performed by the arbiter system 625 and can be based on one or more mathematical functions, such as a sum of the risks, a mean of the risks, a median of the risks, a mode of the risks, or some other mathematical function, etc. In another embodiment, the total level of risk is identified based on only the highest level of risk of the first level of risk and the second level of risk.

[0135] The process 700 also includes determining, at element 745, whether the total level of risk will place the vehicle in a safe state or an unsafe state. This determination is performed by the arbiter system 625. In one embodiment, this determination is based on a comparison of the total level of risk to a threshold value related to the identified maneuver action. In other words, the threshold value is dynamic. In another embodiment, this determination is based on a comparison of the total level of risk to a pre-identified threshold value that is independent of the maneuver action. In other words, the threshold value is static. If the total level of risk satisfies (or exceeds) the threshold value, the arbiter system 625 identifies that the AV is in or will be in an unsafe state and proceeds to element 715 by, for example, providing unsafe indicator data to the intervention request system 630. However, if the total level of risk is below (or, at or below) the threshold value, the arbiter system 625 identifies that the AV is in or will be in a safe state and threads the trajectory (e.g., performs the maneuver action) at 750. Specifically, the arbiter system 625 facilitates threading the trajectory by another system or subsystem of the AV, such as the control system 408, etc.

[0136] In general, the PVM for a region of interest indicates a probability that a detection of an object in the perception data is visible to at least one sensor of the AV in the region of interest. The probability can be based on one or more of various factors, such as sensor condition data associated with an operational state of the at least one sensor of the AV, etc. As used herein, the operational state indicates whether a field of view of the at least one sensor is occluded (and, if so, the operational state can indicate to what extent the field of view is occluded), and / or whether the at least one sensor has failed. In another embodiment, the probability is determined based on at least one environmental condition, such as a weather condition (e.g., whether it is raining or sunny), an illumination condition (e.g., whether sunlight is available), or some other condition, etc.

[0137] Example Perception System

[0138] As previously described, in embodiments, the perception system 402 provides data related to the extent and distance that a sensor (e.g., sensor 605) of the AV is able to perceive an object at a given time. More specifically, the perception system 402 is responsible for characterizing the sensing capabilities of the AV by modeling and monitoring the capabilities of the perception system 402 to detect objects and environmental features of interest around the AV. The capabilities are modeled as a function of sensor conditions (e.g., sensor blockage, sensor dirt, sensor failure, etc.), environmental conditions (e.g., weather, time of day, sensor occlusion, etc.), and capabilities of the sensor 605 (e.g., field of view (FOV), mounting location of the sensor, etc.). In alternative embodiments, the perception system 402 is responsible for object detection, and another system, separate from the perception system 402, is responsible for characterizing the sensing capabilities of the AV by modeling and monitoring the capabilities of the perception system 402 to detect objects and environmental features of interest around the AV.

[0139] In embodiments, the perception system 402 generates a PVM for encapsulating the environment outside of the AV and the perception system 402 sensing capabilities at runtime. The PVM can be generated based on online and / or offline data. In embodiments, a set of sensor detectors are used to detect and monitor sensor conditions and environmental conditions. The sensor conditions are conditions of the sensor including, but not limited to, blockage / occlusion, sensor dirt, sensor failure, etc. The environmental conditions include, but are not limited to, conditions of weather, time of day, other objects that are occluding the sensor (e.g., occluding the FOV of the sensor), etc.

[0140] Reference Figure 8 , Figure 8A block diagram of an example sensor pipeline 800 including a sensing system 402 is depicted. The sensing system 402 includes at least one sensor system 810 configured to receive sensor data 606 from one or more sensors 605 and detect / monitor sensor conditions and / or environmental conditions. Specifically, sensor conditions such as blockage / occlusion, dirt, and malfunction, or environmental conditions such as glare from sunlight, can reduce the ability of the sensing system 402 to detect objects. The sensor system 810 is configured to detect the presence and location of these conditions based on the sensor data 606 (e.g., which sensors 605 are affected, and how or where the problem is occurring (e.g., locations where sunlight or light is causing a glare effect, etc.)). In embodiments, the sensor detector 810 is configured to identify these sensor conditions or environmental conditions through, for example, data analysis associated with the sensor data 606, analysis of metadata associated with the sensor data 606, control data associated with the one or more sensors 605, or some other data output by the one or more sensors 605. Then, (one or more) sensor systems 810 provide data related to sensor / environment data 840 to PVM subsystem 820, which will be described in further detail below.

[0141] In some embodiments, as described above, such as in Figure 8 As can be seen, sensor / environmental data 840 is directly provided to the intervention request system 630. In one embodiment, data 840 is continuously provided to the intervention request system 630, while in another embodiment, data 840 is provided to the intervention request system 630 only when the parameters of the data reach or exceed a threshold, which can be dynamic or static. In this embodiment, the intervention request system 630 is configured to act on data 840 received from one or more sensor systems 810, which indicates situations such as sensor failure (e.g., extreme environmental conditions, sensor malfunction, etc.) that may require intervention as described above.

[0142] The perception system 402 also includes at least one perception pipeline 815. The perception pipeline 815 is configured to receive sensor data and perform actions related to object detection. More specifically, the perception pipeline 815 is configured to generate intermediate perception results as perception data 619 as output. This perception data 619 includes, but is not limited to, LiDAR semantic point clouds carrying ground data and data on detected objects. In other embodiments, the perception data 845 includes, for example, data related to radar detection, data related to one or more cameras, etc. The perception data 619 is output by the perception pipeline(s) 815 to the PVM subsystem 820 and the planning system 404.

[0143] The PVM uses the perception data 619 to identify environmental occlusions and to identify / interpret LiDAR data. The PVM can also use the perception data 619 output by the perception pipeline 815 to construct a perception map 607, such as a PoD map related to objects within a vicinity of the AV. Such a PoD map will be described in further detail below. The perception pipeline 815 is also configured to perform similar functions, or generate similar data related to radar data, camera data, or some other type of data generated by or related to the sensor(s) 605. It should also be understood that although Figure 8 Only a single perception pipeline is shown, but in other embodiments, the perception system 402 includes multiple perception pipelines 815. In some embodiments, individual sensors of the sensor(s) 605 have their own perception pipeline, while in other embodiments, at least one of the sensor(s) share a perception pipeline.

[0144] The perception system 402 also includes a PVM subsystem 820. The PVM subsystem 820 is configured to receive the perception data 619 and the sensor / environment data 840 and generate a PVM. Generally, the PVM is a model that indicates a viewable region of the sensors 605 of the AV. As will be described in further detail below, the PVM is used to generate at least one perception map 607, such as a PoD vicinity map for individual objects of interest (e.g., vehicles, pedestrians, objects adjacent to vehicles (such as mailboxes or street lights, etc.), etc.). The PVM further generates perception maps 607, such as an occlusion level map that models environmental occlusions and severity around the AV, etc. The PoD vicinity maps and the occlusion maps are described and discussed in further detail below.

[0145] Generally, the PVM subsystem 820 generates the PVM based on data such as the sensor / environment data 840 and the perception data 619. In embodiments, the PVM is also based on additional data such as localization data 801 provided by the localization system 406. The localization data 801 includes data related to a location or environment of the AV and can be used by the PVM subsystem 820 to identify whether one or more objects (e.g., buildings, overpasses, etc.) are present in a vicinity of the AV that affect the viewability of one or more sensors of the AV. Such data can be provided by, for example, a global navigation satellite system (e.g., GPS).

[0146] In embodiments, the PVM is also based on environmental data 802 indicative of environmental conditions. Such environmental data 802 includes, for example, a weather forecast generated by a weather service with which the environmental subsystem 850 is communicatively coupled. Additionally or alternatively, the environmental data 802 can include data related to (or communicatively coupled to) sensors of the AV, such as a barometer, a humidity sensor, or some other type of sensor indicative of environmental conditions in the vicinity of the AV.

[0147] In embodiments, the PVM subsystem 820 is configured to further base the PVM or its synthesized maps (such as the PoD vicinity map or the occlusion map, etc.) on prior data 806 stored in the PVM prior database 825. Such prior data 806 can include prior PVMs, prior PoD vicinity maps, prior occlusion maps, and / or some other type of data.

[0148] The synthesized map(s) of the PVM subsystem are output by the perception system 402 to the arbiter system 625 to determine the first risk level as described above with respect to the element 720. In some embodiments, one or more of the maps 607 can also be output directly to the intervention request system 630 to determine an intervention based on the results of the maps. For example, in embodiments, the PVM subsystem 820 continuously provides one or more of the maps generated by the PVM to the intervention request system 630, while in another embodiment, the map(s) are provided to the intervention request system 630 only when a parameter of the map(s) reaches or exceeds a threshold, which can be dynamic or static. In this embodiment, the intervention request system 630 is configured to act on the map(s) received from the PVM subsystem 820 that indicate a situation such as extreme occlusion, an object in very close proximity to the AV, etc. that can require an intervention as described above.

[0149] Reference is made to Figure 9 , Figure 9 An example 900 of a PoD map is depicted. As previously described, the perception system 402, and in particular the PVM subsystem 820, is configured to generate at least one PoD map for different objects (e.g., one PoD map for a vehicle, another PoD map for a pedestrian, etc.). Additionally, as noted, the PoD map can be based on one or more prior PoD maps supplied from the PVM prior database 825.

[0150] Example 900 depicts a Point-of-Depth (PoD) map of a vehicle. Specifically, vehicle 200 (e.g., AV) is located on road 930, which can be a street, driveway, highway, avenue, etc. For the purposes of this example 900, the environment adjacent to road 930 is further labeled "off-road" 925. Vehicle 940 (e.g., another vehicle) is located near vehicle 200. Additionally, for the purposes of this example 900, the location of the sun 920 is depicted in front of vehicle 200.

[0151] In this embodiment, the PoD map 905 is a PoD map associated with an agent such as vehicle 940 (e.g., vehicle, pedestrian, bicycle, motorcycle, etc.). Specifically, the PoD map 905 represents a distance threshold at which vehicle 200 is detected by sensors of vehicle 200, such as one or more sensors 605, with a probability of approximately N% (N = 75%) or greater. In some embodiments, the PoD map 905 may be influenced by or based on an azimuth angle (i.e., the angle between the object and the horizon), while in other embodiments, the azimuth angle is not a factor used in the calculation of the PoD map 905.

[0152] As in Figure 9 As can be seen in the example, PoD map 905 roughly covers the surrounding area around vehicle 200. However, it should be noted that PoD map 905 has... Figure 9 The three limitations are depicted. The first limitation is caused by the presence of vehicle 940, which reduces the ability of vehicle 200's sensors to detect vehicles located between vehicle 200 and other vehicles. Another limitation is seen at the sun 920, which reduces the extent of PoD map 905 due to, for example, glare or other lighting-related environmental conditions. Another limitation is seen to the left of vehicle 200, where PoD map 905 generally coincides with the edge of road 930. This limitation may be based, for example, on the presence of obstacles, buildings, or other constraints. It should be understood that this illustration is intended only as an example, and the specific angles, size, or shape of PoD map 905, factors affecting PoD map 905, etc., may vary in different embodiments. Furthermore, while the values ​​used for the PoD map in this example 900 are described as having an example threshold of 75%, other values ​​may be used in other embodiments. In some embodiments, the threshold is selected based on the type of sensor, the type of object associated with the PoD map, or other factors. In an embodiment, the PoD value can be selected based on requirements from modules upstream or downstream of the sensing system 402.

[0153] As previously described, in embodiments, the PVM subsystem 820 is configured to use data related to prior PVMs, prior PoD maps, or some other data in identifying PVMs or PoD maps. Such data can be stored in the PVM prior database 825. Embodiment 900 depicts two such examples of prior PoD maps. Specifically, element 915 depicts an online prior PoD map of small cars with a probability of detection greater than, for example, 75%. As used herein, an "online" prior PoD map describes a prior PoD map related to previous calculations by the PVM subsystem 820. Such a map can be stored in memory of the PVM subsystem 820, the PVM prior database 825, or both. In embodiments, the values of the online prior PoD map 915 are based on an aggregation of values from current usage by vehicles 200 during a particular past window of time (e.g., during the past 5 minutes of usage), or a particular number of samples. In this embodiment, the online prior PoD map 915 can be based on an average of these values, while in other embodiments, the map 915 can be based on a mean, median, maximum, or some other function related to the prior values.

[0154] Element 910 depicts an offline prior PoD map of small motorized vehicles with a probability of detection greater than, for example, 75%. As used herein, an "offline" prior PoD map describes a PoD map stored in the PVM prior database 825 and intended for use in instances where the online map (e.g., element 915) is not available in situations where environmental conditions or sensor conditions have changed significantly such that there is not a large enough number of samples to generate an online map 915. Such an offline map can be considered a default map or a backup map used by the PVM subsystem 820 in the absence of an online map 915. In embodiments, the values related to the offline map 910 are related to a large number of trips, and can be further based on particular environmental conditions such as weather, location, time of day, etc. Similar to the online map 915, the map 910 can be based on a mean, median, maximum, or some other function related to the prior values.

[0155] In embodiments, the map 905 is initially generated based on, for example, perception data 619, sensor / environment data 840, offline prior PoD map 910, some other factor, and / or some combination thereof. The map 905 is then updated based on additional data such as additional perception data 619 and / or additional sensor / environment data 840. In other words, the map 905 is updated as additional data is received. This updating can be periodic (e.g., every x time units), or can be dynamic such as only when new data is received, etc. Un-updated map regions (e.g., map regions for which additional data is not present in received perception data 619 or sensor / environment data 840) can be adjusted toward a prior value such as a value in the offline prior PoD map 910 or online prior PoD map 915. This offset can be incremental according to a predefined measurement unit, or can be dynamic for a midpoint between the current value and the prior value, etc. In embodiments, the PoD can be updated by iteratively combining a prior probability with new information / evidence using a Bayesian update according to equation [1].

[0156] P(H|D) = P(D|H) x P(H) / P(D) [1]

[0157] where H is the event of detecting a particular object (e.g., a pedestrian), D is the data / new information, P(D) is the probability of the observed event (e.g., rain, all lighting) occurring (e.g., calculated using the total probability formula), P(D|H) is the likelihood of that data given the detected event, P(H) is the prior probability, and P(H|D) is the new detection probability.

[0158] Turning to Figure 10 depicts an example 1000 occlusion map. As described above, this occlusion map is generated by the PVM subsystem 820 and depicts areas in which the sensors 605 of the vehicle 200 are occluded or blocked by objects in the vicinity of the vehicle 200. The vicinity of the vehicle 200 includes several objects of varying height that form occlusion zones such as low occlusion zone(s) 1030, medium occlusion zone(s) 1035, and high occlusion zone(s) 1040. As used herein, the terms “low,” “medium,” and “high” are used to differentiate relative heights from one another. In one embodiment, the low occlusion zone 1030 relates to a height between about ground level and about 1 meter off the ground. The medium occlusion zone 1035 relates to a height between about 1 meter and 2 meters off the ground. The high occlusion zone 1040 relates to a height greater than about 2 meters. However, it should be understood that other embodiments have more or fewer occlusion zones, zones with different height parameters, etc.

[0159] Objects within the vicinity of vehicle 200 include small pedestrians 1005 (e.g., children) that will generate a low obstruction zone 1030. These objects also include tall pedestrians 1010 (e.g., adults) and one or more vehicles (such as a car 1020 that includes a medium obstruction zone 1035). These objects also include one or more large vehicles 1015 (such as moving trucks, tractor-trailers, etc.) that include a high obstruction zone 1040.

[0160] It should be noted that in some embodiments, the obstruction area may change over time. For example, as seen at 1045, as the range from the vehicle 200 increases, the obstruction area generated by the tall pedestrian 1010 changes from a medium obstruction area 1035 to a high obstruction area 1040. This change may be caused by factors such as changes in road slope, the type of sensor being obstructed, or other factors. Similarly, at 1050, it can be seen that the obstruction area changes from a medium obstruction area 1035 to a low obstruction area 1030. This change may be caused by road upslope, the shape of the car 1020 (e.g., the obstruction area caused by the interior of the car 1020 differs from the obstruction area caused by the hood or rear of the car 1020), the type of sensor being obstructed, or other factors.

[0161] Go to Figure 11 This describes an example process 1100 related to the generation and use of PVM. In some embodiments, this is achieved by an example sensor pipeline 800 (and more specifically...). Figure 8 The sensing system 402 and PVM subsystem 820 (e.g., fully and / or partially, etc.) perform one or more of the elements described in processing 1100. Additionally or alternatively, in some embodiments, a separate device or set of devices (such as, separate from the sensor pipeline 800) is used. Figure 1 , 2 One or more of the devices in any of the figures 1, 2, 3, 4 and 6, or the sensor pipeline 800, to (e.g., fully and / or partially, etc.) perform one or more elements described with respect to the processing 1200.

[0162] In an embodiment, process 1100 includes determining at least one sensor condition (such as the sensor conditions described above) at 1105. The sensor condition is based on sensor data 606 associated with at least one measurement of the environment, and the sensor data 606 is generated by at least one of the sensors 605. In an embodiment, element 1105 is performed by one or more sensor detectors 810, PVM subsystem 820, or some combination thereof.

[0163] Processing 1100 also includes determining at least one environmental condition (such as the aforementioned environmental conditions) at 1110 based on at least one measurement of the environment. The measurement may be based on, for example, environmental data 802 or sensor data 606. Similar to element 1105, element 1110 may be performed by one or more sensor detectors 810, PVM subsystem 820, or some combination thereof.

[0164] Processing 1100 also includes generating a PVM at 1115 based on the at least one sensor condition, the at least one environmental condition, and the location of the at least one vehicle. In an embodiment, element 1115 is performed by the PVM subsystem 820, and the positioning data 801 is based on data provided by the positioning system 406.

[0165] Processing 1100 also includes identifying a trajectory to be traversed by at least one vehicle at 1120 based on PVM. Element 1120 can be performed, for example, by planning system 404 or by intervention request system 630. In one embodiment, the trajectory is similar to trajectory 520 described above. In another embodiment, as described above, the trajectory involves maneuvering actions such as MRM, RVA request, DMO, etc.

[0166] Go to Figure 12 This describes an alternative example process 1200 related to the generation and use of PVM. In some embodiments, this is achieved by an example sensor pipeline 800 (and more specifically...). Figure 8 The sensing system 402 and PVM subsystem 820 (e.g., fully and / or partially, etc.) perform processing of one or more of the elements described in processing 1200. Additionally or alternatively, in some embodiments, a separate device or set of devices (such as, separate from the sensor pipeline 800) is used. Figure 1 , 2 One or more of the devices in any of the figures 1, 2, 3, 4 and 6, or the sensor pipeline 800, to (e.g., fully and / or partially, etc.) perform one or more elements described with respect to the processing 1200.

[0167] Processing 1200 includes receiving sensor data at 1205. The sensor data is, for example, sensor data 606 received from one or more sensors 605.

[0168] Processing 1200 also includes receiving sensing data from the vehicle's sensing pipeline at 1210. The sensing data is, for example, sensing data 619 received from one or more sensing pipelines 815.

[0169] Processing 1200 also includes detecting at least one sensor condition at 1215 based on sensor data. The sensor condition is detected by a sensor detector such as sensor detector 810, and involves sensor conditions such as faults, blockages, etc., as described above.

[0170] Processing 1200 also includes detecting at 1220 at least one environmental condition based on sensor data. The environmental condition is as described above and indicates the characteristics of the operating environment of the vehicle. In some embodiments, additionally or alternatively, the environmental condition may be based on environmental data 802.

[0171] Processing 1200 also includes generating a current PVM for at least one object detection at 1225 based on the location of the vehicle, at least one sensor condition, and at least one environmental condition. Similar to element 1115, the location data is based on data provided by positioning system 406.

[0172] Processing 1200 also includes determining the vehicle's trajectory at 1230, at least in part, based on perception data 619 and PVM data 618. Similar to element 1230, element 1230 can be performed, for example, by planning system 404 or by intervention request system 630. In one embodiment, the trajectory is similar to trajectory 520. In another embodiment, as described above, the trajectory involves maneuvering actions such as MRM, RVA requests, DMO, etc.

[0173] Example Scenario Assessment

[0174] Figure 13 This is a diagram of an evaluation pipeline 1300 according to some embodiments. The behavior requirements of the AV at a given moment take into account the maneuvering the AV is performing and the AV's current environment. For example, the behavior requirements are relatively fewer when the AV is performing a "car around a parked car" driving scenario in the absence of other vulnerable road users (such as pedestrians or cyclists) compared to when the AV is performing a "lane behind the highway" driving scenario in a busy urban environment. The evaluation system 620 has the task of understanding the maneuvering the AV is performing in the current environment and verifying whether the behavior requirements are met for the current driving scenario.

[0175] The evaluation pipeline 1300 includes a planning system 404, an evaluation system 620, an arbitrator system 625, and an intervention request system 630. For example... Figure 13 As shown, the evaluation system 620 includes a maneuver evaluation subsystem 1303 and an anomaly detection subsystem 1304. Although in Figure 13 Not explicitly shown, but in embodiments, planning system 404 includes control system 408, or is replaced by control system 408.

[0176] Trajectory data 609 is generated by the planning system 404 (and / or the control system 408) based on perception data, map data, and localization data. The trajectory data 609 is input into the maneuver assessment subsystem 1303. The maneuver assessment subsystem 1303 also inputs current and future world states of various agents (e.g., other vehicles) in the vicinity of the trajectory, map data, target assignment, and rulebook, and outputs detection results for the ill-defined scenario (e.g., unsafe indicator data 616) and non-compliance risk analysis (e.g., minimum perception zone data 611). In some embodiments, updates to the perception system 402 are also output.

[0177] The maneuver assessment subsystem 1303 generates minimum perception zone data 611 that is output to the arbiter system 625. In embodiments, the maneuver assessment subsystem 1303 determines whether the trajectory indicated by the trajectory data 609 is in the ODD and whether it complies with pre-defined behavioral requirements, including but not limited to: checking for compliance with regulatory, safety, and comfort rules. For example, if the maneuver assessment subsystem 1303 can verify that the AV took the appropriate action, it can avoid triggering an RVA request or MRM during a challenging situation, e.g., sharing a lane with a cyclist. In another embodiment, the determination is made by the arbiter system 625 based on the minimum perception zone data 611 output by the maneuver assessment subsystem 1303.

[0178] Only the maneuver detection is used to assess the poorly understood scenario (e.g., no pre-defined behavioral requirements). For example, an RVA request or MRM can be expected when a traffic signal officer appears at a malfunctioning traffic light intersection.

[0179] Some examples of maneuver assessment include, but are not limited to, gap analysis and attention zone assessment. Gap analysis checks whether the vehicle 200 maintains a safe gap with other road users. For example, when the vehicle 200 is making a lane change, gap analysis checks whether the vehicle 200 maintains a safe gap with upcoming vehicles in front of and behind the vehicle 200. Attention zone analysis is used to specify the minimum sensor perception zone (e.g., the area covered by the sensors) required for the vehicle 200 to make a safe maneuver.

[0180] The detection results for the ill-defined scenario are input into the anomaly detection subsystem 1304 along with the planner / controller internal state. The anomaly detection subsystem 1304 outputs contextual data, such as unsafe indicator data 616 for the detected anomaly, etc. Examples of anomaly detection include detecting unusual road traffic scenarios, such as traffic accidents, construction zones, and other road users breaking priority, such as “jaywalkers,” or vehicles attempting to run a red light, etc.

[0181] Another example of anomaly detection is "stuck" detection used to determine whether the vehicle 200 is in (or will soon enter) an irresolvable immobility state as described below. These scenarios can require remote intervention, and context related to why the vehicle 200 is immobile that can be used to determine appropriate intervention tasks.

[0182] Context data is input into the intervention request system 630, which selects appropriate interventions for the detected anomaly based on the context data. The intervention request system 630 initiates at least one intervention, including but not limited to remote RVA requests, MRM, and DMO tasks.

[0183] Figure 14 is a flowchart of a process 1400 for scenario evaluation according to some embodiments. The process 1400 can be implemented by Figure 13 the evaluation pipeline 1300 shown.

[0184] The process 1400 begins by receiving input data at 1401. The input data includes trajectories, map data, current / predicted state of the agent(s), and target assignments for vehicles (e.g., destination locations for vehicles).

[0185] The process 1400 continues by evaluating a driving scenario involving a vehicle based on the input data at 1402.

[0186] The process 1400 continues by determining whether the scenario has a defined set of behavioral requirements at 1403.

[0187] According to scenarios that do not have a defined set of behavioral requirements, the process 1400 continues by generating context data for the anomaly event (1404) and assigning intervention tasks at 1405.

[0188] According to scenarios that have a defined set of behavioral requirements (1403), it is determined whether the trajectories / state comply with the defined set of behavioral requirements (1406), and a quantitative measure of non-compliance risk is provided (1407).

[0189] Example immobility detection with scenario context

[0190] ​As previously described, when an AV faces a challenging road situation, the AV can "stall" (e.g., not move for an extended duration). In such a case, it is desirable to understand the context behind the stall (e.g., the cause of the stop) so that a suitable intervention can be requested or made. In this embodiment, it is also desirable to identify a wait time associated with the situation before determining that the AV is stalled. As one example, the wait time used before identifying that a vehicle is "stalled" can be different for different situations, such as waiting at a red traffic light, waiting for a pedestrian to pass at a crosswalk, waiting for a jaywalker to pass through the road, or waiting at an intersection without a traffic light (e.g., all-way stop). For an intersection without a traffic light, the wait time can vary depending on the number of other vehicles present at the intersection. Similarly, certain causes of stalling can be immediately identified as expected to not be resolved, and thus the wait time before requesting intervention can be very short. Such situations can be, for example, a traffic accident, road debris, construction, or an illegally parked vehicle blocking the lane.

[0191] In general, different interventions can be used for different situations in the above situations. For example, if one or more lanes in the road are blocked, the intervention mechanism can be or include rerouting the vehicle. If only one lane is blocked, the intervention mechanism can provide a trajectory for going around the blockage. If the cause of the stop includes yielding to a car without an identified intent to proceed, the intervention mechanism can remove the stop constraint on the AV. Similarly, if the cause of the stop includes yielding to a pedestrian on the sidewalk without an intent to proceed, the intervention mechanism can include marking the pedestrian as not intending to cross.

[0192] At a high level, embodiments include the following techniques. It should be appreciated that this technique, as well as other techniques described with respect to context-dependent immobility detection, can be performed by the immobility detection system 635 and / or the intervention request system 630. In another embodiment, the techniques described herein can additionally or alternatively be performed by at least one other processor, system, or subsystem of the vehicle 200 or a system that is remote from the vehicle 200 but communicatively coupled to the vehicle 200. First, a spatial, velocity, and temporal corridor constraint is constructed using metadata in the form of constraint object identifiers (e.g., agents and map constructs). Then, the technique includes checking for current constraints that result in zero velocity. The cause of the stop can be inferred based on metadata related to these constraints. In particular, the cause of the stop can be inferred without considering the full amount of data outside of these constraints. Then, depending on the context, the process can include applying a timeout mechanism as a threshold for stuck detection (e.g., a vehicle can be considered stuck from the instant of immobility, but only identified as stuck after the timeout mechanism expires). In one embodiment, the threshold depends on an expected “normal” wait time for a given context. In another embodiment, additionally or alternatively, the threshold depends on an expected risk of staying at a current location for too long (e.g., a vehicle should not stop for an extended duration within an intersection due to the risk of obstructing traffic). Then, the technique includes reporting instances of stuck, as well as duration and cause of immobility, to an intervention request subsystem such as the intervention request system 630.

[0193] Figure 15 An example block diagram related to immobility detection with situational context is depicted in accordance with various embodiments. An example of an immobility detection pipeline 1500 includes the immobility detection system 635 and the intervention request system 630.

[0194] The immobility detection system 635 is configured to receive trajectory data 609, for example, from the planning system 404. In particular, the immobility detection system 635 includes a constraint checking subsystem 1510 that is configured to receive data related to a trajectory 1505. The trajectory 1505 can indicate that the vehicle is going to stop (e.g., become immobile).

[0195] The constraint checking subsystem 1510 is configured to identify at least one constraint related to the trajectory 1505. For example, the constraint can include a spatial constraint, a velocity constraint, or a temporal corridor constraint. In some embodiments, the constraint can also include a constraint related to a first time at which the vehicle 200 stopped (e.g., based on a timestamp stored in the constraint database 1515), a most recent time at which the vehicle 200 stopped, a duration for which the vehicle 200 has been stopped, an indication of whether the vehicle 200 has been determined to be immobile, and the like.

[0196] More generally, constraints such as speed constraints can be represented by a function describing a constraint property with respect to path length (S) (e.g., velocity (V), time (T), or lateral offset (D) which can indirectly affect stop decisions if the lateral separation between left / right offsets is too narrow for the AV). For example, if the constraint is a linear function, it can be described by a slope plus an axis intersection, plus an effective range [Smin, Smax].

[0197] Based on the identified constraints, the constraint checking subsystem 1510 identifies a stop cause 1520, which can be a stop cause as described above. The stop cause 1520 data can include, for example, an identifier of the object (e.g., a label, a bounding box), data related to a map location (e.g., a location of the vehicle), or some other data.

[0198] In an embodiment, the stop cause data 613 is provided directly from the constraint checking subsystem 1510 to the intervention request system 630. This can occur, for example, if the stop cause relates to a situation that is not expected to resolve, as described above. In another embodiment, this provision can occur if the stop cause is identified as relating to a situation that is expected to require immediate intervention (e.g., an imminent collision or some other situation).

[0199] Additionally, the stop cause 1520 can be provided to a timeout threshold generator 1525. The timeout threshold generator 1525 is configured to determine a timeout until intervention is to be requested based on the stop cause 1520. Specifically, the timeout threshold generator 1525 is communicatively coupled with a timeout database 1530 configured to store timeout values related to different stop causes. In one embodiment, various timeout values are predefined, while in another embodiment, the timeout database 1530 stores data that can be used by the timeout threshold generator 1525 to calculate a timeout value based on the stop cause (e.g., based on a queue length of cars at an omnidirectional stop, based on a number of pedestrians in the vicinity of the vehicle, etc.). As used herein, a timeout threshold is an amount of time that the vehicle will wait before initiating at least one remedial action, such as a request for an RVA, an MRM, a change in route, etc.

[0200] The timeout threshold 1535 can then be provided to an immobilization timer 1540, which is capable of tracking the length of time that the vehicle is immobilized. If the immobilization situation is resolved, the timeout sequence is cancelled and normal action of the vehicle is resumed. However, if the immobilization situation is not resolved and the time of immobilization meets or exceeds the timeout threshold, the immobilization detection system 635 transmits trigger signal data 614 to the intervention request system 630. The intervention request system 630 is configured to cause intervention based on the trigger signal data 614, including generation and transmission of at least one intervention request 612.

[0201] The following example script describes how the constraint checking subsystem 1510 and / or the constraint database 1515 analyze various speed constraints. It should be understood that the following script is merely an example and can differ according to different programming languages, data structures, values, etc.

[0202]

[0203]

[0204] TrackInfo and MapInfo contain at least a unique identifier (ID). These IDs are used to look up further detailed properties if needed. For example, it can be desirable to retrieve further data related to the track (object / agent) type classification (pedestrian, vehicle, etc.), location, speed, etc. Map objects are stored similarly with unique identifiers, geometry, and relationship properties. Some example map types that can partially describe the reason why the AV stopped in MAP_TYPE enum are shown below.

[0205]

[0206] Figure 16 is a flowchart of an example process 1600 related to immobility detection with situational context according to various embodiments. Specifically, Figure 16 An example is depicted that can describe upon which concepts herein can be based. It should be understood that the example described below is merely intended to be illustrative and that other examples include more, fewer, or different elements, constraints, thresholds, etc. In some embodiments, one or more elements described with respect to the process 1600 are performed by the system depicted in the immobility detection pipeline 1500 (e.g., in whole and / or in part, etc.). Additionally or alternatively, in some embodiments, one or more elements described with respect to the process 1600 are performed (e.g., in whole and / or in part, etc.) by another device or set of devices separate from the example OED framework 600, such as one or more devices of any of the diagrams 1, Figure 1 , 2 , 3, and 4, etc.) or the example OED framework 600.

[0207] Process 1600 includes determining a need for the vehicle 200 to stop and / or yield at 1605. For example, the need for the vehicle 200 to stop and yield to a pedestrian is determined based on the vehicle’s 200 projected time to cross the crosswalk compared to the time for the nearby pedestrian to reach the crosswalk. Thus, a speed constraint is imposed from 2 meters (m) in front of the crosswalk (as measured along the AV path length) to dictate a zero speed rate. This constraint is published with metadata specifying the constraint is caused by the particular crosswalk and pedestrian, as well as the unique IDs of the respective objects. The constraint is republished for several time stamp iterations of the planning system 404, the immobility detection system 635, or some other system or subsystem.

[0208] Process 1600 also includes checking the constraints on respective time stamps (e.g., on respective time stamp iterations) at 1610. In particular, while there can be other valid constraints for limiting speed, spatial offset, etc., the constraint related to the crosswalk is identified as the primary constraint causing the immobility of the vehicle 200 (a zero speed rate constraint that is valid at the current location, other less relevant constraints that can be further calculated along the path, or that are controlled by the zero speed rate constraint and thus similarly less relevant if valid at the current location). Based on the metadata indicating the type of logical constraint (including data related to the identified pedestrian and crosswalk), the vehicle 200 (and in particular the constraint checking subsystem 1510) is configured to know the stop cause 1520 for the current time stamp immobility.

[0209] The stop cause 1520 is output to a timeout threshold generator 1525, and process 1600 further includes assigning a wait time threshold(s) at 1615. In particular, the timeout threshold generator 1525 assigns different wait time thresholds for different stop causes or categories of stop causes. For example, the wait time threshold for yielding at a crosswalk can be different from the wait time threshold for yielding to a jaywalker or some other classification of stop cause. The wait time threshold 1535 is output to the immobility timer 1540 for processing. The immobility timer 1540 tracks the same stop cause persisting over sequential time steps, and compares the current accumulated wait time to the wait time threshold. If the accumulated wait time exceeds the threshold, the immobility detection system 635 determines that the vehicle is “stuck.” For example, a crosswalk user can be assumed to cross the road at a slower pace than a jaywalker, and thus have a longer time threshold. Or as a further refinement, if the crosswalk is a signalized crosswalk, the immobility detection system 635 considers the typical duration of the “walk / green” signal, and does not expect to wait much longer than that.

[0210] The process 1600 also includes generating an intervention request, such as intervention request 612, at 1620. Specifically, upon determining that the vehicle is stuck, the trigger signal data 614 that the timeout is satisfied is output to the intervention request system 630 for generating the intervention request 612. The intervention request 612 can include the context of the stop reason (e.g., stop reason data 613), which in this case includes an indication that the vehicle is waiting for the identified pedestrian at the identified crosswalk. This can be useful for an RVA operator to initiate an override for further progress. For example, if the identified pedestrian can be seen on a video feed and it is clear that the pedestrian is not intending to cross (talking to another stationary person, waiting to cross in an orthogonal direction, or some other visual indication), then the operator can choose to unblock, send an acceleration command, or use some other means to get the vehicle across the crosswalk anyway. Or similarly, the perception system can have misclassified a nearby sign or mailbox as a pedestrian, which can also take a similar visual affirmative and override. Note that in the case where the context of the reason for being immobile is a pedestrian at a crosswalk that is not present, these intervention means can not be obvious and the chosen intervention mechanism can also be different for other immobile reasons (e.g., avoiding a static or slow jaywalker can require changing paths).

[0211] Reference Figure 17 , Figure 17 is a flowchart of an alternative example process 1700 related to immobile detection with contextual context, in accordance with various embodiments. Specifically, Figure 17 Depicted is process 1700 related to immobile timer 1540 and / or constraint check subsystem 1510, and process 1700 can be performed by immobile timer 1540 and / or constraint check subsystem 1510. In some embodiments, one or more elements described in relation to process 1700 are performed by immobile detection pipeline 1500 (e.g., in whole and / or in part, etc.). Additionally or alternatively, in some embodiments, one or more elements described in relation to process 1700 are performed (e.g., in whole and / or in part, etc.) by another device or set of devices separate from immobile detection pipeline 1500, such as one or more devices of any of FIGS. 1-4, etc. Figure 1 , 2 , 3, and 4, etc.) or immobile detection pipeline 1500.

[0212] Process 1700 includes identifying a status of a vehicle at 1705. Such identification includes whether the vehicle has been previously identified as immobile, whether there is data related to an existing constraint or an existing stop reason, etc.

[0213] The process 1700 also includes updating existing constraints at 1710. In particular, if an existing constraint is identified at 1705, the stop reason(s) associated with that existing constraint is checked at 1725. If the stop reason is true (e.g., if the stop reason still exists), the "last seen time" field associated with the constraint or stop reason is updated at 1730. If the stop reason is false (e.g., the stop reason no longer exists), the stop constraint is removed at 1735 such that the inaction timer related to that particular stop constraint or stop reason is canceled.

[0214] The process 1700 also includes identifying remaining stop constraints at 1715. For example, the process includes checking at 1740 whether there are any additional stop situations. For example, in the process, the vehicle can already know of one situation that requires the vehicle to be inaction (e.g., a pedestrian at a crosswalk as described above). However, at 1740, the vehicle will identify whether there are any additional or new situations that will generate a new stop constraint at 1745 (e.g., an additional pedestrian at the crosswalk or a car about to enter the intersection).

[0215] The process 1700 also includes identifying that the vehicle is inaction or stuck at 1720. This identification occurs after the timeout threshold 1535 expires and results in the generation and transmission of the trigger signal data 614 to the intervention request system 630.

[0216] Reference Figure 18 , Figure 18 is a flowchart of an alternative example process 1800 related to inaction detection with situational context in accordance with various embodiments. In some embodiments, one or more elements described with respect to the process 1800 are performed (e.g., entirely and / or partially, etc.) by the inaction detection pipeline 1500. Additionally or alternatively, in some embodiments, one or more elements described with respect to the process 1800 are performed (e.g., entirely and / or partially, etc.) by another device or set of devices separate from the inaction detection pipeline 1500, such as one or more devices of any of the diagrams 1-3 and 4, etc. Figure 1 、 2 , 3 and 4, etc.) or the inaction detection pipeline 1500.

[0217] The process 1800 includes determining at least one current or future constraint of a trajectory of a vehicle in an environment associated with the vehicle becoming inaction for an extended period of time at 1805. As noted, the constraint relates to a spatial, velocity, or time constraint of the vehicle, or some other constraint. The constraint is a constraint of the trajectory that has resulted or will result in the vehicle becoming inaction.

[0218] The process 1800 also includes determining, at 1810, a stationary stop cause for the vehicle 200 based on the determination of the at least one current or future constraint of the trajectory of the vehicle in the environment. As noted, the stop cause is a moving cause. Examples of stop causes include a vehicle 200 waiting at a traffic light, a pedestrian / jaywalker crossing a crosswalk, a vehicle waiting at an intersection without a traffic light, an accident involving the vehicle or closest to the vehicle, a parked vehicle, yielding to another vehicle that does not have forward intent, or some other stop cause. In embodiments, determining the stop cause includes identifying a first time and a last time at which the vehicle became stationary due to the at least one current or future constraint. In embodiments, the stop cause includes an identifier associated with the at least one object or map constraint.

[0219] The process 1800 also includes identifying, at 1815, a timeout threshold based on the stop cause. The timeout threshold is an amount of time that a system of the vehicle (e.g., the intervention request system 630) will wait before initiating at least one remedial action to address the stationarity. In embodiments, element 1815 optionally further includes determining, based on the stop cause, that the stop cause is associated with one of a first set of stop causes and a second set of stop causes. In this embodiment, the first set of stop causes are stop causes that are expected to resolve the stationarity, and the second set of stop causes are stop causes that are not expected to resolve the stationarity. The timeout threshold is then identified based on whether the stop cause is in the first set of stop causes or the second set of stop causes. The timeout threshold can be based on a nominal traffic light wait time, a nominal pedestrian or jaywalker walking rate, a number of vehicles ahead of the AV, a nominal wait time (which can be less than or equal to 1 second in situations where the stationarity is not expected to resolve), a predetermined wait time, etc. In embodiments, the timeout threshold depends on a risk associated with the vehicle being stationary for an extended period of time (e.g., a risk of blocking an intersection). The remedial action can include generating a new route for the vehicle, generating a new trajectory that bypasses the stopped object, removing a constraint related to a vehicle or pedestrian that is identified as not having forward intent, reporting to an RVA assistant that the vehicle is stationary (and the moving cause and / or duration), etc.

[0220] The technique 1800 also includes identifying, at 1820, that the timeout threshold is satisfied, e.g., as described above with respect to the stationarity timer 1540 and the trigger signal data 614. The technique 1800 also includes initiating, at 1825, at least one remedial action for the vehicle based on the identification that the timeout threshold is satisfied (such as the intervention request 612, etc.).

[0221] In the foregoing description, aspects and embodiments of the present disclosure have been described with reference to a number of specific details that can vary depending on implementation. Thus, the description and drawings should not be seen as constituting a limitation on the scope of the application. The sole and exclusive indicator of the scope of the application, and what is intended by the applicants to be the scope of the application, is set forth in the claims issued from this application as published by the United States Patent and Trademark Office, including any subsequent correction of such claim(s). Any definitions of terms set forth in this detailed description are expressly incorporated herein by reference only to the extent that the terms are expressly used in the claims. In addition, whenever a clause contains the phrase "at least one of A and B," this shall be interpreted to mean that "A" or "B" or "A and B" are possible, and any of the foregoing (A and B) can be used in the claims. Further, the use of "and / or", "and", "or", "either" and "one of" in the description and claims is to be taken as setting forth alternatives, and not as being exclusive.

Claims

1. A method for a vehicle, comprising: Using at least one processor, the trajectory of the vehicle is determined based on the location of the vehicle and sensor data associated with at least one sensor; Using the at least one processor, determine whether the trajectory is associated with at least one defined functional requirement; as well as The trajectory is determined to be associated with at least one defined functional requirement: Using the at least one processor, a first risk level of non-compliance of the vehicle is determined based on the at least one defined functional requirement and the execution of the trajectory; Using the at least one processor, an area of ​​interest is determined based on the location of the vehicle and its trajectory; Using the at least one processor, a minimum sensing area is generated based on the trajectory; Using the at least one processor, a second risk level of non-compliance of the vehicle, associated with the vehicle's performance of the trajectory, is assessed based on the minimum sensing area; Using the at least one processor, a total risk level is determined based on a combination of the first risk level and the second risk level; Using the at least one processor, it is determined whether the overall risk level will place the vehicle in a safe or unsafe state; The vehicle is brought to a safe state based on the overall risk level, and travels along the trajectory; as well as Based on the total risk level, the vehicle is placed in an unsafe state, and intervention is requested to bring the vehicle to a safe state.

2. The method according to claim 1, further comprising: Based on the determination that the trajectory is not associated with at least one defined functional requirement, intervention is requested to bring the vehicle into a safe state.

3. The method according to claim 1, wherein, The first risk level is based on a perceptual visibility model of the region of interest, which indicates the probability that at least one sensor of the vehicle in the region of interest detects an object in the sensor data as visible. The assessment of the first risk level includes: The first risk level is assessed based on the probability that at least one sensor of the vehicle in the area of ​​concern detects a visible object in the environment.

4. The method according to claim 3, wherein, Determining a first risk level of non-compliance of the vehicle with the at least one defined functional requirement, while the vehicle is traversing the trajectory, further includes: Determine whether at least one quantitative metric associated with the defined functional requirements is met.

5. The method according to claim 4, wherein, The at least one quantitative metric is associated with at least one regulatory rule, safety rule, or occupant comfort rule.

6. The method according to claim 3, wherein, The perceptual visibility model is based on at least one prior perceptual visibility model.

7. The method according to claim 3, wherein, The probability is determined based on sensor condition data associated with the operating state of at least one sensor of the vehicle.

8. The method according to claim 7, wherein, The operating status indicates whether the field of view of the at least one sensor is obstructed.

9. The method according to claim 7, wherein, The operating status indicates whether at least one sensor has malfunctioned.

10. The method according to claim 3, wherein, The probability is determined based on at least one environmental condition.

11. The method according to claim 10, wherein, The at least one environmental condition is a weather condition.

12. The method according to claim 10, wherein, The at least one environmental condition is a lighting condition.

13. The method according to claim 1, wherein, The intervention is a minimum-risk maneuver.

14. The method according to claim 1, further comprising: Based on the request for intervention, data associated with the control of the vehicle is received, wherein the data is configured to cause the vehicle to take one or more actions.

15. The method according to claim 1, wherein, The minimum sensing area is based on the trajectory.

16. A system for a vehicle, comprising: At least one processor; as well as At least one non-transitory computer-readable medium including instructions that, when executed by at least one processor, cause the vehicle to operate, the operation including: The trajectory of the vehicle is determined based on the location of the vehicle and sensor data associated with at least one sensor; Determine whether the trajectory is associated with at least one defined functional requirement; and The trajectory is determined to be associated with at least one defined functional requirement: The first level of risk of non-compliance of the vehicle is determined based on the at least one defined functional requirement and the execution of the trajectory; The area of ​​interest is determined based on the location of the vehicle and its trajectory; The minimum sensing area is generated based on the trajectory; The second risk level of non-compliance of the vehicle is assessed based on the minimum sensing area and its association with the vehicle's conduct of the trajectory. The total risk level is determined based on the combination of the first risk level and the second risk level. Determine whether the overall risk level will place the vehicle in a safe or unsafe state; The vehicle is brought to a safe state based on the overall risk level, traversing the trajectory; and Based on the total risk level, the vehicle is placed in an unsafe state, and intervention is requested to bring the vehicle to a safe state.

17. The system of claim 16, further comprising: Based on the determination that the trajectory is not associated with at least one defined functional requirement, intervention is requested to bring the vehicle into a safe state.

18. The system according to claim 16, wherein, The first risk level is based on a perceptual visibility model of the region of interest, which indicates the probability that at least one sensor of the vehicle in the region of interest detects an object in the sensor data as visible. The assessment of the first risk level includes: The first risk level is assessed based on the probability that at least one sensor of the vehicle in the area of ​​concern detects a visible object in the environment.

19. The system according to claim 18, wherein, Determining a first risk level of non-compliance of the vehicle with the at least one defined functional requirement, while the vehicle is traversing the trajectory, further includes: Determine whether at least one quantitative metric associated with the defined functional requirements is met.

20. The system according to claim 19, wherein, The at least one quantitative metric is associated with at least one regulatory rule, safety rule, or occupant comfort rule.

Images