Systems and methods for intersection management of autonomous vehicles

Abstract

Systems and methods for autonomously or semi-autonomously navigating an intersection can include, but are not limited to including accessing data related to the geographic location and traffic management features of an intersection; performing autonomous actions to navigate the intersection; and, if necessary, coordinating with one or more processors and / or operators to perform remote actions. Traffic management features can be identified through the use of various types of images, such as oblique images.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 001,329 entitled “Traffic Management Feature Identification”, filed on March 28, 2020 (Agent File No. AA226). Background Technology

[0003] This instruction generally deals with driving through intersections, and specifically with autonomous driving through intersections.

[0004] To safely navigate autonomous vehicles (AVs), caution must be exercised when crossing intersections, paying attention to traffic signs and signals, and avoiding moving and static obstacles. Crossing intersections can be dangerous for any vehicle. According to a 2018 Department of Transportation study, nearly a quarter of fatal traffic accidents and almost 30% of non-fatal traffic accidents occur at intersections. Therefore, AVs must determine, at least based on traffic signs / signals and obstacles within and around the intersection, when it is safe to cross and when it is not. Any type of obstacle avoidance requires knowledge of the position and movement of static and dynamic obstacles near the intersection. Obeying traffic rules requires at least knowledge of the status of any traffic signals. For AVs traveling on pedestrian walkways and motor vehicle lanes, different traffic rules need to be followed. In some cases, it may be necessary to reposition the AV to view traffic signals to correctly determine their status.

[0005] User-operated vehicle drivers use cues such as lane markings on roads, traffic lights indicating priority at intersections, traffic signs, and pedestrian signs and signals. Autonomous perception systems can use these cues in real time using sensors such as radar, cameras, and lidar. These cues can be pre-recorded in maps to simplify and accelerate the operation of real-time systems. Traffic management features marked on the map can be used to indicate when and where vehicles should be able to see traffic management features. By using a map of traffic management features, vehicles can predict when they should see traffic management features and respond appropriately, such as gradually braking to a stop. Using the map, vehicles can predict where traffic management features (such as red lights) might appear in camera images.

[0006] To create maps, perception systems can use driver assistance systems, such as lines painted on roads, and can use alternative sensing modes, such as radar or lidar, instead of vision, to locate traffic management features. Traffic management features can include, but are not limited to, vehicle traffic management features such as lights and signs, and pedestrian traffic management features such as lights and signs. The 3D location and orientation of traffic management features can be estimated from aerial and / or satellite imagery. To estimate the height of traffic management features, a car can be driven through an intersection, and the feature location can be estimated by triangulating multiple images.

[0007] There is a need for systems and methods that enable AVs to safely navigate through intersections of vehicles and pedestrians, determine the location of traffic signals before the navigation session, sense the state of traffic signals, and take appropriate action. Summary of the Invention

[0008] The intersection navigation system described in this teaching solves the problems described herein and other problems through one or a combination of the features described herein.

[0009] Systems and methods for autonomously or semi-autonomously navigating intersections may include, but are not limited to, accessing data related to the geographic location of the intersection; performing autonomous actions to navigate the intersection; and coordinating remote actions with one or more processors and / or operators when necessary.

[0010] Accessing data related to the geographic location of an intersection can include determining the location of traffic signals. Identifying the location of traffic signals is crucial for determining their status. The method of this teaching for identifying traffic management features from at least one oblique image can improve traffic signal recognition. Oblique images can be used as part of a dataset that can reduce data loss due to, for example, sensor congestion. For example, a particular sensor may not recognize a feature covered by vegetation, while a sensor located elsewhere may be able to see the same feature. One possibility is to combine views and data from ground-based sensors with views and data from aerial sensors. Another possibility is to combine views and data from one perspective with views and data from another perspective, possibly object views taken from an oblique angle relative to the other views. A method for viewing objects such as traffic management features from different perspectives can include, but is not limited to, determining the latitude / longitude of the corner of an aerial image area of ​​interest from at least one aerial image; and determining the oblique image area of ​​interest from at least one oblique image based on latitude / longitude. The method may include determining the estimated height of traffic management features; generating bounding box pixels for traffic management features within an oblique image region of interest based on a machine learning process; and calculating the coordinates of the traffic management features based on homography transformation, using the estimated height and the bounding box pixels. Optionally, the method may include using a segmentation model to determine the latitude and longitude of the airspace of interest and discarding traffic management features with heights lower than the estimated height. Traffic management features may optionally include traffic lights, traffic signs, pedestrian signals, and pedestrian markers. The oblique image region of interest may optionally include traffic intersections and / or highway merges.

[0011] In an alternative configuration, the systems and methods of this teaching can use tilted images of intersections to detect the location of traffic management features. The method may include analyzing aerial imagery to determine the location of the intersection. A subdivision model can be used to perform the analysis. The intersection location can be used to locate regions of interest (ROIs) within the tilted image. These ROIs, along with the estimated heights of the traffic management features of interest, can be fed into a machine learning algorithm to select potential features. Features shorter than the estimated height of the traffic management features of interest can be discarded. Homography transformation can be used to compute the coordinates of the traffic management features of interest based on the bounding box pixels and the estimated height.

[0012] When navigating an intersection, an autonomous vehicle (AV) can drive autonomously within a specific lane and perform autonomous actions. The AV can become aware of an approaching intersection and can begin to consider various possibilities. Depending on the expected behavior of the AV and potentially associated remote systems, the list of possibilities presented herein can be reduced or expanded. Ultimately, if the AV is under autonomous control, it can distinguish between different categories of intersections and act accordingly. For example, if the intersection has signs, the AV can use various navigation strategies. Alternatively, if the intersection has signals, another set of strategies can be used. Different strategies can be used if the intersection is a signalized road intersection versus a signalized pedestrian intersection. The system taught in this paper is flexible enough to incorporate various navigation strategies. Furthermore, other users of the road may require the AV to execute specific strategies. For example, by evaluating the time-based persistence of objects at the intersection and assessing the impact of objects present from one time period to the next, static and dynamic objects can be avoided during navigation. Attached Figure Description

[0013] This teaching will be more easily understood by referring to the following description in conjunction with the accompanying drawings, wherein:

[0014] Figure 1A and 1B , Figure 2 and Figure 3 This is a graphical representation of the intersection definition applicable to this teaching;

[0015] Figure 4 This is a schematic block diagram of a configuration for autonomous and remote handshakes in this teaching.

[0016] Figures 5A to 5H This is a flowchart of the teaching method;

[0017] Figure 5I It is a graph of the obstacle weighting factor, which relates the speed of obstacles at an intersection to the distance between the obstacle and the AV.

[0018] Figure 6A and 6B It is a graphical representation of the location of possible obstacles at an intersection;

[0019] Figure 7 This is a schematic diagram of the system of this teaching;

[0020] Figure 8 This is a schematic block diagram illustrating the implementation of the system of this teaching.

[0021] Figure 9 This is a graphical representation of the method taught in this course;

[0022] Figure 10This is an image representation of the bounding box of the traffic management features of this teaching;

[0023] Figure 11 This is a visual representation of the oblique images of the intersections and traffic management features of this teaching;

[0024] Figure 12 yes Figure 11 Image representation of the bounding box of traffic management features;

[0025] Figure 13 This is an image representation of a tilted image of an intersection with pedestrian management features as described in this teaching;

[0026] Figure 14 yes Figure 13 Image representation of the bounding boxes of pedestrian management features;

[0027] Figure 15 This is a visual representation of a type of traffic sign management feature of this teaching;

[0028] Figure 16 yes Figure 15 Image representation of the bounding box of traffic sign management features;

[0029] Figure 17 This is another type of traffic sign management feature represented by a tilted image;

[0030] Figure 18 yes Figure 17 Image representation of the bounding box of traffic sign management features;

[0031] Figure 19 This is a flowchart of the teaching method;

[0032] Figure 20 This is a schematic diagram of the system of this teaching.

[0033] Figures 21 to 24 This is a state diagram of the implementation plan for this teaching;

[0034] Figure 25 This is a schematic block diagram of a system for implementing this teaching; and

[0035] Figure 26A and 26B This is a perspective view of two possible implementations of the AV that can be used to carry out the teachings herein. Detailed Implementation

[0036] The intersection navigation system taught in this guide involves autonomous vehicle navigation. However, various types of applications can utilize the features of this guide.

[0037] Systems and methods for autonomously or semi-autonomously navigating intersections may include, but are not limited to, autonomous vehicles (AVs) performing autonomous actions and, where necessary, coordinating with one or more processors and / or operators to perform remote actions. As a starting point for describing intersection navigation, an AV can autonomously drive within a specific lane, thereby performing autonomous actions. The AV can become aware of an approaching intersection and can begin to consider various possibilities. Depending on the expected behavior of the AV and associated remote systems, the list of possibilities provided herein can be reduced or expanded. Ultimately, if the AV is under autonomous control, it can distinguish between different categories of intersections and act accordingly. For example, if the intersection is marked, the AV can use various navigation strategies. Alternatively, if the intersection is signaled, another set of strategies can be used. If the intersection is a signalized road intersection versus a signalized pedestrian intersection, different strategies can be used. The systems taught in this paper are flexible enough to incorporate various navigation strategies. Furthermore, other users of the road may require the AV to perform specific strategies. For example, by assessing the persistence of objects in the intersection based on time periods, and evaluating the impact of the object's presence from one time period to the next, static and dynamic objects can be avoided during navigation.

[0038] Now for reference Figure 1A To provide a foundation for understanding the problems of autonomous intersection navigation, this paper introduces and describes related terminology. The terminology is not intended to limit the scope of this teaching, but simply to provide a way of referring to terms such as... Figure 1A and 1B The intersection 8003 is illustrated in the diagram. For example, intersection 8003 may include intersection approach lines 8012 and traffic lights 8007. Stop line 8011 is a location that an AV can reach to make a decision about whether to enter intersection 8003 (referred to herein as a pass / don't pass decision). Safe intersection navigation may depend on determining the stop line 8011 of intersection 8003 and providing real-time information about the position of stop line 8011 relative to moving AV 8009. Figure 1B In some configurations, in AV 8009 ( Figure 1B Before reaching intersection 8003, one can know the location of intersection 8003, the traffic control devices located at intersection 8003 (e.g., but not limited to traffic light 8007), the geometry of intersection 8003, and its relationship with AV8009. Figure 1B The associated perception limitations and reference points related to intersection 8003. Using these known values ​​and possibly other known or measured values, such as those for stop line 8011, the position can be determined in real time. In some configurations, all parameters can be determined in real time. AV 8009 can be determined based on known information. Figure 1BThe sensing range around AV 8005 is defined as AV 8009. Figure 1B The area from which traffic light 8007 can be seen is referred to in this paper as the minimum sensing range. The minimum sensing range is the closest distance from which traffic light 8007 can be detected, and is the height of traffic light 8007, AV 8009 ( Figure 1B The field of view of the camera that detects traffic lights 8007 and the relationship between traffic lights 8007 and AV 8009. Figure 1B The stop line 8011 is a function of the distance between the intersection entrance 8012 and the sensing range 8005. The stop line 8011 can be determined by locating a line 8002 between the intersection entrance 8012 and the sensing range 8005, and drawing a tangent 8011 to the sensing range 8005 at the intersection 8013 of the line 8002 and the sensing range 8005. The intersection entrance 8012 can be known in advance, for example, from historical mapping data. An optimal stop line 8011 based on the intersection entrance 8012 can be determined. The intersection 8003 and the intersection entrance 8012 can coincide. If the intersection 8003 includes traffic signals, such as a traffic light 8007, then the stop line 8011 and the intersection entrance 8012 can coincide. The stop line 8011 can be as close to the traffic light 8007 as the intersection entrance 8012, but no closer. If the intersection 8003 includes traffic signs, then, for example, the stop line 8011 and the intersection entrance 8012 are in the same location. If the path taken by the AV requires the AV to travel from the sidewalk to another surface, then stop line 8011 may include discontinuous surface features, such as, but not limited to, curbs. In such cases, if no feature allowing crossing of the discontinuous surface feature does not exist without navigating the discontinuous surface feature (e.g., but not limited to, a curb cut), then stop line 8011 can be determined to be located on the target surface side of the discontinuous surface feature. This teaching assumes that other curb cuts are determined.

[0039] Now for reference Figure 1B If AV 8009 is within sensing range 8005 ( Figure 1A If AV 8009 is too close to traffic light 8007 and cannot see it, then stop line 8011 ( Figure 1A The position of ) can be considered irrelevant to this scene. If AV 8009 is within the perception range of 8005 ( Figure 1A In addition to ), it can be based on the sensing range of 8005 ( Figure 1A The tangent intersection point 8013 Figure 1A Draw stop line 8011 ( Figure 1A ), making stop line 8011 ( Figure 1A The direction of the traffic light 8007 is oriented to achieve optimal visibility. If the perception range 8005 ( Figure 1AThe radius of the intersection point is 8013. Figure 1A ) fell at the intersection 8003 ( Figure 1A Within ) then stop line 8011 ( Figure 1A It can be set as intersection entrance 8012. Figure 1A In some configurations, the calculated sensing range is 8005 ( Figure 1A The radius 8016 can include determining the field of view of the sensor associated with AV 8009, such as the sensor height acquired at the midpoint of the sensor, and the signal height acquired, for example, at the top of the signal. The radius 8016 can be calculated according to the following formula:

[0040] Radius = (Signal height - Sensor height) / (tan(Sensor field of view in radians) / 2)

[0041] In some configurations, the sensor may include a camera mounted on the directional forward side of the AV 8009. Other sensor options are contemplated in this teaching. For example, the signal may include a road signal or a pedestrian signal.

[0042] Now for reference Figure 2 Generally, given a distance of 9405 to stop line 8011, the position of non-return point (PNR) 9401 can be calculated as AV 8009 moves. The purpose of PNR 9401 is to stop as smoothly as possible at stop line 8011. PNR 9401 is the intersection entry point 8012 ( Figure 1AThe PNR 9401 is a point between the maximum sensing range and the stop line 8011. The PNR 9401 can fall anywhere between the maximum sensing range and the stop line 8011. When the AV 8009 is at the stop line 8011, the PNR 9401 is juxtaposed with the stop line 8011. The PNR 9401 is a point relative to the stop line 8011 where, if the brakes are applied, the AV 8009 will stop before the stop line 8011, thus avoiding damage to the stop line 8011 and allowing for a smooth stop at the stop line 8011. The PNR 9401 is a function of the current speed of the AV 8009. The PNR 9401 is a point behind or at the stop line 8011, whose distance 9405 from the stop line 8011 is equal to the braking distance 9407 of the AV 8009 at the current speed. The PNR 9401 can include a range or area 9417, for example, within 0.5m of the PNR 9401. Therefore, any PNR-related speed adjustments begin from the PNR 9401 range before AV 8009 reaches the actual PNR. If AV 8009 reaches PNR zone 9417, the braking sequence can begin when deceleration is applied to the current speed. The primary objective of PNR 9401 is to prevent AV 8009 from abruptly stopping / interrupting / hard stopping as much as possible. PNR 9401 is located at the braking distance after stop line 8011, such that if deceleration 9403 (e.g., but not limited to, -0.5 m / s) is iteratively applied while AV 8009 is traveling, the braking distance is such that... 2 up to -3.1m / s 2 If a pre-selected deceleration is applied to the speed of AV 8009 at PNR 9401, AV 8009 can stop at stop line 8011. Therefore, PNR 9401 changes with the current speed of AV 8009. When AV 8009 approaches stop line 8011, the traffic signal suddenly turns yellow / red / unknown, making a hard stop the correct choice to protect the safety of AV 8009 and other AVs sharing the route. In some configurations,

[0043] Braking distance = distance between PNR 9401 and stop line 8011 (in meters)

[0044] = (Current speed of AV 8009 (in m / s)) 2 / (2.0 * deceleration rate (in m / s) 2 (as a unit)

[0045] The deceleration rate can be determined based on pre-selected criteria. PNR 9401 is evaluated to ensure that AV 8009 recognizes its location at PNR 9401 so that AV 8009 can begin braking before reaching the PNR 9401 zone. The PNR 9401 zone can begin approximately 0.5 m ahead of the actual PNR position. The speed is changed based on the AV's position relative to PNR 9401, as shown in Table I. The reduced speed is calculated as follows:

[0046] MMS(t) = max(0.0, MMS(t-ΔT) – (deceleration * ΔT))

[0047] The maximum speed of the manager (MMS) in the first deceleration cycle is equal to the speed of AV 8009 at time t. In subsequent iterations, the speed can be expressed in m / s over a time interval ΔT between two consecutive pre-selected time intervals. 2 The deceleration rate decreases.

[0048] Now for reference Figure 3 When deceleration is required, the distance to stop line 8011 is needed. When AV 8009 is traveling in lane 8022, multiple parameters can be accessed and / or determined regardless of AV 8009's position relative to road 8024. In some configurations, for example, the bottom boundary 8006 can be derived from the left boundary 8018 and right boundary 8020, which can be determined based on historical map data. (As can be...) Figure 3 As seen in the diagram, when the lane includes a turn, distance 8015 is longer than distance 8032 because distance represents the minimum and maximum length of lane 8022. Given the current position of AV 8009 and other map parameters described herein, dF 8004 (the shortest distance between AV 8009 and the front boundary), dL 8002 (the distance between AV 8009 and the left lane boundary 8018), dR 8024 (the distance between AV 8009 and the right lane boundary 8020), and dB 8026 (the distance between AV 8009 and the bottom boundary 8006) can be calculated. These calculations can be used to calculate the distance to stop line 8011 as a weighted average of the distances along the left boundary 8018 and the right boundary 8020. For example, the distance from the indicated position of AV 8009 to stop line 8011 can be calculated as follows:

[0049] Ratio = dR / (dR / )

[0050] Distance = Ratio * Distance 8015 + (1 - Ratio) * Distance 8032

[0051] Continue to refer to Figure 3Before reaching PNR 9401, AV 8009 continued to travel at the maximum speed described above. Upon reaching PNR 9401, the maximum speed was, for example, -3.1 m / s. 2 to -0.5m / s 2 When the deceleration setting in the range is 0.0 m / s or during deceleration, deceleration can be applied to the speed of the AV 8009 until the actual or virtual stop line. The table below lists the actions taken by the AV8009 in some of the situations described herein.

[0052]

[0053]

[0054] Table I

[0055] Now for reference Figure 4 The AV can drive autonomously until it encounters an intersection, at which point it invokes the intersection navigation procedures described herein. These procedures are provided when the intersection navigation procedures are invoked, and possibly throughout the navigation process at the intersection, and include, but are not limited to, information on driving lanes, traffic light status, maximum speed, the presence of dynamic and static obstacles, and the current stop line. Lane information and obstacles can be used to help the intersection navigation procedures determine whether remote assistance is needed to cross the intersection. The intersection navigation procedures can use maximum speed, traffic light status, and the current stop line to enforce rules associated with various types of intersections. Intersection categories can include marked, signaled, and remotely controlled. As the AV approaches the current stop line, the AV determines the type of intersection it encounters, for example, by evaluating information provided to the AV. For example, marked intersections can include right-of-way, stop, coasting stop, and yield. For example, signaled intersections can include intersections with both traffic lights and pedestrian lights. Remotely controlled intersections can include intersections of unknown type. When the AV determines that an intersection is appearing in its navigation path, if the AV is not remotely controlled at or within the intersection, it begins intersection navigation processing based on intersection classification. Traffic lights and pedestrian lights are processed according to road traffic light and pedestrian traffic light rules, respectively. Right-of-way, stop, coasting stop, and yield are processed according to the rules for marked intersections. These processing steps can be interrupted by hard exit, for example, but not limited to route recalculation, or if the current stop line has a processed stop line. After reaching the minimum sensing range of the current intersection, an upcoming intersection's stop line may be encountered. The AV processes the stop line of the current intersection before processing the stop line of the upcoming intersection.

[0056] Continue to refer to Figure 4In some configurations, when the AV anticipates crossing an intersection, it can come to a complete stop and alert the remote control processor. The remote control processor can confirm sufficient cellular signal strength, prevent signal degradation, and, at some types of intersections, can drive the AV through the intersection. At some intersection types, the AV can plan its route and cross autonomously. Intersections that the AV can autonomously cross may include (but are not limited to) traffic lights (pedestrian and road) with or without pedestrian crossings, four-way stop signs, two-way stop signs, yield signs, and virtual right-of-way.

[0057] Continue to refer to Figure 4The situations that can be adapted by implementing the system of this teaching may include (but are not limited to) invalid or potentially invalid sensing data collected by the AV, difficult intersections, obstacles in the intersection, complex intersections, and AV orientation problems. Regarding invalid or potentially invalid sensing data, the system of this teaching may transfer control to a remote system, which may stop the AV, for example, but not limited to, when it is detected that the AV is making an incorrect decision, for example, based on invalid data. Other responses to potentially invalid sensing data are considered, including repositioning the AV and / or executing internal filtering and correction instructions. If the remote system is invoked, the remote system may stop the AV at the current stop line. Autonomous control may be returned to the AV when it is certain that the data received by the AV is correct and that it is safe to cross the intersection. If the AV has already crossed the line representing the minimum sensing range marker (also referred to herein as the distance from the current stop line) and a stop request is sent from the remote system to the AV, control may be retained in the remote system until the AV is no longer located at the intersection. When it is known in advance that the intersection is difficult to cross autonomously, such as, but not limited to, a railway crossing, the AV may wait at the current stop line and request remote assistance to determine whether it is safe to cross the intersection. If there are numerous obstacles at the intersection, the AV can request to transfer control to a remote system that can determine whether it is safe to cross the intersection. When the remote system determines that the intersection is clear or at least safe, it can return autonomous control to the AV. When it is known in advance that the AV will enter a complex intersection, the AV can transfer control to the remote system until the remote system determines that the AV has crossed the intersection. Such complex intersections may include left-turn intersections. When it is known that the curb ramp between the pedestrian crossing and the crosswalk does not face the pedestrian traffic light, the AV can rotate at the stop line to bring the pedestrian traffic light into the AV's field of view. Rotation may occur if the AV is not within the pre-selected field of view of the pedestrian traffic light, for example, but not limited to 40°. When the rotation is complete, the AV can wait a pre-selected amount of time to gain stability from the rotation and can then detect the traffic light and act accordingly. For example, according to U.S. Patent Application #17 / 214045, filed concurrently with this application and entitled "System and Method for Navigating a Turn by an Autonomous Vehicle", the AV can rotate autonomously.

[0058] Further reference is needed. Figure 4In one implementation of the system taught herein, method 9150 may include a handshake protocol between the autonomous AV system and a remote system / person (collectively referred to herein as a remote control processor). If 9151 the AV is not approaching an intersection, method 9150 may include continuing 9165 autonomous driving and continuing 9151 looking out at the intersection. If the AV is approaching the intersection, and if 9175 the remote control processor notices that the decision being made is inconsistent with the expected behavior of the AV, the remote control processor sends a stop command 9177 to the AV. In some configurations, control does not need to be transferred from the AV to the remote control processor when the remote control processor suspects that the AV will make an inconsistent decision. Similarly, if the remote control processor finds that the actions taken by the AV are consistent, control does not need to be transferred from the remote control processor to the AV. If the remote control processor has taken over, the AV waits at the current stop line until the remote control processor can transfer control to the AV. If 9176 the AV has the right of way at the intersection, the remote control processor can transfer control to the AV, allowing the AV to drive autonomously again 9165. If 9176AV does not have the right-of-way at the intersection, the remote control processor may maintain control of the AV until the AV has the right-of-way. If 9151AV is approaching the intersection without intervening with the remote control processor, and if 9159 classifies the intersection as one requiring remote control (i.e., unable to drive autonomously) based on factors such as whether the AV's route requires a left turn across the intersection, then method 9150 may include stopping 9167AV at the current stop line and transferring control to the remote system. If 9179AV has already crossed the minimum perception line relative to the stop line, then method 9150 may include the remote system sending command 9181 to the AV to drive through the intersection under the control of the remote system, and then returning control to the AV system upon completion of the intersection crossing. If 9179AV has not yet crossed the minimum perception line, then method 9150 may include returning autonomous control to the AV. If intersection 9159 falls under the category of marked intersections, then method 9150 may include subsequent rule 9169 for marked intersections, and if a hard exit is received at 9171, then method 9150 may include returning to autonomous driving at 9165. If a hard exit is not recognized at 9171, then method 9150 may include continuing to test whether the AV is approaching the intersection. If intersection 9159 falls under the category of signalized intersections, and if intersection 9161 falls under the category of pedestrian intersections, then method 9150 may include subsequent rule 9170 for signalized pedestrian intersections, and if a hard exit is received at 9171, then method 9150 may include returning to autonomous driving at 9165.If intersection 9161 falls into the category of road intersections, then method 9150 may include the following rule 9163 for signalized road intersections, and if a hard exit is received at 9171, then method 9150 may include returning to autonomous driving 9165.

[0059] Now for reference Figures 5A to 5I Implementations of the autonomous navigation methods taught in this specification may include exemplary methods for handling various situations that an AV may encounter when navigating an intersection. Variations of the example methods for intersection navigation taught in this specification are considered and covered in this specification.

[0060] Now for reference Figure 5A Method 9050 initiates the implementation of the method of this teaching. The description is not intended to be limiting and other features and processing steps are considered in this teaching. Method 9050 for navigating an intersection may include (but is not limited to) receiving 9051 an alert of an expected intersection in the travel path. If 9053 a traffic sign is present at the intersection, method 9050 may include performing steps to navigate the AV through the signified intersection. If 9053 a traffic signal is present at the intersection, method 9050 may include performing steps to navigate the AV through the signalized intersection. If 9053 the type of intersection is unknown, or if the AV is under remote control, method 9050 may include performing steps to navigate the AV through an unknown type of intersection. Type-specific processing provides the travel speed of the AV in the case of the intersection type. When the type-specific processing is complete, method 9050 may include sending a speed value 9055 to a controller, which may instruct the AV to travel through the intersection at the desired speed. When the speed is sent to the controller, method 9050 may include waiting to receive notification of another intersection in the driving path.

[0061] Now for reference Figure 5B When the AV is approaching an intersection including traffic signals, embodiments of this teaching may include processing steps specific to signalized intersections. Other embodiments are contemplated. Method 9450 for navigating a signalized intersection may include, but is not limited to, determining, for example, a point of no return (PNR) 9451 as described herein or by other methods. If 9453 the AV is not traveling on the road, method 9450 may include rotating 9455 the AV at the stop line to face the traffic light so that the AV can clearly see the light's status. If 9453 the AV is traveling on the road, or when the AV has already rotated to face the traffic light, and if 9459 the traffic light is green, method 9450 may include setting the speed 9457 to the maximum speed for the road type and sending the speed 9055 ( Figure 5AIf the traffic light state is unknown (9459), and if the AV is located at the stop line provided to the AV based on the current driving rules (9461), and if the time the traffic light has been in an unknown state is greater than or equal to a pre-selected time amount (9463), then method 9450 may include setting the AV's speed (9471) to zero, sending the speed (9473) to the AV's speed controller, and transferring control (9475) to a remote control to request check-in. If the traffic light state is unknown (9459), and if the AV is not at the stop line (9461), then method 9450 may include calculating the AV's reduced speed based on the PNR's position and sending the speed (9055) to the AV's speed controller. Figure 5A If the traffic light status is unknown (9459), and if AV is at the stop line (9461), and if the time the traffic light has been in the unknown state is less than a preselected time, then method 9450 may include reducing the speed of AV to 0.0 m / s, sending the speed (9473) to the speed controller of AV, and transferring control (9475) to remote control. If the traffic light status is red / yellow (9459), and if AV has reached the PNR (Personal Noise Reduction Network) (9253), then method 9250 (…). Figure 5D This can include calculating 9255 based on PNR. Figure 5D The speed decreases, and the speed is sent to 9055 ( Figure 5A (to the AV speed controller. Otherwise, method 9250) Figure 5D This can include setting the AV speed to 9257 ( Figure 5D The maximum speed described above is achieved, and the speed is sent to 9055 ( Figure 5A () to the AV speed controller.

[0062] Now for reference Figure 5C If 9053 ( Figure 5A If a traffic sign is present at the intersection, and if 9551AV has the right-of-way at the intersection, then method 9550 may include setting the speed 9553 to the maximum speed for that situation, and sending the speed 9055. Figure 5A ) to the speed controller of AV. If 9551AV does not have right-of-way, and if 9253 ( Figure 5D If AV has reached PNR, then method 9250 ( Figure 5D This can include setting the AV speed to 9255 ( Figure 5D The speed decreases until it reaches the stop line, and the speed is sent to 9055 ( Figure 5A ) to the speed controller of AV. If 9551AV does not have right-of-way, and if 9253 ( Figure 5D If AV has not yet reached PNR, method 9250 may include setting the speed of AV to 9257. Figure 5D) to the maximum value used for this situation, and send the speed 9055 ( Figure 5A The speed controller of AV is then used. If 9557AV has not yet reached the stop line, method 9550 may include returning to method 9250. Figure 5D The method 9550 determines whether the PNR is ahead of or has been reached on the driving path. If 9557AV has reached the stop line, method 9550 may include sending a stop line arrival message 9561 to the dynamic obstacle processor, receiving the obstacle's location 9563, and waiting at the stop line for a preselected time amount 9565. In some configurations, the preselected time amount is 5 seconds. Other waiting times are also possible. If 9557AV has reached the stop line, and if 9561 there is no obstacle at the intersection, method 9550 may include waiting at the stop line for the preselected time amount 9565, and then managing the case where there is no obstacle at the intersection.

[0063] Now for reference Figure 5EManaging marked (virtual or physical) intersections where obstacles may exist may require the AV to avoid encountering obstacles in its projection path. Virtual traffic signs can be specified in areas where there may be no physical signs, but this may require similar or identical processing as when physical signs are present at the intersection. Navigating intersections with or without obstacles involves deciding whether to enter the intersection based on factors including the presence of obstacles within the intersection. In real-world traffic conditions, the decision must include whether the obstacle will obstruct the navigation path. The method described herein can be used to quantify the likelihood of road congestion and, by doing so, to achieve safe navigation at intersections. The method can determine whether an obstacle is predicted in the navigation path during a certain percentage of a sliding window within the decision time frame. In some configurations, the sliding window can be a 3-second window, and the decision time frame can be 20 seconds. Any time value can be used for the window and time frame, at least based on local conditions. The window can slide over time, and the AV can use information about the obstacle during the sliding time window to determine whether to enter the intersection. One objective is to avoid initiating a new sliding window each time a pre-selected window expires. In some configurations, the decision time frame corresponds to the time the AV waits at the stop line before requesting remote assistance, for example, but not limited to, check-in. The remote control process can assess the surrounding environment and may send a start request to trigger the AV to autonomously cross the intersection. If the AV is informed of an obstacle in the intersection, the data provided to the AV may still include false or illusory obstacles. Due to the potential for unreliable data, there may be a built-in time for the AV to wait for the sensors to stabilize or other methods that enable reliable obstacle readings. When requesting assistance from the remote control process, the remote control process can assess the AV's surrounding environment and send a start request, enabling the AV to autonomously cross the intersection. A pre-selected time amount can be chosen at 20 seconds.

[0064] Continue to refer to Figure 5EThe persistent presence of objects at intersections can be used to fine-tune object avoidance. The basic idea behind obstacle persistence is that if at least one obstacle exists at the intersection within a pre-selected time window within a pre-selected time frame, the AV will not enter the intersection. Calculating the persistence value may include calculating a value referred to herein as a weighting factor, which provides a measure of the importance of the appearance and disappearance of obstacles at the intersection. Since the appearance and disappearance of obstacles occur over a period of time, the AV can maintain at least two timers associated with obstacles at the intersection. The first timer is called a time frame, which is divided into windows with a second timer. The time frame is used to measure the time period during which obstacles and intersections are observed. The amount of time in the time frame and the amount of time in the window can be constant, can vary during navigation, can be dynamically determined, or any other method suitable for a particular navigation. When the amount of time in the time frame has expired and the AV is still at the stop line waiting for the obstacle to be cleared at the intersection, control can be transferred to a remote control processor that may navigate the AV through the intersection. If the amount of time in the time frame has not yet expired, the obstacle weighting factor process can continue.

[0065] Continue to refer to Figure 5E The AV can receive a trajectory information vector, where each element of the vector represents a static / dynamic obstacle trajectory. Static obstacle trajectory information can provide instantaneous object information without the associated object velocity. Among other things, the trajectory information can be used to determine (1) the velocity of the fastest dynamic obstacle, (2) the distance between the AV and the closest dynamic obstacle to the AV, and (3) the distance between the AV and the closest static obstacle to the AV. This information can be converted into a weighting factor that can be between 0 and 1. Figure 5I The weighting factor function 9201 is shown. Figure 5I ) and speed 9203 ( Figure 5I ) or distance 9205 ( Figure 5I (The graphic)

[0066] Continue to refer to Figure 5E One of the weighting factors, w time Weight the progress achieved through the window. time The value of w can decrease linearly or according to any other function, and can eventually decrease to zero at the end of the window. As the amount of time the window occupies elapses, that is, when a subsequent time period begins and a new window and possibly a new timeframe begins, w... time The value can be reset. Through a time window, w time= 1.0 – (Current Loop Count / Total Loop Count). This is the number of obstacles within the AV's sensor range. The total loop count is calculated as the loop rate multiplied by the number of seconds within the window. For example, at a loop rate of 20Hz, the loop count is 40 at 2 seconds into a 3-second window. During this process, weights are assigned to the type of obstacle, such as, but not limited to, dynamic and static obstacles, typically depending on the distance between the AV and the obstacle. Weighting factors can also be assigned to the speed at which the obstacle moves. The final weighting factor calculation is the maximum value of the calculated weighting factors. The obstacle weighting factor (moving (dynamic) and static) w at an intersection can be calculated as follows: obstacle :

[0067] w speed =1.0–e -Av It is a weighting factor based on the fastest dynamic obstacle speed v (in meters per second), and the value of A is a function of the allowable speed of the obstacle on the road. In some configurations, A can be determined empirically and can take values ​​such as, but not limited to, 0.16.

[0068] w dynamicDist =-d d / B+1.0 is based on the fastest dynamic distance (in meters) between the obstacle and the AV. d The weighting factor, B, is a function of the AV's sensing range. In some configurations, the sensor is radar with a maximum range of 40 meters.

[0069] w staticDist =-d s / B+1.0 is based on the distance (in meters) between the nearest static obstacle and the AV. s The weighting factor, B, is based on the perception range of AV.

[0070] w obstacleMax (t)=max(w speed, w dynamicDist, w staticDist ) is the weighting factor for obstacles in the current window at the current time.

[0071] w obstacle =(w obstacleMax(t) )

[0072] Using these weighting factors, persistence can be calculated as (w time +w obstacle ) / 2.

[0073] When persistence is calculated as w time and w obstacle When the average value is used, the result is that AV must wait a certain amount of time before it can proceed through the intersection. Even if wobstacle The value is 0.0 when using w. time and w obstacle When the average value is reached, AV may not proceed through the intersection until w time Reaching a non-zero value, such as 0.4, corresponds to a weighting factor of 1.8 seconds. A persistently low value indicates that it is safe for the AV to enter the intersection, while a high value indicates that it is safer to remain at the stop line. Specifically, in some configurations, a value greater than 0.2 can indicate that the AV should remain at the stop line. In some configurations, when A = 0.16 and B = 40m, an obstacle 23m away from the AV traveling at 3.2m / s will trigger a decision to remain at the stop line.

[0074] Continue to refer to Figure 5E Method 8150 can provide one implementation of persistence calculation. Specifically, if 8151AV has been waiting at the stop line for more than a preselected time frame, method 8150 may include setting the maximum speed 8153 to 0.0 m / s, i.e., stopping at the stop line, and transferring control 8155 to a remote control processor. If 8151AV has not been waiting at the stop line for more than a preselected time frame, method 8150 may include calculating 8157 a weighting factor for calculating obstacle persistence. The weighting factor may include (but is not limited to) a weighting factor based on the speed of the fastest dynamic obstacle, the distance between the AV and the nearest dynamic obstacle, the distance between the AV and the nearest static obstacle, and the maximum value of the weighting factor calculated at the current time.

[0075] Now for reference Figure 5F After calculating the weighting factor, the time segments at the intersection where there are no obstacles can be evaluated. If 8351 the difference between the current time and the start time of the previous time window is greater than a pre-selected time amount, and if 8353 the difference between the current time and the start time of the obstacle-free segment is less than a pre-selected time amount, then method 8350 may include setting 8355 the start time of the window in the current cycle as the time of the obstacle-free segment. Otherwise, method 8350 may include setting 8357 the start time of the obstacle-free segment to the current cycle time, and setting the start time of the window in the current cycle time to the start time of the obstacle-free segment.

[0076] Currently, the main reference is... Figure 5GIf an obstacle exists at the intersection, method 8450 can calculate the weighting factor for the obstacle segment. If an obstacle exists at the intersection (8451), method 8450 can include determining an influence factor and setting the weighting factor for the obstacle-free segment at the current loop time (8455) to the obstacle weighting factor at the current loop time. If no obstacle exists at the intersection (8451), method 8250 can include setting the time of the obstacle-free segment (8452) to the current loop time, and further determining an influence factor and setting the weighting factor for the obstacle-free segment at the current time (8454) to the obstacle weighting factor at the current time. The 3-second time window within a 20-second timeframe is a sliding window. This window slides over time, and the AV makes consecutive no-pass decisions when an obstacle exists at the intersection. The goal is to avoid starting a new window every 3 seconds. This is because even if a previous window resulted in a high persistence value, there may only be an obstacle at the beginning of the window. To determine the position of the sliding 3-second window, track the most recent obstacle-free segment. This segment points to the portion of the recent past when there were no obstacles, and the accumulated w can be used as part of the next 3-second window. obstacle Weighting factor. This process may result in more timely entry into intersections.

[0077] Continue to refer to Figure 5G To determine where to slide the window to subsequent time periods, the latest obstacle-free segment statistics can be tracked to identify the portion of the sliding window that was obstacle-free in the most recent past. In some configurations, for each 3-second time period, the last time period during which no obstacle was detected can be recorded. For example, if there is no obstacle at the end of a 3-second window, then it is not necessary to restart each new 3-second window, even if the cumulative weighting factor of the 3-second window exceeds 8. One way to accommodate this is to track the last obstacle-free segment. The last obstacle-free segment may include the time between the observation of obstacles. The segment may end or may continue. If at least one boundary of the segment is determined, the cumulative w from the start of the segment to the current time is calculated. obstacle When the next 3-second window begins, if there is an unobstructed section, the accumulated w... obstacle It can be associated with the start time of the window. The latestObstacleAbsentStreak object can include the following information:

[0078] mStreakStartTimestamp = the start time of the latest segment. This is updated whenever a break occurs in a segment and a new segment begins;

[0079] mCycleCount = The number of cycles that have occurred since mStreakStartTimestamp;

[0080] mTotalWeight = w of the current loop obstacle ;as well as

[0081] mStreakEnd = True if the latest segment has ended, otherwise False.

[0082] If false, the latest segment remains the current segment, and no obstacles have been seen since the start of the latest segment. When the current 3-second window slides due to expiration and makes a no-pass decision, the window slides to mStreakStartTimestamp, provided mStreakStartTimestamp is not 3 seconds or older. If in the past mStreakStartTimestamp ≥ 3 seconds, mStreakStartTimestamp is reset, and the sliding window moves to start at the current timestamp. This process indicates the accumulated w obstacle Could it be used as part of the next sliding window, and could potentially lead to more timely passage decisions?

[0083] Currently, the main reference is... Figure 5H Method 8250 can determine the impact factor and use it to modify the obstacle weighting factor. The impact factor indicates the importance of the obstacle observation weighting factor. In some configurations, possible values ​​for the impact factor can be obtained empirically. In some configurations, the impact factor can vary between 0.5 and 1.2. time The value of w allows persistence to decrease as the time window ends. When persistence decreases, it can be decided to continue through the intersection. For example, if an obstacle with a high weighting factor suddenly appears, AV should ideally remain at the stop line due to high speed or very close proximity. A high impact factor increases w obstacle The importance of the weighting factor. If an obstacle providing a high weighting factor suddenly disappears, this could be due to a variety of reasons. For example, the sensor may malfunction, or the obstacle may have moved to a location where the available sensors cannot receive data. The disappearance leads to w obstacleMax The value of (t) = 0.0, which may change w. obstacleThe value of the impact factor is used to prevent the AV from inappropriately entering the intersection. Therefore, in some cases, the impact factor of suddenly disappearing obstacles can be reduced. Increasing the impact factor can provide a safety measure to ensure that there are no obstacles before the AV enters the intersection. If the obstacle persistence in the previous time window is less than or equal to 0.2, a first pre-selected value can be assigned to the impact factor. When the obstacle persistence in the previous window is greater than 0.2 and the obstacle persistence in the current time window is less than or equal to 0.2, a second pre-selected value can be assigned to the impact factor. The first pre-selected value can be less than the second pre-selected value. The values ​​and their magnitudes relative to each other can be constants determined empirically, or they can be dynamically determined and change during navigation. When using the impact factor, the obstacle weighting factor in the current loop can be modified by the obstacle weighting factor in the previous loop, so that the weighting factor reflects the importance of obstacles in both the previous and current loops. In some configurations, the impact factor can take values ​​such as 1.2 as the default and second preselected value, and 0.5 as the first preselected value if, for example, one or more obstacles suddenly disappear. This may cause the AV to enter the intersection when the weighted factor of the input observed obstacles is low.

[0084] Continue to refer to Figure 5H The implementation of the persistence and impact factor strategy of this teaching may include method 8250 for modifying the barrier weighting factor at the current time. In method 8150 ( Figure 5E ) Increased by 8159 ( Figure 5E After the loop count, method 8250 can be called. If the persistence of 8251 as a function of the obstacle weighting factor in the previous loop is greater than a preselected value, for example, but not limited to 0.2, and if the persistence of 8251 as a function of the maximum weighting factor in the current loop is less than or equal to the preselected value, and if the difference between the obstacle weighting factor in the previous loop and the maximum weighting factor in the current loop is greater than or equal to the preselected value, for example, 0.2, then method 8250 may include setting the influence factor (IF) 8257 to a preselected first value, for example, but not limited to 0.5, and setting the obstacle weight in the current loop to w in the current loop. obstacle (t)=(w obstacle (t-1)+IF*w obstacleMax (t)) / (1+IF). Otherwise, method 8250 may include setting the impact factor (IF) to a pre-selected second value, such as, but not limited to, 1.2, and setting the obstacle weighting factor at the current time to w. obstacle (t)=(w obstacle (t-1)+IF*w obstacleMax (t)) / (1+IF).

[0085] Refer again Figure 5E Method 8150 can perform persistence calculations by setting the obstacle weighting factor of the previous loop to the obstacle weighting factor of the current loop; setting the time weighting factor of the current loop to w. time (t) = 1.0 - (percentage of the process in the 3-second window); and the persistence calculation 8165 is the average of the obstacle weighting factor and the time weighting factor in the current cycle.

[0086] Now for reference Figure 5E , 5F For example, if, within the first 3-second window of a 20-second decision timeframe, it is determined whether an obstacle is too large for a given period of time, a no-passage decision can be made based on the obstacle. If an obstacle exists for a pre-selected amount of time or more within the 3-second window, subsequent 3-second windows in the 20-second decision timeframe can be evaluated and weighted based on the obstacle assessment results of the previous windows. If necessary, this step-by-step decision-making process can continue until the entire 20-second timeframe is evaluated. An obstacle may be detected at the beginning of the sliding window even if the previous window may have resulted in a high persistence value.

[0087] Refer again Figure 5C If the persistence of 9567 is less than or equal to a preselected value, such as, but not limited to, 0.2, then method 9550 may include setting the speed of AV 9571 to the maximum speed of the condition. Otherwise, method 9550 may include setting the speed of AV 9569 to 0.0 m / s.

[0088] Now for reference Figure 6A and 6B DOP 8471 ( Figure 8 The system provides dynamic obstacles within the lane of interest (LOI) 9209 belonging to the AV. Static obstacles 9219 can also be provided. Criteria for autonomously navigating the AV through intersections 9201 when obstacles are present can include whether the AV occupies a vehicular or pedestrian lane. When the AV is acting as a vehicular vehicle (e.g., in...) Figure 6A When crossing a road, obstacles 9211 / 9213 behind AV 8009 in the same lane 9209 can be ignored. Whether an obstacle is considered to be behind the AV is based on whether the obstacle is located within the pre-selected driving area. In some configurations, when AV 8009 is acting as a pedestrian (such as in...), Figure 6B When crossing the road, pedestrian 9215 is considered to share the road with AV and therefore is not considered in the persistence calculation. There is a waiting time, such as, but not limited to, 5 seconds, before entering the intersection to ensure that the received obstacle is caused by a stable sensor reading. Obstacles can be received during this 5-second waiting time.

[0089] Now for reference Figure 7 The system 9700 for autonomous vehicle navigation according to this teaching may include (but is not limited to) autonomy management 9701, autonomous intersection navigation 9703, and optionally, remote intersection navigation 9705. Autonomy management 9701 can process incoming data from, for example, but not limited to, sensors, historical geographic information, and route information, and create commands to safely move the AV on roads and sidewalks. Autonomy management 9701 can invoke various subsystems for assistance, including autonomous intersection navigation 9703, which can resolve situations when the AV encounters an intersection. When the AV can autonomously navigate an intersection, autonomous intersection navigation 9703 can, for example, but not limited to, determine the AV's speed in the intersection situation, which obstacles to avoid and ignore, and whether to request assistance from a remote processor. When assistance is requested or when remote intersection navigation 9705 determines that the operation of the AV may be affected, such as when sensors cannot correctly distinguish the AV's surroundings, remote intersection navigation 9705 can safely navigate the AV. Autonomous intersection navigation 9703 can manage situations where the AV is on road 9711, possibly in traffic, or navigating on pedestrian crossing 9713, potentially encountering pedestrians, cyclists, etc. Furthermore, autonomous intersection navigation 9703 can manage situations where the AV is navigating through a marked intersection 9709 or a signalized intersection 9707. Persistent processing 9715 can persistently manage the presence of obstacles at the intersection by evaluating obstacles over a time period, and impact processing 9717 can manage the presence of obstacles at the intersection from one time period to the next. In some configurations, the presence of obstacles can be evaluated at both marked and signalized intersections. In some configurations, when a vehicle occupies an intersection (marked or signalized), the AV can follow a first set of policies, and when a pedestrian occupies an intersection (marked or signalized), the AV can follow a second set of policies.

[0090] Continue to refer to Figure 7Autonomous navigation can be enhanced with remote assistance. Exemplary scenarios are shown below. Other scenarios are also possible. In some configurations, these exemplary scenarios can be handled entirely autonomously. In some configurations, complex intersection structures, such as, but not limited to, left-turn intersections, can be pre-established as intersections managed by the remote intersection navigation 9705. When encountering a complex intersection, the AV can stop at the stop line and request remote control assistance. The remote intersection navigation 9705 can safely control and drive the AV through the intersection, then return to the autonomous intersection navigation 9703 after crossing the intersection. When the AV stops at an intersection to wait for obstacles to be cleared, for example, if there are many targets, the AV can request the remote intersection navigation 9705 to check in. When the remote intersection navigation 9705 deems it safe to cross the intersection, it can return control to the autonomous intersection navigation 9703. If, for any reason, the remote intersection navigation 9705 detects that the AV is making an incorrect decision based on an AV perception problem, a stop request can be issued from the remote intersection navigation 9705. The remote intersection navigation 9705 can then stop the AV at the stop line. The AV can wait to return to autonomous control until the remote intersection navigation 9705 determines (e.g., but not limited to) that the perception prediction (e.g., for traffic lights, signs) and / or the decision made by the AV is correct. If the AV has crossed the minimum perception distance line, for example, if the AV is in the middle of the intersection and the remote intersection navigation 9705 issues a stop request, the remote intersection navigation 9705 can control the AV to pass through the rest of the intersection and can return the AV to autonomous control upon completion of the intersection crossing. At intersections where the AV has difficulty making a decision, the AV can wait at the stop line and request assistance from the remote intersection navigation 9705 to determine whether it is safe to cross the intersection. Examples of such intersections are railway crossings and pedestrian-to-road transitions, where parked cars may block the transition area. These intersections can be determined in advance or dynamically. Autonomous control can be resumed after the remote intersection navigation system 9705 determines that it is safe to cross the intersection.

[0091] Now for reference Figure 8This illustrates an exemplary implementation of the autonomous navigation system of this teaching. System 800 provides an AV architecture that enables an AV to safely traverse intersections. In some configurations, a manager layer 801 of system 800 may provide an interface between a manager 803 and the rest of a communication network that couples the components of system 800 to each other. Manager layer 801 may receive a map of the area the AV is traversing, traffic light locations and statuses, lane information from provider layer 808, the AV's initial maximum speed, and a responsible manager from AD 805 (selected from, for example, but not limited to, road manager 8453, pedestrian manager 8450, remote control manager 8421, and managerless 8445), and a filtered list of dynamic obstacles from dynamic obstacle provider 8471.

[0092] Continue to refer to Figure 8 Regarding signals, the manager 803 expects to receive traffic light status (red / green / unknown / yellow) and other data, such as illuminated arrows, timers, and other possible lighting options. Based on, for example, if the distance from AV to stop line 8011 is greater than, less than, or equal to the braking distance, the manager 803 ( Figure 8 The maximum speed reduction can be published. Manager 803 can operate in the various states discussed herein. The manager FSM 804 state is indicated by AD 805, and the intersection state 8461 is indicated by the stop line module (SLM) 807 discussed herein.

[0093] Continue to refer to Figure 8In some configurations, manager 803 may include, for example (but not limited to), road manager 8453, pedestrian manager 8450, remote control manager 8421, and no manager 8445. In some configurations, AD 805 provides manager type 803. Each manager 803 corresponds to manager FSM state 804. For example, road manager 8453 corresponds to road state 8455, pedestrian manager 8450 corresponds to pedestrian state 8449, and remote control manager 8421 and no manager 8445 correspond to no state 8447. The autonomy controller (AD) 805 selects an activity manager based at least on the lane type the AV is traveling in and sets the maximum speed of the selected activity manager. The initial maximum speed is based at least on the lane the AV is traveling in, intersections near the AV, and possibly other factors. Lane categories may include (but are not limited to) unknown, pedestrian, bicycle, road, parking lot, and parking space. For example, in some configurations, pedestrian lanes have a pedestrian lane category, road lanes belong to the road lane category, roads within a parking lot have a parking lot category, and parking spaces within a parking lot have a parking space lane category. For safety reasons, lane categories can determine the upper limit of AV speed in each different lane category. For example, in some configurations, the AV's speed should not exceed the speed of a person walking on a sidewalk, so the AV's maximum speed on a sidewalk may be lower than its maximum speed on a highway. An intersection can include the junction of multiple roads. The state machine can modify the initial maximum speed based on the current context, which includes, for example, but not limited to, the presence and state of traffic lights, and the presence and type of traffic signs. Regardless of which manager is in control, the maximum speed can be provided to other parts of the system controlling the AV's speed. The primary function of manager 803 when approaching and crossing an intersection is to provide MPC 812 with the AV's maximum speed at each point in the process. To stop the AV, manager 803 can be arranged to send a speed of 0.0 m / s to MPC 812. As a result, the AV will not continue forward until manager 803 determines that it can navigate autonomously or until control of the AV is remotely taken over. To continue forward, manager 803 can be arranged to send a non-zero speed to MPC 812. As a result, the AV continues forward, for example, crossing the intersection.

[0094] Continue to refer to Figure 8The pedestrian crossing manager 8450 may include processing for traveling with pedestrians on a pedestrian crossing, and the road manager 8453 may include processing for traveling with vehicles on a road. When the AV is traveling on a road, the AD 805 sets an initial maximum road speed, and when the AV is traveling on a pedestrian crossing, the AD 805 sets an initial maximum pedestrian crossing speed. In some configurations, the initial maximum road speed may include (but is not limited to) 3.1 m / s. The initial maximum pedestrian crossing speed may include (but is not limited to) 1.5 m / s. Although most features of intersection processing are shared between road and pedestrian crossing management, in some configurations, if there is a pedestrian on the pedestrian crossing in front of the AV, the AV may, if necessary, navigate around the pedestrian and continue navigating the intersection. However, if there are vehicles on the road, the AV may stop at the stop line until the intersection is clear. In some configurations, when the AV is traveling on a road, obstacles within the AV's perception range but behind the AV are not considered when the AV decides whether to cross the intersection. Obstacles are considered to be behind the AV when they occupy lanes of interest that the AV has already traversed.

[0095] Continue to refer to Figure 8 The stop line module (SLM) 807 provides stop lines 8011 to the manager layer 801. Figure 1A The SLM 807 can determine the location of stop line 8011 in real time. Figure 1A The SLM 807 can determine the location of AV 8009. Figure 1B The surrounding perception range is 8005 ( Figure 1A The sensing range is limited to AV 8009 ( Figure 1A You might see traffic light 8007. Figure 1A The area is 8012 (intersection entry point). Figure 1A This can be provided by navigation map 806. SLM 807 can be based on intersection entrance 8012 ( Figure 1A ) Calculate the optimal stop line 8011 ( Figure 1A The SLM 807 can calculate the sensing range of 8005 as follows: Figure 1A radius of )

[0096] Radius = (Traffic Light 8007) Figure 1A (height of ) - height of the associated camera on AV) / (tan(field of view of the associated camera / 2)

[0097] Both heights are measured in meters, and the camera's field of view is measured in radians. Manager layer 801 needs to reach stop line 8011. Figure 1A The distance is a precise and smooth / predictable value so that deceleration can be commanded when needed.

[0098] Continue to refer to Figure 8 Provide SLM 807 with a pre-created map and left boundary 8018 ( Figure 3 ) and right boundary 8020 ( Figure 3 SLM 807 can calculate dF 8004 (). Figure 3 (AV 8009) Figure 3 (shortest distance between ) and the front boundary), dL 8002 ( Figure 3 (AV 8009) Figure 3 ) and left lane boundary 8018 ( Figure 3 The distance between ) and dR 8024 ( Figure 3 (AV 8009) Figure 3 ) and right lane boundary 8020 ( Figure 3 The distance between them and dB 8026 Figure 3 (AV 8009) Figure 3 ) and bottom boundary 8006 ( Figure 3 The distance between )). When AV 8009 ( Figure 3 When the SLM 807 reaches the pre-selected distance from the intersection, it can begin sending information about the stop line 8011 to the manager layer 801. Figure 3 Information from AV 8009. The pre-selected distance is based at least on information related to AV 8009. Figure 3 The maximum field of view of the associated sensor and the types of traffic control available at the intersection. For example, traffic control types may include (but are not limited to) at least one traffic light, at least one traffic sign (e.g., a stop sign or yield sign), and virtual right-of-way indication. The SLM 807 can be connected to the AV 8009 ( Figure 3 The pre-created information associated with the current location of the traffic control allows access to the current type of traffic control. In some configurations, when the traffic control type is a traffic light, the pre-selected distance can include approximately 50m. In some configurations, when the traffic control type is a sign or virtual right-of-way, the pre-selected distance can include approximately 15m. When the SLM 807 provides information about stop line 8011 (… Figure 3 Other information provided to the manager layer 801 when providing information may include (but is not limited to) the classification of traffic signals (if the traffic control type is signal), the classification of traffic signs (if the traffic control type is sign), the identification of road 8024, whether the AV is not remotely controlled, and whether mandatory check-in is required. Mandatory check-in is required if it is predetermined that the remote control processor (e.g., but not limited to an operator) should verify whether the AV should enter the intersection. When SLM 807 begins sending stop line 8011 according to the guidelines set forth herein... Figure 3During position updates, the SLM 807 continues to send stop line position updates, route updates, and AV 8009 updates at preselected intervals, such as, but not limited to, 10Hz. Figure 3 The location of ). If you are remotely controlling AV 8009 ( Figure 3 If the SLM 807 does not send an update, then the SLM 807 will not send an update.

[0099] Continue to refer to Figure 8 Instance messages from SLM 807 to Manager Layer 801 may include a signal classification field indicating whether the intersection status is marked (limiters 3-6) or signaled (limiters 1-2). Stop line information may also include a timestamp, stop point, road sign, and yaw within the perception range to move the AV toward the road / pedestrian traffic signal. Stop line messages may also include a direction to intersection status 8417 to request remote control assistance at the stop line (isRemoteControl = True), where the remote control processor crosses the intersection and returns control to the AV after completing the intersection crossing. Stop line messages may also include a direction to the AV to check in at the stop line via the remote control processor (forcedCheckin = True), where the remote control processor may send an initiation request if the intersection is clear and the AV has the right of way, such as at a railway crossing and the merging of a pedestrian and bicycle path. SLM 807 may infer the information it provides to Manager Layer 801 from a map associated with the driving geography. An example stop line message is as follows:

[0100] secs:1587432405 (timestamp in seconds)

[0101] nsecs:787419316 (timestamp, in nanoseconds)

[0102] signalClass:1

[0103] wayID:"110"

[0104] isRemoteControl: Fake

[0105] virtualStopPointXY:

[0106] x:-4.59256887436

[0107] y:-6.44708490372

[0108] yawAtPerceptionRangeMin_rad:2.24184179306

[0109] forcedCheckIn: True

[0110] signalYaw_rad:1.24184179306

[0111] perceptionRangeMinSeq:108

[0112] The timestamp can be used for comparison with the arrival time of the current stop line. If a new stop line is available, the road ID indicates the road where the stop line is located, and the sequence number indicates the route point closest to the stop line. The virtual stop point indicates the x, y position of the point located on stop line 8011. The manager uses this information to determine whether the stop line has been reached or crossed. If the orientation of the traffic signal (signalYaw) is within a pre-selected threshold, such as, but not limited to, ±20° of the AV orientation, it is assumed that the AV can observe the front of the traffic signal. If the orientation of the traffic signal and the current orientation of the AV differ from the pre-selected threshold, the AV can reorient itself based on the difference. If the AV is navigating on a pedestrian crossing, the AV can wait for a pre-selected amount of time, depending on, for example, a known sensor stabilization delay used to observe the state of the traffic lights, such as 1 second.

[0113] Continue to refer to Figure 8 One scenario where an AV (Action Viewer) can be reoriented to face a traffic signal is if the navigation path includes a curb ramp between a pedestrian crossing and a crosswalk. In this scenario, the curb ramp may not always face the pedestrian traffic light. For example, the AV can be reoriented at a stop line so that it faces the pedestrian traffic signal so that it can observe the pedestrian crossing. The reorientation angle allows the AV to achieve optimal orientation so that the AV sensor can obtain a preselected field of view, such as, but not limited to, 40°. After reorientation, the AV can wait for a preselected amount of time, depending on, for example, a known sensor settling delay used to observe the signal's state, such as 1 second.

[0114] Now for reference Figure 9Furthermore, one implementation for determining the location of a traffic management feature relative to map data may include using information from different types of images, including oblique images. Using various types of images can improve the accuracy of locating traffic management features. The method 150 of this teaching for identifying the three-dimensional location of at least one traffic management feature 107 from at least one oblique image 111 may include (but is not limited to) determining the latitude / longitude of a corner 117 of an intersection 101 from at least one aerial image 115. Method 150 may include determining an intersection area of ​​interest 103 (with a corner 105) from at least one oblique image 111 based on latitude / longitude, and determining an estimated height of the traffic management feature 107. The estimated height may be based on the average height of the feature type, empirical data associated with the area of ​​interest, data from a general area containing the area of ​​interest, or any other suitable estimation method. Method 150 may include generating a traffic management feature bounding box pixel 107 for the traffic management feature 108 within the intersection area of ​​interest 103 based on machine learning 116. Method 150 may include calculating the coordinates of traffic management feature 108, including elevation 121, based on homography transformation 113, based on the estimated elevation and traffic management feature bounding box pixels 107. For example, a subdivision model may optionally be used to determine the latitude and longitude of corner 117 of intersection 103. Identified features 109 shorter than the estimated elevation may optionally be discarded. Traffic management feature 108 may include (but is not limited to) traffic signs and signals, and pedestrian signs and signals.

[0115] Now for reference Figure 10 According to the annotation process specific to the type of traffic management feature 108, each tilted image 111 associated with the intersection can be annotated. Figure 9 Annotate ) In some configurations, oblique images can be collected from an oblique camera system that is looking down at the Earth at a certain angle.111( Figure 9 In some configurations, the angle can include 45°. Each tilted image 111 ( Figure 9 The image can include important objects that can be annotated with bounding boxes 107 and optionally text labels. The bounding box can include top-left coordinates (x1, y1) 1101 and bottom-right coordinates (x2, y2) 1103. In some configurations, the image can include a .png image, and the annotation output can be stored in ".json" / ".csv" format. For example, each image can be stored in a JSON file with a key ("label") followed by a value as the object category ("Traffic Light") and bounding box information ("type", "index", and "points") for the corresponding box describing the traffic management feature 108.

[0116] [{"tags":{"Labels":"Traffic_Light"},"type":"rect","index":1,"points":[[x1,y1],[x2,y2]]},

[0117] {"tags":{"Labels":"Traffic_Light"},"type":"rect","index":2,"points":[[x1,y1],[x2,y2]]}]

[0118] Unknown or unclear images that cannot be accurately annotated can be discarded.

[0119] Continue to refer to Figure 10 Traffic management feature 108 may include any signaling device located at a road intersection that can control the flow of traffic for vehicles and pedestrians. Traffic signs and signals can control traffic flow, and pedestrian signs and signals can control pedestrian flow. Traffic signals may include (but are not limited to) traffic lights suspended on power lines and traffic lights suspended on poles. In some configurations, features that appear to be traffic lights suspended on power lines may be annotated even if the front of the traffic light is not visible, unless they are obscured by objects other than the traffic light. In some configurations, features that appear to be traffic lights suspended on poles (vertical and horizontal poles) may be annotated if the front of the feature is visible and if they are not obscured by objects other than the traffic light. If the front of the feature is not visible, the traffic light pole may obscure part of the traffic light. In some configurations, features that appear to be traffic lights but are obscured by more than 50% of objects other than those that appear to be traffic lights are not annotated as traffic lights. In some configurations, if there are multiple features that appear to be traffic lights that overlap with each other, the image may be annotated with multiple bounding boxes. In some configurations, if a feature that appears to be a traffic light appears at the edge of an image, then the feature can be labeled as such if 80% or more of the object is within the image.

[0120] Now for reference Figures 11 to 18 Regardless of whether the traffic is vehicles or pedestrians, tilted images can be automatically annotated based on the type of traffic management features located within them. The process for optimally annotating tilted images can include the factors listed in this paper. This paper discusses specific types of traffic management features, but these automatic annotation techniques can be applied to traffic management features of any size, shape, and type.

[0121] Now for reference Figure 11 and Figure 12 The original tilted image 111 may include intersection 103 and traffic management features 108. Intersection 103 may include intersection path 125. Figure 12 ) and 127 ( Figure 12 The intersection path may include annotated traffic management features 107. Figure 12 In these examples, traffic management feature 107 ( Figure 12 This includes traffic lights.

[0122] Now for reference Figure 13 and Figure 14 The tilted images 131 / 133 may include pedestrian signs / signals. Pedestrian signs / signals may include any signaling device located at road intersections or crosswalks 135 / 137 to allow pedestrians, bicycles, and / or autonomous vehicles to cross the road. In some configurations, pedestrian signs / signals 108 may be obscured by more than 50% of objects other than pedestrians / traffic signs / signals. Figure 14 ) can be annotated as pedestrian signs / signals. In some configurations, if there are multiple pedestrian signs / signals 108 that overlap somewhat with each other... Figure 14 ), then multiple bounding boxes 107 ( Figure 14 (Note: The image is tilted, as shown in image 131 / 133.) In some configurations, if 80% or more of the pedestrian signs / signals 108 ( Figure 14 Within the tilted image 131 / 133, pedestrian signs / signals 108 ( Figure 14 This can be annotated as follows.

[0123] Now for reference Figure 15 and Figure 16 The tilted images 141 / 143 may include traffic management features 108. Figure 16 The traffic management feature may include traffic signs. Traffic management feature 108 ( Figure 16 This can include signs of any shape that can manage traffic at, for example, but not limited to, intersections 145 / 147. In some configurations, traffic management feature 108 ( Figure 16 () may include an octagonal sign, optionally including the word "Stop". Traffic management features 108 may not be obscured by more than 20%. Figure 16 The annotation is for traffic signs. In some configurations, if 80% or more of the traffic management features are 108 ( Figure 16 Within the tilted images 141 / 143, traffic signs can be annotated as follows. In some configurations, traffic sign 149 ( Figure 16 (This might not be noted because the traffic sign is not facing forward in the tilted image 141.)

[0124] Now for reference Figure 17 and Figure 18 The tilted images 201 / 203 include traffic management features 108 ( Figure 18 Traffic management characteristics 108 ( Figure 16This can include signs of any shape that can manage traffic flow at, for example, but not limited to, yield zones 207 / 205. In some configurations, traffic management feature 108 ( Figure 18 () may include an inverted triangle sign, optionally including the word "yield". Traffic management features 108 (which may not be obscured by more than 10%) may not be obscured. Figure 18 The annotation is for traffic signs. In some configurations, if 90% or more of the traffic management features are 108 ( Figure 18 Within the tilted image 201 / 203, traffic management feature 108 ( Figure 18 This can be annotated as follows.

[0125] Now for reference Figure 19 The method 150 of this teaching for identifying traffic management features may include (but is not limited to) determining 151 the latitude / longitude of a corner of an aerial image region of interest from at least one aerial image; determining 153 the oblique image region of interest from at least one oblique image based on the latitude / longitude; determining 155 the estimated altitude of the traffic management feature; generating 157 traffic management feature bounding box pixels for the traffic management feature within the oblique image region of interest based on a machine learning process; and calculating 159 the coordinates of the traffic management feature based on a homography transformation based on the estimated altitude and the traffic management feature bounding box pixels. Homography can be used to insert a model of a 3D object into an image or video such that the object is rendered with the correct perspective and displayed as part of the original scene. A homography matrix is ​​a matrix that maps a given set of points in one image to a corresponding set of points in another image. A homography matrix is ​​a 3x3 matrix that maps each point in a first image to a corresponding point in a second image. Homography can perform transformations on images taken from different perspectives. For example, a homography matrix can be calculated between two images of the same location taken from different angles. A homography transformation matrix can be applied to transform pixels from one image to have the same viewpoint as pixels from another image. Typically, homography between images is estimated by finding feature correspondences within the images.

[0126] Now for reference Figure 20The system 300 of this teaching for identifying traffic management features may include (but is not limited to) an aerial image processor 303 that determines the latitude / longitude 317 of a corner of an aerial image region of interest from at least one aerial image 309; and an oblique image processor 305 that determines an oblique image region of interest from at least one oblique image 311 based on the latitude / longitude 317. The system 300 may include an estimated altitude processor 307 that determines an estimated altitude 325 of a traffic management feature 108; and a traffic management feature processor 327 that generates a traffic management feature bounding box pixel 333 for the traffic management feature 108 within the oblique image region of interest 321 by a machine learning process 331. The traffic management feature processor 327 may calculate the coordinates 345 of the traffic management feature 108 based on a homography transformation 347. The homography processor 341 may provide the homography transformation 347 based at least on the estimated altitude 325 and the traffic management bounding box pixel 333. The traffic management feature processor 327 can provide the map processor 343 with the 3D coordinates of traffic management features.

[0127] Now for reference Figure 21 The state transition diagram depicts the transitions and intersection finite state machine (FSM) states within each manager. The manager FSM states are initialized to stateless 8447, and in AD 805 ( Figure 8 The conversion occurs when the AD 805 is converted to a running state and a non-manager (Road Manager 8453 or Sidewalk Manager 8450) is released. Figure 8 ) Switch to remote control mode 8425 ( Figure 8 If so, the Activity Manager will be published as a remote control manager 8421. Figure 8 The remote control manager causes the manager FSM to become stateless 8447. When control returns from remote to local, i.e., when traversing the remote control portion of the route, AD 805 ( Figure 8 Returning to the running state and issuing a non-none (non-remote control) manager will trigger the manager FSM to switch to road state 8455 or sidewalk state 8449.

[0128] Continue to refer to Figure 21 Intersection FSM 8461 ( Figure 8 Initialized to the initial state of no-intersection state 8415, by SLM 807 ( Figure 8 ) Stop line 8011 ( Figure 1A If a transition to an intersection state is required afterward, the starting state is a placeholder. The no-intersection state 8415 maintains the same speed set by the road state 8455 or the pedestrian crossing state 8449. If stop line 8011 ( Figure 1A ) by SLM 807 ( Figure 8 If a crossroads FSM is published, the crossroads FSM state is transformed into a sub / derived state of crossroads state 8417 based on the manager context and decision-making process within crossroads state 8417 as described herein. In some configurations, crossroads FSM 8461( Figure 8 ) from SLM 807 ( Figure 8 Received new stop line 8011 ( Figure 1A When the intersection transitions from state 8415 (no intersection) to state 8417 (intersection), and crosses stop line 8011 (… Figure 1A Then return to the no-intersection state 8415. For each new stop line 8011 ( Figure 1A The system transitions from a no-intersection state 8415 to a crossroads state 8417. For each intersection of stop line 8011, the system transitions from a crossroads state 8417 to a no-crossroads state 8415. This is only necessary when AD 805 ( Figure 8 By assigning the Activity Manager as either Remote Control Manager 8421 or Managerless 8445, System 801 can remain stateless 8447. In stateless 8447, AD 805 ( Figure 8 The maximum speed can be set to a preset value. In some configurations, the preset value can be 0.0 m / s. If AD 805 ( Figure 8 If the activity manager is assigned as the road manager 8415, then system 801 can enter road state 8455 and remain in road state 8455 until AD 805. Figure 8 Assign the Activity Manager as either Remote Control Manager 8421 or No Manager 8445, thereby enabling the Manager Layer 801 ( Figure 8 Return to stateless 8447. If AD 805( Figure 8 If the activity manager is assigned as the pedestrian walkway manager 8450, then the manager layer 801 can enter the pedestrian walkway state 8449 and remain in the pedestrian walkway state 8449 until AD 805. Figure 8 Assign the Activity Manager as either Remote Control Manager 8421 or No Manager 8445, thereby enabling the Manager Layer 801 ( Figure 8 Return to state 8447 (no state) or road state 8455. If intersection FSM 8461 ( Figure 8 If the intersection is in state 8415 (no intersection), then SLM 807 indicates that 8441AV has reached the new stop line 8011. Figure 1A This can trigger FSM 8461 at the intersection. Figure 8 ) Entering intersection state 8417. In some configurations, intersection FSM 8461 ( Figure 8) can be used in the no-intersection state 8415 (when crossing and addressing stop line 8011) Figure 1A When AV is not in an intersection, or when the intersection state 8417 is in a sub-state (when a new stop line 8011 is received). Figure 1A Cycle between () and () times. When AV reaches the stop line 8011 ( Figure 1A When ), it is considered to have crossed the stop line 8011. Figure 1A The system then determines whether to enter the intersection. The decision is based at least on the location information of the AV and information about stop line 8011. Figure 1A The information is as follows: The intersection state 8417 is a transient state, meaning that the manager FSM804 can never be in this state, but can only be in one of the sub-states (8425, 8432, 8431) or the no-intersection state 8415. In some configurations, when addressing each new stop line 8011 (… Figure 1A Before transitioning to the no-intersection state 8415, the addressing of the previous stop line 8011 can be marked. Figure 1A (The end of )

[0129] Now for reference Figure 22 Crossroads state 8417 is a transient state, and in conjunction with crossroads FSM 8461, the transient state determines which state to transition to based at least on the manager context. Components of the manager context that help determine the next crossroads sub-state may include (1) if AD 805 ( Figure 8 (1) In remote control state 8425, and (2) if the signal category of the current stop line has a specific value. If, for any reason, the AV is remotely controlled while it is in intersection state 8417, then enter remote control intersection state 8425. If the AV is not remotely controlled, the signal category value and the activity manager are used to determine which sub-state to switch to: pedestrian signal intersection state 8428, road signal intersection state 8430, or marked intersection state 8431. If, for any reason, the route has changed, or if the AV has reached stop line 8011 ( Figure 1A If (isIntersection = false 8411), or a hard exit has been encountered (8413), the sub-intersection state is switched. In these cases, AV switches to the no-intersection state (8415). When the stop line (8011) is reached... Figure 1A When AV crosses stop line 8011, a decision can be made regarding the intersection. Some possible decisions include invoking remote control, proceeding through the intersection, and initiating a request from the remote control. Based on this decision, when AV crosses stop line 8011... Figure 1A ) indicates a hard exit 8413.

[0130] Continue to refer to Figure 22A remote control parking request can be sent from the remote control processor to the AV to help the manager make decisions based on the context. If the remote control processor determines that the AV is making an incorrect decision, the remote control processor can intervene by sending a parking request that causes the AV to stop at stop line 8011. Figure 1A The AV can stop at stop line 8011 when this series of events occurs. Figure 1A The remote control processor waits for remote control action at the intersection. In some configurations, if, for example, at an unsignaled intersection, the intersection is clear, or if at a signalized intersection, the traffic light is green, the remote control processor can send a start request. While the AV is being remotely controlled, the remote control processor can determine whether the intersection is signaled or marked. If the intersection is signaled, the remote control processor can determine the signal status, such as whether the traffic light is green and whether the intersection is clear. If the light is green and the intersection is clear, the remote control processor can drive the AV through the intersection, or it can return control to the AV that might automatically cross the intersection. If the intersection is marked, the remote control processor can determine whether the right-of-way is clear. If the right-of-way is clear, the remote control processor can drive the AV through the intersection. In some configurations, the remote control processor can maintain control throughout the navigation of the intersection. Other configurations are anticipated. In some configurations, if the AV is simply waiting for a start request from the remote control processor, autonomous control can be returned to the AV. If the AV has crossed the minimum perception range line, and the remote control processor sends a stop request, the AV may request to relinquish control of the remote control processor. If the remote control processor sends a stop request to the AV while the AV is in the middle of the intersection, the AV may request to relinquish control of the remote control processor until navigation of the intersection is complete.

[0131] Continue to refer to Figure 22 AD 805 ( Figure 8 It can be based on providing the 808 layer ( Figure 8 The lane classification provided is used to assign the manager as the activity manager 8401. In some configurations, this information can be stored in a pre-created map or navigation map 806. Figure 8 In ), if control is taken due to, for example but not limited to, lane classification or a remote control processor, AD 805 ( Figure 8 If the remote control manager 8421 is assigned as the activity manager 8401, then system 8461 ( Figure 8 ) Enters remote-controlled intersection state 8425, where the remote control device guides AV through the intersection. If AD 805 ( Figure 8The activity manager 8401 is assigned as either the road manager 8453 or the pedestrian crossing manager 8450, and if there is a traffic signal (SLM 807) at the intersection... Figure 8 If traffic signal categories 1 and 2 (8429 / 8427) are provided, then system 8461 ( Figure 8 The intersection enters a signalized state 8432, i.e., road traffic light state 8430 or pedestrian traffic light state 8428. If the intersection has traffic signs (SLM 807... Figure 8 If traffic sign categories 3-6 (8433) are provided, then system 8461 ( Figure 8 Entering a marked intersection state 8431. For each pre-selected time period, a check is performed to determine if system changes are necessary 8461. Figure 8 ) and manager FSM 804 ( Figure 8 Additionally, there is a check for hard exits. If a hard exit occurs, the system can switch from the no-intersection state 8415. The no-intersection state 8415 handles situations where the AV has not encountered an intersection. In this case, the manager layer 801 ( Figure 8 ) for driver's level 813 ( Figure 8 Provided by AD 805 ( Figure 8 The maximum speed is set without modification. When the AV is at its maximum sensing range, the manager layer 801 receives a stop line message from the SLM 807. The intersection state 8417 indicates that the AV has encountered an intersection, which is equivalent to receiving a stop line message from the SLM 807. Figure 8 ) Receives a stop line message. In this case, the manager layer 801 ( Figure 8 Analysis of SLM 807 ( Figure 8 The message is sent to select the intersection status type. The intersection status type may include (but is not limited to) remote control status 8425, signaled road status 8430, signaled pedestrian status 8428, and marked status 8431. Each intersection status may be further quantified as, for example, but not limited to (0) unknown, (1) traffic light, (2) pedestrian light, (3) right-of-way, (4) stop, (5) coasting stop, and (6) yield. Under remote control intersection status 8425, limiters (0) through (6) apply, and remote control is active at the intersection. Under signaled road / pedestrian intersection status 8430 / 8428, limiters (1) and (2) apply, and the upcoming intersection includes at least one traffic light. Under marked intersection status 8431, limiters (3) through (6) apply, and the upcoming intersection includes at least one traffic sign, which may be supplied to SLM 807 ( Figure 8Then, it is supplied to the manager layer 801 via SLM messages. Figure 8 Virtual markers are introduced into the predefined map.

[0132] Now for reference Figure 23 In one embodiment of this teaching, the system may include a data repository 802 that may retain information generated and used by state machines in the system, such as manager state machine 804 and intersection FSM 8461. In this embodiment, data repository 802 may store data generated by state machines published by manager layer 801 to, for example, but not limited to, ROS networks. Other networks and structures that are expected to move data from one system facility to another are also considered. In some configurations, data repository 802 may be omitted entirely or may store any type of subset of data passed between manager layer 801 and the state machines. In the illustrated embodiment, the published data may include data calculated by the system of this teaching and provided to MPC 812 (…). Figure 8 The maximum speed of the AV. As described herein, for example, the speed of the AV may vary as the AV approaches an intersection. The published data may also include stop line information, such as when the AV has reached the stop line. Other published data may include turn signals and the preferred side of the AV's travel. The preferred side relates to, for example, the size of the area around an obstacle.

[0133] Continue to refer to Figure 23 The data that can be received by the manager layer 801 may include (but is not limited to) the maximum speed based on the type of road surface the AV is navigating (e.g., but not limited to, a road or pedestrian walkway). The manager layer 801 may also receive information about dynamic obstacles traveling across the intersection and the status of traffic lights associated with the intersection. This data may be collected, among other possibilities, by sensors located on the AV, such as, but not limited to, mounted sensors and / or aerial sensors installed near the navigation route. Other data may include internal communication speed, lane of interest, and driver controller state. Internal communication speed is the current speed of the AV. Lane of interest is the lane of interest at the intersection. The driver control state relates to the position of the stop line when the curb or curb ramp is on the navigation route. The driver control state indicates when the system recognizes that it has reached the intersection. The system can only recognize that the AV has reached the intersection after completing a curb crossing. Based on the received data, the manager state machine 804 can determine the context of the manager with control, and the intersection FSM 8461 can switch to a state machine that processes the current intersection type within the context of the controlling manager.

[0134] Now for reference Figure 24Implementations of the systems described herein may include various state machines, as discussed herein. An AV can find itself in any of the described states and may suffer from various listed events while in said state. The states and events listed herein are exemplary and non-limiting. Figure 24 The architecture described herein can be adapted to various other states and events.

[0135] Now for reference Figure 25 An exemplary AV 9800 capable of autonomously navigating intersections may include, but is not limited to, an external interface 9801, a control 9803, and a mobile device 9805. The external interface 9801 can provide the exemplary AV 9800 with its eyes, ears, other senses, and digital communications. The external interface 9801 may include (but is not limited to) long-range and short-range sensors, such as, but not limited to, radar, lidar, cameras, ultrasonic sensors, thermometers, audio sensors, and odor sensors, as well as digital communications such as Bluetooth and satellite. It may also include a microphone and a visual display. The external interface 9801 can provide perception and other data to the control 9803. The control 9803 can process the perception and other data, which can provide real-time and historical information to inform the AV 9800 of possible routes. The control 9803 may include a processor, such as, but not limited to, a perception processor 9807, an autonomy processor 9809, a mobile device 9811, and a remote device 9813. The perception processor 9807 can receive, filter, process, and fuse incoming sensor data that can inform the navigation process. For example, for intersection navigation, this data may include images of traffic lights. The autonomy processor 9809 can process the information required for the AV to drive autonomously and can guide other processors to achieve autonomous driving. The mobile processor 9811 can cause the AV to follow the commands of the autonomy processor 9809, and the remote processor 9813 can manage any remote control required for safe navigation. The processor performing the work of the control 9803 may include executing software, firmware, and hardware instructions. The mobile device 9805 can implement the commands of the mobile processor 9811. The mobile device 9805 may include wheels, movable tracks, or any other form of mobility device.

[0136] Now for reference Figure 26A and 26B An example AV 9800 is shown. Figure 14Two configurations of the AV are shown. The configurations in the figures include similar but not identical components referred to herein by the same reference numerals, as variations in style do not alter the functionality of the AV when traversing intersections. In some configurations, the AV may be configured to transport goods and / or perform other functions involving navigation through intersections. In some configurations, the AV may include a container 20110 that can be remotely opened automatically or manually in response to user input to allow the user to place or remove packages and other items. The container 20110 is mounted on a cargo platform 20160, which is operatively coupled to a power base 20170. The power base 20170 includes four powered wheels 20174 and two casters 20176. The power base 20170 provides speed and direction control to move the container 20110 along the ground above obstacles including discontinuous surface features. The cargo platform 20160 is connected to the power base 20170 via two U-shaped frames 20162. Each U-shaped frame 20162 is rigidly attached to the structure of the cargo platform 20160 and includes two holes that allow for the formation of a rotatable joint 20164 with the end of each arm 20172 on the power base 20170. The power base 20170 controls the rotational position of the arms, thereby controlling the height and attitude of the container 20110. The AV may include one or more processors that can implement the intersection crossing strategy described herein. In some configurations, the AV may navigate, for example, using a different number of wheels or different wheel sets. Wheel selection may be based on the terrain of the lane and the road intersection. In some configurations, when the AV is located at an intersection and on the road, the AV may use a wheel configuration capable of adapting to relatively flat terrain to cross the intersection. This configuration may include rear wheels 20174 ( Figure 26B ) and casters connected to the ground 20176 ( Figure 26B ), while the front wheel 20174A ( Figure 26B The AV is raised from the ground. In some configurations, when the AV is located at an intersection, such as one involving pedestrian walkways or discontinuous surface features including curbs, the AV can use a wheel configuration capable of adapting to challenging terrain to traverse the intersection. This configuration may include rear wheels 20174 ( Figure 26A ) and the front wheel connected to the ground 20174A ( Figure 26A ), and casters 20176 ( Figure 26A It is raised from the ground.

[0137] The configuration described herein relates to a computer system for implementing the methods discussed herein, and a computer-readable medium containing programs for implementing these methods. Raw data and results can be stored for future retrieval and processing, printing, displaying, transfer to another computer, and / or to other locations. Communication links can be wired or wireless, such as using cellular communication systems, military communication systems, and satellite communication systems. Parts of the system can run on a computer with a variable number of CPUs. Other alternative computer platforms can be used.

[0138] The configuration of this invention also relates to software / firmware / hardware for implementing the methods discussed herein, and a computer-readable medium for storing the software for implementing these methods. The various modules described herein may be implemented on the same CPU or on different CPUs. According to regulations, the configuration of this invention has been described in more or less specific language regarding its structural and methodological features. However, it should be understood that the configuration of this invention is not limited to the specific features shown and described, as the apparatus disclosed herein includes preferred forms for implementing the configuration of this invention.

[0139] The method may be implemented electronically, in whole or in part. Signals indicating actions taken by system elements and other disclosed configurations may be transmitted over at least one real-time communication network. Control and data information may be executed electronically and stored on at least one computer-readable medium. The system may be implemented to execute on at least one computer node in at least one real-time communication network. Common forms of at least one computer-readable medium may include, for example, but not limited to, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic media, optical disc read-only memory or any other optical media, punched cards, paper tape, or any other physical media with perforated patterns, random access memory, programmable read-only memory and erasable programmable read-only memory (EPROM), flash EPROM or any other memory chip or cassette tape, or any other medium that a computer can read. Furthermore, at least one computer-readable medium may contain any form of graphics that requires appropriate licensing where necessary, including but not limited to Graphics Exchange Format (GIF), Joint Image Experts Group (JPEG), Portable Web Graphics (PNG), Scalable Vector Graphics (SVG), and Tagged Image File Format (TIFF).

[0140] While the teachings have been described above with respect to specific configurations, it should be understood that they are not limited to these disclosed configurations. Many modifications and other configurations will occur to those skilled in the art to which this invention pertains, and these modifications and other configurations are contemplated to be covered by this disclosure and the appended claims. The scope of the teachings is to be determined by proper interpretation and construction of the appended claims and their legal equivalents, as understood by those skilled in the art based on the disclosure in this specification and the accompanying drawings.

[0141] Claims:

Claims

1. A system for navigating an autonomous vehicle through an intersection, the system comprising a first processor configured for execution on the autonomous vehicle and navigating a class of intersections comprising: navigating the autonomous vehicle in a motor vehicle traffic lane; navigating the autonomous vehicle in a pedestrian walkway; and identifying a traffic management feature from a tilted image, determining a latitude / longitude of a corner of an aerial image region of interest from an aerial image; determining a tilted image region of interest from the tilted image based on the latitude / longitude; determining an estimated height of the traffic management feature; generating, based on a machine learning process, traffic management feature bounding box pixels for the traffic management feature within the tilted image region of interest; and calculating, based on a homography transform, coordinates of the traffic management feature based on the estimated height and the traffic management feature bounding box pixels. the first processor is configured for navigating the autonomous vehicle when an obstacle is present in the class of intersections. the first processor is configured for determining an intersection entry decision for the autonomous vehicle entering the class of intersections based on a distance between the obstacle present in the intersection at a current cycle and the autonomous vehicle, a speed of the obstacle present at the current cycle, and a presence of the obstacle present in the intersection at a previous cycle.

2. The system of claim 1, wherein, the first processor is configured for weighting the intersection entry decision at the current cycle based on at least the intersection entry decision at the previous cycle.

3. The system of claim 2, wherein, the first processor is configured for sending a command to a controller of the autonomous vehicle to stop the autonomous vehicle during a time window while determining a plurality of the intersection entry decisions.

4. The system of claim 3, wherein, the first processor is configured for receiving map data for a travel route of the autonomous vehicle, stop lines specifying a deceleration sequence and a stop position for the autonomous vehicle relative to the intersection, and locations of dynamic and static obstacles.

5. The system of claim 3, wherein, the first processor is configured for navigating the autonomous vehicle based on a travel lane of the autonomous vehicle.

6. The system of claim 1, wherein, 8. The system of claim 7, further comprising a second processor configured for:

7. The system of claim 1, wherein, execution external to the autonomous vehicle; and commanding the autonomous vehicle when the first processor requests a transfer of control to the second processor.

9. The system of claim 1, further comprising: a first finite state machine comprising a first set of states of the autonomous vehicle associated with navigating the autonomous vehicle in a motor vehicle traffic lane; and a second finite state machine comprising a second set of states of the autonomous vehicle associated with navigating the autonomous vehicle in a pedestrian walkway.

10. The system of claim 1, further comprising: a subdivision model configured for determining the latitude / longitude of the aerial region of interest. the first processor is configured for discarding the traffic management feature that is shorter than the estimated height. ​ ​ 11. The system of claim 1, wherein, ​ 12. The system of claim 1, wherein, The traffic management features include traffic lights, traffic signs, pedestrian signals, pedestrian signs, and combinations thereof.

13. The system of claim 1, wherein, The oblique image region of interest includes a traffic intersection.

14. The system of claim 1, wherein, The oblique image region of interest includes a highway merge.

15. A system for navigating an autonomous vehicle through an intersection, comprising: an autonomous processor configured to autonomously navigate the autonomous vehicle through the intersection; a remote control processor configured to navigate the autonomous vehicle through the intersection by remote control; and a protocol between the autonomous processor and the remote control processor configured for establishing a transfer of control between the autonomous processor and the remote control processor, wherein the autonomous processor comprises: a first processor configured for: executing on the autonomous vehicle; and navigating a class of intersections, the class of intersections including: a motor vehicle traffic lane; and a pedestrian walkway, wherein the first processor is configured for: identifying traffic management features from oblique images, including: determining latitudes / longitudes of corners of an aerial image region of interest from aerial images; determining an oblique image region of interest from the oblique images based on the latitudes / longitudes; determining an estimated height of the traffic management features; generating traffic management feature bounding box pixels for the traffic management features within the oblique image region of interest based on a machine learning process; and computing coordinates of the traffic management features based on a homography transformation based on the estimated height and the traffic management feature bounding box pixels.

16. The system of claim 15, wherein, the first processor is configured for navigating the autonomous vehicle when an obstacle is present in the class of intersections.

17. The system of claim 16, wherein, the first processor is configured for determining an intersection entry decision for the autonomous vehicle to enter the class of intersections based on a distance between the obstacle present in the intersection at a current cycle and the autonomous vehicle, a speed of the obstacle present at the current cycle, and a presence of the obstacle present in the intersection at a previous cycle.

18. The system of claim 17, wherein, the first processor is configured for weighting the intersection entry decision at the current cycle based on the intersection entry decision at the previous cycle.

19. The system of claim 17, wherein, the first processor is configured for sending a command to a controller of the autonomous vehicle to stop the autonomous vehicle during a time window while determining a plurality of the intersection entry decisions.

20. The system of claim 15, wherein, the first processor is configured for receiving map data for a travel route of the autonomous vehicle, stop lines specifying a deceleration sequence and a stop position for the autonomous vehicle relative to the intersection, and locations of dynamic and static obstacles.

21. The system of claim 15, wherein, the first processor is configured for navigating the autonomous vehicle based on a travel lane of the autonomous vehicle.

22. The system of claim 15, wherein, the remote control processor comprises a second processor configured for: executing external to the autonomous vehicle; and commanding the autonomous vehicle when the autonomous processor requests a transfer of control to the second processor.

23. The system of claim 15, further comprising: a first finite state machine comprising a first set of states of the autonomous vehicle associated with navigating the autonomous vehicle in a motor vehicle traffic lane; and a second finite state machine comprising a second set of states of the autonomous vehicle associated with navigating the autonomous vehicle in a pedestrian walkway.

24. A method for navigating an autonomous vehicle through an intersection as part of a travel path, comprising: determining an intersection type; commanding the autonomous vehicle based on the intersection type, identifying a traffic management feature from a tilted image, comprising: determining a latitude / longitude of a corner of an aerial image region of interest from an aerial image; determining a tilted image region of interest from the tilted image based on the latitude / longitude; determining an estimated height of the traffic management feature; generating traffic management feature bounding box pixels for the traffic management feature within the tilted image region of interest based on a machine learning process; and calculating coordinates of the traffic management feature based on a homography transform based on the estimated height and the traffic management feature bounding box pixels.

25. The method of claim 24, wherein, the intersection type comprises a signed intersection.

26. The method of claim 25, further comprising: commanding the autonomous vehicle to cross the intersection if the autonomous vehicle has the right of way at the intersection; decelerating the autonomous vehicle from a home point to a stop line if the home point is in the travel path; receiving information about an obstacle in the intersection; waiting a first amount of time; calculating a weighting factor at a first time; modifying the first time based on when the obstacle is not in the intersection; modifying the weighting factor at the modified first time based on the weighting factor at a second time, the second time occurring before the modified first time; calculating an intersection decision threshold based on the weighting factor; and commanding the autonomous vehicle to perform an action with respect to the intersection based on the intersection decision threshold.

27. The method of claim 24, wherein, the intersection type comprises a signaled intersection.

28. The method of claim 24, further comprising: determining the latitude / longitude of the aerial region of interest using a subdivision model.

29. The method of claim 24, further comprising: discarding the traffic management feature if the estimated height is shorter than the estimated height.

30. The method of claim 24, wherein, the traffic management feature comprises a traffic light, a traffic sign, a pedestrian signal, a pedestrian sign, and combinations thereof.

31. The method of claim 24, wherein, the tilted image region of interest comprises a traffic intersection.

32. The method of claim 24, wherein, the tilted image region of interest comprises a highway merge.

Images