Prediction and navigation in congested environments for driving applications

Abstract

This facilitates efficient prediction of congestion behavior in congested areas of the driving environment, and enables safe and precise navigation. [Solution] The exemplary disclosed system includes a vehicle sensing system and a data processing system. The sensing system acquires sensing data associated with the vehicle's driving environment. Based on the sensing data, the data processing system detects the presence of vulnerable road users (VRUs) in the driving environment. The data processing system applies one or more clustering metrics to form clusters of VRUs, each cluster associated with a geometric shape enclosing one or more VRUs, and speed associated with the collective movement of these VRUs. Using the geometric shapes and associated speeds, the data processing system predicts one or more VRU block regions and determines the driving path of the vehicle in the driving environment.

Description

[Technical Field]

[0001] This specification generally relates to autonomous vehicles. More specifically, it relates to the detection, evaluation, prediction, and navigation of driving environments, including congestion. [Background technology]

[0002] Autonomous (fully or partially autonomous) vehicles (AVs) operate by sensing the driving environment using a variety of electromagnetic (e.g., radar and optical) and non-electromagnetic (e.g., sound and humidity) sensors. Some autonomous vehicles chart a driving path within the environment based on the sensed data. The driving path may be determined based on Global Positioning System (GPS) data and roadmap data. GPS and roadmap data can provide information about static aspects of the environment (e.g., buildings, street layout, road closures), while dynamic information (e.g., information about other vehicles, pedestrians, streetlights) is obtained from simultaneously collected sensed data. The accuracy and safety of the driving path, as well as the accuracy and safety of the speed regime selected by the autonomous vehicle, depend on the ability of the driving algorithm to timely and accurate identification of various objects present in the driving environment and to process information about the environment to provide correct instructions to vehicle control and the drivetrain. [Brief explanation of the drawing]

[0003] This disclosure is presented as an example, not an limitation, and may be understood more fully by referring to the following detailed description when considered in relation to the figures.

[0004] [Figure 1] Figure 1 shows exemplary vehicle components that enable efficient prediction of congestion behavior and safe navigation in congested areas of a driving environment, as demonstrated by several implementations of the present disclosure. [Figure 2]Figure 2 shows an exemplary system architecture of a congestion analyzer, which can predict congestion behavior and safely navigate congested areas of the driving environment, based on several implementations of the present disclosure. [Figure 3A] Figure 3A illustrates the identification of individual pedestrians and / or other vulnerable road users (VRUs) as part of predicting congestion behavior and safe navigation in congested areas of a driving environment, as demonstrated by several implementations of this disclosure. [Figure 3B] Figure 3B illustrates the identification of groups of pedestrians and / or other VRUs having similar behavioral patterns as part of predicting congestion behavior and safe navigation in congested areas of a driving environment, by several implementations of the present disclosure. [Figure 3C] Figure 3C illustrates the formation of clusters based on groups of pedestrians and / or other VRUs having similar behavioral patterns, as demonstrated by several implementations of the present disclosure. [Figure 3D] Figure 3D illustrates the identification of individual clusters as part of cluster aggregation for efficient prediction of congestion behavior in congested areas in a driving environment and for safe navigation, as demonstrated by several implementations of this disclosure. [Figure 3E] Figure 3E illustrates the formation of aggregated clusters based on the merging of individual clusters, as demonstrated by several implementations of this disclosure. [Figure 3F] Figure 3F shows the temporal evolution of an exemplary cluster in several implementations of this disclosure. [Figure 3G] Figure 3G shows exemplary timelines of cluster formation and behavior prediction in several implementations of this disclosure. [Figure 4A] Figure 4A schematically shows a portion of the driving environment, including an area where multiple pedestrians and / or other VRUs are present, as in some implementations of the present disclosure. [Figure 4B] Figure 4B schematically shows an example of a relevant area that may be used in some implementations of this disclosure to select a safe operating speed for an autonomous vehicle when multiple pedestrians and / or other VRUs are present. [Figures 5A-5C] Figures 5A–5C show exemplary clusters of non-VRU objects traveling on a highway or street, according to several implementations of the present disclosure. [Figure 6] Figure 6 illustrates exemplary methods, based on several implementations of this disclosure, for predicting congestion behavior in a driving environment and safely navigating congested areas. [Figure 7] Figure 7 shows a block diagram of an exemplary computer device in which a congestion analyzer can be deployed to predict congestion behavior in a driving environment and to safely navigate congested areas, according to several implementations of the present disclosure. [Overview of the Initiative]

[0005] One implementation discloses a system including a vehicle sensing system, the sensing system configured to acquire sensing data associated with the driving environment. The system further includes a vehicle data processing system, the data processing system configured to apply one or more clustering metrics to multiple vulnerable road users (VRUs) in the driving environment to form one or more clusters of VRUs, each of which one or more clusters of VRUs is associated with the geometric shape surrounding one or more VRUs in each cluster of VRUs, and the velocity associated with the collective movement of one or more VRUs in each cluster of VRUs. The data processing system is further configured to use the geometric shape and velocity associated with one or more clusters of VRUs to predict one or more VRU block regions over a time interval, and to determine the driving path of a vehicle in the driving environment over that time interval, taking into account one or more VRU block regions.

[0006] In another implementation, a method is disclosed that includes using a vehicle's sensing system to obtain sensing data associated with the driving environment and using a processing device to detect a plurality of VRUs within the driving environment based on the sensing data. The method further includes applying one or more clustering metrics to the plurality of VRUs within the driving environment to form one or more clusters of VRUs, where each of the one or more clusters of VRUs is associated with a geometric shape that encloses one or more of the VRUs of each cluster of VRUs and a speed associated with the collective behavior of the one or more VRUs of each cluster of VRUs. The method further includes using the geometric shape and speed associated with the one or more clusters of VRUs to predict one or more VRU block regions for a time interval and determining a driving path of a vehicle within the driving environment considering the one or more VRU block regions for that time interval.

[0007] In another implementation, an AV is disclosed that includes a sensing system configured to obtain sensing data associated with the driving environment. The AV further includes a data processing system configured to apply one or more clustering metrics to the plurality of VRUs within the driving environment to form one or more clusters of VRUs, where each of the one or more clusters of VRUs is associated with a geometric shape that encloses one or more of the VRUs of each cluster of VRUs and a speed associated with the collective behavior of the one or more VRUs of each cluster of VRUs. The data processing system is further configured to use the geometric shape and speed associated with the one or more clusters of VRUs to predict one or more VRU block regions for a time interval. The data processing system is further configured to determine a driving path of a vehicle within the driving environment considering the one or more VRU block regions for that time interval. The AV further includes a vehicle control system configured to direct the autonomous vehicle along the determined driving path.

Best Mode for Carrying Out the Invention

[0008] Autonomous vehicles, or vehicles that deploy various advanced driver assistance features, can use multiple sensor modalities to facilitate the detection of objects in the driving environment and predict the future trajectories of such objects. The sensors can include wireless detection and ranging (radar) sensors, optical detection and ranging (lidar) sensors, digital cameras, ultrasonic sensors, position sensors, and the like. Different types of sensors can provide different complementary benefits. For example, radar and lidar emit electromagnetic signals (radio signals or optical signals) that reflect off objects and return information about the distance to the objects (e.g., determined from the time of flight of the signal) and the velocity of the objects (e.g., from the Doppler shift of the frequency of the reflected signal). Radar and lidar can scan the entire 360-degree field of view by using a series of continuous sensing frames. The sensing frames can include a number of reflections that cover the driving environment within a high-density grid of return points. Each return point can be associated with the distance to the corresponding reflecting object and the line-of-sight velocity of the reflecting object (the component of the velocity along the line of sight).

[0009] Lidar has high spatial resolution due to its sub-micron or micron optical wavelengths, which makes it easy to obtain many closely spaced return points from the same object. This enables accurate detection and tracking of objects when they enter the range of the lidar sensor. Radar sensors are inexpensive, require less maintenance than lidar sensors, have a larger operating distance range, and are also more resistant to adverse weather conditions. Cameras (e.g., still cameras or video cameras) can capture a two-dimensional projection of the three-dimensional external space on an image plane (or some other non-planar imaging surface) and can acquire high-resolution images at both close and far distances.

[0010] Various sensors in a vehicle's sensing system (e.g., LiDAR, radar, cameras, and / or other sensors such as sonar) capture a complementary depiction of objects located in the vehicle's environment. The vehicle's sensing system identifies objects based on their appearance, movement, trajectory, and / or other characteristics. For example, LiDAR can accurately map the shape of one or more objects (using multiple return points) and may also determine the distance to those objects and / or their velocity. Cameras can acquire visual images of objects. The sensing system can map the shape and position of various objects in the environment (obtained from LiDAR data) to their visual depictions (obtained from camera data), and can perform several computer vision operations, such as segmenting (clustering) the sensing data between individual objects (clusters), identifying the type / manufacturer / model / etc. of individual objects, and / or similar ones. Predictive and planning systems can track the movement of various objects over multiple cycles (including, but not limited to, position and velocity) and then infer future movements from previously observed movements. This predicted behavior can be used by various vehicle control systems to take these objects into account (e.g., avoid the objects, slow down the vehicle in the presence of the objects, and / or take some other suitable action) and select a driving path.

[0011] As the number of objects in the driving environment increases, the computational load on perception and planning systems increases significantly. Particularly challenging are high-traffic situations on or near roads involving pedestrians (e.g., popular tourist attractions, sporting events, parades, protests, proximity to pedestrian areas, railway or bus terminals, airport pickup / drop-off points, and / or similar). Such situations are referred to herein as “congested environments,” or simply “crowded.” Crowding can include members or participants ranging from a few to tens or hundreds (or more). The presence of congestion presents specific technical and operational challenges for autonomous vehicles. For example, predicting the behavior of all individual participants in real time is extremely difficult and may require significant processing and onboard memory resources. The presence of nearby congestion reduces the acceptable range of driving paths (tracks), including areas where the vehicle cannot or should not move, and limitations on the speed at which the vehicle can move safely. The restrictions imposed on such congestion include, for example, when pedestrians are walking on the road (e.g., during a protest or political rally) or when there is a possibility of such a situation, or when congestion is currently confined to a sidewalk or other non-driveable area but may move toward the road in the near future (e.g., suddenly moving toward a bus or railcar stopped across the road in response to encountering an obstacle on the sidewalk, and / or similar). As a result, the challenge of safely navigating such areas is complicated by the high computational resources required to track and predict the behavior of many congestion participants.

[0012] Aspects and implementations of this disclosure address these and other challenges of modern perceptual, predictive, and planning technologies by systems and technologies for efficient prediction of congestion behavior and safe navigation in congested areas. Despite congestion potentially having a large number of participants, the behavior of congestion in typical driving situations is usually restricted, at least to some extent, and as a result, the number of degrees of freedom of congestion is significantly reduced compared to the same number of independent objects. In particular, congestion (or at least a portion thereof) may have a common purpose (e.g., to reach the entrance of a stadium, to queue to a political rally venue, and / or to remain at a political rally venue). Correspondingly, like the behavior of a fluid, with velocities around some average flow patterns and local variations in the direction of motion, the behavior of congestion may have a dominant general direction and velocity (which may vary in space and time). In some cases, congestion may have a common purpose, e.g., a crowd of travelers, but may also include multiple smaller groups that primarily stay together while moving (even if their shape changes). In some implementations, an object detection system may process sensory data of an autonomous vehicle (e.g., a car, truck, etc.) into its environment to identify the location and type of various objects in the environment and estimate various characteristics of the objects' motion, such as speed and direction. An autonomous vehicle behavior prediction (BP) system may further predict the motion (behavior) of objects over a specific time range, such as a few seconds. Various objects, identified as pedestrians and / or any other vulnerable road users (VRUs), such as cyclists, scooter riders, people in wheelchairs (propelled by muscle power or any motor), animals (livestock and / or wild), horses or other equestrian riders, may be grouped into clusters, where it is assumed that individual members move in a similar manner to other objects in the same cluster. Throughout this disclosure, the term “pedestrian” should be understood to include any VRU, which is not protected by the body of a vehicle and often lacks the ability to move at significant speeds, accelerations, and / or maneuverability (e.g., due to the lack of a high-power engine).

[0013] Initial clusters can be formed using one or more heuristics and / or outputs of one or more machine learning models, including, but not limited to, the distance from a candidate pedestrian to at least one pedestrian already included in the cluster, not exceeding a (first) threshold distance, but projected after a certain time range, from the candidate pedestrian to at least one pedestrian in the cluster (or the centroid of the cluster), and the candidate pedestrian's velocity (including both speed and direction of movement) being within a certain tolerance range from the representative velocity V of the pedestrians in the cluster, or speed and / or similar. Objects identified as belonging to a given individual cluster can be collectively characterized by a preferred boundary shape and a representative velocity. For example, the boundary shape could be a convex hull, e.g., a polyline enclosed around the outer boundary of the individual objects in the cluster. The representative velocity could be the centroid (e.g., mean, or median) velocity of the individual pedestrians in the cluster, or the velocity V of a particular pedestrian in the cluster whose velocity is closest to the centroid velocity. Furthermore, each cluster can be characterized by the diffusion ΔV of the velocities of the individual pedestrians. Diffusion ΔV can indicate the likelihood of a cluster maintaining its shape and area. For example, a positive diffusion ΔV > 0 may indicate that the cluster size is increasing along one or more dimensions, a negative diffusion ΔV < 0 may indicate that each cluster may decrease along one or more dimensions (becoming more compact and denser), and (approximately) zero diffusion ΔV ≈ 0 may indicate that the cluster size remains substantially unchanged (e.g., over a time range, such as a few seconds). In some implementations, individual initial clusters can be further combined into aggregated clusters using one or more techniques, such as hierarchical aggregate clustering (HAC) or other similar techniques. The behavior of each cluster can then be predicted, for example, after time t, using the initial position R0 of a representative member (or centroid) of the cluster and a representative velocity V, and the position of the representative member

number

[0014] Additional techniques may be deployed to limit the speed of the autonomous vehicle, taking into account the density of pedestrians and / or other VRUs near the autonomous vehicle's driving path. In some implementations, a certain distance L of the path can be set empirically, for example, and the number of objects within this distance can be calculated. The speed of the autonomous vehicle can be determined based on several heuristics. In one embodiment, the distance d from the driving path to the nearest pedestrian is used. MIN This can be determined. In another embodiment, the (linear) density n of pedestrians is, for example, n = N LX2D / L is, for example, the total number of pedestrians N within a rectangle of length L and width 2D of the driving path within a relevant area of ​​the environment. LX2D This can be determined as follows (the width parameter D is set empirically, for example, based on field tests). The speed of the autonomous vehicle U is then determined by the minimum distance d MIN , and a preferred function of density n is f(.), U=f(d MIN It can be set as ,n). The function f(.) is the distance d MIN It can decrease (increase) along with a decrease (increase) of and / or an increase (decrease) in density n.

[0015] The benefits of the disclosed implementation include, but are not limited to, efficient modeling of congestion behavior when a large number of pedestrians are present in the driving environment, and facilitating navigation in congested areas with a safe, courteous, and comfortable speed (relative to other road users), determined considering the actual congestion density in the most relevant areas near the autonomous vehicle's driving path. Slowing down the autonomous vehicle in areas of high congestion density has the additional benefit that the onboard perception system has more time to process more objects in the relevant area. This results in improved driving path selection and increased safety of driving operations.

[0016] In these cases where the implementation description refers to autonomous vehicles, it should be understood that similar techniques may be used in various driver assistance systems that do not reach the level of fully autonomous driving systems. In some embodiments, the disclosed techniques may be used in Level 2 driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver assistance. In some embodiments, the disclosed techniques may be used in Level 3 driver assistance systems that are capable of autonomous driving under limited conditions (e.g., highways). Such systems may use high-speed, accurate detection and tracking of objects to inform the driver of approaching vehicles and / or other objects, and allow the driver to make final driving decisions (e.g., in a Level 2 system) or certain driving decisions (e.g., in a Level 3 system), such as deceleration or lane changes, without requiring driver feedback.

[0017] Figure 1 shows the components of an exemplary vehicle 100 that enables efficient prediction of congestion behavior and safe navigation in congested areas of a driving environment, as demonstrated by several implementations of the present disclosure. In some implementations, vehicle 100 may be an autonomous vehicle. Autonomous vehicles may include automobiles (such as cars, trucks, buses, motorcycles, all-terrain vehicles, recreational vehicles, and any special agricultural or construction vehicles), or any other self-driving vehicle capable of operating in an automated driving mode (without or with reduced human input) (e.g., robots, factory or warehouse robotic vehicles, sidewalk delivery robotic vehicles, etc.).

[0018] The driving environment 101 may include any (moving or stationary) objects located outside the vehicle 100, such as roads, buildings, trees, bushes, sidewalks, bridges, mountains, other vehicles, and pedestrians. The driving environment 101 may be an urban area, suburban area, or rural area. In some implementations, the driving environment 101 may be an off-road environment (e.g., agriculture or farmland). In some implementations, the driving environment may be an indoor environment, such as an industrial plant, a shipping warehouse, or a hazardous area of ​​a building. In some implementations, the driving environment 101 may be substantially flat, with various objects moving parallel to the surface (e.g., parallel to the ground). In other implementations, the driving environment may be three-dimensional, with objects capable of moving along all three directions (e.g., balloons, leaves). Hereafter, the term “driving environment” should be understood to include all environments in which autonomous operation of a self-propelled vehicle may occur. For example, “driving environment” may include any possible flight environment for an aircraft or a marine environment for a marine vessel. Objects in the driving environment 101 may be located at any distance from the vehicle 100, from a short distance of a few feet (or less) to a distance of several miles (or more).

[0019] As described herein, in semi-autonomous or partially autonomous driving modes, the vehicle assists with one or more driving operations (e.g., steering, braking, and / or acceleration for lane centering, adaptive cruise control, advanced driver-assistance systems (ADAS), or emergency braking), but the human driver is expected to situationally perceive the vehicle's surroundings and supervise the assisted driving operations. Here, the vehicle may perform all driving tasks in certain situations, but the human driver is expected to take control as needed.

[0020] For the sake of simplification and brevity, various systems and methods may be described below in conjunction with autonomous vehicles, but similar techniques may be used in various driver assistance systems that do not reach the level of fully autonomous driving systems. In the United States, the Society of Automotive Engineers (SAE) defines different levels of autonomous driving operations to indicate how much or how little a vehicle controls the driving, although different organizations in the United States or other countries may classify the levels differently. More specifically, the disclosed systems and methods may be used in SAE Level 2 (L2) driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver assistance. The disclosed systems and methods may be used in SAE Level 3 (L3) driver assistance systems that enable autonomous driving under limited conditions (e.g., highways). Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 (L4) autonomous driving systems that operate autonomously under most normal driving conditions and require only occasional attention from a human operator. In all such driver assistance systems, accurate lane estimation can be performed automatically without driver input or control (e.g., while the vehicle is moving), resulting in improved reliability of vehicle positioning and navigation, as well as overall safety for autonomous, semi-autonomous, and other driver assistance systems. As stated above, in addition to the way the SAE classifies levels of autonomous driving operations, other organizations in the United States or other countries may classify levels of autonomous driving operations differently. The systems and methods disclosed herein may be used in driver assistance systems defined by the levels of autonomous driving operations of these other organizations, but are not limited to these.

[0021] An exemplary vehicle 100 may include a sensing system 110. The sensing system 110 may include various electromagnetic (e.g., optical) and non-electromagnetic (e.g., audio) sensing subsystems and / or devices. The sensing system 110 may include radar (or multiple radars) 112, which may be any system that utilizes radio or microwave frequency signals to sense objects in the driving environment 101 of the vehicle 100. The radar(s) 112 may be configured to sense both the spatial position of an object and the velocity of an object (e.g., using Doppler shift technology). Hereinafter, “velocity” refers to both how fast an object is moving (object velocity) and the direction of the object’s motion. In some implementations, the sensing system 110 may include a lidar 114, which may be a laser-based unit capable of determining the distance (including their spatial dimensions) to an object in the driving environment 101 and the velocity of the object. Each of the radars 112 and lidars 114 may include a coherent sensor, such as a frequency-modulated continuous-wave (FMCW) lidar or radar sensor. For example, radar 112 may use heterodyne detection for velocity determination. In some implementations, the functionality of ToF and coherent radars is combined into a radar unit that can simultaneously determine both the distance to a reflective object and the line-of-sight velocity of the reflective object. Such a unit may be configured to operate in a non-coherent sensing mode (ToF mode) and / or a coherent sensing mode (e.g., a mode using heterodyne detection), or both modes simultaneously. In some implementations, multiple radars 112 or lidars 114 may be mounted on the vehicle 100.

[0022] Lidar 114 may include one or more light sources that generate and emit signals, and one or more detectors that receive signals reflected back from objects. In some implementations, Lidar 114 may perform a 360-degree scan in the horizontal direction. In some implementations, Lidar 114 may have the ability to perform spatial scans along both the horizontal and vertical directions. In some implementations, the field of view may be up to 90 degrees vertically (e.g., at least a portion of the area above the horizon is scanned by the Lidar signal). In some implementations, the field of view may be spherical (consisting of two hemispheres).

[0023] The sensing system 110 may further include one or more cameras 118 for capturing images of the driving environment 101. The images may be two-dimensional projections of the driving environment 101 (or a portion of the driving environment 101) onto the imaging surface (flat or non-flat) of the camera(s). Some of the cameras 118 of the sensing system 110 may be video cameras configured to capture a continuous (or quasi-continuous) stream of images of the driving environment 101. The sensing system 110 may also include one or more infrared (IR) sensors 119. The sensing system 110 may further include one or more microphone sensors 116 that may be used to capture audio data for the driving environment, such as emergency vehicle sirens and other sounds.

[0024] The sensing data obtained by the sensing system 110 may be processed by the vehicle's data processing system 120. For example, the data processing system 120 may include a perception and planning system 130. The perception and planning system 130 may be configured to detect and track objects in the driving environment 101 and recognize the detected objects. For example, the perception and planning system 130 may be able to analyze images captured by the camera 118 and have the ability to detect traffic signals, road signs, road layout (e.g., traffic lane boundaries, intersection topology, parking lot designations, etc.), the presence of obstacles, etc. The perception and planning system 130 may also receive radar sensing data (Doppler data and ToF data) and determine the distance to various objects in the driving environment 101 and the velocity of such objects (radially, and in some implementations, laterally, as described below). In some implementations, the perception and planning system 130 may use radar data in combination with data captured by the camera(s) 118, as described in more detail below.

[0025] The perception and planning system 130 monitors how the driving environment 101 unfolds over time by, for example, tracking the position and velocity of moving objects (e.g., the Earth, and / or AV) and predicting how various objects will move in the future over a specific time range, e.g., 1 to 10 seconds or longer. The perception and planning system 130 may include a congestion analyzer 132 that predicts congestion behavior and identifies a safe speed regime for navigation in the driving environment 101, for example, as disclosed below in conjunction with Figures 2 to 5. The congestion analyzer 132 may include one or more heuristic modules and one or more trainable MLMs that can process data from multiple modalities, e.g., camera data, radar data, lidar data, voice data, road graph data, and / or similar.

[0026] The perception and planning system 130 may also receive information from a positioning subsystem 122, which may include a GPS transceiver and / or an inertial measuring unit (IMU) (not shown in Figure 1) configured to obtain information about the position of the AV relative to the Earth and its surroundings. The positioning subsystem 122 can use positioning data, e.g., GPS data and IMU data, in conjunction with the perception data to help accurately determine the position of the vehicle 100 relative to fixed objects in the driving environment 101 (e.g., roads, lane boundaries, intersections, sidewalks, crosswalks, road signs, curves, surrounding buildings, etc.), whose positions may be provided by road graph information 124. In some implementations, the data processing system 120 may receive non-electromagnetic data, such as audio data (e.g., ultrasonic sensor data, or data from one or more microphone sensors 116 that detect emergency vehicle sirens), temperature sensor data, humidity sensor data, pressure sensor data, and weather data (e.g., wind speed and direction, precipitation data).

[0027] Various systems and subsystems of the data processing system 120 may have software stored in one or more system memory devices 126. The system memory 126 may include any volatile or non-volatile memory devices, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, flip-flop memory, or any other device capable of storing data. RAM may be static memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), or static random access memory (SRAM). In some implementations, the system memory 126 may be on-chip memory.

[0028] The operation of the data processing system 120 may be carried out by one or more processors 128, which may include CPUs, GPUs, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc. As used herein, “processor” refers to a device capable of executing instructions that code arithmetic, logical, or I / O operations, for example, stored in the system memory 126. In some implementations, the processors 126 and the system memory 130 may be implemented as a single controller, for example, as an FPGA.

[0029] Data generated by the perception and planning system 130, the positioning subsystem 122, and / or other systems, and components of the data processing system 120 may be used by an autonomous driving system, such as a vehicle control system (VCS) 140. The VCS 140 may include one or more algorithms that control how the vehicle 100 should behave in various driving situations and environments. For example, the VCS 140 may include a navigation system for determining a global driving route to a destination. The VCS 140 may also include a driving route selection system for selecting a specific path through the immediate driving environment, which may include selecting a traffic lane, navigating through traffic congestion, selecting a place to make a U-turn, and selecting a trajectory for parking maneuvers. The VCS 140 may also include an obstacle avoidance system for safely avoiding various obstacles in the AV's driving environment, such as stones, stalled vehicles, and pedestrians crossing in disregard of traffic rules and signals. An obstacle avoidance system may be configured to evaluate the size of an obstacle and its trajectory (if the obstacle is moving) and to select the optimal driving strategy (e.g., braking, steering, acceleration) to avoid the obstacle.

[0030] The VCS140 algorithm and modules can generate instructions for various systems and components of a vehicle, including the powertrain, brakes, and steering 150, vehicle electronics 160, signaling 170, and other systems and components not explicitly shown in Figure 1. The powertrain, brakes, and steering 150 may include the engine (internal combustion engine, electric engine, etc.), transmission, differential, axles, wheels, steering mechanism, and other systems. The vehicle electronics 160 may include the onboard computer, engine management, ignition, communication systems, car computer, telematics, in-car entertainment system, and other systems and components. The signaling 170 may include high and low headlights, stop lights, turn signals and taillights, horns and alarms, interior lighting systems, dashboard notification systems, passenger notification systems, radio and wireless network transmission systems, etc. Some of the commands output by the VCS140 can be delivered directly to the powertrain, brakes, and steering 150 (or signaling 170), while other commands output by the VCS140 are first delivered to the vehicle electronics 160, which generates commands for the powertrain, brakes, and steering 150, and / or signaling 170.

[0031] In one example, the VCS 140 may determine that an obstacle identified by the data processing system 120 should be avoided by slowing the vehicle down to a safe speed, and then steering the vehicle to bypass the obstacle. The VCS 140 may output commands to the powertrain, brakes, and steering 150 (directly or via the vehicle electronics 160) to (1) reduce the fuel flow to the engine and decrease the number of engine turns by modifying the throttle setting, (2) downshift the drivetrain to a lower gear via the automatic transmission, (3) engage the brake unit to reduce the vehicle's speed to a safe speed (working in conjunction with the engine and transmission), and (4) use the power steering mechanism to perform steering until the obstacle is safely bypassed. The VCS 140 may then output commands to the powertrain, brakes, and steering 150 to resume the vehicle's previous speed setting.

[0032] In the following diagram descriptions, the term “vehicle” is used to refer to an automotive machine deploying the disclosed technology for identifying and navigating the BL. The term “object” is used to refer to any road user who, intentionally or accidentally, could block the road, or any part thereof. “Object” may include any type of vehicle, such as cars, trucks, vans, SUVs, trailers, motorcycles, scooters, and bicycles, but may also include police officers, emergency responders, pedestrians, animals, and / or similar entities.

[0033] Figure 2 shows an exemplary system architecture 200 of a congestion analyzer capable of predicting congestion behavior and safely navigating congested areas of a driving environment, according to several implementations of the present disclosure. Input to the congestion analyzer (referring to the congestion analyzer 132 in Figure 1) may include data obtained by a sensing system 110 (e.g., by radar 112, lidar 114, camera(s) 118, and / or other sensors). The obtained data may be provided via a sensing data acquisition module 210, which can decode, preprocess (e.g., denoising, upsampling, or downsampling), reformatize, crop, etc., the sensing data into a format accessible to the congestion analyzer. In one exemplary implementation, the sensing data acquisition module 210 can obtain a series of camera images 202, for example, a two-dimensional projection of the driving environment (or a portion thereof) on an array of sensing detectors (e.g., charge-coupled elements, or CCD detectors, complementary metal-oxide-semiconductor, or CMOS detectors, and / or similar). Individual camera images may have pixels of varying intensities, either in one color (for black and white images) or multiple colors (for color images). Camera image 202 may be a panoramic (360-degree) image or an image showing a specific part of the driving environment. Camera image 202 may contain a large number of pixels. The number of pixels may depend on the image resolution. Each pixel may be characterized by one or more intensity values. Black and white pixels may be characterized by a single intensity value representing the brightness of the pixel, for example, where a value of 1 corresponds to a white pixel and a value of 0 corresponds to a black pixel (and vice versa). Intensity values ​​may be continuous (or discrete) values ​​between 0 and 1 (or between any other selected limits, e.g., 0 and 255). Similarly, color pixels may be represented by multiple intensity values, for example, three intensity values ​​(e.g., when using the RGB color encoding scheme) or four intensity values ​​(when using the CMYK color encoding scheme). The camera image 202 can be preprocessed, for example, by downscaling (combining multiple pixel intensity values ​​into a single pixel value), upsampling, filtering, and denoising.The camera image 202 can be in any suitable digital format (JPEG, TIFF, GIG, BMP, CGM, SVG, etc.).

[0034] The sensing data acquisition module 210 can further obtain lidar and / or radar images 204, which may include a set of return points (point clouds) corresponding to radar (lidar) beam reflections from various objects in the driving environment. Each return point can be understood as a data unit (pixel), including the coordinates of the reflective surface, line-of-sight velocity data, intensity data, and / or similar. For example, the sensing data acquisition module 210 can provide a lidar / radar image 204 including a lidar (and / or radar) intensity map I(R,θ,Φ), where R, θ, and Φ may be a set of spherical coordinates. In some implementations, Cartesian coordinates, elliptic coordinates, parabolic coordinates, or other preferred coordinates may be used instead. The lidar (radar) intensity map identifies the intensity of radar (lidar) reflections for various points within the radar (lidar) field of view. The coordinates of an object reflecting a lidar (and / or radar) signal can be determined from directional data (e.g., polar angle θ and azimuth angle Φ in the direction of signal transmission) and distance data (e.g., radial distance R determined from the time of flight of the signal). The lidar / radar image 204 may further include velocity data of various reflecting objects identified based on the detected Doppler shift of the reflected signal.

[0035] The camera image 202 and lidar / radar image 204 may be a large image of the entire driving environment or an image of a smaller portion of the driving environment (e.g., camera images acquired by the front camera(s) of the sensing system 110). In some implementations, the sensing data acquisition module 210 may crop the camera image 202 and lidar / radar image 204 to correspond to a specific segment around the direction of the vehicle's movement. For example, in one exemplary non-limiting implementation, since the relevant object of interest is typically located around the direction of the vehicle's movement, the sensing data acquisition module 210 may crop the camera image 202 and lidar / radar image 204 within a forward-view segment 200-250 m long and 20-40 m wide. The size of the segment may depend on the vehicle's speed and the type of driving environment, and may differ between highway driving environments and urban driving environments.

[0036] Camera images 202, LiDAR / radar images 204, and in some implementations, road graph information 124 can be used as input to an object detection module 220 that identifies individual objects in the driving environment. The object detection module 220 may be (or may include) any suitable computer vision model, such as a machine learning model trained to identify areas containing objects of interest, such as vehicles, pedestrians, or animals. Predictions of the actions (behavior) of vehicles and other non-pedestrian objects may be handled by other modules of the perception and planning system 130 (see Figure 1).

[0037] A VRU (or other object) 230 may be processed by an object-level behavior prediction (BP) 240. Inputs to the object-level BP 240 may include the location (e.g., coordinates) of the VRU / object 230, and various objects in the environment, e.g., the layout of buildings, roads, and sidewalks, and / or similar. In some implementations, inputs to the object-level BP 240 may include the speed of pedestrians, as well as various other objects, e.g., vehicles, construction staff and equipment, and / or similar. In some implementations, inputs to the object-level BP 240 may include the aforementioned information, and / or time t1, t2, ... t, in order to inform the object-level BP 240 about the pedestrian's motion history in the context of a changing environment. N It may include any other suitable data for the number of (timestamps, sensing frames, etc.). The object-level BP 240 can predict the behavior (operation) of the VRU / object 230 over a specific future time range, such as several seconds or more.

[0038] Clustering 250 can group VRUs / objects 230 into clusters in which individual members move in a similar manner to other members of the same cluster. Figure 3A illustrates the identification of individual pedestrians and / or other VRUs as part of predicting congestion behavior and safe navigation in a congested area of ​​a driving environment, as demonstrated by several implementations of the present disclosure. Schematically shown in Figure 3A are seven pedestrians 301, 302, ... 307 in a region of an autonomous vehicle environment. In some implementations, pedestrians may be identified using coordinates, boundary shape, class (e.g., adult, child, wheelchair pedestrian, etc.), and / or similar. The boundary shape is shown as a circle in Figure 3A, but any other suitable geometric shape (e.g., boundary rectangle, convex hull, etc.) may be used. Figure 3A shows pedestrians 301–307 using a top view (bird's-eye view, BEV) in which pedestrians and / or boundary shapes can be identified using Cartesian in-plane coordinates X and Y, but various other coordinates, e.g., spherical coordinates, coordinates of an oblique view of the environment (e.g., camera), and / or similar may also be used. Identification of individual pedestrians 301–307 may further include the direction and velocity V that each pedestrian faces, which includes the direction of motion (which may be different from the opposing direction) and the velocity of that motion.

[0039] Figure 3B illustrates the identification of groups of pedestrians and / or other VRUs having similar behavioral patterns as part of predicting congestion behavior and safe navigation in congested areas of a driving environment, by several implementations of the present disclosure. Such groups of pedestrians may be formed using one or more heuristics. For example, groups of pedestrians may be identified based on (1) the distance from a candidate pedestrian to other potential candidate pedestrians (or pedestrians already included in that group), (2) the speed of the candidate pedestrian, (3) the expected distance to other candidate pedestrians after a specific set time t, e.g., 1 second, 1.5 seconds, 2 seconds, etc., and / or other suitable criteria. For example, an initial clustering criterion may include the distance from a candidate object to at least one pedestrian already included in a group that does not exceed a (first) threshold distance D1. Similarly, another clustering criterion may include the expected (intended, predicted) distance after time t to at least one other pedestrian in the group being less than a second threshold distance D2, which may be the same as or different from the first threshold distance D1. Further clustering criteria may include the candidate pedestrian's velocity (both speed and direction of movement) and / or similar, being within a specific tolerance range from the center of gravity velocities of other pedestrians in the group. The tolerance range may be specified in absolute units (e.g., 0.3 m / s, 0.5 m / s, or some other value) or relative units (e.g., 25% of the candidate pedestrian's velocity, or some other value). In some implementations, at least a minimum number (e.g., 1, 2, etc.) of the above criteria (and / or any other suitable criteria) must be met before a candidate pedestrian is included in a group of other pedestrians. In some implementations, a more flexible approach may be used, for example, lowering the threshold for the remaining criteria if one or more of the clustering criteria are met. For example, if a candidate object has a highly matching velocity, the threshold distance between pedestrians that are still grouped together may be increased, and vice versa.

[0040] As schematically shown in FIG. 3B, pedestrians 301-303 can be grouped into a first group 310 based on their spatial proximity and / or speed similarity, pedestrians 304-305 can be grouped into a second group 311, and pedestrians 306-307 can be grouped into a third group 312. Pedestrian 303, having a different position and / or speed compared to other groups, can remain in its own single object group. In FIG. 3B, further schematically shown by dashed lines is that the outer boundaries of each group can be joined to form a convex hull, or other boundary shape, to form a cluster of pedestrians. FIG. 3C shows the formation of clusters based on groups of pedestrians and / or other VRUs having similar operating patterns according to some implementations of the present disclosure. The convex hull can be any polyline or curve that closely encloses the individual pedestrians within each cluster. FIG. 3C shows three multi-pedestrian clusters 320-322 having a pedestrian 303 associated with its own single pedestrian cluster 323. In some implementations, pedestrians and / or other VRMs of a particular type can be excluded from grouping with other pedestrians and tracked separately. For example, such particular types of VRMs can include children, VRMs using wheelchairs and / or other mobility aids, VRMs guided by service animals, and / or other types of VRMs that are likely to have unexpected (e.g., similar to children) and / or individualized (similar to wheelchair users) operating patterns.

[0041] The individual clusters have a representative cluster speed V 1、 V 2、V3, etc., may be assigned. In some implementations, the representative cluster velocity can be the centroid velocity (e.g., mean or median) of the pedestrians within each cluster, V. In some implementations, the representative velocity can be the velocity V of a specific pedestrian within the cluster. The representative pedestrian may be a pedestrian located near the center of the cluster, whose velocity is closest to the centroid velocity, and / or similar, or a pedestrian selected using some other set of conditions (or a weighted combination of conditions).

[0042] Furthermore, in some implementations, each individual multi-pedestrian cluster can be characterized by the respective diffusion 4V of the velocities of its members. For example, if the minimum velocity of pedestrians in a cluster is 1.2 m / s and the maximum velocity of pedestrians in the same cluster is 1.6 m / s, the diffusion could be 4V = 0.4 m / s. Indicators of diffusion 4V can track whether the cluster size is growing (4V > 0) or shrinking (4V < 0).

[0043] In some implementations, referring again to Figure 2, individual initial clusters may undergo cluster aggregation 260. In some implementations, cluster aggregation 260 may involve combining tightly located clusters with not significantly different operational characteristics into an aggregated cluster, for example, using one or more of the techniques of Hierarchical Agglomerative Clustering (HAC) or other preferred techniques. More specifically, each cluster may be associated with a point in an M-dimensional cluster space, where the number of dimensions M may correspond to several cluster characteristics, e.g., cluster location, cluster velocity V, type of object in the cluster, and / or similar. Cluster aggregation 260 may involve identifying clusters whose distance (e.g., Euclidean distance) in the M-dimensional cluster space is less than a certain empirically defined distance, and then combining such clusters with the aggregated cluster.

[0044] In some implementations, the operation of clustering 250 and / or cluster aggregation 260 may be performed by a trained machine learning model, such as a neural network-based model. Input to the machine learning model may include, some or all of, the position of individual VRUs, the velocity of individual VRUs, the direction of motion of individual VRUs, the type of individual VRUs, and / or similar. Output data generated by the trained machine learning model may further include the position and / or velocity of clusters of VRUs, for example, as shown in Figures 3D-3F, the boundary shape encompassing the members of the cluster(s), and the cluster centroid (and / or other reference point). The machine learning model may be trained using logs of sensory data collected during actual driving missions, for example, using various vehicle sensors such as cameras, radar, lidar, ultrasonic sensors, and / or similar. In some implementations, training may be supervised, for example, by using clusters annotated by the developer as ground truth. In some implementations, training may not be supervised. In such cases, the identified clusters can be compared to sensing data (logs) over a relatively short time range (e.g., 1 to a few seconds) that have the behavior of individual members of the cluster, compared to tracking predictions made using the model-identified clusters. A score may be assigned to a given cluster predicted by the model, and the score indicates how well the model-based prediction matches the actual behavior of the individual VRUs of the cluster. For example, if all (or most) VRUs contained within a cluster move in substantially the same direction (at most within a set empirically defined tolerance), the model-based prediction may receive a high score. Conversely, if some VRUs contained within a cluster move in different directions (beyond the set tolerance), the model-based prediction may receive a low score. The score can then be used in place of ground truth, and training may be carried out using various training techniques, such as backpropagation, stochastic gradient descent, and / or similar, until the model consistently achieves a high score.

[0045] Figure 3D illustrates the identification of individual clusters as part of cluster aggregation for efficient prediction of congestion behavior in congested areas in a driving environment and for safe navigation, as demonstrated by several implementations of this disclosure. As shown, cluster 321 and single pedestrian cluster 323 can be identified as clusters having close similarity (calculated, for example, based on the proximity of the relative positions of the two clusters and the corresponding representative velocities V2 and V4).

[0046] Figure 3E illustrates the formation of an aggregated cluster based on the merging of individual clusters, as demonstrated by several implementations of this disclosure. As shown, cluster 321 and single pedestrian cluster 323 are merged into aggregated cluster 325. The representative velocity V5 of aggregated cluster 325 can be the average or weighted average of the individual cluster velocities V2 and V4, for example, weighted by the number of pedestrians in each cluster. For example, in the case of two-pedestrian cluster 321 and single pedestrian cluster 323, the representative velocity V5 is:

number

[0047] In some implementations, referring again to Figure 2, cluster-level behavior prediction 270 can be performed using individual clusters and / or aggregated clusters. More specifically, the behavior of each cluster can be predicted using the cluster's initial position R0 and representative velocity V, and as a result, after time t, the cluster's position (e.g., reference point, centroid, representative member of the cluster, etc.) is:

number

[0048] The cluster-level behavior prediction 270 may be repeated periodically. Figure 3G shows an exemplary timeline of cluster formation and behavior prediction in some implementations of the present disclosure. New cluster formation may be performed at regular time intervals, e.g., a first cluster formation 330, a second cluster formation 331, a third cluster formation 332, and / or similar. In some implementations, cluster formation may be repeated when the object detection module 220 receives and processes new sensing data, e.g., every 0.3 seconds, every 0.5 seconds, every 1 second, and / or similar. Following the first cluster formation 330 (which may be performed using operations 220-260, e.g., as shown in Figures 3A-3E), the cluster-level behavior prediction 270 in Figure 2 may generate predictions for a first prediction range 340, e.g., 5 seconds, 10 seconds, and / or similar. After the object detection module 220 generates a new set of VRUs / objects 230, it may perform a second cluster formation 331, followed by a new behavior prediction for a second prediction range 341, and / or similar actions.

[0049] The output of the cluster-level behavior prediction 270 indicates the congestion block zones 282 that the autonomous vehicle should avoid at various time ts. For example, the planner 280 may determine a driving path 284 for the autonomous vehicle that avoids intersections of any clusters and, in some implementations, with a specific safety margin around the clusters (e.g., 1m).

[0050] The VRU / object 230 may also be used to perform congestion density mapping 290 to identify deceleration zones 286, taking into account the density of pedestrians near the driving path 284, and to determine a safe speed for the autonomous vehicle traveling through or near areas where pedestrian congestion exists. Deceleration zones 286 refer to areas where safety considerations indicate an additional deceleration of the autonomous vehicle (compared to legal speed limits), even if legal speed limits do not require such deceleration.

[0051] Figure 4A schematically shows a portion of the driving environment 400, including areas where multiple pedestrians and / or other VRUs are present, according to several implementations of the present disclosure. The autonomous vehicle 402 may select a speed regime in different areas based on the density of pedestrians (indicated by circles) in those areas. For example, the autonomous vehicle 402 may select a medium speed in a medium-density area 404, a high speed (e.g., the maximum legal speed limit) in a medium-density area 406, a low speed in a high-density area 408, and / or similar. In some implementations, the speed selection of the autonomous vehicle 402 may be communicated using one or more suitably selected functions, for example, based on relevant areas near the vehicle's (expected) driving path 403.

[0052] Figure 4B schematically shows an example of a related region 410 that, according to some implementations of this disclosure, may be used to select a safe operating speed for the autonomous vehicle when multiple pedestrians and / or other VRUs are present. In some implementations, the related region 410 may have the form of a rectangle with length L and lateral width 2D along the driving path 403 of the autonomous vehicle 402. The length L (e.g., 20–30 m, or some other preferred value) and width 2D (e.g., 15–20 m, or some other preferred value) may be set empirically. In some implementations, the length L and / or width 2D may depend on the speed of the autonomous vehicle 402. The congestion density mapping module of the perception system of the autonomous vehicle 402 determines the number of pedestrians N detected within the related region 410. Lx2D The safe speed of the autonomous vehicle 402 is the distance d from the driving path 403 to the nearest pedestrian. MINThe safe speed of the autonomous vehicle 402 can be determined based on the following: In one implementation, the safe speed of the autonomous vehicle 402 is determined by a preferred defined density of pedestrians n, for example, n = N Lx2D Further determinations can be made based on / L. The speed of the autonomous vehicle U is determined by this minimum distance d. MIN , and a suitable function f(.) of density n, for example, U=f(d MIN, n) can be set as follows: The function f(.) is the minimum distance d MIN It can be an increasing function of and a decreasing function of density n. In some implementations, the function f(.) is a function of its variable d MIN , and can be defined as a model function (e.g., a continuous function) of n. In some implementations, the function f(.) is defined by its variable d MIN With respect to n, it can be defined as a discrete set of bins. In some implementations, the associated region 410 may have a non-rectangular form, for example, a two-dimensional shape with a width that varies along the driving path 403 such that the width is greater in the vicinity of the autonomous vehicle 402 and smaller further away from the autonomous vehicle 402.

[0053] In some implementations, clustering (e.g., clustering 250 in Figure 2) can be used to group other moving objects (not limited to VRUs) into clusters, where individual members move in a manner similar to other objects in the cluster, including (but not limited to) vehicles moving on a road. A planner (e.g., planner 280 in Figure 2) can then predict the collective behavior of such clusters by treating the clusters as a single entity (substantially similar to VRU clusters, as disclosed above in relation to Figures 3F and 3G).

[0054] Clustering of moving objects may be performed using several clustering criteria or metrics, which may include, but are not limited to, any, some, or all of the following:

[0055] One of the clustering metrics includes the distance D from a candidate object to one or more other objects within the same cluster (the expected cluster to which the candidate object belongs). For example, a group of vehicles moving in the same direction along a road may belong to the same cluster, provided that the distance D between the vehicles is not too large. In one embodiment, the distance D between the candidate object and all other objects within the cluster is D MAX If it exceeds this value, the candidate object will be excluded from the cluster, using the hard cutoff maximum distance D. MAX This can be used. In another embodiment, the distance D to the nearest object in a cluster may be evaluated along with other metrics (e.g., for calculating distances in the clustering space, as described below), while a maximum distance does not need to be imposed. In some implementations, if the distance from an object to the furthest object in the cluster exceeds some maximum distance, the object is not assigned to the cluster. This can prevent the formation of overly large clusters (e.g., long lines of heavy traffic). Distance measurements may prioritize candidate objects that are close to other objects assigned to the cluster, and disregard candidate objects that are further away from such objects.

[0056] Another clustering metric includes the velocity of the candidate object, or the velocity of the object relative to some reference velocity for the cluster, for example, the average velocity of the objects assigned to the cluster. In cases where the movement of objects is constrained to one direction (e.g., vehicles on a highway), the velocity of the object (absolute value, or magnitude of velocity) may be used instead of those velocities. This measure can prioritize candidate objects moving at velocities similar to those of other objects assigned to the cluster, and disregard candidate objects moving at velocities different from those of the cluster.

[0057] Another clustering metric may include, for example, the acceleration of a candidate object relative to the acceleration of other objects assigned to the cluster (e.g., average acceleration). Acceleration should be understood to include deceleration (negative acceleration). This measure can prioritize candidate objects whose behavior changes in the same way as other objects assigned to the cluster, and not those whose behavior changes differently from such objects.

[0058] Another clustering metric may include lane match / mismatch between the lane occupied by a candidate object and the lane occupied by other objects. For example, various lanes for a given direction of movement can be numbered, e.g., 1, 2, 3, up to the number of lanes on a road. The average number of lanes, LaneN, can be determined for a cluster. For example, if all objects in the cluster are moving in the right lane (lane 1), the average number of lanes can be LaneN = 1.0; if two objects in the cluster are moving in the right lane and three are moving in the middle lane (lane 2), LaneN = 1.6; and if one object in the cluster is moving in the right lane, one in the middle lane, and three are moving in the left lane (lane 3), LaneN = 2.4. The lane match / mismatch measurement may include the difference between the number of lanes for a candidate object and the average number of lanes for the cluster, LaneN. This measure prioritizes candidate objects that move in the same lane as other objects assigned to the cluster, and does not prioritize candidate objects that move in other lanes.

[0059] Other clustering metrics may include object type, lane destination mismatch, and / or similar factors. For example, trucks may be preferred to cluster with other trucks, but not with cars, motorcycles, or buses. Similarly, vehicles moving in forward lanes may not be preferred to be included in the cluster with vehicles moving in left-turn lanes.

[0060] In one exemplary implementation, the values ​​M1...M indicate the degree of discrepancy between various metrics for candidate objects and each metric for clusters (e.g., cluster centroids). n This can be considered as a distance along the corresponding dimension of the clustering space, where dimension n is determined by the number of measurement criteria. The clustering measurement (distance within the clustering space) can be calculated as follows:

number

[0061] Figures 5A–5C illustrate exemplary clusters of non-VRU objects traveling on a highway or street, according to several implementations of the present disclosure. Figure 5A shows an exemplary section of a two-lane road 500 occupied by nine vehicles. Arrows indicate the direction of vehicle movement, and the length of the arrows roughly indicates the vehicle speed. Figure 5A depicts two clusters, each confined to its own lane: namely, cluster 502 of vehicles traveling in the left lane and cluster 504 of vehicles traveling in the right lane. Although all five vehicles are positioned in close proximity, the vehicles in the slower cluster 504 may prevent cluster 504 from coupling with cluster 502. Other vehicles, shown in Figure 5A, with speeds substantially different from the speeds of other nearby vehicles, will not couple with cluster 502 or 504 (equivalently, these vehicles may be treated as single-object clusters).

[0062] Figure 5B shows another exemplary section of a two-lane road 510 occupied by eight vehicles. Figure 5B shows two multi-lane clusters, namely cluster 512 for slow-moving vehicles and cluster 514 for fast-moving vehicles. Figure 5C shows an exemplary section of a three-lane road 520, including two forward-moving lanes and a left-turn lane. Figure 5C shows two multi-lane clusters: namely cluster 522 for slow-moving forward vehicles and cluster 524 for fast-moving forward vehicles. Furthermore, Figure 5C shows cluster 526 of vehicles preparing to make a left turn. The vehicles in cluster 526 may be moving at the same (or similar) speeds and may be located near the vehicles in cluster 522, but the two clusters are not joined together due to a metric that does not prioritize mismatch in lane destinations.

[0063] In some implementations, a method using object (including non-VRU) clustering may include using the vehicle's sensing system to acquire sensing data associated with the vehicle's driving environment. The method may further include using a processing device and, based on the sensing data, detecting multiple objects in the driving environment. The method may further include applying one or more clustering criteria (measurements) to multiple objects in the driving environment to form clusters of one or more objects. Individual clustering metrics may include the distance between an individual object and the reference point location of each cluster (e.g., center of gravity location) or one or more objects in each cluster. Individual clustering metrics may further include, for example, a reference velocity (e.g., center of gravity velocity), or the velocity of each object relative to the velocity of one or more objects in each cluster. Individual clustering metrics may further include, for example, a reference acceleration (e.g., center of gravity acceleration), or the acceleration of each object relative to the acceleration of one or more objects in each cluster. Each clustering metric may further include, for example, the lanes occupied by each object for each cluster occupied by one or more objects. Each clustering metric may further include, for example, the destination(s) of the lanes occupied by each object for each cluster occupied by one or more objects. Each clustering metric may further include, for example, the type(s) of each object (e.g., car, truck, bus, motorcycle, scooter, and / or similar) for each cluster occupied by one or more objects. In some implementations, applying one or more clustering criteria (metrics) to multiple objects may involve evaluating the Euclidean distance in multidimensional index space between a point representing a candidate object and a reference point(e.g., centroid) of one or more objects already assigned to each cluster of each object in each cluster and / or object.The method may further include predicting the movement of one or more formed clusters over a time interval based, for example, on the size of the cluster, the position of the cluster's center of gravity, the velocity of the cluster's center of gravity, the diffusion (e.g., dispersion) of the cluster's position / velocity, and / or other characteristics of the cluster, and / or similar. The method may further include determining the driving path of a vehicle in the driving environment, taking into account the predicted movement of the clusters. For example, a vehicle may choose a driving path that avoids the clusters by avoiding entering the space occupied by the clusters, for example by moving in a lane not occupied by the clusters, or by waiting until the clusters move away from the vehicle.

[0064] In some implementations, the method may involve forming multiple initial clusters of objects and then using HAC to aggregate the multiple initial clusters of objects into a larger cluster (for example, as disclosed above in conjunction with VRU).

[0065] In some implementations, applying one or more clustering metrics to multiple objects may involve using a trained machine learning model to process input data, including object locations (including the lanes occupied by the objects), road graph information (e.g., a map of road lanes), object velocities, direction of motion of objects, and / or similar. The machine learning model may be trained, for example, in conjunction with VRUs as disclosed above.

[0066] Figure 6 shows examples of Method 600, as implemented in several ways of this disclosure, for predicting congestion behavior in a driving environment and safely navigating congested areas in a driving environment. A processing device having one or more processing units (CPUs), one or more graphics processing units (GPUs), one or more parallel processing units (PPUs), and memory devices communicably coupled to the CPU(s), GPU(s), and / or PPU(s) may implement Method 600 and / or each of its individual functions, routines, subroutines, or operations. Method 600 may implement a vehicle system and its components. In some implementations, the vehicle may be an autonomous vehicle. In some implementations, the vehicle may be a driver-operated vehicle with driver assistance systems, e.g., Level 2 or Level 3 driver assistance systems, that provide limited assistance by specific vehicle systems (e.g., steering, braking, acceleration, etc.) or under limited driving conditions (e.g., highway driving). A processing device that performs Method 600 (e.g., processor 128 in Figure 1) may execute instructions issued by the perception and planning system 130 in Figure 1, more specifically, the congestion analyzer 132, during the operation of a vehicle. The operation of Method 600 may be performed in response to instructions stored in non-volatile computer-readable memory (e.g., system memory 126 in Figure 1). In a particular implementation, a single processing thread may perform Method 600. Alternatively, two or more processing threads may perform Method 600, each thread performing one or more individual functions, routines, subroutines, or operations of the Method. In a descriptive example, the processing threads performing Method 600 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads performing Method 600 may be executed asynchronously with respect to each other. Some operations of Method 600 may be performed in an order different from the order shown in Figure 6. Some operations of Method 600 may be performed simultaneously with other operations. Some actions can be optional.

[0067] In block 610, method 600 may include using the vehicle's sensing system to acquire sensing data associated with the driving environment (e.g., camera images 202, lidar, and / or radar images 204, see Figure 2). In block 620, method 600 may include using a processing device (e.g., processor 128 in Figure 1) and based on the sensing data to detect multiple vulnerable road users (VRUs) (e.g., VRU / object 230 in Figure 2) in the driving environment. In some implementations, detecting VRUs may be done using an object detection module 220 (shown in Figure 2). VRUs may include, but are not limited to, pedestrians, bike cyclists, scooter riders, wheelchair users, animals, horse / donkey riders, and / or any other road / sidewalk users that may be traveling collectively as a group of two or more entities.

[0068] In block 630, method 600 may be continued by applying one or more clustering metrics to multiple VRUs in a driving environment to form one or more clusters of VRUs (e.g., as shown in Figures 3A-3E). Each of the one or more clusters of VRUs may be associated with a geometric shape enclosing one or more VRUs in each cluster of VRUs (e.g., as shown in Figures 3A-3C), and with a velocity associated with the collective motion of one or more VRUs in each cluster of VRUs. In some implementations, the geometric shape may be a rectangle, square, circle, ellipse, parallelepiped, and / or any other suitable geometric shape. In some implementations, the velocity associated with the collective motion of one or more VRUs in each cluster may be calculated as the average velocity of one or more VRUs in each cluster. In some implementations, the velocity associated with the collective motion of one or more VRUs may be a weighted average velocity, where the velocity of individual VRUs near the center of the cluster is given a greater weight than the velocity of VRUs near the edges of the cluster. In some implementations, the velocity associated with the collective motion of one or more VRUs can be a one-dimensional velocity determined along the direction of the road (sidewalk, road intersection, etc.). In some implementations, the velocity associated with the collective motion of one or more VRUs can be a multi-dimensional velocity with components along multiple spatial directions (e.g., along and perpendicular to roads / sidewalks / intersections, etc.).

[0069] In some implementations, applying one or more clustering metrics to multiple VRUs may involve operations shown in the upper callout portion of Figure 6. More specifically, in block 632, method 600 may include determining that the distance from each of one or more VRUs to the reference point location of each cluster of the VRUs is below a threshold distance. For example, the reference point may be the centroid of the cluster or a representative member of the cluster. The centroid (or other reference point) does not have to be static and may change when one or more VRUs are added to (or removed from) a given cluster. In some implementations, the threshold distance may be a fixed empirical distance (e.g., measured in meters). In other implementations, the threshold distance may be defined for the size of the cluster, including a specific percentage of the cluster's dimensions, such as the longest, shortest, average, etc. In some implementations, the threshold distance can be defined in relation to the maximum size of a cluster; for example, clusters exceeding the maximum distance may be split into two or more smaller clusters (e.g., to improve the predictability of the future behavior of the clusters).

[0070] In some implementations, applying clustering metrics may involve determining in block 634 that the difference in speed between one or more VRUs in each cluster of VRUs falls below a threshold difference. In some implementations, the threshold difference may be a fixed empirical value (e.g., measured in meters per second, e.g., 1 m / s, 0.5 m / s, etc.). In other implementations, the threshold difference may be defined relative to the average speed of the cluster (e.g., 20%, 15%, etc.).

[0071] In some implementations, applying clustering metrics to multiple VRUs may involve, in block 636, evaluating the Euclidean distance in position-velocity space between each of one or more clusters of each VRU and the centroid of each cluster of each VRU. For example, the position-velocity space is a vector (x,y,v) for a given VRU. x ,v y) to characterize the operating state of the VRU, any, some, or all, x coordinates, y coordinates, v x velocity, V y Velocity may be included. The centroid of a particular cluster is a similar vector (X, Y, V x ,V Y It can also be associated with the following, where X and Y are the coordinates of the center of mass, and Vx and VY are the velocities of the center of mass. The Euclidean distance can be calculated as follows.

number

[0072] In some implementations, the operation of block 630 may further include forming multiple initial clusters of VRUs and aggregating at least two of the multiple initial clusters of VRUs using hierarchical aggregate clustering (HAC) (as shown, for example, in Figures 3D-3E).

[0073] In some implementations, the operation of block 630 may involve processing input data with a trained machine learning model to obtain output data. The input data may include (i) the locations of multiple VRUs, (ii) the velocities of multiple VRUs, and (iii) the operating directions of multiple VRUs, and / or similar. The output data may include one or more clusters of VRUs.

[0074] In some implementations, method 600 may be continued in block 640 by predicting one or more VRU block regions for a time interval (e.g., 2 seconds, 3 seconds, 5 seconds, and / or similar) using the geometric shape and velocity associated with one or more clusters of the VRU. For example, the coordinates (X,Y) of the cluster (e.g., the centroid of the cluster) and velocity (V) in the current time instance. X, V Y Using knowledge of ), the position of the cluster (e.g., the centroid) at a later time t can be predicted.

[0075] In some implementations, each of one or more clusters of VRUs is further associated with the velocity diffusion of one or more VRUs in each cluster of VRUs, with a time interval X(t) = X + V x t;Y(t)=Y+Y x Predict one or more VRU block regions in time t. In some implementations, predicting one or more VRU block regions may involve the operation of intermediate callout block 642, which includes estimating the change in the geometric shape of each cluster of VRUs over a time interval. For example, velocity diffusion may be used to estimate the increase (or decrease) in the area occupied by the clusters (as shown, e.g., in Figure 3F).

[0076] In block 650, method 600 may include determining the driving path of a vehicle in a driving environment, taking into account a time interval and one or more VRU block regions. For example, a vehicle processing system may predict regions occupied by various formed clusters at different moments within a time interval and avoid these regions. In some implementations, determining the driving path of a vehicle may include operations in bottom callout block 652, which include determining the speed of the vehicle's driving path based on the minimum distance from the vehicle's driving path to multiple VRUs, or VRUs of many VRUs, located in the relevant region around the vehicle's driving path (for example, as shown in Figure 4). In some implementations, the speed of the vehicle's driving path may be a decreasing function of the number of VRUs in the relevant region, for example, as the number of VRUs in the relevant region increases (decreases), and the maximum speed of the vehicle (different from the legal speed limit, e.g., lower than the legal speed limit) may decrease (increase).

[0077] In some implementations, for example in autonomous vehicles, determining the vehicle's driving path can be performed by the autonomous vehicle's vehicle control system (e.g., VCS140 in Figure 1). In some implementations, for example in driver-operated vehicles, determining the vehicle's driving path can be performed by the driver according to a driver assistance system that outputs driving path suggestions to the driver.

[0078] Figure 7 shows a block diagram of an exemplary computer device 700 in which a congestion analyzer can be deployed to predict congestion behavior and to safely navigate congested areas of a driving environment, according to several implementations of the present disclosure. The exemplary computing device 700 may be a computing device that implements the data processing system 120 of Figure 1. The exemplary computer device 700 may be connected to other computer devices in a LAN, intranet, extranet, and / or the Internet. The computer device 700 may operate with server capabilities in a client-server network environment. The computer device 700 may be a personal computer (PC), set-top box (STB), server, network router, switch or bridge, or any device capable of executing a set of instructions (sequentially or otherwise) that specify the actions to be taken by such device. Furthermore, although only a single embodiment of a computer device is shown, the term “computer” shall also be considered to include any collection of computers that execute a set of instructions (or sets of instructions), individually or together, to implement any one or more of the methods discussed herein.

[0079] An exemplary computer device 700 may include a processing device 702 (also referred to as a processor or CPU), main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), static memory 706 (e.g., flash memory, static random access memory (SRAM), etc.), and secondary memory (e.g., a data storage device 718), which may communicate with each other via bus 730. In some implementations, the processing device 702 may be the processor 128 in Figure 1, or may include the processor 128, and the main memory 704 may be the system memory 126 in Figure 1, or may include the system memory 126.

[0080] The processing device 702 (which may include a logic processor 703) represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or similar. More specifically, the processing device 702 may be a composite instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that executes other instruction sets, or a processor that executes combinations of instruction sets. The processing device 702 may also be one or more application-specific processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or similar. According to one or more aspects of this disclosure, the processing device 702 may be configured to execute instructions that perform various methods for predicting congestion behavior and safely navigating congested areas in a driving environment, as disclosed herein.

[0081] An embodiment of the computer device 700 may further include a network interface device 708 that can be communicatively coupled to a network 720. The exemplary computer device 700 may further include a video display 710 (e.g., a liquid crystal display (LCD), a touchscreen, or a cathode ray tube (CRT)), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), and an audio signal generator 716 (e.g., a speaker).

[0082] The data storage device 718 may include a computer-readable storage medium (or more specifically, a non-temporary computer-readable storage medium) 728 on which one or more sets of executable instructions 722 are stored. According to one or more aspects of the present disclosure, the executable instructions 722 may include executable instructions that perform various methods for predicting congestion behavior and safely navigating congested areas in a driving environment, as disclosed herein.

[0083] The executable instructions 722 may also reside, entirely or at least partially, in main memory 704 and / or processing device 702 during their execution, for example, in an exemplary computer device 700, and main memory 704 and processing device 702 also constitute computer-readable storage media. The executable instructions 722 may also be transmitted or received over a network via network interface device 708.

[0084] Although the computer-readable storage medium 728 is shown as a single medium in Figure 7, the term “computer-readable storage medium” should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of operational instructions. The term “computer-readable storage medium” should also be understood to include any medium that has the ability to store or encode a set of instructions for machine execution that causes a machine to perform any one or more of the methods described herein. Thus, the term “computer-readable storage medium” should be understood to include, but not be limited to, solid-state memory, as well as optical media and magnetic media.

[0085] Some parts of the detailed description above are presented in terms of algorithms of operation and symbolic representations of data bits in computer memory. These algorithmic descriptions and representations are means used by those skilled in the field of data processing to most effectively communicate the content of the work to others skilled in the art. An algorithm is understood here, and generally, to be a step-self-consistent sequence that produces a desired result. A step is one that requires the physical manipulation of a physical quantity. Usually, but not always, these quantities take the form of electrical or magnetic signals that can be stored, moved, combined, compared, and otherwise manipulated. Referring to these signals as bits, values, elements, symbols, characters, terms, numbers, or similar has sometimes proven convenient, primarily for reasons of general use.

[0086] However, it should be noted that all these terms, and similar terms, are associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Otherwise, unless specifically stated, as will be evident from the following considerations, any considerations throughout the explanation using terms such as “specify,” “determine,” “store,” “adjust,” “produce,” “return,” “compare,” “generate,” “stop,” “load,” “copy,” “insert,” “replace,” “implement,” or similar terms will be understood to refer to the operation and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of a computer system to other data similarly represented as physical quantities in computer system memory or registers or other such information storage, transmission, or display devices.

[0087] Examples of this disclosure also relate to apparatus for carrying out the methods described herein. Such apparatus may be specifically constructed for a required purpose or may be a general-purpose computer system selectively programmed by computer programs stored within the computer system. Such computer programs may be stored on computer-readable storage media, including, but not limited to, any type of disk, including optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic disk storage media, optical storage media, flash memory devices, other types of machine-accessible storage media, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0088] The methods and representations presented herein are not inherently related to any particular computer or other device. Various general-purpose systems may be used with the programs taught herein, or it may be convenient to construct more specialized devices to carry out the necessary method steps. The necessary structures for various such systems will appear as described below. In addition, the scope of this disclosure is not limited to any particular programming language. It will be understood that various programming languages ​​can be used to carry out the teachings of this disclosure.

[0089] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. While this disclosure describes specific examples, it will be recognized that the systems and methods of this disclosure are not limited to the examples described herein and can be modified and implemented within the scope of the appended claims. Therefore, the specification and drawings should be considered illustrative, not restrictive. Accordingly, the scope of this disclosure should be determined by reference to the appended claims, along with the entire scope of equivalents to which such claims are entitled.

Claims

1. It is a system, A vehicle sensing system configured to acquire sensing data associated with the driving environment, The data processing system for the vehicle, wherein the data processing system is Applying one or more clustering metrics to multiple vulnerable road users (VRUs) within the aforementioned driving environment to form one or more clusters of VRUs, wherein each of the one or more clusters of VRUs is The geometric shape surrounding one or more VRUs in each cluster of VRUs, The speed associated with the collective operation of one or more VRUs in each cluster of the VRUs, and the associated application, Using the geometric shape and velocity associated with one or more clusters of the VRU, predict one or more VRU block regions over a time interval. A system comprising: a data processing system configured to determine the driving path of the vehicle in the driving environment, taking into account the aforementioned time interval and the one or more VRU block regions.

2. In order to apply the one or more clustering metrics to the one or more VRUs of each cluster of the VRU, the data processing system: The distance from each of the one or more VRUs to the reference point position of each cluster of the VRUs is less than the threshold distance, or The system according to claim 1, for determining that the speed difference of one or more VRUs in each cluster of the VRUs is below a threshold difference, and at least one of the following:

3. In order to apply the one or more clustering metrics to the multiple VRUs, the data processing system: The system according to claim 1, for evaluating the Euclidean distance in position-velocity space between each of the one or more VRUs in each cluster of the VRUs and the centroid of each cluster of the VRUs.

4. In order to apply the one or more clustering metrics to the multiple VRUs, the data processing system: Applying hierarchical clustering (HAC) to multiple initial clusters of a VRU to aggregate at least two initial clusters to obtain one or more clusters of the VRU, or The system according to claim 1, which is for performing at least one of the following: processing input data with a trained machine learning model to obtain output data, wherein the input data includes (i) the locations of the plurality of VRUs, (ii) the velocities of the plurality of VRUs, and (iii) the direction of operation of the plurality of VRUs, and the output data includes one or more clusters of the VRUs.

5. The system according to claim 1, wherein the speed associated with the collective operation of the one or more VRUs of each of the clusters includes the average speed of the one or more VRUs of each of the clusters.

6. Each of the one or more clusters of the VRU is further associated with the velocity spread of one or more VRUs in each cluster of the VRU, and the data processing system predicts the one or more VRU block regions for the time interval, The system according to claim 1, for estimating the change in the geometric shape of each cluster of the VRU over the aforementioned time interval.

7. In order to determine the driving route of the vehicle, the data processing system Determining the speed of the vehicle's operation along the aforementioned driving path, and at least, The minimum distance from the aforementioned driving path to one of the VRUs, or The system according to claim 1, further configured to make a determination based on the number of VRUs in a relevant area around the driving path of the vehicle.

8. The system according to claim 7, wherein the speed of the operation of the vehicle's driving path decreases as the number of VRUs in the relevant region increases.

9. It is a method, Using the vehicle's sensing system, we acquire sensing data related to the driving environment, Using a processing device and based on the sensing data, detect multiple vulnerable road users (VRUs) within the driving environment, Applying one or more clustering metrics to the multiple VRUs in the operating environment to form one or more clusters of VRUs, wherein each of the one or more clusters of VRUs is The geometric shape surrounding one or more VRUs in each cluster of VRUs, The speed associated with the collective operation of one or more VRUs in each cluster of the VRUs, and the associated status. Using the geometric shape and velocity associated with one or more clusters of the VRU, predict one or more VRU block regions over a time interval. A method comprising determining the driving path of the vehicle in the driving environment with respect to the aforementioned time interval and taking into consideration the one or more VRU block regions.

10. Applying the aforementioned one or more clustering metrics to the aforementioned multiple VRUs is It is determined that the distance from each of the one or more VRUs to the reference point position of each cluster of the VRUs is less than the threshold distance, The method according to claim 9, comprising at least one of the following: determining that the difference in speed of one or more VRUs in each cluster of the VRUs is below a threshold difference.

11. Applying the aforementioned one or more clustering metrics to the aforementioned multiple VRUs is The method according to claim 9, comprising evaluating the Euclidean distance in position-velocity space between each of the one or more VRUs in each cluster of the VRUs and the centroid of each cluster of the VRUs.

12. Applying the aforementioned one or more clustering metrics to the aforementioned multiple VRUs is Applying hierarchical clustering (HAC) to multiple initial clusters of a VRU to aggregate at least two initial clusters to obtain one or more clusters of the VRU, or The method according to claim 9, comprising processing input data with a trained machine learning model to obtain output data, wherein the input data includes (i) the locations of the plurality of VRUs, (ii) the velocities of the plurality of VRUs, and (iii) the direction of operation of the plurality of VRUs, and the output data includes one or more clusters of the VRUs.

13. The method according to claim 9, wherein the rate associated with the collective operation of the one or more VRUs of each of the clusters includes the average rate of the one or more VRUs of each of the clusters.

14. Each of the one or more clusters of the VRU is further associated with the velocity diffusion of one or more VRUs in each cluster of the VRU, and the prediction of the one or more VRU block regions over the time interval is made. The method according to claim 9, comprising estimating the change in the geometric shape of each cluster of the VRU over the aforementioned time interval.

15. Determining the aforementioned driving route of the aforementioned vehicle is Determining the speed of the vehicle's operation along the aforementioned driving path, at least The minimum distance from the aforementioned driving path to one of the VRUs, or The method according to claim 9, comprising determining based on the number of VRUs in a relevant area around the driving path of the vehicle.

16. The method according to claim 15, wherein the speed of the operation of the vehicle's driving path decreases as the number of VRUs in the relevant region increases.

17. It is an autonomous vehicle, A sensing system configured to acquire sensing data associated with the driving environment, A data processing system, Applying one or more clustering metrics to multiple vulnerable road users (VRUs) within the aforementioned driving environment to form one or more clusters of VRUs, wherein each of the one or more clusters of VRUs is The geometric shape surrounding one or more VRUs in each cluster of VRUs, The speed associated with the collective operation of one or more VRUs in each cluster of the VRUs, and the associated status. Using the geometric shape and velocity associated with one or more clusters of the VRU, predict one or more VRU block regions over a time interval. A data processing system configured to determine the driving path of the autonomous vehicle within the driving environment, taking into account the aforementioned time interval and the one or more VRU block regions, A vehicle control system, An autonomous vehicle comprising: a vehicle control system configured to guide the autonomous vehicle along the determined driving path.

18. In order to apply the one or more clustering metrics to the one or more VRUs of each cluster of the VRU, the data processing system: The distance from each of the one or more VRUs to the reference point position of each cluster of the VRUs is less than the threshold distance, or The autonomous vehicle according to claim 17, which determines at least one of the following: the speed difference of one or more VRUs in each cluster of the VRUs is below a threshold difference.

19. In order to apply the one or more clustering metrics to the multiple VRUs, the data processing system: The autonomous vehicle according to claim 17, for evaluating the Euclidean distance in position-velocity space between each of the one or more VRUs in each cluster of the VRUs and the centroid of each cluster of the VRUs.

20. The aforementioned data processing system Determining the speed of the autonomous vehicle's operation along the driving path, and at least, The minimum distance from the aforementioned driving path to one of the VRUs, or The autonomous vehicle according to claim 17, further configured to make a determination based on the number of VRUs in a relevant area around the driving path of the autonomous vehicle.

Images