Method and system for traffic light marking via motion inference

Abstract

A method for offline perceptual motion inference is provided that may include obtaining map data indicative of an environment and obtaining data associated with at least one agent. The method may include determining a trajectory for the agent and matching the trajectory of the agent to a lane connection line. The method may also include determining traffic light parameters. A system and a computer program product are also provided.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the priority benefit of U.S. Application No. 18 / 146,364, filed on December 24, 2022, and U.S. Provisional Application No. 63 / 416,248, filed on October 14, 2022, both entitled "METHODS AND SYSTEMS FOR PERCEPTION MOTION INFERENCE", the entire disclosures of which are incorporated herein by reference. Background Art

[0003] For an autonomous vehicle, it is crucial to be able to determine the status of traffic lights in its environment during operation (e.g., whether the traffic light is green or red). Radio - and image - based analyses are currently being used to detect the status of traffic lights. However, radio communications such as dedicated short - range communications (DSRC) are not available for all traffic lights. Additionally, image - based detection only covers visible traffic lights and requires specific hardware to use. It may be difficult to determine the status of traffic lights using only these technologies. Brief Description of the Drawings

[0004] Figure 1 is an example environment of a vehicle that can implement one or more components of an autonomous system;

[0005] Figure 2 is a diagram of one or more example systems of a vehicle including an autonomous system;

[0006] Figure 3 is Figure 1 and Figure 2 a diagram of one or more example devices and / or components of one or more example systems;

[0007] Figure 4 is a diagram of certain components of an example autonomous system;

[0008] Figures 5A to 5B is a diagram of an example implementation of a process for offline perception motion inference;

[0009] Figure 6 is a diagram of an example implementation of a process for offline perception motion inference;

[0010] Figure 7 is a diagram of an example implementation of a process for offline perception motion inference; and

[0011] Figure 8 is a flowchart of an example process for offline perception motion inference. Detailed Description

[0012] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the embodiments described herein may be practiced without these specific details. In some instances, well-known structures and devices are illustrated in block diagram form in order to avoid unnecessarily obscuring aspects of the present disclosure.

[0013] In the drawings, for ease of description, a specific arrangement or order of illustrative elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements, etc.) is illustrated. However, those skilled in the art will understand that unless explicitly described, the specific order or arrangement of the illustrative elements in the drawings is not intended to imply a required order or sequence of processing, or separation of processing. Further, unless explicitly described, the inclusion of illustrative elements in the drawings is not intended to imply that such elements are required in all embodiments, nor that the features represented by such elements cannot be included in some embodiments or combined with other elements in some embodiments.

[0014] Moreover, in the drawings, connecting elements (such as solid lines, dashed lines, or arrows, etc.) are used to illustrate connections, relationships, or associations between or among two or more other illustrative elements. The absence of any such connecting element is not intended to imply that no connection, relationship, or association can exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the present disclosure. Further, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of a signal, data, or instruction (e.g., "software instruction"), those skilled in the art will understand that such an element may represent one or more than one signal path (e.g., a bus) that may be required to affect the communication.

[0015] Although terms such as "first", "second", and / or "third", etc. are used to describe various elements, these elements should not be limited by these terms. The terms "first", "second", and / or "third" are only used to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0016] The terms used in the description of the various embodiments described herein are included only for the purpose of describing particular embodiments and are not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms "a", "an", and "the" are also intended to include the plural forms and may be used interchangeably with "one or more than one" or "at least one", unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms "comprises", "comprising", "includes", and / or "having" are used in this specification, it specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0017] As used herein, the terms "communicate" and "communicating" refer to at least one of receiving, receipt, transmission, conveyance, and / or provision of information (or information represented by, for example, data, signals, messages, instructions, and / or commands, etc.). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof, etc.) that is to communicate with another unit, this means that the unit can directly or indirectly receive information from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is inherently wired and / or wireless. Additionally, two units can communicate with each other even if the information transmitted between the first unit and the second unit is modified, processed, relayed, and / or routed. For example, even if the first unit passively receives information and does not actively transmit information to the second unit, the first unit can communicate with the second unit. As another example, if at least one intermediate unit (e.g., a third unit located between the first unit and the second unit) processes the information received from the first unit and transmits the processed information to the second unit, the first unit can communicate with the second unit. In some embodiments, a message can refer to a network packet (e.g., a data packet, etc.) that includes data.

[0018] As used herein, depending on the context, the term "if" is optionally interpreted to mean "when", "at the time of", "in response to determining that", and / or "in response to detecting", etc. Similarly, depending on the context, the phrase "if it has been determined" or "if [the stated condition or event] is detected" is optionally interpreted to mean "at the time of determining...", "in response to determining that", or "at the time of detecting [the stated condition or event]" and / or "in response to detecting [the stated condition or event]", etc. Further, as used herein, the terms "has", "having", or "owns", etc. are intended to be open-ended terms. Additionally, unless otherwise explicitly stated, the phrase "based on" is intended to mean "at least partially based on".

[0019] "At least one" and "one or more than one" include the function being performed by one element, the function being performed by more than one element, e.g., in a distributed manner, the function being performed by one element for several functions, the function being performed by several elements for several functions, or any combination of the above.

[0020] Some embodiments of the present disclosure are described herein in connection with thresholds. As described herein, meeting (such as conforming to, etc.) a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, and / or equal to the threshold, etc.

[0021] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to those of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0022] General Overview

[0023] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement perception motion inference such as for the determination of traffic light conditions (e.g., states). For example, the present disclosure relates to systems, methods, and computer program products for providing traffic light inference based on the determined trajectories of agents. In certain examples, the present disclosure includes obtaining data indicative of the environment around an autonomous vehicle and data associated with agents in the environment. The present disclosure also includes: determining the trajectories of the agents and then matching the trajectories with known lane connection lines to determine traffic light conditions.

[0024] Implementations of the systems, methods, and computer program products described herein provide for the determination of traffic light status, such as via sensorimotor inference. Some embodiments of the present disclosure allow for online sensorimotor inference based on sensor data generated for the operation of an autonomous vehicle. Some embodiments of the present disclosure allow for offline sensorimotor inference, which can utilize a powerful sensing system to train a model to improve inference and is not limited by in-vehicle hardware constraints. Some advantages of these techniques include improved determination of traffic signals, especially for traffic lights that are not visible and traffic lights lacking radio communication systems. This can advantageously optimize the operation of an autonomous vehicle by eliminating rate bottlenecks in repeated matching of motion trajectories and lane connection lines. Additionally, these techniques can allow for improved traffic light labeling by inferring traffic lights from the past and future trajectories of surrounding agents. Further, past and future sensor data and / or prediction data can be used for post-analysis to improve tracking performance, especially in the first few frames of data where tracking confidence may be low. Such sensing techniques can be used for inference of both red and green lights. Additionally, the present disclosure can be used to automatically label the status of traffic lights in a scene by inferring the status of traffic lights in the scene from the past and future trajectories of surrounding agents and / or actors. Further, using the past and future trajectories of agents, the present disclosure allows for the determination of traffic light status for all lanes in a given intersection, not just a single lane.

[0025] Reference is now made to Figure 1 , illustrative example environment 100, in which vehicles including an autonomous system and vehicles not including an autonomous system operate. As illustrated, environment 100 includes vehicles 102a - 102n, objects 104a - 104n, routes 106a - 106n, area 108, vehicle-to-infrastructure (V2I) devices 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a - 102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via a wired connection, a wireless connection, or a combination of wired and wireless connections (e.g., establishing a connection for communication, etc.). In some embodiments, objects 104a - 104n are interconnected with at least one of vehicles 102a - 102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0026] Vehicles 102a - 102n (individually referred to as vehicle 102 and collectively referred to as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, vehicle 102 is the same as or similar to vehicle 200 described herein (see Figure 2 ). In some embodiments, vehicles 200 in a set of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicle 102 travels along corresponding routes 106a - 106n (individually referred to as route 106 and collectively referred to as routes 106). In some embodiments, one or more than one vehicle 102 includes an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0027] Objects 104a - 104n (individually referred to as object 104 and collectively referred to as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.), etc. Each object 104 is stationary (e.g., located at a fixed location for a period of time) or moving (e.g., having a speed and associated with at least one trajectory). In some embodiments, object 104 is associated with a corresponding location in area 108.

[0028] Routes 106a - 106n (individually referred to as Route 106 and collectively as Routes 106) are each associated with (e.g., define) a series of actions (also referred to as a trajectory) along which an AV can be navigated. Each Route 106 starts from an initial state (e.g., a state corresponding to a first spatio - temporal location and / or speed, etc.) and terminates at a final target state (e.g., a state corresponding to a second spatio - temporal location different from the first) or a target zone (e.g., a subspace of acceptable states (e.g., termination states)). In some embodiments, the first state includes a location where one or more individuals will board the AV, and the second state or zone includes one or more locations where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple sequences of acceptable states (e.g., multiple sequences of spatio - temporal locations), which are associated with (e.g., define) multiple trajectories. In an example, Route 106 includes only high - level actions or imprecise state locations, such as a series of connected roads indicating a direction change at a roadway intersection, etc. Additionally or alternatively, Route 106 can include more precise actions or states, such as, for example, a specific target lane or precise location within a lane area and a target speed at those locations. In an example, Route 106 includes multiple precise state sequences along at least one high - level action with a finite look - ahead horizon to reach an intermediate target, where the combination of consecutive iterations of the finite - horizon state sequences cumulatively corresponds to multiple trajectories that together form a high - level route terminating at the final target state or zone.

[0029] Region 108 includes a physical area (e.g., a geographical area) in which vehicle 102 can be navigated. In an example, Region 108 includes at least one state (e.g., a country, a province, an individual state among multiple states included in a country, etc.), at least a part of a state, at least one city, at least a part of a city, etc. In some embodiments, Region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, an interstate highway, a parkway, an urban street, etc. Additionally or alternatively, in some examples, Region 108 includes at least one unnamed road, such as a lane, a section of a parking lot, a section of an open space and / or undeveloped area, a dirt road, etc. In some embodiments, a road includes at least one lane (e.g., the part of the road through which vehicle 102 can pass). In an example, a road includes at least one lane associated with (e.g., identified based on) at least one lane marking line.

[0030] A vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to-infrastructure or vehicle-to-everything (V2X) device) includes at least one device configured to communicate with a vehicle 102 and / or a V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, a remote AV system 114, a queue management system 116, and / or the V2I system 118 via a network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), lane markings, streetlights, a parking meter, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the queue management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0031] The network 112 includes one or more wired and / or wireless networks. In an example, the network 112 includes a cellular network (e.g., a Long-Term Evolution (LTE) network, a third-generation (3G) network, a fourth-generation (4G) network, a fifth-generation (5G) network, a Code Division Multiple Access (CDMA) network, etc.), a Public Land Mobile Network (PLMN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a telephone network (e.g., a Public Switched Telephone Network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of some or all of these networks.

[0032] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the network 112, the queue management system 116, and / or the V2I system 118 via the network 112. In an example, the remote AV system 114 includes a server, a server group, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including an autonomous system, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0033] The queue management system 116 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the V2I system 118. In an example, the queue management system 116 includes a server, a server group, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ridesharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems), etc.).

[0034] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipal authority or a private institution (e.g., a private institution for maintaining the V2I device 110, etc.).

[0035] In some embodiments, as Figure 8 illustrated, the device 300 is configured to execute software instructions for one or more steps of the disclosed method.

[0036] Provide Figure 1 The number and arrangement of the illustrated elements are provided as an example. Compared with the Figure 1 illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements with a different arrangement. Additionally or alternatively, at least one element of the environment 100 may perform one or more functions described as being performed by Figure 1 at least one different element. Additionally or alternatively, at least one set of elements of the environment 100 may perform one or more functions described as being performed by at least one different set of elements of the environment 100.

[0037] Now referring to Figure 2 , the vehicle 200 (which may be the same as or similar to the Figure 1 vehicle 102) includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208, or is associated with the autonomous system 202, the powertrain control system 204, the steering control system 206, and the braking system 208. In some embodiments, the vehicle 200 is the same as the vehicle 102 (see Figure 1)Same or similar. In some embodiments, the autonomous system 202 is configured to endow the vehicle 200 with autonomous driving capabilities (e.g., implement at least one of the following driving automation or maneuver-based functions, features, and / or devices, etc., where the at least one driving automation or maneuver-based function, feature, and / or device enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon reliance on human intervention, such as level 5 ADS-operated vehicles, etc.), highly autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in certain situations, such as level 4 ADS-operated vehicles, etc.), and / or conditionally autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in limited situations, such as level 3 ADS-operated vehicles, etc.), etc.). In one embodiment, the autonomous system 202 includes the operational or tactical functionality required to operate the vehicle 200 in road traffic and continuously perform a part or all of the dynamic driving task (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) that includes driver support features. The autonomous system 202 supports various levels of driving automation ranging from no driving automation (e.g., level 0) to full driving automation (e.g., level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference can be made to SAE International standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire content of which is incorporated by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ridesharing company.

[0038] The autonomous system 202 includes a sensor suite that includes one or more devices such as a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, and a microphone 202d. In some embodiments, the autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance the vehicle 200 has traveled, etc.). In some embodiments, the autonomous system 202 uses one or more devices included in the autonomous system 202 to generate data associated with the environment 100 described herein. The data generated by one or more devices of the autonomous system 202 may be used by one or more systems described herein to observe the environment (e.g., environment 100) in which the vehicle 200 is located. In some embodiments, the autonomous system 202 includes a communication device 202e, an autonomous vehicle computing 202f, a drive-by-wire (DBW) system 202h, and a safety controller 202g.

[0039] The camera 202a includes at least one device configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., a bus 302 that is the same as or similar to Figure 3 the bus). The camera 202a includes at least one camera (e.g., a digital camera using an optical sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images including physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, the camera 202a generates camera data as output. In some examples, the camera 202a generates camera data including image data associated with the image. In such an example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or an image timestamp, etc.). In such an example, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a includes a plurality of independent cameras configured on (e.g., positioned on) the vehicle for capturing images for the purpose of stereovision (stereo vision). In some examples, the camera 202a includes generating image data and transmitting the image data to the autonomous vehicle computing 202f and / or a queue management system (e.g., to Figure 1A plurality of cameras of the queue management system 116 that are the same as or similar to the queue management system). In such an example, the autonomous vehicle computing 202f determines the depth to one or more objects in the fields of view of at least two of the plurality of cameras based on image data from at least two cameras. In some embodiments, the camera 202a is configured to capture images of objects within a distance relative to the camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Thus, the camera 202a includes features such as sensors and lenses that are optimized for sensing objects at one or more distances relative to the camera 202a.

[0040] In an embodiment, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects that provide visual navigation information. In some embodiments, the camera 202a generates traffic light data associated with one or more images. In some examples, the camera 202a generates TLD (Traffic Light Detection) data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data is different from other systems incorporating cameras described herein in that: the camera 202a may include one or more cameras having a wide field of view (e.g., a wide-angle lens, a fish-eye lens, and / or a lens having a viewing angle of about 120 degrees or greater, etc.) to generate images related to as many physical objects as possible.

[0041] The Light Detection and Ranging (LiDAR) sensor 202b includes being configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with Figure 3at least one device that communicates via a bus (e.g., a bus identical or similar to bus 302). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b further includes at least one light detector that detects the light after the light emitted from the light emitter encounters a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing the objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface), etc. In such examples, the image is used to determine the boundary of the physical object in the field of view of the LiDAR sensor 202b.

[0042] A radio detection and ranging (Radar) sensor 202c includes at least one device configured to communicate with a communication device 202e, an autonomous vehicle computer 202f, and / or a safety controller 202g via a bus (e.g., a bus identical or similar to Figure 3 bus 302). The Radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the Radar sensor 202c encounter a physical object and are reflected back to the Radar sensor 202c. In some embodiments, the radio waves emitted by the Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the Radar sensor 202c generates a signal representing the objects included in the field of view of the Radar sensor 202c. For example, at least one data processing system associated with the Radar sensor 202c generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface), etc. In some examples, the image is used to determine the boundary of the physical object in the field of view of the Radar sensor 202c.

[0043] The microphone 202d includes at least one device configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., a bus identical or similar to the Figure 3 bus 302). The microphone 202d includes one or more microphones (e.g., an array microphone and / or an external microphone, etc.) that capture an audio signal and generate data associated with (e.g., representing) the audio signal. In some examples, the microphone 202d includes a transducer device and / or a similar device. In some embodiments, one or more of the systems described herein can receive the data generated by the microphone 202d and determine the position (e.g., distance, etc.) of an object relative to the vehicle 200 based on the audio signal associated with the data.

[0044] The communication device 202e includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. For example, the communication device 202e can include a device identical or similar to the Figure 3 communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0045] The autonomous vehicle computing 202f includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the communication device 202e, the safety controller 202g, and / or the DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as a client device, a mobile device (e.g., a cellular phone and / or a tablet, etc.), and / or a server (e.g., a computing device including one or more central processing units and / or graphics processing units, etc.). In some embodiments, the autonomous vehicle computing 202f is identical or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to the Figure 1 remote AV system 114), a queue management system (e.g., a queue management system identical or similar to the Figure 1 queue management system 116), a V2I device (e.g., a V2I device identical or similar to the Figure 1 V2I device 110), and / or a V2I system (e.g., a V2I system identical or similar to the Figure 1communicate with a V2I system 118 that is the same as or similar to the V2I system).

[0046] The safety controller 202g includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the communication device 202e, the autonomous vehicle computing 202f, and / or the DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (such as an electrical controller and / or an electromechanical controller, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (such as the powertrain control system 204, the steering control system 206, and / or the braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing 202f.

[0047] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (such as an electrical controller and / or an electromechanical controller, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (such as the powertrain control system 204, the steering control system 206, and / or the braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (such as turn signals, headlights, door locks, and / or windshield wipers, etc.).

[0048] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle movements (such as starting to move forward, stopping moving forward, starting to move backward, stopping moving backward, accelerating in a certain direction, decelerating in a certain direction, etc.), or perform lateral vehicle movements (such as making a left turn and / or making a right turn, etc.). In an example, the powertrain control system 204 increases, maintains the same, or decreases the energy (such as fuel and / or electricity, etc.) provided to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate. In other words, the steering control system 206 causes the activities required to regulate the y-axis component of the vehicle movement.

[0049] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 rotates two front wheels and / or two rear wheels of the vehicle 200 left or right to turn the vehicle 200 left or right.

[0050] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate the vehicle 200 and / or keep it stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on corresponding rotors of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0051] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or condition of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), a wheel speed sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor, etc. Although the braking system 208 is illustrated as being located Figure 2 proximal to the vehicle 200 in, the braking system 208 can be located anywhere in the vehicle 200.

[0052] Now refer to Figure 3, a schematic diagram of an exemplary device 300. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of a system of vehicle 102); at least one device of remote AV system 114, queue management system 116, V2I system 118; and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of a system of vehicle 102 (such as at least one device of remote AV system 114, queue management system 116, and V2I system 118, etc.)), and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112) include at least one device 300 and / or at least one component of device 300. As Figure 3 shown, device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0053] Bus 302 includes components that permit communication between components of device 300. In some cases, processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), etc.). Memory 306 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic and / or static storage device that stores data and / or instructions for use by processor 304 (e.g., flash memory, magnetic memory, and / or optical memory, etc.).

[0054] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk, etc.), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cassette tape, a magnetic tape, a CD-ROM, RAM, PROM, EPROM, FLASH-EPROM, NV-RAM, and / or another type of computer readable medium, and corresponding drives.

[0055] The input interface 310 includes components of the enabling device 300 that receive information via a user input (e.g., a touchscreen display, keyboard, keypad, mouse, button, switch, microphone, and / or camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator, etc.). The output interface 312 includes components for providing output information from the device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs), etc.).

[0056] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receiver and transmitter, etc.) that enable the device 300 to communicate with other devices via a wired connection, wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the device 300 to receive information from another device and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, optical interface, coaxial interface, infrared interface, radio frequency (RF) interface, Universal Serial Bus (USB) interface, interface, and / or a cellular network interface, etc.

[0057] In some embodiments, the device 300 performs one or more of the processes described herein. The device 300 performs these processes based on software instructions executed by a processor 304 that are stored in a computer-readable medium such as a memory 306 and / or a storage component 308. A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes a storage space located within a single physical storage device or a storage space distributed across multiple physical storage devices.

[0058] In some embodiments, software instructions are read into the memory 306 and / or the storage component 308 from another computer-readable medium or from another device via the communication interface 314. When executed, the software instructions stored in the memory 306 and / or the storage component 308 cause the processor 304 to perform one or more of the processes described herein. Additionally or alternatively, hardwired circuitry is used instead of or in combination with software instructions to perform one or more of the processes described herein. Thus, unless otherwise expressly stated, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0059] Memory 306 and / or storage component 308 includes a data store or at least one data structure (e.g., a database, etc.). Device 300 is capable of receiving information from the data store or at least one data structure in memory 306 or storage component 308, storing information in the data store or at least one data structure, communicating information to the data store or at least one data structure, or searching for information stored in the data store or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0060] In some embodiments, device 300 is configured to execute software instructions stored in memory 306 and / or the memory of another device (e.g., another device that is the same as or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in memory 306 and / or the memory of another device, which, when executed by processor 304 and / or the processor of another device (e.g., another device that is the same as or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes described herein. In some embodiments, the module is implemented in software, firmware, and / or hardware, etc.

[0061] Provided Figure 3 The number and arrangement of the illustrated components are provided as an example. In some embodiments, compared to Figure 3 the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another set of components of device 300.

[0062] Now refer Figure 4, an example block diagram illustrating autonomous vehicle computing 400 (sometimes referred to as the "AV stack"). As illustrated, autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as the perception module), a planning system 404 (sometimes referred to as the planning module), a localization system 406 (sometimes referred to as the localization module), a control system 408 (sometimes referred to as the control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in one or more separate systems located in the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), and / or a field programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., an autonomous vehicle system that is the same as or similar to the remote AV system 114, a queue management system that is the same as or similar to the queue management system 116, and / or a V2I system that is the same as or similar to the V2I system 118, etc.).

[0063] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object), and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a), the image being associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies the at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of the physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0064] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the perception system 402 (e.g., the data associated with the classification of the physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to the tactical functions required to operate the vehicle 102 in road traffic. Tactical efforts involve maneuvering the vehicle in traffic during the journey, which includes but is not limited to deciding whether and when to overtake another vehicle, change lanes, or select an appropriate speed, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with the updated position of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0065] In some embodiments, the positioning system 406 receives data associated with (e.g., representing) the location of a vehicle (e.g., vehicle 102) in an area. In some examples, the positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In certain examples, the positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the positioning system 406 generates a combined point cloud based on the respective point clouds. In these examples, the positioning system 406 compares the at least one point cloud or the combined point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in the database 410. Then, based on the positioning system 406 comparing the at least one point cloud or the combined point cloud with the map, the positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated prior to the navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of roadway geometry, a map describing the connectivity of the road network, a map describing the physical properties of roadways (such as traffic speed, traffic flow, the number of vehicle and bicycle traffic lanes, lane width, lane traffic direction, or the type and location of lane markings, or a combination thereof, etc.), and a map describing the spatial location of road features (such as crosswalks, traffic signs, or various other driving signals, etc.). In some embodiments, the map is generated in real time based on the data received by the perception system.

[0066] In another example, the positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, the positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and the positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, the positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, the positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the positioning system 406 determining the location of the vehicle, the positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0067] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to cause the powertrain control system (e.g., the DBW system 202h and / or the powertrain control system 204, etc.), the steering control system (e.g., the steering control system 206), and / or the braking system (e.g., the braking system 208) to operate. For example, the control system 408 is configured to perform operating functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes activities required to regulate the y-axis component of the vehicle motion. Longitudinal vehicle motion control causes activities required to regulate the x-axis component of the vehicle motion. In an example, in the case where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.) to change states.

[0068] In some embodiments, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model alone or in combination with one or more of the above systems. In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.).

[0069] The database 410 stores data transmitted to, received from, and / or updated by the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408. In some examples, the database 410 includes a storage component for storing data and / or software related to operations and using the autonomous vehicle computing 400 of at least one system (e.g., related to Figure 3the same or similar storage components as the storage component 308). In some embodiments, the database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, the database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., nations), etc. In such examples, a vehicle (e.g., a vehicle the same or similar to the vehicle 102 and / or the vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, back roads, and / or off-road roads, etc.), and cause at least one LiDAR sensor (e.g., a LiDAR sensor the same or similar to the LiDAR sensor 202b) to generate data associated with an image representing the objects included in the field of view of the at least one LiDAR sensor.

[0070] In some embodiments, the database 410 can be implemented across multiple devices. In some examples, the database 410 is included in a vehicle (e.g., a vehicle the same or similar to the vehicle 102 and / or the vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system the same or similar to the remote AV system 114), a queue management system (e.g., a queue management system the same or similar to Figure 1 the queue management system 116), and / or a V2I system (e.g., a V2I system the same or similar to Figure 1 the V2I system 118), etc.

[0071] Now refer to Figures 5A to 5B , a diagram illustrating the system 500 according to the present disclosure. In some embodiments, the system 500 is a system for determining traffic light parameters (e.g., status) based on the movement of one or more agents (referred to herein as sensed motion inference). In some embodiments, the system 500 is a system for determining traffic light parameters (e.g., status) during an offline (after-driving) analysis of a driving log or during an online (real-time) operation of an autonomous vehicle. In some embodiments, the system 500 is connected to and / or incorporated into a vehicle 550 (e.g., an autonomous vehicle the same or similar to Figure 2 the vehicle 200). In one or more embodiments or examples, the system 500 is associated with an autonomous vehicle (e.g., such as Figure 2 the autonomous system 202 illustrated in Figure 3 the device 300), an AV system, AV computing (such as Figure 2 the AV computing 202f and / or Figure 4 the AV computing 400), a remote AV system (such as Figure 1a remote AV system 114, etc.), a queue management system (such as Figure 1 the queue management system 116, etc.) and a V2I system (such as Figure 1 the V2I system 118, etc.) and / or is part of it. In one or more examples, the system 500 can be used to operate an autonomous vehicle.

[0072] In one or more embodiments or examples, the system 500 communicates with and / or includes one or more of the following: a device (such as Figure 3 the device 300, etc.), a positioning system (such as Figure 4 the positioning system 406, etc.), a planning system ( Figure 4 the planning system 404), a sensing system (such as Figure 5A the sensing system 504 and / or Figure 4 the sensing system 402, etc.), a database (such as Figure 4 the database 410 and / or Figure 5A the database 508, etc.) and a control system (such as Figure 4 the control system 408, etc.).

[0073] In one or more embodiments or examples, the system 500 includes at least one processor and at least one memory, and instructions are stored on the at least one memory, and when the instructions are executed by the at least one processor, the at least one processor is caused to operate. In one or more embodiments or examples, the operations include: obtaining map data 505 indicating the environment in which the autonomous vehicle 550 can operate. In one or more embodiments or examples, the map data 505 includes a plurality of lane connection lines associated with intersections in the environment. In one or more embodiments or examples, the operations include: obtaining data 507 associated with at least one agent in the environment, the data 507 indicating the agent relative to the environment. In one or more embodiments or examples, the operations include: determining a trajectory 511 of at least one agent relative to the environment based on the data 507 associated with the at least one agent. In one or more embodiments or examples, the operations include: matching the trajectory 511 of at least one agent with one of the plurality of lane connection lines to determine a matching trajectory 513. In one or more embodiments or examples, the operations include: determining traffic light parameters indicating the status of traffic lights at intersections based on the matching trajectory 513.

[0074] In other words, the system 500 obtains environmental data and data indicating any agents within the environment. In some examples, the system 500 uses the trajectories of agents (including the autonomous vehicle 550 of which the system 500 is a part) (such as Figure 5Bto determine the status of the traffic light (e.g., whether the traffic light is green or red) based on the trajectories 511, etc. of the agent. For example, the map data 505 includes a plurality of lane connection lines that illustrate potential paths through an intersection. By matching the trajectory of the agent with the lane connection lines, the system 500 can infer the traffic light status. As an example, if the system 500 determines that the agent is taking the trajectory 511 through the intersection, the system 500 (e.g., by inference) determines that the traffic light status for the lane through which the agent is traveling is green. As another example, if all agents have trajectories 511 that are being stopped, the system 500 (e.g., by inference) determines that the traffic light status for the lane through which the agent is traveling is red. The system 500 can make these determinations online by using the sensor data 509 and / or offline to train and improve the determinations of the system 500. In the case of offline use, the system 500 can develop an inference dataset that can later be used to train one or more machine learning models based on the determinations made by the system 500.

[0075] In one or more examples or embodiments, the system 500 is used to train and / or validate downstream planning and prediction models conditioned on traffic light status. The disclosed system 500 allows for realistic simulation of traffic from all directions.

[0076] In one or more embodiments or examples, the system 500 obtains the map data 505 from a database 508 (such as Figure 4 the database 410, etc.). The database 508 can be within the system 500 or from a separate server communicating with the system 500. In some examples, the map data 505 is a ground truth map of the environment in which the autonomous vehicle 550 can operate. In other words, the map data 505 is the area of the environment in which the autonomous vehicle 550 is currently operating. The map data 505 is obtained from an offline source and / or from an online source. In one or more examples or embodiments, the map data 505 indicates drivable areas of the environment, such as roads, crosswalks, lanes, parking lots, etc.

[0077] In one or more embodiments or examples, the map data 505 includes a plurality of lane connection lines. Figure 7Examples of lane connectors are illustrated. In particular, the map data 505 may include lane connectors at one or more intersections indicated by the map data 505. In some examples, the lane connectors indicate one or more potential paths that an agent and / or an autonomous vehicle 550 may take at an intersection. As an example, the lane connectors indicate that an agent may take a stop, a right turn, a left turn, a straight path, etc. through the intersection. The lane connectors may indicate any maneuver within the intersection (e.g., a U-turn). In one or more examples or embodiments, the system 500 is configured to filter out lane connectors that do not correspond to the nearest intersection to the autonomous vehicle 550.

[0078] In one or more examples or embodiments, the system 500 obtains information indicating an agent in the environment, such as via data 507 associated with at least one agent. An agent may be considered any object in the environment that is capable of moving dynamically. Examples of agents include pedestrians, vehicles, and bicycles. In some examples, the data indicates the agent relative to the environment. In one or more examples or embodiments, the data includes one or more of the following: the position, orientation, speed, and direction of the agent. In some examples, the system 500 interprets the autonomous vehicle 550 itself as an agent in the environment. For example, the system 500 uses positioning data from a positioning system (such as Figure 4 the positioning system 406, etc.) to determine data associated with the autonomous vehicle 550. In an example, the system 500 determines other agents in the environment based on perception data (such as sensor data 509 generated from sensors 510, etc.) from a perception system (such as Figure 4 the perception system 402, etc.). In some examples, the system 500 obtains data 507 associated with at least one agent for all lanes and all directions at a particular intersection.

[0079] In one or more examples or embodiments, the system 500 obtains data 507 associated with at least one agent from a database 508 (such as Figure 4 the database 410, etc.). Thus, the system 500 may not obtain data 507 associated with at least one agent from the sensors 510. For example, the data 507 associated with at least one agent indicates the past routes and / or trajectories taken by the agent at an intersection (e.g., recorded intersection data).

[0080] In one or more embodiments or examples, system 500 determines a trajectory 511 of at least one agent relative to an environment. For example, system 500 includes a trajectory generator 512 (which may be part of the perception system 504 and / or the planning system 506) configured to determine the trajectory 511 of at least one agent. The perception system 504 may obtain map data 505 and data 507 associated with at least one agent. In some examples, such as Figure 5A as shown in

[0081]

[0082] the perception system 504 transmits the map data 505 and the data 507 associated with at least one agent to the planning system 506. In certain examples, the planning system 506 outputs planning data 520, which is received as input by the trajectory generator 512. In one or more examples or embodiments, the planning data 520 indicates a predicted or planned trajectory of the agent. For example, system 500 determines the trajectory 511 of each agent among at least one agent in the environment (e.g., the trajectory generator 512 generates the trajectory 511). For example, the trajectory of at least one agent is determined based on the data 507 associated with at least one agent. In some examples, the trajectory 511 is one or more of the following: a past trajectory, a current trajectory, and a predicted (e.g., future) trajectory. In one or more examples or embodiments, system 500 is configured to match the trajectory 511 with a plurality of lane connection lines of the map data 505. As an example, system 500 includes a system 514 that matches the trajectory 511 with the lane connection line that is the closest to the trajectory 511 in the environment. In one or more examples or embodiments, system 500 uses the directed Hausdorff distance between the trajectory 511 and all of the lane connection lines in the plurality of lane connection lines to find the best-matched (e.g., closest) lane connection line. The directed Hausdorff distance provides a distance between two subsets of a metric space, where the first subset indicates the trajectory 511 and the second subset indicates the lane connection line. In one or more examples or embodiments, system 500 compares the directed Hausdorff distance with a threshold to obtain the best-matched lane connection line. The best-matched lane connection line can be considered as the lane connection line having the closest distance to the trajectory 511. In some examples, system 500 is configured to provide a matching trajectory 513 for at least one agent. The matching trajectory 513 may be equivalent to one of the lane connection lines in the plurality of lane connection lines.In one or more examples or embodiments, system 500 determines traffic light parameters based on matching trajectory 513. For example, system 500 includes a traffic light condition generator 518 configured to determine traffic light parameters based on matching trajectory 513. In some examples, the traffic light parameters are associated with a lane connection line. For example, the traffic light parameters indicate the condition of the traffic light at an intersection. In one or more examples or embodiments, the traffic light parameters indicate that the traffic light is green, the traffic light is red, or an unknown traffic light state. For example, system 500 determines traffic light parameters indicating a red light based on system 500 determining that trajectory 511 matches a trajectory that stops at an intersection (e.g., matching trajectory 513). For example, trajectory 511 that matches a lane connection line passing through an intersection is used to determine traffic light parameters indicating a green light. However, trajectory 511 that matches a lane connection line for a right turn at an intersection may not be sufficient to determine a green or red light because turning would be legal in both cases, so system 500 may determine the traffic parameter as indicating an unknown traffic light state.

[0083] When initializing system 500, in certain examples or embodiments, all traffic light parameters are set to indicate an unknown traffic light state. System 500 may be configured to update traffic light parameters when more data is obtained. In one or more examples or embodiments, the traffic light parameters are stored in association with a lane connection line rather than just storing the traffic light condition. For example, the traffic light parameters may indicate the lane connection line where the vehicle stops at an intersection. In some examples, system 500 updates the association of multiple lane connection lines with the relevant traffic light parameters of an intersection.

[0084] In one or more examples or embodiments, when system 500 is determining traffic light parameters, system 500 uses many different types of criteria. These criteria may vary based on the type of agent (e.g., vehicle, bicycle, pedestrian) and may include different ways of determining how the agent is reacting relative to the intersection. In some examples, these criteria allow filtering out (e.g., removing, discarding) irrelevant agents to analyze the intersection more efficiently. If the agent parameters do not meet the criteria, the agent is filtered out. If the agent parameters do meet the criteria, the agent is not filtered out and processing continues. Filtering out agents can advantageously result in improved performance. For example, system 500 may match only the unfiltered agents to a lane connection line. Non-limiting illustrative examples of such criteria are provided below, each having specific advantages.

[0085] In one or more embodiments or examples, the matching trajectory 511 includes: determining agent parameters of at least one agent based on the trajectory 511. In one or more embodiments or examples, the matching trajectory 511 includes: determining whether the agent parameters meet the criteria. In one or more embodiments or examples, the matching trajectory 511 includes: filtering out at least one agent in response to determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, the matching trajectory 511 includes: not filtering out at least one agent in response to determining that the agent parameters meet the criteria. In one or more embodiments or examples, matching the trajectory 511 to a lane connection line includes: matching the trajectory 511 of at least one unfiltered agent among at least one agent to a lane connection line.

[0086] In one or more embodiments or examples, the agent parameters include a class parameter indicating whether at least one agent is a vehicle, a bicycle, or a pedestrian. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining whether the class parameter indicates that at least one agent is a pedestrian. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining that the agent parameters do not meet the criteria in response to determining that the class parameter indicates a pedestrian. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining that the agent parameters meet the criteria in response to determining that the class parameter does not indicate a pedestrian. In other words, the system 500 is configured to filter out pedestrians who generally do not indicate traffic light conditions, but continue to analyze vehicles such as cars, trucks, etc. The system 500 may be configured to determine that the agent parameters do not meet the criteria in response to determining that the class parameter indicates a bicycle. If the agent is not one of a vehicle, a bicycle, and a pedestrian, the system 500 determines in some examples that the agent parameters do not meet the parameters. In some examples, the agent parameters meet the criteria only if the class parameter indicates that at least one agent is a vehicle.

[0087] In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: determining a distance parameter indicative of a distance between at least one agent and an intersection. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: determining whether the distance parameter is within a first threshold. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: in response to determining that the distance parameter is not within the first threshold, determining that the agent parameters do not satisfy the criteria. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: in response to determining that the distance parameter is within the first threshold, determining that the agent parameters satisfy the criteria. For example, the agent parameters include the distance parameter. In other words, system 500 can be configured to use the distance between the agent and the intersection to determine whether the agent parameters satisfy or do not satisfy the criteria. This can be used to filter out agents that are not close enough to the intersection and thus may not be useful for traffic light signal determination. The first threshold can be a filtering criterion based on a specific distance from the intersection. The first threshold can be regarded as a distance threshold. For example, the first threshold is one or more of 10 feet, 20 feet, 30 feet, 40 feet, and 50 feet. The first threshold can vary according to the speed limit of the road converging with the intersection. For example, due to the required stopping time, an agent on a road with a slower speed may not be affected by the traffic light until it is closer to the intersection compared to an agent on a road with a higher speed limit.

[0088] In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: determining a time parameter indicative of the time at which at least one agent will be within the intersection. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: determining whether the time parameter satisfies a time threshold. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: in response to determining that the time parameter is below the time threshold, determining that the agent parameters do not satisfy the criteria. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: in response to determining that the time parameter satisfies the time threshold, determining that the agent parameters satisfy the criteria. For example, the agent parameters include the time parameter. In other words, system 500 can determine whether the agent parameters satisfy or do not satisfy the criteria by using the time of the agent within the intersection. This can be used to filter out agents that have passed through the intersection or are currently passing through the intersection, which may not provide a correct indication of the traffic light. For example, when the time parameter indicates a duration that is not too short (such as in seconds, etc.), the agent parameters will satisfy the criteria. The time threshold can be expressed in seconds.

[0089] In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: determining the distance between the starting position and the ending position of the trajectory 511. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: determining whether the distance between the starting position and the ending position satisfies a second threshold. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: in response to determining that the distance does not satisfy the second threshold, determining that the agent parameters do not satisfy the criteria. In one or more embodiments or examples, determining whether agent parameters satisfy criteria includes: in response to determining that the distance satisfies the second threshold, determining that the agent parameters satisfy the criteria. In other words, the system 500 can determine whether the agent parameters satisfy or do not satisfy the criteria by using the distance between the starting position and the ending position of the trajectory. For example, the system 500 determines whether the starting and ending points of the trajectory (e.g., past and / or future waypoints within the target intersection) are far enough apart to ensure that the agent moves sufficiently. In other words, for example, when the agent does not move enough, the data is discarded by the agent's system.

[0090] In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining agent position parameters indicating the position of at least one agent relative to an intersection. In other words, the agent parameters include agent position parameters indicating the position of at least one agent relative to an intersection. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining whether the agent position parameters meet the distance from the stop line of the intersection. For example, when the agent position parameters indicate that the agent is at a distance greater than the stop line distance from the intersection, system 500 determines that the stop line distance is met. For example, when the agent position parameters indicate that the agent is at a distance less than or equal to the stop line distance from the intersection, system 500 determines that the stop line distance is not met. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining whether the agent position parameters indicate that at least one agent is in a traffic lane of the intersection. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters do not meet the stop line distance, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters meet the stop line distance, determining that the agent parameters meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters do not indicate that at least one agent is in a traffic lane of the intersection, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters indicate that at least one agent is in a traffic lane of the intersection, determining that the agent parameters meet the criteria. For example, the agent position parameters are based on one or more of the front, rear, and center of the agent. Each lane of the intersection may have a stop line associated therewith. The stop line distance may indicate a stop area. In other words, by way of example, system 500 is configured to determine whether the agent is before or after the stop line distance (e.g., whether the agent position parameters meet the stop line distance). In some examples, when the agent does not meet the stop line distance, system 500 filters out the agent. In one or more examples or embodiments, system 500 further determines whether the agent is in a lane of the intersection. The agent may be passing outside the intersection or may be traveling in a manner such that the agent's trajectory 511 will not intersect the intersection. These agents may be irrelevant to traffic light condition analysis and may be filtered out. It can be understood that the criterion of the stop line distance (e.g., between the start point and the end point of the trajectory) is to reduce false matches of overly short trajectories.

[0091] In some embodiments, system 500 is configured to determine whether a traffic light parameter indicates a green light or a red light. In one or more embodiments or examples, determining the traffic light parameter includes determining whether a matching trajectory 513 of at least one agent is within an intersection. In one or more embodiments or examples, determining the traffic light parameter includes, in response to determining that the matching trajectory 513 is within the intersection, determining the traffic light parameter as indicating a green light. In one or more embodiments or examples, determining the traffic light parameter includes, in response to determining that the matching trajectory 513 is not within the intersection, not determining the traffic light parameter as indicating a green light. In other words, in certain examples, system 500 is configured to determine whether at least one agent is within an intersection at a particular time. In other words, system 500 clips the trajectory to be within the intersection. In some examples, system 500 may use the center of at least one agent to determine whether the at least one agent is within the intersection. In some examples, if at least one agent is within the intersection, system 500 is configured to determine the traffic light parameter as indicating green. For example, system 500 determines that at least one agent is legally passing through the intersection and thus determines (e.g., infers) that the traffic light is green to allow traffic to pass through the intersection.

[0092] In one or more embodiments or examples, determining the traffic light parameter includes determining whether at least one agent is within a stop region of the intersection. In one or more embodiments or examples, determining the traffic light parameter includes, in response to determining that at least one agent is within the stop region, determining whether the speed of the at least one agent meets a speed threshold. In one or more embodiments or examples, determining the traffic light parameter includes, in response to determining that at least one agent is not within the stop region, not determining whether the speed of the at least one agent meets the speed threshold. In one or more embodiments or examples, determining the traffic light parameter includes, in response to determining that the speed meets the speed threshold and the traffic light parameter does not indicate a green light, determining the traffic light parameter as indicating a red light. In one or more embodiments or examples, determining the traffic light parameter includes, in response to determining that the speed does not meet the speed threshold or the traffic light parameter indicates a green light, not determining the traffic light parameter as indicating a red light. Determining the traffic light parameter as indicating a red light may be more complex than determining the traffic light parameter as indicating a green light.

[0093] In one or more examples or embodiments, system 500 is configured to determine whether at least one agent is located within a stop region of an intersection. The stop region may be predefined in system 500. The stop region may be an area defined by a stop line distance, such as an area where an agent intends to dwell before traveling through the intersection. The stop region may indicate a normal stop region at the intersection. If system 500 determines that the agent is not located within the stop region, system 500 may not continue to determine the traffic light parameter as indicating a red light. In some examples, if the agent is within the stop region, system 500 determines whether the speed (e.g., velocity) of at least one agent meets a speed threshold. The speed threshold may be the maximum velocity an agent may have within the stop region. This may eliminate agents within the stop region that are capable of traveling through the intersection normally, which would not be a red light. The speed threshold may be a specific set velocity and / or a specific velocity under the speed limit of the road. If the speed is below the speed threshold, the speed meets the speed threshold. If the speed is at or above the speed threshold, the speed does not meet the speed threshold. In one or more examples or embodiments, if system 500 determines that at least one agent is within the stop region and below the speed threshold, system 500 determines the traffic light parameter as indicating a red light.

[0094] In one or more embodiments or examples, matching trajectory 511 to a lane connection line includes: determining a plurality of matching parameters indicative of the distance between trajectory 511 and each lane connection line among a plurality of lane connection lines. In one or more embodiments or examples, matching trajectory 511 to a lane connection line includes: matching trajectory 511 to the lane connection line among the plurality of lane connection lines having the lowest matching parameter among the plurality of matching parameters. For example, system 500 is configured to compare trajectory 511 of at least one agent to each lane connection line among the plurality of lane connection lines. Since there may be many different lane connection lines (e.g., a plurality of lane connection lines), each trajectory 511 of the agent will include a plurality of different matching parameters. For example, the matching parameter indicates the distance between trajectory 511 and a specific lane connection line. For example, the matching parameter includes the distance between the trajectory and each lane connection line. The directed Hausdorff distance may be used to determine the distance. In one or more examples or embodiments, system 500 determines the total gap (e.g., area) between trajectory 511 and a specific lane connection line for use as a matching parameter. A lower matching parameter may indicate a closer match between trajectory 511 and the lane connection line. In certain examples, system 500 matches (e.g., selects) the lane connection line among the plurality of lane connection lines having the lowest (e.g., minimum, closest match) associated matching parameter.

[0095] In one or more embodiments or examples, matching the trajectory 511 to a lane connection line includes: obtaining movement data indicative of the past movement of at least one agent. In one or more embodiments or examples, matching the trajectory 511 includes: matching the trajectory 511 to at least one lane connection line based on the movement data. The movement data may be stored in a database 508 (such as Figure 4 database 410, etc.). In some examples, the system 500 uses the movement data to account for the driver's slow reaction time. In some examples, the system 500 includes a machine learning model that uses the movement data to improve the prediction of the trajectory 511. In some examples, for offline perception, the system uses actual movement data from past, present, and future instances. In one or more examples or embodiments, the system 500 matches the trajectory 511 to the lane connection line by using the movement data.

[0096] In one or more embodiments or examples, matching the trajectory 511 to a lane connection line includes: determining prediction movement parameters indicative of the predicted movement of at least one agent. In one or more embodiments or examples, matching the trajectory 511 includes: matching the trajectory 511 to at least one lane connection line based on the prediction movement parameters. For example, the system 500 predicts the trajectory 511 in the perception system 504 (such as using Figure 4 perception system 402, etc.) and / or in the planning system 506 (such as using Figure 4 planning system 404, etc.), and / or matches the trajectory 511 to the lane connection line 514 using the predicted movement parameters. In some examples, the system 500 utilizes machine learning techniques or models to determine the prediction movement parameters. The prediction movement parameters may be based on the movement data.

[0097] In one or more embodiments or examples, obtaining data 507 associated with at least one agent includes obtaining data associated with a plurality of agents. In one or more embodiments or examples, determining the trajectory 511 includes determining the trajectory 511 for each agent among the plurality of agents. In one or more embodiments or examples, matching the trajectory 511 to a lane connection line includes: matching the trajectory 511 of each agent among the plurality of agents to a lane connection line to provide a plurality of matching trajectories. In one or more examples or embodiments, the system 500 is configured to obtain data associated with a plurality of agents. In some examples, the system 500 determines the trajectory 511 and then matches the trajectory 511 to a lane connection line for each corresponding agent among the plurality of agents.

[0098] In one or more embodiments or examples, the intersection includes a plurality of traffic lights. In one or more embodiments or examples, the operation includes grouping the trajectories of the agents that are parallel in direction. In one or more embodiments or examples, the operation includes: matching the grouped trajectories with the corresponding lane connection lines. Advantageously, grouping can improve processing efficiency because the system 500 only needs to calculate one in the group. In some examples, the matched grouped trajectories indicate parallel (or substantially parallel) trajectories. Since parallel trajectories can share the same traffic light conditions, the system 500 can group them together to provide the matched grouped trajectories. Then, instead of the system 500 comparing each trajectory 511 with the lane connection line, the system 500 can only have to match each grouped trajectory with the lane connection line. In one or more embodiments or examples, determining the traffic light parameters includes: determining the traffic light parameters of each traffic light among the plurality of traffic lights based on the matched grouped trajectories. In other words, as an example, the system 500 is configured to determine the conditions of many different traffic lights at a given intersection. For example, a four-way intersection can have at least four different traffic lights, and may have more traffic lights if restricted left turns and / or right turns are used.

[0099] In one or more embodiments or examples, determining the traffic light parameters includes: determining the current traffic light parameters based on the previously determined traffic light parameters. For example, the system 500 uses post-processing backfill to account for the driver's slow reaction time. If it is advantageous for the system 500 to avoid having the traffic light parameters be "unknown", then in some examples, the system 500 is configured to determine (e.g., estimate) the current traffic light parameters based on the previous traffic light parameters.

[0100] In one or more embodiments or examples, the operation further includes obtaining sensor data 509 indicative of the environment. In one or more embodiments or examples, obtaining map data 505 includes: obtaining map data 505 based on sensor data 509. In one or more embodiments or examples, obtaining data 507 associated with at least one agent includes: obtaining data 507 associated with at least one agent based on sensor data 509. For example, the system 500 is configured to act on an autonomous vehicle 550 online. In one or more examples or embodiments, the system 500 obtains sensor data 509 via a perception system 504 (such as Figure 4 the perception system 402, etc.). For example, the system 500 obtains sensor data 509 from one or more sensors 510 of the autonomous vehicle 550 (such as Figure 2One or more of the camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d, etc.) obtain sensor data 509. In one or more examples, the system 500 obtains map data 505 based on the sensor data 509. This can provide a more accurate understanding of the environment. In one or more examples, the system 500 obtains data 507 associated with at least one agent based on the sensor data 509. This can allow the autonomous vehicle 550 to operate in real time and can actively determine the agent status and trajectory 511. In some examples, the perception system 504 is configured to obtain the sensor data 509.

[0101] In one or more examples or embodiments, the sensor data 509 is one or more of the following: Radar sensor data, camera sensor data, image sensor data, audio sensor, and LiDAR sensor data. The specific type of sensor data is not restrictive. The sensor data 509 can indicate the environment around the autonomous vehicle. For example, the sensor data 509 indicates objects (such as agents, etc.) and / or multiple objects in the environment around the autonomous vehicle.

[0102] In one or more examples or embodiments, the sensor 510 can be one or more sensors such as on-board sensors. The sensor 510 can be associated with the autonomous vehicle. The autonomous vehicle can include one or more sensors, and the one or more sensors can be configured to monitor the environment of the autonomous vehicle operation via the sensor data 509, such as via the sensor 510, etc. The sensors can include Figure 2 one or more of the sensors illustrated in

[0103] In one or more embodiments or examples, as Figure 5B shown, the operation further includes determining the autonomous vehicle trajectory based on traffic light parameters. In one or more embodiments or examples, the operation further includes: (e.g., to Figure 4A control system 408 similar to or the same as the control system provides data 522 associated with the autonomous vehicle trajectory. The data 522 associated with the autonomous vehicle trajectory can be control data (e.g., for controlling and / or operating the autonomous vehicle). In one or more embodiments or examples, the data 522 associated with the autonomous vehicle trajectory is configured such that the autonomous vehicle operates along the autonomous vehicle trajectory. In other words, the system 500 is configured to provide operation information to the vehicle 550. The system 500 can be configured to determine the trajectory of the autonomous vehicle 550 based on traffic light parameters. For example, based on traffic light parameters indicating a red light, the trajectory is determined to be associated with the stopping and / or deceleration of the vehicle. As another example, based on traffic light parameters indicating a green light, the trajectory is determined to be associated with the continued movement and / or acceleration of the vehicle.

[0104] In one or more examples or embodiments, causing (e.g., controlling) the operation includes: generating control data for the control system of the autonomous vehicle 550 based on data associated with the AV trajectory. In one or more examples or embodiments, causing (e.g., controlling) the operation includes: providing control data to the control system of the autonomous vehicle 550. In one or more examples or embodiments, causing (e.g., controlling) the operation includes: transmitting the control data to, for example, the control system of the autonomous vehicle 550 and / or an external system. In one or more examples or embodiments, causing (e.g., controlling) the operation includes: controlling the control system of the autonomous vehicle 550 and / or an external system based on the control data. In one or more examples or embodiments, the system is optimized for performance. For example, as all trajectories 511 are repeatedly matched with the lane connection lines, a rate bottleneck occurs. In an embodiment, the system is configured to timestamp the matched trajectories 513. The consecutive timestamps of the matched trajectories 513 can reuse previously determined matches. The consecutive timestamps can be timestamps within 10, 9, 8, 7, 6, 5, 4, 3, 2, or 1 second.

[0105] Figure 6 It is a diagram of an example implementation of a process 600 for determining traffic light parameters via offline and / or online perception motion inference. In one or more embodiments or examples, the process 600 is performed by Figures 5A to 5B the system 500. The input 602 of the process 600 includes map data 606 and data indicating an agent (indicated as agent trace 604 in Figure 6 and / or a predicted agent trajectory 607).

[0106] In one or more examples or embodiments, process 600 involves: determining a traffic light parameter (TLP) inference indicating a green light 608 or a red light 609 based on input 602. Regardless of the determination, process 600 may filter (610) irrelevant agents such as pedestrians as discussed above with respect to Figures 5A to 5B In some examples, process 600 includes: updating situation inference 612 as needed based on any filtering performed (610). Additionally, in some examples, process 600 involves: applying any required post-processing 614, including any grouping and / or backfilling as discussed herein. In one or more examples or embodiments, process 600 involves: determining final traffic light parameters 616, such as traffic light parameters indicating a particular traffic light situation, etc. For online inference, a system that performs process 600 includes, for example, a deep learning-based online traffic light detector 620 (e.g., detector inference, e.g., a vision-based online traffic light detector), which detector 620 serves as an input for integrating (e.g., aggregating) with the output from situation inference 618 to provide traffic light parameters 616. Integration 618 can be considered aggregating matching trajectories with detector inference data from 620 to determine traffic light parameters, for example, via weighted average, bagging, boosting, training of a machine learning model. At 618, integration techniques are applied to the matching trajectories and detector inference data from 620. The traffic light parameters are determined, for example, by: determining a first confidence parameter indicating the confidence level of the matching trajectory (e.g., using directed Hausdorff distance for the green light and deviation from a 0 rate for the red light) and a second confidence parameter indicating the confidence level of the detector inference data; and aggregating the matching trajectory with the detector inference data to determine the traffic light parameters based on the first confidence parameter and / or the second confidence parameter (e.g., performing integration using the confidence parameters).

[0107] Additionally, these techniques can allow for improved labeling of traffic lights by inferring the label of a traffic light from the past trajectories of surrounding actors and optionally future trajectories for offline perception motion inference. The disclosed method is lightweight and fast (and thus can be used online), and can be used as an integration tool for deep learning-based traffic light detectors. This can improve the performance of the entire traffic light detection system.

[0108] Figure 7 is an example implementation of a process for determining traffic light parameters via offline and / or online perception motion inference performed by systems 500 such as Figures 5A to 5B and systems 600 such as Figure 6 The implementation shown in Figure 7 can be performed offline or online (e.g., using sensor data). As Figure 7As shown in the example, the system obtains map data indicating an environment (e.g., intersection 701) and data indicating at least one agent in the environment (shown as vehicles 702, 702A, 702B). The map data includes a plurality of lane connection lines 703, which are illustrated as lines passing through intersection 701. For clarity, Figure 7 Additionally includes arrows indicating the direction of traffic passing through intersection 701. Each of the agents 702, 702A, 702B can have a determined trajectory, which then matches a specific lane connection line 703. For example, as shown, agent 702A will match the straight lane connection line 703A passing through intersection 701. Based on this matching trajectory, the traffic light parameter 704 of the lane parallel to agent 702A can indicate a green light.

[0109] In addition, Figure 7 Agent 702 located within the stop area 710 at a speed of 0 is illustrated. Since the traffic light parameter of the lane where agent 702 is located is not green, agent 702 is within the stop area 710 and the speed of agent 702 is 0, the traffic light parameter 706 is determined to indicate a red light.

[0110] Now referring to Figure 8 , a flowchart of a method or process 800 for offline perception motion inference such as for operating and / or controlling an AV etc. is illustrated. The method can be performed by the systems disclosed herein (such as Figure 2 AV calculation 202f and Figure 4 AV calculation 400, Figure 1 and Figure 2 corresponding vehicles 102, 200, Figure 3 device 300, Figures 5A to 5B system 500, Figure 6 process 600 and Figure 7 implementation etc.). The disclosed systems can include at least one processor, which can be configured to perform one or more operations of method 800. Method 800 can be performed by another device or group of devices (e.g., completely and / or partially etc.) that are separate from or include the systems disclosed herein.

[0111] In one or more embodiments or examples, method 800 includes: at step 802, obtaining, using at least one processor, map data indicative of an environment in which an autonomous vehicle can operate. In one or more embodiments or examples, the map data includes a plurality of lane connection lines associated with an intersection in the environment. For example, the lane connection lines are obtained from the map data. In one or more embodiments or examples, method 800 includes: at step 804, obtaining, using at least one processor, data associated with at least one agent in the environment, the data indicative of the agent relative to the environment. In one or more embodiments or examples, method 800 includes: at step 806, determining, using at least one processor, a trajectory of the at least one agent relative to the environment based on the data. For example, the trajectory of the at least one agent is determined based on the data associated with the at least one agent. In one or more embodiments or examples, method 800 includes: at step 808, matching, using at least one processor, the trajectory of the at least one agent with one of the plurality of lane connection lines to determine a matching trajectory. In one or more embodiments or examples, method 800 includes: at step 810, determining, using at least one processor, traffic light parameters indicative of a condition of a traffic light at the intersection based on the matching trajectory. The traffic light parameters can be a lane connection line. The condition of the traffic light includes (e.g., green and / or red). Advantageously, method 800 can be performed offline using offline data such as from a database. In some examples, the plurality of lane connection lines indicate potential paths that the autonomous vehicle and / or the agent can take at the intersection. Example lane connection lines include right turn, left turn, and straight. Method 800 can include filtering lanes that will not enter the nearest signalized intersection to the autonomous vehicle.

[0112] In one or more examples or embodiments, the data associated with at least one agent includes past and future trajectories (e.g., agent traces). For example, the data includes data for all lanes and all directions. A positioning system (such as Figure 4 positioning system 406, etc.) can be used to obtain data related to the autonomous vehicle. Method 800 can operate in a closed loop or an open loop. In some examples, method 800 uses a directed Hausdorff distance to match the trajectory with one of the lane connection lines.

[0113] In one or more embodiments or examples, matching the trajectory at step 808 includes: determining agent parameters of at least one agent based on the trajectory. In one or more embodiments or examples, matching the trajectory at step 808 includes: determining whether the agent parameters satisfy criteria. In one or more embodiments or examples, matching the trajectory at step 808 includes: filtering out at least one agent in response to determining that the agent parameters do not satisfy the criteria. In one or more embodiments or examples, matching the trajectory at step 808 includes: not filtering out at least one agent in response to determining that the agent parameters satisfy the criteria. In one or more embodiments or examples, matching the trajectory to a lane connection line at step 808 includes: matching the trajectory of at least one unfiltered agent among at least one agent to a lane connection line. In some examples, filtering out at least one agent includes: removing and / or discarding the agent. Example criteria are discussed below. Example criteria include whether the center of the agent is sufficiently within the target intersection, whether the duration of the trajectory is not too short, and whether the start and end points of the trajectory are far enough apart.

[0114] In one or more embodiments or examples, the agent parameters include a class parameter indicating that at least one agent is a vehicle, a bicycle, or a pedestrian. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining whether the class parameter indicates that at least one agent is a pedestrian. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: in response to determining that the class parameter indicates a pedestrian, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: in response to determining that the class parameter does not indicate a pedestrian, determining that the agent parameters meet the criteria. In other words, method 800 filters out pedestrians, but focuses on vehicles such as cars, trucks, etc. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining a distance parameter indicating the distance between at least one agent and the intersection. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining whether the distance parameter is within a first threshold. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: in response to determining that the distance parameter is not within the first threshold, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: in response to determining that the distance parameter is within the first threshold, determining that the agent parameters meet the criteria. In other words, method 800 determines whether the agent is in or close enough to the intersection. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining a time parameter indicating the time at which at least one agent will be within the intersection. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: determining whether the time parameter meets a time threshold. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: in response to determining that the time parameter is below the time threshold, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether the agent parameters meet the criteria includes: in response to determining that the time parameter is equal to or higher than the time threshold, determining that the agent parameters meet the criteria.

[0115] In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining the distance between a starting position and an ending position of a trajectory. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining whether the distance between the starting position and the ending position meets a second threshold. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the distance does not meet the second threshold, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the distance meets the second threshold, determining that the agent parameters meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining agent position parameters indicative of the position of at least one agent relative to an intersection. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining whether the agent position parameters meet a distance from a stop line of the intersection. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: determining whether the agent position parameters indicate that at least one agent is in a traffic lane of the intersection. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters do not meet the stop line distance, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters meet the stop line distance, determining that the agent parameters meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters do not indicate that at least one agent is in a traffic lane of the intersection, determining that the agent parameters do not meet the criteria. In one or more embodiments or examples, determining whether agent parameters meet criteria includes: in response to determining that the agent position parameters indicate that at least one agent is in a traffic lane of the intersection, determining that the agent parameters meet the criteria.

[0116] In one or more embodiments or examples, determining traffic light parameters at step 810 includes: determining whether a matching trajectory of at least one agent is within an intersection. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: in response to determining that the matching trajectory is within the intersection, determining the traffic light parameters to indicate a green light. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: in response to determining that the matching trajectory is not within the intersection, not determining the traffic light parameters to indicate a green light. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: determining whether at least one agent is within a stop region of the intersection. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: in response to determining that at least one agent is within the stop region, determining whether the speed of at least one agent meets a speed threshold. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: in response to determining that at least one agent is not within the stop region, not determining whether the speed of at least one agent meets the speed threshold. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: in response to determining that the speed meets the speed threshold and the traffic light parameters do not indicate a green light, determining the traffic light parameters to indicate a red light. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: in response to determining that the speed does not meet the speed threshold or the traffic light parameters indicate a green light, not determining the traffic light parameters to indicate a red light. In some examples or embodiments, the stop region is a location where a vehicle typically stops for a red light. The stop region can be predefined.

[0117] In one or more embodiments or examples, matching a trajectory with a lane connection line at step 808 includes: determining a plurality of matching parameters indicating distances between the trajectory and each lane connection line among a plurality of lane connection lines. In one or more embodiments or examples, matching a trajectory with a lane connection line at step 808 includes: matching the trajectory with the lane connection line among the plurality of lane connection lines having the lowest matching parameter among the plurality of matching parameters. Method 800 can utilize the closest match to determine the correct match. For example, method 800 uses the Hausdorff distance.

[0118] In one or more embodiments or examples, matching the trajectory to a lane connection line at step 808 includes: obtaining movement data indicative of the past movement of at least one agent. In one or more embodiments or examples, matching the trajectory at step 808 includes: matching the trajectory to at least one lane connection line based on the movement data. In one or more embodiments or examples, matching the trajectory to a lane connection line at step 808 includes: determining prediction movement parameters indicative of the predicted movement of at least one agent. In one or more embodiments or examples, matching the trajectory at step 808 includes: matching the trajectory to at least one lane connection line based on the prediction movement parameters. In one or more embodiments or examples, obtaining data associated with at least one agent at step 804 includes: obtaining data associated with a plurality of agents. In one or more embodiments or examples, determining a trajectory at step 806 includes: determining a trajectory for each agent among the plurality of agents. In one or more embodiments or examples, matching the trajectory to a lane connection line at step 808 includes: matching the trajectory of each agent among the plurality of agents to a lane connection line to provide a plurality of matching trajectories.

[0119] Method 800 may use certain post-processing as discussed herein. In one or more embodiments or examples, the intersection includes a plurality of traffic lights. In one or more embodiments or examples, method 800 includes: grouping the trajectories in the parallel direction among the trajectories of each agent. In one or more embodiments or examples, method 800 includes: matching the grouped trajectories to the corresponding lane connection lines. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: determining traffic light parameters for each traffic light among the plurality of traffic lights based on the matched grouped trajectories. The map data may indicate information related to traffic lights in the "parallel" direction and thus sharing the same traffic light conditions indicated by the traffic light parameters. Method 800 may use this information to set all traffic lights in the parallel direction to have the same conditions. In one or more embodiments or examples, determining traffic light parameters at step 810 includes: determining current traffic light parameters based on previously determined traffic light parameters. For example, method 800 uses post-processing backfill after a certain time for drivers with slow starts. Method 800 updates the current traffic light parameters to adopt the values of the previous traffic light parameters, in particular to account for the slow reaction time of the driver, where the traffic light parameters are determined according to the green / red light conditions in a past time window.

[0120] In one or more examples or embodiments, method 800 is performed online. In one or more embodiments or examples, method 800 further includes: obtaining sensor data indicative of an environment. In one or more embodiments or examples, obtaining map data at step 802 includes: obtaining map data based on the sensor data. In one or more embodiments or examples, obtaining data associated with at least one agent at step 804 includes: obtaining data associated with at least one agent based on the sensor data. For example, the sensor data includes LiDAR sensor data, camera sensor data, and / or Radar sensor data. In one or more embodiments or examples, method 800 further includes: determining an autonomous vehicle trajectory based on traffic light parameters. In one or more embodiments or examples, method 800 further includes: providing, using at least one processor, data associated with the autonomous vehicle trajectory. In one or more embodiments or examples, the data associated with the autonomous vehicle trajectory is configured such that the autonomous vehicle operates along the autonomous vehicle trajectory.

[0121] In the foregoing description, aspects and embodiments of the present disclosure have been described with reference to numerous specific details, which may vary depending on implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive in a limiting sense. The sole and exclusive indication of the scope of the invention, and what the applicant intends to be the scope of the invention, is the literal and equivalent scope of the claims as issued from this application in the specific form of the allowed claims, including any subsequent amendments. Any definition explicitly set forth herein for terms to be included in such claims shall govern the meaning of such terms as used in the claims. Additionally, when the term "further includes" is used in the foregoing specification or the appended claims, the text following that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously recited step or entity.

[0122] A non-transitory computer-readable medium is disclosed, including instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform operations in accordance with one or more of the methods disclosed herein.

[0123] Also disclosed are a method, a non-transitory computer-readable medium, and a system according to any one of the following.

[0124] Item 1. A method, comprising:

[0125] using at least one processor, obtaining map data indicative of an environment in which an autonomous vehicle is capable of operating, the map data including a plurality of lane connection lines associated with intersections in the environment;

[0126] Using the at least one processor, obtain data associated with at least one agent in the environment, indicating the agent's data relative to the environment;

[0127] Using the at least one processor, determine a trajectory of the at least one agent relative to the environment based on the data;

[0128] Using the at least one processor, match the trajectory of the at least one agent with one of the plurality of lane connection lines to determine a matching trajectory; and

[0129] Using the at least one processor, determine traffic light parameters indicating the status of a traffic light at the intersection based on the matching trajectory.

[0130] Item 2. The method according to item 1, wherein matching the trajectory includes:

[0131] Determine agent parameters of the at least one agent;

[0132] Based on the trajectory, determine whether the agent parameters meet the criteria; and

[0133] In response to determining that the agent parameters do not meet the criteria, filter out the at least one agent;

[0134] Wherein, matching the trajectory with one lane connection line includes matching the trajectory of at least one unfiltered agent among the at least one agent with one lane connection line.

[0135] Item 3. The method according to item 2, wherein the agent parameters include class parameters indicating whether the at least one agent is a vehicle, a bicycle or a pedestrian; wherein, determining whether the agent parameters meet the criteria includes:

[0136] Determine whether the class parameters indicate that the at least one agent is a pedestrian; and

[0137] In response to determining that the class parameters indicate a pedestrian, determine that the agent parameters do not meet the criteria.

[0138] Item 4. The method according to any one of items 2 to 3, wherein determining whether the agent parameters meet the criteria includes:

[0139] Determine a distance parameter indicating the distance between the at least one agent and the intersection;

[0140] Determine whether the distance parameter is within a first threshold; and

[0141] In response to determining that the distance parameter is not within the first threshold, it is determined that the agent parameter does not meet the criterion.

[0142] Item 5. The method according to any one of Items 2 to 4, wherein determining whether the agent parameter meets the criterion includes:

[0143] Determining a time parameter indicating the time when the at least one agent will be within the intersection;

[0144] Determining whether the time parameter meets a time threshold; and

[0145] In response to determining that the time parameter is below the time threshold, it is determined that the agent parameter does not meet the criterion.

[0146] Item 6. The method according to any one of Items 2 to 5, wherein determining whether the agent parameter meets the criterion includes:

[0147] Determining the distance between the starting position and the ending position of the trajectory;

[0148] Determining whether the distance between the starting position and the ending position meets a second threshold; and

[0149] In response to determining that the distance does not meet the second threshold, it is determined that the agent parameter does not meet the criterion.

[0150] Item 7. The method according to any one of Items 2 to 6, wherein determining whether the agent parameter meets the criterion includes:

[0151] Determining an agent position parameter indicating the position of the at least one agent relative to the intersection;

[0152] Determining whether the agent position parameter meets the distance to the stop line of the intersection;

[0153] Determining whether the agent position parameter indicates that the at least one agent is in the traffic lane of the intersection; and

[0154] In response to determining that the agent position parameter does not meet the stop line distance or the agent position parameter does not indicate that the at least one agent is in the traffic lane of the intersection, it is determined that the agent parameter does not meet the criterion.

[0155] Item 8. The method according to any one of the preceding items, wherein determining the traffic light parameter includes:

[0156] Determining whether the matching trajectory of the at least one agent is within the intersection; and

[0157] In response to determining that the matching trajectory is within the intersection, determine the traffic light parameter to indicate a green light.

[0158] Item 9. The method according to item 8, wherein determining the traffic light parameter includes:

[0159] Determine whether the at least one agent is located within the stop area of the intersection;

[0160] In response to determining that the at least one agent is located within the stop area, determine whether the speed of the at least one agent meets a speed threshold; and

[0161] In response to determining that the speed meets the speed threshold and the traffic light parameter does not indicate a green light, determine the traffic light parameter to indicate a red light.

[0162] Item 10. The method according to any one of the preceding items, wherein matching the trajectory with a lane connection line includes:

[0163] Determine a plurality of matching parameters indicating the distance between the trajectory and each lane connection line among the plurality of lane connection lines; and

[0164] Match the trajectory with the lane connection line having the lowest matching parameter among the plurality of matching parameters.

[0165] Item 11. The method according to any one of the preceding items, wherein matching the trajectory with a lane connection line includes:

[0166] Obtain movement data indicating the past movement of the at least one agent;

[0167] Wherein, matching the trajectory includes matching the trajectory with at least one lane connection line based on the movement data.

[0168] Item 12. The method according to any one of the preceding items, wherein matching the trajectory with a lane connection line includes:

[0169] Determine prediction movement parameters indicating the predicted movement of the at least one agent;

[0170] Wherein, matching the trajectory includes matching the trajectory with at least one lane connection line based on the prediction movement parameters.

[0171] Item 13. The method according to any one of the preceding items, further comprising:

[0172] Obtaining data associated with the at least one agent includes obtaining data associated with a plurality of agents;

[0173] Determining the trajectories includes determining a trajectory for each of the plurality of agents; and

[0174] Matching the trajectories to a lane connection line includes matching the trajectory of each of the plurality of agents to a lane connection line to provide a plurality of matched trajectories.

[0175] Item 14. The method according to item 13, wherein the intersection includes a plurality of traffic lights, and wherein the method includes:

[0176] Grouping the trajectories in the trajectories of each agent that are in a parallel direction; and

[0177] Matching the grouped trajectories to corresponding lane connection lines.

[0178] Item 15. The method according to item 14, wherein determining the traffic light parameters includes:

[0179] Determining traffic light parameters for each of the plurality of traffic lights based on a plurality of matched grouped trajectories.

[0180] Item 16. The method according to item 15, wherein determining the traffic light parameters includes:

[0181] Determining current traffic light parameters based on previously determined traffic light parameters.

[0182] Item 17. The method according to any one of the preceding items, further comprising:

[0183] Obtaining sensor data indicative of the environment,

[0184] wherein obtaining the map data includes obtaining the map data based on the sensor data; and

[0185] wherein obtaining data associated with the at least one agent includes obtaining data associated with the at least one agent based on the sensor data.

[0186] Item 18. The method according to item 17, further comprising:

[0187] Determining an autonomous vehicle trajectory based on the traffic light parameters; and

[0188] Using the at least one processor, providing data associated with the autonomous vehicle trajectory, the data associated with the autonomous vehicle trajectory being configured to cause the autonomous vehicle to operate along the autonomous vehicle trajectory.

[0189] Item 18a. The method according to any one of items 17 to 18, wherein obtaining data associated with the at least one agent includes:

[0190] Predict a future trajectory of the at least one agent; and

[0191] Include the future trajectory in data associated with the at least one agent.

[0192] Item 18b. The method according to any one of Items 17 to 18a, wherein determining the traffic light parameter includes:

[0193] Obtain detector inference data indicating the status of the traffic light; and

[0194] Aggregate the matching trajectory with the detector inference data to determine the traffic light parameter.

[0195] Item 18c. The method according to Item 18b, wherein aggregating the matching trajectory with the detector inference data includes applying an integration technique to the matching trajectory and the detector inference data.

[0196] Item 18d. The method according to any one of Items 18b to 18c, wherein determining the traffic light parameter includes:

[0197] Determine a first confidence parameter indicating a confidence level of the matching trajectory;

[0198] Determine a second confidence parameter indicating a confidence level of the detector inference data; and

[0199] Aggregate the matching trajectory with the detector inference data to determine the traffic light parameter based on the first confidence parameter and / or the second confidence parameter.

[0200] Item 19. A non-transitory computer-readable medium including instructions stored thereon, which when executed by at least one processor cause the at least one processor to perform operations, the operations including:

[0201] Obtain map data indicating an environment in which an autonomous vehicle can operate, the map data including a plurality of lane connection lines associated with intersections in the environment;

[0202] Obtain data indicating the at least one agent in the environment with respect to the environment;

[0203] Based on the data, determine a trajectory of the at least one agent with respect to the environment;

[0204] Match the trajectory of the at least one agent with one of the plurality of lane connection lines to determine a matching trajectory; and

[0205] Determine traffic light parameters indicating the status of the traffic light at the intersection based on the matching trajectory.

[0206] Item 20. The non-transitory computer-readable medium according to item 19, wherein matching the trajectory includes:

[0207] Determine the agent parameters of the at least one agent;

[0208] Based on the trajectory, determine whether the agent parameters meet the criteria; and

[0209] In response to determining that the agent parameters do not meet the criteria, filter out the at least one agent;

[0210] Wherein, matching the trajectory with a lane connection line includes matching the trajectory of at least one unfiltered agent among the at least one agent with a lane connection line.

[0211] Item 21. The non-transitory computer-readable medium according to item 20, wherein the agent parameters include class parameters indicating whether the at least one agent is a vehicle, a bicycle, or a pedestrian; wherein, determining whether the agent parameters meet the criteria includes:

[0212] Determine whether the class parameters indicate that the at least one agent is a pedestrian; and

[0213] In response to determining that the class parameters indicate a pedestrian, determine that the agent parameters do not meet the criteria.

[0214] Item 22. The non-transitory computer-readable medium according to any one of items 20 to 21, wherein determining whether the agent parameters meet the criteria includes:

[0215] Determine a distance parameter indicating the distance between the at least one agent and the intersection;

[0216] Determine whether the distance parameter is within a first threshold; and

[0217] In response to determining that the distance parameter is not within the first threshold, determine that the agent parameters do not meet the criteria.

[0218] Item 23. The non-transitory computer-readable medium according to any one of items 20 to 22, wherein determining whether the agent parameters meet the criteria includes:

[0219] Determine a time parameter indicating the time when the at least one agent will be within the intersection;

[0220] Determine whether the time parameter meets a time threshold; and

[0221] In response to determining that the time parameter is below the time threshold, it is determined that the agent parameter does not meet the criterion.

[0222] Item 24. The non-transitory computer-readable medium according to any one of Items 20 to 23, wherein determining whether the agent parameter meets the criterion includes:

[0223] Determine the distance between the starting position and the ending position of the trajectory;

[0224] Determine whether the distance between the starting position and the ending position meets a second threshold; and

[0225] In response to determining that the distance does not meet the second threshold, it is determined that the agent parameter does not meet the criterion.

[0226] Item 25. The non-transitory computer-readable medium according to any one of Items 20 to 24, wherein determining whether the agent parameter meets the criterion includes:

[0227] Determine an agent position parameter indicating the position of the at least one agent relative to the intersection;

[0228] Determine whether the agent position parameter meets the distance from the stop line of the intersection;

[0229] Determine whether the agent position parameter indicates that the at least one agent is in the traffic lane of the intersection; and

[0230] In response to determining that the agent position parameter does not meet the stop line distance or the agent position parameter does not indicate that the at least one agent is in the traffic lane of the intersection, it is determined that the agent parameter does not meet the criterion.

[0231] Item 26. The non-transitory computer-readable medium according to any one of Items 19 to 25, wherein determining the traffic light parameter includes:

[0232] Determine whether the matching trajectory of the at least one agent is within the intersection; and

[0233] In response to determining that the matching trajectory is within the intersection, determine the traffic light parameter as indicating a green light.

[0234] Item 27. The non-transitory computer-readable medium according to Item 26, wherein determining the traffic light parameter includes:

[0235] Determine whether the at least one agent is located within the stop area of the intersection;

[0236] In response to determining that at least one agent is located within the stop region, determining whether the speed of the at least one agent meets a speed threshold; and

[0237] In response to determining that the speed meets the speed threshold and the traffic light parameter does not indicate a green light, determining the traffic light parameter to indicate a red light.

[0238] Item 28. The non-transitory computer-readable medium according to any one of items 19 to 27, wherein matching the trajectory with a lane connection line includes:

[0239] Determining a plurality of matching parameters indicating distances between the trajectory and each lane connection line among the plurality of lane connection lines; and

[0240] Matching the trajectory with the lane connection line having the lowest matching parameter among the plurality of matching parameters.

[0241] Item 29. The non-transitory computer-readable medium according to any one of items 19 to 28, wherein matching the trajectory with a lane connection line includes:

[0242] Obtaining movement data indicating past movements of the at least one agent;

[0243] Wherein matching the trajectory includes matching the trajectory with at least one lane connection line based on the movement data.

[0244] Item 30. The non-transitory computer-readable medium according to any one of items 19 to 29, wherein matching the trajectory with a lane connection line includes:

[0245] Determining prediction movement parameters indicating predicted movements of the at least one agent;

[0246] Wherein matching the trajectory includes matching the trajectory with at least one lane connection line based on the prediction movement parameters.

[0247] Item 31. The non-transitory computer-readable medium according to any one of items 19 to 30, further comprising:

[0248] Obtaining data associated with the at least one agent includes obtaining data associated with a plurality of agents;

[0249] Determining the trajectory includes determining trajectories for each agent among the plurality of agents; and

[0250] Matching the trajectory with a lane connection line includes matching the trajectories of each agent among the plurality of agents with a lane connection line to provide a plurality of matching trajectories.

[0251] Item 32. The non-transitory computer-readable medium according to Item 31, wherein the intersection includes a plurality of traffic lights, and the non-transitory computer-readable medium includes:

[0252] Grouping the trajectories in the trajectories of each agent that are in parallel directions; and

[0253] Matching the grouped trajectories with corresponding lane connection lines.

[0254] Item 33. The non-transitory computer-readable medium according to Item 32, wherein determining the traffic light parameters includes:

[0255] Determining traffic light parameters for each of the plurality of traffic lights based on the matched grouped trajectories.

[0256] Item 34. The non-transitory computer-readable medium according to Item 33, wherein determining the traffic light parameters includes:

[0257] Determining current traffic light parameters based on previously determined traffic light parameters.

[0258] Item 35. The non-transitory computer-readable medium according to any one of Items 19 to 34, further comprising:

[0259] Obtaining sensor data indicating the environment,

[0260] wherein obtaining the map data includes obtaining the map data based on the sensor data; and

[0261] wherein obtaining data associated with the at least one agent includes obtaining data associated with the at least one agent based on the sensor data.

[0262] Item 36. The non-transitory computer-readable medium according to Item 35, further comprising:

[0263] Determining an autonomous vehicle trajectory based on the traffic light parameters; and

[0264] Providing data associated with the autonomous vehicle trajectory, the data associated with the autonomous vehicle trajectory being configured to cause the autonomous vehicle to operate along the autonomous vehicle trajectory.

[0265] Item 37. A system, comprising: at least one processor; and at least one memory having instructions stored thereon, the instructions when executed by the at least one processor cause the at least one processor to perform operations, the operations including:

[0266] Obtain map data indicating an environment in which an autonomous vehicle can operate, the map data including a plurality of lane connection lines associated with intersections in the environment;

[0267] Obtain data indicating at least one agent in the environment with respect to the environment;

[0268] Determine a trajectory of the at least one agent with respect to the environment based on the data;

[0269] Match the trajectory of the at least one agent with one of the plurality of lane connection lines to determine a matching trajectory; and

[0270] Determine traffic light parameters indicating the status of a traffic light at the intersection based on the matching trajectory.

[0271] Item 38. The system according to item 37, wherein matching the trajectory includes:

[0272] Determine agent parameters of the at least one agent based on the trajectory;

[0273] Determine whether the agent parameters meet criteria; and

[0274] In response to determining that the agent parameters do not meet the criteria, filter out the at least one agent;

[0275] Wherein matching the trajectory with one lane connection line includes matching the trajectory of at least one unfiltered agent among the at least one agent with one lane connection line.

[0276] Item 39. The system according to item 38, wherein the agent parameters include a class parameter indicating whether the at least one agent is a vehicle, a bicycle, or a pedestrian; wherein determining whether the agent parameters meet the criteria includes:

[0277] Determine whether the class parameter indicates that the at least one agent is a pedestrian; and

[0278] In response to determining that the class parameter indicates a pedestrian, determine that the agent parameters do not meet the criteria.

[0279] Item 40. The system according to any one of items 38 to 39, wherein determining whether the agent parameters meet the criteria includes:

[0280] Determine a distance parameter indicating the distance between the at least one agent and the intersection;

[0281] Determine whether the distance parameter is within a first threshold; and

[0282] In response to determining that the distance parameter is not within the first threshold, it is determined that the agent parameter does not meet the criterion.

[0283] Item 41. The system according to any one of Items 38 to 40, wherein determining whether the agent parameter meets the criterion includes:

[0284] Determining a time parameter indicating the time when the at least one agent will be within the intersection;

[0285] Determining whether the time parameter meets a time threshold; and

[0286] In response to determining that the time parameter is below the time threshold, it is determined that the agent parameter does not meet the criterion.

[0287] Item 42. The system according to any one of Items 38 to 41, wherein determining whether the agent parameter meets the criterion includes:

[0288] Determining the distance between the starting position and the ending position of the trajectory;

[0289] Determining whether the distance between the starting position and the ending position meets a second threshold; and

[0290] In response to determining that the distance does not meet the second threshold, it is determined that the agent parameter does not meet the criterion.

[0291] Item 43. The system according to any one of Items 38 to 42, wherein determining whether the agent parameter meets the criterion includes:

[0292] Determining an agent position parameter indicating the position of the at least one agent relative to the intersection;

[0293] Determining whether the agent position parameter meets the distance from the stop line of the intersection;

[0294] Determining whether the agent position parameter indicates that the at least one agent is in the traffic lane of the intersection; and

[0295] In response to determining that the agent position parameter does not meet the stop line distance or the agent position parameter does not indicate that the at least one agent is in the traffic lane of the intersection, it is determined that the agent parameter does not meet the criterion.

[0296] Item 44. The system according to any one of Items 37 to 43, wherein determining the traffic light parameter includes:

[0297] Determining whether the matching trajectory of the at least one agent is within the intersection; and

[0298] In response to determining that the matching trajectory is within the intersection, determine the traffic light parameter to indicate a green light.

[0299] Item 45. The system according to item 44, wherein determining the traffic light parameter includes:

[0300] Determine whether the at least one agent is located within the stop area of the intersection;

[0301] In response to determining that the at least one agent is located within the stop area, determine whether the speed of the at least one agent meets a speed threshold; and

[0302] In response to determining that the speed meets the speed threshold and the traffic light parameter does not indicate a green light, determine the traffic light parameter to indicate a red light.

[0303] Item 46. The system according to any one of items 37 to 45, wherein matching the trajectory with a lane connection line includes:

[0304] Determine a plurality of matching parameters indicating the distances between the trajectory and each lane connection line among the plurality of lane connection lines; and

[0305] Match the trajectory with the lane connection line having the lowest matching parameter among the plurality of matching parameters.

[0306] Item 47. The system according to any one of items 37 to 46, wherein matching the trajectory with a lane connection line includes:

[0307] Obtain movement data indicating the past movement of the at least one agent;

[0308] Wherein, matching the trajectory includes matching the trajectory with at least one lane connection line based on the movement data.

[0309] Item 48. The system according to any one of items 37 to 47, wherein matching the trajectory with a lane connection line includes:

[0310] Determine prediction movement parameters indicating the predicted movement of the at least one agent;

[0311] Wherein, matching the trajectory includes matching the trajectory with at least one lane connection line based on the prediction movement parameters.

[0312] Item 49. The system according to any one of items 37 to 48, the operation further includes:

[0313] Obtaining data associated with the at least one agent includes obtaining data associated with a plurality of agents;

[0314] Determining the trajectories includes determining a trajectory for each of the plurality of agents; and

[0315] Matching the trajectories to a lane connection line includes matching the trajectory of each of the plurality of agents to a lane connection line to provide a plurality of matching trajectories.

[0316] Item 50. The system according to item 49, wherein the intersection includes a plurality of traffic lights, and wherein the operation includes:

[0317] Grouping the trajectories in the trajectories of each agent that are in a parallel direction; and

[0318] Matching the grouped trajectories to corresponding lane connection lines.

[0319] Item 51. The system according to item 50, wherein determining the traffic light parameters includes:

[0320] Determining traffic light parameters for each of the plurality of traffic lights based on the matching grouped trajectories.

[0321] Item 52. The system according to item 51, wherein determining the traffic light parameters includes:

[0322] Determining current traffic light parameters based on previously determined traffic light parameters.

[0323] Item 53. The system according to any one of items 37 to 52, the operation further includes:

[0324] Obtaining sensor data indicative of the environment,

[0325] wherein obtaining the map data includes obtaining the map data based on the sensor data; and

[0326] wherein obtaining data associated with the at least one agent includes obtaining data associated with the at least one agent based on the sensor data.

[0327] Item 54. The system according to item 53, the operation further includes:

[0328] Determining an autonomous vehicle trajectory based on the traffic light parameters; and

[0329] Using the at least one processor, providing data associated with the autonomous vehicle trajectory, the data associated with the autonomous vehicle trajectory being configured to cause the autonomous vehicle to operate along the autonomous vehicle trajectory.

Claims

1. A method, comprising: Using at least one processor, obtaining map data indicating an environment in which an autonomous vehicle can operate, the map data including a plurality of lane connection lines associated with intersections in the environment; Using the at least one processor, obtaining data indicating at least one agent in the environment, the data indicating the at least one agent also indicating the location of the agent relative to the environment; Using the at least one processor, determining a trajectory of the at least one agent relative to the environment based on the data indicating the at least one agent; Using the at least one processor, matching the trajectory of the at least one agent with a lane connection line among the plurality of lane connection lines to determine a matching trajectory; And Using the at least one processor, determining traffic light parameters indicating the status of a traffic light at the intersection based on the matching trajectory.

2. The method according to claim 1, wherein, Matching the trajectory includes: Determining agent parameters of the at least one agent based on the trajectory; Determining whether the agent parameters meet criteria; and In response to determining that the agent parameters do not meet the criteria, filtering the at least one agent; Wherein, matching the trajectory with the lane connection line includes matching the trajectory of at least one unfiltered agent among the at least one agent with the lane connection line.

3. The method according to claim 2, wherein, The agent parameters include class parameters indicating that the at least one agent is a vehicle, a bicycle, or a pedestrian; And Wherein, determining whether the agent parameters meet the criteria includes: Determining that the class parameters indicate that the at least one agent is a pedestrian; and In response to determining that the class parameters indicate that the at least one agent is a pedestrian, determining that the agent parameters do not meet the criteria.

4. The method according to any one of claims 2 to 3, wherein Determining whether the agent parameters meet the criteria includes: Determining a distance parameter indicating the distance between the at least one agent and the intersection; Determining that the distance parameter is within a first threshold; and In response to determining that the distance parameter is not within the first threshold, determining that the agent parameters do not meet the criteria.

5. The method according to any one of claims 2 to 4, wherein Determining whether the agent parameters meet the criteria includes: Determining a time parameter indicating the time when the at least one agent will be within the intersection; Determining that the time parameter meets a time threshold; and In response to determining that the time parameter is below the time threshold, determining that the agent parameters do not meet the criteria.

6. The method according to any one of claims 2 to 5, wherein, Determining whether the agent parameters meet the criteria includes: Determining the distance between the starting position and the ending position of the trajectory; Determining that the distance between the starting position and the ending position meets a second threshold; and In response to determining that the distance does not meet the second threshold, determining that the agent parameters do not meet the criteria.

7. The method according to any one of claims 2 to 6, wherein Determining whether the agent parameters meet the criteria includes: Determining an agent position parameter indicating the position of the at least one agent relative to the intersection; Determining that the agent position parameter meets the distance from the stop line of the intersection; Determining that the agent position parameter indicates that the at least one agent is in a traffic lane at the intersection; and In response to determining that the agent position parameter does not satisfy the stop line distance or the agent position parameter does not indicate that the at least one agent is in the traffic lane of the intersection, it is determined that the agent parameter does not satisfy the criterion.

8. The method according to any one of the preceding claims, wherein, Determining the traffic light parameter includes: Determining that the matching trajectory of the at least one agent is within the intersection; and In response to determining that the matching trajectory is within the intersection, determining the traffic light parameter to indicate a green light.

9. The method according to claim 8, wherein Determining the traffic light parameter includes: Determining whether the at least one agent is located within the stop area of the intersection; In response to determining that the at least one agent is located within the stop area, determining whether the speed of the at least one agent satisfies a speed threshold; and In response to determining that the speed satisfies the speed threshold and the traffic light parameter does not indicate a green light, determining the traffic light parameter to indicate a red light.

10. The method according to any one of the preceding claims, wherein Matching the trajectory with a lane connection line includes: Determining a plurality of matching parameters indicating the distances between the trajectory and each lane connection line among the plurality of lane connection lines; and Matching the trajectory with the lane connection line having the lowest matching parameter among the plurality of matching parameters.

11. The method according to any one of the preceding claims, wherein, Matching the trajectory with a lane connection line includes: Obtaining movement data indicating the past movement of the at least one agent; wherein matching the trajectory includes matching the trajectory with at least one lane connection line based on the movement data.

12. The method according to any one of the preceding claims, wherein, Matching the trajectory with a lane connection line includes: Determining predicted movement parameters indicating the predicted movement of the at least one agent; wherein matching the trajectory includes matching the trajectory with at least one lane connection line based on the predicted movement parameters.

13. The method according to any one of the preceding claims, wherein, Obtaining data associated with the at least one agent includes obtaining data associated with a plurality of agents; Determining the trajectory includes determining the trajectory for each agent among the plurality of agents; and Matching the trajectory with a lane connection line includes matching the trajectory of each agent among the plurality of agents with a lane connection line to provide a plurality of matching trajectories.

14. The method according to claim 13, wherein, The intersection includes a plurality of traffic lights, and the method further includes: Grouping the trajectories associated with the parallel directions in the trajectories of each agent; and Matching the grouped trajectories with the corresponding lane connection lines.

15. The method according to claim 14, wherein, Determining the traffic light parameter includes: Determining the traffic light parameter for each traffic light among the plurality of traffic lights based on the matching grouped trajectories.

16. The method according to claim 13, wherein Determining the traffic light parameter includes: Determining the current traffic light parameter based on the previously determined traffic light parameter.

17. The method according to any one of the preceding claims, further includes: Obtaining sensor data indicating the environment in which the autonomous vehicle is operating, wherein obtaining the map data includes obtaining the map data based on the sensor data; and wherein obtaining data associated with the at least one agent includes obtaining data associated with the at least one agent based on the sensor data.

18. The method according to claim 17, further comprising: determining an autonomous vehicle trajectory based on the traffic light parameters; and providing data indicative of the autonomous vehicle trajectory, the data associated with the autonomous vehicle trajectory being configured to cause the autonomous vehicle to operate along the autonomous vehicle trajectory.

19. The method according to any one of claims 17 to 18, wherein, Obtaining data associated with the at least one agent includes: predicting a future trajectory of the at least one agent; and including the future trajectory in the data associated with the at least one agent.

20. The method according to any one of claims 17 to 19, wherein Determining the traffic light parameters includes: obtaining detector inference data indicative of the condition of the traffic light; and aggregating the matching trajectory with the detector inference data to determine the traffic light parameters.

Images