Hierarchical multi-object tracker for automatic annotation

Abstract

Methods are provided for a hierarchical multi-object tracker for automatic annotation. Some methods include generating trajectory segments based on detections in sensor data. A weighted graph of trajectories is generated based on the trajectory segments, and the weighted graph is converted to a bipartite graph. The bipartite graph is solved to determine at least one optimal path cover corresponding to respective observed objects, and the detections in the sensor data are annotated with trajectory identifiers corresponding to trajectories representing the optimal path cover of the respective observed objects. Systems and computer program products are also provided.

Description

Background Technology

[0001] Labeled or annotated data is used to train autonomous systems. Labeled data enables machine learning models to learn how to perform various functions. Manually labeling data can be expensive and time-consuming. Attached Figure Description

[0002] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system;

[0003] Figure 2 It is a diagram of one or more systems that include autonomous vehicles;

[0004] Figure 3 yes Figure 1 and Figure 2 A diagram of one or more devices and / or one or more system components;

[0005] Figure 4 It is a diagram of some components of an autonomous system;

[0006] Figure 5 This is a diagram of the implementation of a hierarchical multi-object tracker for automatic annotation;

[0007] Figure 6 It is a block diagram of the workflow for a hierarchical multi-object tracker used for automatic annotation;

[0008] Figure 7 The detection results in the sensor data frame are shown;

[0009] Figure 8 The weighted graph is shown;

[0010] Figure 9A This shows a weighted graph where nodes are divided into two disjoint and independent sets;

[0011] Figure 9B This shows a bipartite graph;

[0012] Figure 10A The workflow corresponding to the first stage tracker is shown;

[0013] Figure 10B The workflow corresponding to the second stage tracker is shown;

[0014] Figure 11 A flowchart of the process for a hierarchical multi-object tracker used for automatic annotation is shown. Detailed Implementation

[0015] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0016] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0017] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0018] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0019] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0020] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet (e.g., a data packet) that includes data.

[0021] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0022] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0023] General Overview

[0024] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement a hierarchical offline multi-object tracker for automatic annotation. Data captured by at least one sensor associated with a vehicle is used to identify and make decisions related to features of the environment. In some embodiments, the sensor data is used for object detection. In object detection, the sensor data is analyzed to annotate it using confidence scores, which indicate the presence of instances of a specific object category within a region of the environment corresponding to the captured sensor data. For example, the sensor data is divided into one or more bounding boxes, and each bounding box is labeled with a likelihood that the bounding box contains an object of a specific category.

[0025] A hierarchical multi-object tracker automatically determines at least one object track associated with each detected object. In the example, sensor data is automatically labeled with at least one object track associated with each detected object. For example, detection results are fed into a first-level tracker to generate tracklets. The tracklets are then fed into a second-level tracker, where they are associated with the tracks of the detected objects. Sensor data is labeled with at least one track representing the motion of the detected objects. In the example, the labeled data is used to train or otherwise evaluate the autonomous system.

[0026] This paper describes a technique for automatically labeling sensor data using hierarchical offline multi-object trackers, based on the implementation of the systems, methods, and computer program products described herein. Some advantages of this technique include automatic labeling of trajectories in the sensor data. The use of pairwise-likelihood graphs for trajectory segment stitching allows for the discarding of less-than-best trajectory segments to achieve better tracking performance. Therefore, the tracker's runtime is reduced. Furthermore, false positives are reduced by discarding frames of the trajectories before using them to label the sensor data. In this way, the hierarchical tracker operates on trajectories input from different types of trackers, such as Kalman filter-based trackers, first-stage stitchers with short scan windows, deep learning-based trackers, or any combination thereof.

[0027] Now for reference Figure 1 Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0028] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2 The vehicles 200 in the collection of vehicles 200 are similar or identical. In some embodiments, vehicles 200 in the collection of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0029] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., moving with a speed and associated with at least one trajectory). In some embodiments, object 104 is associated with a corresponding location in area 108.

[0030] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with (e.g., defining) a series of actions (also referred to as a trajectory line) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will board the AV, and a second state or area includes a location where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Alternatively, route 106 may include more precise actions or states, such as, for example, a specific target lane or precise location within a lane area and a target rate at those locations. In the example, route 106 includes a plurality of precise state sequences along at least one high-level action having a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to a plurality of trajectories that collectively form a high-level route terminating at a final target state or region.

[0031] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In the example, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driving lane, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In the example, a road includes at least one lane associated with at least one lane marking line (e.g., identified based on at least one lane marking line).

[0032] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Infrastructure or Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with vehicle 102 and / or V2I system 118. In some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, V2I device 110 includes radio frequency identification (RFID) devices, signs, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markings, streetlights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicle 102. Alternatively, in some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, and / or queue management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0033] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0034] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0035] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including and / or not including autonomous systems)).

[0036] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0037] supply Figure 1 The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements with different arrangements. Alternatively or additionally, at least one element of environment 100 may be described as being composed of… Figure 1 One or more functions performed by at least one different element of environment 100. Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0038] Now for reference Figure 2 Vehicle 200 (which can be connected with) Figure 1 The vehicle 200 (which is the same as or similar to vehicle 102) includes, or is associated with, an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 is associated with, or is similar to, vehicle 102 (see...). Figure 1 The autonomous system 202 is configured to endow the vehicle 200 with autonomous driving capabilities (e.g., implementing at least one driving automation or maneuver-based function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention, such as Level 5 ADS-operated vehicles), highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations, such as Level 4 ADS-operated vehicles), and / or conditionally autonomous vehicles (e.g., vehicles that abandon human intervention in limited situations, such as Level 3 ADS-operated vehicles)). In one embodiment, the autonomous system 202 includes some or all of the operational or tactical functionality required for the vehicle 200 to operate in road traffic and continuously perform dynamic driving tasks (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) incorporating driver support features. The autonomous system 202 supports various levels of driver automation, ranging from no driver automation (e.g., Level 0) to full driver automation (e.g., Level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0039] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0040] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision) purposes. In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0041] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD (Traffic Light Detection) data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other systems incorporated into cameras described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., wide-angle lens, fisheye lens, and / or lenses with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0042] The Light Detection and Ranging (LiDAR) sensor 202b includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object within the field of view of the LiDAR sensor 202b.

[0043] The radio detection and ranging (RADR) sensor 202c includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). Radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by Radar sensor 202c encounter a physical object and are reflected back to Radar sensor 202c. In some embodiments, the radio waves emitted by Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with Radar sensor 202c generates a signal representing objects included in the field of view of Radar sensor 202c. For example, at least one data processing system associated with Radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of Radar sensor 202c.

[0044] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0045] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, an autonomous vehicle computing unit 202f, a safety controller 202g, and / or a DBW (drive-by-wire) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0046] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1 V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1The V2I system 118 communicates with the same or similar V2I system.

[0047] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0048] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200. Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0049] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle movement (such as starting to move forward, stopping to move forward, starting to move backward, stopping to move backward, accelerating in a certain direction, decelerating in a certain direction, etc.) or lateral vehicle movement (such as making a left turn and / or making a right turn, etc.). In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0050] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to rotate left or right, thereby causing the vehicle 200 to turn left or right. In other words, the steering control system 206 causes the activity required to regulate the y-axis component of the vehicle's motion.

[0051] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0052] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), wheel rate sensors, wheel brake pressure sensors, wheel torque sensors, engine torque sensors, and / or steering angle sensors. Although the braking system 208 is illustrated as being located at... Figure 2 The braking system 208 is located near the vehicle 200, but it can be located anywhere within the vehicle 200.

[0053] Now for reference Figure 3 A schematic diagram of device 300 is illustrated. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to at least one device of vehicle 102 (e.g., at least one device of the system of vehicle 102) and / or one or more devices of network 112 (e.g., one or more devices of the system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of the system of vehicle 102) and / or one or more devices of network 112 (e.g., one or more devices of the system of network 112) include at least one device 300 and / or at least one component of device 300. Figure 3As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0054] Bus 302 includes components for communication between the components of the licensed device 300. In some cases, processor 304 includes a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or accelerated processing unit (APU), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC)). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores data and / or instructions for use by processor 304.

[0055] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0056] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0057] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, or a Wi-Fi interface. ® Interfaces and / or cellular network interfaces, etc.

[0058] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 306 and / or storage component 308, performed by processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0059] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. The software instructions stored in memory 306 and / or storage component 308, when executed, cause processor 304 to perform one or more processes as described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes as described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0060] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0061] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or the memory of another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the memory of the other device, which, when executed by the processor of processor 304 and / or the processor of the other device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0062] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3 Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, the set of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another set of components of device 300.

[0063] Now for reference Figure 4Example block diagrams of autonomous vehicle computing 400 (sometimes referred to as the “AV stack”) are illustrated. As illustrated, autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the autonomous navigation system of the vehicle (e.g., the autonomous vehicle computing 202f of vehicle 200). Alternatively or additionally, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0064] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0065] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the perception system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory line or generates at least one different trajectory line based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to the tactical functions required for the operation of vehicle 102 in road traffic. Tactical efforts involve maneuvering the vehicle in traffic during a journey, including but not limited to deciding whether and when to overtake another vehicle, change lanes, or select appropriate speed, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory line or generates at least one different trajectory line based on the data generated by the positioning system 406.

[0066] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various other types of traffic signals). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0067] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the vehicle's latitude and longitude. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0068] In some embodiments, the control system 408 receives data associated with at least one trajectory line from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory line from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate a powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), a steering control system (e.g., steering control system 206), and / or a braking system (e.g., braking system 208). For example, the control system 408 is configured to perform operational functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes activities required to regulate the y-axis component of the vehicle's motion. Longitudinal vehicle motion control causes activities required to regulate the x-axis component of the vehicle's motion. In an example, where the trajectory line includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers).

[0069] In some embodiments, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.).

[0070] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3(The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0071] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0072] Now for reference Figure 5 A diagram illustrates an implementation 500 of a hierarchical multi-object tracker for automatic annotation. In some embodiments, implementation 500 includes AV computation 510, a hierarchical multi-object tracker 504, and a planning system 506. In some embodiments, AV computation 510 and... Figure 2 The autonomous vehicle calculation 202f is the same as or similar to that of the autonomous vehicle. In some embodiments, the planning system 506 is the same as... Figure 4 The planning system 404 is the same as or similar to that described later. Additionally, in some embodiments, the hierarchical multi-object tracker 504 is similar to that described later. Figure 6 The hierarchical multi-object tracker 612 is the same as or similar.

[0073] A value of 500 indicates an online or offline implementation of a hierarchical multi-object tracker. Sensors (e.g., Figure 2The vehicle 502 uses cameras 202a, LiDAR sensors 202b, radar sensors 202c, and microphones 202d to capture data from the environment for mapping features in the environment to quantitative measurement results. In the online implementation 500, sensor data is evaluated in real time as the vehicle 502's sensors capture data. For example, sensor data is streamed from multiple sensor sources, and AV computation 510 makes predictive decisions to control the vehicle 502. In the offline implementation 500, sensor data is recorded over a period of time as the vehicle 502 navigates through the environment. The sensor data is stored in a driving log for offline processing and / or analysis.

[0074] For ease of description, sensor data captured by specific sensors is described. However, sensors may include, for example, sensors that capture dynamics associated with the vehicle (e.g., inertial measurement units (IMUs), encoders, inertial sensors, and positioning sensors (GNSS receivers)) and sensors that capture information associated with the environment surrounding the vehicle (e.g., cameras, LiDAR, radar, and ultrasonic sensors). In the example, a camera sensor continuously records the surrounding environment in real time. A GPS sensor tracks the vehicle's position in real time. Additionally, in the example, LiDAR and / or radar sensors sense the position and distance of other objects relative to the vehicle in real time. In the example, real-time data is recorded and used for offline implementation.

[0075] In the offline implementation 500, a hierarchical multi-object tracker 504 obtains detection results extracted from recorded sensor data. In some embodiments, the detection results refer to a representation of objects detected in the environment. The detection results are generated by at least one object detection network. The object detection network identifies and locates objects in various frames of sensor data. The hierarchical multi-object tracker 504 determines a trajectory 514 associated with the detection results. In the example, a trajectory refers to a representation of an object trajectory line across sensor data frames. Object tracking associates the detected object with a path in the environment across multiple frames over time. The association of detection results corresponding to the same object forms a trajectory corresponding to that object over a period of time. In some embodiments, the hierarchical multi-object tracker 504 includes multiple tracking stages that iteratively determine trajectory segments (e.g., a series of consecutive detection results) based on the detection results. The trajectory segments are stitched together at the tracking stages to obtain a trajectory. The trajectory is output by the hierarchical multi-object tracker 504. In some embodiments, the trajectory is stored in a driving log using a trajectory identifier (ID), wherein the captured sensor data is labeled with a trajectory ID issued by the hierarchical multi-object tracker 504. In some embodiments, the stored sensor data is labeled based on the detection results output by the object detection network and the tracking ID published by the hierarchical multi-object tracker 504.

[0076] In the online implementation 500, as the vehicle 502 navigates through the environment in real time, the perception system of the vehicle 502 (e.g., Figure 4 The perception system 402) obtains the detection results. The hierarchical multi-object tracker 504 generates a trajectory, and the trajectory is transmitted in real time to the planning system 506 (e.g., Figure 4 Planning system 404). Planning system 506 takes into account trajectory 516 to generate a route with at least one path (e.g., Figure 1 The data associated with route 106 allows the vehicle 502 to travel along at least one route toward its destination. In some embodiments, the route generated by the planning system 506 is at least in part based on the avoidance of traffic conflicts with the trajectory generated by the hierarchical multi-object tracker 504.

[0077] like Figure 5 As described in the example, the hierarchical multi-object tracker 504 is used in an offline implementation to annotate sensor data. Annotating sensor data enables the development and training of various vehicle capabilities. In some embodiments, an automated annotation pipeline including the hierarchical multi-object tracker 504 generates annotations for recorded data, such as driving logs. In the example, the automated annotation pipeline also includes evaluating and annotating a constant-velocity motion model of the recorded sensor data. The automated annotation pipeline reduces the manual resources consumed in annotating data while maintaining a high level of accuracy and consistency in the generated annotations. Furthermore, the automated annotation pipeline enables the rapid and efficient annotation of large datasets, further enabling the rapid development and training of machine learning models. For example, the labeled dataset can be used as training data for neural networks in supervised learning algorithms, as ground truth annotations for evaluating AV stacks, and for other testing and evaluation purposes. In the example, the labeled sensor data is used to train or evaluate components of the AV stack, including, as per [reference to...] Figure 4 The described autonomous navigation system of the vehicle includes a perception system 402, a planning system 404, a positioning system 406, or a control system 408.

[0078] Figure 6 This is a block diagram of the workflow 600 for a hierarchical multi-object tracker used for automatic annotation. The hierarchical multi-object tracker 612 generates trajectories 622 for automatically annotating sensor data. Figure 6 In workflow 600, the hierarchical multi-object tracker 612 outputs a trajectory identifier (ID) for the corresponding detection result 604 input to the hierarchical multi-object tracker 612. In the example, the detection result 604 is input to the first-level tracker 614 to generate trajectory segments 616. The second-level stitcher 618 stitches together trajectory segments associated with the same object to generate a trajectory corresponding to the same object. In some embodiments, sensor data is labeled with trajectories.

[0079] exist Figure 6 In the example, detector 602 is used to generate detection result 604. In the example, detector 602 is an object detection network that uses sensor data to discover objects in the environment. The object detection network generates detection results for individual frames of sensor data. In the example, a frame is a collection of one or more data values ​​captured at a timestamp. A record of sensor data (e.g., a driving log) comprises multiple data frames captured at a series of consecutive timestamps. In the example, the detector is a semantic network used to detect objects in the captured sensor data. The detector outputs detection result 604, which includes bounding boxes associated with the corresponding objects in the environment and a confidence score that quantifies the likelihood of the presence of object category instances (e.g., cars, pedestrians, or bicycles) within the bounding boxes. In the example, the bounding boxes are either two-dimensional or three-dimensional bounding boxes.

[0080] For example, an Image Semantic Network (ISN) or a LiDAR Semantic Network (LSN) outputs a detection result 604, which includes oriented bounding boxes and the probability that an object within the bounding box belongs to a specific category. In this example, the ISN takes image data (e.g., captured by a camera sensor) as input and outputs a set of predicted 2D or 3D bounding boxes of potential objects in the environment, and a set of corresponding confidence scores indicating the presence of object category instances within the bounding boxes. The 2D or 3D bounding boxes include information about the size (e.g., dimensions), orientation, and location of the respective object. In this example, the ISN predicts the category of each pixel in the image and outputs a confidence score for each pixel in the image. Example confidence scores are probability values ​​indicating the probability that a pixel's category is correctly predicted.

[0081] Similarly, an LSN takes at least one point cloud (e.g., captured by LiDAR) as input and outputs a set of predicted 2D or 3D bounding boxes of potential objects in the environment, and a set of confidence scores representing the presence of object class instances within the bounding boxes. In the example, the LSN receives multiple data points representing the environment. For example, each of the multiple data points in at least one point cloud is a set of 3D spatial coordinates (e.g., x, y, z coordinates). Using one or more point clouds, the LSN estimates oriented 2D or 3D bounding boxes of object class instances (such as cars, pedestrians, and cyclists) based on the point clouds. Similar to an ISN, the predicted 2D or 3D bounding boxes output by the LSN include information about the size, orientation, and location of the object. The set of predicted 2D or 3D bounding boxes also includes confidence scores representing the presence of object class instances within the bounding boxes.

[0082] The hierarchical multi-object tracker 612 includes a first-level tracker 614 and a second-level stitcher 618. In the example, the first-level tracker 614 obtains detection results 604. In the example, the first-level tracker 614 is a Kalman filter-based tracker, a stitcher with a short scan time, or a deep learning-based tracker. For example, given object detection results in individual frames of sensor data, a Kalman filter-based tracker uses a Kalman filter to predict and update the position and velocity of the object. The Kalman filter-based tracker predicts the future state (position, velocity, etc.) of the object based on its current state and a mathematical model of its motion. When new measurements (e.g., from sensors) become available, the Kalman filter updates its predictions by combining the predicted state with the new measurements to produce a more accurate estimate using a weighted average, where higher weights are assigned to the estimate with greater certainty. In this way, trajectory segments associated with the object are generated.

[0083] In another example, a stitcher is used as a first-stage tracker, where the stitcher includes a series of filters that determine the likelihood that a detected object belongs to a trajectory segment. These filters include, for example, temporal filters and kinematic filters. The stitcher's scan time refers to the length of time (and the final number of frames) spent evaluating sensor data. A short scan window can be, for example, less than one second or less than half a second. In this example, the length of the scan window is chosen to reduce the runtime of the first-stage tracker 614. The temporal filter evaluates the temporal information associated with the trajectory segments to ensure that the trajectory segments to be stitched are associated with different timestamps (e.g., no identical frames exist among the various trajectory segments). The kinematic filter is applied to trajectory segments that are not associated with the same frame (e.g., trajectory segments appear at different points in time). The kinematic filter evaluates the trajectory segments and prevents the stitching of trajectories that are kinematically infeasible.

[0084] In the example, the deep learning-based tracker includes a machine learning model that receives a set of initial object detection results as input and determines a unique identifier associated with each initial detection result. The deep learning-based tracker then tracks the detected objects as moving objects across sensor data frames.

[0085] like Figure 6As shown in the example, the first-stage tracker 614 outputs a trajectory segment 616. In some embodiments, the trajectory segment is constrained by detection results associated with a corresponding object, because the corresponding trajectory segment is a series of consecutive detection results of a corresponding object in a sensor data frame. A trajectory segment terminates due to a false detection result (e.g., incorrectly detecting a previously detected object as a newly detected object), a false positive detection result (e.g., incorrectly detecting an object as a previously detected object), or a missing detection result (e.g., failing to detect an object present in the data frame). In some embodiments, false detection results, false positive detection results, or missing detection results occur due to sudden changes in the shape and motion of the object (such as during partial occlusion of the object). False detection results, false positive detection results, or missing detection results may also occur due to changes in ambient lighting or illumination. Figure 7 Further description of the trajectory fragments, and about Figure 10A Describe the workflow of the first-level tracker.

[0086] The second-stage stitcher 618 of the hierarchical multi-object tracker 612 stitches trajectory segments 616 from the first-stage tracker 614 to generate an object trajectory. Even with false detections, false positives, or missing detections, the evaluation of the trajectory segments allows it to be determined that the trajectory segments are associated with the same object and therefore correspond to the same trajectory. Instead of stitching together detection results generated by the detector to obtain a trajectory, this technique generates trajectory segments for stitching to reduce the runtime of the hierarchical multi-object tracker 612. Stitching the trajectory segments 616 output by the first-stage tracker 614 consumes fewer resources compared to directly stitching together detection results 604 generated from the detector to form a trajectory. In some embodiments, the second-stage stitcher 618 associates trajectory segments to generate a trajectory by constructing a graph using the trajectory segments 616. This graph further relates to... Figures 11-1 The graph is described in 3. It is used to find the corresponding trajectories associated with each object. In the example, the graph is iteratively updated based on captured sensor data.

[0087] Figure 7 The detection results in a sensor data frame are shown. In the example, the detection results are objects detected in the sensor data and are generated by at least one object detection network. Figure 7In the example, columns 702, 703, 704, 705, 706, and 707 represent sensor data frames at corresponding timestamps in a series of consecutive timestamps. For each timestamp, the box in each row corresponds to the detection result of an object in the environment. The same shading of each box indicates that the box belongs to a specific trajectory segment. The number inside each box represents the trajectory segment ID. Trajectory segments are generated based on the detection results, where the trajectory segments correspond to the detection results at consecutive timestamps. Missing detection results in a frame cause the termination of a trajectory segment. In response to missing detection results, a new trajectory segment is initialized.

[0088] like Figure 7 As shown in the example, the first row 712 of the detection results creates a single trajectory segment with trajectory segment ID 0, where there are no detection results associated with trajectory segment ID 0 in data frame 702. The second row 714 shows the final five detection results associated with the same object. In the second row 714, a detection result is missing at frame 705. For the first three detection results in the second row 714, trajectory segment ID 1 is assigned to the detected object. After the missing detection result at frame 705, a new trajectory segment is initiated for the object with trajectory segment ID 2. Therefore, the detection results in the second row 714 correspond to two trajectory segments, a first trajectory segment with trajectory segment ID 1 and a second trajectory segment with trajectory segment ID 2.

[0089] Similarly, the object detected in row 716 corresponds to two trajectory segments: the first trajectory segment with trajectory segment ID 3 and the second trajectory segment with trajectory segment ID 4. The detection result in row 718 corresponds to a single trajectory segment with trajectory segment ID 5. The detection result in row 720 corresponds to two trajectory segments: the first trajectory segment with trajectory segment ID 6 and the second trajectory segment with trajectory segment ID 7. Figure 7 As shown in the example, the trajectory segment corresponds to a series of consecutive detection results and includes one or more detection results.

[0090] Such as about Figure 6 The trajectory segments discussed are generated by a first-level tracker (e.g., Figure 6 The first-level tracker (614) is generated. The first-level tracker can be, for example, a Kalman filter-based tracker, a stitcher with a short scan window, or a deep learning-based tracker. Figure 7 As shown, each trajectory segment is a series of one or more consecutive detection results. Figure 7 In the example, detection results with the same trajectory segment ID are used to display the individual trajectory segments. In some embodiments, the second-level stitcher (e.g., Figure 6The second-level stitcher 618 generates trajectories from trajectory fragments. In the second-level tracker, trajectory fragments are stitched together to form the corresponding trajectories for each detected object. In some embodiments, a bipartite graph is constructed based on the trajectory fragments to determine object trajectories using graph-based object tracking. The trajectory fragments form nodes of a weighted graph, where each node of the weighted graph corresponds to the detection result of the corresponding trajectory fragment, and the edges of the graph are weighted according to a first cost metric generated by the first-level tracker. The weighted trajectory graph is converted into a bipartite graph, where each node of the weighted trajectory graph is represented by two disjoint and independent sets (U, V), where each edge connects to a node in the first set of nodes (U) and a node in the second set of nodes (V). Trajectories are generated from the bipartite graph by determining the optimal path cover using the bipartite graph. In the example, the optimal path cover in the bipartite graph is the set of vertex-disjoint paths that cover the vertices in the graph and minimize the number of paths.

[0091] Figure 8 A weighted graph is shown. Figure 8 In the example, the weighted trajectory map 800 includes detection results for three frames at timestamps t0, t1, and t2. For each frame, two detection results are shown. Therefore, at timestamp t0, detection results z1 and z4 appear; at timestamp t1, detection results z2 and z5 appear; and at timestamp t2, detection results z3 and z6 appear. The first-level tracker generates two trajectory segments: a first trajectory segment 802 (shown by dashed lines) that associates detection results z1, z2, and z6; and a second trajectory segment 804 (shown by dashed lines) that associates detection results z4 and z5. Detection result z3 represents a third single detection result trajectory segment 806.

[0092] The corresponding weights associated with each edge of the weighted graph are shown as w0, w1, w2, w3, w4, w5, w6, and w7. For example... Figure 8 As shown in the example, each detection result is a node in a weighted graph because each edge is associated with a weight, which is the cost representing the likelihood that the first and second detection results are associated with the same object.

[0093] Figure 9A The diagram shows a weighted graph 900A where nodes are divided into two disjoint and independent sets. Each node corresponds to the detection result of a trajectory segment. Figure 9A As shown, the nodes of a weighted graph are represented by a first set of nodes (') associated with incoming edges and a second set of nodes ('') associated with outgoing edges. Figure 9A As shown, the weights are updated, and each edge is associated with a weight.

[0094] Figure 9B A bipartite graph 900B is shown. Each node in the bipartite graph is associated with a timestamp, trajectory segment ID, and state distribution. In some embodiments, the edge weights of the weighted graph are assumed to be Gaussian. In this example, the assumed distribution allows for the prediction of the likelihood of the corresponding weight occurring. The transformation from a weighted graph to a bipartite graph can leverage the properties of the Gaussian distribution to improve the performance or accuracy of the trajectory derived from the bipartite graph. Figure 9A The shown dividing node, Figure 8 The weighted graph is transformed into Figure 9B The diagram shows a bipartite graph. Using a bipartite graph, the trajectory of each object is determined by finding the optimal path cover. In some embodiments, a path cover is a set of directed paths such that each node U, V belongs to at least one path. The optimal path cover is the path with the lowest cost. By selecting the path with the lowest cost, trajectory segments are stitched together to form a trajectory, because the cost between two trajectory segments associated with the same object will be the lowest cost. In some embodiments, the optimal path cover is found by solving a bipartite matching problem.

[0095] Figure 10A The workflow 1000A corresponding to the first-level tracker is shown. In this example, the first-level tracker is... Figure 6 The first-stage tracker 614. The first-stage tracker can be, for example, a Kalman filter-based tracker. At block 1002, the detection result is obtained. In some embodiments, the detection result is obtained from a detector such as an object detection network.

[0096] At block 1004, the trajectory is correlated with the detection result.

[0097] The unmatched detection result 1006 is input into block 1010. At block 1010, a trajectory is established.

[0098] At block 1012, the unmatched trajectory 1008 remains an unassociated trajectory. In some embodiments, the unassociated trajectory (1018) is terminated after a predetermined time period. In some embodiments, the predetermined time period is two seconds.

[0099] At block 1014, a 3D Kalman filter is used to generate trajectory segment 1016.

[0100] In some embodiments, for each detection result, a state vector is generated to be input into a Kalman filter, wherein the state vector is as follows: X=(x, y, z, theta, L, W, H, confidence, vx, vy, vz).

[0101] From the detector (e.g., Figure 6 The detector 602) obtains this information X. The state space and measurement space of the Kalman filter are identical. In the example, x, y, and z indicate the position of the detection result, theta represents the orientation of the detection result, and L, W, and H represent the length, width, and height associated with the detection result. Confidence represents the likelihood that the detection result is correctly classified as an instance of a specific object category. Variables vx, vy, and vz represent the velocity of the object in the x, y, and z directions. The state vector represents the state space of the Kalman filter updated over time, thus combining information from a constant velocity motion model to track detected objects. The measurement space represents the variables directly measured by the sensor. In some embodiments, the state space and measurement space are identical to the state vector extracted from the captured sensor data.

[0102] In the example, the 3D Kalman filter predicts trajectory segment 1016 and provides measurement updates to the associated trajectory. The measurement updates are input to block 1004, where the trajectory and detection results are associated. As a first-stage tracker, the 3D Kalman filter generates short trajectory segments. In this example, the short trajectory segments are consecutive detection results associated with the object. For each trajectory segment, if there is no detection result associated with a timestamp, the trajectory segment is terminated and output as trajectory segment 1016.

[0103] For the associated trajectory at block 1004, a measurement update is applied from the output of the Kalman filter. For unassociated trajectories, such as the unmatched trajectory 1008, they are propagated without a measurement update. For example, an unassociated trajectory coasts, meaning there are no measurements associated with the trajectory propagated by the constant velocity motion model. Therefore, trajectory segments are allowed to float for two or three seconds until the trajectory terminates due to a lack of measurements. When a measurement is detected for a trajectory, a new trajectory is created (e.g., generated) for the new measurement period.

[0104] Figure 10B The workflow of the second-level tracker 1000B is shown. In the example, the second-level tracker is... Figure 6 The stitcher 618. The second-level tracker is graph-based, using the binary representation described above. Each trajectory segment 1016 is stored as a node in the stitcher, containing: timestamp, trajectory segment ID, and state distribution (assumed to be Gaussian). ).

[0105] When constructing a (1020) bipartite graph, edge weights are calculated based on the cost. Edges with corresponding weights in the weighted graph are transformed or mapped to corresponding weighted edges in the bipartite graph. In the example, the second cost metric used to weight the edges of the bipartite graph is the negative logarithm of kinematic likelihood, type likelihood, likelihood based on intersection over union (IoU) / generalized intersection over union (GIoU), etc. The second cost metric is derived at least in part based on the first cost metric used to weight the edges of the corresponding weighted graph (e.g., timestamps, headings, or velocities of detection results). In the example, kinematic likelihood refers to the likelihood that motion from one trajectory segment leads to the next trajectory segment. Type likelihood refers to the predicted classification match from one trajectory segment to the next. IoU / GIoU is defined using the location of the bounding boxes. For example, if the bounding boxes from one trajectory segment to the next trajectory segment are located close to each other, they may come from the same object, depending on how much they overlap. To construct the graph, newly added nodes are trajectory segments created from new output trajectory segments from the first-level tracker. Newly added edges lie between existing and new nodes and satisfy evaluation criteria to ensure feasible time, position, and velocity variations across these edges.

[0106] Once the bipartite graph is established, it is solved (1022). To solve the bipartite graph, a cost matrix is ​​constructed. In the cost matrix, for each edge, the "from" node is in the row and the "to" node is in the column. Minimum-cost bipartite matching is used to find the optimal path cover. In minimum-cost bipartite matching, the minimum cost is found for a complete matching between two subsets of vertices in a given weighted bipartite graph.

[0107] The bipartite graph (1024) is parsed to eliminate false positives in tracking. To parse the graph, the spliced ​​trajectory obtained according to the optimal path coverage is pruned. In some embodiments, if the head trajectory fragment (e.g., the initial trajectory fragment associated with the corresponding trajectory) has exceeded a predetermined time length (e.g., nscan_time) in the past, the accumulated state history stored in the circular buffer for the head trajectory fragment is passed to its descendant trajectory fragments. In the example, the descendant trajectory fragment is the next trajectory fragment of the corresponding trajectory in chronological order. The head trajectory fragment ID is used as the ID of the descendant node. The node corresponding to the head trajectory fragment is deleted, and the incoming edges to the descendant nodes are also deleted.

[0108] In some embodiments, parsing the bipartite graph includes removing the tail of a trajectory (e.g., the end of the trajectory) to improve the recall of the tracker output, thereby achieving higher quality automatically labeled data. In some embodiments, the tracker outputs a trajectory when it is confirmed. To reduce false positives, a predetermined number of measurement updates are used to confirm the trajectory. Additionally, during graph parsing, the first two frames of each trajectory are discarded, which reduces recall. Therefore, to increase recall, the confirmation status of the first two frames of each trajectory is changed from "provisional" to "confirmed" by backtracking the trajectory using future information to output the confirmed trajectory.

[0109] In some embodiments, the trajectory is terminated after a predetermined time period (such as one or two seconds) after no additional measurements are associated with the corresponding trajectory. Since the bounding box propagation is based on a constant velocity motion model, the output consists of the bounding boxes from the trajectory at the timestamp without any measurement updates. Using a constant velocity motion model, bounding boxes exist even without measurement updates. However, these boxes are likely to lack matching ground truth values ​​and are considered false positives. Post-processing is used to eliminate these bounding boxes in the absence of measurements from the output, thereby reducing false positives. This is similar to removing the first few frames of the trajectory, as the tracker uses some frames with measurements to confirm the trajectory. Therefore, to increase recall, tail cutting is implemented for each trajectory to prevent these boxes from being output when no additional measurement updates are available. After parsing the bipartite graph, the trajectory is output. In some embodiments, the output trajectory includes annotation of the sensor data using trajectory IDs.

[0110] Now for reference Figure 11 A flowchart of a process 1100 for an automatic annotation hierarchical offline multi-object tracker is shown. In some embodiments, one or more steps described with respect to process 1100 are performed by, for example... Figure 6 The hierarchical offline multi-object tracker 612 and other hierarchical multi-object trackers (e.g., fully and / or partially, etc.) are used. Alternatively or additionally, in some embodiments, one or more steps described with respect to processing 600 are performed by a hierarchical offline multi-object tracker 612 that is separate from or includes the hierarchical offline multi-object tracker 612. Figure 5 The AV calculation is performed by other devices or groups of devices such as 510 (e.g., completely and / or partially).

[0111] At block 1102, a detection result corresponding to the observed object in the sensor data captured from the environment is obtained. In this example, the detection result represents the detected object representing a potential object of interest identified in the sensor data. In some embodiments, the detection result includes the output of an ISN, LSN, or other object detector.

[0112] At block 1104, a trajectory segment is generated based on the detection results. In the example, methods such as... Figure 6 A first-level tracker, such as 614, is used to generate trajectory fragments. The tracker outputs trajectory fragments to determine which detection corresponds to the object being tracked. In some embodiments, the first-level tracker is a stitcher, a machine learning-based tracker, or a Kalman filter-based tracker. In the example, the trajectory fragment includes nodes representing detections corresponding to a single object at a timestamp.

[0113] At block 1106, a weighted trajectory graph is generated based on the trajectory segments. Weights are assigned to the edges between nodes of the trajectory segments according to at least one first cost metric associated with a node. In the example, the cost metric corresponds to the timestamp, heading, or velocity of the detection result.

[0114] At block 1108, the weighted trajectory graph is converted into a bipartite graph, where nodes of the bipartite graph correspond to corresponding detection results of trajectory segments, and weights are assigned to the edges of the bipartite graph based on at least one second cost metric associated with the nodes. In the example, the transformation creates a bipartite graph from all trajectory segments in the weighted graph, where all trajectory segments include multiple trajectory segments corresponding to a single object (e.g., an object with two trajectory segments due to missing detection results or false detection results).

[0115] In the example, at least a first cost metric (e.g., a timestamp, heading, or velocity of a detection result) is used in part to derive at least one second cost metric associated with the corresponding node. For example, the weighted graph is converted into a bipartite graph by partitioning the individual vertices in the weighted graph. In the bipartite graph, the two vertices belong to different sets. In some embodiments, for each edge a with weights in the weighted graph, a corresponding edge with weights as a second cost metric is created in the bipartite graph, at least in part based on the first cost metric. Alternatively, in some embodiments, for each edge a with weights in the weighted graph, a corresponding edge with the same weights as in the weighted graph is created in the bipartite graph, thereby preserving the weighted nature of the graph.

[0116] At block 1110, the bipartite graph is solved to determine at least one optimal path coverage. Solving the bipartite graph is used to determine the complete trajectory of each detected object. At block 1112, the solved bipartite graph is parsed to obtain a trajectory representing at least one optimal path coverage corresponding to the respective observed object, wherein frames of the trajectory that do not meet a predetermined threshold are discarded. In the example, the parsing includes backtracking the trajectory and removing the tail of the trajectory to improve the recall of the tracker output, which will provide automatically labeled data with higher quality.

[0117] At block 1114, sensor data is updated by labeling the detection results with trajectory identifiers corresponding to the trajectory covered by the best path representing the corresponding observed object.

[0118] Example

[0119] According to some non-limiting embodiments or examples, a method is provided, comprising: obtaining detection results corresponding to observed objects in sensor data; generating a trajectory segment based on the detection results, wherein the trajectory segment includes nodes and edges, the nodes representing corresponding observed objects at a series of consecutive timestamps, the edges connecting the nodes at the series of consecutive timestamps; generating a weighted trajectory graph based on the trajectory segment, wherein weights are assigned to the edges between the nodes according to at least one first cost metric associated with the nodes of the trajectory segment; converting the weighted trajectory graph into a bipartite graph, wherein the nodes of the bipartite graph correspond to the corresponding detection results of the trajectory segment, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes; solving the bipartite graph to determine at least one optimal path coverage corresponding to the corresponding observed object; parsing the solved bipartite graph to obtain a trajectory representing the at least one optimal path coverage corresponding to the corresponding observed object, wherein frames of the trajectory that do not meet a predetermined threshold are discarded; and labeling the detection results using a trajectory identifier corresponding to the trajectory representing the optimal path coverage of the corresponding observed object.

[0120] According to some non-limiting embodiments or examples, a system is provided, comprising: at least one processor and at least one non-transitory storage medium storing instructions, which, when executed by the at least one processor, cause the at least one processor to: obtain detection results corresponding to observed objects in sensor data; generate trajectory segments based on the detection results, wherein the trajectory segments include nodes and edges, the nodes representing corresponding observed objects at a series of consecutive timestamps, and the edges connecting the nodes at the series of consecutive timestamps; and generate a weighted trajectory graph based on the trajectory segments, wherein, according to at least one first cost metric associated with a node of the trajectory segment, the graph is weighted according to the node. Weights are assigned to the edges between points; the weighted trajectory graph is converted into a bipartite graph, wherein the nodes of the bipartite graph correspond to the corresponding detection results of the trajectory segments, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes; the bipartite graph is solved to determine at least one optimal path coverage corresponding to the corresponding observation; the solved bipartite graph is parsed to obtain trajectories representing the at least one optimal path coverage corresponding to the corresponding observation, wherein frames of the trajectories that do not meet a predetermined threshold are discarded; and the detection results are labeled using trajectory identifiers corresponding to the trajectories representing the optimal path coverage of the corresponding observations.

[0121] According to some non-limiting embodiments or examples, at least one non-transitory storage medium is provided, which stores instructions that, when executed by at least one processor, cause the at least one processor to: obtain detection results corresponding to observed objects in sensor data; generate a trajectory segment based on the detection results, wherein the trajectory segment includes nodes and edges, the nodes representing corresponding observed objects at a series of consecutive timestamps, and the edges connecting the nodes at the series of consecutive timestamps; and generate a weighted trajectory graph based on the trajectory segment, wherein weights are assigned to the edges between the nodes according to at least one first cost metric associated with the nodes of the trajectory segment. The weighted trajectory graph is converted into a bipartite graph, wherein nodes of the bipartite graph correspond to corresponding detection results of the trajectory segments, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes; the bipartite graph is solved to determine at least one optimal path coverage corresponding to the corresponding observation; the solved bipartite graph is parsed to obtain trajectories representing the at least one optimal path coverage corresponding to the corresponding observation, wherein frames of the trajectories that do not meet a predetermined threshold are discarded; and the detection results are labeled using trajectory identifiers corresponding to the trajectories representing the optimal path coverage of the corresponding observations.

[0122] Further non-limiting aspects or embodiments are illustrated in the following numbered examples:

[0123] Example 1: A method comprising: obtaining detection results corresponding to observed objects in sensor data; generating a trajectory segment based on the detection results, wherein the trajectory segment includes nodes and edges, the nodes representing corresponding observed objects at a series of consecutive timestamps, the edges connecting the nodes at the series of consecutive timestamps; generating a weighted trajectory graph based on the trajectory segment, wherein weights are assigned to the edges between the nodes according to at least one first cost metric associated with the nodes of the trajectory segment; converting the weighted trajectory graph into a bipartite graph, wherein the nodes of the bipartite graph correspond to the corresponding detection results of the trajectory segment, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes; solving the bipartite graph to determine at least one optimal path coverage corresponding to the corresponding observed object; parsing the solved bipartite graph to obtain a trajectory representing the at least one optimal path coverage corresponding to the corresponding observed object, wherein frames of the trajectory that do not meet a predetermined threshold are discarded; and labeling the detection results using a trajectory identifier corresponding to the trajectory representing the optimal path coverage of the corresponding observed object.

[0124] Example 2: The method according to any of the preceding claims, wherein obtaining the detection result includes: obtaining a detection result comprising a three-dimensional bounding box corresponding to the position, size and velocity of the object.

[0125] Example 3: The method according to any of the preceding claims, wherein generating the trajectory segment includes: generating the trajectory segment using a three-dimensional Kalman filter.

[0126] Example 4: The method according to any of the preceding claims includes: assigning corresponding timestamps, corresponding trajectory segment identifiers and corresponding state distributions to the nodes of the trajectory segment.

[0127] Example 5: The method according to any of the preceding claims, wherein the at least one second cost metric includes kinematic likelihood, type likelihood, likelihood based on the cross-union ratio associated with the trajectory segment, or any combination thereof.

[0128] Example 6: The method according to any of the preceding claims, wherein solving the bipartite graph includes: minimum bipartite matching to determine the at least one optimal path cover corresponding to the respective observation object.

[0129] Example 7: The method according to any of the preceding claims, wherein parsing the bipartite graph includes: removing nodes corresponding to the discarded frames from the bipartite graph, wherein when removing a node from the bipartite graph, an identifier and state history are passed to descendant nodes, and the incoming edges of the removed nodes are deleted from the bipartite graph.

[0130] Example 8: A system comprising: at least one processor and at least one non-transitory storage medium storing instructions, which, when executed by the at least one processor, cause the at least one processor to: obtain detection results corresponding to observed objects in sensor data; generate a trajectory segment based on the detection results, wherein the trajectory segment includes nodes and edges, the nodes representing corresponding observed objects at a series of consecutive timestamps, and the edges connecting the nodes at the series of consecutive timestamps; and generate a weighted trajectory graph based on the trajectory segment, wherein edges between the nodes are assigned according to at least one first cost metric associated with a node of the trajectory segment. Weighting; converting the weighted trajectory graph into a bipartite graph, wherein nodes of the bipartite graph correspond to corresponding detection results of the trajectory segments, and assigning weights to the edges of the bipartite graph according to at least one second cost metric associated with the nodes; solving the bipartite graph to determine at least one optimal path coverage corresponding to the corresponding observation; parsing the solved bipartite graph to obtain trajectories representing the at least one optimal path coverage corresponding to the corresponding observation, wherein frames of the trajectories that do not meet a predetermined threshold are discarded; and labeling the detection results using trajectory identifiers corresponding to the trajectories representing the optimal path coverage of the corresponding observations.

[0131] Example 9: The system according to any of the preceding claims, wherein the detection result is a three-dimensional bounding box corresponding to the position, size, and velocity of the object.

[0132] Example 10: The system according to any of the preceding claims, wherein the trajectory segment is generated using a three-dimensional Kalman filter.

[0133] Example 11: The system according to any of the preceding claims, wherein a corresponding timestamp, a corresponding trajectory segment identifier, and a corresponding state distribution are assigned to the nodes of the trajectory segment.

[0134] Example 12: A system according to any of the preceding claims, wherein the at least one second cost metric includes kinematic likelihood, type likelihood, likelihood based on the cross-union ratio associated with the trajectory segment, or any combination thereof.

[0135] Example 13: The system according to any of the preceding claims, wherein solving the bipartite graph includes: minimum bipartite matching to determine the at least one optimal path cover corresponding to the respective observation object.

[0136] Example 14: The system according to any of the preceding claims, wherein parsing the bipartite graph includes: removing nodes corresponding to discarded frames from the bipartite graph, wherein when removing a node from the bipartite graph, an identifier and state history are passed to descendant nodes, and the incoming edges of the removed nodes are deleted from the bipartite graph.

[0137] Example 15: At least one non-transitory storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtain detection results corresponding to observed objects in sensor data; generate trajectory segments based on the detection results, wherein the trajectory segments include nodes and edges, the nodes representing corresponding observed objects at a series of consecutive timestamps, and the edges connecting the nodes at the series of consecutive timestamps; generate a weighted trajectory graph based on the trajectory segments, wherein weights are assigned to the edges between the nodes according to at least one first cost metric associated with the nodes of the trajectory segments; convert the weighted trajectory graph into a bipartite graph, wherein the nodes of the bipartite graph correspond to the corresponding detection results of the trajectory segments, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes; solve the bipartite graph to determine at least one optimal path coverage corresponding to the corresponding observed object; parse the solved bipartite graph to obtain trajectories representing the at least one optimal path coverage corresponding to the corresponding observed object, wherein frames of the trajectory that do not meet a predetermined threshold are discarded; and label the detection results using trajectory identifiers corresponding to the trajectories representing the optimal path coverage of the corresponding observed objects.

[0138] Example 16: At least one non-transitory storage medium according to any of the preceding claims, wherein obtaining the detection result includes: obtaining a detection result including a three-dimensional bounding box corresponding to the position, size and velocity of the object.

[0139] Example 17: At least one non-transitory storage medium according to any of the preceding claims, wherein generating a trajectory segment includes: generating a trajectory segment using a three-dimensional Kalman filter.

[0140] Example 18: At least one non-transitory storage medium according to any of the preceding claims, comprising: assigning a corresponding timestamp, a corresponding trajectory segment identifier, and a corresponding state distribution to the nodes of the trajectory segment.

[0141] Example 19: At least one non-transitory storage medium according to any of the preceding claims, wherein the at least one second cost metric includes kinematic likelihood, type likelihood, likelihood based on the cross-union ratio associated with the trajectory segment, or any combination thereof.

[0142] Example 20: At least one non-transitory storage medium according to any of the preceding claims, wherein solving the bipartite graph includes: minimum bipartite matching to determine the at least one optimal path coverage corresponding to the corresponding observation object.

[0143] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and the applicant's expectation that the scope of this invention is defined in the specific form of the claims published in this application, is the literal and equivalent scope of the claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as used in the claims. Furthermore, when the term "comprising" is used in the preceding specification or appended claims, the following phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method comprising: Obtain detection results corresponding to the observed objects in the sensor data; A trajectory segment is generated based on the detection results. The trajectory segment includes nodes and edges. The nodes represent corresponding observation objects at a series of consecutive timestamps, and the edges connect the nodes at the series of consecutive timestamps. A weighted trajectory graph is generated based on the trajectory segment, wherein weights are assigned to edges between nodes according to at least one first cost metric associated with a node of the trajectory segment; The weighted trajectory graph is converted into a bipartite graph, wherein the nodes of the bipartite graph correspond to the corresponding detection results of the trajectory segments, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes. Solve the bipartite graph to determine at least one optimal path cover corresponding to the respective observation object; The solved bipartite graph is analyzed to obtain a trajectory representing the coverage of at least one optimal path corresponding to the respective observed object, wherein frames of the trajectory that do not satisfy a predetermined threshold are discarded; and The detection results are labeled using trajectory identifiers corresponding to the trajectory covered by the best path representing the corresponding observed object.

2. The method according to claim 1, wherein, The test results include: Obtain detection results including three-dimensional bounding boxes corresponding to the position, size, and velocity of the object.

3. The method according to claim 1, wherein, The generation of trajectory segments includes: generating trajectory segments using a three-dimensional Kalman filter.

4. The method according to claim 1, comprising: Assign corresponding timestamps, corresponding trajectory segment identifiers, and corresponding state distributions to the nodes of the trajectory segment.

5. The method according to claim 1, wherein, The at least one second cost metric includes kinematic likelihood, type likelihood, likelihood based on the crossover ratio associated with the trajectory segment, or any combination thereof.

6. The method according to claim 1, wherein, Solving the bipartite graph involves: minimum bipartite matching to determine at least one optimal path cover corresponding to the respective observation.

7. The method according to claim 1, wherein, Parsing the bipartite graph includes: removing the node corresponding to the discarded frame from the bipartite graph, wherein when removing a node from the bipartite graph, the identifier and state history are passed to descendant nodes, and the incoming edges of the removed node are deleted from the bipartite graph.

8. A system comprising: At least one processor, and At least one non-transitory storage medium, which stores instructions that, when executed by the at least one processor, cause the at least one processor to: Obtain detection results corresponding to the observed objects in the sensor data; A trajectory segment is generated based on the detection results. The trajectory segment includes nodes and edges. The nodes represent corresponding observation objects at a series of consecutive timestamps, and the edges connect the nodes at the series of consecutive timestamps. A weighted trajectory graph is generated based on the trajectory segment, wherein weights are assigned to edges between nodes according to at least one first cost metric associated with a node of the trajectory segment; The weighted trajectory graph is converted into a bipartite graph, wherein the nodes of the bipartite graph correspond to the corresponding detection results of the trajectory segments, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes. Solve the bipartite graph to determine at least one optimal path cover corresponding to the respective observation object; The solved bipartite graph is analyzed to obtain a trajectory representing the coverage of at least one optimal path corresponding to the respective observed object, wherein frames of the trajectory that do not satisfy a predetermined threshold are discarded; and The detection results are labeled using trajectory identifiers corresponding to the trajectory covered by the best path representing the corresponding observed object.

9. The system according to claim 8, wherein, The detection result is a three-dimensional bounding box corresponding to the position, size, and velocity of the object.

10. The system according to claim 8, wherein, The trajectory segment is generated using a three-dimensional Kalman filter.

11. The system according to claim 8, wherein, Assign corresponding timestamps, corresponding trajectory segment identifiers, and corresponding state distributions to the nodes of the trajectory segment.

12. The system according to claim 8, wherein, The at least one second cost metric includes kinematic likelihood, type likelihood, likelihood based on the crossover ratio associated with the trajectory segment, or any combination thereof.

13. The system according to claim 8, wherein, Solving the bipartite graph involves: minimum bipartite matching to determine at least one optimal path cover corresponding to the respective observation.

14. The system according to claim 8, wherein, Parsing the bipartite graph includes: removing the node corresponding to the discarded frame from the bipartite graph, wherein when removing a node from the bipartite graph, the identifier and state history are passed to descendant nodes, and the incoming edges of the removed node are deleted from the bipartite graph.

15. At least one non-transitory storage medium, which stores instructions that, when executed by at least one processor, cause the at least one processor to: Obtain detection results corresponding to the observed objects in the sensor data; A trajectory segment is generated based on the detection results, wherein, The trajectory segment includes nodes and edges, where the nodes represent corresponding observation objects at a series of consecutive timestamps, and the edges connect the nodes at the series of consecutive timestamps; A weighted trajectory graph is generated based on the trajectory segment, wherein weights are assigned to edges between nodes according to at least one first cost metric associated with a node of the trajectory segment; The weighted trajectory graph is converted into a bipartite graph, wherein the nodes of the bipartite graph correspond to the corresponding detection results of the trajectory segments, and weights are assigned to the edges of the bipartite graph according to at least one second cost metric associated with the nodes. Solve the bipartite graph to determine at least one optimal path cover corresponding to the respective observation object; The solved bipartite graph is analyzed to obtain a trajectory representing the coverage of at least one optimal path corresponding to the respective observed object, wherein frames of the trajectory that do not satisfy a predetermined threshold are discarded; and The detection results are labeled using trajectory identifiers corresponding to the trajectory covered by the best path representing the corresponding observed object.

16. The at least one non-transitory storage medium according to claim 15, wherein, The test results include: Obtain detection results including three-dimensional bounding boxes corresponding to the position, size, and velocity of the object.

17. The at least one non-transitory storage medium according to claim 15, wherein, The generation of trajectory segments includes: generating trajectory segments using a three-dimensional Kalman filter.

18. The at least one non-transitory storage medium according to claim 15, comprising: Assign corresponding timestamps, corresponding trajectory segment identifiers, and corresponding state distributions to the nodes of the trajectory segment.

19. The at least one non-transitory storage medium according to claim 15, wherein, The at least one second cost metric includes kinematic likelihood, type likelihood, likelihood based on the crossover ratio associated with the trajectory segment, or any combination thereof.

20. The at least one non-transitory storage medium according to claim 15, wherein, Solving the bipartite graph involves: minimum bipartite matching to determine at least one optimal path cover corresponding to the respective observation.

Images