Tracking object trace segment cleanup

Abstract

Methods for clearing trace segments of tracked objects using neural networks are provided. These methods may include detecting a first trace segment and a second trace segment. The method includes applying a trained machine learning model to determine whether the first and second trace segments capture a real object and whether the first and second trace segments represent the same object outside a vehicle. The method also includes combining the first and second trace segments to form a single trace segment with a single trajectory in response to the first and second trace segments being determined to represent the same object. System and computer program products are also provided.

Description

Technical Field

[0001] This invention relates to a technique for cleaning up trace segments of tracked objects. Background Technology

[0002] Autonomous vehicles are vehicles capable of sensing and navigating their environment without human input. Autonomous vehicles rely on multiple types of sensors to perceive their surroundings. These sensors provide the autonomous vehicle with data representative of its environment. The autonomous vehicle then applies various processing techniques to this data to identify objects in its vicinity. Identifying these objects provides the autonomous vehicle with the information needed to safely navigate driving scenarios.

[0003] Modern object detection and tracking systems require large amounts of data on the tracked objects to train neural network models for online deployment. Automatic labeling systems, or offline sensing, have been developed to automatically annotate tracking boxes based on raw data. The quality of annotation is crucial for the successful training of models used in online deployment. Segmented tracks and false alarms are common occurrences in both online and offline object detection and tracking systems. Segmented tracks from the same road user, along with false alarms, can lead autonomous vehicles to make incorrect decisions for safe navigation. Summary of the Invention

[0004] According to a first aspect of the invention, a method includes: using at least one processor to detect a first trace segment and a second trace segment; using the at least one processor to determine that the first trace segment and the second trace segment represent the same object outside a vehicle; using the at least one processor to determine a first trace segment quality score associated with the first trace segment and a second trace segment quality score associated with the second trace segment, as well as a splicing score between the first trace segment and the second trace segment; and in response to determining that the first trace segment and the second trace segment represent the same object because the first trace segment quality score and the second trace segment quality score meet a quality score threshold and the splicing score meets a splicing score threshold, using the at least one processor to combine the first trace segment and the second trace segment to form a single trace segment having a single trajectory.

[0005] According to a second aspect of the invention, a system includes: at least one processor; and at least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to operate, the operation including: detecting a first track segment and a second track segment; determining that the first track segment and the second track segment represent the same object outside a vehicle; determining a first track segment quality score associated with the first track segment and a second track segment quality score associated with the second track segment, and a splicing score between the first track segment and the second track segment; and, in response to determining that the first track segment and the second track segment represent the same object because the first track segment quality score and the second track segment quality score satisfy a quality score threshold and the splicing score satisfies a splicing score threshold, combining the first track segment and the second track segment to form a single track segment having a single trajectory and a single cloud point feature.

[0006] According to a third aspect of the invention, a non-transitory computer-readable storage medium includes at least one program executable by one or more processors of a first device, the at least one program including instructions that, when executed by the one or more processors, cause the first device to operate, the operation including: detecting a first trace segment and a second trace segment; determining that the first trace segment and the second trace segment represent the same object outside a vehicle; determining a first trace segment quality score associated with the first trace segment and a second trace segment quality score associated with the second trace segment, and a splicing score between the first trace segment and the second trace segment; and, in response to determining that the first trace segment and the second trace segment represent the same object because the first trace segment quality score and the second trace segment quality score satisfy a quality score threshold and the splicing score satisfies a splicing score threshold, combining the first trace segment and the second trace segment to form a single trace segment having a single trajectory and a single cloud point feature.

[0007] According to a fourth aspect of the invention, a method includes: using at least one processor to detect a first trace segment and a second trace segment; using the at least one processor to determine a first trace segment quality score associated with the first trace segment and a second trace segment quality score associated with the second trace segment; using the at least one processor to determine a splicing score between the first trace segment and the second trace segment; and in response to the first trace segment quality score and the second trace segment quality score satisfying a quality score threshold and the splicing score satisfying a splicing score threshold, using the at least one processor to combine the first trace segment and the second trace segment to form a single trace segment representing the same object outside a vehicle. Attached Figure Description

[0008] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system;

[0009] Figure 2 It is a diagram of one or more systems that include autonomous vehicles;

[0010] Figure 3 yes Figure 1 and Figure 2 A diagram of one or more devices and / or one or more system components;

[0011] Figure 4A It is a diagram of some components of an autonomous system;

[0012] Figure 4B This is a diagram illustrating the implementation of a neural network;

[0013] Figure 5 This is a diagram illustrating the implementation of trace segment cleanup;

[0014] Figure 6 This is a diagram illustrating the implementation of the trace segment cleanup data stream;

[0015] Figure 7 This is a diagram illustrating the implementation of the trace feature extraction network;

[0016] Figure 8 This is a diagram illustrating the implementation of the trace segment cleanup module;

[0017] Figure 9 It is a diagram of the implementation of a training sample generator configured to generate samples for training machine learning models;

[0018] Figure 10 This is a flowchart used to match trace segments with ground truth traces to generate training samples;

[0019] Figure 11 It is a graph illustrating the implementation of matching ground truth traces with trace segments; and

[0020] Figure 12 This is a flowchart of the process used for cleanup of trace segments using a machine learning model. Detailed Implementation

[0021] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0022] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0023] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0024] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0025] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0026] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet that includes data (e.g., a data packet, etc.).

[0027] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0028] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0029] General Overview

[0030] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement trace segment cleanup of the data representation of a tracked object. The trace segment cleanup system can use machine learning systems to identify and remove potentially erroneous trace segments. A trace segment is data associated with a perceived object encountered by a vehicle (such as an autonomous vehicle). Trace segment cleanup is a technique that eliminates erroneous trace segments or combines erroneous trace segments with other trace segments to capture the complete trace of the tracked object. Examples of erroneous trace segments include redundant, disconnected, or false trace segments caused by poor perception of objects outside the vehicle or failure to identify object movement as belonging to the same object. To eliminate erroneous trace segments, the trace segment cleanup system can eliminate or combine trace segments. This elimination or combination reduces the computational resources required to monitor objects outside the vehicle and ensures that the vehicle's planning system makes safe planned movements in response to various driving scenarios.

[0031] As an example technique, a track segment cleanup system detects two track segments as potential candidates to be combined into a single track segment. After detecting track segment candidates, the track segment cleanup system applies a trained machine learning model to determine whether the first and second track segments represent the same object around the autonomous vehicle. The track segment cleanup system can combine the first and second track segments to form a single track segment. For example, the track segment cleanup system determines to combine the first and second track segments based on the determination that the first track segment is associated with a bicycle in a first time period and the second track segment is associated with the same bicycle in a second time period. The track segment cleanup system combines the individual trajectory and / or cloud point features of the first and second track segments. The combined track segment then has a single set of trajectories and a single set of cloud point features. Track segment cleanup increases the likelihood of accurate movement planning by a planning system utilizing the vehicle by removing erroneous track segments. In this way, the planning system has an accurate representation of the objects around the vehicle. In addition, the trace segment cleanup system eliminates or combines erroneous trace segments, thereby reducing the computational resources required to run the planning system and ensuring that the planning system makes safe planned movements in response to various driving scenarios.

[0032] This paper provides techniques for cleaning up trace segments of tracked objects through the implementation of the systems, methods, and computer program products described herein. Unlike other object tracking modules in machine learning models, the trace segment cleaning framework includes techniques for eliminating erroneous trace segments and combining them into a single trace. Instead of potentially tracking spurious objects or making multiple trace segments represent the same object, the trace segment cleaning framework eliminates redundant, broken, and spurious trace segments. Eliminating these erroneous trace segments reduces the computational resources required for motion planning and improves the safety of planned motion in response to various driving scenarios. More specifically, the trace segment cleaning framework can compare the trajectory features, cloud point features, and / or image features of trace segments to determine which trace segments will be stitched together or otherwise eliminated. These technical improvements enhance the decision-making of planning systems.

[0033] Furthermore, the track segment cleanup system addresses technical issues associated with the planning module configured to perform planned movements. These technical issues include obtaining metrics for evaluating track quality to ensure the vehicle is responding to new object types. For example, the planning system may make poorly planned movements when the planning module fails to identify erroneous track segments. Without evaluation metrics for identifying track segments (e.g., track segment quality scores, patchwork scores), especially when the vehicle encounters new objects, the planning module cannot distinguish between false and accurate track segments. Consequently, erroneous track segments can adversely affect the planning module's ability to safely navigate autonomous vehicles.

[0034] Thus, a trace segment cleanup system needs to be configured to evaluate tracked object segments to eliminate or combine erroneous trace segments.

[0035] Training samples are needed to train the machine learning model. These training samples are generated by adding track quality labels to track segments generated by the perception detection and tracking system. The labels added to track segments can be values ​​representing the quality of the track sample compared to ground truth tracks. For example, a value of "1" means the training sample corresponds to a ground truth track, and a value of "0" means the training sample corresponds to a false track. However, without a sufficient number of labeled training samples, the machine learning model cannot accurately determine the quality score of track segments representing objects encountered by the vehicle. Thus, a training sample generator is implemented to correct for the insufficiency of labeled training samples.

[0036] The training sample generator produces labeled training samples to train the machine learning model. The training sample generator is configured to associate labels with track segments to generate training samples. The training sample generator utilizes ground truth tracks that include known, well-represented representations of objects observed by the vehicle. The training sample generator compares the ground truth tracks with one or more track segments to generate training samples with assigned labels.

[0037] Additionally, the training sample generator utilizes data augmentation techniques to generate labeled training samples. For example, the training sample generator segments a single track into two or more track segments to create training samples. The training samples are associated with labels used to identify two or more track segments as belonging to a single track. For example, a training sample comprising two or more track segments might have a ground truth splicing score of "1" to indicate that the two or more track segments come from the same track. Alternatively, the training samples can be associated with labels used to identify two or more track segments as belonging to a single track. For example, a training sample comprising two or more track segments belonging to a single track might have a ground truth splicing score of "0" to indicate that the two or more track segments come from different tracks. These technical improvements provide accurate training tools for machine learning models. The trained machine learning model can evaluate the quality of track segments and, based on training with the training samples, evaluate the segments to be spliced ​​together.

[0038] Now for reference Figure 1Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0039] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2 The vehicles 102 are the same as or similar to autonomous vehicles 202. In some embodiments, vehicles 200 in a group of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0040] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0041] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with (e.g., defining) a series of actions (also referred to as trajectories) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will board the AV, and a second state or area includes a location where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Additionally or alternatively, route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within a lane area and target rates at those locations. In the example, route 106 includes multiple precise state sequences along at least one high-level action with a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to multiple trajectories that collectively form a high-level route terminating at a final target state or region.

[0042] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In the example, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driving lane, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In the example, a road includes at least one lane associated with at least one lane marking (e.g., identified based on at least one lane marking).

[0043] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with vehicle 102 and / or V2I infrastructure system 118. In some embodiments, the V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, the V2I device 110 includes radio frequency identification (RFID) devices, signs, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markings, streetlights, parking meters, etc. In some embodiments, the V2I device 110 is configured to communicate directly with vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, and / or queue management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0044] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0045] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0046] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I infrastructure system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including and / or not including autonomous systems)).

[0047] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0048] supply Figure 1 The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements arranged differently. Additionally or alternatively, at least one element of environment 100 may be described as being composed of… Figure 1 One or more functions performed by at least one different element of environment 100. Additionally or alternatively, at least one group of elements of environment 100 may perform one or more functions described as performed by at least one different group of elements of environment 100.

[0049] Now for reference Figure 2 The vehicle 200 includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 and the vehicle 102 (see...) Figure 1 The vehicle 200 is similar to or the same as the vehicle in question. In some embodiments, the vehicle 200 has autonomous capabilities (e.g., implementing at least one function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention) and / or highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations)). For a detailed description of fully autonomous and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0050] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, and safety controller 202g.

[0051] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision). In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1 The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0052] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other systems containing cameras described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0053] The laser detection and ranging (LiDAR) sensor 202b includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object within the field of view of the LiDAR sensor 202b.

[0054] The radio detection and ranging (radar) sensor 202c includes components configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the radar sensor 202c encounter a physical object and are reflected back to the radar sensor 202c. In some embodiments, the radio waves emitted by the radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the radar sensor 202c generates a signal representing objects included in the field of view of the radar sensor 202c. For example, at least one data processing system associated with the radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of the radar sensor 202c.

[0055] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0056] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, an autonomous vehicle computing system 202f, a safety controller 202g, and / or a drive-by-wire (DBW) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0057] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1 V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1 The V2I system 118 communicates with the same or similar V2I system.

[0058] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0059] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200, including one or more devices (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0060] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a certain direction, decelerate in a certain direction, make a left turn and / or make a right turn, etc. In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0061] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to turn left or right, thereby causing the vehicle 200 to turn left or right.

[0062] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0063] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), a wheel rate sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor.

[0064] Now for reference Figure 3 A schematic diagram of device 300 is illustrated. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of system of vehicle 102); and / or one or more devices of network 112 (e.g., one or more devices of system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of system of vehicle 102), and / or one or more devices of network 112 (e.g., one or more devices of system of network 112) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0065] Bus 302 includes components for communication between the components of the licensed device 300. In some embodiments, processor 304 is implemented in hardware, software, or a combination of hardware and software. In some examples, processor 304 includes a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or accelerated processing unit (APU), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC), etc.). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by processor 304.

[0066] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0067] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0068] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, etc. Interfaces and / or cellular network interfaces, etc.

[0069] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 306 and / or storage component 308, executed by processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0070] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. When executed, the software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0071] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0072] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the other device, which, when executed by the processor of processor 304 and / or the processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0073] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a group of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another group of components of device 300.

[0074] Now for reference Figure 4A The diagram illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0075] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0076] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the sensing system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the sensing system 402. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0077] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various types of other traffic lights). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0078] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0079] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206), and / or the braking system (e.g., braking system 208). In an example, where the trajectory includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.).

[0080] In some embodiments, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.).

[0081] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3(The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0082] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0083] Now for reference Figure 4B The diagram illustrates an implementation of a machine learning model. More specifically, it illustrates an implementation of a convolutional neural network (CNN) 420. For illustrative purposes, the following description of CNN 420 will concern the implementation of CNN 420 via a perception system 402. However, it will be understood that in some examples, CNN 420 (e.g., one or more components of CNN 420) is implemented by systems other than or besides perception system 402 (such as planning system 404, positioning system 406, and / or control system 408, etc.). Although CNN 420 includes certain features as described herein, these features are provided for illustrative purposes and are not intended to limit this disclosure.

[0084] CNN 420 includes multiple convolutional layers comprising a first convolutional layer 422, a second convolutional layer 424, and a convolutional layer 426. In some embodiments, CNN 420 includes a subsampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, subsampling layer 428 and / or other subsampling layers have a dimension smaller than that of the upstream system (i.e., the number of nodes). By means of subsampling layer 428 having a dimension smaller than that of the upstream layers, CNN 420 combines the amount of data associated with the initial input and / or output of the upstream layers, thereby reducing the computational cost required for downstream convolution operations by CNN 420. Additionally or alternatively, by means of subsampling layer 428 associated with at least one subsampling function (e.g., configured to perform at least one subsampling function), CNN 420 combines the amount of data associated with the initial input.

[0085] Based on the perception system 402 providing corresponding inputs and / or outputs associated with each of the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426 to generate corresponding outputs, the perception system 402 performs convolution operations. In some examples, based on the perception system 402 providing data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426, the perception system 402 implements a CNN 420. In such examples, based on the perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle identical or similar to vehicle 102, a remote AV system identical or similar to remote AV system 114, a queue management system identical or similar to queue management system 116, and / or a V2I system identical or similar to V2I system 118, etc.), the perception system 402 provides data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426. A detailed description of the convolution operation is included below.

[0086] In some embodiments, the perception system 402 provides data associated with an input (referred to as initial input) to a first convolutional layer 422, and the perception system 402 uses the first convolutional layer 422 to generate data associated with an output. In some embodiments, the perception system 402 provides the output generated by the convolutional layer as input to different convolutional layers. For example, the perception system 402 provides the output of the first convolutional layer 422 as input to a subsampling layer 428, a second convolutional layer 424, and / or a convolutional layer 426. In such an example, the first convolutional layer 422 is referred to as the upstream layer, and the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426 are referred to as downstream layers. Similarly, in some embodiments, the perception system 402 provides the output of the subsampling layer 428 to the second convolutional layer 424 and / or the convolutional layer 426, and in this example, the subsampling layer 428 will be referred to as the upstream layer, and the second convolutional layer 424 and / or the convolutional layer 426 will be referred to as the downstream layer.

[0087] In some embodiments, before providing input to the CNN 420, the perception system 402 processes the data associated with the input provided to the CNN 420. For example, the perception system 402 processes the data associated with the input provided to the CNN 420 based on the normalization of sensor data (e.g., image data, LiDAR data, and / or radar data, etc.) by the perception system 402.

[0088] In some embodiments, CNN 420 generates outputs by performing convolution operations associated with each convolutional layer based on the perception system 402. In some examples, CNN 420 generates outputs by performing convolution operations associated with each convolutional layer and an initial input based on the perception system 402. In some embodiments, the perception system 402 generates outputs and provides these outputs to a fully connected layer 430. In some examples, the perception system 402 provides the outputs of convolutional layer 426 to the fully connected layer 430, wherein the fully connected layer 430 includes data associated with multiple feature values ​​referred to as F1, F2, ..., FN. In this example, the outputs of convolutional layer 426 include data associated with multiple output feature values ​​representing predictions.

[0089] In some embodiments, the perception system 402 identifies a prediction from among a plurality of predictions based on a feature value identified as the highest probability of being the correct prediction among multiple predictions. For example, if the fully connected layer 430 includes feature values ​​F1, F2, ..., FN and F1 is the largest feature value, the perception system 402 identifies the prediction associated with F1 as the correct prediction among multiple predictions. In some embodiments, the perception system 402 trains the CNN 420 to generate predictions. In some examples, the perception system 402 trains the CNN 420 to generate predictions based on training data associated with predictions provided to the CNN 420.

[0090] Now for reference Figure 5The diagram illustrates the implementation of trace segment cleanup. Trace segment cleanup uses a machine learning model to identify and eliminate potentially erroneous trace segments. The perception system can generate trace segments as representations of objects encountered by the vehicle. However, sometimes these trace segments are inaccurate or erroneous. Examples of erroneous trace segments include redundant, broken, or false trace segments. Trace segment cleanup is a technique that eliminates redundant, broken, or false trace segments, or combines these trace segments with other trace segments to capture a single, complete, and accurate trace of the tracked object. Erroneous trace segments may result from poor perception of objects outside the vehicle, splicing errors in object information across time frames, or the perception system 402 and / or planning system 404 failing to identify object movement as belonging to the same object. To correct this problem, trace segment cleanup includes identifying false traces (e.g., traces with quality below a threshold). Furthermore, trace segment cleanup includes identifying two or more separate trace segments that can be combined into a trace segment.

[0091] The pre-cleanup track segment 510 indicates multiple track segments with potential errors. The pre-cleanup track segment 510 can be created by an automatic tagging and annotating system, which can be implemented by the perception system 402 in response to the detection of an object. The automatic tagging and annotating system is configured to associate the track segments with the object when the sensors at the vehicle perceive the object. Figure 5 As shown, the trace segment 510 before cleanup includes four trace segments: the first trace segment 515, the second trace segment 525, the third trace segment 535, and the fourth trace segment 545.

[0092] The first trace segment 515 includes the first object within its frame, indicating that the first trace segment 515 is tracking the same object. The first object is included in each frame of the first trace segment 515, and the first object is not included in any other trace segment. The consistency that the first object occupies only one trace indicates that the single object is correctly paired with the single trace, and no further cleanup of the trace is required to obtain the complete trace segment of the first object.

[0093] The second trace segment 525 and the third trace segment 535 depict two trace segments representing the movement of a second object. The second object in the second trace segment 525 and the second object in the third trace segment 535 are the same object. However, associating two separate trace segments of the same object is inefficient and erroneous. Tracking multiple trace segments of the same object may consume excessive computer resources of the planning system 404 and potentially lead to navigation decision difficulties for the planning system 404, the positioning system 406, and / or the control system 408. As a result, the second object in both the second trace segment 525 and the third trace segment 535 exhibits various inefficiencies and safety problems. The second trace segment 525 and the third trace segment 535 should be combined so that the second object is represented by only one trace.

[0094] The fourth trace segment 545 is a spurious trace. No object corresponds to the fourth trace segment 545. That is, the spurious trace indicates that the object does not exist in real life. As a result, the fourth trace segment 545 complicates the navigation decisions of the planning system 404, the positioning system 406, and / or the control system 408. The fourth trace segment 545 should be eliminated so that the spurious trace does not consume excessive computing resources. For offline perception automatic labeling systems, spurious traces may reduce the quality of output object annotations, which may reduce the effectiveness of machine learning-based online object detection and tracking models trained using this automatically labeled data.

[0095] Post-cleaning trace segment 520 shows the pre-cleaning trace segment 510 after trace cleanup processing. Post-cleaning trace segment 520 exemplifies an object represented by only one trace segment. Post-cleaning trace segment 520 has a 1:1 correspondence between the number of tracked objects and the number of trace segments. Post-cleaning trace segment 520 does not include any spurious trace segments or trace segments that do not correspond to real-world objects. Figure 5 As shown, the first trace segment 515 includes the first object within its frame, indicating that the first trace segment 515 is associated with the same object. The first object is included in each frame of the first trace segment 515, and the first object is not included in any other trace segment. The consistency of the first object occupying only one trace indicates that the single object is correctly paired with the single trace, and no further merging of traces is required to obtain the complete trace segment of the first object.

[0096] Similarly, the second trace segment 525 includes the second object within its frame, indicating that the second trace segment 525 is tracking the same object. The second object is included in each frame of the second trace segment 525, and the second object is not included in any other trace segment. The consistency of the second object occupying only one trace indicates that the single object is correctly paired with the single trace, and no further merging of traces is required to obtain a complete trace segment for the second object. The trace segment cleanup process combines the third trace segment 535 with the second trace segment 525 to obtain a complete trace. In addition, since there is no real-world object corresponding to the fourth trace segment 545, the trace segment cleanup process removes the fourth trace segment 545. The cleaned-up trace segment 520 increases the accuracy of the planned movement of the planning system 404, reduces the processing and storage resources required to monitor objects outside the vehicle, and increases the possibility of safe planned movement in response to various driving scenarios.

[0097] Now for reference Figure 6A diagram illustrating the implementation of trace segment cleanup data stream 600 is provided. Trace segment cleanup data stream 600 identifies potentially redundant, broken, or spurious trace segments. These trace segments represent sensed objects encountered by the vehicle. Trace segment cleanup data stream 600 is configured to eliminate or combine potentially redundant, broken, or spurious trace segments. Additionally, as described herein, trace segment cleanup data stream 600 applies a trained machine learning model to determine whether a trace segment represents a real object outside the vehicle. The trained machine learning model determines the quality of the trace segments and determines whether these segments should be stitched together.

[0098] Input tracing module 610 is configured to receive tracing segments. A tracing segment represents an object perceived external to the vehicle. Tracing feature extraction network 620 receives input tracings selected by input tracing module 610. The input tracing includes a first tracing segment 612 and a second tracing segment 614. Tracing feature extraction network 620 extracts features from the first tracing segment 612 and the second tracing segment 614. Tracing segment cleanup module 630 determines whether the tracing segments represent the same object by comparing features from the first tracing segment 612 and the second tracing segment 614. Tracing segment cleanup module 630 applies a trained machine learning model to determine whether the first tracing segment 612 and the second tracing segment 614 represent the same object. Tracing segment cleanup module 630 determines whether to combine the first tracing segment 612 and the second tracing segment 614 and output cleaned tracing 640. Additionally and / or alternatively, tracing segment cleanup module 630 may determine which tracing segments to eliminate based on tracing segment quality scores.

[0099] In an embodiment, the trace segment cleanup data stream 600 receives trace segments as input traces at the input trace module 610. The input traces include trace segments representing objects tracked by the perception system 402 and / or the planning system 404. For example, a bicycle outside a vehicle is an object tracked in the perception system 402. A trace segment represents a tracked object outside a vehicle that is spatially monitored relative to the vehicle's location over a period of time. Tracked objects can be detected by an autonomous system 202, which may include at least one of a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, and a microphone 202d. For example, a bicycle outside a vehicle is initially detected by the camera 202a and subsequently monitored using the LiDAR sensor 202b.

[0100] In an embodiment, the automatic tagging system automatically annotates data collected from vehicle sensors. The automatic tagging annotator can be implemented as an offline version of the sensing system 402. The automatic tagging annotator is configured to annotate tracked objects representing real objects and form track segments. For example, the automatic tagging annotator annotates a first track segment 612 representing a first object. In an embodiment, a first automatic tagging annotation is appended to the first track segment 612, and a second automatic tagging annotation is appended to the second track segment 614. The annotated track segment can be an input track received at the input track module 610 to initiate the track segment cleanup data stream 600.

[0101] The input tracing module 610 is configured to select two or more trace segments (e.g., a first trace segment 612 and a second trace segment 614) from a plurality of trace segments that can potentially be combined. The input tracing module 610 can examine characteristics or features of the plurality of trace segments to determine whether the two or more trace segments might represent the same object. For example, the perception system 402 detects a bicycle within a first time period represented by the first trace segment 612, and the perception system 402 detects a bicycle within a second time period represented by the second trace segment 614. The input tracing module 610 determines that the two traces can potentially be combined based on the temporal proximity of the two trace segments. In another embodiment, the first trace segment 612 has a first set of timestamps, and the second trace segment 614 has a second set of timestamps, wherein at least one timestamp in the second set of timestamps is different from a timestamp in the first set of timestamps. The input trace module 610 determines that the two traces can potentially be combined based on the fact that the closest timestamp does not exceed a certain time period (e.g., 3 seconds) and that the types of objects being tracked are the same between the first trace segment 612 and the second trace segment 614.

[0102] Additionally, the input tracing module 610 determines whether the two traces can potentially be combined based on the physical proximity of objects in the first trace segment 612 and objects in the second trace segment 614. Objects in the first trace segment 612 and objects in the second trace segment 614 may satisfy a distance threshold. For example, if a distance threshold of five meters is met, the input tracing module 610 determines that the first trace segment 612 and the second trace segment 614 may represent the same object. In an embodiment, the first trace segment 612 and the second trace segment 614 are selected from multiple trace segments based on satisfying a timestamp threshold and / or a physical proximity threshold, etc.

[0103] After the input tracking module 610 identifies potentially combinable track segments (i.e., track segment candidates), the track segment cleaning data stream 600 extracts features from the track segment candidates in the track feature extraction network 620. The features extracted in the track feature extraction network 620 may include trajectory features, cloud point features, and image features. For example, the track feature extraction network 620 extracts first trajectory features and first cloud point features from a first track segment 612 representing a bicycle within a first time period. Additionally, the track feature extraction network 620 extracts second trajectory features and second cloud point features from a second track segment 614 representing a bicycle within a second time period.

[0104] After extraction and processing, the trace segment cleanup data stream 600 sends the extracted features to the trace segment cleanup module 630. The trace segment cleanup module 630 determines, based on the extracted features, whether to combine the first trace segment 612 and the second trace segment 614 into a single trace. For example, the trace segment cleanup module 630 compares first trajectory features and second trajectory features, and similarly, compares first cloud point features and second cloud point features corresponding to bicycles in two time periods.

[0105] The trace segment cleanup module 630 can apply a trained machine learning model to determine, based on extracted features, whether the first trace segment 612 and the second trace segment 614 represent the same object outside the vehicle. The machine learning model can provide analysis for generating scores representing the probability that the first trace segment 612 and the second trace segment 614 represent the same object outside the vehicle. For example, the trace segment cleanup module 630 assigns a trace quality score of 0.9 to the first trace segment 612 and a trace quality score of 0.87 to the second trace segment 614. The trace segment cleanup module 630 assigns a splicing score of 0.93 to both the first trace segment 612 and the second trace segment 614.

[0106] After generating a score, the track segment cleaning module 630 determines whether the score meets a threshold. For example, if the track quality score threshold is 0.75 and the stitching score threshold is 0.8, then the score threshold generated by the machine learning model is met. If the score meets the threshold, the track segment cleaning module 630 is configured to combine the first track segment 612 and the second track segment 614 into a single track segment with a single trajectory and a single cloud point feature. For example, two track segments representing a bicycle (e.g., the first track segment 612 and the second track segment 614) are combined with all extracted features into a single track segment. The combined track segment then has a single trajectory set and a single cloud point feature set.

[0107] The trace segment cleanup data stream 600 increases the likelihood of accurate perception of objects and their trajectories, thereby preventing erroneous planned movements from being exploited by the planning system 404. Furthermore, the trace segment cleanup data stream 600 identifies redundant, broken, or spurious trace segments to eliminate or combine them, thereby reducing the computational resources required to run the planning system 404 and ensuring safe planned movements in response to various types of objects.

[0108] Now for reference Figure 7 The diagram illustrates the implementation of the trace feature extraction network 620. The trace feature extraction network 620 is configured to determine features from trace segments. The features determined or extracted from the trace segments indicate the quality of the trace. Additionally, these features indicate the possibility of combining trace segments with other trace segments. Features extracted from one trace segment can be fused, concatenated, or averaged with features from different trace segments in the trace feature fusion network 750. The extracted features may include trajectory features, cloud point features, and image features.

[0109] Track feature extraction network 620 uses trajectory feature network 710 to extract trajectory features. The trajectory features include trace boxes from trace segments, which can be cascaded to learn the motion characteristics of the object. For example, a trace box can be extracted from the second trace segment 614, determining that the bicycle's motion in a direction perpendicular to the vehicle is 10 km / h. Perception system 402 can instruct planning system 404 to add rectangles to the image of the bicycle to indicate the object's trajectory features.

[0110] A tracking bounding box consists of a rectangle with a center, size, and heading angle. The center of the rectangle has (x, y) coordinates to determine the midpoint of the object. The size of the rectangle has a width and height that determine the overall size of the object. The heading angle determines the direction of travel. Metadata associated with the rectangle determines the type of object within the rectangle. For example, a tracking bounding box monitoring a bicycle has a center at the bicycle seat, a size that surrounds the bicycle wheel-to-wheel, a heading angle perpendicular to the vehicle, and metadata identifying the object as a bicycle. Tracking bounding boxes can include timestamps. Multiple tracking bounding boxes can be chained together, each with its own timestamp. For all tracking bounding boxes, a machine learning model (as an example, a multilayer perceptron network) can be applied to learn robust trajectory features that better represent the trajectory and motion of tracking segments.

[0111] The trace feature extraction network 620 uses the point cloud feature network 720 to extract cloud point features. Cloud point features include trace points representing the 3D characteristics and motion of the object along the trace segment. Trace points include (x, y, z) coordinates, intensity, and a timestamp. For example, cloud point features for determining the structure of a bicycle can be extracted from the second trace segment 614. Determining the bicycle's structure involves determining how the bicycle will be represented in a 3D plane. Trace points are extracted along the bicycle's tires, frame, rider, and handlebars to represent the bicycle when it is drawn in a 3D plane. Multiple cloud point features can be chained together, each with its own timestamp. For all trace points, a machine learning model (as an example, the PointNet network) can be applied to learn robust point features that better represent the object's structure and trace motion.

[0112] The trace feature extraction network 620 can use the image feature network 730 to extract image features. Image features include extracting corresponding images. Extracting corresponding images includes extracting segmentation, embedding features, or pixels associated with the object. For example, capturing an image of a bicycle to represent the bicycle. Object image regions or pixels can be obtained by projecting 3D trace points of cloud points onto the image plane. Pixels can be represented using RGB values, and timestamps can be associated with the images. Multiple images are strung together, each with its own timestamp. A machine learning model (as an example, a convolutional neural network) is used to learn image features that better represent the appearance features of the tracked object.

[0113] For all trajectory features learned from the trajectory feature network 710, point features learned from the point cloud feature network 720, and image features learned from the image feature network 730, the trace feature fusion network 750 uses a machine learning model (for example, a multilayer perceptron network) to combine these multimodal features into a single feature vector that better depicts the trace segment. The trajectory feature network 710, point cloud feature network 720, image feature network 730, and trace feature fusion network 750 can be learned together. The fused features learned by the trace feature fusion network 750 can utilize all trajectory, point cloud, and image features and better complement each modality. For example, a white vehicle trace segment and a black trace segment can be easily distinguished from image features. For another example, a large bus traveling at a long distance and a small bus traveling at a medium distance may look similar in image appearance, but can be easily distinguished by point features. Even if objects look and are similar in size, trajectory features can easily distinguish static and dynamic segments. In practice, the feature fusion network can flexibly discard a feature modality. For example, when the data only contains point clouds, training can be performed using only trajectory features and point cloud features. Furthermore, the trace feature fusion network can be extended to include other trace features such as radar features. Following the trace feature fusion network 750, trace segments can be represented by trace features 760.

[0114] Trace feature 760 is the output of trace feature fusion network 750. Trace feature 760 includes features extracted from trace segments. For example, trace feature 760 includes fused features learned from trajectory features and point cloud features of a bicycle in two time periods corresponding to the first trace segment 612 and the second trace segment 614.

[0115] Now for reference Figure 8 The diagram illustrates the implementation of the trace segment cleaning module 630. The trace segment cleaning module 630 includes trace features 760, a machine learning model 820, and a trace score 830. The trace score 830 includes a first trace quality score 832, a second trace quality score 834, and a trace splicing score 836.

[0116] Trace features 760 are received from trace feature extraction network 620. Trace features 760 include a first trace feature 812 and a second trace feature 814. The first trace feature 812 corresponds to features extracted from a first trace segment 612, and the second trace feature 814 corresponds to features extracted from a second trace segment 614. The first trace feature 812 and the second trace feature 814 are evaluated to determine the quality of the trace segment. The quality of the trace segment can indicate the likelihood that the trace segment represents a real-life object. The first trace feature 812 and the second trace feature 814 are compared to determine a splicing score for at least two trace segments. The splicing score of the two traces indicates the likelihood that the spliced ​​trace segment thus obtained represents the same real-life object.

[0117] The trace segment cleaning module 630 applies a machine learning model 820 to determine quality scores and splicing scores for a first trace feature 812 and a second trace feature 814. More specifically, the trace segment cleaning module 630 applies the machine learning model 820 to determine a first trace quality score 832 associated with the first trace segment 612 and a second trace quality score 834 associated with the second trace segment 614. In one embodiment, the machine learning model 820 has a network architecture including a common convolutional network to consume the first trace feature 812 and the second trace feature 814. The network architecture is configured to output an analysis of the trace quality score and the trace splicing score for each trace. In another embodiment, the machine learning model 820 has a network architecture including a two-branch convolutional network. One branch regresses the quality score of each trace, and the other branch regresses the splicing score. In another embodiment, the machine learning model 820 includes one or more of the following: a multilayer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, and / or a transformer.

[0118] A machine learning model 820 is trained to determine whether a first trace segment 612 and a second trace segment 614 represent the same object based on a first trace feature 812 and a second trace feature 814. For example, the machine learning model 820 determines that the first trace segment 612 and the second trace segment 614 represent the same object based on the trace points and intensity of cloud point features from the first trace feature 812 and the second trace feature 814. In another example, the machine learning model 820 determines that the probability that the first trace segment 612 and the second trace segment 614 are the same object is low based on the center position and timestamp of the trajectory features from the first trace feature 812 and the second trace feature 814.

[0119] The trace segment cleanup module transforms the comparative and qualitative analysis from the machine learning model 820 into a trace score 830. The trace score 830 includes three scores: a first trace quality score 832, a second trace quality score 834, and a trace splicing score 836. The first trace quality score 832 and the second trace quality score 834 indicate the probability that the trace segment represents a real-life object. The trace splicing score 836 of the two traces indicates the probability that the spliced ​​trace segment of the two traces thus obtained represents the same real-life object. The machine learning model 820 and the trace segment cleanup module 630 can generate the first trace quality score 832, the second trace quality score 834, and the trace splicing score 836 to determine the probability that the first trace segment 612 and the second trace segment 614 represent the same object outside the vehicle. For example, the machine learning model 820 and the trace segment cleanup module 630 determine that 0.9 is the first trace quality score 832 and 0.87 is the second trace quality score 834. Machine learning model 820 and trace segment cleaning module 630 determined that the first trace feature 812 and the second trace feature 814 together have a splicing score of 0.93. In this embodiment, a score in the range between 0 and 1 can be generated.

[0120] A low trace quality score indicates a high probability that the trace segment is a false positive of a real-world object. For example, the first trace feature 812 includes several unstable cloud features that significantly alter the structure of the object between at least two timestamps. On the other hand, a high trace quality score indicates a high probability that the trace segment represents a real-world object. For example, the first trace feature 812 includes consistent cloud features that maintain the structure of the object across multiple timestamps. In embodiments, the quality of a trace segment indicates whether the trace segment will be combined with another trace segment. The quality of a trace segment is evaluated independently of other trace segments. For example, the quality of the first trace feature 812 is evaluated independently of the second trace feature 814.

[0121] A low splicing score indicates a low probability that at least two trace segments represent the same object. For example, the difference in the magnitude and heading angle of the trajectory features of the first trace feature 812 and the second trace feature 814 indicates a low probability that the spliced ​​trace segments thus obtained represent the same real-world object. A high splicing score indicates a high probability that at least two trace segments represent the same object. For example, the similarity in the magnitude and heading angle of the trajectory features of the first trace feature 812 and the second trace feature 814 indicates a high probability that the spliced ​​trace segments thus obtained represent the same real-world object.

[0122] The trace segment cleanup module 630 can determine that some trace segments need to be eliminated due to low trace segment quality scores. In an embodiment, the trace segment cleanup module 630 applies a machine learning model 820 to determine a third trace segment quality score associated with a third trace segment. The machine learning model 820 can generate a third trace segment quality score to determine the likelihood that the third trace segment represents a real-life object outside the vehicle. For example, the machine learning model 820 and the trace segment cleanup module 630 determine that 0.3 is a third trace quality score. With a trace quality score threshold of 0.75, the third trace quality score does not meet the threshold. As a result, the trace segment cleanup module 630 eliminates the third trace segment.

[0123] The track segment cleaning module 630 determines whether the first track quality score 832, the second track quality score 834, and the track splicing score 836 meet thresholds. For example, if the track quality score threshold is 0.75 and the splicing score threshold is 0.8, then the scores are met. If the scores meet the thresholds, the track segment cleaning module 630 is configured to combine the first track segment 612 and the second track segment 614 into a single track segment with a single trajectory and a single cloud point feature. For example, two track segments representing a bicycle (e.g., the first track segment 612 and the second track segment 614) are combined with all extracted features into a single track segment. The combined track segment then has a single trajectory set and a single cloud point feature set. In an embodiment, the combined single track segment has a single trajectory and a single cloud point feature.

[0124] In one embodiment, the first trace segment 612 and the second trace segment 614 are combined by at least concatenating the trajectory features of the first trace segment 612 with the trajectory features of the second trace segment 614. In another embodiment, the first trace segment 612 and the second trace segment 614 are combined by at least concatenating the cloud point features of the first trace segment 612 with the cloud point features of the second trace segment 614.

[0125] In this embodiment, the track segment cleanup module 630 identifies false or low-quality track segments for elimination. The track segment cleanup module 630 determines whether to eliminate a track segment based on extracted features. For example, the track segment cleanup module 630 evaluates trajectory features, and similarly, evaluates cloud point features corresponding to the bicycle within a first time period.

[0126] In one embodiment, the track segment cleanup module 630 may apply a trained machine learning model 820 to determine whether a track segment represents an object outside the vehicle based on extracted features. The machine learning model 820 can provide analysis for generating scores representing the probability that a false track segment represents the same object outside the vehicle. For example, the track segment cleanup module 630 may assign a track quality score of 0.25 to a track segment.

[0127] In this embodiment, the track segment cleaning module 630 determines whether the score meets a threshold. For example, if the track quality score threshold is 0.75, then the score generated by the machine learning model 820 for the track segment does not meet the threshold. If the score does not meet the threshold, the track segment cleaning module 630 is configured to eliminate false track segments. For example, eliminating track segments that incorrectly represent bicycles and all extracted features.

[0128] Now for reference Figure 9 The diagram illustrates the implementation of a training sample generator configured to generate samples for training a machine learning model. Training data stream 900 utilizes data generator 910 to generate training samples for training the machine learning model. Training samples can be generated by training sample generator 912. Training the machine learning model involved in trace feature extraction network 620 and trace segment cleaning module 630 improves the accuracy of trace segment cleaning. Furthermore, training sample generator 912 corrects the problem of insufficient labeled training samples for training machine learning model 820.

[0129] Training sample generator 912 generates labeled training samples for training the machine learning model 820 of trace feature extraction network 620 and trace segment cleaning module 630. Training sample generator 912 is configured to associate labels with trace segments to generate training samples. Training sample generator 912 may utilize ground truth traces to generate labels for training samples. Ground truth traces may include an accurate representation of the movement of a perceived object. Ground truth traces may also be at least partially artificially synthesized and correspond to the movement of a simulated object. Training sample generator 912 compares ground truth traces with one or more trace segments to generate training samples. If training sample generator 912 performs matching between ground truth traces from multiple ground truth traces and one or more trace segments, training sample generator 912 assigns a first label to the trace segment to generate a training sample with the first label. Additionally, if the training sample generator 912 fails to match ground truth traces from multiple ground truth traces with one or more trace segments, the training sample generator 912 assigns a second label to the trace segment to generate training samples with the second label.

[0130] Training sample generator 912 generates training samples that indicate false trace segments. For example, a training sample with a quality score label of "0" indicates that the training sample does not correspond to the ground truth trace (i.e., the training sample is a false positive). In another example, a training sample with a quality score label of "1" indicates that the training sample corresponds to the ground truth trace (i.e., the training sample matches the real object). Training samples can be a standard used for comparison when determining the accuracy and quality of other trace segments input to the machine learning model 820 of the trace feature extraction network 620 and the trace segment cleaning module 630.

[0131] In addition to labels indicating whether a trace segment is a false trace, the training samples generated by the training sample generator 912 can also include labels for any two input trace segments indicating whether they should be concatenated. For example, two trace segments matching the same ground truth trace will be assigned a concatenation score of "1" to indicate that they should be combined. Two trace segments matching different ground truth traces will be assigned a concatenation score of "0" to indicate that they should not be combined. A trace segment with a matching ground truth trace and another unmatched trace segment will be assigned a concatenation score of "0" because they come from different traces. For two trace segments that do not have a matching ground truth trace, the concatenation score between these two trace segments can be ignored during training because the trace quality of these two trace segments will be 0, and thus the trace quality score will eliminate these two trace segments. There is no need to concatenate these unmatched trace segments.

[0132] Training samples can be the same as other trace segments and include trajectory features, cloud point features of other trace segments, and / or image features. Training samples are used to train the trace feature extraction network 620 and the machine learning model 820 together to, for example, correctly identify false trace segments and trace segments to be combined. In embodiments, training samples can also be used as validation samples to determine whether the performance of the trace feature extraction network 620 and the machine learning model 820 of the trace segment cleaning module 630 is satisfactory in identifying false trace segments and trace segments to be spliced.

[0133] In an embodiment, the training sample generator 912 generates training samples used by the machine learning model 820 of the trace feature extraction network 620 and the trace segment cleaning module 630. In this embodiment, the training samples generated by the training sample generator 912 include at least two trace segments. The training sample generator 912 compares the set of ground truth traces with one or more trace segments to generate training samples with quality score labels. The training sample generator 912 compares the trace segments with each ground truth trace to determine if a matching ground truth trace exists to assign a trace quality label. To determine a matching ground truth trace, the training sample generator 912 iterates through the ground truth trace matching flowchart 1000 to match the correct ground truth trace with the trace segment and / or the training sample with the quality score label. For example, if the training sample generator 912 identifies a trace segment that matches a ground truth trace, the training sample generator 912 assigns the quality score label "1" to the training sample that includes the trace segment based on the match between the trace segment and the ground truth trace.

[0134] Additionally, the training sample generator 912 compares two or more trace segments to generate training samples with splicing scores. The training samples generated by the training sample generator 912 include at least two trace segments. The training sample generator 912 can compare the trace segments of the training samples to assign splicing score labels. To determine two or more matching trace segments in the training samples, the training sample generator 912 checks whether the two trace segments match the same ground truth trace. For example, if the training sample generator 912 identifies that a trace segment can be combined with another trace segment, then the training sample generator 912 assigns the splicing score label "1" to the two trace segments of the training sample based on the two trace segments matching the same ground truth trace. If the training sample generator 912 does not match a trace segment with another trace segment, then the training sample generator 912 assigns the splicing score label "0" to the two trace segments of the training sample based on the two or more trace segments not matching the same ground truth trace. A trace quality label of "0" can indicate that a trace segment is a false trace.

[0135] In an embodiment, the training sample generator 912 is configured to segment a single track into two or more track segments to create training samples. The training sample generator 912 is configured to add labels to two or more track segments to create two or more training samples. For example, the label for a training sample comprising two or more track segments has a ground truth splicing score of "1" to indicate that the two or more track segments come from the same track. Alternatively, the training samples are associated with labels used to identify two or more track segments as belonging to a single track. For example, the label for a training sample comprising two or more track segments belonging to a single track has a ground truth splicing score of "0" to indicate that the two or more track segments come from different tracks. In an implementation, the training sample generator 912 may randomly segment a track segment with a matching ground truth label into at least two training samples.

[0136] Segmenting trace segments to create two or more training samples creates diversity in the training data. Without a sufficient number of labeled training samples, the trace feature extraction network 620 and the machine learning model 820 cannot accurately determine the quality score of the trace segments representing objects encountered by the vehicle. In an embodiment, if a trace segment fails to meet a frame threshold, it is not eligible for segmentation to generate training samples. For example, if a trace segment has fewer than five frames, the training sample generator 912 ignores attempts to segment the trace segment. Frames can be timestamps. In an embodiment, the trace sample generator 912 has a rule that each of at least two segmented trace segments has at least three time frames.

[0137] Now for reference Figure 10 The flowchart illustrates a process for generating training samples with ground truth labels to train a machine learning model. Ground truth trace matching flowchart 1000 results in the selection of ground truth traces that match the input trace segment to output training samples. Training sample generator 912 can compare the trace segment representation with the ground truth trace representation to generate labeled training samples. In this embodiment, ground truth trace matching flowchart 1000 eliminates ineligible ground truth traces to determine the ground truth trace corresponding to the input trace segment.

[0138] At 1010, the training sample generator 912 compares the timestamps of the trace segment and the ground truth traces. If there is no common timestamp between the trace segment and the ground truth traces from multiple ground truth traces, the training sample generator 912 determines that the ground truth traces from multiple ground truth traces are unqualified matches and continues to the next ground truth trace. The training sample generator 912 compares the timestamps of each ground truth trace in the multiple ground truth traces and determines a subset of ground truth traces as potential qualified matches based on these timestamps. Ground truth traces outside the subset of ground truth traces are eliminated because they are unqualified matches. In an embodiment, there are no remaining ground truth traces in the subset of ground truth traces. In this case, since no ground truth trace is a qualified match, the training sample generator 912 assigns a quality score label "0" to the trace segment.

[0139] At 1020, the training sample generator 912 determines whether a ground truth track from a subset of ground truth tracks has trajectory features exceeding a distance threshold relative to the trajectory features of a track segment. The distance threshold can be Euclidean distance. The distance threshold can also be a similarity threshold representing any similarity or dissimilarity function. If the trajectory features of a ground truth track from a subset of ground truth tracks exceed the distance threshold, the training sample generator 912 determines that the ground truth track from the subset of ground truth tracks is an unqualified match and continues to the next ground truth track. The training sample generator 912 compares the distances of the trajectory features of each ground truth track among multiple ground truth tracks and determines a second subset of ground truth tracks as potential qualified matches based on the trajectory features. Ground truth tracks outside the second subset of ground truth tracks are eliminated because they are unqualified matches. In an embodiment, there are no remaining ground truth tracks in the second subset of ground truth tracks. In this case, since no ground truth trace is a qualified match, the training sample generator 912 assigns the quality score label "0" to the trace segment.

[0140] At 1030, the training sample generator 912 determines the number of boxes close to the trace segment based on the trajectory features of the ground truth trace segments from the second ground truth trace subset. If the ground center distance or intersection-over-union (IoU) between two boxes is below a certain threshold, the ground truth box is close to the trace segment box. If the ground center distance between two boxes is less than scale_factor*(diagonal(box1)+diagonal(box2)) / 2.0, the ground truth box is close to the trace segment. A scale_factor of 1.5 can be used. The boxes of the ground truth traces and trace segments are compared, which can be used to filter the difference matches between the ground truth traces from the second ground truth trace subset and the trace segments. If the number of close boxes between the ground truth traces and trace segments drops below the box threshold, the training sample generator 912 determines that the ground truth traces from the second ground truth trace subset are unqualified matches and continues to the next ground truth trace. The training sample generator 912 compares the proximity boxes of ground truth traces from the second ground truth trace subset and determines a third ground truth trace subset as a potential qualified match based on the number of proximity boxes. Ground truth traces outside the third ground truth trace subset are eliminated as unqualified matches. In an embodiment, there are no remaining ground truth traces in the third ground truth trace subset. In this case, since no ground truth trace is a qualified match, the training sample generator 912 assigns a quality score label "0" to the trace segment.

[0141] At position 1040, the training sample generator 912 selects ground truth traces from the third ground truth trace subset that have the highest number of nearest-boxes relative to the trace segment. In this embodiment, the selected trace segment has a trace quality label assigned as "1" based on the trace segment matching the best ground truth trace. If no ground truth trace matches the trace segment, the training sample generator 912 assigns a quality score of "0". Additionally, the training sample generator 912 assigns a stitching score label of "1" based on a trace segment that matches the same ground truth trace as another trace segment. If two or more trace segments correspond to the same ground truth trace, a stitching score label of "1" is assigned to these two or more trace segments. If no two trace segments match the ground truth trace, the training sample generator 912 assigns a stitching score of "0" to the training sample.

[0142] Now for reference Figure 11 The diagram illustrates the implementation of matching ground truth traces with trace segments. Three ground truth traces are depicted: a first ground truth trace 1115, a second ground truth trace 1125, and a third ground truth trace 1135. Trace segments are compared with each of the three ground truth traces to generate labeled training samples. This is achieved by performing targeted... Figure 10 The ground truth trace matching flowchart 1000 describes a training sample generator 912 that selects ground truth traces that match the trace segments. Ground truth traces are depicted using the trajectories of 3D bounding boxes. Ground truth traces are then sequentially sorted based on timestamps.

[0143] The first ground truth trace 1115, the second ground truth trace 1125, and the third ground truth trace 1135 can be compared with the trace segment. The comparison between the bounding box of the trace segment and the bounding box of the ground truth trace can be based on the steps of the ground truth trace matching flowchart 1000. The training sample generator 912 uses the ground truth trace matching flowchart 1000 to determine that the second ground truth trace 1125 is a matched ground truth trace. Additionally, the training sample generator 912 determines that the second ground truth trace 1125 is a matched ground truth trace based on timestamps, trajectory features that meet distance thresholds, and / or the number of proximity boxes between the trace segment and the ground truth trace. In this embodiment, the bounding boxes of each ground truth trace are aligned with the trace segment to determine the best fit.

[0144] Now for reference Figure 12 The diagram illustrates a flowchart of a process for cleaning up trace segments using a machine learning model. In some embodiments, one or more of the steps described in process 1200 are performed by a vehicle scenario mining data stream (e.g., entirely and / or partially). Additionally or alternatively, in some embodiments, one or more of the steps described in process 1200 are performed by another means or group of means (e.g., entirely and / or partially) separate from or including the vehicle scenario mining data stream.

[0145] At point 1202, the first and second track segments are detected. A track segment represents a tracked object outside the vehicle that is spatially monitored relative to the vehicle's position over a period of time. For example, a bicycle detected in the first time period is the first track segment, and a bicycle detected in the second time period is the second track segment.

[0146] At point 1204, the first trace segment and the second trace segment are determined to represent the same object. The first and second trace segments are identified as candidates for combination into a single trace segment. Features can be extracted from the first and second trace segments for evaluation by machine learning model 820. Machine learning model 820 analyzes and determines whether the first and second trace segments meet score thresholds that indicate whether the first and second trace segments will be combined. For example, machine learning model 820 determines that the bicycle from the first trace segment and the bicycle from the second trace segment represent the same object.

[0147] At point 1206, the first and second track segments are combined to form a single track segment. This is achieved by concatenating features of the first and second track segments into a single feature set. For example, features of a bicycle tracked by the first track segment are concatenated with features of a bicycle tracked by the second track segment to form a single track segment comprising a single feature set.

[0148] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and what the applicant expects to be the scope of this invention, is the literal and equivalent scope of the claims published from this application in the specific form of the published claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as such terms are used in the claims. Furthermore, when the term “comprising” is used in the preceding specification or appended claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method for clearing trace segments for planned movement of a vehicle, comprising: Use at least one processor to detect the first trace segment and the second trace segment; Using the at least one processor, the trace points and intensity of cloud point features corresponding to the first trace feature extracted from the first trace segment and the second trace feature corresponding to the second trace feature extracted from the second trace segment, or based on the center position and timestamp of the trajectory features from the first trace feature and the second trace feature, or based on the size and heading angle of the trajectory features from the first trace feature and the second trace feature, are determined to represent the same object outside the vehicle. Using the at least one processor, determine the quality score of a first trace segment associated with the first trace segment and the quality score of a second trace segment associated with the second trace segment, as well as the splicing score between the first trace segment and the second trace segment; and In response to determining that the first trace segment and the second trace segment represent the same object outside the vehicle because the quality scores of the first trace segment and the second trace segment meet a quality score threshold and the splicing score meets a splicing score threshold, the first trace segment and the second trace segment are combined using the at least one processor to form a single trace segment with a single trajectory, wherein the quality score of the first trace segment indicates the probability that the first trace segment represents a real-life object, and the quality score of the second trace segment indicates the probability that the second trace segment represents a real-life object.

2. The method according to claim 1, wherein, The first trace segment and the second trace segment are combined by at least concatenating a first trajectory feature of the first trace segment with a second trajectory feature of the second trace segment, wherein the first trajectory feature and the second trajectory feature include at least one of a center point, a size relative to the center point, and a heading angle.

3. The method according to claim 1, wherein, The first trace segment and the second trace segment are combined by at least concatenating a first cloud point feature of the first trace segment with a second cloud point feature of the second trace segment, wherein the first cloud point feature and the second cloud point feature include at least one of trace points, 3D feature features and motion features.

4. The method according to claim 1, further comprising: Using the at least one processor, it is determined that the first trace segment satisfies a distance threshold and the second trace segment satisfies the distance threshold; as well as In response to the first trace segment and the second trace segment satisfying the distance threshold, the at least one processor is used to determine that the first trace segment and the second trace segment represent the same object.

5. The method according to claim 1, wherein, The first trace segment includes a first plurality of timestamps, and the second trace segment includes a second plurality of timestamps.

6. The method according to claim 1, further comprising: Using the at least one processor, a quality score for the third trace segment associated with the third trace segment is determined; as well as In response to the third trace segment's quality score failing to meet the quality score threshold, the at least one processor is used to eliminate the third trace segment.

7. The method according to claim 1, wherein, The first trace segment and the second trace segment are selected from a plurality of trace segments, and wherein the first trace segment and the second trace segment represent tracked objects outside the vehicle that are spatially monitored relative to the position of the vehicle over a period of time.

8. The method according to claim 1, further comprising: Using the at least one processor, at least one training sample is generated by segmenting a known single trace into at least two training trace segments. The known single trace includes ground truth annotations, which indicate that the at least two training trace segments form the known single trace.

9. The method according to claim 8, further comprising: Using the at least one processor and based on the at least one training sample, a machine learning model is trained to identify at least two training trace segments that form the known single trace.

10. A track segment clearing system for planned movement of a vehicle, comprising: At least one processor; as well as At least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform an operation, the operation including: Detect the first trace segment and the second trace segment; Based on the trace points and intensity of cloud point features corresponding to the first trace feature extracted from the first trace segment and the second trace feature corresponding to the second trace feature extracted from the second trace segment, or based on the center position and timestamp of the trajectory features from the first trace feature and the second trace feature, or based on the size and heading angle of the trajectory features from the first trace feature and the second trace feature, it is determined that the first trace segment and the second trace segment represent the same object outside the vehicle. Determine the quality score of the first trace segment associated with the first trace segment and the quality score of the second trace segment associated with the second trace segment, as well as the splicing score between the first trace segment and the second trace segment; and In response to the determination that the first trace segment and the second trace segment represent the same object outside the vehicle because the quality scores of the first trace segment and the second trace segment meet a quality score threshold and the splicing score meets a splicing score threshold, the first trace segment and the second trace segment are combined to form a single trace segment with a single trajectory and a single cloud point feature, wherein the quality score of the first trace segment indicates the probability that the first trace segment represents a real-life object, and the quality score of the second trace segment indicates the probability that the second trace segment represents a real-life object.

11. The system according to claim 10, wherein, The first trace segment and the second trace segment are combined by at least concatenating a first trajectory feature of the first trace segment with a second trajectory feature of the second trace segment, wherein the first trajectory feature and the second trajectory feature include at least one of a center point, a size relative to the center point, and a heading angle.

12. The system according to claim 10, wherein, The first trace segment and the second trace segment are combined by at least concatenating a first cloud point feature of the first trace segment with a second cloud point feature of the second trace segment, wherein the first cloud point feature and the second cloud point feature include at least one of trace points, 3D feature features and motion features.

13. The system according to claim 10, further comprising: It is determined that the first trace segment satisfies the distance threshold and the second trace segment satisfies the distance threshold; as well as In response to the first trace segment and the second trace segment satisfying the distance threshold, it is determined that the first trace segment and the second trace segment represent the same object.

14. The system according to claim 10, wherein, The first trace segment includes a first plurality of timestamps, and the second trace segment includes a second plurality of timestamps, wherein at least one of the second plurality of timestamps is different from the timestamps in the first plurality of timestamps.

15. The system according to claim 10, further comprising: Determine the quality score of the third trace segment associated with the third trace segment; as well as In response to the third trace segment's quality score failing to meet the quality score threshold, the third trace segment is eliminated.

16. The system according to claim 10, wherein, The first trace segment and the second trace segment are selected from a plurality of trace segments, and wherein the first trace segment and the second trace segment represent tracked objects outside the vehicle that are spatially monitored relative to the position of the vehicle over a period of time.

17. The system of claim 10, further comprising: At least one training sample is generated by segmenting a known single trace into at least two training trace segments. The known single trace includes ground truth annotations, which indicate that the at least two training trace segments form the known single trace.

18. The system of claim 17, further comprising: Based on the at least one training sample, a machine learning model is trained to identify at least two training trace segments that form the known single trace.

19. A non-transitory computer-readable storage medium comprising at least one program executable by one or more processors of a first device, the at least one program comprising instructions that, when executed by the one or more processors, cause the first device to perform a track segment clearing operation for a planned movement of a vehicle, the operation comprising: Detect the first trace segment and the second trace segment; Based on the trace points and intensity of cloud point features corresponding to the first trace feature extracted from the first trace segment and the second trace feature corresponding to the second trace feature extracted from the second trace segment, or based on the center position and timestamp of the trajectory features from the first trace feature and the second trace feature, or based on the size and heading angle of the trajectory features from the first trace feature and the second trace feature, it is determined that the first trace segment and the second trace segment represent the same object outside the vehicle. Determine the quality score of the first trace segment associated with the first trace segment and the quality score of the second trace segment associated with the second trace segment, as well as the splicing score between the first trace segment and the second trace segment; and In response to the determination that the first trace segment and the second trace segment represent the same object outside the vehicle because the quality scores of the first trace segment and the second trace segment meet a quality score threshold and the splicing score meets a splicing score threshold, the first trace segment and the second trace segment are combined to form a single trace segment with a single trajectory and a single cloud point feature, wherein the quality score of the first trace segment indicates the probability that the first trace segment represents a real-life object, and the quality score of the second trace segment indicates the probability that the second trace segment represents a real-life object.

20. The non-transitory computer-readable storage medium according to claim 19, wherein, The first trace segment and the second trace segment are combined by at least concatenating a first trajectory feature of the first trace segment with a second trajectory feature of the second trace segment, wherein the first trajectory feature and the second trajectory feature include at least one of a center point, a size relative to the center point, and a heading angle.

21. The non-transitory computer-readable storage medium according to claim 19, wherein, The first trace segment and the second trace segment are combined by at least concatenating a first cloud point feature of the first trace segment with a second cloud point feature of the second trace segment, wherein the first cloud point feature and the second cloud point feature include at least one of trace points, 3D feature features and motion features.

22. The non-transitory computer-readable storage medium of claim 19, further comprising: It is determined that the first trace segment satisfies the distance threshold and the second trace segment satisfies the distance threshold; as well as In response to the first trace segment and the second trace segment satisfying the distance threshold, it is determined that the first trace segment and the second trace segment represent the same object.

23. The non-transitory computer-readable storage medium according to claim 19, wherein, The first trace segment includes a first plurality of timestamps, and the second trace segment includes a second plurality of timestamps, wherein at least one of the second plurality of timestamps is different from the timestamps in the first plurality of timestamps.

24. The non-transitory computer-readable storage medium of claim 19, further comprising: Determine the quality score of the third trace segment associated with the third trace segment; as well as In response to the third trace segment's quality score failing to meet the quality score threshold, the third trace segment is eliminated.

25. The non-transitory computer-readable storage medium according to claim 19, wherein, The first trace segment and the second trace segment are selected from a plurality of trace segments, and wherein the first trace segment and the second trace segment represent tracked objects outside the vehicle that are spatially monitored relative to the position of the vehicle over a period of time.

26. The non-transitory computer-readable storage medium of claim 19, further comprising: At least one training sample is generated by dividing a known single trace into at least two training trace segments.

27. A method for clearing trace segments for planned movement of a vehicle, comprising: Use at least one processor to detect the first trace segment and the second trace segment; Using the at least one processor, the trace points and intensity of cloud point features corresponding to the first trace feature extracted from the first trace segment and the second trace feature corresponding to the second trace feature extracted from the second trace segment, or based on the center position and timestamp of the trajectory features from the first trace feature and the second trace feature, or based on the size and heading angle of the trajectory features from the first trace feature and the second trace feature, are determined to represent the same object outside the vehicle. Using the at least one processor, a first trace segment quality score associated with the first trace segment and a second trace segment quality score associated with the second trace segment are determined; Using the at least one processor, determine the splicing score between the first trace segment and the second trace segment; and In response to the first trace segment quality score and the second trace segment quality score satisfying a quality score threshold and the splicing score satisfying a splicing score threshold, the at least one processor is used to combine the first trace segment and the second trace segment to form a single trace segment representing the same object outside the vehicle, wherein the first trace segment quality score indicates the probability that the first trace segment represents a real-life object, and the second trace segment quality score indicates the probability that the second trace segment represents a real-life object.

28. A computer program product comprising a computer program configured, when executed by a processor, to perform the trace segment cleanup method according to any one of claims 1-9, 27.

Images