Method, vehicle and storage medium for object detection

Abstract

The invention relates to a method, a vehicle and a storage medium for object detection. Therein, a technique for cross-modal active learning for object detection is described. In an example, a first set of predicted bounding boxes and a second set of predicted bounding boxes are generated. The first set of predicted bounding boxes and the second set of predicted bounding boxes are projected into a same representation. The projections are filtered, wherein predicted bounding boxes satisfying a maximum confidence score are selected for inconsistency computation. Based on the filtering of the projections, an inconsistency is computed across the projected bounding boxes. An informative scene is extracted based on the computed inconsistency. The informative scene is used to train a first object detection neural network or a second object detection neural network.

Description

Technical Field

[0001] This document relates to cross-modal active learning for object detection. Background Technology

[0002] Active learning is a form of machine learning in which a system can query for labels to be applied to new data points. Often, large amounts of unlabeled data are available, but manually labeling data is costly. With active learning, the system can issue queries to obtain labels for the data. Therefore, active learning is an iterative supervised learning technique where input data is analyzed and labeled according to an inference function, and the inference function is used to label new data. Summary of the Invention

[0003] According to one aspect of the present invention, a method includes: generating a first set of predicted bounding boxes and a second set of predicted bounding boxes by a processor, wherein corresponding predicted bounding boxes in the first set of predicted bounding boxes and the second set of predicted bounding boxes are assigned classification scores indicating the presence of object class instances within the corresponding predicted bounding boxes; projecting the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same representation by the processor; filtering the projection by the processor, wherein predicted bounding boxes satisfying the maximum confidence score are selected for inconsistency calculation; calculating inconsistency across the projected bounding boxes by the processor based on the filtering of the projections; extracting an informative scene by the processor based on the calculated inconsistency; and training a first object detection neural network or a second object detection neural network using the informative scene by the processor.

[0004] According to one aspect of the present invention, a non-transitory computer-readable storage medium includes at least one program for execution by at least one processor of a first device, the at least one program including instructions that, when executed by the at least one processor, perform a method comprising: generating a first set of predicted bounding boxes and a second set of predicted bounding boxes, wherein corresponding predicted bounding boxes in the first set of predicted bounding boxes and the second set of predicted bounding boxes are assigned classification scores indicating the presence of object class instances within the corresponding predicted bounding boxes; projecting the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same representation; filtering the projections, wherein predicted bounding boxes satisfying the maximum confidence score are selected for inconsistency calculation; calculating inconsistency across the projected bounding boxes based on the filtering of the projections; extracting an informative scene based on the calculated inconsistency; and training a first object detection neural network or a second object detection neural network using the informative scene.

[0005] According to one aspect of the present invention, a vehicle includes: at least two sensors, wherein a first sensor is communicatively coupled to a first object detection neural network for generating a first set of predicted bounding boxes, and a second sensor is communicatively coupled to a second object detection neural network for generating a second set of predicted bounding boxes, wherein corresponding predicted bounding boxes in the first and second sets of predicted bounding boxes are assigned classification scores indicating the presence of object class instances within the corresponding predicted bounding boxes; at least one computer-readable medium storing computer-executable instructions; and at least one processor communicatively coupled to the at least two sensors and configured to execute the computer-executable instructions, the execution comprising: projecting the first and second sets of predicted bounding boxes into the same representation; filtering the projections, wherein predicted bounding boxes satisfying the maximum confidence score are selected for inconsistency calculation; calculating inconsistency across the projected bounding boxes based on the filtered projections; extracting an informative scene based on the calculated inconsistency; and training the first or second object detection neural network using the informative scene. Attached Figure Description

[0006] Figure 1 An example of an autonomous vehicle (AV) with autonomous capabilities is shown.

[0007] Figure 2 An example "cloud" computing environment is shown.

[0008] Figure 3 The computer system is shown.

[0009] Figure 4 An example architecture for AV is shown.

[0010] Figure 5 Examples of inputs and outputs that can be used by a sensing system are shown.

[0011] Figure 6 An example of a LiDAR system is shown.

[0012] Figure 7 The LiDAR system in operation is shown.

[0013] Figure 8 The operation of the LiDAR system is shown in more detail.

[0014] Figure 9 A block diagram illustrating the relationship between the inputs and outputs of the planning system is shown.

[0015] Figure 10 The directed graph used in path planning is shown.

[0016] Figure 11 A block diagram showing the inputs and outputs of the control system is provided.

[0017] Figure 12 A block diagram of the controller's inputs, outputs, and components is shown.

[0018] Figure 13 It is a system 1300 that enables cross-modal active learning for object detection.

[0019] Figure 14A It is a front view representation of the environment.

[0020] Figure 14B It is a bird's-eye view representation of the environment.

[0021] Figure 14C It is a heatmap representation of the environment as a grid.

[0022] Figure 15 This is a flowchart of the processing flow for cross-modal active learning for object detection. Detailed Implementation

[0023] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that this disclosure may be practiced without these specific details. In other instances, well-known constructions and apparatuses are shown in block diagram form to avoid unnecessarily obscuring this disclosure.

[0024] In the accompanying drawings, for ease of description, a specific arrangement or order of schematic elements (such as those representing devices, modules, systems, instruction blocks, and data elements) is shown. However, those skilled in the art will understand that the specific order or arrangement of the schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or a separation of processing procedures. Furthermore, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0025] Furthermore, in the accompanying drawings, connecting elements, such as solid or dashed lines or arrows, are used to illustrate connections, relationships, or associations between two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, connections, relationships, or associations between some elements are not shown in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions, those skilled in the art will understand that such an element represents one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0026] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0027] The features described below can each be used independently of each other or in any combination with other features. However, any individual feature may not solve any of the problems discussed above, or may only solve one of the problems discussed above. Some of the problems discussed above may not be adequately solved by any of the features described herein. Although headings are provided, information relating to specific headings but not found in the sections bearing those headings can be found elsewhere in this specification. Embodiments are described herein based on the following summary:

[0028] 1. General Overview

[0029] 2. System Overview

[0030] 3. AV Architecture

[0031] 4. AV Input

[0032] 5. AV Planning

[0033] 6. AV Control

[0034] 7. Cross-modal active learning system

[0035] 8. Cross-modal active learning processing

[0036] General Overview

[0037] Vehicles (such as autonomous vehicles) typically have multiple sensors mounted at various locations on the vehicle. Data from these sensors is used for object detection. In object detection, sensor data is analyzed to label the data with confidence scores, which indicate the presence of instances of a specific object class within the regions of data captured by the sensors. For example, sensor data is segmented into one or more dimensional bounding boxes, and each box is labeled with the probability that it contains a specific object class. Objects include, but are not limited to, pedestrians, vehicles, and bicycles. Active learning is implemented across multiple sensor modalities to determine the informative data output by object detection techniques. Cross-modal inconsistencies in the output data are calculated, and the captured scenes are identified as informative scenes using these inconsistencies. Inconsistencies are weighted according to the known strengths and weaknesses of the respective modalities.

[0038] Some advantages of these techniques include a general implementation of black-box methods for determining which scenes are informative and which are not. This technique can be used to complement existing engineering work in collision detection, general object detection, anomaly detection, and sensor fault detection. Furthermore, this technique encompasses any network performing two-dimensional (2D) or three-dimensional (3D) object detection, including but not limited to camera object detection networks, radar object detection networks, and LiDAR object detection networks. Implementations of the systems and techniques described herein can make datasets more informative, thereby enabling more efficient operation of systems using those datasets.

[0039] System Overview

[0040] Figure 1 An example of an AV with autonomous capabilities is shown.

[0041] As used herein, the term “autonomy” refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including but not limited to full AV, high AV, and conditional AV.

[0042] As used in this article, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0043] As used in this article, "vehicle" includes any mode of transport for goods or people. Examples include cars, buses, trains, airplanes, drones, trucks, ships, vessels, submersibles, and spacecraft. Driverless cars are an example of vehicles.

[0044] As used herein, a “track” refers to a path or route that navigates an AV from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as the initial location or starting point, and the second spatiotemporal location is referred to as the destination, final location, target, target location, or target position. In some examples, a track consists of one or more road segments (e.g., segments of a road), each segment consisting of one or more blocks (e.g., a lane or part of an intersection). In embodiments, spatiotemporal locations correspond to real-world locations. For example, a spatiotemporal location is a pick-up or drop-off point for people or goods to board or alight.

[0045] As used herein, “(one or more) sensors” includes one or more hardware components for detecting information relating to the environment surrounding the sensor. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmission and / or receiving components (e.g., laser or radio frequency wave transmitters and receivers), electronic components (such as analog-to-digital converters), data storage devices (such as RAM and / or non-volatile memory), software or firmware components, and data processing components such as ASICs (Application-Specific Integrated Circuits), microprocessors, and / or microcontrollers.

[0046] As used herein, a “scene description” is a data structure (e.g., a list) or data stream that includes one or more classified or tagged objects detected by one or more sensors on an AV vehicle, or one or more classified or tagged objects provided by a source outside the AV.

[0047] As used in this article, a "road" is a physical area that can be traversed by a vehicle and can correspond to a named passageway (e.g., a city street, an interstate highway, etc.) or an unnamed passageway (e.g., a driveway in a house or office building, a section of a parking lot, a section of an vacant parking lot, a waste path in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, SUVs, etc.) can traverse a variety of physical areas that are not particularly suitable for vehicle travel, a "road" can be any physical area that is not formally defined as a passageway by any municipality or other government or administrative agency.

[0048] As used herein, a “lane” is the portion of a road that can be traversed by vehicles. Sometimes lanes are identified based on lane markings. For example, a lane may correspond to most or all of the space between lane markings, or only a portion of the space between lane markings (e.g., less than 50%). For instance, a road with lane markings spaced far apart may accommodate two or more vehicles between the markings, allowing one vehicle to overtake another without crossing the lane markings; therefore, this could be interpreted as a lane being narrower than the space between lane markings, or as having two lanes between the markings. Lanes can also be interpreted in the absence of lane markings. For example, a lane may be defined based on the physical characteristics of the environment (e.g., rocks and trees along a main road in a rural area, or natural obstacles that should be avoided, for example, in underdeveloped areas). Lanes can also be interpreted independently of lane markings or physical characteristics. For example, a lane may be interpreted based on any unobstructed path in an area that would otherwise lack the characteristics that would be interpreted as a lane boundary. In the example scenario, an AV could interpret a lane as a lane traversing an unobstructed portion of a field or open space. In another example scenario, an AV can interpret lanes that pass through a wide road (e.g., wide enough for two or more lanes) without lane markings. In this scenario, an AV can communicate lane-related information to other AVs, allowing them to coordinate route planning using the same lane information.

[0049] The term “over-the-air (OTA) client” includes any AV, or any electronic device embedded in, coupled to, or communicating with an AV (e.g., computer, controller, IoT device, electronic control unit (ECU)).

[0050] The term “over-the-air (OTA) update” means any update, alteration, deletion, or addition to software, firmware, data, or configuration settings, or any combination thereof, delivered to an OTA client using proprietary and / or standardized wireless communication technologies, including but not limited to: cellular mobile communications (e.g., 2G, 3G, 4G, 5G), radio local area networks (e.g., WiFi), and / or satellite Internet.

[0051] The term "edge node" refers to one or more edge devices coupled to a network that provide a portal for communicating with AV and can communicate with other edge nodes and cloud-based computing platforms to schedule OTA updates and deliver OTA updates to OTA clients.

[0052] The term "edge device" refers to a device that implements an edge node and provides a physical wireless access point (AP) to the core network of an enterprise or service provider (such as Verizon or AT&T). Examples of edge devices include, but are not limited to: computers, controllers, transmitters, routers, routing switches, integrated access devices (IADs), multiplexers, metropolitan area network (MAN) and wide area network (WAN) access devices.

[0053] "One or more" includes functions performed by a single element, functions performed by multiple elements, such as in a distributed manner, several functions performed by a single element, several functions performed by several elements, or any combination of the above.

[0054] It will also be understood that, although in some cases the terms first, second, etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0055] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that “and / or” as used herein refers to and includes any and all possible combinations of one or more of the relevant list items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0056] As used herein, depending on the context, the term "if" may optionally be understood as meaning "when" or "at that time" or "in response to being determined" or "in response to being detected." Similarly, depending on the context, the phrase "if determined" or "if [the stated condition or event] has been detected" may optionally be understood as meaning "when determined" or "in response to being determined" or "when [the stated condition or event] is detected" or "in response to being detected."

[0057] This document describes some embodiments of the present disclosure in conjunction with thresholds. As described herein, a threshold can refer to a value greater than a threshold, more than a threshold, higher than a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, lower than a threshold, less than or equal to a threshold, and equal to a threshold, etc.

[0058] As used herein, an AV system refers to an AV system and an array of hardware, software, stored data, and real-time generated data that support AV operation. In embodiments, the AV system is incorporated within an AV unit. In embodiments, the AV system is distributed across several locations. For example, some software of the AV system is discussed below. Figure 2 The cloud computing environment described is implemented in 200 similar cloud computing environments.

[0059] Generally, this document describes technologies applicable to any vehicle with one or more autonomous capabilities, including fully automated vehicle (AV), highly automated vehicle (AV), and conditionally automated vehicle (AV), such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Classification and Definition of Terms Related to Automated Driving Systems for Motor Vehicles on Roads, the entire contents of which are incorporated herein by reference for further details on vehicle autonomy levels). The technologies described in this document are also applicable to partially automated vehicle (AV) and driver-assisted vehicle (MAV) vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Classification and Definition of Terms Related to Automated Driving Systems for Motor Vehicles on Roads). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems may automatically perform certain vehicle operations (e.g., steering, braking, and map usage) under certain operating conditions based on the processing of sensor inputs. The technologies described in this document can benefit vehicles of any level, ranging from fully automated vehicle (AV) to human-operated vehicles.

[0060] AVs have advantages over vehicles that require human drivers. One advantage is safety. For example, in 2016, the U.S. experienced 6 million car accidents, 2.4 million injuries, 40,000 deaths, and 13 million vehicle collisions, with an estimated social cost of over $910 billion. From 1965 to 2015, the number of traffic fatalities per 100 million miles driven in the U.S. decreased from about 6 to about 1, partly due to additional safety features deployed in vehicles. For example, an extra half-second of warning associated with an impending collision is believed to mitigate 60% of front and rear collisions. However, passive safety features (such as seat belts and airbags) may have reached their limits in improving these figures. Therefore, active safety measures, such as automated vehicle controls, are a possible next step in improving these statistics. Since human drivers are considered to be responsible for serious pre-collision events in 95% of collisions, autonomous driving systems could potentially achieve better safety outcomes by: identifying and avoiding emergencies more reliably than humans; making better decisions, obeying traffic regulations better than humans, and predicting future events better than humans; and controlling vehicles more reliably than humans.

[0061] refer to Figure 1 The AV system 120 enables the vehicle 100 to operate along a trajectory 198, traversing the environment 190 to the destination 199 (sometimes referred to as the final location), while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists and other obstacles) and complying with road rules (e.g., operating rules or driving preferences).

[0062] In an embodiment, the AV system 120 includes means 101 for receiving and operating operation commands from and on a computer processor 146. The term "operation command" is used to refer to executable instructions (or a set of instructions) that cause a vehicle to perform actions (e.g., driving maneuvers). Operation commands may, without limitation, include instructions for causing the vehicle to begin moving forward, stop moving forward, begin moving backward, stop moving backward, accelerate, decelerate, make a left turn, and make a right turn. In an embodiment, the computer processor 146 is referenced below. Figure 3 The processor 304 described is similar. Examples of the device 101 include a steering controller 102, a brake 103, a gear, an accelerator pedal or other acceleration control mechanism, a windshield wiper, a side door lock, a window controller, and a turn indicator.

[0063] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the vehicle 100, such as the AV's position, linear velocity and angular velocity, linear acceleration and angular acceleration, and heading (e.g., the orientation of the front end of the vehicle 100). Examples of sensors 121 are GPS, inertial measurement units (IMUs) that measure both linear acceleration and angular rate of the vehicle, wheel rate sensors for measuring or estimating wheel slip ratio, wheel braking pressure or braking torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0064] In an embodiment, sensor 121 also includes sensors for sensing or measuring properties of the environment of the AV. Examples include a monocular or stereo camera 122 in the visible, infrared, or thermal (or both) spectrum, LiDAR 123, radar, ultrasonic sensors, time-of-flight (TOF) depth sensors, rate sensors, temperature sensors, humidity sensors, and precipitation sensors.

[0065] In one embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with a computer processor 146 or data collected by the sensor 121. In another embodiment, the data storage unit 142 is associated with the following... Figure 3 The described ROM 308 or storage device 310 is similar. In this embodiment, memory 144 is similar to main memory 306 described below. In this embodiment, data storage unit 142 and memory 144 store historical, real-time, and / or predictive information related to environment 190. In this embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In this embodiment, data related to environment 190 is transmitted from remote database 134 to vehicle 100 via a communication channel.

[0066] In an embodiment, AV system 120 includes communication devices 140 for communicating measured or inferred attributes of the state and conditions of other vehicles (such as position, linear velocity and angular velocity, linear acceleration and angular acceleration, and linear heading and angular heading) to vehicle 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices, as well as devices for wireless communication via point-to-point or ad hoc networks or both. In an embodiment, communication device 140 communicates across the electromagnetic spectrum (including radio and optical communications) or other media (e.g., air and acoustic media). Combinations of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communication (and in some embodiments, one or more other types of communication) are sometimes referred to as vehicle-to-all-things (V2X) communication. V2X communication typically conforms to one or more communication standards for communication with, between, and within AVs.

[0067] In an embodiment, the communication device 140 includes a communication interface. For example, this may be a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near-field, infrared, or radio interface. The communication interface transmits data from a remote database 134 to the AV system 120. In an embodiment, the remote database 134 is embedded in, for example... Figure 2 In the cloud computing environment 200 described herein, communication device 140 transmits data collected from sensor 121 or other data related to the operation of vehicle 100 to remote database 134. In some embodiments, communication device 140 transmits information related to teleoperation to vehicle 100. In some embodiments, vehicle 100 communicates with other remote (e.g., "cloud") servers 136.

[0068] In this embodiment, the remote database 134 also stores and transmits digital data (e.g., data such as road and street locations). Such data is stored on memory 144 on the vehicle 100 or transmitted from the remote database 134 to the vehicle 100 via a communication channel.

[0069] In one embodiment, the remote database 134 stores and transmits historical information (e.g., rate and acceleration distribution) related to driving attributes of vehicles that previously traveled along trajectory 198 at similar times of day. In one implementation, such data may be stored on memory 144 on vehicle 100 or transmitted from the remote database 134 to vehicle 100 via a communication channel.

[0070] The computer processor 146 located on the vehicle 100 generates control actions in an algorithmic manner based on both real-time sensor data and prior information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0071] In one embodiment, the AV system 120 includes a computer peripheral device 132 coupled to a computer processor 146 for providing information and alerts to a user of the vehicle 100 (e.g., a passenger or a remote user) and receiving input from that user. In another embodiment, the peripheral device 132 is similar to the one described in the following reference. Figure 3 The discussed display 312, input device 314, and cursor controller 316 are coupled wirelessly or wiredly. Any two or more interface devices can be integrated into a single device.

[0072] In one embodiment, the AV system 120 receives and enforces a privacy level for an occupant, such as one specified by the occupant or stored in a profile associated with the occupant. The occupant's privacy level determines how access is permitted to specific occupant-related information (e.g., occupant comfort data, biometric data, etc.) stored in the occupant profile and / or stored on cloud server 136 and associated with the occupant profile. In one embodiment, the privacy level specifies specific occupant-related information that is deleted once the ride is complete. In another embodiment, the privacy level specifies specific occupant-related information and identifies one or more entities authorized to access that information. Examples of the specified entities authorized to access the information may include other AV systems, third-party AV systems, or any entity that could potentially access the information.

[0073] An occupant's privacy level can be specified at one or more granular levels. In one embodiment, the privacy level identifies specific information to be stored or shared. In another embodiment, the privacy level applies to all information associated with the occupant, allowing the occupant to specify that her personal information should not be stored or shared. The designation of entities authorized to access specific information can also be specified at various granular levels. The various sets of entities authorized to access specific information may include, for example, other AVs, cloud server 136, specific third-party AV systems, etc.

[0074] In an embodiment, AV system 120 or cloud server 136 determines whether AV 100 or another entity can access certain information associated with an occupant. For example, a third-party AV system attempting to access occupant input related to a specific time and place must, for example, obtain authorization from AV system 120 or cloud server 136 to access occupant-related information. For example, AV system 120 uses a specified privacy level for the occupant to determine whether location- and time-related occupant input can be presented to a third-party AV system, AV 100, or another AV. This allows the occupant's privacy level to specify which other entities are allowed to receive data related to the occupant's actions or other data associated with the occupant.

[0075] Figure 2An example "cloud" computing environment is shown. Cloud computing is a service delivery model that enables convenient, on-demand access over a network to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, memory, storage, applications, virtual machines, and services). In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver the services provided by the cloud. Now refer to... Figure 2 The cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected via cloud 202. Data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

[0076] A cloud computing environment 200 includes one or more cloud data centers. Generally, a cloud data center (e.g.) Figure 2 The cloud data center 204a shown refers to the cloud (e.g., Figure 2 The physical arrangement of servers in cloud 202 (or a specific portion of the cloud) is illustrated. For example, servers are physically arranged in rooms, groups, rows, and racks within a cloud data center. A cloud data center has one or more regions, each containing one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in regions, rooms, racks, and / or rows are grouped based on the physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In this embodiment, server nodes are similar to... Figure 3 The computer system described herein. Data center 204a has many computing systems distributed across multiple racks.

[0077] Cloud 202 includes cloud data centers 204a, 204b, and 204c, and networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) interconnected with cloud data centers 204a, 204b, and 204c and facilitating access to cloud computing services by computing systems 206a–206f. In embodiments, the network represents any combination of one or more local area networks, wide area networks, or internetworks coupled by wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network is transmitted using various network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, etc. Furthermore, in embodiments where the network represents a combination of multiple subnetworks, different network layer protocols are used on each of the underlying subnetworks. In some embodiments, the network represents one or more interconnected internetworks (such as the public Internet).

[0078] Computing systems 206a-206f or cloud computing service consumers connect to the cloud 202 via network links and network adapters. In embodiments, computing systems 206a-206f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, AV (including cars, drones, space shuttles, trains, buses, etc.), and consumer electronics. In embodiments, computing systems 206a-206f are implemented in other systems or as part of other systems.

[0079] Figure 3 Computer system 300 is illustrated. In implementation, computer system 300 is a dedicated computing device. The dedicated computing device is hardwired to perform these techniques, or includes a digital electronic device persistently programmed to perform the techniques, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), or may include one or more general-purpose hardware processors programmed to execute the techniques according to program instructions in firmware, memory, other memory, or a combination thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to perform the techniques. In various embodiments, the dedicated computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that includes hardwired and / or program logic to implement the techniques.

[0080] In an embodiment, computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a processor 304 coupled to the bus 302 to process information. Processor 304 is, for example, a general-purpose microprocessor. Computer system 300 also includes main memory 306, such as random access memory (RAM) or other dynamic storage device, coupled to the bus 302 to store information and instructions to be executed by processor 304. In one implementation, main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 304. When such instructions are stored in a non-transitory storage medium accessible to processor 304, computer system 300 becomes a dedicated machine customized to perform the operations specified in the instructions.

[0081] In an embodiment, the computer system 300 further includes a read-only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions for the processor 304. A storage device 310, such as a disk, optical disk, solid-state drive, or three-dimensional cross-point memory, is provided and coupled to the bus 302 to store information and instructions.

[0082] In this embodiment, the computer system 300 is coupled via a bus 302 to a display 312, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light-emitting diode (LED) display, or an organic light-emitting diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric keys and other keys, is coupled to the bus 302 for communicating information and command selection to the processor 304. Another type of user input device is a cursor controller 316, such as a mouse, trackball, touchscreen, or cursor arrow keys, for communicating directional information and command selection to the processor 304 and for controlling the movement of the cursor on the display 312. This input device typically has two degrees of freedom on two axes (a first axis (e.g., the x-axis) and a second axis (e.g., the y-axis)), which allow the device to specify a position in a plane.

[0083] According to one embodiment, the techniques described herein are performed by a computer system 300 in response to a processor 304 executing one or more sequences of one or more instructions contained in main memory 306. Such instructions are read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequence of instructions contained in main memory 306 causes the processor 304 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry is used instead of software instructions or in combination with software instructions.

[0084] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media include non-volatile and / or volatile media. For example, non-volatile media include optical discs, magnetic disks, solid-state drives, or three-dimensional cross-point memory such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, floppy disks, floppy hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with perforations, RAM, PROMs and EPROMs, FLASH-EPROMs, NV-RAMs, or any other memory chips or memory cartridges.

[0085] Storage media differ from transmission media, but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, which include conductors forming bus 302. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0086] In embodiments, various forms of media involve carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions may initially be executed on a disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and transmits them over a telephone line using a modem. A local modem of computer system 300 receives the data over the telephone line and converts the data into an infrared signal using an infrared transmitter. An infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 before or after execution by processor 304.

[0087] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides bidirectional data communication coupled to network link 320 connected to local network 322. For example, communication interface 318 is an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem used to provide data communication connectivity to a corresponding type of telephone line. As another example, communication interface 318 is a Local Area Network (LAN) card used to provide data communication connectivity to a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0088] Network link 320 typically provides data communication to other data devices via one or more networks. For example, network link 320 provides connectivity to host computer 324 or to a cloud data center or device operated by Internet Service Provider (ISP) 326 via local network 322. ISP 326, in turn, provides data communication services via a worldwide packet data communication network now commonly referred to as the "Internet" 328. Both local network 322 and Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks and signals on network link 320 via communication interface 318 are example forms of transmission media carrying digital data entering and leaving computer system 300. In embodiments, network 320 includes the aforementioned cloud 202 or a portion of cloud 202.

[0089] Computer system 300 sends messages and receives data including program code through one or more networks, network links 320, and communication interfaces 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310 or other non-volatile storage device for later execution.

[0090] AV architecture

[0091] Figure 4 It shows a method for AV (e.g., Figure 1 The example architecture 400 of the vehicle 100 shown is illustrated. Architecture 400 includes a sensing system 402 (sometimes referred to as a sensing circuit), a planning system 404 (sometimes referred to as a planning circuit), a control system 406 (sometimes referred to as a control circuit), a positioning system 408 (sometimes referred to as a positioning circuit), and a database system 410 (sometimes referred to as a database circuit). Each system plays a role in the operation of the vehicle 100. Commonly, systems 402, 404, 406, 408, and 410 can be... Figure 1 This is a portion of the AV system 120 shown. In some embodiments, any of systems 402, 404, 406, 408, and 410 is a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application-specific integrated circuits (ASICs), hardware memory devices, other types of integrated circuits, other types of computer hardware, or any or all combinations of these hardware). Each of systems 402, 404, 406, 408, and 410 is sometimes referred to as processing circuitry (e.g., computer hardware, computer software, or a combination of both). Any or all combinations of systems 402, 404, 406, 408, and 410 are also examples of processing circuitry.

[0092] In use, the planning system 404 receives data representing the destination 412 and determines data representing the trajectory 414 (sometimes called a route) that the vehicle 100 can travel to reach (e.g., arrive at) the destination 412. In order for the planning system 404 to determine the data representing the trajectory 414, the planning system 404 receives data from the sensing system 402, the positioning system 408, and the database system 410.

[0093] The sensing system 402 uses, for example, as Figure 1 One or more sensors 121 are shown to identify nearby physical objects. The objects are classified (e.g., grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.), and a scene description including the classified objects 416 is provided to the planning system 404.

[0094] The planning system 404 also receives data representing the location 418 of the AV from the positioning system 408. The positioning system 408 calculates the location to determine the AV's position using data from sensor 121 and data from database system 410 (e.g., geographic data). For example, the positioning system 408 uses data from GNSS (Global Navigation Satellite System) sensors and geographic data to calculate the longitude and latitude of the AV. In embodiments, the data used by the positioning system 408 includes high-precision maps with lane geometry properties, maps describing road network connectivity properties, maps describing lane physical properties (such as traffic speed, traffic volume, number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or combinations thereof), and maps describing the spatial locations of road features (such as intersections, traffic signs, or various other types of traffic signals). In embodiments, the high-precision map is constructed by adding data to a low-precision map via automatic or manual annotation.

[0095] The control system 406 receives data representing trajectory 414 and data representing AV position 418, and operates the AV control functions 420a-420c (e.g., steering, throttle, braking, ignition) in a manner that will cause the vehicle 100 to travel along trajectory 414 to reach destination 412. For example, if trajectory 414 includes a left turn, the control system 406 will operate the control functions 420a-420c in such a way that the steering angle of the steering function will cause the vehicle 100 to turn left, and the throttle and brake will cause the vehicle 100 to stop and wait for passing pedestrians or vehicles before making the turn.

[0096] AV input

[0097] Figure 5 The sensing system 402 is shown. Figure 4 The inputs used are 502a to 502d (for example, Figure 1 Examples of sensor 121 and outputs 504a-504d (e.g., sensor data) are shown. One input 502a is a LiDAR (light detection and ranging) system (e.g., Figure 1 The LiDAR system shown is 123. LiDAR is a technique that uses light (e.g., a beam of light such as infrared light) to obtain data related to physical objects in its line of sight. The LiDAR system produces LiDAR data as output 504a. For example, LiDAR data is a collection of 3D or 2D points (also called point clouds) used to construct a representation of environment 190.

[0098] Another input 502b is a RADAR (radar) system. RADAR is a technology that uses radio waves to acquire data related to nearby physical objects. RADAR can acquire data related to objects that are not within the line of sight of a LiDAR system. The RADAR system generates RADAR data as output 504b. For example, RADAR data is one or more radio frequency electromagnetic signals used to construct a representation of environment 190.

[0099] Another input 502c is a camera system. The camera system uses one or more cameras (e.g., a digital camera using a light sensor such as a charge-coupled device [CCD]) to acquire information related to nearby physical objects. The camera system produces camera data as output 504c. Camera data is typically in the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, for example, for stereoscopic imaging (stereoscopic vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as "nearby," this is relative to the AV (view of objects). In some embodiments, the camera system is configured to "see" distant objects (e.g., objects as far as 1 kilometer or more in front of the AV). Therefore, in some embodiments, the camera system has features such as sensors and lenses optimized for perceiving distant objects.

[0100] Another input 502d is a Traffic Light Detection (TLD) system. The TLD system uses one or more cameras to acquire information related to traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system produces TLD data as output 504d. TLD data is often in the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). The TLD system differs from systems that include cameras in that it uses cameras with a wide field of view (e.g., using a wide-angle lens or fisheye lens) to acquire information related to as many physical objects as possible that provide visual navigation information, enabling the vehicle 100 to access all relevant navigation information provided by these objects. For example, the TLD system has a field of view of approximately 120 degrees or greater.

[0101] In some embodiments, sensor fusion technology is used to combine outputs 504a-504d. Thus, individual outputs 504a-504d are provided to other systems of the vehicle 100 (e.g., to systems such as...). Figure 4The planning system 404 shown may provide combined outputs to other systems in the form of single or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs or both) or single or multiple combined outputs of different types (e.g., using different corresponding combination techniques or combining different corresponding outputs or both). In some embodiments, an early fusion technique is used. The early fusion technique is characterized by combining the outputs before applying one or more data processing steps to the combined outputs. In some embodiments, a late fusion technique is used. The late fusion technique is characterized by combining the outputs after applying one or more data processing steps to the individual outputs.

[0102] Figure 6 An example of a LiDAR system 602 is shown (e.g., Figure 5 The input 502a is shown. The LiDAR system 602 emits light 604a-604c from a emitter 606 (e.g., a laser emitter). The light emitted by the LiDAR system is typically not in the visible spectrum; for example, infrared light is often used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and is reflected back to the LiDAR system 602. (The light emitted from the LiDAR system typically does not penetrate the physical object, e.g., a solid physical object.) The LiDAR system 602 also has one or more photodetectors 610 for detecting the reflected light. In an embodiment, one or more data processing systems associated with the LiDAR system generate an image 612 representing the field of view 614 of the LiDAR system. Image 612 includes information representing the boundary 616 of the physical object 608. Thus, image 612 is used to determine the boundary 616 of one or more physical objects near the AV.

[0103] Figure 7 The diagram illustrates a LiDAR system 602 in operation. In the scenario shown, the vehicle 100 receives both a camera system output 504c in the form of an image 702 and a LiDAR system output 504a in the form of LiDAR data points 704. In use, the vehicle 100's data processing system compares the image 702 with the data points 704. Specifically, physical objects 706 identified in the image 702 are also identified within the data points 704. Thus, the vehicle 100 perceives the boundaries of physical objects based on the contours and density of the data points 704.

[0104] Figure 8 Additional details of the operation of the LiDAR system 602 are shown. As described above, the vehicle 100 detects the boundaries of physical objects based on the characteristics of the data points detected by the LiDAR system 602. Figure 8As shown, a flat object, such as ground 802, will reflect light 804a-804d emitted from LiDAR system 602 in a consistent manner. In other words, because LiDAR system 602 emits light at a consistent interval, ground 802 will reflect light back to LiDAR system 602 at the same consistent interval. When vehicle 100 travels on ground 802, LiDAR system 602 will continue to detect light reflected by the next effective surface point 806 if nothing obstructs its path. However, if object 808 obstructs its path, the light 804e-804f emitted by LiDAR system 602 will be reflected from points 810a-810b in a manner inconsistent with the expected consistency. Based on this information, vehicle 100 can determine the presence of object 808.

[0105] Path planning

[0106] Figure 9 Show (for example, as) Figure 4 The diagram 900 illustrates the relationship between the inputs and outputs of the planning system 404. Generally, the output of the planning system 404 is a route 902 from a starting point 904 (e.g., a source location or initial location) to an ending point 906 (e.g., a destination or final location). Route 902 is typically defined by one or more road segments. For example, a road segment refers to a distance to be traveled over at least a portion of a street, road, highway, driveway, or other physical area suitable for vehicle travel. In some examples, such as if the vehicle 100 is an off-road capable vehicle such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck, route 902 includes “off-road” segments such as unpaved paths or open fields.

[0107] In addition to route 902, the planning system also outputs lane-level route planning data 908. Lane-level route planning data 908 is used to navigate segments of route 902 at specific times based on conditions. For example, if route 902 comprises a multi-lane highway, lane-level route planning data 908 includes trajectory planning data 910, which vehicle 100 can use to select lanes from the multiple lanes based on factors such as whether an exit is nearby, whether other vehicles are present in one or more lanes, or other factors that change over a period of minutes or less. Similarly, in some implementations, lane-level route planning data 908 includes a speed constraint 912 specific to a segment of route 902. For example, if the segment includes pedestrians or unexpected traffic, speed constraint 912 can limit vehicle 100 to a slower speed than expected, such as a speed limit based on the segment's speed limit data.

[0108] In an embodiment, the inputs to the planning system 404 include (e.g., from...) Figure 4 The database system 410 shown contains database data 914 and current location data 916 (for example, Figure 4 The AV position shown is 418), (for example, for use with Figure 4 The destination data 918 and object data 920 shown for destination 412 (e.g., as shown) Figure 4 The perception system 402 shown perceives classified objects 416. In some embodiments, database data 914 includes rules used during planning. The rules are specified using a formal language (e.g., Boolean logic). At least some of these rules will apply to any given situation encountered by vehicle 100. A rule applies to a given situation if it has conditions satisfied based on information available to vehicle 100 (e.g., information about the surrounding environment). Rules can have priorities. For example, a rule “move to the leftmost lane if the road is a highway” can have a lower priority than “move to the rightmost lane if the exit is within a mile.”

[0109] Figure 10 This is illustrated in path planning (e.g., by planning system 404). Figure 4 The directed graph used is 1000. Generally speaking, such as... Figure 10 The directed graph 1000 shown is used to determine any path between a starting point 1002 and an ending point 1004. In the real world, the distance separating the starting point 1002 and the ending point 1004 may be relatively large (e.g., in two different urban areas) or relatively small (e.g., two intersections adjacent to a city block or two lanes of a multi-lane road).

[0110] In an embodiment, the directed graph 1000 has nodes 1006a-1006d representing different locations that a vehicle 100 may occupy between a starting point 1002 and an ending point 1004. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different urban areas, nodes 1006a-1006d represent road segments. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different locations on the same road, nodes 1006a-1006d represent different locations on that road. Thus, the directed graph 1000 includes information at different levels of granularity. In an embodiment, the directed graph with higher granularity is also a subgraph of another directed graph with a larger scale. For example, most of the information in a directed graph where the starting point 1002 and the ending point 1004 are far apart (e.g., many miles apart) is at a low granularity, and the directed graph is based on stored data, but the directed graph also includes some high-granularity information for representing a portion of the physical location in the field of view of the vehicle 100.

[0111] Nodes 1006a-1006d are distinct from objects 1008a-1008b that cannot overlap with nodes. In an embodiment, at a low granularity, objects 1008a-1008b represent areas that vehicles cannot pass through, such as areas without streets or roads. At a high granularity, objects 1008a-1008b represent physical objects within the field of view of vehicle 100, such as other vehicles, pedestrians, or other entities with which vehicle 100 cannot share physical space. In an embodiment, some or all of objects 1008a-1008b are static objects (e.g., objects that do not change position, such as streetlights or utility poles) or dynamic objects (e.g., objects that can change position, such as pedestrians or other cars).

[0112] Nodes 1006a-1006d are connected by edges 1010a-1010c. If two nodes 1006a-1006b are connected by edge 1010a, then vehicle 100 can travel between one node 1006a and the other node 1006b, for example, without having to travel to an intermediate node before reaching the other node 1006b. (When it is mentioned that vehicle 100 travels between nodes, it means that vehicle 100 travels between two physical locations represented by the respective nodes.) Edges 1010a-1010c are typically bidirectional, meaning that vehicle 100 can travel from a first node to a second node, or from a second node to a first node. In an embodiment, edges 1010a-1010c are unidirectional, meaning that vehicle 100 can travel from a first node to a second node, but not from a second node to a first node. When edges 1010a-1010c represent, for example, a one-way street, a single lane of a street, road, or highway, or other features that can only be traversed in one direction due to legal or physical constraints, edges 1010a-1010c are one-way.

[0113] In an embodiment, the planning system 404 uses a directed graph 1000 to identify a path 1012 consisting of nodes and edges between a start point 1002 and an end point 1004.

[0114] Edges 1010a-1010c have associated costs 1014a-1014b. Costs 1014a-1014b represent the resources that would be spent if vehicle 100 selected that edge. A typical resource is time. For example, if the physical distance represented by one edge 1010a is twice the physical distance represented by another edge 1010b, then the associated cost 1014a of the first edge 1010a can be twice the associated cost 1014b of the second edge 1010b. Other factors affecting time include anticipated traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. The two edges 1010a-1010b can represent the same physical distance, but due to factors such as road conditions and anticipated weather, one edge 1010a may require more fuel than the other edge 1010b.

[0115] When the planning system 404 identifies the path 1012 between the starting point 1002 and the ending point 1004, the planning system 404 typically selects the path that is optimized for cost, such as the path that has the minimum total cost when the individual costs of the edges are added together.

[0116] AV control

[0117] Figure 11 Show (for example, as) Figure 4The diagram shows a block diagram 1100 of the inputs and outputs of the control system 406. The control system operates according to a controller 1102, which includes, for example: one or more processors similar to processor 304 (e.g., one or more computer processors such as a microprocessor or microcontroller or both); short-term and / or long-term data storage devices similar to main memory 306, ROM 308 and storage device 310 (e.g., memory, random access memory or flash memory or both); and instructions stored in the memory that, when executed (e.g. by one or more processors), perform the operation of controller 1102.

[0118] In one embodiment, controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes speed, such as rate and heading. The desired output 1104 may be based, for example, from (e.g., as...) Figure 4 The data received by the planning system 404 (as shown) is used as follows. Based on the desired output 1104, the controller 1102 generates data that can be used as throttle input 1106 and steering input 1108. Throttle input 1106 indicates, for example, engaging the throttle of the vehicle 100 (e.g., acceleration control) to achieve the magnitude of the desired output 1104 by engaging the steering pedal or another throttle control. In some examples, throttle input 1106 also includes data that can be used to engage the brakes of the vehicle 100 (e.g., deceleration control). Steering input 1108 indicates the steering angle, such as the steering control of the AV (e.g., steering wheel, steering angle actuator, or other function for controlling the steering angle), which should be positioned to achieve the desired output 1104.

[0119] In one embodiment, controller 1102 receives feedback used when adjusting inputs provided to throttle and steering. For example, if vehicle 100 encounters an obstacle 1110 such as a hill, the measured rate 1112 of vehicle 100 drops below the desired output rate. In another embodiment, any measured output 1114 is provided to controller 1102 so that necessary adjustments can be made, for example, based on the difference 1113 between the measured rate and the desired output. The measured output 1114 includes measured position 1116, measured speed 1118 (including rate and heading), measured acceleration 1120, and other sensor-measurable outputs of vehicle 100.

[0120] In one embodiment, information related to interference 1110 is detected in advance, for example, by a sensor such as a camera or LiDAR sensor, and this information is provided to a predictive feedback system 1122. The predictive feedback system 1122 then provides information that the controller 1102 can use to make appropriate adjustments. For example, if the vehicle 100's sensors detect ("see") a hill, the controller 1102 can use this information to prepare to engage the throttle at an appropriate time to avoid significant deceleration.

[0121] Figure 12 A block diagram 1200 shows the inputs, outputs, and components of controller 1102. Controller 1102 has a rate analyzer 1202 that affects the operation of throttle / brake controller 1204. For example, the rate analyzer 1202 instructs throttle / brake controller 1204 to accelerate or decelerate using throttle / brake 1206 based on feedback received by, for example, controller 1102 and processed by the rate analyzer 1202.

[0122] Controller 1102 also has a lateral tracking controller 1208 that affects the operation of steering controller 1210. For example, lateral tracking controller 1208 instructs steering controller 1210 to adjust the position of steering angle actuator 1212 based on feedback received by controller 1102 and processed by lateral tracking controller 1208.

[0123] Controller 1102 receives several inputs for determining how to control the throttle / brake 1206 and the steering angle actuator 1212. Planning system 404 provides controller 1102 with information such as selecting the heading of vehicle 100 at the start of operation and determining which road segment vehicle 100 will cross when it reaches an intersection. Positioning system 408 provides controller 1102 with information describing the current location of vehicle 100, such as allowing controller 1102 to determine whether vehicle 100 is at a location expected based on positive control of the throttle / brake 1206 and steering angle actuator 1212. In embodiments, controller 1102 receives information from other inputs 1214, such as information received from a database, computer network, etc.

[0124] Cross-modal active learning system

[0125] Figure 13 This is system 1300, which implements cross-modal active learning for object detection. In system 1300, active learning is performed across multiple sensor modalities to determine informative data. Vehicles (e.g., Figure 1 The vehicle 100 can be detected by multiple sensors (e.g., Figure 1Sensor 121 records a large amount of data. However, data storage space is limited, and often constrained. The active learning technique described herein reduces the amount of data being stored to informative data, enabling the provision of high-quality data for improving other vehicle functions, such as collision detection, general object detection, and sensor failure detection, while reducing the required data storage space. Furthermore, the reduction in data storage allows this technique to be implemented using vehicle hardware instead of dedicated servers in a data center.

[0126] System 1300 (for example, Figure 1 The system 120) includes at least one camera 1302 (e.g., Figure 1 The camera 122) and at least one LiDAR 1304 (e.g., Figure 1 (LiDAR 123). Camera 1302 outputs image data 1306 in the form of images in image formats such as RAW, JPEG, PNG, etc. (e.g., Figure 5 The output is 504c). LiDAR 1304 outputs LiDAR data in the form of point cloud 1308 (e.g., Figure 5 The output 504a). At least one camera 1302 and at least one LiDAR 1304 are, for example, a sensing system 402 ( Figure 4 Components of a perception system, such as camera 1302 and LiDAR 1304. Therefore, camera 1302 and LiDAR 1304 output data for identifying classified objects such as pedestrians, bicycles, cars, traffic signs, etc. Figure 4 The data of the classified objects (416). For ease of explanation, specific sensor modes such as cameras, LiDAR, and radar are described according to this technique. However, this technique can be implemented via any sensor mode that is utilized to capture data analyzed for object detection.

[0127] Image data 1306 typically provides accurate measurements of edges, color, and illumination, ultimately producing accurate object classification and localization on the image plane (e.g., as determined by the localization system 408). However, 3D localization is often inaccurate when based solely on camera data. Compared to image data, point clouds 1308 typically contain less semantic information and can instead achieve highly accurate 3D localization. Point clouds are generally sparse, and their range is typically limited to 50–100 meters (m). This technique leverages the advantages of multiple sensor modalities to achieve robust object detection across sensor modalities.

[0128] System 1300 includes two object detection networks: an image semantic segmentation network (ISN) 1310 and a LiDAR semantic segmentation network (LSN) 1312. Generally, the object detection neural network is configured to receive sensor data and process the sensor data to detect at least one object (e.g., in the 3D space surrounding the sensor) in the image semantic segmentation network. Figure 1 Natural obstacles 191, vehicles 193, pedestrians 192; cyclists and other obstacles). In this embodiment, the object detection neural network is a feedforward convolutional neural network that, given outputs 504a–504d (e.g., sensor data), generates a set of bounding boxes for potential objects in 3D space and confidence scores for the presence of object class instances (e.g., cars, pedestrians, or bicycles) within the bounding boxes. The higher the classification score, the greater the likelihood that the corresponding object class instance exists in the box. An example object detection neural network is described in detail in W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, CY Fu, and ACBerg. SSD: Single Shot MultiboxDetector. Springer, 201 (available at https: / / arxiv.org / pdf / 1512.02325.pdf).

[0129] ISN 1310 takes image data 1306 as input and outputs a set of predicted 3D bounding boxes 1314 for potential objects in 3D space, along with corresponding confidence scores for the presence of object class instances within the bounding boxes. The 3D bounding boxes include information related to the size, orientation, and location of the object's 3D bounding box, in a format that can be projected into the same data representation described below. Specifically, ISN 1310 takes a set of images as input, predicts the class of each pixel in the images, and outputs semantic segmentation data (e.g., confidence scores) for each pixel in the output images. ISN 1310 is trained using an image dataset comprising images, each augmented with bounding boxes and segmentation labels for the classes in the image dataset. Example confidence scores are probability values ​​indicating the probability of correctly predicting a pixel class.

[0130] Similarly, LSN 1312 takes at least one point cloud 1308 as input and outputs a set of predicted 3D bounding boxes 1316 for potential objects in 3D space, along with confidence scores for the presence of object class instances within the bounding boxes. In the example, the LSN receives multiple data points representing 3D space. For example, each data point is a set of 3D spatial coordinates (e.g., x, y, z coordinates). As described in “PointPillars: Fast Encoders for ObjectDetection from Point Clouds” (arXiv:1812.05784v2[cs.LG]) dated May 7, 2019, the data is encoded into pillars and output as pseudo-images. Generally, pillar encoders take one or more point clouds as input and estimate oriented 3D bounding boxes for cars, pedestrians, and cyclists based on the point clouds. Similar to ISN 1310, the predicted 3D bounding box 1316 output by LSN includes information related to the size, orientation, and location of the object's 3D bounding box, in a format that can be projected into the same data representation as described below. The predicted 3D bounding box set also includes a confidence score for the presence of object class instances within the bounding box.

[0131] Post-processing is used to project the predicted 3D bounding boxes 1314 and 1316 onto the same representation. In system 1300, the predicted 3D bounding box 1314 based on image data is sent to post-processing 1318. The predicted 3D bounding box 1316 based on point cloud is sent to post-processing 1320. As used herein, a representation is a transformation applied to the predicted 3D bounding boxes. When applied to predicted 3D bounding boxes obtained from multiple object detection networks, this transformation ensures that the scene is interpreted in the same format. In embodiments, the representation is at least partially based on camera angles. In embodiments, the representation is a probability distribution applied to confidence scores associated with the predicted bounding boxes. For example, the representation could be a front view representation, a bird's-eye view representation, or a heatmap representation. Although a particular representation has been described, this technique is not limited to a single representation used for the projection of predicted 3D bounding boxes. Instead, this technique transforms the predicted 3D bounding boxes into the same data representation that can be compared across multiple object detection networks.

[0132] In the example, the predicted 3D bounding box is projected into the front view representation. The front view representation is a graph of the bounding box projected onto the image plane. Figure 14AThis is the front view representation 1400A of the environment 1402A. As illustrated, the front view representation 1400A includes multiple projected bounding boxes 1404A. Bounding boxes 1404A are predicted 3D bounding boxes projected onto the front view representation 1400A from the object detection network. In the front view representation, bounding boxes 1404A are aligned along the x-axis of coordinate 1406A. In other words, the bounding boxes are not rotated along the x-axis and have edges parallel to the yz plane.

[0133] In the example, the predicted 3D bounding boxes are projected onto a bird's-eye view representation. Generally, a bird's-eye view representation is a rendering of the environment projected onto a ground plane. Bird's-eye views eliminate occlusion between detected objects and enable semantically and geometrically consistent relationships between vehicles and obstacles. Figure 14B This is a bird's-eye view representation 1400B of the environment 1402B. As illustrated, the bird's-eye view representation 1400B includes multiple bounding boxes 1404B. Bounding boxes 1404B are predicted 3D bounding boxes projected onto the bird's-eye view representation 1400B from the object detection network. In the bird's-eye view representation, bounding boxes 1404B are not aligned along the z-axis of coordinate 1406B. In other words, the bounding boxes are rotated along the z-axis and have no edges parallel to the xy-plane. The projected bounding boxes 1404B are non-axis-aligned.

[0134] In the example, a heatmap representation is generated. The heatmap representation is generated by transforming the bird's-eye view representation based on confidence scores. Generally, the heatmap represents size as color in two dimensions. Sizes are inserted into a matrix with one or more fixed cells. The columns and rows of the heatmap are discrete phenomena and categories. Figure 14C This is a heatmap of the environment 1400C, representing grid 1402C. The discrete phenomena and categories represented on grid 1402C are the classifications and confidence scores of the objects detected in the bird's-eye view representation. Figure 14C In the example, each cell represents one or more projected bounding boxes in the bird's-eye view (e.g., Figure 14B The confidence score of the bounding box (1404B) is used. In the example, a confidence score of 1.0 is assigned to the background region without a bounding box. The cells of mesh 1402C are distinguished based on the confidence scores of their associated projected bounding boxes.

[0135] Refer again Figure 13 The predicted 3D bounding boxes 1314 and 1316, output by ISN 1310 and LSN 1312 respectively, overlap when projected onto the same representation (e.g., respectively in...). Figure 14A and 14BThe projected bounding boxes 1404A and 1404B overlap. Therefore, post-processing 1318 and post-processing 1320 apply nonmaximum suppression to filter redundant projected bounding boxes belonging to the same object. Filtering redundant projected bounding boxes removes them from the list of suggested bounding boxes. Nonmaximum suppression varies depending on the same representation selected. Figure 14A , 14B As illustrated in 14C, the same representation can be a front view representation, a bird's-eye view representation, or a heatmap representation. Generally, during object detection, confidence scores are created for each bounding box, resulting in many high scores that are close to each other, thus generating a large number of predicted 3D bounding boxes. Ultimately, this creates a large number of projected bounding boxes with associated confidence scores in the same representation. Non-maximum suppression suppresses scores that are not maximum values ​​within local ranges.

[0136] In an embodiment, post-processing 1318 and 1320 includes filtering via non-maximum suppression, where the intersection-to-union ratio (IoU) is calculated for each group of overlapping boxes. IoU is the ratio between the intersection and union of the projected bounding boxes and the bounding boxes with the highest confidence scores. In the example, the front view represents the projected bounding boxes (e.g., box A) that include the highest confidence scores. Bounding box A is selected and removed from a list of projected bounding boxes (e.g., a list of B) and added to a final list of projected bounding boxes (e.g., a list of F). The projected list of F is initially empty. The projected bounding box A with the highest confidence score is compared with all the remaining projected bounding boxes in the list of B to determine the IoU between boxes A and the list of B. If the IoU between A and each of the Bs is greater than an IoU threshold, the projected bounding box B is removed from the list of B. IoU is a predetermined threshold that can tune the strength of the final predicted bounding boxes. This process is repeated until there are no remaining projected bounding boxes in the list of B. Specifically, the projected bounding box with the second-highest confidence (e.g., the next box A) is selected again from the remaining projected bounding boxes (e.g., the rest of the list of B) and added to the final list of projected bounding boxes F. The IoU between the next box A and the rest of the list of B is calculated, and boxes with IoU above the IoU threshold are removed again. This process is repeated until there are no more recommendations in the remaining list of suggested bounding boxes B. Thus, redundant projected bounding boxes are removed.

[0137] In embodiments, post-processing 1318 and 1320 includes filtering via non-maximum suppression, where the filtering is due to non-axis-aligned bounding boxes (e.g., Figure 14B The IoU calculation is modified for the bounding box (1404B). In the bird's-eye view representation, the projected bounding box may be rotated along the z-axis instead of aligned to the axis. The nonmaximum suppression applied to the predicted 3D bounding box projected onto the bird's-eye view is modified to account for the rotation that occurs.

[0138] For example, this modification determines the IoU of convex polygons to account for rotations between projected bounding boxes. In the modified IoU determination, the recommender (e.g., box A) with the highest confidence is iteratively selected from the list of projected bounding boxes (e.g., the list of B) added to the final projection list (e.g., list F). The IoU of all recommenders in the list of B with box A is found, and boxes with IoUs above the IoU threshold are removed again. This process is repeated until there are no more recommenders in the list of projected bounding boxes (B). When determining the IoU between bounding boxes in the list of B and box A, all corners of box A contained within box B are found. All corners of box B contained within box A are found. The intersection points between box A and box B are found, and all points are sorted clockwise using arctan2.

[0139] In the embodiments, post-processing 1318 and post-processing 1320 enable post-processing for the heatmap representation. If more than one bounding box is assigned to the same cell in the heatmap, the bounding box with the highest confidence score is selected as the final bounding box associated with that cell. In this way, projected bounding boxes that do not meet the threshold of the highest confidence score are removed.

[0140] Generally, post-processing 1318 and post-processing 1320 can be configured based on the same representation used for projection. The outputs of post-processing 1318 and post-processing 1320 are the final set of predicted bounding boxes projected onto the same representation and filtered to select the best projected bounding boxes for each sensor modality. The projected bounding boxes are associated with confidence scores indicating the presence of object class instances within the bounding boxes. Object class instances are used to label the projected bounding boxes with a specific class. The outputs of post-processing 1318 and post-processing 1320 are used for inconsistency calculation 1322. During inconsistency calculation 1322, the predictions of ISN 1310 and LSN 1312 are compared. This comparison is based on error or uncertainty. In an embodiment, the inconsistencies obtained from error-based or uncertainty-based calculations are weighted according to the known strengths and weaknesses of the respective modalities.

[0141] In this embodiment, the output of the inconsistency calculation 1322 is an active learning score estimating the inconsistency between sensor modes. The projected bounding boxes of cross-modal inconsistencies are used to determine whether the scene is informative. As used herein, a scene refers to a portion or the entire environment. In this embodiment, a scene refers to a portion or the entire environment at a specific time instance. Generally, inconsistency refers to a divergence in corresponding information between a first sensor mode and a second sensor mode.

[0142] The bounding boxes are ranked using active learning scores, and the N bounding boxes with the highest information content are selected as informational samples and designated as such informational samples within the scene. In some examples, a scene is identified as an informational scene if it includes at least one active learning score indicating inconsistencies between identical representations across sensor modalities. In embodiments, a scene is considered informational if inconsistencies occur in a portion of the scene exceeding a predetermined threshold. For example, if the calculated number of inconsistencies exceeds a predetermined threshold, the scene can be labeled as informational and used to train an object detection neural network. A scene can be labeled as informational if errors or uncertainties are found in more than 25% of the projected bounding boxes. A scene is also labeled as informational if errors or uncertainties are found in projected bounding boxes within a predetermined distance of a vehicle. For example, a scene is identified as informational if the location associated with the calculated inconsistency is within a predetermined distance of a vehicle. Informational scenes provide additional or supplementary labeled data from which the object detection network can learn. In addition, informational scenarios provide supplementary or additional labeled data, for which known object detection networks are affected by inherent limitations.

[0143] Error-based inconsistency calculations are represented as:

[0144] E(ISN(S i ),LSN(S l )) (1)

[0145] Where E is a function used to calculate the error between ISN 1310 and LSN 1312, and S i and S l It is the set of the first set of projected bounding boxes and the second set of projected bounding boxes. In the embodiment, function E is any one or more statistical classifications. Generally, the predictions of object detection networks of multiple modalities are compared. The predictions of the object detection network are predicted 3D bounding boxes that have been projected onto the same representation and post-processed. Regarding inconsistency calculation 1322, predicted 3D bounding boxes that have been projected onto the same representation and post-processed are generally referred to as predictions.

[0146] In error-based inconsistency calculations, the prediction of the first object detection network for the sensor modality is labeled as a false true value, while the predictions of the remaining object detection networks are labeled as true predictions. The error between the false true value and the predictions of the remaining object detection networks is determined. For example, the predictions of ISN and LSN are compared, and either the ISN prediction or the LSN prediction is labeled as a false true value, and many error types are counted.

[0147] When calculating the error, the bounding boxes are weighted. In this embodiment, the closer the bounding box is to the vehicle, the higher its weight. This gives greater importance to potentially informative scenarios when inconsistencies occur closer to the vehicle compared to inconsistencies farther away. An exemplary linear weighting function is provided below:

[0148]

[0149] In equation (2), in the bird's-eye view representation, distance is the calculated distance between the center of the bounding box and the position of the vehicle. In this embodiment, no weighting function is applied, and a weight value of 1 is assigned to each prediction.

[0150] Implement machine learning statistical metrics to determine error-based uncertainty. Machine learning statistical metrics include, but are not limited to, false positives and false negatives (FP+FN), precision, recall, and F1 score, or any combination thereof. Generally, machine learning statistical metrics are based on the ground truth compared to the prediction. When evaluating machine learning statistical metrics, a false positive (FP) is an error used to indicate the existence of a condition when that condition actually does not exist. A false negative (FN) is an error that incorrectly indicates the absence of a condition. A true positive is a correctly indicated positive condition, and a true negative is a correctly indicated negative condition. Therefore, for the FP+FN statistical metric, a false positive is a prediction that does not have a sufficiently high IoU with any ground truth box. A false negative occurs when the confidence score of the detection used to detect the ground truth is below a predetermined threshold. The number of FP+FN errors is counted between false ground truth modal predictions and other modal predictions.

[0151] Generally, precision is the number of true positives divided by the sum of true positives and false positives. Subtracting precision from 1 yields an active learning score where higher precision results in lower inconsistency. Lower inconsistency indicates that the associated projections are consistent. Similarly, recall is the number of true positives divided by the sum of true positives and false positives. Subtracting recall from 1 yields an active learning score where higher recall results in lower inconsistency. The F1 score is a balanced F-score and is the harmonic mean of precision and recall. In this embodiment, the F1 score is a measure of precision. Precision is the probability that a randomly selected instance (positive or negative, relevant or irrelevant) will be correct. Similar to the precision and recall statistics, subtracting the F1 score from 1 yields an active learning score where higher F1 scores result in lower inconsistency. Lower inconsistency indicates that the associated predictions are consistent across modalities. In this example, a predetermined threshold is applied to the active learning score. In this example, if the score is higher than 0.5, the associated prediction is determined to be inconsistent. To account for relationships across the entire range of confidence scores, uncertainty-based inconsistency computation is applied. Generally, uncertainty-based inconsistency computation is used to compute cross-modal inconsistencies with the probability distributions of individual true positive pairs from each object detection network. Uncertainty-based inconsistency computation can also be divided into two subgroups: bounding boxes and heatmaps.

[0152] In the example, inconsistency calculation based on the first uncertainty is used to compute the JS divergence between each pair of true positives. The Jensen-Shannon (JS) divergence measures the similarity between two probability distributions, such as the probability distribution associated with the bounding box. The JS divergence is symmetric and always has a finite value. In this embodiment, a maximum value is assigned to each false positive and false negative. In the example, the maximum value is 1.0.

[0153] Alternatively, when predicting 3D bounding boxes mapped to heatmap representations (probabilistic maps), mutual information (MI) is used to compute the active learning score. Generally, mutual information is a measure of the interdependence between two variables. In this embodiment, these two variables are two heatmaps. Mutual information quantifies the amount of information about a random variable gained by observing another random variable. The probabilities from the two heatmaps in the class are averaged. Specifically, the mutual information of each unit is computed using the following equation:

[0154]

[0155] Mutual information is calculated for each unit as follows:

[0156]

[0157] Where H represents the entropy function. Finally, the final mutual information of the heatmap is calculated as:

[0158]

[0159] Where I represents the total number of units in the heatmap.

[0160] Figure 13 The diagram is not intended to indicate that active learning as described in this article will include Figure 13 All components shown. Conversely, cross-modal active learning can include fewer or... Figure 13 Additional components not illustrated herein (e.g., additional modalities, detection networks, post-processing, inconsistency calculation, etc.). Depending on the specific implementation details, active learning may include any number of additional components not shown. Furthermore, any of the ISN, LSN, post-processing, inconsistency calculation, and other described functions may be implemented partially or wholly in hardware and / or a processor. For example, the function may be implemented using an application-specific integrated circuit, in logic implemented in a processor, in logic implemented in a dedicated graphics processing unit, or in any other device.

[0161] Figure 14A , 14B The block diagrams for 14C are not intended to indicate that the representation described herein will include Figure 14A , 14B And all the components shown in 14C. Conversely, these representations may include fewer or... Figure 14A , 14B And additional components not illustrated in 14C (e.g., objects, object classes, bounding boxes, etc.). Depending on the specific implementation details, these representations may include any number of additional components not shown. Furthermore, any functionality described may be implemented, in part or in whole, in hardware and / or a processor. For example, the functionality may be implemented using an application-specific integrated circuit, in logic implemented in a processor, in logic implemented in a dedicated graphics processing unit, or in any other device.

[0162] Cross-modal active learning processing

[0163] Figure 15 This is a flowchart of the processing 1500 for cross-modal active learning for object detection. At box 1502, a first set of predicted bounding boxes and a second set of predicted bounding boxes are obtained. These are communicated with the object detection neural network (e.g., ISN 1310, LSN 1312) via corresponding sensors (e.g., [missing information]). Figure 1 The first sensor (e.g., sensor 121) is used to generate the predicted bounding box. Figure 13 The camera 1302) and the first object detection neural network (e.g., Figure 13 The first object detection neural network is communicatively coupled to the ISN 1310, and generates a first set of predicted bounding boxes (e.g., Figure 13The predicted 3D box 1314). Additionally, a second sensor (e.g., Figure 13 LiDAR 1304) and a second object detection neural network (e.g., Figure 13 The second object detection neural network is communicatively coupled to the LSN1312, and generates a second set of predicted bounding boxes (e.g., ...). Figure 13 The predicted 3D bounding boxes (1316) are each assigned a confidence score to indicate the presence of an object class instance within the corresponding predicted bounding box.

[0164] At box 1504, the first set of predicted bounding boxes and the second set of predicted bounding boxes are projected onto the same representation (e.g., Figure 14A The front view shows 1400A, Figure 14B The bird's-eye view indicates 1400B or Figure 14C The heatmap is shown at 1400°C. In post-processing (e.g., ...), Figure 13 During post-processing 1318 and post-processing 1320, the predicted bounding boxes are projected onto the same representation.

[0165] At box 1506, the projection is filtered, where predicted bounding boxes that satisfy the maximum confidence score are selected for inconsistency calculation. In this embodiment, the filtering is based on type-adapted nonmaximum suppression of the same representation. In other words, the form of nonmaximum suppression can be varied based on the same representation to which the bounding boxes are projected. For example, for a front view representation, filtering via nonmaximum suppression includes determining the intersection-over-union ratio (IoU) between each set of overlapping bounding boxes and selecting the bounding box with the maximum confidence score as the final score. In another example, filtering via nonmaximum suppression for a bird's-eye view includes determining the IoU between each set of overlapping bounding boxes, where the IoU calculation is modified to determine the intersection points of convex polygons, thereby taking into account rotations between predicted boxes when projected onto the bird's-eye view representation. The bounding box with the maximum confidence score is selected as the final score. In another example, filtering via nonmaximum suppression is applied to a heatmap representation. For example, in the case where more than one predicted bounding box is assigned to the same cell in the heatmap, the box with the highest confidence score is selected as the final bounding box associated with that cell.

[0166] At box 1508, inconsistencies are calculated across the filtered projected bounding boxes based on the filtered projection. Inconsistency calculations (e.g., Figure 13(1322) is based on error, uncertainty, or any combination thereof. Inconsistency is used to obtain the active learning score. At box 1510, the scene extracted based on the calculated inconsistency is an informative scene. For example, a scene is informative if it includes at least one active learning score indicating inconsistency between the same representations across sensor modalities. In an embodiment, a scene is informative if inconsistency occurs in a portion of the scene exceeding a predetermined threshold. For example, a scene can be marked as informative if an error or uncertainty is found in more than 25% of the projected bounding box. A scene can also be marked as informative if an error or uncertainty is found in the projected bounding box within a predetermined distance of the vehicle. By determining whether a scene is informative, only meaningful data is stored, thus preventing the need to store all recorded data. Therefore, this technique can operate within a limited storage space. Due to the reduction in storage space, this technique is implemented on a vehicle.

[0167] Annotated scenes are used in machine learning to train models and improve their understanding of various environments. Annotation is typically done by human operators, which is costly and susceptible to human bias and error. Generally, the more exposure a machine learning model has to labeled datasets, the more robust and complete the model becomes. This technique reduces annotation budgets by selecting and annotating data of interest based on model understanding. In this embodiment, the technique mines raw data from logs using a given set of images and point clouds instead of a single input modality.

[0168] Because it doesn't need to store all the data, the ability to automatically filter out meaningless data creates a more efficient system with sufficient storage capacity. Each object detection network uses active learning to select data for learning, enabling the core machine learning algorithm of the object detection network to achieve high accuracy.

[0169] Figure 15 The processing flowchart is not intended to indicate that the boxes in example process 1500 will be executed in that order or that all boxes will be included in every case. Furthermore, depending on the specific implementation details, any number of additional boxes not shown may be included within example process 1500.

[0170] In the preceding description, embodiments of the invention have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of the invention, and what the applicant expects to be the scope of the invention, is the literal and equivalent scope of the claims published from this application in the specific form of the claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as such terms are used in the claims. Furthermore, when the term “comprising” is used in the preceding specification or appended claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method comprising: The processor generates a first set of predicted bounding boxes based on images from an image sensor and a second set of predicted bounding boxes based on point clouds from a LiDAR sensor, wherein the corresponding predicted bounding boxes in the first set and the second set are assigned classification scores to indicate the presence of object class instances within the corresponding predicted bounding box. The processor projects the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same representation. The processor filters the projection, wherein a first subset of predicted bounding boxes that satisfy the maximum confidence score is selected from the first set of predicted bounding boxes, and a second subset of predicted bounding boxes that satisfy the maximum confidence score is selected from the second set of predicted bounding boxes. The processor calculates, based on filtering the projection, the inconsistency between a first subset of predicted bounding boxes associated with the image from the image sensor and a second subset of predicted bounding boxes associated with the point cloud from the LiDAR sensor; The processor extracts informational scenarios based on the calculated number of inconsistencies exceeding a predetermined threshold; and The processor is used to train a first object detection neural network or a second object detection neural network using the information-rich scene.

2. The method according to claim 1, further comprising: When the number of inconsistencies calculated exceeds a predetermined threshold, the information-rich scenarios are extracted.

3. The method according to claim 1 or 2, further comprising: Weights are assigned to each predicted bounding box in the first subset and the second subset based on the distance between each projected bounding box and the vehicle.

4. The method according to claim 1 or 2, wherein, The same representation is one of the front view representation, bird's-eye view representation, and heat map representation.

5. The method according to claim 1, further comprising: Project the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same front view representation; The projections are filtered using nonmaximum suppression to eliminate redundant projections belonging to the same object; and The inconsistency of the confidence score of the filtered projection is calculated by error-based inconsistency calculation.

6. The method according to claim 1, further comprising: Project the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same bird's-eye view representation; The projections are filtered using nonmaximum suppression with a modified intersection-to-union ratio to eliminate redundant projections belonging to the same object; and The inconsistency of the confidence score of the filtered projection is calculated by error-based inconsistency calculation.

7. The method according to claim 1, further comprising: Project the first set of predicted bounding boxes and the second set of predicted bounding boxes onto the same heatmap representation; The projection is filtered by selecting the highest confidence score of multiple bounding boxes assigned to the cells in the heatmap representation; and The inconsistency of the confidence score of the filtered projection is calculated by error-based inconsistency calculation.

8. The method according to claim 1, further comprising: Project the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same front view representation; The projections are filtered using nonmaximum suppression to eliminate redundant projections belonging to the same object; and The inconsistency of the confidence score of the filtered projection is calculated by inconsistency calculation based on uncertainty.

9. The method according to claim 1, further comprising: Project the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same bird's-eye view representation; The projections are filtered using nonmaximum suppression with a modified cross-union ratio to eliminate redundant projections belonging to the same object; and The inconsistency of the confidence score of the filtered projection is calculated by inconsistency calculation based on uncertainty.

10. The method according to claim 1 or 2, further comprising: When the location associated with the calculated inconsistency is within a predetermined distance of the vehicle, the information-rich scenario is extracted.

11. A non-transitory computer-readable storage medium comprising at least one program executable by at least one processor of a first device, the at least one program comprising instructions that, when executed by the at least one processor, perform a method comprising: Generate a first set of predicted bounding boxes based on images from an image sensor and a second set of predicted bounding boxes based on point clouds from a LiDAR sensor, wherein the corresponding predicted bounding boxes in the first set and the second set are assigned classification scores to indicate the presence of object class instances within the corresponding predicted bounding box. Project the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same representation; The projection is filtered, wherein a first subset of predicted bounding boxes that satisfy the maximum confidence score is selected from the first set of predicted bounding boxes, and a second subset of predicted bounding boxes that satisfy the maximum confidence score is selected from the second set of predicted bounding boxes. Based on filtering the projection, inconsistencies are calculated between a first subset of predicted bounding boxes associated with the image from the image sensor and a second subset of predicted bounding boxes associated with the point cloud from the LiDAR sensor. Scenarios that extract information based on the calculated number of inconsistencies exceeding a predetermined threshold; and The informational scenario is used to train the first object detection neural network or the second object detection neural network.

12. The non-transitory computer-readable storage medium of claim 11, further comprising: When the number of inconsistencies calculated exceeds a predetermined threshold, the information-rich scenarios are extracted.

13. The non-transitory computer-readable storage medium according to claim 11 or 12, the method further comprising: Weights are assigned to each predicted bounding box in the first subset and the second subset based on the distance between each projected bounding box and the vehicle.

14. The non-transitory computer-readable storage medium according to claim 11 or 12, wherein, The same representation is one of the front view representation, bird's-eye view representation, and heat map representation.

15. A vehicle comprising: At least two sensors, including an image sensor and a LiDAR sensor, wherein the image sensor is communicatively coupled to a first object detection neural network for generating a first set of predicted bounding boxes based on an image from the image sensor, and the LiDAR sensor is communicatively coupled to a second object detection neural network for generating a second set of predicted bounding boxes based on a point cloud from the LiDAR sensor, wherein corresponding predicted bounding boxes in the first set and the second set of predicted bounding boxes are assigned classification scores to indicate the presence of object class instances within the corresponding predicted bounding box; At least one computer-readable medium storing computer-executable instructions; and At least one processor, communicatively coupled to the image sensor and the LiDAR sensor, and configured to execute the computer-executable instructions, the execution of which includes: Project the first set of predicted bounding boxes and the second set of predicted bounding boxes into the same representation; The projection is filtered, wherein a first subset of predicted bounding boxes that satisfy the maximum confidence score is selected from the first set of predicted bounding boxes, and a second subset of predicted bounding boxes that satisfy the maximum confidence score is selected from the second set of predicted bounding boxes. Based on filtering the projection, inconsistencies are calculated between a first subset of predicted bounding boxes associated with the image from the image sensor and a second subset of predicted bounding boxes associated with the point cloud from the LiDAR sensor. Scenarios that extract information based on the calculated number of inconsistencies exceeding a predetermined threshold; and The informational scenario is used to train the first object detection neural network or the second object detection neural network.

16. The vehicle according to claim 15, wherein, The operation also includes: When the number of inconsistencies calculated exceeds a predetermined threshold, the information-rich scenarios are extracted.

17. The vehicle according to claim 15 or 16, wherein, The same representation is one of the front view representation, bird's-eye view representation, and heat map representation.

18. The vehicle according to claim 15 or 16, wherein, The first object detection neural network is an image semantic segmentation network, and the second object detection neural network is a LiDAR semantic segmentation network.

19. The vehicle according to claim 15 or 16, wherein the operation further comprises: Weights are assigned to each predicted bounding box in the first subset and the second subset based on the distance between each projected bounding box and the vehicle.

20. A computer program product comprising a program configured to cause a computer to perform the method according to any one of claims 1 to 10.

Images