Using scene dependent object queries to generate bounding boxes

Abstract

A perception system may be used to generate bounding boxes for objects in a vehicle scene. A perceptual system may receive an image and a feature map corresponding to the received image. A perception system may generate a scene dependent object query. The perception system may generate one or more bounding boxes for objects in the vehicle scene using the generated scene-dependent object queries.

Description

Background Art

[0001] An autonomous vehicle can use images obtained from one or more image sensors to generate bounding boxes for objects in a vehicle scenario. Brief Description of the Drawings

[0002] Figure 1 is an example environment of a vehicle that can implement one or more components of an autonomous system.

[0003] Figure 2 is a diagram of one or more systems of a vehicle including an autonomous system.

[0004] Figure 3 is Figure 1 and Figure 2 diagrams of one or more devices and / or components of one or more systems of.

[0005] Figure 4A is a diagram of certain components of an autonomous system.

[0006] Figure 4B is a diagram of an implementation of a neural network.

[0007] Figure 4C and Figure 4D are diagrams illustrating example operations of a CNN.

[0008] Figure 5 is a block diagram of an example perception environment illustrating that a perception system receives and processes images to provide one or more (3D) bounding boxes for objects in a vehicle scenario.

[0009] Figure 6A is a data flow diagram of an example of a perception environment illustrating that a perception system generates a bounding box from an image.

[0010] Figure 6B is a data flow diagram of an example of a perception environment illustrating that a perception system generates a bounding box from an image.

[0011] Figure 7 is a flowchart of an example of a routine implemented by at least one processor to navigate a vehicle based on at least one bounding box generated from one or more images.

[0012] Figure 8 is a flowchart of an example of a routine implemented by at least one processor to navigate a vehicle based on at least one bounding box generated from one or more images. Detailed Description

[0013] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the embodiments described herein may be practiced without these specific details. In some instances, well-known structures and devices are illustrated in block diagram form in order to avoid unnecessarily obscuring aspects of the present disclosure.

[0014] In the drawings, for ease of description, specific arrangements or orderings of illustrative elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements, etc.) are illustrated. However, those skilled in the art will understand that unless explicitly described, the specific order or arrangement of the illustrative elements in the drawings is not intended to imply a required order or sequence of processing, or a separation of processing. Further, unless explicitly described, the inclusion of illustrative elements in the drawings is not intended to imply that such elements are required in all embodiments, nor that the features represented by such elements cannot be included in some embodiments or combined with other elements in some embodiments.

[0015] In addition, in the drawings, connecting elements (such as solid lines, dashed lines, or arrows, etc.) are used to illustrate connections, relationships, or associations between or among two or more other illustrative elements. The absence of any such connecting element is not intended to mean that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the present disclosure. Further, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of a signal, data, or instruction (e.g., “software instruction”), those skilled in the art will understand that such an element may represent one or more than one signal path (e.g., a bus) that may be required to affect the communication.

[0016] Although terms such as “first,” “second,” and / or “third,” etc. are used to describe various elements, these elements should not be limited by these terms. The terms “first,” “second,” and / or “third” are only used to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0017] The terms used in the description of the various embodiments herein are included only for the purpose of describing particular embodiments and are not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms "a", "an", and "the" are also intended to include the plural forms and may be used interchangeably with "one or more than one" or "at least one", unless the context clearly dictates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense) such that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Unless specifically stated otherwise, disjunctive language such as the phrase "at least one of X, Y, and Z" is generally understood, in the context in which it is used, to mean that the items, terms, etc. can be X, Y, or Z, or any combination thereof (e.g., X, Y, or Z). Thus, such disjunctive language is generally not intended to and should not imply that certain embodiments require the presence of at least one of each of X, at least one of Y, and at least one of Z. It will also be understood that when the terms "comprises", "comprising", "includes", and / or "including" are used in this specification, they specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0018] As used herein, the terms "communicate" and "communicating" refer to at least one of receiving, receipt, transmission, conveyance, and / or provision of information (or information represented by, for example, data, signals, messages, instructions, and / or commands, etc.). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof, etc.) to communicate with another unit, this means that the unit can directly or indirectly receive information from and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Additionally, two units can communicate with each other even if the information transmitted between the first unit and the second unit is modified, processed, relayed, and / or routed. For example, even if the first unit receives information passively and does not actively transmit information to the second unit, the first unit can communicate with the second unit. As another example, if at least one intermediate unit (e.g., a third unit located between the first unit and the second unit) processes the information received from the first unit and transmits the processed information to the second unit, the first unit can communicate with the second unit. In some embodiments, a message can refer to a network packet (e.g., a data packet, etc.) that includes data.

[0019] The conditional language used herein (such as "can", "may", "might", "could", and "for example", etc.) is generally intended to convey that certain embodiments include certain features, elements, or steps, while other embodiments do not include certain features, elements, or steps, unless specifically stated otherwise or understood otherwise within the context in which it is used. Thus, such conditional language is generally not intended to imply that the features, elements, or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether to include or perform these features, elements, or steps in any particular embodiment in the presence or absence of other inputs or cues. As used herein, depending on the context, the term "if" is optionally interpreted to mean "when", "upon", "in response to determining that", and / or "in response to detecting", etc. Similarly, depending on the context, the phrase "if it has been determined" or "if [the stated condition or event] has been detected" is optionally interpreted to mean "upon determining...", "in response to determining that", or "upon detecting [the stated condition or event]" and / or "in response to detecting [the stated condition or event]", etc. Additionally, as used herein, the terms "has", "have", or "having", etc. are intended to be open-ended terms. Further, unless otherwise expressly stated, the phrase "based on" is intended to mean "at least partially based on".

[0020] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0021] Overview

[0022] To effectively navigate through various scenarios, an autonomous vehicle uses computer vision to identify objects in the scene and then navigates the scene based on the identified objects. As part of the navigation process, the autonomous vehicle can draw three-dimensional (3D) bounding boxes around the objects in the image to understand the spatial relationship between the objects and the autonomous vehicle.

[0023] In some cases, it can be challenging to draw accurate 3D bounding boxes for objects in a real-time driving environment because the neural network cannot obtain sufficient semantic and local information about the objects. Additionally, individual cameras on an autonomous vehicle may not be able to capture the entire object, which increases the difficulty of identifying objects in the vehicle's scene. Thus, individual feature maps corresponding to different cameras may not have sufficient semantic data to enable the drawing of accurate bounding boxes.

[0024] To address these issues, an autonomous vehicle can utilize scene-dependent object queries. Scene-independent object queries can be enriched, or scene-independent object queries can be cross-correlated with each other and with other image features to generate scene-dependent queries. Then, the scene-dependent object queries can be used at subsequent time steps of the scene analysis.

[0025] In the present disclosure, a system is configured to generate and utilize a combination of scene-dependent object queries and scene-independent object queries to generate bounding boxes. An autonomous vehicle can collect scene data associated with an image capture device (e.g., a camera) obtained at a time step, and the scene data can be collected at a defined interval (such as 10 hz). The scene data can be collected and object queries can be generated, which are used to generate 3D bounding boxes associated with objects within the vehicle's field of view.

[0026] Object queries can be calculated based on scene-independent queries. Scene-independent object queries are general queries that are processed at each time step without reference to the scene data collected during previous time steps. Compared with scene-dependent object queries, converging scene-independent object queries may require a large amount of processing. The system can use a vision transformer to converge the object queries, and the vision transformer can include self-attention and cross-attention functions to converge the object queries and generate bounding boxes for objects within the image scene. For example, scene-independent object queries can utilize a vision transformer with six decoder layers in sequence to generate enriched object queries. This may result in a processing delay for detecting objects within a time step. When the object queries are scene-independent, the system processes the scene data at each time step and does not benefit from previous calculations. As a result, at each time step, the system can newly process 900 or more queries.

[0027] After initializing the system using scene-independent queries, the system can generate scene-dependent queries. From a defined number of scene-independent object queries (such as, for example, 900), a subset of scene-dependent object queries (such as, for example, 200) is generated. The scene-dependent object queries can be selected from the set of enriched scene-independent object queries based on a defined number of enriched queries and / or a confidence threshold associated with the enriched queries. For example, the system can select the top 200 queries with the highest confidence thresholds, a number of queries that meet the confidence threshold, or a number of queries that meet the confidence threshold up to a threshold number of queries.

[0028] By using scene-dependent object queries (e.g., a subset of enriched scene-independent queries), the detection process for objects within a scene can be improved. The queries can be used to generate bounding boxes for objects having a feature map or a set of feature maps. An autonomous vehicle can reduce processing requirements and increase the speed and efficiency of processing the queries. These efficiencies can increase the rate at which the autonomous vehicle can accurately identify objects in an image. For example, in some cases, the use of scene-dependent object queries can reduce the number of iterations required for query convergence, such as from six iterations to two iterations.

[0029] Additionally, scene-dependent object queries can improve the ability of an autonomous vehicle to identify objects within the scene of the autonomous vehicle. For example, scene-dependent object queries can improve the ability of an autonomous vehicle to identify objects and determine corresponding bounding boxes.

[0030] General Overview

[0031] By virtue of the implementations of the systems, methods, and computer program products described herein, an autonomous vehicle can more accurately identify objects within an image, more accurately identify the locations of the identified objects within the image, more accurately predict the trajectories of the identified objects within the image, determine additional features of the identified objects, and infer additional information related to the scene of the image.

[0032] Now refer to Figure 1, an exemplary environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a - 102n, objects 104a - 104n, routes 106a - 106n, area 108, vehicle - to - infrastructure (V2I) devices 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a - 102n, vehicle - to - infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via a wired connection, a wireless connection, or a combination of wired and wireless connections (e.g., establishing a connection for communication, etc.). In some embodiments, objects 104a - 104n are interconnected with at least one of vehicles 102a - 102n, vehicle - to - infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0033] Vehicles 102a - 102n (individually referred to as vehicle 102 and collectively as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, vehicle 102 is the same as or similar to vehicle 200 described herein (see Figure 2 ). In some embodiments, vehicles 200 in the set of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicle 102 travels along corresponding routes 106a - 106n (individually referred to as route 106 and collectively as routes 106). In some embodiments, one or more than one vehicle 102 includes an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0034] Objects 104a - 104n (individually referred to as object 104 and collectively as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., building, sign, fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and over a period of time) is stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in region 108.

[0035] Routes 106a - 106n (individually referred to as route 106 and collectively as routes 106) are each associated with (e.g., define) a series of actions (also referred to as a trajectory) that a connected AV can navigate along. Each route 106 begins at an initial state (e.g., a state corresponding to a first spatio - temporal location and / or speed, etc.) and ends at a final target state (e.g., a state corresponding to a second spatio - temporal location different from the first spatio - temporal location) or a target zone (e.g., a subspace of acceptable states (e.g., termination states)). In some embodiments, the first state includes a location where one or more individuals will board the AV, and the second state or zone includes one or more locations where one or more individuals boarding the AV will disembark. In some embodiments, route 106 includes a plurality of acceptable state sequences (e.g., a plurality of spatio - temporal location sequences) that are associated with (e.g., define) a plurality of trajectories. In an example, route 106 includes only high - level actions or imprecise state locations, such as a series of connected roads indicating a direction change at a roadway intersection. Additionally or alternatively, route 106 can include more precise actions or states, such as, for example, a specific target lane or precise location within a lane region and a target rate at those locations. In an example, route 106 includes a plurality of precise state sequences along at least one high - level action with a finite look - ahead horizon to reach an intermediate target, where the combination of successive iterations of the finite - horizon state sequences cumulatively corresponds to a plurality of trajectories that together form a high - level route terminating at the final target state or zone.

[0036] Region 108 includes a physical region (e.g., a geographical region) that the vehicle 102 can navigate. In an example, region 108 includes at least one state (e.g., a country, a province, an individual state among a plurality of states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial (referred to herein as a "road"), such as a highway, an interstate highway, a parkway, a city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of a vacant and / or undeveloped area, a dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In an example, a road includes at least one lane associated with (e.g., identified based on) at least one lane marking line.

[0037] A vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to-everything (V2X) device) includes at least one device configured to communicate with the vehicle 102 and / or the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, the platoon management system 116, and / or the V2I system 118 via the network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), a lane marking, a streetlight, a parking meter, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the platoon management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0038] The network 112 includes one or more wired and / or wireless networks. In an example, the network 112 includes a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of some or all of these networks.

[0039] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the network 112, the queue management system 116, and / or the V2I system 118 via the network 112. In an example, the remote AV system 114 includes a server, a server group, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing, etc.). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0040] The queue management system 116 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the V2I system 118. In an example, the queue management system 116 includes a server, a server group, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ridesharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems), etc.).

[0041] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipal authority or a private institution (e.g., a private institution for maintaining the V2I device 110, etc.).

[0042] Provide Figure 1 The number and arrangement of the illustrated elements are provided as examples. Compared with Figure 1 the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements in a different arrangement. Additionally or alternatively, at least one element of the environment 100 may perform one or more functions described as being performed by Figure 1 at least one different element. Additionally or alternatively, at least one set of elements of the environment 100 may perform one or more functions described as being performed by at least one different set of elements of the environment 100.

[0043] Now refer toFigure 2 , the vehicle 200 includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 is the same as or similar to the vehicle 102 (see Figure 1 ). In some embodiments, the vehicle 200 has autonomous capabilities (e.g., implements at least one of the following functions, features, and / or devices, etc., where the at least one function, feature, and / or device enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that dispense with reliance on human intervention) and / or highly autonomous vehicles (e.g., vehicles that dispense with reliance on human intervention in certain situations), etc.). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference can be made to SAE International Standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire content of which is incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ridesharing company.

[0044] The autonomous system 202 includes a sensor suite that includes one or more devices such as a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, and a microphone 202d. In some embodiments, the autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by the vehicle 200, etc.). In some embodiments, the autonomous system 202 uses one or more of the devices included in the autonomous system 202 to generate data associated with the environment 100 described herein. The data generated by one or more of the devices of the autonomous system 202 can be used by one or more of the systems described herein to observe the environment in which the vehicle 200 is located (e.g., the environment 100). In some embodiments, the autonomous system 202 includes a communication device 202e, an autonomous vehicle computing 202f, and a safety controller 202g.

[0045] The camera 202a includes being configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with Figure 3at least one device that communicates over a bus 302 that is the same as or similar to the bus). The camera 202a includes at least one camera (e.g., a digital camera using an optical sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) configured to capture an image including a physical object (e.g., a car, a bus, a curb, and / or a person, etc.). In some embodiments, the camera 202a generates camera data as output. In some examples, the camera 202a generates camera data including image data associated with the image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, and / or an image timestamp, etc.). In such an example, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a includes a plurality of independent cameras configured on (e.g., positioned on) a vehicle to capture images for the purpose of stereoscopic vision (stereo vision). In some examples, the camera 202a includes a plurality of cameras that generate image data and transmit the image data to the autonomous vehicle computing 202f and / or a queue management system (e.g., a queue management system that is the same as or similar to the Figure 1 queue management system 116 of). In such an example, the autonomous vehicle computing 202f determines the depth to one or more objects in the field of view of at least two of the plurality of cameras based on the image data from at least two cameras. In some embodiments, the camera 202a is configured to capture an image of an object within a distance (e.g., up to 100 meters and / or up to 1 kilometer, etc.) relative to the camera 202a. Thus, the camera 202a includes features such as sensors and lenses optimized to sense objects at one or more distances relative to the camera 202a.

[0046] In an embodiment, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, the camera 202a generates traffic light data associated with one or more images. In some examples, the camera 202a generates TLD data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data is different from other systems incorporating cameras described herein in that: the camera 202a may include one or more cameras having a wide field of view (e.g., a wide-angle lens, a fish-eye lens, and / or a lens having a viewing angle of about 120 degrees or greater, etc.) to generate images related to as many physical objects as possible.

[0047] A Light Detection and Ranging (LiDAR) sensor 202b includes at least one device configured to communicate with a communication device 202e, an autonomous vehicle computing 202f, and / or a safety controller 202g via a bus (e.g., a bus 302 that is the same as or similar to Figure 3 the bus). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object that the light encounters. The LiDAR sensor 202b further includes at least one light detector that detects the light after the light emitted from the light emitter encounters a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing the objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface), etc. In such examples, the image is used to determine the boundary of the physical object in the field of view of the LiDAR sensor 202b.

[0048] A Radio Detection and Ranging (Radar) sensor 202c includes at least one device configured to communicate with a communication device 202e, an autonomous vehicle computing 202f, and / or a safety controller 202g via a bus (e.g., a bus that is the same as or similar to Figure 3At least one device that communicates via a bus 302 that is the same as or similar to the bus). The Radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the Radar sensor 202c encounter physical objects and are reflected back to the Radar sensor 202c. In some embodiments, the radio waves emitted by the Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the Radar sensor 202c generates a signal representing the objects included in the field of view of the Radar sensor 202c. For example, at least one data processing system associated with the Radar sensor 202c generates an image representing the boundaries of the physical object and / or the surface of the physical object (e.g., the topology of the surface), etc. In some examples, the image is used to determine the boundaries of the physical objects in the field of view of the Radar sensor 202c.

[0049] The microphone 202d includes at least one device configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., a bus Figure 3 that is the same as or similar to the bus 302). The microphone 202d includes one or more microphones (e.g., an array microphone and / or an external microphone, etc.) that capture an audio signal and generate data associated with (e.g., representing) the audio signal. In some examples, the microphone 202d includes a transducer device and / or a similar device. In some embodiments, one or more of the systems described herein may receive the data generated by the microphone 202d and determine the position (e.g., distance, etc.) of the object relative to the vehicle 200 based on the audio signal associated with the data.

[0050] The communication device 202e includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the drive-by-wire (DBW) system 202h. For example, the communication device 202e may include a device Figure 3 that is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0051] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, a safety controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets, etc.), and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units, etc.). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., an autonomous vehicle system the same as or similar to the remote AV system 114 of Figure 1 ), a queue management system (e.g., a queue management system the same as or similar to the queue management system 116 of Figure 1 ), a V2I device (e.g., a V2I device the same as or similar to the V2I device 110 of Figure 1 ), and / or a V2I system (e.g., a V2I system the same as or similar to the V2I system 118 of Figure 1 ).

[0052] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, the autonomous vehicle computing 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (such as electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., a powertrain control system 204, a steering control system 206, and / or a braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing 202f.

[0053] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., an electrical controller and / or an electromechanical controller, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., the powertrain control system 204, the steering control system 206, and / or the braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.).

[0054] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives a control signal from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a certain direction, decelerate in a certain direction, make a left turn, and / or make a right turn, etc. In an example, the powertrain control system 204 increases, maintains the same, or decreases the energy (e.g., fuel and / or electricity, etc.) provided to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0055] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes two front wheels and / or two rear wheels of the vehicle 200 to rotate left or right, so that the vehicle 200 makes a left turn or a right turn.

[0056] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate the vehicle 200 and / or keep it stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the corresponding rotors of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0057] In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring properties of the state or condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), a wheel speed sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor.

[0058] Now referring to Figure 3 , a schematic diagram of illustrative device 300 is shown. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of a system of vehicle 102); and / or one or more than one device of network 112 (e.g., one or more than one device of a system of network 112). In some embodiments, one or more than one device of vehicle 102 (e.g., at least one device of a system of vehicle 102), and / or one or more than one device of network 112 (e.g., one or more than one device of a system of network 112) includes at least one device 300 and / or at least one component of device 300. As Figure 3 shown, device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0059] Bus 302 includes components that permit communication between the components of device 300. In some instances, processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), etc.). Memory 306 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic and / or static storage device that stores data and / or instructions for use by processor 304 (e.g., flash memory, magnetic memory, and / or optical memory, etc.).

[0060] The storage component 308 stores data and / or software related to the operation and use of the device 300. In some examples, the storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk, etc.), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette tape, a magnetic tape, a CD-ROM, a RAM, a PROM, an EPROM, a FLASH-EPROM, an NV-RAM, and / or another type of computer-readable medium, and corresponding drives.

[0061] The input interface 310 includes components that permit the device 300 to receive information such as via a user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, and / or a camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes sensors for sensing information (e.g., a global positioning system (GPS) receiver, an accelerometer, a gyroscope, and / or an actuator, etc.). The output interface 312 includes components for providing output information from the device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs), etc.).

[0062] In some embodiments, the communication interface 314 includes transceiver-like components that permit the device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection (e.g., a transceiver and / or separate receivers and transmitters, etc.). In some examples, the communication interface 314 permits the device 300 to receive information from another device and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, an interface, and / or a cellular network interface, etc.

[0063] In some embodiments, the device 300 performs one or more of the processes described herein. The device 300 performs these processes based on software instructions stored by a computer-readable medium such as the memory 306 and / or the storage component 308 and executed by the processor 304. A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes a storage space located within a single physical storage device or a storage space distributed across multiple physical storage devices.

[0064] In some embodiments, software instructions are read into memory 306 and / or storage component 308 from another computer-readable medium or from another device via communication interface 314. When the software instructions stored in memory 306 and / or storage component 308 are executed, they cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used in place of or in combination with software instructions to perform one or more processes described herein. Accordingly, unless otherwise expressly stated, embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0065] Memory 306 and / or storage component 308 includes a data store or at least one data structure (e.g., a database, etc.). Device 300 is capable of receiving information from the data store or at least one data structure in memory 306 or storage component 308, storing information in the data store or at least one data structure, communicating information to the data store or at least one data structure, or searching for information stored in the data store or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0066] In some embodiments, device 300 is configured to execute software instructions stored in memory 306 and / or the memory of another device (e.g., another device that is the same as or similar to device 300). As used herein, the term “module” refers to at least one instruction stored in memory 306 and / or the memory of another device, which when executed by processor 304 and / or the processor of another device (e.g., another device that is the same as or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes described herein. In some embodiments, a module is implemented in software, firmware, and / or hardware, etc.

[0067] Provided Figure 3 The number and arrangement of the illustrated components are provided as an example. In some embodiments, compared to Figure 3 the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another set of components of device 300.

[0068] Now refer to Figure 4A, an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack") is illustrated. As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a localization system 406 (sometimes referred to as a localization module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of the vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in one or more independent systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 are included in one or more independent systems located in the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), and / or a field-programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., an autonomous vehicle system that is the same as or similar to the remote AV system 114, a queue management system that is the same as or similar to the queue management system 116, and / or a V2I system that is the same as or similar to the V2I system 118, etc.).

[0069] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect the at least one physical object), and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a), the image being associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies the at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of the physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0070] In some embodiments, the planning system 404 receives data associated with a destination, and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the perception system 402 (e.g., the data associated with the classification of the physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In some embodiments, the planning system 404 receives data associated with an updated position of a vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0071] In some embodiments, the positioning system 406 receives data associated with (e.g., representing) the location of a vehicle (e.g., vehicle 102) in an area. In some examples, the positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In certain examples, the positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the positioning system 406 generates a combined point cloud based on the respective point clouds. In these examples, the positioning system 406 compares the at least one point cloud or the combined point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in the database 410. Then, based on the positioning system 406 comparing the at least one point cloud or the combined point cloud with the map, the positioning system 406 determines the position of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated prior to the navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the roadway geometry, a map describing the connection properties of the road network, a map describing the physical properties of the roadway (such as traffic rate, traffic flow, the number of vehicle and bicycle traffic lanes, lane width, lane traffic direction, or the type and location of lane markings, or a combination thereof, etc.), and a map describing the spatial location of road features (such as crosswalks, traffic signs, or various other driving signals, etc.). In some embodiments, the map is generated in real time based on the data received by the perception system.

[0072] In another example, the positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, the positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and the positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, the positioning system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, the positioning system 406 generates data associated with the position of the vehicle. In some examples, based on the positioning system 406 determining the position of the vehicle, the positioning system 406 generates data associated with the position of the vehicle. In such examples, the data associated with the position of the vehicle includes data associated with one or more semantic properties corresponding to the position of the vehicle.

[0073] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to cause the powertrain control system (e.g., the DBW system 202h and / or the powertrain control system 204, etc.), the steering control system (e.g., the steering control system 206), and / or the braking system (e.g., the braking system 208) to operate. In an example, in the case where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.) to change states.

[0074] In some embodiments, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model individually or in combination with one or more of the above systems. In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.). The following are examples of Figures 4B to 4D the implementation of machine learning models.

[0075] The database 410 stores data transmitted to, received from, and / or updated by the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408. In some examples, the database 410 includes a storage component for storing operation-related data and / or software and using the autonomous vehicle computing 400 of at least one system (e.g., related to Figure 3the same or similar storage components as the storage component 308). In some embodiments, the database 410 stores data associated with 2D and / or 3D maps of at least one region. In some examples, the database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., nations), etc. In such examples, a vehicle (e.g., a vehicle the same or similar to the vehicle 102 and / or the vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, backroads, and / or off-road paths, etc.), and cause at least one LiDAR sensor (e.g., a LiDAR sensor the same or similar to the LiDAR sensor 202b) to generate data associated with an image representing the objects included in the field of view of the at least one LiDAR sensor.

[0076] In some embodiments, the database 410 can be implemented across multiple devices. In some examples, the database 410 is included in a vehicle (e.g., a vehicle the same or similar to the vehicle 102 and / or the vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system the same or similar to the remote AV system 114), a queue management system (e.g., a queue management system the same or similar to Figure 1 the queue management system 116), and / or a V2I system (e.g., a V2I system the same or similar to Figure 1 the V2I system 118), etc.

[0077] Now refer to Figure 4B FIG., a diagram illustrating the implementation of a machine learning model. More specifically, a diagram illustrating the implementation of a convolutional neural network (CNN) 420. For illustrative purposes, the following description of the CNN 420 will be with respect to implementing the CNN 420 via the perception system 402. However, it will be understood that in some examples, the CNN 420 (e.g., one or more components of the CNN 420) is implemented by other systems different from or in addition to the perception system 402, such as the planning system 404, the positioning system 406, and / or the control system 408, etc. Although the CNN 420 includes certain features as described herein, these features are provided for illustrative purposes and are not intended to limit the present disclosure.

[0078] The CNN 420 includes a plurality of convolutional layers including a first convolutional layer 422, a second convolutional layer 424, and a convolutional layer 426. In some embodiments, the CNN 420 includes a subsampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, the subsampling layer 428 and / or other subsampling layers have dimensions smaller than the dimensions of the upstream system (i.e., the amount of nodes). By means of the subsampling layer 428 having dimensions smaller than the dimensions of the upstream layer, the CNN 420 combines the amount of data associated with the initial input and / or output of the upstream layer, thereby reducing the amount of computation required for the CNN 420 to perform downstream convolutional operations. Additionally or alternatively, by means of the subsampling layer 428 being associated with at least one subsampling function (e.g., being configured to perform at least one subsampling function) (as described below with respect to Figure 4C and Figure 4D ), the CNN 420 combines the amount of data associated with the initial input.

[0079] Based on the perception system 402 providing corresponding inputs and / or outputs associated with the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426 respectively to generate corresponding outputs, the perception system 402 performs convolutional operations. In some examples, based on the perception system 402 providing data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426, the perception system 402 implements the CNN 420. In such examples, based on the perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle the same or similar to the vehicle 102, a remote AV system the same or similar to the remote AV system 114, a queue management system the same or similar to the queue management system 116, and / or a V2I system the same or similar to the V2I system 118, etc.), the perception system 402 provides the data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426. The following is a detailed description of Figure 4C including convolutional operations.

[0080] In some embodiments, the perception system 402 provides data associated with an input (referred to as an initial input) to the first convolutional layer 422, and the perception system 402 uses the first convolutional layer 422 to generate data associated with an output. In some embodiments, the perception system 402 provides the output generated by the convolutional layer as an input to a different convolutional layer. For example, the perception system 402 provides the output of the first convolutional layer 422 as an input to the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426. In such an example, the first convolutional layer 422 is referred to as an upstream layer, and the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426 are referred to as downstream layers. Similarly, in some embodiments, the perception system 402 provides the output of the subsampling layer 428 to the second convolutional layer 424 and / or the convolutional layer 426, and in this example, the subsampling layer 428 will be referred to as an upstream layer, and the second convolutional layer 424 and / or the convolutional layer 426 will be referred to as downstream layers.

[0081] In some embodiments, before the perception system 402 provides an input to the CNN 420, the perception system 402 processes the data associated with the input provided to the CNN 420. For example, based on the perception system 402 normalizing sensor data (such as image data, LiDAR data, and / or Radar data, etc.), the perception system 402 processes the data associated with the input provided to the CNN 420.

[0082] In some embodiments, based on the perception system 402 performing convolutional operations associated with each convolutional layer, the CNN 420 generates an output. In some examples, based on the perception system 402 performing convolutional operations associated with each convolutional layer and the initial input, the CNN 420 generates an output. In some embodiments, the perception system 402 generates an output and provides the output to the fully connected layer 430. In some examples, the perception system 402 provides the output of the convolutional layer 426 to the fully connected layer 430, where the fully connected layer 430 includes data associated with a plurality of eigenvalues referred to as F1, F2,..., FN. In this example, the output of the convolutional layer 426 includes data associated with a plurality of output eigenvalues representing predictions.

[0083] In some embodiments, based on the perception system 402 identifying an eigenvalue associated with the highest likelihood as the correct prediction among a plurality of predictions, the perception system 402 identifies the prediction from among the plurality of predictions. For example, in the case where the fully connected layer 430 includes eigenvalues F1, F2, ..., FN and F1 is the largest eigenvalue, the perception system 402 identifies the prediction associated with F1 as the correct prediction among the plurality of predictions. In some embodiments, the perception system 402 trains the CNN 420 to generate predictions. In some examples, the perception system 402 trains the CNN 420 to generate predictions based on the perception system 402 providing training data associated with the prediction to the CNN 420.

[0084] Now refer to Figure 4C and Figure 4D , a diagram illustrating an example operation of the CNN 440 that utilizes the perception system 402. In some embodiments, the CNN 440 (e.g., one or more components of the CNN 440) is the same as or similar to the CNN 420 (e.g., one or more components of the CNN 420) (see Figure 4B ).

[0085] In step 450, the perception system 402 provides data associated with an image as an input to the CNN 440 (step 450). For example, as illustrated, the perception system 402 provides data associated with an image to the CNN 440, where the image is a grayscale image represented as values stored in a two-dimensional (2D) array. In some embodiments, the data associated with the image may include data associated with a color image, which is represented as values stored in a three-dimensional (3D) array. Additionally or alternatively, the data associated with the image may include data associated with an infrared image and / or a Radar image, etc.

[0086] In step 455, the CNN 440 performs a first convolution function. For example, based on the CNN 440 providing the values representing the image as inputs to one or more neurons (not explicitly illustrated) included in the first convolutional layer 442, the CNN 440 performs the first convolution function. In this example, the values representing the image may correspond to the values of a region (sometimes referred to as a receptive field) representing the image. In some embodiments, each neuron is associated with a filter (not explicitly illustrated). The filter (sometimes referred to as a kernel) can be represented as an array of values corresponding in size to the values provided as inputs to the neuron. In one example, the filter can be configured to identify edges (e.g., horizontal lines, vertical lines, and / or straight lines, etc.). In successive convolutional layers, the filters associated with the neurons can be configured to successively identify more complex patterns (e.g., arcs and / or objects, etc.).

[0087] In some embodiments, based on the CNN 440, the values provided as input to each neuron among one or more neurons included in the first convolutional layer 442 are multiplied by the values of the filters corresponding to each neuron among the same one or more neurons, and the CNN 440 performs a first convolutional function. For example, the CNN 440 may multiply the values provided as input to each neuron among one or more neurons included in the first convolutional layer 442 by the values of the filters corresponding to each neuron among the same one or more neurons to generate a single value or an array of values as output. In some embodiments, the collective output of the neurons of the first convolutional layer 442 is referred to as the convolutional output. In some embodiments, when each neuron has the same filter, the convolutional output is referred to as a feature map.

[0088] In some embodiments, the CNN 440 provides the output of each neuron of the first convolutional layer 442 to the neurons of the downstream layer. For clarity, an upstream layer may be a layer that transmits data to a different layer (referred to as the downstream layer). For example, the CNN 440 may provide the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the subsampling layer. In an example, the CNN 440 provides the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the first subsampling layer 444. In some embodiments, the CNN 440 adds a bias value to the aggregated set of all values provided to each neuron of the downstream layer. For example, the CNN 440 adds a bias value to the aggregated set of all values provided to each neuron of the first subsampling layer 444. In such an example, the CNN 440 determines the final value to be provided to each neuron of the first subsampling layer 444 based on the aggregated set of all values provided to each neuron and the activation function associated with each neuron of the first subsampling layer 444.

[0089] In step 460, the CNN 440 performs a first subsampling function. For example, based on the CNN 440 providing the values output by the first convolutional layer 442 to the corresponding neurons of the first subsampling layer 444, the CNN 440 may perform a first subsampling function. In some embodiments, the CNN 440 performs the first subsampling function based on an aggregation function. In an example, based on the CNN 440 determining the maximum input among the values provided to a given neuron (referred to as the max pooling function), the CNN 440 performs the first subsampling function. In another example, based on the CNN 440 determining the average input among the values provided to a given neuron (referred to as the average pooling function), the CNN 440 performs the first subsampling function. In some embodiments, based on the CNN 440 providing values to each neuron of the first subsampling layer 444, the CNN 440 generates an output, which is sometimes referred to as the subsampled convolutional output.

[0090] At step 465, CNN 440 performs a second convolution function. In some embodiments, CNN 440 performs the second convolution function in a manner similar to how CNN 440 performs the first convolution function as described above. In some embodiments, based on the values output by the first subsampling layer 444 being provided as inputs to one or more neurons (not explicitly illustrated) included in the second convolutional layer 446, CNN 440 performs the second convolution function. In some embodiments, as described above, each neuron of the second convolutional layer 446 is associated with a filter. As described above, the (one or more) filters associated with the second convolutional layer 446 may be configured to identify more complex patterns compared to the filters associated with the first convolutional layer 442.

[0091] In some embodiments, based on CNN 440 multiplying the values provided as inputs to each of the one or more neurons included in the second convolutional layer 446 by the values of the filters corresponding to each of the one or more neurons, CNN 440 performs the second convolution function. For example, CNN 440 may multiply the values provided as inputs to each of the one or more neurons included in the second convolutional layer 446 by the values of the filters corresponding to each of the one or more neurons to generate a single value or an array of values as output.

[0092] In some embodiments, CNN 440 provides the output of each neuron of the second convolutional layer 446 to the neurons of a downstream layer. For example, CNN 440 may provide the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the subsampling layer. In an example, CNN 440 provides the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the second subsampling layer 448. In some embodiments, CNN 440 adds a bias value to the aggregated set of all values provided to each neuron of the downstream layer. For example, CNN 440 adds a bias value to the aggregated set of all values provided to each neuron of the second subsampling layer 448. In such an example, CNN 440 determines the final value provided to each neuron of the second subsampling layer 448 based on the aggregated set of all values provided to each neuron and the activation function associated with each neuron of the second subsampling layer 448.

[0093] At step 470, CNN 440 performs a second subsampling function. For example, based on the values output by the second convolutional layer 446 being provided to the respective neurons of the second subsampling layer 448 by CNN 440, CNN 440 may perform the second subsampling function. In some embodiments, based on CNN 440 using an aggregation function, CNN 440 performs the second subsampling function. In an example, as described above, based on CNN 440 determining the maximum input or average input among the values provided to a given neuron, CNN 440 performs the first subsampling function. In some embodiments, based on CNN 440 providing values to the respective neurons of the second subsampling layer 448, CNN 440 generates an output.

[0094] At step 475, CNN 440 provides the output of each neuron of the second subsampling layer 448 to the fully connected layer 449. For example, CNN 440 provides the output of each neuron of the second subsampling layer 448 to the fully connected layer 449 such that the fully connected layer 449 generates an output. In some embodiments, the fully connected layer 449 is configured to generate an output associated with a prediction (sometimes referred to as classification). The prediction may include an indication that an object included in the image provided as input to CNN 440 includes an object and / or a collection of objects, etc. In some embodiments, the perception system 402 performs one or more operations and / or provides data associated with the prediction to different systems described herein.

[0095] Generate bounding boxes for navigation

[0096] As described herein, to improve the functionality of an autonomous vehicle and its ability to generate bounding boxes in real time and navigate the environment, the autonomous vehicle may be configured to use feature maps and object queries to combine or associate features within an input image. In some cases, multiple feature maps are cross-correlated with each other and with the object query. The object query may be a scene-independent object query, which is general and not based on image data associated with a previous time step of the scene of the autonomous vehicle. The scene-independent object query may be enriched and used to generate a scene-dependent object query, which may be used in subsequent time steps.

[0097] By generating scene-dependent object queries, the autonomous vehicle can use fewer computational resources to generate bounding boxes, which can increase the speed of object detection processing and bounding box generation.

[0098] In addition, the scene-dependent object query can improve the autonomous vehicle's ability to identify objects within the scene of the autonomous vehicle. For example, the scene-dependent object query can improve the autonomous vehicle's ability to determine the bounding boxes for objects in the vehicle's scene.

[0099] The autonomous vehicle can also generate object queries and enrich the object queries using the feature map and / or features from other object queries to generate scene-dependent object queries. The scene-dependent object queries can improve the ability of the autonomous vehicle to identify objects within the scene of the autonomous vehicle.

[0100] Figure 5 FIG. is a block diagram of an example perception environment 500 in which the perception system 402 receives and processes an image 502 to provide one or more (3D) bounding boxes 520 for objects in a vehicle scene (corresponding to the image 502). In the example shown, the perception system 402 includes an image feature extractor 504, an object query formulation stage 505, a decoder stage 506, and a detection stage 510. However, it will be understood that the perception system may include fewer or more components. In some cases, the perception system 402 may omit the object query formulation stage 505. For example, in some cases, the decoder stage 506 may output one or more object queries and may have predefined queries and may not formulate new object queries. The various components of the perception system 402 described herein may be implemented using one or more processors and / or as one or more layers or stages of a machine learning model or neural network.

[0101] An image 502 for a particular scene (also referred to herein as an image set 502) may include image data from one or more sensors in the sensor suite. The image 502 may include different types of images corresponding to the sensors or devices used to generate these images. For example, the image 502 may be a camera image generated from one or more cameras (such as camera 202a, etc.), or a LiDAR image generated from one or more LiDAR sensors (such as LiDAR sensor 202b, etc.). Other image types may be used, such as Radar images generated from one or more Radar sensors (e.g., generated from Radar sensor 202c). Each image may correspond to a different image sensor (or camera) placed at different locations around the autonomous vehicle. In some cases, a combination of images may form a 360-degree view of the scene of the autonomous vehicle from the perspective of the autonomous vehicle. Thus, each image in the image 502 may be a neighbor or border of another image in the image 502, and some objects (or parts of objects) may appear in different images in the image 502.

[0102] In addition, the image 502 in the image collection can be generated at approximately the same time and can form part of different image streams. In this way, the image 502 can represent the scene of the vehicle at a specific time or time step. When the perception system 402 uses the image to generate the bounding box 520 and navigate the vehicle, it will be understood that the perception system 402 can process the image 502 in real time or near real time to generate the bounding box 520.

[0103] Multi-view level

[0104] The image feature extractor 504 can be implemented using one or more neural networks or layers of a neural network to extract features from the image 502. In some cases, the image feature extractor 504 can be implemented using a backbone with a Feature Pyramid Network (FPN), Residual Network (Resnet), or Swin Transformer, CSWin Transformer, Vision Transformer (ViT), etc. The image feature extractor 504 can use the image 502 to generate one or more feature maps. In some cases, the image feature extractor 504 generates at least one feature map for each image 502. For example, if the image feature extractor 504 receives six images corresponding to six cameras placed at different positions around the vehicle and oriented in different ways (e.g., to obtain a 360-degree view of the area around the vehicle), the image feature extractor 504 can generate six feature maps respectively.

[0105] The feature maps can have the same or different shapes as the images used to generate them, and / or can have the same or different shapes from each other. For example, if each image 502 has a shape [900, 1600, 3], the corresponding feature map can have a shape [45, 80, 256]. However, it will be understood that the feature maps can have different shapes from each other.

[0106] Each of the generated feature maps can be divided into an array of grid cells with a specific channel depth. The grid cells can include semantic data (or features) extracted from the pixels in the image (or images) used to generate the feature map. The features of the grid cells can be organized as vectors or some other tensor shape. For example, the features (or semantic data) of the grid cells can indicate the shape, light, texture, reflectivity, edges, object class, location, etc. of something detected by the image feature extractor 504.

[0107] The multi-view stage 503 can enrich the feature maps by comparing and / or correlating features from different feature maps. In some cases, the multi-view stage 503 uses the features of the grid cells in a group of grid cells to update each other (also referred to as self-attention herein). For example, the multi-view stage 503 can use the features of a group of grid cells in one or more than one feature map to enrich or modify the features of a specific grid cell in the group of grid cells.

[0108] In some cases, the multi-view stage 503 can group the grid cells based on objects (e.g., group the grid cells corresponding to (or seemingly corresponding to) the same object or the contour of the same object). In some cases, the multi-view stage 503 can group the grid cells by dividing the feature map into multiple regions (also referred to as windows herein) and / or assigning different grid cells of the feature map to different regions or windows. In some cases, different regions or windows of the feature map can be mutually exclusive (e.g., a grid cell can be assigned to only one region or window). In some cases, the multi-view stage 503 can divide the feature map into multiple rows or multiple columns of regions or windows. Some or all of the regions or windows can have the same (or different) sizes (e.g., width and height), and one or more than one of the regions can overlap with multiple feature maps corresponding to different images. The window rows can be aligned or offset from each other.

[0109] As described herein, instead of the entire feature map or a set of feature maps, by using a group of grid cells (e.g., a window or a subset of the image / feature map) for comparison and enrichment (e.g., performing self-attention), the decoder stage 506 can reduce the processing requirements and increase the speed and efficiency of the decoder stage 506 in processing the feature maps. These efficiencies can increase the rate at which the perception system 402 can accurately identify the objects in the image 502.

[0110] The multi-view stage 503 can compare the semantic data of groups of grid cells (e.g., different grid cells within a specific window or region) with each other. Based on this comparison, the multi-view stage 503 can modify the semantic data of different grid cells. For example, the multi-view stage 503 can compare certain features of a grid cell (e.g., color, reflectivity, shape, etc.) with the corresponding features of different grid cells in the same group (e.g., compare the features of the grid cells within window 552a with the corresponding features of different grid cells within window 552a). Based on the similarity, the multi-view stage 503 can determine the probability relationships between the grid cells in the group (e.g., the probability that a grid cell is part of the same object (such as a vehicle, bicycle, pedestrian, construction cone, etc.)). Based on this determination, the multi-view stage 503 can update one or more features of the grid cell. For example, one grid cell can be updated to indicate that it is the middle part of an object, and another grid cell can be updated to indicate that it is the start of the same object, and so on.

[0111] In some cases, the multi-view stage 503 can cross-correlate some or all of the features of various grid cells within a group (e.g., within a specific region) with each other. In this way, the multi-view stage 503 can enrich some or all of the grid cells within a specific group. Additionally, the multi-view stage 503 can repeat the comparison for each group within a group of feature maps (e.g., a window) and / or across some or all of the feature maps, such that some or all of the grid cells of the feature maps are compared / updated based on the comparison with the features of other grid cells from the same group (e.g., a window or region).

[0112] As described above, referring to the self-attention of object queries, in some cases, the multi-view stage 503 can generate a matrix including some or all of the grid cells within a group. Then, the multi-view stage 503 can determine the weight or probability relationships between the grid cells and include the weights in the matrix. The multi-view stage 503 can use the weights / relationships in the matrix (indicating the relationships or weights between the grid cells) to calculate the updated values of the features of different grid cells. For example, the multi-view stage 503 can update a specific value of a specific grid cell using the corresponding weighted values of some or all of the other grid cells in the group. Examples of such matrices and calculations (but for object queries) are described herein with reference to the self-attention of object queries. Additionally, this process can be repeated across some or all of the grid cell groups within a feature map and across some or all of the feature maps. For example, the image feature extractor 504 can generate multiple feature maps for each image, where each feature map corresponds to one or more detected characteristics of the image. In some such cases, a window (or other form of grouping) can be applied to some or all of the updated feature maps and the grid cells of the feature maps as described herein.

[0113] In some cases, the decoder stage 506 may include multiple layers of the multi-view stage 503. In some such cases, the groups (e.g., windows) in different layers of the multi-view stage 503 may be different. In some cases, the size and / or positioning of the windows may be different.

[0114] Object query formulation stage

[0115] The object query formulation stage 505 (also referred to herein as the formulation stage 505) can be used to initialize, seed / modify, and / or enrich object queries. Thus, it will be understood that the formulation stage 505 may include one or more than one sub-stage, including but not limited to an initialization stage, a seed / modification stage, and / or an enrichment stage.

[0116] In some cases, the formulation stage 505 may initiate a specific number of object queries. In certain cases, the formulation stage 505 initiates more object queries than the number of objects expected to be found in the (one or more) images 502. For example, if the formulation stage 505 expects no more than 400 objects to be present in the scene of the image 502, the formulation stage 505 may initiate a number of object queries greater than 400, such as 900 object queries, etc.

[0117] Object queries can be organized as vectors or some other tensor shape and / or can include the same or different numbers of features. For example, object queries can include 256-dimensional features, or fewer or more dimensional features. The features can represent one or more characteristics of an object individually or in combination, such as but not limited to the class of the object, movement, relationship with other objects (whether foreground or background), location, shape, size, color, texture, reflectivity, etc. In some cases, the formulation stage 505 may initiate the features of the object queries randomly and / or pseudo-randomly. For example, the values of the features for the object queries can include random numbers or pseudo-random numbers.

[0118] Additionally, the formulation stage 505 can seed or modify the initial (random or pseudo-random) values of the features for the object queries. In some cases, the formulation stage 505 may include a positioning network (or receive values from a positioning network) that determines (or helps determine) the possible locations of individual object queries within the vehicle scene. In certain cases, the formulation stage 505 can use other data to seed the object queries (or modify the initial values of the object queries). For example, the formulation stage 505 can use a heat map indicating the expected or possible movement or trajectory of the objects within the vehicle scene to formulate the object queries.

[0119] In some cases, the formulation level 505 may include an object query self-attention level (e.g., similar to the object query self-attention level 512 described herein), which enables an object query to self-attention and update itself. For example, the self-attention level of the formulation level 505 may modify the values of the object query group based on the features of the object queries in the object query group. As described herein at least with reference to the object query self-attention level 512, the object query self-attention level of the formulation level 505 may compare the features of a particular object query with the features of some or all of the other object queries (or some or all of the object features in the object feature group) to determine the correlation or similarity between the particular object query and the other object queries, weight the features of the various object queries using the correlation between the particular object query and the other object queries, and use the weighted features to calculate new (or modified) values for the corresponding features of the particular object query. In some cases, the object query self-attention level 512 may update the features of some or all of the object queries in this way. In certain cases, the object query self-attention level 512 may determine a matrix to indicate the relationship (or weight) between the features of the various object queries, and use the matrix to update the features of some or all of the object queries in the object query. The decoder level 506 can be used to enrich the feature map and the object query. In some cases, the decoder level 506 may use self-attention and / or cross-attention techniques to enrich the feature map and the object query.

[0120] In the example shown, the decoder level 506 includes one or more layers of the object query self-attention level 512 and the object query cross-attention level 514.

[0121] Decoder level

[0122] The different layers of the decoder level 506 may include similar components. In Figure 6A and Figure 6B the example shown, the layers of the decoder level 506 include the object query self-attention level 512, the object query cross-attention level 514, and the feed-forward network (FFN) level 516. However, it will be understood that the decoder level 506 and / or the different layers of the decoder level 506 may include different components. For example, the decoder level 506 and / or the different layers of the decoder level 506 may include different components or different relationships between components. In some cases, the individual layers of the decoder level 506 include the same components in the same relationship. In certain cases, the layers of the decoder level 506, the components of the different layers, may be configured differently or use different parameters. For example, the object query self-attention level 512 of the first layer may process the object query using different parameters or configurations than the object query self-attention level 512 of the second layer. Similarly, for the object query cross-attention level 514 and / or the FFN level 516, etc., different parameters may be used in different layers.

[0123] The components of the layers of decoder stage 506 can process data in parallel or sequentially. In some cases, the output of a stage within a layer can be used as the input to another stage within that layer. For example, in the Figure 6A and Figure 6B example shown, the object query cross-attention stage 514 processes the data output by the object query self-attention stage 512, and the FFN stage 516 processes the data output by the object query cross-attention stage 514. However, it will be understood that the components of decoder stage 506 can be aligned in various configurations.

[0124] The output of one layer of decoder stage 506 can be used as the input to a subsequent layer, and the output of the last layer of decoder stage 506 can be provided to detection stage 510. For example, in an N-layer decoder stage 506, the output of the first layer can be used as the input to the second layer, and so on until the output of the N-1th layer is used as the input to the Nth layer. In some such cases, the output of the Nth layer can be used as the input to detection stage 510.

[0125] Object query self-attention stage

[0126] The object query self-attention stage 512 can be configured to generate and / or enrich object queries using features from an object query group (e.g., using self-attention).

[0127] As described herein, there can be many object queries, and some or all of the object queries can be modified by the object query cross-attention stage 514 (e.g., using grid cells from a feature map). In some cases, the object query self-attention stage 512 can modify the features of the object queries by comparing the features of the object queries with each other, determining weighted values based on the comparison, and using the weighted features (weighted based on the determined weighted values). For example, the object query self-attention stage 512 can compare the semantic data (or features) of the object queries to determine relationships between the object queries, such as the likelihood that different object queries correspond to the same object or different objects. In some cases, this can include comparing features of the object queries corresponding to the class of the object, movement, relationship with other objects, being foreground or background, color, light reflectivity, edges, texture, shape, etc. Based on the comparison, the object query self-attention stage 512 can update the object queries. In some cases, this can include modifying one or more values of a tensor corresponding to the object queries.

[0128] In some cases, the object query self-attention level 512 compares the features of a specific object query with the features of some or all of the other object queries (or some or all of the object features in the object feature group) to determine the correlation or similarity between the specific object query and the other object queries. In some cases, the correlation or similarity can be expressed as a probability or a weight. Using the correlation between the specific object query and the other object queries, the features of the object queries (including the specific object query) can be weighted, and the weighted features can be used to calculate new (or modified) values for the corresponding features of the specific object query. For example, the first feature of some or all of the object queries in the object query can be weighted (relative to the specific object query), and the weighted value is used to determine the value of the first feature for the specific object query. Similarly, other features of the specific object query can be updated (e.g., using the same or different weighting). In some cases, the object query self-attention level 512 can update the features of some or all of the object queries in this way. In certain cases, the object query self-attention level 512 can determine a matrix to indicate the relationship (or weight) between the features of various object queries, and use the matrix to update the features of some or all of the object queries.

[0129] As a non-limiting example, consider the following three object queries and the values of their features: Object Query 1[1,4] = (.2,.2,.4,.7); Object Query 2[1,4] = (.3,.4,.6,.7); Object Query 3[1,4] = (.1,.8,.9,.7).

[0130] After analyzing the features of the three object queries, assume that the object query self-attention level 512 generates the following relationship (or weighting) matrix between these object queries:

[0131] Object Query 1 Object Query 2 Object Query 3 Object Query 1 .7 .2 .1 Object Query 2 .2 .6 .2 Object Query 3 .1 .2 .7

[0132] Based on the determined relationship or weighting, the object query self-attention level 512 can update the values of the features for the object queries as follows:

[0133] Object Query 1[1,4] = (.21,.3,.49,.7) or (.7*.2 +.2*.3 +.1*.1,.7*.2 +.2*.4 +.1*.8,.7*.4 +.2*.6 +.1*.9,.7*.7 +.7*.2 +.7*.1)

[0134] Object Query 2[1,4] = (.24,.44,.62,.7) or (.2*.2 +.6*.3 +.2*.1,.2*.2 +.6*.4 +.2*.8,.2*.4 +.6*.6 +.2*.9,.2*.7 +.6*.7 +.2*.7

[0135] Object query 3[1,4] = (.15,.66,.79,.7) or (.1*.2+.2*.3+.7*.1,.1*.2+.2*.4+.7*.8,.1*.4+.2*.6+.7*.9,.1*.7+.7*.7

[0136] Object query cross-attention level

[0137] The object query cross-attention level 514 can be configured to enrich the object query (set) (e.g., using cross-attention and / or using data from another level such as the object query self-attention level 512 and / or the multi-view level 503, etc.).

[0138] In some cases, the object query cross-attention level 514 can enrich the object query based on data received from the object query self-attention level 512 and / or the multi-view level 503. For example, the object query cross-attention level 514 can use semantic data corresponding to one or more feature maps output by the multi-view level 503 to modify or edit the object query. In some cases, this can include modifying one or more features of the tensor corresponding to the object query.

[0139] In some cases, the object query cross-attention level 514 can associate data from one or more feature maps enriched by the multi-view level 503. As part of correlating the object query with one or more feature maps, the object query cross-attention level 514 can use one or more linear layers to identify one or more features in one or more feature maps. For example, the object query cross-attention level 514 can multiply the tensor [1,N] corresponding to the object query by a learnable linear layer matrix [N,2] to determine the position (x,y) in the (enriched) feature map corresponding to the feature to be correlated with the object query.

[0140] The object query cross-attention level 514 can use the features of the identified feature maps to modify some or all of the features of the object query. In some cases, this can include assigning weights to specific features of the object query and to the corresponding features of the identified feature maps, and using the result (non-limiting example: sum of products) to modify or assign new values to specific features of the object query. In certain cases, the object query cross-attention level 514 can use a learnable linear layer matrix to identify multiple features of one or more feature maps, and use the identified features to modify the features of the object query. In some such cases, the object query cross-attention level 514 can assign different weights to the corresponding features of different feature maps, and use the weighted features to determine the corresponding features of the object query.

[0141] Detection level

[0142] The detection stage 510 uses the output of the cross-attention stage 514 to determine the bounding box for the object query, and a detector such as a center point detector (an example of a center point detector is described in "Center-based 3D Object Detection and Tracking" by Yin et al., arXiv:2006.11275v2, January 6, 2021) can be used to implement the detection stage 510.

[0143] Fewer, more, or different components can be used as part of the perception system 402. For example, in some cases, the perception system 402 can omit one or more layers of the decoder stage. The object queries output by the decoder stage 506 can be communicated to the detection stage 510 to detect the bounding box 520. As another example, in certain cases, the multi-view stage 503 can be omitted or combined. For example, the feature maps generated by the image feature extractor 504 can be enriched in one or more layers during the multi-view stage 503.

[0144] Figure 6A FIG. 600 is a data flow diagram of an example perception environment that illustrates the perception system 402 generating the bounding box 520 from the image 502 at the initialization stage at an initial time step (t = 0). Figure 6B Illustrates the operation stage where the perception system operates at subsequent time steps (t = 1, 2, 3, etc.) after the initialization stage. In the example shown, two general data paths are shown: the object query data path and the feature map data path. Although there is data crossover between the two general data paths, for simplicity, the feature map data path (including the image 502, the image feature extractor 504, the feature map 602, and the multi-view stage 503) will be described first, and then the object query data path (including the formulation stage 505, the object query 604, the positional encoding 606, the object query cross-attention stage 514, the object query self-attention stage 512, and the FFN stage 516) will be described.

[0145] As described herein, the image 502 can correspond to images received from different cameras (or other image sensors) around the autonomous vehicle. In the example shown, six images are shown; however, it will be understood that fewer or more images can be used. The image 502 can correspond to images obtained at the same (or approximately the same) time (e.g., within milliseconds of each other). In this way, the images can correspond to the same scene of the vehicle. Additionally, the perception system 402 can repeatedly receive images and perform the functions described herein multiple times per second when a new image is received. Thus, it will be understood that the perception system 402 can operate in real time or near real time to generate the bounding box 520 from the image 502.

[0146] As described herein, the image feature extractor 504 generates a feature map 602 from the image 502. In the example shown, the image feature extractor 504 generates one feature map from each of the images in the image 502. However, it will be understood that the image feature extractor 504 may generate multiple feature maps 602 from each of the images in the image 502 and communicate the multiple feature maps 602 to the decoder stage 506.

[0147] Each of the feature maps in the feature map 602 can be divided into an array of grid cells having a particular channel depth. The grid cells can include semantic data (or features) extracted from the pixels in the (one or more) images from which the feature map is generated. The features can be organized as vectors or some other tensor shape. For example, the features (or semantic data) of a grid cell can indicate the shape, light, texture, reflectivity, edges, object class, location, etc. of something detected by the image feature extractor 504.

[0148] The multi-view stage 503 can enrich the feature map. In some cases, the multi-view stage 503 can enrich the feature map by modifying the features in a grid cell with features from another grid cell (and vice versa). As described herein, the multi-view stage 503 can cross-correlate the features in the grid cells within different groups (e.g., windows) of the feature map 602.

[0149] In addition, one or more of the groups of grid cells can cause the multiple feature maps to overlap such that the grid cells in one feature map are cross-correlated with the grid cells from another feature map. In some such cases, the multi-view stage 503 can use windows that cause the feature maps 602 corresponding to the images from cameras adjacent to each other to overlap (e.g., one window can cause the feature map corresponding to the front view image of a vehicle and the feature map corresponding to the left front view of the vehicle to overlap). Thus, the features from the grid cells in one feature map can be propagated to (or correlated with) the features from the grid cells in another (e.g., neighboring, adjacent, or contiguous) feature map.

[0150] Additionally, a group of grid cells (e.g., a window) for self-attention grid cells in one layer can be different from that in another layer. For example, the group of grid cells used by the multi-view stage 503 in the first layer can be different from that used in the second layer. In some cases, the window used by the multi-view stage 503 in the second layer can be shifted in one or more directions relative to the window used by the multi-view stage 503 in the first layer. Subsequent layers can include additional shifts or oscillations between the placement of the window in the first layer and the placement of the window in the second layer. Additionally, depending on how the windows are aligned, a portion of a window can include opposite ends or corners of a feature map. For example, a window can include the lower right corner of one feature map and the upper left corner of another (e.g., adjacent or neighboring) feature map.

[0151] Referring to the object query data path, the formulation stage 505 generates the object query 604. As described herein, the formulation stage 505 can generate and / or initialize the object query 604 simultaneously with the image feature extractor 504 generating the feature map 602. As described herein, when generating the object query 604, the formulation stage 505 can initialize the features of the object query randomly or pseudo-randomly.

[0152] The formulation stage 505 can also use features from the feature map 602, other feature maps (e.g., from the localization network), or other data (e.g., heatmap data associated with a heatmap) to generate the object query 604. As described herein, in some cases, this can include using a linear layer matrix to identify grid cells and cross-attention the features of the identified grid cells with the features of the object query using the weighted features of the grid cells. Additionally, in some cases, the formulation stage 505 can include a self-attention stage to enable the grid cells to cross-correlate or associate features and update themselves.

[0153] The position encoding 606 can provide context data associated with the position of the input vector within the dataset. The position encoding can be used to identify the position of a grid cell within the feature map. The position encoding can be added to the corresponding input vector. For example, the position encoding can have the same dimension as the query. The position encoding can depend on the position of the vector, the index within the vector, and the dimension of the input.

[0154] The formulation stage 505 communicates the object query 604 to the decoder stage 506 (e.g., the object query of the decoder stage 506 from the self-attention stage 512) for further processing. The object query self-attention stage 512 can enrich the object query 604. For example, the object query self-attention stage 512 can compare the features of the object query 604 to determine the probabilistic relationships between them, generate weighted values based on the probabilistic relationships, and modify the features of one object query based on the weighted features (weighted by the weighted values) from other object queries. In some cases, the object query self-attention stage 512 can use similar techniques to enrich some or all of the object queries 604 to provide the enriched object query 614 such that the semantic data from some or all of the object queries in the object query 604 is updated or enriched.

[0155] As described herein, the object query cross-attention stage 514 can use the feature map 602 to enrich the object query 604. In some cases, the object query cross-attention stage 514 enriches the object query 604 in a manner similar to the way the formulation stage 505 uses the feature map 602 to generate the object query 604. For example, the object query cross-attention stage 514 can identify the (one or more) grid cells in the feature map 602 corresponding to a particular object query in the object query 604, weight the features, and / or use the (weighted) features of the identified (one or more) grid cells to modify or enrich the features of the particular object query. In some cases, the object query cross-attention stage 514 can use similar techniques to enrich some or all of the object queries 604.

[0156] After the object query cross-attention stage 514, the FFN stage 516 can transform the object query into a defined dimensional output. The FFN stage can converge the output of the cross-attention stage 514 to a 256-value vector. The FFN stage 516 can converge the vector to a lower spatial embedding. The output of the FFN stage 516 can be provided to the next layer of the decoder for processing until the total number of layers is completed.

[0157] Following each stage (such as the self-attention stage 512, the cross-attention stage 514, and the FFN stage 516) can be layer normalization processing. The normalization processing can be normalization processing used in the art, such as batch normalization, weight normalization, layer normalization, group normalization, weight standardization, or other normalization processing.

[0158] As described herein, there may be multiple layers in decoder stage 506 for object query cross-attention stage 514 and object query self-attention stage 512 such that rich post-object queries 614 generated in the first layer of decoder stage 506 are communicated to a second layer (e.g., second object query cross-attention stage 514 and / or second object query self-attention stage 512), as shown by dashed line 610A etc. In some cases, there may be 2, 4, 6 or more layers. During the initialization stage at t = 0, more layers are used to initialize the system and generate initial scene-dependent object queries 614. At subsequent time steps (as shown in Figure 6B ), decoder stage 506 has fewer layers compared to the initialization stage. For example, in one embodiment, at the initialization stage, decoder stage 506 has six layers, and at the operational stage, decoder stage 506 has two layers.

[0159] (During initialization or operation) After processing all layers at the decoder stage, the output is a rich post-query set 614. The rich post-query set 614 is a machine representation of the queries (e.g., 256-value vector). There is a rich post-query 614 for each object query 604 input to the decoder stage.

[0160] Detection stage 510 outputs bounding boxes 520 based on the rich post-queries. Detection stage 510 can be a feed-forward network. In some embodiments, the FFN can predict the coordinates of the bounding box relative to the input image and predict a classification 522 associated with the bounding box.

[0161] Each query has a determined confidence value associated with each recognized class. The FFN can use the class to determine the type of object identified by the query. The FFN can have a defined number of detectable classes. For a query that identifies an object within a class, the FFN can have a confidence threshold for determining whether to generate a bounding box associated with a particular query. The FFN can use only the highest confidence value of the class to determine whether to generate a bounding box. For example, the confidence threshold can be 0.5, and if the class with the maximum value does not meet the threshold, no bounding box will be generated and the query will be discarded. The final output of the detection stage can be a defined set of regression parameters representing the bounding box 520 and the classification 522 identifying the object type.

[0162] In one embodiment, the set of parameters has the following parameters:

[0163] Regression parameters

[0164] 0, 1, 2 = X, Y, Z coordinates of the box centroid

[0165] 3, 4, 5 = width, length, and height of the box

[0166] 6, 7, 8 = velocity X, velocity Y, velocity Z

[0167] 9, 10, 11 = sin(yaw_orientation), cos(yaw_orientation), orientation interval

[0168] class parameter

[0169] Class number (7) + background class (1)

[0170] After processing the enriched query 614 at the detection level 510 and then querying 614, the perception system can determine a subset of the scene-dependent object queries 616 in the subsequent time steps to be used for processing. For example, the number of queries in the subset can be a fixed number (such as 150, 200, 300) or any other subset of the enriched query 614 generated by the decoder level. In some embodiments, the subset can be based on the confidence value associated with the query. For example, the perception system 402 can select queries for which there is sufficient confidence that the query represents an object detected within the scene. For example, a confidence threshold of 0.5 can be used for testing to determine the existence of an object. By selecting a defined number of enriched queries 614 to be included in the subset, the cut-off confidence threshold for the queries included in the scene-dependent object query may not always have the same confidence value. In some embodiments, queries can be selected based on a confidence value threshold. In this case, the number of scene-dependent object queries selected for the subsequent time step can change. A machine representation of the query encoding (e.g., a 256-value vector) is provided for each scene-dependent object query 616 provided to the subsequent time step.

[0171] As Figure 6B shown, the system is at t = 1 or any subsequent time step after the initialization step. The scene-dependent object queries are a subset of the enriched queries from the previous time step. These queries are a subset of the enriched query 614 determined at t = 0. These queries are scene-dependent object queries representing the objects identified in the previous scene. These queries provide the scene dependence that allows the decoder level to converge more quickly on the identification of the objects in the scene and provide better object identification. The scene-dependent object queries 616 are added to the set of queries 604 generated for the next time step. The total number of queries at the next time step will include the scene-independent object queries 604 and the scene-dependent object queries 616. The decoder level will process these two sets of queries. For example, the total number of queries being processed can be 900 scene-independent object queries and 200 scene-dependent object queries. The processing of the queries is as Figure 6A described.

[0172] When scene data (e.g., images) are obtained at intervals where the differences between the scene data do not change significantly, scene-dependent object queries can assist in making predictions at subsequent time steps. For example, in one embodiment, time steps can be executed at a rate of 10 Hz. Based on the operation of various systems, the rate of time steps can be faster or slower. The network is initialized at t = 0, and at subsequent time steps, scene-dependent object queries are used to better detect objects within the scene because scene-dependent object queries can converge more quickly compared to using scene-independent object queries.

[0173] Scene-dependent object queries from previous time steps assist in focusing on previously detected objects. Scene-independent object queries can assist in detecting objects missed during previous detection levels and / or any objects that have recently appeared within the field of view of the autonomous vehicle. In some instances, more than 90% of the scene-dependent object queries are passed to the next time step.

[0174] Initialization stage

[0175] Figure 7 is a flowchart illustrating an example of routine 700 implemented by at least one processor to navigate a vehicle based on at least one bounding box generated from one or more images at an initialization stage. Figure 7 The flowchart shown is provided for illustrative purposes only. It will be understood that one or more of the steps of the routine shown can be removed Figure 7 or the order of the steps can be changed. Additionally, for purposes of clarity of illustration of the example, one or more specific system components are described in the context of performing various operations during various data flow stages. However, other system arrangements and distributions of processing steps across system components and / or the autonomous vehicle computing 400 can be used.

[0176] At block 702, the perception system 402 receives an image of the vehicle scene. As described herein, the image can correspond to different image sensors or cameras located at different positions around the vehicle. Collectively, the images can represent a 360-degree view of the environment from the perspective of the vehicle.

[0177] At block 704, the perception system 402 generates a feature map based on the image. As described herein, in some cases, the perception system 402 generates at least one feature map for each received image. In some cases, the feature map includes a positional relationship corresponding to the image used to generate the feature map. For example, adjacent or neighboring images may correspond to adjacent or neighboring feature maps. In certain cases, the perception system 402 uses a feature pyramid network (such as but not limited to Resnet or Feature Pyramid Network (FPN), etc.) to generate the feature map. The feature map may have a specific channel depth (e.g., 256, 512, etc.). As described herein, the feature map may include features indicating the extracted characteristics of the image, such as but not limited to color, texture, position, reflectivity, shape, edges, etc.

[0178] The perception system 402 determines grid cells for the feature map. As described herein, the grid cells may have a specific size and shape and may cover the entire set of feature maps. In some cases, there may be multiple rows and / or multiple columns of grid cells, where the rows are aligned or offset from each other.

[0179] At block 706, the perception system 402 generates a scene-independent object query. As described herein, the perception system 402 may use the features of a specific object query to identify one or more features corresponding to the specific object query from the feature map. The perception system 402 may use the identified grid cells to modify the features of the specific object query. In some cases, the perception system 402 may weight the features of the identified one or more grid cells and use the weighted features to modify the features of the specific object query. In a similar manner, the perception system 402 may modify some or all of the features of the object queries. In certain cases, as part of generating the object query, the perception system 402 may use different feature maps and / or different data (such as heatmap data associated with a heatmap, for example, generated from a localization network different from the image feature extractor 504). Additionally, in certain cases, as part of the generation process, the perception system 402 may cross-attention (e.g., using the feature map generated at block 804) or self-attention the object query.

[0180] At block 708, the perception system 402 enriches the object query. As described herein, the perception system 402 may enrich the object query in various ways. The perception system may use a vision transformer with a decoder that has a defined number of layers. The perception system 402 may iterate through the various layers of the decoder until the defined number of layers has been completed. During the initialization stage, the defined number of layers used is greater than that during (such as herein regarding Figure 6B and Figure 8The number of layers used during the described operational levels. For example, during the initialization level, the perception system may iterate through six layers of the decoder to enrich scene-independent object queries. During the initialization level, the perception system is limited to using scene-independent object queries, which are used to generate scene-dependent object queries that can be used in subsequent time steps.

[0181] In some cases, the perception system 402 may enrich an object query based on any one or any combination of the following: (enriched) feature maps (non-limiting examples described herein at least with reference to the object query cross-attention level 514), features from other object queries (non-limiting examples described herein at least with reference to the object query self-attention level 512), positional encodings of the object query, and / or one or more processes (non-limiting examples described herein at least with reference to the FFN level 516). In certain cases, the perception system 402 may enrich an object query based on features from other object queries. In some such cases, a feature map may be generated from a feature map generated by an image feature network (e.g., the image feature extractor 504).

[0182] As described herein, to enrich an object query based on a feature map, the perception system 402 may perform a cross-attention function to determine the relationship between the object query and the feature map. The perception system 402 may identify (one or more) grid cells corresponding to different object queries from the feature map and use the features of the identified grid cells to enrich or modify the features of the corresponding object queries.

[0183] As described herein, to enrich an object query based on features from other object queries, the perception system 402 may perform a self-attention function to determine the relationship between object queries (e.g., by comparing the features of different object queries). Based on the determined relationship, the perception system 402 may weight the features from the object queries relative to each other and use the weighted features from some or all of the object queries to modify the features of a particular object query. For example, to enrich a first object query, the perception system 402 may compare the features of the first object query with the features of a set of at least one second object query and determine a relationship based on the comparison. The perception system 402 may further determine a weighting value to be applied to the features of the set of at least one second object query to generate weighted features from the set of at least one second object query. Then, the perception system 402 may use the weighted values from the set of at least one object query to determine or modify one or more features of the first object query. For example, as described herein, the perception system 402 may multiply a weighting value associated with a second object query by a particular feature of the second object query and use the weighted feature to determine or modify the corresponding feature of the first object query.

[0184] As described herein, to enrich object queries based on feature maps, the perception system 402 can generate a feature map (e.g., feature map 602) during the multi-view stage 503. The perception system 402 can use the features of the object query to identify one or more grid cells in the feature map corresponding to the object query, and use the features of the grid cells to modify the features of the corresponding object query. In some cases, the features from the grid cell / object query can be weighted based on the determined relationship between the grid cell / object query and the corresponding object query being modified.

[0185] At block 710, the perception system outputs scene-dependent object queries for subsequent time steps. After enriching the enriched query 614, the perception system 402 can determine a subset of the enriched query 614 that will be used as the scene-dependent object query 616 in the subsequent time step of processing. For example, the number of queries in the subset can be a fixed number (such as 150, 200, 300) or any other subset of the enriched query 614 generated by the decoder stage. In some embodiments, the subset can be based on a confidence value associated with the query. These queries are scene-dependent object queries representing the objects identified in the current scene. These queries provide the scene-dependence that allows the decoder to converge more quickly to the identification of the objects in the scene and provide better object identification. The scene-dependent object queries are added to the set of queries 604 generated for the next time step. The total number of queries at the next time step will include the scene-independent object queries 604 and the scene-dependent object queries 616.

[0186] At block 712, the perception system 402 generates at least one bounding box based on the enriched object query. As described herein, the perception system 402 can use one or more decoders to identify the bounding boxes for the objects in the image based on the enriched object query. In some cases, the perception system 402 can use a feed-forward network to process the enriched object query. The perception system can generate an object classification associated with the bounding box. In certain cases, more object queries for generating the bounding box can achieve improved accuracy of the bounding box.

[0187] At block 714, the perception system 402 causes the vehicle to be navigated based on the at least one bounding box. In some cases, the perception system 402 can communicate the bounding box to the planning system 404. The planning system 404 can use the bounding box to determine how to navigate the vehicle scene.

[0188] Routine 700 may include fewer, more, or different steps. For example, the use of a reference window and correlating the features of the grid cells within the window. However, it will be understood that other groups of grid cells may be used, such as but not limited to grid cells corresponding to the outline (or approximation of the outline) of an object.

[0189] In addition, the order of the steps may be changed and / or some steps may be repeated. For example, the perception system 402 may repeat blocks 708 corresponding to different layers of the decoder stage 506. The perception system 402 may output a scene-dependent object query at block 710 after block 712.

[0190] Operation level

[0191] Figure 8 is a flowchart illustrating an example of routine 800 implemented by at least one processor to navigate a vehicle based on at least one bounding box generated from one or more images using a scene-dependent object query. Figure 8 The flowchart shown is provided for illustrative purposes only. It will be understood that one or more of the steps of the routine shown may be removed Figure 8 or the order of the steps may be changed. In addition, for the purposes of clarifying the example, one or more specific system components are described in the context of performing various operations during the respective data flow levels. However, other system arrangements and distributions of the processing steps across system components and / or the autonomous vehicle computing 400 may be used.

[0192] At block 802, the perception system 402 receives an image of the vehicle scene. As described herein, the image may correspond to different image sensors or cameras located at different positions around the vehicle. Collectively, the images may represent a 360-degree view of the environment from the perspective of the vehicle.

[0193] At block 804, as described at least with reference to Figure 7 block 704 above, the perception system 402 generates a feature map based on the image.

[0194] At block 806, the perception system 402 generates a scene-independent object query. As described herein, the perception system 402 can use the features of a specific object query to identify one or more features corresponding to the specific object query from the feature map. The perception system 402 can use the identified grid cells to modify the features of the specific object query. In some cases, the perception system 402 can weight the features of the identified one or more grid cells and use the weighted features to modify the features of the specific object query. In a similar manner, the perception system 402 can modify the features of some or all of the object queries in the object query. In certain cases, as part of generating the object query, the perception system 402 can use different feature maps and / or different data (e.g., heatmap data associated with a heatmap) generated, for example, from a localization network different from the image feature extractor 504. Additionally, in certain cases, as part of the generation process, the perception system 402 can cross-attention (e.g., using the feature map generated at block 804) or self-attention the object query.

[0195] At block 808, the perception system 402 receives the scene-dependent object query from the previous time step. As described herein at least with reference to Figure 7 blocks 708 and 710 describe the generation and output of the scene-dependent object query.

[0196] At block 810, the perception system 402 enriches the object query. As described herein, the perception system 402 can enrich the object query in various ways. The perception system can use a vision transformer with a decoder that has a defined number of layers. The perception system 402 can iterate through the various layers of the decoder until the defined number of layers has been completed. During the operation level, the defined number of layers used is less than the number of layers used during the operation level (as described in Figure 6B and Figure 8 ). The perception system 402 can converge more quickly when using the scene-dependent object query and can better detect objects within the scene.

[0197] In some cases, the perception system 402 can enrich an object query based on any one or any combination of the following: (enriched) feature maps (non-limiting examples described herein with at least reference to the object query cross-attention level 514), features from other object queries (non-limiting examples described herein with at least reference to the object query self-attention level 512), the positional encoding of the object query, and / or one or more processes (non-limiting examples described herein with at least reference to the FFN level 516). In certain cases, the perception system 402 can enrich an object query based on features from other object queries. In some such cases, a feature map can be generated from the feature maps generated by an image feature network (e.g., the image feature extractor 504).

[0198] As described herein, to enrich an object query based on a feature map, the perception system 402 can perform a cross-attention function to determine the relationship between the object query and the feature map. The perception system 402 can identify the (one or more) grid cells corresponding to different object queries from the feature map, and use the features of the identified grid cells to enrich or modify the features of the corresponding object queries.

[0199] As described herein, to enrich an object query based on features from other object queries, the perception system 402 can perform a self-attention function to determine the relationship between the scene-dependent object query and the scene-independent object query (e.g., by comparing the features of different object queries). Based on the determined relationship, the perception system 402 can weight the features from the scene-dependent object query and the scene-independent object query relative to each other, and use the weighted features from some or all of the object queries to modify the features of a specific object query. For example, to enrich a first scene-dependent object query, the perception system 402 can compare the features of the first scene-dependent object query with the features of a set of at least one second object query (scene-dependent object query or scene-independent object query), and determine the relationship based on the comparison. The perception system 402 can further determine the weighting values to be applied to the features of the set of at least one second object query to generate the weighted features of the set of at least one second object query. Then, the perception system 402 can use the weighted values from the set of at least one object query to determine or modify one or more features of the first scene-dependent object query. For example, as described herein, the perception system 402 can multiply the weighting value associated with the second object query by a specific feature of the second object query, and use the weighted feature to determine or modify the corresponding feature of the first scene-dependent object query.

[0200] As described herein, to enrich object queries based on feature maps, the perception system 402 can generate feature maps (e.g., feature map 602) during the multi-view stage 503. The perception system 402 can use the features of object queries (scene-dependent object queries and scene-independent object queries) to identify one or more grid cells in the feature map corresponding to the object queries, and use the features of the grid cells to modify the features of the corresponding object queries. In some cases, the features from the grid cell / object query can be weighted based on the determined relationship between the grid cell / object query and the corresponding object query being modified. The perception system 402 can use the FFN to generate the enriched object queries.

[0201] At block 812, as described at least with reference to Figure 7 block 710 herein, the perception system 402 outputs scene-dependent object queries for use in subsequent time steps.

[0202] At block 814, the perception system 402 generates at least one bounding box based on the enriched object queries. As described herein, the perception system 402 can use one or more decoders to identify the bounding boxes for the objects in the image based on the enriched object queries. In some cases, the perception system 402 can use a feed-forward network to process the enriched object queries. The perception system can generate an object classification associated with the bounding box. In certain cases, more object queries for generating the bounding box can achieve improved accuracy of the bounding box.

[0203] At block 816, as described at least with reference to Figure 7 block 714 herein, the perception system 402 causes the vehicle to be navigated based on the at least one bounding box.

[0204] Fewer, more, or different blocks can be used with routine 800. In some cases, any one or any combination of the blocks from routine 700 can be combined with the blocks from routine 800, and vice versa.

[0205] Examples

[0206] The various example embodiments of the present disclosure can be described by the following clauses:

[0207] Clause 1. A method, comprising:

[0208] Receiving a first plurality of images from a plurality of image sensors, the first plurality of images corresponding to a plurality of views of a scene of a vehicle at a first time step;

[0209] Generating a first plurality of feature maps based on the first plurality of images;

[0210] Receiving a first set of object queries associated with the first time step;

[0211] Enrich the first object query set based on the first plurality of feature maps to generate a first enriched object query set;

[0212] Generate at least one bounding box for an object in the scene of the vehicle at the first time step based on the first enriched object query set;

[0213] Generate a second enriched object query set based on the first enriched object query set, wherein the second enriched object query set is a subset of the first enriched object query set;

[0214] Receive a second plurality of images from the plurality of image sensors, the second plurality of images corresponding to a plurality of views of the scene of the vehicle at a second time step;

[0215] Generate a second plurality of feature maps based on the second plurality of images;

[0216] Receive a second object query set associated with the second time step;

[0217] Receive the second enriched object query set;

[0218] Enrich the second object query set and the second enriched object query set based on the second plurality of feature maps to generate a third enriched object query set;

[0219] Generate at least one bounding box for an object in the scene of the vehicle at the second time step based on the third enriched object query set; and

[0220] Cause the vehicle to be controlled based on the at least one bounding box.

[0221] Clause 2. The method according to clause 1, wherein enriching the second object query set and the second enriched object query set includes: enriching the first object query set based on the second enriched object query set.

[0222] Clause 3. The method according to any one of clauses 1 and 2, wherein enriching the second object query set and the second enriched object query set based on the second plurality of feature maps includes:

[0223] Performing a cross-attention calculation function between the second object query set and the second plurality of feature maps, and

[0224] Performing a cross-attention calculation function between the second enriched object query set and the second plurality of feature maps.

[0225] Clause 4. The method according to any one of Clauses 1 to 3, wherein enriching the first set of object queries includes: iterating through a first number of layers of a decoder to provide the first enriched set of object queries.

[0226] Clause 5. The method according to Clause 4, wherein enriching the second set of object queries and the second enriched set of object queries includes: iterating through a second number of layers of the decoder to provide the third enriched set of object queries, wherein the second number of layers is less than the first number of layers.

[0227] Clause 6. The method according to any one of Clauses 1 to 5, wherein generating at least one bounding box for an object in a scene of the vehicle based on the third enriched set of object queries includes: generating a classification of the object type of an individual bounding box, and

[0228] wherein causing the vehicle to be controlled based on the at least one bounding box includes: causing the vehicle to be controlled based on the classification.

[0229] Clause 7. The method according to any one of Clauses 1 to 6, wherein the second enriched set of object queries is a defined number of queries in the first enriched set of object queries.

[0230] Clause 8. The method according to any one of Clauses 1 to 7, wherein individual image sensors among the plurality of image sensors are placed around the vehicle at different orientations.

[0231] Clause 9. The method according to any one of Clauses 1 to 8, wherein the first set of object queries and the second set of object queries are the same.

[0232] Clause 10. A system, comprising:

[0233] A data memory that stores computer-executable instructions; and

[0234] A processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:

[0235] Receive a first plurality of images from a plurality of image sensors, the first plurality of images corresponding to a plurality of views of a scene of a vehicle at a first time step;

[0236] Generate a first plurality of feature maps based on the first plurality of images;

[0237] Receive a first set of object queries associated with the first time step;

[0238] Enrich the first object query set based on the first plurality of feature maps to generate a first enriched object query set;

[0239] Generate at least one bounding box for an object in the scene of the vehicle at the first time step based on the first enriched object query set;

[0240] Generate a second enriched object query set based on the first enriched object query set, wherein the second enriched object query set is a subset of the first enriched object query set;

[0241] Receive a second plurality of images from the plurality of image sensors, the second plurality of images corresponding to a plurality of views of the scene of the vehicle at a second time step;

[0242] Generate a second plurality of feature maps based on the second plurality of images;

[0243] Receive a second object query set associated with the second time step;

[0244] Receive the second enriched object query set;

[0245] Enrich the second object query set and the second enriched object query set based on the second plurality of feature maps to generate a third enriched object query set;

[0246] Generate at least one bounding box for an object in the scene of the vehicle at the second time step based on the third enriched object query set; and

[0247] Cause the vehicle to be controlled based on the at least one bounding box.

[0248] Clause 11. The system according to clause 10, wherein, in order to enrich the second object query set and the second enriched object query set, the processor is configured to enrich the first object query set based on the second enriched object query set.

[0249] Clause 12. The system according to any one of clauses 10 and 11, wherein, in order to enrich the second object query set and the second enriched object query set based on the second plurality of feature maps, the processor is configured to:

[0250] Perform a cross-attention calculation function between the second object query set and the second plurality of feature maps, and

[0251] Perform a cross-attention calculation function between the second enriched object query set and the second plurality of feature maps.

[0252] Clause 13. The system according to any one of Clauses 10 to 12, wherein, in order to enrich the first object query set, the processor is configured to iterate through a first number of layers of the decoder to provide the first enriched object query set.

[0253] Clause 14. The system according to Clause 13, wherein, in order to enrich the second object query set and the second enriched object query set, the processor is configured to iterate through a second number of layers of the decoder to provide the third enriched object query set, wherein the second number of layers is less than the first number of layers.

[0254] Clause 15. The system according to any one of Clauses 10 to 14, wherein, in order to generate at least one bounding box for an object in a scene of the vehicle based on the second enriched object query set, the processor is configured to generate a classification of the object type of an individual bounding box, and

[0255] wherein, in order to control the vehicle based on the at least one bounding box, the processor is configured to cause the vehicle to be controlled based on the classification.

[0256] Clause 16. The system according to any one of Clauses 10 to 15, wherein the second enriched object query set is a defined number of queries in the first enriched object query set.

[0257] Clause 17. The system according to any one of Clauses 10 to 16, wherein individual image sensors among the plurality of image sensors are placed around the vehicle in different orientations.

[0258] Clause 18. The system according to any one of Clauses 10 to 17, wherein the first object query set and the second object query set are the same.

[0259] Clause 19. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:

[0260] Receive a first plurality of images from a plurality of image sensors, the first plurality of images corresponding to a plurality of views of a scene of a vehicle at a first time step;

[0261] Generate a first plurality of feature maps based on the first plurality of images;

[0262] Receive a first object query set associated with the first time step;

[0263] Enrich the first object query set based on the first plurality of feature maps to generate a first enriched object query set;

[0264] Generate at least one bounding box for an object in the scene of the vehicle at the first time step based on the first enriched object query set;

[0265] Generate a second enriched object query set based on the first enriched object query set, wherein the second enriched object query set is a subset of the first enriched object query set;

[0266] Receive a second plurality of images from the plurality of image sensors, the second plurality of images corresponding to multiple views of the scene of the vehicle at a second time step;

[0267] Generate a second plurality of feature maps based on the second plurality of images;

[0268] Receive a second object query set associated with the second time step;

[0269] Receive the second enriched object query set;

[0270] Enrich the second object query set and the second enriched object query set based on the second plurality of feature maps to generate a third enriched object query set;

[0271] Generate at least one bounding box for an object in the scene of the vehicle at the second time step based on the third enriched object query set; and

[0272] Cause the vehicle to be controlled based on the at least one bounding box.

[0273] Clause 20. The non-transitory computer-readable medium according to clause 19, wherein, in order to enrich the second object query set and the second enriched object query set, the computing system is configured to enrich the second object query set based on the second enriched object query set.

[0274] Clause 21. The non-transitory computer-readable medium according to any one of clauses 19 and 20, wherein, in order to enrich the second object query set and the second enriched object query set based on the second plurality of feature maps, the computing system is configured to:

[0275] Perform a cross-attention calculation function between the second object query set and the second plurality of feature maps, and

[0276] Perform a cross-attention calculation function between the second enriched object query set and the second plurality of feature maps.

[0277] Clause 22. The non-transitory computer-readable medium according to any one of Clauses 19 to 21, wherein, in order to enrich the first object query set, the computing system is configured to iterate through a first number of layers of the decoder to provide the first enriched object query set.

[0278] Clause 23. The non-transitory computer-readable medium according to Clause 22, wherein, in order to enrich the second object query set and the second enriched object query set, the computing system is configured to iterate through a second number of layers of the decoder to provide the third enriched object query set, wherein the second number of layers is less than the first number of layers.

[0279] Clause 24. The non-transitory computer-readable medium according to any one of Clauses 19 to 23, wherein, in order to generate at least one bounding box for an object in a scene for the vehicle based on the second enriched object query set, the computing system is configured to generate a classification of the object type of an individual bounding box, and

[0280] wherein, in order to control the vehicle based on the at least one bounding box, the computing system is configured to cause the vehicle to be controlled based on the classification.

[0281] Clause 25. The non-transitory computer-readable medium according to any one of Clauses 19 to 24, wherein the second enriched object query set is a defined number of queries in the first enriched object query set.

[0282] Clause 26. The non-transitory computer-readable medium according to any one of Clauses 19 to 25, wherein individual image sensors among the plurality of image sensors are placed around the vehicle at different orientations.

[0283] Clause 27. The non-transitory computer-readable medium according to any one of Clauses 19 to 26, wherein the first object query set and the second object query set are the same.

[0284] Additional examples

[0285] All of the methods and tasks described herein can be performed and fully automated by a computer system. In some cases, the computer system can include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, cloud computing resources, etc.) that communicate and interoperate over a network to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in an execution memory or other non-transitory computer-readable storage medium or device (e.g., solid-state storage device, disk drive, etc.). The various functions disclosed herein can be embodied in such program instructions or can be implemented in the dedicated circuitry (e.g., ASIC or FPGA) of the computer system. In cases where the computer system includes multiple computing devices, these devices can, but need not, be located in the same location. The results of the disclosed methods and tasks can be persistently stored by transforming a physical storage device such as a solid-state memory chip or disk into a different state. In some embodiments, the computer system can be a cloud-based computing system whose processing resources are shared by multiple different business entities or other users.

[0286] The processes described herein or illustrated in the figures of the present disclosure can be initiated in response to events (such as according to a pre-determined or dynamically determined schedule, on demand when initiated by a user or system administrator, or in response to certain other events). When such a process is initiated, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard disk drive, flash memory, removable media, etc.) can be loaded into the memory (e.g., RAM) of a server or other computing device. These executable instructions can then be executed by a hardware-based computer processor of the computing device. In some embodiments, such a process or a portion thereof can be implemented serially or in parallel on multiple computing devices and / or multiple processors.

[0287] According to this embodiment, certain actions, events, or functions of any process or algorithm described herein can be performed in a different sequence, can be added, combined, or completely omitted (e.g., not all of the described operations or events are necessary for the practice of the algorithm). Additionally, in certain embodiments, operations or events can occur simultaneously rather than sequentially, for example, through multi-threading, interrupt handling, or multiple processors or processor cores or on other parallel architectures.

[0288] The various illustrative logical blocks, modules, routines, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware (e.g., an ASIC or FPGA device), computer software running on computer hardware, or a combination of both. In addition, the various illustrative logical blocks and modules described in connection with the embodiments disclosed herein can be implemented or performed by: a machine, such as a processor device, a digital signal processor ("DSP"), an application specific integrated circuit ("ASIC"), a field programmable gate array ("FPGA"), or other programmable logic device; discrete gate or transistor logic; discrete hardware components; or any combination thereof designed to perform the functions described herein. The processor device can be a microprocessor, but in an alternative, the processor device can be a controller, a microcontroller, or a state machine, or a combination thereof, etc. The processor device can include circuitry configured to process computer executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logical operations without processing computer executable instructions. The processor device can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more than one microprocessor in combination with a DSP core, or any other such configuration. Although described herein primarily in terms of digital technology, the processor device can also include primarily analog components. For example, a portion or all of the rendering techniques described herein can be implemented in analog circuitry or in mixed analog and digital circuitry. The computing environment can include any type of computer system, including but not limited to a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computing engine within an appliance (to name just a few).

[0289] The elements of a method, process, routine, or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor device, or in a combination of both. The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer readable storage medium. An example storage medium can be coupled to the processor device such that the processor device can read information from, and write information to, the storage medium. In an alternative, the storage medium can be integral with the processor device. The processor device and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In an alternative, the processor device and the storage medium can reside as discrete components in a user terminal.

[0290] In the foregoing description, aspects and embodiments of the present disclosure have been described with reference to numerous specific details, which may vary depending on the implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive in nature. The sole and exclusive indication of the scope of the invention, and what the applicant desires to be the scope of the invention, is the literal and equivalent scope of the terms of the claims that are issued from this application in the specific form of the granted claims, including any subsequent amendments. Any definitions expressly set forth herein for terms to be included in such claims shall be controlling as to the meaning of such terms as used in the claims. Additionally, when the term "further comprises" is used in the foregoing specification or the appended claims, the remainder of the phrase can be additional steps or entities, or sub-steps / sub-entities of the previously recited steps or entities.

Claims

1. A method, comprising: Receiving a first plurality of images from a plurality of image sensors, the first plurality of images corresponding to a plurality of views of a scene of a vehicle at a first time step; Generating a first plurality of feature maps based on the first plurality of images; Receiving a first set of object queries associated with the first time step; Enriching the first set of object queries based on the first plurality of feature maps to generate a first enriched set of object queries; Generating at least one bounding box for an object in the scene of the vehicle at the first time step based on the first enriched set of object queries; Generating a second enriched set of object queries based on the first enriched set of object queries, wherein the second enriched set of object queries is a subset of the first enriched set of object queries; Receiving a second plurality of images from the plurality of image sensors, the second plurality of images corresponding to a plurality of views of the scene of the vehicle at a second time step; Generating a second plurality of feature maps based on the second plurality of images; Receiving a second set of object queries associated with the second time step; Receiving the second enriched set of object queries; Enriching the second set of object queries and the second enriched set of object queries based on the second plurality of feature maps to generate a third enriched set of object queries; Generating at least one bounding box for an object in the scene of the vehicle at the second time step based on the third enriched set of object queries; and Causing the vehicle to be controlled based on the at least one bounding box.

2. The method according to claim 1, wherein, Enriching the second set of object queries and the second enriched set of object queries includes: enriching the first set of object queries based on the second enriched set of object queries.

3. The method according to any one of claims 1 and 2, wherein Enriching the second set of object queries and the second enriched set of object queries based on the second plurality of feature maps includes: Performing a cross-attention calculation function between the second set of object queries and the second plurality of feature maps, and Performing a cross-attention calculation function between the second enriched set of object queries and the second plurality of feature maps.

4. The method according to any one of claims 1 to 3, wherein, Enriching the first set of object queries includes: iterating through a first number of layers of a decoder to provide the first enriched set of object queries.

5. The method according to claim 4, wherein, Enriching the second set of object queries and the second enriched set of object queries includes: iterating through a second number of layers of the decoder to provide the third enriched set of object queries, wherein the second number of layers is less than the first number of layers.

6. The method according to any one of claims 1 to 5, wherein Generating at least one bounding box for an object in the scene of the vehicle based on the third enriched set of object queries includes: generating a classification of the object type of an individual bounding box, and Wherein causing the vehicle to be controlled based on the at least one bounding box includes: causing the vehicle to be controlled based on the classification.

7. The method according to any one of claims 1 to 6, wherein The second enriched set of object queries is a defined number of queries in the first enriched set of object queries.

8. The method according to any one of claims 1 to 7, wherein Individual image sensors among the plurality of image sensors are placed around the vehicle in different orientations.

9. The method according to any one of claims 1 to 8, wherein, The first object query set and the second object query set are the same.

10. A system comprising: A data memory that stores computer-executable instructions; And A processor configured to execute the computer-executable instructions, wherein the execution of the computer-executable instructions causes the system to: Receive a first plurality of images from a plurality of image sensors, the first plurality of images corresponding to a plurality of views of a scene of a vehicle at a first time step; Generate a first plurality of feature maps based on the first plurality of images; Receive a first object query set associated with the first time step; Enrich the first object query set based on the first plurality of feature maps to generate a first enriched object query set; Based on the first enriched object query set, generate at least one bounding box for an object in the scene of the vehicle at the first time step; Generate a second enriched object query set based on the first enriched object query set, wherein the second enriched object query set is a subset of the first enriched object query set; Receive a second plurality of images from the plurality of image sensors, the second plurality of images corresponding to a plurality of views of the scene of the vehicle at a second time step; Generate a second plurality of feature maps based on the second plurality of images; Receive a second object query set associated with the second time step; Receive the second enriched object query set; Enrich the second object query set and the second enriched object query set based on the second plurality of feature maps to generate a third enriched object query set; Based on the third enriched object query set, generate at least one bounding box for an object in the scene of the vehicle at the second time step; and Cause the vehicle to be controlled based on the at least one bounding box.

11. The system according to claim 10, wherein, To enrich the second object query set and the second enriched object query set, the processor is configured to enrich the first object query set based on the second enriched object query set.

12. The system according to any one of claims 10 and 11, wherein To enrich the second object query set and the second enriched object query set based on the second plurality of feature maps, the processor is configured to: Perform a cross-attention calculation function between the second object query set and the second plurality of feature maps, and Perform a cross-attention calculation function between the second enriched object query set and the second plurality of feature maps.

13. The system according to any one of claims 10 to 12, wherein, To enrich the first object query set, the processor is configured to iterate through a first number of layers of a decoder to provide the first enriched object query set.

14. The system according to claim 13, wherein, To enrich the second object query set and the second enriched object query set, the processor is configured to iterate through a second number of layers of the decoder to provide the third enriched object query set, wherein the second number of layers is less than the first number of layers.

15. The system according to any one of claims 10 to 14, wherein, To generate at least one bounding box for an object in a scene of the vehicle based on the second enriched object query set, the processor is configured to generate a classification of an object type of an individual bounding box, and wherein, to control the vehicle based on the at least one bounding box, the processor is configured to control the vehicle based on the classification.

16. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:[[]] Receive a first plurality of images from a plurality of image sensors, the first plurality of images corresponding to a plurality of views of a scene of a vehicle at a first time step; Generate a first plurality of feature maps based on the first plurality of images; Receive a first object query set associated with the first time step; Enrich the first object query set based on the first plurality of feature maps to generate a first enriched object query set; Generate at least one bounding box for an object in the scene of the vehicle at the first time step based on the first enriched object query set; Generate a second enriched object query set based on the first enriched object query set, wherein the second enriched object query set is a subset of the first enriched object query set; Receive a second plurality of images from the plurality of image sensors, the second plurality of images corresponding to a plurality of views of the scene of the vehicle at a second time step; Generate a second plurality of feature maps based on the second plurality of images; Receive a second object query set associated with the second time step; Receive the second enriched object query set; Enrich the second object query set and the second enriched object query set based on the second plurality of feature maps to generate a third enriched object query set; Generate at least one bounding box for an object in the scene of the vehicle at the second time step based on the third enriched object query set; and Cause the vehicle to be controlled based on the at least one bounding box.

17. The non-transitory computer-readable medium according to claim 16, wherein, To enrich the second object query set and the second enriched object query set, the computing system is configured to enrich the second object query set based on the second enriched object query set.

18. The non-transitory computer-readable medium according to any one of claims 16 and 17, wherein, To enrich the second object query set and the second enriched object query set based on the second plurality of feature maps, the computing system is configured to:[[]] Perform a cross-attention calculation function between the second object query set and the second plurality of feature maps, and Perform a cross-attention calculation function between the second enriched object query set and the second plurality of feature maps.

19. The non-transitory computer-readable medium according to any one of claims 16 to 18, wherein, To enrich the first object query set, the computing system is configured to iterate through a first number of layers of a decoder to provide the first enriched object query set.

20. The non-transitory computer-readable medium according to claim 19, wherein, To enrich the second object query set and the second enriched object query set, the computing system is configured to iterate through a second number of layers of the decoder to provide the third enriched object query set, where the second number of layers is less than the first number of layers.

Images