Deep learning based beam control for autonomous vehicles

Abstract

The invention provides deep learning based light beam control for autonomous vehicles. Systems and methods for deep learning based light beam control are provided. Sensor data associated with an environment and corresponding detected objects from a perception system are obtained. Object features and image features are extracted. The extracted object features and image features are fused into fused features. A light beam control state is predicted from the fused features, where the light beam control state indicates a high beam illumination intensity or a low beam illumination intensity of a light emitting device.

Description

Technical Field

[0001] This invention relates to deep learning-based beam control for autonomous vehicles. Background Technology

[0002] Vehicles typically include one or more light-emitting devices to illuminate their interior or exterior. For example, internal light-emitting devices provide illumination inside the vehicle, increasing visibility within. External light-emitting devices, such as headlights, provide illumination of the surrounding environment. Specifically, headlights illuminate the area in front of the vehicle. Control of light emission by the autonomous vehicle is an essential capability for its safe navigation of its environment. Summary of the Invention

[0003] A system includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the following operations: extracting object features from a perception system output using an object feature network to identify objects in an environment, wherein the object feature network outputs object features characterizing object information related to a beam control state, and wherein the object features include data associated with at least location and classification data associated with at least one object; extracting image features from sensor data using an image feature network to identify ambient lighting information, wherein the image feature network outputs image features characterizing the ambient lighting information related to the beam control state; fusing the object features and the image features into a fused feature using a feature fusion network, the feature fusion network taking the object features and the image features as input and outputting the fused feature; and predicting a beam control state based on the fused feature, wherein the beam control state indicates the high beam illumination intensity or low beam illumination intensity of a light-emitting device; and a control circuit communicatively coupled to the at least one processor, wherein the control circuit is configured to operate the light-emitting device based on the beam control state.

[0004] A method includes: using at least one processor, extracting object features from the output of a perception system using an object feature network to identify objects in an environment, wherein the object feature network outputs object features characterizing object information related to a beam control state, and wherein the object features include data associated with at least location and classification data associated with at least one object; using the at least one processor, extracting image features from sensor data using an image feature network to identify ambient lighting information, wherein the image feature network outputs image features characterizing the ambient lighting information related to the beam control state; using the at least one processor, fusing the object features and the image features into a fused feature using a feature fusion network, the feature fusion network taking the object features and the image features as input and outputting the fused feature; using the at least one processor, predicting the beam control state based on the fused feature, wherein the beam control state indicates the high beam illumination intensity or low beam illumination intensity of a light-emitting device; and using the at least one processor, operating the light-emitting device based on the beam control state.

[0005] A non-transitory computer program product storing instructions that, when executed by at least one programmable processor of a vehicle, cause the at least one programmable processor to operate, the operation comprising: extracting object features from a perception system output using an object feature network to identify objects in an environment, wherein the object feature network outputs object features characterizing object information related to a beam control state, and wherein the object features include data associated with at least location and classification data associated with at least one object; extracting image features from sensor data using an image feature network to identify ambient lighting information, wherein the image feature network outputs image features characterizing the ambient lighting information related to the beam control state; fusing the object features and the image features into a fused feature using a feature fusion network, the feature fusion network taking the object features and the image features as input and outputting the fused feature; predicting the beam control state based on the fused feature, wherein the beam control state indicates the high beam illumination intensity or low beam illumination intensity of a light-emitting device; and operating the light-emitting device based on the beam control state. Attached Figure Description

[0006] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system;

[0007] Figure 2 It is a diagram of one or more systems that include autonomous vehicles;

[0008] Figure 3 yes Figure 1 and Figure 2 A diagram of one or more devices and / or one or more system components;

[0009] Figure 4A It is a diagram of some components of an autonomous system;

[0010] Figure 4B This is a diagram illustrating the implementation of a neural network;

[0011] Figure 4C and Figure 4D This is a diagram illustrating example operations of a CNN;

[0012] Figure 5 This is a diagram illustrating the implementation of processing for a deep learning-based beam control system; and

[0013] Figure 6 This is a diagram of the beam illumination intensity control system; and

[0014] Figure 7 This is a diagram of the beam control network;

[0015] Figure 8 This is a block diagram of a system used for training and deploying beam control networks;

[0016] Figure 9 It is a block diagram of the perception model; and

[0017] Figure 10 This is a block diagram of the processing used in a deep learning-based beam control system. Detailed Implementation

[0018] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0019] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0020] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0021] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0022] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet that includes data (e.g., a data packet, etc.).

[0024] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0025] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0026] General Overview

[0027] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement a deep learning-based beam control system. Generally, a beam refers to light emitted by a light-emitting device. In the example, a beam is a cone of light guided to a predetermined location. Examples of light-emitting devices include headlights, taillights, daytime running lights, fog lights, signal lights, brake lights, hazard warning lights, driving lights, etc. For ease of description, the light-emitting device according to this technology is described as a headlight. However, any light-emitting device that changes one or more states (e.g., illumination intensity) during operation can be used. Furthermore, this technology is described as applied to a beam of light output by a headlight. However, this technology is applicable to any emitted light, such as diffused light and ambient light.

[0028] Vehicles (such as autonomous vehicles) include one or more light-emitting devices that illuminate the environment surrounding the vehicle. For example, headlights are typically located at the front of the vehicle and produce a beam of light capable of maximizing visibility of the road ahead. The beam of light emitted by the headlights at the front of the vehicle is generally referred to as the headlight beam. The intensity of the beam emitted by the headlights (e.g., the headlight beam) can vary depending on various environmental conditions. Sensor data is obtained from at least one sensor. Image features and object features are extracted from the sensor data and the objects detected by the perception system, respectively. The image features and object features are fused and fed into a beam classifier. Once trained, the beam classifier takes the fused features associated with the environment as input and classifies the corresponding output beam of the headlights as either high beam intensity or low beam intensity. In this example, the beam classifier is iteratively updated using additional incremental hard sample data acquired during the real-time deployment of the beam classifier.

[0029] By utilizing the systems, methods, and computer program products described herein, a lightweight deep learning system for automatic beam illumination intensity control is realized using deep learning-based beam control systems. The system according to this technique reuses data typically extracted for other autonomous vehicle operations, thereby reducing or eliminating the need for additional sensors and / or datasets for controlling the operation of the illumination device. Furthermore, this technique is closed-loop and enables continuous improvement of the deep learning model without requiring manual labeling of training data. This technique ensures the use of high beams or low beams during autonomous vehicle operation as expected (e.g., in compliance with vehicle traffic laws or local road regulations) and ensures that only rigorously tested software is used in deployment.

[0030] Now for reference Figure 1Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0031] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2 The vehicles 102 are the same as or similar to autonomous vehicles 202. In some embodiments, vehicles 200 in a group of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0032] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0033] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with (e.g., defining) a series of actions (also referred to as trajectories) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will board the AV, and a second state or area includes a location where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Additionally or alternatively, route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within a lane area and target rates at those locations. In the example, route 106 includes multiple precise state sequences along at least one high-level action with a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to multiple trajectories that collectively form a high-level route terminating at a final target state or region.

[0034] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In the example, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driving lane, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In the example, a road includes at least one lane associated with at least one lane marking (e.g., identified based on at least one lane marking).

[0035] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with vehicle 102 and / or V2I infrastructure system 118. In some embodiments, the V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, the V2I device 110 includes radio frequency identification (RFID) devices, signs, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markings, streetlights, parking meters, etc. In some embodiments, the V2I device 110 is configured to communicate directly with vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, and / or queue management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0036] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0037] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing, etc.). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0038] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I infrastructure system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including and / or not including autonomous systems)).

[0039] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0040] supply Figure 1 The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements arranged differently. Additionally or alternatively, at least one element of environment 100 may be described as being composed of… Figure 1 One or more functions performed by at least one different element of environment 100. Additionally or alternatively, at least one group of elements of environment 100 may perform one or more functions described as performed by at least one different group of elements of environment 100.

[0041] Now for reference Figure 2 The vehicle 200 includes an autonomous system 202, a powertrain control system 204, a steering control system 206, a braking system 208, and a lighting system 210. In some embodiments, the vehicle 200 and the vehicle 102 (see...) Figure 1 The vehicle 200 is similar to or the same as the vehicle in question. In some embodiments, the vehicle 200 has autonomous capabilities (e.g., implementing at least one function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention) and / or highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations)). For a detailed description of fully autonomous and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road MotorVehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0042] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, and drive-by-wire (DBW) system 202h.

[0043] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision). In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1 The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0044] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other camera-included systems described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., wide-angle lens, fisheye lens, and / or a lens with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0045] The laser detection and ranging (LiDAR) sensor 202b includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object in the field of view of the LiDAR sensor 202b.

[0046] The radio detection and ranging (radar) sensor 202c includes components configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with...). Figure 3At least one device that communicates with the same or similar bus (bus 302). The radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the radar sensor 202c encounter a physical object and are reflected back to the radar sensor 202c. In some embodiments, the radio waves emitted by the radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the radar sensor 202c generates a signal representing objects included in the field of view of the radar sensor 202c. For example, at least one data processing system associated with the radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of the radar sensor 202c.

[0047] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0048] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, an autonomous vehicle computing system 202f, a safety controller 202g, and / or a drive-by-wire (DBW) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0049] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1 V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1 The V2I system 118 communicates with the same or similar V2I system.

[0050] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0051] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200. Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0052] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a certain direction, decelerate in a certain direction, make a left turn and / or make a right turn, etc. In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0053] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to turn left or right, thereby causing the vehicle 200 to turn left or right.

[0054] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0055] The lighting system 210 includes at least one light-emitting device configured to output light. In some examples, according to the present technology, the lighting system 210 includes at least one controller and / or actuator configured to cause one or more light-emitting devices of the vehicle 200 to emit light at at least one illumination intensity. Additionally or alternatively, in some examples, the lighting system 210 includes headlights, taillights, daytime running lights, fog lights, signal lights, brake lights, hazard warning lights, driving lights, etc. Illumination intensity generally refers to the amount of light output by the light-emitting device. In the examples, illumination intensity is the number of lumens output by the light-emitting device. Generally, high beam illumination intensity outputs more light (e.g., a higher number of lumens) compared to low beam illumination intensity. High beam illumination intensity and low beam illumination intensity are collectively referred to as beam control state. Beam control state refers to the state of output of the light-emitting device. In the examples, the state of the light-emitting device is relative to other states of the same light-emitting device. In the examples, the state of the light-emitting device is relative to the states of other light-emitting devices.

[0056] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), a wheel rate sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor.

[0057] Now for reference Figure 3 A schematic diagram of device 300 is illustrated. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of vehicle 102 system); at least one device of lighting system 210; and / or one or more devices of network 112 (e.g., one or more devices of network 112 system). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of vehicle 102 system), at least one device of lighting system 210, and / or one or more devices of network 112 (e.g., one or more devices of network 112 system) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0058] Bus 302 includes components for communication between the components of the licensed device 300. In some embodiments, processor 304 is implemented in hardware, software, or a combination of hardware and software. In some examples, processor 304 includes a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or accelerated processing unit (APU), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC), etc.). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by processor 304.

[0059] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0060] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0061] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, etc. Interfaces and / or cellular network interfaces, etc.

[0062] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 305 and / or storage component 308, executed by processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0063] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. When executed, the software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0064] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0065] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the other device, which, when executed by the processor of processor 304 and / or the processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0066] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a group of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another group of components of device 300.

[0067] Now for reference Figure 4A The diagram illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 41 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0068] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0069] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the sensing system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the sensing system 402. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0070] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various other types of traffic signals). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0071] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0072] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206), and / or the braking system (e.g., braking system 208) and / or the lighting system (e.g., lighting system 210). In an example, where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.). For example, when a change in environmental conditions occurs (e.g., as detected by one or more sensors or devices of vehicle 200), control system 408 sends a control signal to lighting system 210 to adjust the lighting intensity of the light-emitting device of vehicle 200.

[0073] In some embodiments, the sensing system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the sensing system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the sensing system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.). For example, a lighting system (e.g., lighting system 210) may implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more lighting intensities within the environment, etc.). The following is about Figures 4B to 4D This includes examples of implementations of machine learning models.

[0074] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3 (The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable zones (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or trails off-road, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0075] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0076] Now for reference Figure 4B The diagram illustrates an implementation of a machine learning model. More specifically, it illustrates an implementation of a convolutional neural network (CNN) 420. For illustrative purposes, the following description of CNN 420 will concern the implementation of CNN 420 via a perception system 402. However, it will be understood that in some examples, CNN 420 (e.g., one or more components of CNN 420) is implemented by systems other than or besides perception system 402 (such as planning system 404, positioning system 406, and / or control system 408, etc.). Although CNN 420 includes certain features as described herein, these features are provided for illustrative purposes and are not intended to limit this disclosure.

[0077] CNN 420 includes multiple convolutional layers comprising a first convolutional layer 422, a second convolutional layer 424, and a convolutional layer 426. In some embodiments, CNN 420 includes a subsampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, subsampling layer 428 and / or other subsampling layers have a dimension smaller than that of the upstream system (i.e., the number of nodes). By means of subsampling layer 428 having a dimension smaller than that of the upstream layers, CNN 420 combines the amount of data associated with the initial input and / or output of the upstream layers, thereby reducing the computational cost required for downstream convolutional operations by CNN 420. Additionally or alternatively, subsampling layer 428 is associated with at least one subsampling function (e.g., configured to perform at least one subsampling function) (as described below). Figure 4C and Figure 4D As described, CNN 420 combines the amount of data associated with the initial input.

[0078] Based on the perception system 402 providing corresponding inputs and / or outputs associated with the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426 to generate corresponding outputs, the perception system 402 performs convolution operations. In some examples, based on the perception system 402 providing data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426, the perception system 402 implements a CNN 420. In such examples, based on the perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle identical or similar to vehicle 102, a remote AV system identical or similar to remote AV system 114, a queue management system identical or similar to queue management system 116, and / or a V2I system identical or similar to V2I system 118, etc.), the perception system 402 provides data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426. The following is about Figure 4C Includes a detailed explanation of convolution operations.

[0079] In some embodiments, the perception system 402 provides data associated with an input (referred to as initial input) to a first convolutional layer 422, and the perception system 402 uses the first convolutional layer 422 to generate data associated with an output. In some embodiments, the perception system 402 provides the output generated by the convolutional layer as input to different convolutional layers. For example, the perception system 402 provides the output of the first convolutional layer 422 as input to a subsampling layer 428, a second convolutional layer 424, and / or a convolutional layer 426. In such an example, the first convolutional layer 422 is referred to as the upstream layer, and the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426 are referred to as downstream layers. Similarly, in some embodiments, the perception system 402 provides the output of the subsampling layer 428 to the second convolutional layer 424 and / or the convolutional layer 426, and in this example, the subsampling layer 428 will be referred to as the upstream layer, and the second convolutional layer 424 and / or the convolutional layer 426 will be referred to as the downstream layer.

[0080] In some embodiments, before providing input to the CNN 420, the perception system 402 processes the data associated with the input provided to the CNN 420. For example, the perception system 402 processes the data associated with the input provided to the CNN 420 based on the normalization of sensor data (e.g., image data, LiDAR data, and / or radar data, etc.) by the perception system 402.

[0081] In some embodiments, the perception system 402 generates an output by performing convolution operations associated with each convolutional layer of the CNN 420. In some examples, the CNN 420 generates an output by performing convolution operations associated with each convolutional layer and an initial input based on the perception system 402. In some embodiments, the perception system 402 generates an output and provides that output to a fully connected layer 430. In some examples, the perception system 402 provides the output of a convolutional layer 426 to a fully connected layer 430, wherein the fully connected layer 430 includes data associated with multiple feature values ​​referred to as F1, F2, ..., FN. In this example, the output of the convolutional layer 426 includes data associated with multiple output feature values ​​representing a prediction.

[0082] In some embodiments, the perception system 402 identifies a prediction from among a plurality of predictions based on a feature value identified as the highest probability of being the correct prediction among multiple predictions. For example, if the fully connected layer 430 includes feature values ​​F1, F2, ..., FN and F1 is the largest feature value, the perception system 402 identifies the prediction associated with F1 as the correct prediction among multiple predictions. In some embodiments, the perception system 402 trains the CNN 420 to generate predictions. In some examples, the perception system 402 trains the CNN 420 to generate predictions based on training data associated with predictions provided to the CNN 420.

[0083] Now for reference Figure 4C and Figure 4D A diagram illustrating example operation of a CNN 440 utilizing a perception system 402. In some embodiments, CNN 440 (e.g., one or more components of CNN 440) and CNN 420 (e.g., one or more components of CNN 420) (see...) Figure 4B (Same or similar)

[0084] In step 450, the perception system 402 provides image-associated data as input to the CNN 440 (step 450). For example, as illustrated, the perception system 402 provides image-associated data to the CNN 440, wherein the image is a grayscale image represented as values ​​stored in a two-dimensional (2D) array. In some embodiments, the image-associated data may include data associated with a color image, which is represented as values ​​stored in a three-dimensional (3D) array. Additionally or alternatively, the image-associated data may include data associated with infrared images and / or radar images, etc. Furthermore, the image-associated data includes ambient lighting conditions and objects identified in the environment.

[0085] In step 455, CNN 440 executes a first convolution function. For example, CNN 440 executes the first convolution function based on the fact that CNN 440 provides values ​​representing the image as input to one or more neurons (not explicitly illustrated) included in the first convolutional layer 442. In this example, the values ​​representing the image may correspond to values ​​representing regions of the image (sometimes called receptive fields). In some embodiments, individual neurons are associated with filters (not explicitly illustrated). A filter (sometimes called a kernel) may be represented as an array of values ​​corresponding in size to the values ​​provided as input to the neuron. In one example, the filter may be configured to recognize edges (e.g., horizontal lines, vertical lines, and / or straight lines, etc.). In successive convolutional layers, the filters associated with neurons may be configured to successively recognize more complex patterns (e.g., arcs and / or objects, etc.).

[0086] In some embodiments, the CNN 440 performs a first convolution function by multiplying the values ​​of each neuron in one or more neurons included in the first convolutional layer 442, which are provided as input, with the values ​​of the filters corresponding to each of the neurons in the same or more neurons. For example, the CNN 440 may multiply the values ​​of each neuron in one or more neurons included in the first convolutional layer 442, which are provided as input, with the values ​​of the filters corresponding to each of the neurons in the same or more neurons to generate a single value or an array of values ​​as output. In some embodiments, the collective output of the neurons in the first convolutional layer 442 is referred to as the convolutional output. In some embodiments, when the neurons have the same filters, the convolutional output is referred to as a feature map.

[0087] In some embodiments, CNN 440 provides the outputs of each neuron in the first convolutional layer 442 to neurons in downstream layers. For clarity, an upstream layer may be a layer that transmits data to a different layer (referred to as a downstream layer). For example, CNN 440 may provide the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in a subsampling layer. In this example, CNN 440 provides the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in the first subsampling layer 444. In some embodiments, CNN 440 adds a bias value to the set of all values ​​provided to the neurons in the downstream layer. For example, CNN 440 adds a bias value to the set of all values ​​provided to the neurons in the first subsampling layer 444. In such an example, CNN 440 determines the final values ​​to be provided to the neurons in the first subsampling layer 444 based on the set of all values ​​provided to the neurons and the activation function associated with each neuron in the first subsampling layer 444.

[0088] In step 460, CNN 440 executes a first subsampling function. For example, CNN 440 may execute the first subsampling function based on the values ​​provided by CNN 440 from the output of the first convolutional layer 442 to the corresponding neurons of the first subsampling layer 444. In some embodiments, CNN 440 executes the first subsampling function based on an aggregation function. In one example, CNN 440 executes the first subsampling function based on determining the maximum input (called the max pooling function) among the values ​​provided to a given neuron. In another example, CNN 440 executes the first subsampling function based on determining the average input (called the average pooling function) among the values ​​provided to a given neuron. In some embodiments, based on the values ​​provided by CNN 440 to the individual neurons of the first subsampling layer 444, CNN 440 generates an output, which is sometimes referred to as the subsampling convolution output.

[0089] In step 465, CNN 440 executes a second convolution function. In some embodiments, CNN 440 executes the second convolution function in a manner similar to how CNN 440 executes the first convolution function described above. In some embodiments, CNN 440 executes the second convolution function based on CNN 440 providing the value output by the first subsampling layer 444 as input to one or more neurons (not explicitly illustrated) included in the second convolutional layer 446. In some embodiments, as described above, each neuron in the second convolutional layer 446 is associated with a filter. As described above, the filter (one or more) associated with the second convolutional layer 446 can be configured to recognize more complex patterns compared to the filter associated with the first convolutional layer 442.

[0090] In some embodiments, the CNN 440 performs a second convolution function by multiplying the values ​​of each neuron in one or more neurons included in the second convolutional layer 446 as input with the values ​​of the filters corresponding to each of those neurons. For example, the CNN 440 may multiply the values ​​of each neuron in one or more neurons included in the second convolutional layer 446 as input with the values ​​of the filters corresponding to those neurons to generate a single value or an array of values ​​as output.

[0091] In some embodiments, CNN 440 provides the outputs of each neuron in the second convolutional layer 446 to neurons in downstream layers. For example, CNN 440 may provide the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in the subsampling layer. In this example, CNN 440 provides the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in the second subsampling layer 448. In some embodiments, CNN 440 adds a bias value to the set of all values ​​provided to the neurons in the downstream layers. For example, CNN 440 adds a bias value to the set of all values ​​provided to the neurons in the second subsampling layer 448. In such an example, CNN 440 determines the final values ​​provided to the neurons in the second subsampling layer 448 based on the set of all values ​​provided to the neurons and the activation function associated with each neuron in the second subsampling layer 448.

[0092] In step 470, CNN 440 executes a second subsampling function. For example, CNN 440 may execute a second subsampling function based on the values ​​provided by the output of the second convolutional layer 446 to the corresponding neurons of the second subsampling layer 448. In some embodiments, CNN 440 executes a second subsampling function based on the use of an aggregation function. In the example, as described above, CNN 440 executes a first subsampling function based on determining the maximum or average input among the values ​​provided to a given neuron. In some embodiments, CNN 440 generates an output based on the values ​​provided to the individual neurons of the second subsampling layer 448.

[0093] In step 475, CNN 440 provides the outputs of each neuron in the second subsampling layer 448 to the fully connected layer 449. For example, CNN 440 provides the outputs of each neuron in the second subsampling layer 448 to the fully connected layer 449, causing the fully connected layer 449 to generate an output. In some embodiments, the fully connected layer 449 is configured to generate an output associated with a prediction (sometimes called a classification). The prediction may include indications of objects included in the image provided as input to CNN 440, such as objects and / or a group of objects. In some embodiments, the perception system 402 performs one or more operations and / or provides data associated with the prediction to different systems described herein.

[0094] Now for reference Figure 5 A diagram illustrating an implementation 500 of processing for a deep learning-based beam control system is shown. In some embodiments, implementation 500 includes a control system 504b and an illumination system 506. In some embodiments, illumination system 506 is connected to illumination system 210 ( Figure 2 (Same or similar)

[0095] In one embodiment, the control system 504b sends a control signal 520 to the lighting system 506, which causes at least one light-emitting device to modify the illumination intensity. The light-emitting device is, for example, a headlight, taillight, daytime running light, fog light, signal light, brake light, hazard warning light, driving light, or any combination thereof. In this embodiment, the illumination intensity is based at least in part on fused feature data representing real-time environmental conditions, including objects in the environment.

[0096] In this embodiment, the sensing system 402 of the autonomous vehicle ( Figure 4A The output of the system obtains the detected road objects in the surrounding environment (i.e., data including object type, location, size, heading orientation, etc.). On the other hand, the data is processed via the positioning system 406 and the database 410. Figure 4AThe system extracts corresponding map information associated with detected road objects (i.e., the distance between the detected road object location and the drivable road ahead, which is important for determining the high beam intensity). An object feature network is applied to learn features associated with the high beam intensity based on these detected road objects and their map information. An image feature network is used to extract image features from the sensor data to identify environmental conditions. Image features include, for example, ambient lighting conditions closely related to high beam intensity. A second portion of the sensor data is input to the image feature network, and the image feature network outputs an image feature vector that includes at least one lighting data associated with at least one pixel. The object feature vector and the image feature vector are fused into at least one fused feature vector via a feature fusion network, which takes the object feature vector and the image feature vector as input and outputs the fused feature vector. The fused feature vector is classified as being associated with either the high beam intensity or the low beam intensity of the light-emitting device. Control circuitry is configured to operate at least one light-emitting device based on the classification of the fused feature vector.

[0097] This technology includes systems and algorithms for providing illumination intensity control of a light-emitting device. In the example, the light-emitting device is a component of an autonomous vehicle. Typically, autonomous vehicles operate without human input. For example, the autonomous vehicle can use internal systems (such as...) Figure 2 An autonomous system 202 can be used to navigate roads. For example, the autonomous system 202 may include a lighting system 210 having one or more light-emitting devices.

[0098] This technology is generally applicable to any light-emitting device on an autonomous vehicle. Examples of light-emitting devices include headlights, taillights, daytime running lights, fog lights, signal lights, brake lights, hazard warning lights, driving lights, etc. For ease of description, the light-emitting device is described as a headlight. However, any light-emitting device that allows for the alteration of one or more conditions (e.g., illumination intensity) during operation can be used.

[0099] In typical vehicle operation, human drivers manually alternate between high beam and low beam illumination in response to various environmental conditions. For example, when driving at night, human drivers alternate between high beam and low beam illumination in response to changing lighting conditions encountered in the environment. This technology enables autonomous vehicles to determine the illumination intensity output by their light-emitting devices in real time while navigating roads, especially at night or in poor lighting conditions. This technology includes closed-loop, unlabeled illumination intensity control based on a convolutional neural network learned from manual driving data. In particular, this technology includes a deep neural network for beam illumination intensity control that predicts beam illumination intensity based on image data and perceived object detection. In this way, this technology avoids the manual labeling of ground-based data.

[0100] Figure 6 This is a diagram of a beam illumination intensity control system 600. System 600 includes a beam control network 602. The beam control network is, for example, a convolutional neural network (e.g., a neural network). Figure 4B (CNN 420). The input to the beam control network 602 includes data associated with one or more images 604 and data associated with objects (e.g., perception output data) 606. The beam control network 602 outputs a beam illumination intensity classification 608. Specifically, based on the input image 604 and the perception output data 606, the beam control network 602 classifies the environmental conditions represented by the image 604 and the perception output data 606 as either the high beam illumination intensity output by the vehicle's headlights or the low beam illumination intensity output by the vehicle's headlights.

[0101] When driving at night, human drivers alternate between low beam and high beam intensity of the vehicle's headlights to balance increased visibility along the direction of travel with avoiding interference with other road users. For example, manual driving data (e.g., data captured while a human driver is operating the vehicle) includes selecting low beam intensity when approaching another vehicle from behind, when encountering an oncoming vehicle along the road, and on well-lit roads. Additionally, manual driving data includes selecting high beam intensity when driving on poorly lit roads or when no other vehicles are being dazzled or interfered with by the high beam intensity. In this example, manual driving data is collected using human driving beam control signals as ground realities. This technology enables the automatic selection of illumination intensity based on image and object features detected in the environment.

[0102] exist Figure 6 In the example, image features are extracted from the front-facing camera image 604. Typically, image features encode lighting and environmental information. Figure 6In the example, object features are extracted from perception output data 606. Typically, object features encode road user information. In an embodiment, object features are detected in the environment within the illumination range of the luminous device. In the headlight example, object features are determined for perceived objects detected in the vehicle's direction of travel. In an embodiment, object features are associated with a map prior. The map prior of the autonomous system includes at least road network information used to extract the distance to a drivable road for each detected road object, given its location, size, and heading orientation. In the example, the object feature network takes the distance to the road object on the drivable road as input to learn features. The map prior provides additional information associated with the detected road objects. This additional information is used to determine the importance of the detected road objects to autonomous driving and beam control. For example, when determining the output high beam or low beam intensity, pedestrians or vehicles detected far from the driving path are less important than vehicles in the autonomous vehicle's driving path. While the map prior provides additional information, the object feature network is able to learn features related to beam control states from the detected road objects without map information. Typically, image features and object features are fused and fed into the beam control network.

[0103] Figure 7 This is a diagram of the beam control network 700. The beam control network consists of three components: an image feature network 710, an object feature network 720, and a feature fusion network 730. In an embodiment, the feature fusion network 730 enables the classification of fused features learned from input sensor data and perceived output as being associated with either high beam illumination intensity or low beam illumination intensity. This classification is a prediction of the beam control state of the light-emitting device.

[0104] exist Figure 7 In the example, the image feature network 710 is a CNN with multilayer perceptrons (e.g., Figure 4B (CNN420). As shown in the figure, the image feature network 710 includes ResNet-18 712. ResNet-18 712 is an 18-layer deep CNN (including the first convolutional layer and the final fully connected layer). ResNet-18 712 extracts image features from image data 704. Initially, as shown in the figure... Figure 8The initial dataset described is used to train ResNet-18 712. For ease of illustration, the beam control network 700 is shown as having an image feature network 710, an object feature network 720, and a feature fusion network 730 as separate component networks. In an embodiment, the complete beam control network 700 is executed in real time, with the image feature network 710 and the object feature network 720 operating in parallel, and the parallelized output being sent to the feature fusion network 730. Thus, ResNet-18 712 is applied to the sensor data in real time. The real-time execution of the beam control network 700 enables the autonomous vehicle to respond immediately (e.g., with minimal disruption, similar to human response speed) to changes in conditions and determine whether high beam or low beam illumination intensity is appropriate based on the input image from the camera system and the detected objects from the perception system. In an embodiment, the real-time performance enables the autonomous vehicle to respond rapidly by switching to the optimal beam control state without delay. Additionally, in an embodiment, the beam control network 700 improves the performance of the perception system in detecting road objects by providing illumination according to the beam control state.

[0105] Typically, ambient lighting conditions are derived from camera image data 704. In this embodiment, image data 704 includes data associated with ambient lighting. Comparing the ambient lighting-associated data at multiple timestamps provides an indication to the beam control network that lighting conditions have changed. Therefore, in this embodiment, image data 704 is input to ResNet-18 712, and changes in lighting conditions cause changes in the image features extracted by ResNet-18 712.

[0106] In this embodiment, the output of ResNet-18 712 is one or more image features in the form of a feature vector. For example, the image features output by ResNet-18 are 1024-dimensional (d) vectors. The large image feature vector output by ResNet-18 712 is input to a multilayer perceptron 714. Typically, the multilayer perceptron 714 is a feedforward network comprising one or more fully connected layers. In this example, the last two layers of ResNet-18 712 (including the fully connected layer and the softmax layer) are replaced by the multilayer perceptron 714. The input image is resized to 112×112 to learn features representing overall environmental conditions, particularly lighting conditions related to the intensity of far-field illumination. Therefore, the multilayer perceptron 714 reduces the dimensionality of the large-scale image feature vector (1024d) output by ResNet-18 712. In this example, dimensionality reduction extracts one or more important features from large-scale data in a high-dimensional space. The output of the multilayer perceptron 714 is an image feature vector 716 of size 128d.

[0107] exist Figure 7 In the example, the object feature network 720 includes a multilayer perceptron 722. In the embodiment, the multilayer perceptron 722 will receive features from... Figure 9 The perception system 900 described uses the perception output 706 as input. A multilayer perceptron 722 reduces the dimensionality of the object feature vector (256d) output from the perception output 706, where dimensionality reduction extracts one or more important features from the perception output 706. The output of the multilayer perceptron 722 is an object feature vector 724 of size 128d. Figure 7 In the example, perception and map data (e.g., perception output 706) are used to determine the proximity of the autonomous vehicle to other objects, including vehicles also traveling on the road. In the embodiment, the distance between the autonomous vehicle and another vehicle or object governs the appropriate illumination intensity output by the vehicle's light-emitting device.

[0108] At concatenator 707, image feature vector 716 and object feature vector 724 are concatenated to form a concatenated image-object feature vector of size 256d. The concatenated image-object feature vector is input into feature fusion network 730. At feature fusion network 730, the concatenated image-object feature vector is fused, and the dimensionality of the data is reduced to obtain classification 708 of environmental conditions associated with far-beam or near-beam illumination intensity. In this embodiment, a multilayer perceptron fuses and simultaneously reduces the dimensionality of the concatenated image-object feature vector. At concatenator (707), 128d image feature vector and 128d object feature vector are concatenated to obtain a 256d feature vector. At feature fusion network 730, the concatenated 256d feature vector is input into three multilayer perceptron layers with 256, 64, and 2 output channels, respectively. Therefore, in this embodiment, the output of feature fusion network 730 is a two-dimensional vector providing information on far / near light probabilities. A trained beam control network 700, comprising a trained image feature network 710, a trained object feature network 720, and a trained fusion feature network 730, is operable to classify sensor data into high beam illumination intensity or low beam illumination intensity. In an embodiment, the beam control network 700 is iteratively trained based on the availability of training data.

[0109] Figure 8 This is a block diagram of a system 800 used for training and deploying a beam control network 802. Figure 8In the example, the initial dataset is shown at reference numeral 810. A shadow mode deployment of the trained beam control network 802 is shown at reference numeral 820. The initial dataset shown at reference numeral 810 generates a training dataset for training the beam control network 802 during initial training 804. The shadow mode deployment at reference numeral 820 enables hard sample data mining and training 806. The initially trained beam control network 802 is iteratively trained using the hard sample data created during the shadow mode deployment at reference numeral 820. After one or more safety criteria 832 have been met, the beam control network 802 is made available for safe deployment 830.

[0110] In the example, the initial dataset includes image 812A, perception output 814A, and manual driving beam control data 816. In the embodiment, image 812A, perception output 814A, and manual driving beam control data 816 form a training dataset. The beam control network 802 is initially trained using the first training dataset including image 812A, perception output 814A, and manual driving beam control data 816. In the example, image 812A is a front-facing camera image captured along the vehicle's direction of travel. Typically, image data is captured by one or more camera sensors, and the camera data quantizes the illumination of the captured pixels. Typically, the illumination of the image data represents the luminous flux of visible light received from the environment per unit area. In the example, the magnitude of the illumination is characterized by illuminance values.

[0111] Typically, perception output 814A refers to the output of a perception system (e.g., perception system 402). Perception output 814A is as follows: Figure 7 The input to the described object feature network 720. In the example, the perceptual output 814A is about... Figure 7 The described perception output is 706. Additionally, in this example, the output of the perception system 814A is a list of detected objects, each associated with an object category and bounding box (including center location, size, and heading orientation). Perception output 814A is determined by a perception system that uses one or more raw sensor data (including camera, LiDAR, and radar) to detect objects. In some cases, different perception systems may use different sensor data and fuse data from different sensors at different stages. In this embodiment, the detectors (i.e., image object detector, LiDAR object detector, and radar object detector) apply a CNN to detect objects, where cross-sensor fusion occurs at different stages. For example, a later-stage fusion perception system fuses detected objects from different sensor pipelines. Some perception systems fuse image, LiDAR, and radar data at the detection network features.

[0112] Figure 9This is a block diagram of the perception system 900. The perception system includes raw sensor data inputs, including camera 202a, LiDAR 202b, and radar 202c sensors. Sensor outputs raw data, which is input to the corresponding detectors. Therefore, raw sensor data from camera 202a is input to image object detector 902. Raw sensor data from LiDAR 202b is input to LiDAR object detector 904, and raw sensor data from camera radar 202c is input to radar object detector 906. Detectors 902, 904, and 906 use their respective raw sensor data to detect objects. In some cases, objects detected by detectors 902, 904, and 906 are associated with intermediate features. The detected objects are fused at object fusion 908 to eliminate redundancy or false objects. Object fusion 908 outputs detected objects 910. Detected objects are identified by object type and one or more 3D bounding boxes. In this embodiment, intermediate features associated with objects detected by detectors 902, 904, and 906 are independently fused, as shown by dashed lines 912, 914, 916, and 918. In this example, LiDAR 202b is the primary sensor for object detection. Figure 9 In the example, the alternative sensing system is shown by dashed lines 912, 914, and 918. Dashed line 912 shows the fusion between features output by image object detector 902 and features output by LiDAR object detector 904. Similarly, dashed line 914 shows the fusion between features output by LiDAR object detector 904 and features output by radar object detector 906. Dashed lines 916 and 918 show the fusion of detected objects output by image object detector 902 and radar object detector 906 with objects detected by LiDAR object detector 904, respectively. The sensing output of the detected object 910 is used as the object feature network 706. Figure 7 Input of ).

[0113] exist Figure 9In the example, the detected object 910 is associated with one or more bounding boxes, the location of one or more bounding boxes, and the classification of the object associated with one or more bounding boxes. In the embodiment, classification is the probability that an object is classified as a particular object. Furthermore, each bounding box is defined by the size (w, l, h) associated with the corresponding object, the location (x, y, z) associated with the corresponding object, and the heading orientation associated with the corresponding object. In the example, the bounding box defines the front view and size of the object. The output of the perception model 900 is a fixed-length vector with data indicating the classification, location, ground view size, and heading orientation of one or more objects. In the embodiment, map data is used to obtain location data associated with the object. For example, the beam illumination intensity varies depending on whether the object is on or off the road based on the location data. Specifically, high beam illumination intensity is output when the autonomous vehicle is within a threshold distance of an off-road vehicle. Low beam illumination intensity is output when the autonomous vehicle is within a threshold distance of an on-road vehicle. Typically, the list of objects output by the perception model 900 is located within the threshold distance of the vehicle. Objects within a threshold distance of a vehicle can form a vehicle cluster. The threshold distance could be, for example, objects within two meters of a planned travel path. In the example, objects in the cluster are at a predetermined distance from the road curb or other edges.

[0114] Refer again Figure 8 The initial training dataset at reference numeral 810 also includes manual driving beam control data 816. In this embodiment, the manual beam control driving data 816 is data associated with the selection of lighting intensity during human driving. Figure 8 In the example, image 812A, perception output 814A, and human driving beam control data 816 were initially used to train the beam control network model 802.

[0115] Following initial training at reference numeral 804, the trained beam control network is deployed in a shadow mode at reference numeral 820. Typically, a shadow mode refers to the operation of the trained beam control network 802 while the vehicle is controlled by a human driver. Therefore, in this embodiment, the shadow mode simultaneously executes the trained beam control network and the human driver's operation of the vehicle. During this shadow mode, the output 822 of the trained beam control model is compared with manual driving data 824 generated by the human driver. Images 812B and perception outputs 814B are also captured. Instances where the trained beam control network output 822 conflicts with the manual driving data 824 are extracted and mined as hard sample data. In this embodiment, the hard sample data includes the conflicting trained beam control network output 822 and manual driving data 824, the corresponding image 812B, and the corresponding perception output 814B. In this example, the corresponding image 812B and perception output 814B are captured at the same timestamp of the conflicting trained beam control network output 822 and manual driving data 824. Additionally, in the example, corresponding images 812B and perception outputs 814B are captured within a predetermined time range of the timestamps of the conflicting trained beam control network output 822 and manual driving data 824.

[0116] The beam control network 802 is fine-tuned using incrementally collected hard sample data and an existing database. Specifically, the hard sample data is used to update one or more weights of the beam control network. In this example, fine-tuning refers to modifying the weights of the beam control network by retraining it on the hard sample data. In this example, the hard sample data is data with low frequency of occurrence or long-tail corner case data. During fine-tuning, some top layers of the frozen beam control network are unfrozen, and newly added classifier layers and the last layer of the beam control network are jointly trained. This allows for fine-tuning of higher-order feature representations in the beam control network to make them more relevant to beam control. In this embodiment, the weights are updated by inputting an image 812B and a perception output 814B corresponding to the conflict between the trained beam control network output 822 and human driving data 824 for training. Hard sample data mining and training 806 creates a closed-loop system with feedback that enables automatic and continuous system improvement as hard sample data becomes available. Typically, safety standards 832 are used to analyze the beam control network 802. Once one or more safety criteria 832 are met, the beam control network 802 is securely deployed in production mode at box 830.

[0117] Figure 10This is a block diagram of processing 1000 for a deep learning-based beam control system. At block 1002, object features are extracted from the output of the perception system. An object feature network is used to extract the object features. In this embodiment, the objects detected from the perception system (including the object type, location, size, and heading orientation of each object) are the input to the object feature network, and the object feature network outputs an object feature vector that implicitly represents the detected road objects in relation to beam control.

[0118] At box 1004, an image feature network is used to extract image features from sensor data to identify environmental conditions (e.g., lighting). In embodiments, image features include lighting information (brightness or lack thereof), environmental information (image information of objects), or any combination thereof. In embodiments, the output of the image feature network includes at least an image feature vector of lighting data associated with the sensor data. In an example, the output of the image feature network is a feature vector that implicitly characterizes ambient lighting information.

[0119] At box 1006, object features and image features are fused. In this embodiment, the fusion is feature-level data fusion, where feature vectors are concatenated. The concatenated 256d features are then fed into a feature fusion network comprising three multilayer perceptron layers with 256, 64, and 2 output channels, respectively. At box 1008, the fused features are classified as being associated with either high beam or low beam intensity of a light source (e.g., a headlight, or any other light used for visibility). The output of the last layer of the multilayer perceptron in the feature fusion network is a 2d vector containing the probability that the light source's output is either high beam or low beam intensity. In this embodiment, the high / low beam classification is simply applied by examining which of two values ​​in the 2d output of the last multilayer perceptron layer of the feature fusion network is larger. At box 1010, the light source is operated based on the classification of the fused features.

[0120] The beam control network according to this technology is a lightweight learning architecture for automated beam control. The ability to autonomously control beam illumination intensity increases the operational domain of autonomous vehicles. Furthermore, the closed-loop technique according to this technology enables continuous improvement of the deep learning beam control network without relying on manual annotations as input to the model. This makes it possible to develop deep learning beam control networks faster and cheaper. The process ensures no necessary safety risks are incurred and only rigorously tested software is deployed. Typically, the beam control network is an optical network that can run in real-time on top of existing AV systems. In embodiments, object features and image features can be used to train similar networks for AV systems associated with human driving expertise data during vehicle operation. In the example, human driving expertise data is automatically captured during human operation of the vehicle. For example, object features and image features can be used to train automatic horn control networks and automatic emergency braking control networks.

[0121] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and what the applicant expects to be the scope of this invention, is the literal and equivalent scope of the claims published from this application in the specific form of the published claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as such terms are used in the claims. Furthermore, when the term “comprising” is used in the preceding specification or appended claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A system for beam control, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the following operations: An object feature network is used to extract object features from the output of a perception system to identify objects in the environment. The object feature network output is associated with object features from a map previously used to extract distances associated with the objects, where the distances govern the illumination intensity output by the light-emitting device. An image feature network is used to extract image features from sensor data to identify ambient lighting information. The image feature network outputs image features that include environmental information and data associated with the lighting in that environment. A feature fusion network is used to fuse the object features and the image features into a fused feature. The feature fusion network takes the object features and the image features as input and outputs the fused feature. The beam control state is predicted based on the fusion features, wherein the beam control state indicates the high beam illumination intensity or low beam illumination intensity of the light-emitting device; and A control circuit communicatively coupled to the at least one processor, wherein the control circuit is configured to operate the light-emitting device based on the beam control state.

2. The system according to claim 1, wherein, The at least one processor fuses the object features and the image features by applying a multilayer perceptron to the linked object features and image features.

3. The system according to claim 1 or 2, wherein, The object feature network is trained using data associated with manual driving and at least one corresponding output generated by the perception system.

4. The system according to claim 1 or 2, wherein, The image feature network is trained using data associated with manual driving and at least one corresponding image sensor output generated by an image sensor.

5. The system according to claim 1 or 2, wherein, The object feature network, the image feature network, and the feature fusion network are retrained during shadow mode to meet a predetermined security confidence level.

6. The system according to claim 1 or 2, wherein, The instructions stored in the memory also cause the at least one processor to perform the following operations: The mismatch between the predicted beam control state of the fused features and the corresponding manual driving data samples was determined. In response to a mismatch between the predicted beam control state and the corresponding manual driving data sample, the fused feature is identified as a conflicting sample; and The object feature network, the image feature network, the fusion feature network, or any combination of the three can be fine-tuned based on the conflict samples.

7. A method for beam control, comprising: Using at least one processor, an object feature network is used to extract object features from the output of a perception system to identify objects in the environment, wherein the output of the object feature network is associated with object features from a map previously used to extract distances associated with the objects, wherein the distances govern the illumination intensity output by a light-emitting device. Using the at least one processor, an image feature network is used to extract image features from sensor data to identify ambient lighting information, wherein the image feature network outputs image features including environmental information and data associated with the lighting of the environment; Using the at least one processor, a feature fusion network is used to fuse the object features and the image features into a fused feature, wherein the feature fusion network takes the object features and the image features as input and outputs the fused feature; Using the at least one processor, a beam control state is predicted based on the fusion features, wherein the beam control state indicates the high beam illumination intensity or low beam illumination intensity of the light-emitting device; and The light-emitting device is operated using the at least one processor based on the beam control state.

8. The method according to claim 7, wherein, Fusing the object features and the image features into fused features includes applying a multilayer perceptron to the connected object features and image features.

9. The method according to claim 7 or 8, wherein, The object feature network is trained using data associated with manual driving and at least one corresponding output generated by the perception system.

10. The method according to claim 7 or 8, wherein, The image feature network is trained using data associated with manual driving and at least one corresponding image sensor output generated by an image sensor.

11. The method according to claim 7 or 8, wherein, The object feature network, the image feature network, and the feature fusion network are retrained during shadow mode to meet a predetermined security confidence level.

12. The method according to claim 7 or 8, further comprising: Using the at least one processor, a mismatch is determined between the predicted beam control state of the fused features and the corresponding manual driving data sample; Using the at least one processor, in response to a mismatch between the predicted beam control state of the fused feature and the corresponding manual driving data sample, the fused feature is identified as a conflicting sample; Using the at least one processor, the object feature network, the image feature network, the fusion feature network, or any combination of the three are fine-tuned based on the conflict samples.

13. A non-transitory computer program product storing instructions that, when executed by at least one programmable processor of a vehicle, cause the at least one programmable processor to operate, the operation comprising: Object features are extracted from the output of a perception system using an object feature network to identify objects in the environment, wherein the output of the object feature network is associated with object features from a map previously used to extract distances associated with the objects, wherein the distances govern the illumination intensity output by the light-emitting device. An image feature network is used to extract image features from sensor data to identify ambient lighting information, wherein the image feature network outputs image features that include environmental information and data associated with the lighting of the environment; The object features and the image features are fused into a fused feature using a feature fusion network, wherein the feature fusion network takes the object features and the image features as input and outputs the fused feature; The beam control state is predicted based on the fusion features, wherein the beam control state indicates the high beam illumination intensity or low beam illumination intensity of the light-emitting device; and The light-emitting device is operated based on the beam control state.

14. The computer program product according to claim 13, wherein, Fusing the object features and the image features into fused features includes applying a multilayer perceptron to the connected object features and image features.

15. The computer program product according to claim 13 or 14, wherein, The object feature network is trained using data associated with manual driving and at least one corresponding output generated by the perception system.

16. The computer program product according to claim 13 or 14, wherein, The image feature network is trained using data associated with manual driving and at least one corresponding image sensor output generated by an image sensor.

17. The computer program product according to claim 13 or 14, wherein, The object feature network, the image feature network, and the feature fusion network are retrained during shadow mode to meet a predetermined security confidence level.

18. The computer program product according to claim 13 or 14, wherein the operation further comprises: Using the processor, a mismatch is determined between the predicted beam control state of the fused features and the corresponding manual driving data sample; Using the processor, in response to a mismatch between the predicted beam control state of the fused feature and the corresponding manual driving data sample, the fused feature is identified as a conflicting sample; Using the processor, the object feature network, the image feature network, the fusion feature network, or any combination of the three can be fine-tuned based on the conflict samples.

Images