Color filter array for vehicle image sensor

Abstract

In an example implementation, a method includes a color filter array including a plurality of cells arranged according to a spatially repeating pattern. Each of the cells includes a color filter element having a first area and an optically transparent element having a second area, where the first plane is smaller than the second area. The color filter array is configured to receive light from an environment of the device, filter at least a portion of the light using the unit, and transmit at least a portion of the filtered light to an image sensor.

Description

Background Technology

[0001] Image sensors (such as cameras) detect and transmit information for forming an image. As an example, an image sensor includes one or more photodiodes that are sensitive to light. The image sensor receives light from its environment (e.g., light reflected from one or more objects in the environment), directs the light to the photodiode, and measures the electrical signal output by the photodiode in response to the light. The image sensor generates one or more spatially resolved images based on the electrical signals.

[0002] In some implementations, a vehicle (e.g., an autonomous vehicle) may include one or more image sensors for detecting objects in its environment, such as other vehicles, obstacles, pedestrians, etc. Attached Figure Description

[0003] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system;

[0004] Figure 2 It is a diagram of one or more systems that include autonomous vehicles;

[0005] Figure 3 yes Figure 1 and Figure 2 A diagram of one or more devices and / or one or more system components;

[0006] Figure 4 It is a diagram of some components of an autonomous system;

[0007] Figure 5 This is a diagram of an example image sensor;

[0008] Figure 6 These are diagrams of an example color filter array and an example photodiode array; and

[0009] Figure 7 This is a diagram of example pixel resources for a photodiode. Detailed Implementation

[0010] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0011] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0012] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0013] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0014] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0015] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet (e.g., a data packet) that includes data.

[0016] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0017] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0018] General Overview

[0019] In some aspects and / or embodiments, the systems and devices described herein include and / or implement color filter arrays for vehicle image sensors, such as those used in autonomous vehicles.

[0020] In the example implementation, the image sensor includes a photodiode array and a color filter array. During example operation, the image sensor receives light from the environment, directs the light through the color filter array and toward the photodiode array, and detects the filtered light incident on the photodiode array. The image sensor generates an image based on the detected light.

[0021] The photodiode array comprises several units arranged in a spatially repeating pattern. Each unit includes a pair of photodiodes, one of which is larger than the other (and therefore more sensitive to light). For each pair of photodiodes, the image sensor combines the outputs from the photodiodes differently to generate outputs representing different light dynamic ranges.

[0022] In addition, the color filter array also includes several units arranged in a spatial repeating pattern (e.g., grid, matrix, or mosaic).

[0023] Each unit of the color filter array is configured to filter light from the environment and direct the filtered light to a corresponding unit of the photodiode array (e.g., such that the unit of the photodiode array detects light having one or more specific colors). Furthermore, each unit includes a pair of filter elements, one of which is larger than the other. For each unit, the larger filter element is an optically transparent element (e.g., configured to allow light of substantially all wavelengths to pass through), and the smaller filter element is a color filter element (e.g., configured to allow light of a specific color, such as red, green, or blue, to pass through).

[0024] In at least some embodiments, the color filter array described herein enables an image sensor (e.g., compared to an image sensor without the color filter array described herein) to detect light with higher sensitivity while maintaining the ability to distinguish different colors of light.

[0025] For example, in a color filter array, transparent elements allow virtually all light from the environment to pass through to the corresponding larger photodiode in the photodiode array. This enables an image sensor (e.g., compared to an image sensor that filters light according to a specific color and / or wavelength and then directs the filtered light to a larger photodiode, potentially attenuating the intensity of that light) to detect light monochromaticly with greater sensitivity. This can be particularly suitable for detecting environmental features under low-light conditions.

[0026] Furthermore, in the color filter array, the color filters direct light of a specific color from the environment to the corresponding smaller photodiodes in the photodiode array. This enables the image sensor to distinguish between different colors of light.

[0027] At least some of the implementations described herein are particularly advantageous for use in vehicle image sensors. For example, the implementation of the color filter array described herein enables vehicle image sensors to detect objects in the environment with high sensitivity while maintaining the ability to distinguish different colors of light (e.g., different colors of light emitted by traffic lights). As a result, vehicles (e.g., autonomous vehicles) can sense their environment more accurately and operate more safely under various lighting conditions.

[0028] Now for reference Figure 1Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0029] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2 The vehicles 200 in the collection of vehicles 200 are similar or identical. In some embodiments, vehicles 200 in the collection of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0030] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0031] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with (e.g., defining) a series of actions (also referred to as trajectories) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will board the AV, and a second state or area includes a location where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Additionally or alternatively, route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within a lane area and target rates at those locations. In the example, route 106 includes multiple precise state sequences along at least one high-level action with a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to multiple trajectories that collectively form a high-level route terminating at a final target state or region.

[0032] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In examples, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driving lane, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In examples, a road includes at least one lane associated with at least one lane marking line (e.g., identified based on at least one lane marking line).

[0033] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Infrastructure or Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with the vehicle 102 and / or the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, the queue management system 116, and / or the V2I system 118 via a network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, signage, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), lane markings, streetlights, parking meters, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the queue management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via a network 112.

[0034] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0035] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0036] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including and / or not including autonomous systems)).

[0037] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0038] supply Figure 1 The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements arranged differently. Additionally or alternatively, at least one element of environment 100 may be described as being composed of… Figure 1 One or more functions performed by at least one different element of environment 100. Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0039] Now for reference Figure 2 Vehicle 200 (which can be connected with) Figure 1 The vehicle 200 (which is the same as or similar to vehicle 102) includes, or is associated with, an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 is associated with, or is similar to, vehicle 102 (see...). Figure 1 The autonomous system 202 is configured to endow the vehicle 200 with autonomous driving capabilities (e.g., implementing at least one driving automation or maneuver-based function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention, such as Level 5 ADS-operated vehicles), highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations, such as Level 4 ADS-operated vehicles), and / or conditionally autonomous vehicles (e.g., vehicles that abandon human intervention in limited situations, such as Level 3 ADS-operated vehicles)). In one embodiment, the autonomous system 202 includes some or all of the operational or tactical functionality required for the vehicle 200 to operate in road traffic and continuously perform dynamic driving tasks (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) incorporating driver support features. The autonomous system 202 supports various levels of driver automation, ranging from no driver automation (e.g., Level 0) to full driver automation (e.g., Level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0040] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0041] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision) purposes. In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0042] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD (Traffic Light Detection) data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other systems incorporated into cameras described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., wide-angle lens, fisheye lens, and / or lenses with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0043] The Light Detection and Ranging (LiDAR) sensor 202b includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object within the field of view of the LiDAR sensor 202b.

[0044] The radio detection and ranging (RADR) sensor 202c includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). Radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by Radar sensor 202c encounter a physical object and are reflected back to Radar sensor 202c. In some embodiments, the radio waves emitted by Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with Radar sensor 202c generates a signal representing objects included in the field of view of Radar sensor 202c. For example, at least one data processing system associated with Radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of Radar sensor 202c.

[0045] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0046] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, an autonomous vehicle computing unit 202f, a safety controller 202g, and / or a DBW (drive-by-wire) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0047] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1 V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1The V2I system 118 communicates with the same or similar V2I system.

[0048] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0049] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200. Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0050] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle movement (such as starting to move forward, stopping to move forward, starting to move backward, stopping to move backward, accelerating in a certain direction, decelerating in a certain direction, etc.) or lateral vehicle movement (such as making a left turn and / or making a right turn, etc.). In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0051] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to rotate left or right, thereby causing the vehicle 200 to turn left or right. In other words, the steering control system 206 causes the activity required to regulate the y-axis component of the vehicle's motion.

[0052] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0053] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), wheel rate sensors, wheel brake pressure sensors, wheel torque sensors, engine torque sensors, and / or steering angle sensors. Although the braking system 208 is illustrated as being located at... Figure 2 The braking system 208 is located near the vehicle 200, but it can be located anywhere within the vehicle 200.

[0054] Now for reference Figure 3The diagram illustrates a device 300. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to at least one device of vehicle 102 and / or vehicle 200 (e.g., at least one device of the system of vehicle 102), and / or one or more devices of network 112 (e.g., one or more devices of the system of network 112). In some embodiments, one or more devices of vehicle 102 and / or vehicle 200 (e.g., one or more devices of vehicle 102 and / or vehicle 200 systems (such as autonomous system 202, etc.), remote AV system 114, queue management system 116, V2I system 118, and / or one or more devices of network 112 (e.g., one or more devices of network 112 systems) include at least one device 300 and / or at least one component of device 300.

[0055] like Figure 3 As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0056] Bus 302 includes components for communication between the components of the licensed device 300. In some cases, processor 304 includes a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or accelerated processing unit (APU), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC)). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores data and / or instructions for use by processor 304.

[0057] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0058] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0059] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, or a Wi-Fi interface. ® Interfaces and / or cellular network interfaces, etc.

[0060] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 306 and / or storage component 308, executed by processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0061] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. When executed, the software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0062] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0063] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the other device, which, when executed by the processor of processor 304 and / or the processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0064] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3 Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, the set of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another set of components of device 300.

[0065] Now for reference Figure 4The diagram illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0066] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0067] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the perception system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to the tactical functions required to operate the vehicle 102 in road traffic. Tactical efforts involve maneuvering the vehicle in traffic during a journey, including but not limited to deciding whether and when to overtake another vehicle, change lanes, or select appropriate speed, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0068] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various other types of traffic signals). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0069] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0070] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate a powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), a steering control system (e.g., steering control system 206), and / or a braking system (e.g., braking system 208). For example, the control system 408 is configured to perform operational functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes activities required to regulate the y-axis component of the vehicle's motion. Longitudinal vehicle motion control causes activities required to regulate the x-axis component of the vehicle's motion. In an example, where the trajectory includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers).

[0071] In some embodiments, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.). The following is about Figure 6 A to Figure 6 C includes examples of implementations of machine learning models.

[0072] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3(The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0073] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0074] Example color filter array

[0075] Figure 5 Aspects of an image sensor system 500 are shown. In some implementations, the image sensor system 500 is (e.g., as referenced) Figure 2 One or more of the cameras 202a, or otherwise included (e.g., as referenced) Figure 2 One or more of the cameras 202a described above. In some implementations, an image sensor system is included. Figure 2 200 vehicles Figure 3 Device 300 or Figure 4 The calculation of autonomous vehicles is in 400, or with Figure 2 200 vehicles Figure 3 Device 300 or Figure 4 The autonomous vehicle calculation is related to 40.

[0076] like Figure 5As shown, the image sensor system 500 includes a photodiode array 502 and a color filter array 504 (e.g., to enable the photodiode array 502 to detect light of a specific color or wavelength at specific spatial locations along the photodiode array). During example operation, the image sensor system 500 receives light 550 from an environment 552 (e.g., light reflected and / or scattered from an object 554 in the environment 552), directs the light 550 through the color filter array 504 and toward the photodiode array 502, and detects the filtered light incident on the photodiode array 502. Using the photodiode array 502, the image sensor system 500 generates an electrical signal 556 representing one or more spatially resolved images based on the detected light. For example, one or more images may visually depict the object 554 and / or the environment 552.

[0077] Typically, the photodiode array 502 includes a plurality of photosensitive photodiodes 508a-508n. Each photodiode 508a-508n receives light (e.g., light that has been filtered by the color filter array 504) and generates an electrical signal based on the received light. As an example, each photodiode 508a-508n may be a photosensitive semiconductor diode (e.g., a PIN diode) that generates a current in response to incident photons on a photosensitive region (e.g., on or near the depletion region of the PIN diode).

[0078] Typically, the color filter array 504 includes one or more filter elements 506a-506n to filter light 550 (e.g., by attenuating light having (one or more) a specific color and / or (one or more) a specific wavelength, and / or by allowing light having (one or more) a specific color and / or (one or more) a specific wavelength to pass through).

[0079] In some implementations, at least some of the filter elements 506a-506n may be bandpass filters (e.g., filters that allow light of a specific wavelength range or wavelength within the passband to pass through). As an example, some of the filter elements 506a-506n are bandpass filters configured to allow light of a specific color (e.g., light of a specific wavelength or wavelength range within the passband) to pass through, and to attenuate or block light of other colors (e.g., light of wavelengths outside the passband).

[0080] In some implementations, the color filter array 504 and / or filter elements 506a-506n may include one or more absorption filters, interference filters, and / or dichroic filters. Furthermore, the color filter array 504 and / or filter elements 506a-506n may be at least partially constructed of optically transmitting materials such as glass, plastics, colored dyes, etc.

[0081] In some implementations, the color filter array 504 includes filter elements 506a-506n for selectively transmitting different (one or more) colors of light along different corresponding regions of the color filter array 504. For example, some filter elements 506a-506n may be configured to transmit a first color of light (e.g., red), some other filter elements 506a-506n may be configured to transmit a second color of light (e.g., green), some other filter elements 506a-506n may be configured to transmit a third color of light (e.g., blue), and so on. Thus, different colors of light are incident on different corresponding regions of the photodiode array 502. This can, for example, be advantageous in enabling the photodiode array 502 to detect light of a specific color at a specific region of the photodiode array 502 (e.g., to generate an image with spatially resolved color information).

[0082] In some implementations, filter elements 506a-506n configured to transmit red light can transmit light with wavelengths in the range of 580 nm to 800 nm, and attenuate or block light with wavelengths outside this range. In some implementations, filter elements 506a-506n configured to transmit green light can transmit light with wavelengths in the range of 480 nm to 620 nm, and attenuate or block light with wavelengths outside this range. In some implementations, filter elements 506a-506n configured to transmit blue light can transmit light with wavelengths in the range of 380 nm to 520 nm, and attenuate or block light with wavelengths outside this range.

[0083] Furthermore, in some implementations, the color filter array 504 includes at least some optically transparent elements. As an example, the optically transparent elements may non-selectively attenuate light of any wavelength and / or may transmit light of all (or substantially all) wavelengths. As another example, the optically transparent elements may allow light of all wavelengths within a specific wide range (e.g., the visible light range) to pass through. As yet another example, the optically transparent elements may allow light of various colors (e.g., red, green, blue, etc.) that are individually passed through any one of the filter elements 506a-506n to pass through.

[0084] In some implementations, the optically transparent element itself is configured to transmit light in the wavelength range from 380 nm to infrared light (~1 mm).

[0085] Optically transparent elements can be advantageous, for example, by enabling the photodiode array 502 to detect light with greater sensitivity in specific regions of the photodiode array 502 (e.g., by not attenuating light transmitted to the photodiode array 502 along specific regions of the photodiode array 502). In some implementations, one or more optically transparent elements may be arranged with one or more filter elements 506a-506n in a pattern (e.g., a repeating pattern such as a grid, matrix, or mosaic). In some implementations, the pattern is symmetrical about at least one axis. Additionally, in some embodiments, the optically transparent elements are arranged with one or more filter elements 506a-506n in a pattern corresponding to at least one traffic signal, wherein the pattern enables the generation of an image including at least one traffic signal with spatially resolved color information.

[0086] Figure 6 As shown, from the optical axis parallel to the image sensor system 500 (e.g., reference...) Figure 5 An example arrangement of the photodiode array 502 and the color filter array 504 as seen in a direction extending from object 554 along the axis of photodiode array 502. In this example, the color filter array 504 is depicted as covering the photodiode array 502, wherein the color filter array 504 is shown in shade and the photodiode array 502 is shown in outline.

[0087] In this example, the photodiode array 502 includes several cells arranged in a spatially repeating planar pattern (e.g., a grid, matrix, or mosaic). Each cell includes a pair of photodiodes 602a and 602b, wherein photodiode 602a is larger than photodiode 602b (and therefore more sensitive to light). As an example, each photodiode 602a may have a larger area (e.g., planar area) than each photodiode 602b.

[0088] although Figure 6 An arrangement with 4×4 cells is shown, but in practice, the photodiode array 502 is configured such that it has any number of repeating cells in each dimension (e.g., an arrangement with N×M cells).

[0089] For each pair of photodiodes 602a and 602b, the image sensor system 500 (e.g., according to different weights) combines the outputs from the photodiodes differently to generate outputs representing different light dynamic ranges. In some implementations, this may be referred to as a “split pixel” configuration (e.g., where each pair of photodiodes 602a and 602b represents a single pixel in the image).

[0090] For example, photodiodes 602a and 602b can each output a corresponding signal representing the intensity of light detected by the photodiode. The image sensor system 500 can apply different gains or weights to these signals and combine them to form a composite signal. The gain or weights can be modified to obtain composite signals representing different light dynamic ranges.

[0091] As a simplified example, Figure 7 The pixel response (702c) of the smaller photodiode 602b when high gain is applied to the output signal is shown, (ii) the pixel response (702d) of the smaller photodiode 602b when low gain is applied to the output signal, (iii) the pixel response (702a) of the larger photodiode 602a when high gain is applied to the output signal, and (ii) the pixel response (702b) of the larger photodiode 602a when low gain is applied to the output signal. These signal responses 702a-702d each correspond to a corresponding dynamic range 704a-704d. Output 704f is a combined high dynamic range (HDR) output corresponding to pixel responses 702a to 702d and can be provided as a final output signal for pixels (e.g., for a pair of photodiodes 602a and 602b). The image sensor system 500 can generate different combinations of these weighted signals to obtain information about a wider dynamic range 704e. In some implementations, this technique can be achieved by increasing the effective bit depth of the signal output by the image sensor system 500 (e.g., from a 12-bit signal range to a 24-bit signal range). In some implementations, this technique is used to generate high dynamic range (HDR) images.

[0092] like Figure 6 As shown, the color filter array 504 also includes several units arranged in a spatially repeating planar pattern (e.g., grid, matrix, or mosaic).

[0093] Each unit of the color filter array 502 is configured to filter light from the environment and direct the filtered light to a corresponding unit of the photodiode array (e.g., such that the unit of the photodiode array detects light having one or more specific colors).

[0094] Furthermore, each unit includes a pair of optical elements 604a and 604b, wherein the first optical element 604a is larger than the other optical element 604b. As an example, each optical element 604a may have a larger area (e.g., planar area) than each optical element 604b.

[0095] For each unit, the first optical element 604a is an optically transparent element (e.g., configured to allow light of substantially all wavelengths to pass through), and the second optical element 604b is a color filter element (e.g., configured to allow light of a specific color, such as red, green, or blue, to pass through). Although Figure 6 An arrangement with 4×4 cells is shown, but in practice, the color filter array 504 is configured such that it has any number of repeating cells in each dimension (e.g., an arrangement with N×M cells).

[0096] Generally, the repeating units of the color filter array 504 are aligned with the corresponding units of the photodiode array 502, so that the units of the color filter array 504 selectively filter light and provide light to the corresponding units of the photodiode array 502.

[0097] For example, each of the optical elements 604a is aligned with a corresponding one of the photodiodes 602a. The optical element 604a receives light (e.g., from the environment) and transmits at least a portion of that light to the corresponding photodiode 602a. Since the optical element 604a is an optically transparent element, the light supplied to the photodiode 602a is substantially unfiltered and / or unattenuated.

[0098] As another example, each of the optical elements 604b is aligned with a corresponding one of the photodiodes 602b. The optical element 604b receives light (e.g., from the environment), filters the light, and transmits at least a portion of the filtered light to the corresponding photodiode 602b. Because the optical element 604b is a color filter element, the light supplied to the photodiode 602b has been filtered to include certain colors or wavelengths, while other colors or wavelengths are attenuated or blocked.

[0099] In some implementations, the cells of the color filter array 504 are arranged according to a specific color pattern. As an example, in some implementations, the cells of the color filter array are arranged in groups of 2×2 cells, where (e.g., as...) Figure 6 As shown):

[0100] Two of the smaller optical elements 604b in group (i) are green filters.

[0101] One of the smaller optical elements 604b in group (ii) is a red filter.

[0102] One of the smaller optical elements 604b in group (iii) is a blue filter, and

[0103] (iv) The four larger optical elements 604a in the group are each optically transparent elements.

[0104] Furthermore, for each group, the green filters can be diagonally opposite each other, and the red and blue filters can also be diagonally opposite each other. In some implementations, this arrangement can be referred to as a monochromatic and RGGB (e.g., Bayer) filter arrangement.

[0105] Each group represents a single pixel in the image. In some implementations, the image sensor system 500 determines the intensity and / or color of a pixel by interpolating the signals output by the photodiodes 602a and 602b in each group. For example, color interpolation is performed to reconstruct a color image by sampling color information from neighboring pixels with different colors.

[0106] In some implementations, monochrome and RGGB arrangements can be particularly suitable for vehicle image sensors. For example, compared to other types of filter arrangements (e.g., RCCG, RYYCy), monochrome and RGGB arrangements enable image sensors to generate images with high color accuracy. Therefore, colors can be better distinguished from each other in the image. This color filter array can be beneficial in the context of autonomous vehicles because it enables the autonomous vehicle to distinguish different colors of light, which may have different meanings and may lead to different operational outcomes for the autonomous vehicle. For example, this color filter array enables the autonomous vehicle to better distinguish between red, yellow, and green lights in a traffic light, allowing it to cross intersections more safely. However, in some implementations, the color filter array can have an RCCG arrangement, an RYYCy arrangement, or any other color filter arrangement.

[0107] In some implementations, (e.g., relative to a planar surface) one or more photodiodes 602a may have an octagonal shape, and one or more photodiodes 602b may have a quadrilateral shape. In some implementations, (e.g., relative to a planar surface) one or more optical elements 604a may have an octagonal shape, and one or more optical elements 604b may have a quadrilateral shape.

[0108] In some implementations, (e.g., relative to a planar surface) photodiode 602a and optical element 604a may each have similar size and shape. In some implementations, (e.g., relative to a planar surface) photodiode 602b and optical element 604b may each have similar size and shape.

[0109] The color filter arrays described herein enable image sensors (e.g., compared to image sensors without the color filter arrays described herein) to detect light with higher sensitivity while maintaining the ability to distinguish different colors of light. The color filter arrays described herein enable image sensors (e.g., compared to image sensors without the color filter arrays described herein) to detect light with higher spatial resolution and higher color accuracy.

[0110] For example, in the color filter array 504, the transparent optical element 604a allows substantially all light from the environment to pass onto the corresponding larger photodiode 602a of the photodiode array 502. This enables the image sensor system 500 (e.g., compared to an image sensor that filters light according to a specific color and / or wavelength and then directs the filtered light to the larger photodiode 602a, which may attenuate the intensity of the light) to detect light monochromaticly with greater sensitivity. This may be particularly suitable for detecting environmental features under low-light conditions.

[0111] Furthermore, in the color filter array 504, the color filter optics 604b transmit light of a specific color from the environment to the corresponding smaller photodiode 602b of the photodiode array 502. This enables the image sensor system 500 to distinguish different colors of light, especially high-intensity light, which can be more easily detected by the smaller photodiode 602b of the photodiode array 502.

[0112] At least some of the implementations described herein are particularly advantageous in vehicle image sensors (e.g., reference ). Figure 2 The image sensor described herein is used in the camera 202a). For example, the implementation of the color filter array 504 enables the vehicle image sensor to detect objects in the environment with high sensitivity and high resolution, while maintaining the ability to distinguish different colors of light. In particular, the implementation of the color filter array 504 can be adapted to distinguish between yellow and red light emitted by traffic lights in low light and daytime conditions. Therefore, vehicles (e.g., autonomous vehicles) can sense their environment more accurately and operate more safely under various operating conditions.

[0113] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and the applicant's expectation that the scope of this invention is defined in the specific form of the claims published in this application, is the literal and equivalent scope of the claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as used in the claims. Furthermore, when the term "comprising" is used in the preceding specification or appended claims, the following phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method comprising: A color filter array comprising a plurality of units arranged according to a spatial repeating pattern, wherein each unit comprises: A color filter element having a first area, and An optically transparent element having a second area, wherein the first plane is smaller than the second area. The color filter array is configured to receive ambient light from the device, filter at least a portion of the light using the units, and transmit at least a portion of the filtered light to an image sensor.

2. The device according to claim 1, wherein, For each of the units, the color filter element is configured to filter incident light according to the corresponding color.

3. The device according to claim 1, wherein, For each of the units, the color filter element is configured to filter incident light according to one of the following colors: red, Blue, and green.

4. The device according to claim 1, wherein, For each of the units, the color filter element is adjacent to the optically transparent element.

5. The device according to claim 1, wherein, According to the spatial repeating pattern, the cells of the color filter array are arranged in multiple groups having a width of two cells and a height of two cells.

6. The device according to claim 1, wherein, Each group in the group includes: Two color filter elements configured to filter light based on green. A color filter element configured to filter light based on blue, and A color filter element configured to filter light based on red.

7. The device according to claim 1, wherein, At least some of the color filter elements have a quadrilateral shape.

8. The device according to claim 1, wherein, At least some of the optically transparent elements have an octagonal shape.

9. The device according to claim 1, further comprising the image sensor.

10. The device according to claim 9, wherein, The image sensor includes: A photodiode array comprising a plurality of second units arranged according to a second spatial repeating pattern, wherein each of the second units comprises: A first photodiode having a third area, and A second photodiode having a fourth area, wherein the third area is smaller than the fourth area.

11. The device according to claim 10, wherein, Each of the color filter elements is configured to filter incident light according to a corresponding color and to transmit at least a portion of the filtered light to a corresponding first photodiode in the first photodiode.

12. The device according to claim 11, wherein, Each of the optically transparent elements is configured to receive incident light and transmit at least a portion of the light to a corresponding second photodiode in the second photodiode.

13. The device according to claim 10, wherein, Each color filter element in the color filter element is aligned with a corresponding first photodiode in the first photodiode, and Each of the optically transparent elements is aligned with a corresponding second photodiode in the second photodiode.

14. The device according to claim 10, wherein, For each of the second units in the second unit, the image sensor is configured as follows: The first photodiode is used to obtain the first optical measurement result. The second photodiode is used to obtain the second optical measurement result, and The output value is generated based on the combination of the first optical measurement result and the second optical measurement result.

15. The device according to claim 1, wherein, The color filter array further includes an optical substrate, wherein each color filter element and each optically transparent element of the optically transparent elements are arranged on the optical substrate.

Images