Integrated optical design for visual and thermal imaging fusion

Abstract

An imaging system and method for generating a composite image by fusing two images generated by light beams having different spectral distributions is disclosed. The imaging system generates images of a single scene on two image sensors having different spectral responses through different but partially overlapping optical paths, and uses the resulting image signals to generate a composite image that captures features imaged based on visible light and thermal radiation.

Description

[0001] This application claims the priority benefit of U.S. Provisional Patent Application No. 63 / 626713, filed January 30, 2024, entitled INTEGRATED OPTICS DESIGN FORVISION AND THERMAL IMAGING FUSION, the contents of which are incorporated herein by reference in their entirety. Attached Figure Description

[0002] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system.

[0003] Figure 2 It is a diagram of one or more example systems that include autonomous vehicles.

[0004] Figure 3 yes Figure 1 and Figure 2 A diagram of one or more example devices and / or one or more example system components.

[0005] Figure 4A This is a diagram of some components of an example autonomous system.

[0006] Figure 4B This is a diagram of some components of an example autonomous system.

[0007] Figure 4C This is a diagram of some components of an example autonomous system.

[0008] Figure 4D This is a diagram of some components of an example autonomous system.

[0009] Figure 5 A vehicle equipped with a light detection and ranging (LiDAR) system and an imaging system is illustrated schematically. The illustration shows an image captured by the vehicle's imaging system.

[0010] Figure 6 This is a block diagram illustrating an example imaging system having a visible (VIS) image sensor for capturing visible images and an infrared image sensor for capturing thermal images.

[0011] Figure 7 This is an example that can be derived from... Figure 6 The illustrated imaging system uses an example optical subsystem and a block diagram of the arrangement of optical components therein to capture visible and infrared images of a scene via a single entrance aperture.

[0012] Figure 8 This is an example of use. Figure 6The diagram shows images generated by the VIS image sensor and infrared image sensor of the imaging system, as well as images generated by fusing the VIS image and the thermal image. Detailed Implementation

[0013] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatus are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0014] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0015] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0016] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0017] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0018] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet (e.g., a data packet) that includes data.

[0019] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0020] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0021] General Overview

[0022] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement improvements to the design, manufacture, operation, and / or management of autonomous vehicles or such vehicle queues, components or elements of autonomous vehicles or such vehicle queues, support systems for autonomous vehicles or such vehicle queues, or auxiliary systems associated with autonomous vehicles or such vehicle queues.

[0023] By implementing the systems, methods, and computer program products described herein, techniques for improving autonomous vehicles or such vehicle platoons are realized (including improvements in the design, manufacture, operation, and / or management of autonomous vehicle or such vehicle platoons, components or elements of autonomous vehicle or such vehicle platoons, support systems for autonomous vehicle or such vehicle platoons, or auxiliary systems associated with autonomous vehicle or such vehicle platoons).

[0024] Now for reference Figure 1Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0025] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2 The vehicles 200 in the collection of vehicles 200 are similar or identical. In some embodiments, vehicles 200 in the collection of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0026] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0027] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with (e.g., defining) a series of actions (also referred to as trajectories) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will board the AV, and a second state or area includes a location where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Alternatively, route 106 may include more precise actions or states, such as, for example, a specific target lane or precise location within a lane area and a target rate at those locations. In the example, route 106 includes a plurality of precise state sequences along at least one high-level action having a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to a plurality of trajectories that collectively form a high-level route terminating at a final target state or region.

[0028] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In the example, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driving lane, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In the example, a road includes at least one lane associated with at least one lane marking line (e.g., identified based on at least one lane marking line).

[0029] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Infrastructure or Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with vehicle 102 and / or V2I system 118. In some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, V2I device 110 includes radio frequency identification (RFID) devices, signs, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markings, streetlights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicle 102. Alternatively, in some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, and / or queue management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0030] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0031] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0032] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including and / or not including autonomous systems)).

[0033] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0034] supply Figure 1 The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements with different arrangements. Alternatively or additionally, at least one element of environment 100 may be described as being composed of… Figure 1 One or more functions performed by at least one different element of environment 100. Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0035] Now for reference Figure 2 Vehicle 200 (which can be connected with) Figure 1 The vehicle 200 (which is the same as or similar to vehicle 102) includes, or is associated with, an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 is associated with, or is similar to, vehicle 102 (see...). Figure 1 The autonomous system 202 is configured to endow the vehicle 200 with autonomous driving capabilities (e.g., implementing at least one driving automation or maneuver-based function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention, such as Level 5 ADS-operated vehicles), highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations, such as Level 4 ADS-operated vehicles), and / or conditionally autonomous vehicles (e.g., vehicles that abandon human intervention in limited situations, such as Level 3 ADS-operated vehicles)). In one embodiment, the autonomous system 202 includes some or all of the operational or tactical functionality required for the vehicle 200 to operate in road traffic and continuously perform dynamic driving tasks (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) incorporating driver support features. The autonomous system 202 supports various levels of driver automation, ranging from no driver automation (e.g., Level 0) to full driver automation (e.g., Level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0036] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0037] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision). In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0038] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD (Traffic Light Detection) data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other systems incorporated into cameras described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., wide-angle lens, fisheye lens, and / or lenses with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0039] In some embodiments, camera 202a can generate two or more images captured using light with different spectral distributions, enabling the detection of physical objects that may appear brighter in one of these images. In some embodiments, camera 202a may include at least two image sensors with different spectral responses and configured to generate two images of the same portion of the scene using different spectral portions of light received from the scene. In some such embodiments, camera 202a may combine or fuse two images (e.g., digital images) received from two image sensors to generate a composite image. In some examples, a portion of a captured scene that appears at different brightness levels in images formed on two different image sensors may be represented with a modified (e.g., enhanced) brightness based on the image in which that portion appears at a higher brightness level. In some embodiments, camera 202a may include the following regarding… Figure 6 and Figure 7 The imaging system 600 and example optical subsystem 702 describe one or more features. In some embodiments, camera 202a may include Figure 6and Figure 7 The imaging system 600 and the example optical subsystem 702 are shown in the example.

[0040] The Light Detection and Ranging (LiDAR) sensor 202b includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object within the field of view of the LiDAR sensor 202b.

[0041] The radio detection and ranging (RADR) sensor 202c includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3At least one device that communicates with the same or similar bus (bus 302). Radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by Radar sensor 202c encounter a physical object and are reflected back to Radar sensor 202c. In some embodiments, the radio waves emitted by Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with Radar sensor 202c generates a signal representing objects included in the field of view of Radar sensor 202c. For example, at least one data processing system associated with Radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of Radar sensor 202c.

[0042] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0043] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, an autonomous vehicle computing unit 202f, a safety controller 202g, and / or a DBW (drive-by-wire) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0044] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1 V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1 The V2I system 118 communicates with the same or similar V2I system.

[0045] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0046] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200. Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0047] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle movement (such as starting to move forward, stopping to move forward, starting to move backward, stopping to move backward, accelerating in a certain direction, decelerating in a certain direction, etc.) or lateral vehicle movement (such as making a left turn and / or making a right turn, etc.). In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0048] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to rotate left or right, thereby causing the vehicle 200 to turn left or right. In other words, the steering control system 206 causes the activity required to regulate the y-axis component of the vehicle's motion.

[0049] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0050] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), wheel rate sensors, wheel brake pressure sensors, wheel torque sensors, engine torque sensors, and / or steering angle sensors. Although the braking system 208 is illustrated as being located at... Figure 2 The braking system 208 is located near the vehicle 200, but it can be located anywhere within the vehicle 200.

[0051] Now for reference Figure 3 The diagram illustrates a device 300. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of the system of vehicle 102), at least one device of remote AV system 114, at least one device of queue management system 116, at least one device of vehicle-to-infrastructure system 118, and / or one or more devices of network 112 (e.g., one or more devices of the system of network 112). In some embodiments, one or more devices of the vehicle 102 (e.g., one or more devices of the system of the vehicle 102), at least one device of the remote AV system 114, at least one device of the queue management system 116, at least one device of the vehicle-to-infrastructure system 118, and / or one or more devices of the network 112 (e.g., one or more devices of the system of the network 112) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0052] Bus 302 includes components for communication between the components of the licensed device 300. In some cases, processor 304 includes a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or accelerated processing unit (APU), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC)). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores data and / or instructions for use by processor 304.

[0053] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0054] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0055] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, or a Wi-Fi interface. ® Interfaces and / or cellular network interfaces, etc.

[0056] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 306 and / or storage component 308, executed by processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0057] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. When executed, the software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0058] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0059] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the other device, which, when executed by the processor of processor 304 and / or the processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0060] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, the set of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another set of components of device 300.

[0061] Now for reference Figures 4A to 4D The diagram illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Alternatively or additionally, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0062] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0063] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the perception system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to the tactical functions required to operate the vehicle 102 in road traffic. Tactical efforts involve maneuvering the vehicle in traffic during a journey, including but not limited to deciding whether and when to overtake other vehicles, change lanes, or select appropriate speed, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0064] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various other types of traffic signals). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0065] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0066] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate a powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), a steering control system (e.g., steering control system 206), and / or a braking system (e.g., braking system 208). For example, the control system 408 is configured to perform operational functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes activities required to regulate the y-axis component of the vehicle's motion. Longitudinal vehicle motion control causes activities required to regulate the x-axis component of the vehicle's motion. In an example, where the trajectory includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers).

[0067] In some embodiments, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.). The following is about Figures 4B to 4D Includes examples of machine learning model implementations.

[0068] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3(The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0069] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0070] Now for reference Figure 4B The diagram illustrates an implementation of a machine learning model. More specifically, it illustrates an implementation of a convolutional neural network (CNN) 420. For illustrative purposes, the following description of CNN 420 will concern the implementation of CNN 420 via a perception system 402. However, it will be understood that in some examples, CNN 420 (e.g., one or more components of CNN 420) is implemented by systems other than or besides perception system 402 (such as planning system 404, positioning system 406, and / or control system 408, etc.). Although CNN 420 includes certain features as described herein, these features are provided for illustrative purposes and are not intended to limit this disclosure.

[0071] CNN 420 includes multiple convolutional layers comprising a first convolutional layer 422, a second convolutional layer 424, and a convolutional layer 426. In some embodiments, CNN 420 includes a subsampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, subsampling layer 428 and / or other subsampling layers have a dimension smaller than the dimension of the upstream system (i.e., the number of nodes). By means of subsampling layer 428 having a dimension smaller than that of the upstream layers, CNN 420 combines the amount of data associated with the initial input and / or output of the upstream layers, thereby reducing the computational cost required for downstream convolutional operations by CNN 420. Additionally or alternatively, subsampling layer 428 is associated with at least one subsampling function (e.g., configured to perform at least one subsampling function) (as described below). Figure 4C and Figure 4D As described, the CNN 420 incorporates the amount of data associated with the initial input. In some cases, the initial input may include data from the following... Figure 6 and Figure 7 The imaging system 600 and the example optical subsystem 702 described generate one or both of a thermal image and a visible image (e.g., a digital image). In some cases, the initial input may include the following... Figure 6 and Figure 7 The composite image generated by the described imaging system 600 and example optical subsystem 702.

[0072] Based on the perception system 402 providing corresponding inputs and / or outputs associated with the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426 to generate corresponding outputs, the perception system 402 performs convolution operations. In some examples, based on the perception system 402 providing data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426, the perception system 402 implements a CNN 420. In such examples, based on the perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle identical or similar to vehicle 102, a remote AV system identical or similar to remote AV system 114, a queue management system identical or similar to queue management system 116, and / or a V2I system identical or similar to V2I system 118, etc.), the perception system 402 provides data as input to the first convolutional layer 422, the second convolutional layer 424, and the convolutional layer 426. The following is about Figure 4C Includes a detailed explanation of convolution operations.

[0073] In some embodiments, the perception system 402 provides data associated with an input (referred to as initial input) to a first convolutional layer 422, and the perception system 402 uses the first convolutional layer 422 to generate data associated with an output. In some embodiments, the perception system 402 provides the output generated by the convolutional layer as input to different convolutional layers. For example, the perception system 402 provides the output of the first convolutional layer 422 as input to a subsampling layer 428, a second convolutional layer 424, and / or a convolutional layer 426. In such an example, the first convolutional layer 422 is referred to as the upstream layer, and the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426 are referred to as downstream layers. Similarly, in some embodiments, the perception system 402 provides the output of the subsampling layer 428 to the second convolutional layer 424 and / or the convolutional layer 426, and in this example, the subsampling layer 428 will be referred to as the upstream layer, and the second convolutional layer 424 and / or the convolutional layer 426 will be referred to as the downstream layer.

[0074] In some embodiments, before providing input to the CNN 420, the perception system 402 processes the data associated with the input provided to the CNN 420. For example, the perception system 402 processes the data associated with the input provided to the CNN 420 based on the normalization of sensor data (e.g., image data, LiDAR data, and / or Radar data, etc.) by the perception system 402.

[0075] In some embodiments, CNN 420 generates outputs by performing convolution operations associated with each convolutional layer based on the perception system 402. In some examples, CNN 420 generates outputs by performing convolution operations associated with each convolutional layer and an initial input based on the perception system 402. In some embodiments, the perception system 402 generates outputs and provides these outputs to a fully connected layer 430. In some examples, the perception system 402 provides the outputs of convolutional layer 426 to the fully connected layer 430, wherein the fully connected layer 430 includes data associated with multiple feature values ​​referred to as F1, F2, ..., FN. In this example, the outputs of convolutional layer 426 include data associated with multiple output feature values ​​representing predictions.

[0076] In some embodiments, the perception system 402 identifies a prediction from among a plurality of predictions based on the feature value identified by the perception system 402 as the highest likelihood of being the correct prediction among a plurality of predictions. For example, in the case where the fully connected layer 430 includes feature values ​​F1, F2, ..., FN and F1 is the largest feature value, the perception system 402 identifies the prediction associated with F1 as the correct prediction among a plurality of predictions. In some embodiments, the perception system 402 trains the CNN 420 to generate predictions. In some examples, the perception system 402 trains the CNN 420 to generate predictions based on training data associated with the predictions provided to the CNN 420.

[0077] Now for reference Figure 4C and Figure 4D A diagram illustrating example operation of a CNN 440 utilizing a perception system 402. In some embodiments, CNN 440 (e.g., one or more components of CNN 440) and CNN 420 (e.g., one or more components of CNN 420) (see...) Figure 4B (Same or similar)

[0078] In step 450, the perception system 402 provides image-associated data as input to the CNN 440 (step 450). For example, as illustrated, the perception system 402 provides image-associated data to the CNN 440, wherein the image is a grayscale image represented as values ​​stored in a two-dimensional (2D) array. In some embodiments, the image-associated data may include data associated with a color image, which is represented as values ​​stored in a three-dimensional (3D) array. Additionally or alternatively, the image-associated data may include data associated with infrared images and / or Radar images, etc.

[0079] In step 455, CNN 440 executes a first convolution function. For example, CNN 440 executes the first convolution function based on the fact that CNN 440 provides values ​​representing the image as input to one or more neurons (not explicitly illustrated) included in the first convolutional layer 442. In this example, the values ​​representing the image may correspond to values ​​representing regions (sometimes called receptive fields) of the image. In some embodiments, individual neurons are associated with filters (not explicitly illustrated). A filter (sometimes called a kernel) may be represented as an array of values ​​corresponding in size to the values ​​provided as input to the neuron. In one example, the filter may be configured to recognize edges (e.g., horizontal lines, vertical lines, and / or straight lines, etc.). In successive convolutional layers, the filters associated with neurons may be configured to successively recognize more complex patterns (e.g., arcs and / or objects, etc.).

[0080] In some embodiments, the image provided as input to CNN 440 or another CNN may include images derived from the following: Figure 6 and Figure 7 The described imaging system 600 and example optical subsystem 702 generate one or both of a thermal image and a visible image (e.g., a digital image). In some embodiments, the image provided as input to CNN 440 or another CNN may include images generated by the following... Figure 6 and Figure 7 The composite image generated by the described imaging system 600 and example optical subsystem 702.

[0081] In some embodiments, the CNN 440 performs a first convolution function by multiplying the values ​​of each neuron in one or more neurons included in the first convolutional layer 442, which are provided as input, with the values ​​of the filters corresponding to each of the neurons in the same or more neurons. For example, the CNN 440 may multiply the values ​​of each neuron in one or more neurons included in the first convolutional layer 442, which are provided as input, with the values ​​of the filters corresponding to each of the neurons in the same or more neurons to generate a single value or an array of values ​​as output. In some embodiments, the collective output of the neurons in the first convolutional layer 442 is referred to as the convolutional output. In some embodiments, when the neurons have the same filters, the convolutional output is referred to as a feature map.

[0082] In some embodiments, CNN 440 provides the outputs of each neuron in the first convolutional layer 442 to neurons in downstream layers. For clarity, an upstream layer may be a layer that transmits data to a different layer (referred to as a downstream layer). For example, CNN 440 may provide the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in a subsampling layer. In this example, CNN 440 provides the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in the first subsampling layer 444. In some embodiments, CNN 440 adds a bias value to an aggregate of all values ​​provided to the neurons in the downstream layer. For example, CNN 440 adds a bias value to an aggregate of all values ​​provided to the neurons in the first subsampling layer 444. In such an example, CNN 440 determines the final values ​​to be provided to the neurons in the first subsampling layer 444 based on the aggregate of all values ​​provided to the neurons and the activation function associated with each neuron in the first subsampling layer 444.

[0083] In step 460, CNN 640 executes a first subsampling function. For example, CNN 440 may execute the first subsampling function based on the values ​​provided by CNN 440 from the output of the first convolutional layer 442 to the corresponding neurons of the first subsampling layer 444. In some embodiments, CNN 440 executes the first subsampling function based on an aggregation function. In one example, CNN 440 executes the first subsampling function based on determining the maximum input (called the max pooling function) among the values ​​provided to a given neuron. In another example, CNN 440 executes the first subsampling function based on determining the average input (called the average pooling function) among the values ​​provided to a given neuron. In some embodiments, based on the values ​​provided by CNN 440 to the individual neurons of the first subsampling layer 444, CNN 440 generates an output, which is sometimes referred to as the subsampled convolution output.

[0084] In step 465, CNN 440 executes a second convolution function. In some embodiments, CNN 440 executes the second convolution function in a manner similar to how CNN 440 executes the first convolution function described above. In some embodiments, CNN 440 executes the second convolution function based on CNN 440 providing the value output by the first subsampling layer 444 as input to one or more neurons (not explicitly illustrated) included in the second convolutional layer 446. In some embodiments, as described above, each neuron in the second convolutional layer 446 is associated with a filter. As described above, the filter (one or more) associated with the second convolutional layer 446 may be configured to recognize more complex patterns compared to the filter associated with the first convolutional layer 442.

[0085] In some embodiments, the CNN 440 performs a second convolution function by multiplying the values ​​of each neuron in one or more neurons included in the second convolutional layer 446 as input with the values ​​of the filters corresponding to each of those neurons. For example, the CNN 440 may multiply the values ​​of each neuron in one or more neurons included in the second convolutional layer 446 as input with the values ​​of the filters corresponding to those neurons to generate a single value or an array of values ​​as output.

[0086] In some embodiments, CNN 440 provides the outputs of each neuron in the second convolutional layer 446 to neurons in downstream layers. For example, CNN 440 may provide the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in the subsampling layer. In this example, CNN 440 provides the outputs of each neuron in the first convolutional layer 442 to the corresponding neurons in the second subsampling layer 448. In some embodiments, CNN 440 adds a bias value to an aggregate of all values ​​provided to the neurons in the downstream layers. For example, CNN 440 adds a bias value to an aggregate of all values ​​provided to the neurons in the second subsampling layer 448. In such an example, CNN 440 determines the final values ​​provided to the neurons in the second subsampling layer 448 based on the aggregate of all values ​​provided to the neurons and the activation function associated with each neuron in the second subsampling layer 448.

[0087] In step 470, CNN 440 executes a second subsampling function. For example, CNN 440 may execute a second subsampling function based on the values ​​provided by the output of the second convolutional layer 446 to the corresponding neurons of the second subsampling layer 448. In some embodiments, CNN 440 executes a second subsampling function based on the use of an aggregation function. In the example, as described above, CNN 440 executes a first subsampling function based on determining the maximum or average input among the values ​​provided to a given neuron. In some embodiments, CNN 440 generates an output based on the values ​​provided to the individual neurons of the second subsampling layer 448.

[0088] In step 475, CNN 440 provides the outputs of each neuron in the second subsampling layer 448 to the fully connected layer 449. For example, CNN 440 provides the outputs of each neuron in the second subsampling layer 448 to the fully connected layer 449, causing the fully connected layer 449 to generate an output. In some embodiments, the fully connected layer 449 is configured to generate an output associated with a prediction (sometimes called a classification). The prediction may include an indication that objects included in the image provided as input to CNN 440 include objects and / or sets of objects, etc. In some embodiments, the perception system 402 performs one or more operations and / or provides data associated with the prediction to different systems described herein.

[0089] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and the applicant's expectation that the scope of this invention is defined in the specific form of the claims published in this application, is the literal and equivalent scope of the claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as used in the claims. Furthermore, when the term "comprising" is used in the preceding specification or appended claims, the following phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

[0090] Further details regarding specific anticipated improvements are provided in the following section concerning multispectral imaging. It should be understood that statements made with respect to the imaging systems described below should be construed as referring to the embodiments disclosed below, and not necessarily to all embodiments of this disclosure. For example, in the context of the imaging systems described below, statements indicating that certain features are necessary or required (to the extent such statements are present) should be understood to refer to the necessity or requirement of such features for embodiments of these imaging systems. Such statements do not indicate that such features are necessary or required for all embodiments of this disclosure. Similarly, terminology definitions in the following sections should be understood as defining terms in the context of the corresponding imaging systems, and such definitions may not apply to the remainder of this disclosure.

[0091] The improvements discussed in the following description of the imaging system can be expressed in this paper regarding... Figures 1 to 4D Computer-achievable improvements implemented within the described system. For example, improvements discussed regarding the imaging system described below can be wholly or partially derived from… Figure 3 The device 300 is implemented, such as by means of Figure 3 Execution on processor 304 Figure 3 The device 300 is implemented using computer code stored in memory 306, which implements the improvements to the corresponding imaging system described below. In one embodiment, the device 300 is located in a vehicle (such as...) Figure 2 Inside the vehicle 200, etc. In another embodiment, the device 300 is located outside the vehicle, such as in... Figure 1 Remote AV systems 114, queue management systems 116, or vehicle-to-infrastructure systems 118, etc. Additionally or alternatively, the improvements discussed herein may represent... Figure 1 Improved components of the elements, including Figure 1The improvements can be incorporated into the vehicle 102, remote AV system 114, queue management system 116, or vehicle-to-infrastructure system 118. Therefore, the improvements discussed below can be integrated as a whole into... Figure 1 One or more of the above elements can be used to improve these elements. As an example, improvements to LiDAR sensors can be used to modify... Figure 2 The LiDAR sensor 202b. About... Figure 2 Further details of an example implementation of the LiDAR sensor 202b are provided in U.S. Patent Application No. 17 / 931,051, filed September 9, 2022, entitled “SYSTEMS AND METHODS FOR TIME-OF-FLIGHT (TOF) LIDAR SIGNAL-TO-NOISEIMPROVEMENT” (Attorney’s File No.: MOTN.057A), the entire contents of which are incorporated herein by reference.

[0092] The imaging systems and methods described below can be incorporated into various types of autonomous vehicles and self-driving cars, such as those disclosed in U.S. Patent Application 17 / 444,966, filed August 12, 2021, entitled “END-TO-END SYSTEM TRAININGUSING FUSED IMAGES” and U.S. Patent Application 17 / 443,433, filed July 26, 2021, entitled “VEHICLE LOCATION USING COMBINED INPUTS OF REDUNDANT LOCALIZATION PIPELINES”, the entire contents of which are incorporated herein by reference and form part of this specification.

[0093] A vision system with integrated visible and thermal imaging.

[0094] In various imaging systems, two or more images captured by different image sensors or captured by a single image sensor at two different times can be combined or fused to generate a fused or synthetic image. In some cases, the process of combining image information from two or more images (e.g., relevant information from two or more images) into a single synthetic image can be referred to as image fusion.

[0095] In some cases, a fused image generated by fusing two or more input images can be more informative and accurate than the individual input images. Image fusion can be used in various imaging applications (e.g., remote sensing, medical imaging, military applications, and computer vision) to enhance image quality, improve decision-making, reduce data volume, and provide other improvements. For example, integrating complementary information from different images into a fused image can provide more detailed, clearer, and more comprehensive information, each of which can aid in better analysis and interpretation. In some cases, image fusion may include fusing two input images captured by two different image sensors (e.g., a thermal image sensor and a visible image sensor) with different spectral responses or sensitivities. Advantageously, a synthetic or fused image (referred to herein as a spectrally complementary input image) formed by fusing images captured by two image sensors with different spectral sensitivities can include features that may be unclear or absent in one of the input images captured by one of these image sensors.

[0096] One of the challenges of image fusion is image registration and / or aligning input images (e.g., images captured by different image sensors and / or at different times) to accurately preserve the geometric features of the captured scene in the resulting synthetic image. In some cases, aligning input images can be difficult due to differences in the viewpoint, resolution, and field of view of the imaging systems used to capture the input images. In some cases, inaccurate alignment and / or misalignment between features of different input images in the resulting fused image can lead to low resolution, inaccuracies, and other defects and deficiencies.

[0097] In some cases, when two input images are captured by two separate imaging systems (e.g., two cameras), misalignment between the input images may be caused by relative movement and / or vibration between the two imaging systems. For example, when two cameras are mounted on a moving platform (e.g., a vehicle such as a car), mechanical vibration of the moving platform may cause misalignment between the images captured by these cameras. In some cases, the two cameras may include a visual camera and a thermal camera, the visual camera being configured to capture images based on light having wavelengths in the visible wavelength range (e.g., from 380 nm to 780 nm), and the thermal camera being configured to capture images based on electromagnetic radiation having wavelengths in the long-wave infrared (LWIR) wavelength range (e.g., from 8 μm to 15 μm) or, in some cases, in the mid-wave infrared (MWIR) wavelength range (e.g., from 2.5 μm to 8 μm).

[0098] In some implementations, two imaging systems (e.g., a visual camera and a thermal camera) can be mounted (e.g., separately) on a vehicle to allow the detection of humans and animals, thereby preventing fatal accidents, especially in low-light conditions (e.g., at night). In some cases, the alignment error between the images captured by the two imaging systems (e.g., a visual camera and a thermal camera) (e.g., due to manufacturing tolerances, thermal shift, mechanical vibration (when the camera is mounted on the vehicle)) can be as large as 3 degrees; such a large misalignment can negatively impact the accuracy of image fusion processing, and thus negatively impact the accuracy of the resulting fused image.

[0099] In some imaging systems disclosed herein (described below), by integrating (e.g., optically integrating) two imaging devices, the degree of misalignment between two images captured by two different imaging devices (such as a visual camera and a thermal camera mounted on a vehicle) can be significantly reduced (e.g., reduced to less than 0.5 degrees). In some cases, optically integrating two imaging systems may include redirecting light or electromagnetic radiation received from the environment through a single entrance aperture to two different image sensors (e.g., a VIS image sensor and a thermal image sensor) via two partially overlapping optical paths extending between that single entrance aperture and two image sensors with different spectral responses (e.g., in the VIS wavelength range and the thermal wavelength range). In some embodiments, the overlapping portion of the two optical paths may provide high transmittance for light (or radiation) within both the VIS wavelength range and the thermal wavelength range (e.g., including LWIR and MWIR), and the two non-overlapping portions may each provide high transmittance for light (or radiation) within one of the VIS wavelength range and the thermal wavelength range.

[0100] Some of the imaging systems disclosed herein (described below) may include imaging systems configured to be mounted on a vehicle to image the environment surrounding the vehicle, for example, to image the environment to generate images, information, control signals or commands that can be used to safely drive the vehicle.

[0101] In some embodiments, the imaging system disclosed below can allow the generation of an accurate composite image formed from spectrally complementary images captured by different image sensors with different spectral responses. In some cases, the disclosed imaging system can be mounted on a vehicle and can be configured to generate a composite image formed by fusing accurately aligned visible light and thermal images. The visible light image (referred to herein as the visible image) can be an image formed from light having wavelengths in the visible wavelength range, and the thermal image can be an image formed from thermal radiation or electromagnetic radiation having wavelengths in the MWIR or LWIR wavelength range. In some cases, such a composite (or fused) image can allow the identification of obstacles and moving subjects in the environment surrounding the vehicle (e.g., under low-light conditions (e.g., at night or in a tunnel)). In some cases, the composite image formed by fusing the thermal and visible images can include information and features missing from either the thermal or visible image. For example, certain features (e.g., traffic signs, road markings, or features including multiple colors) may not be identifiable due to a lack of contrast and / or visibility. Similarly, in visible images captured under low-light conditions, certain other features (e.g., humans, animals, colors) may not be easily identifiable.

[0102] In some embodiments, the disclosed imaging system may include an integrated optical design that provides partially overlapping optical paths, including a common field of view (FOV) and an entrance aperture (or window) for forming visible and thermal images of the same scene on two image sensors with different spectral responses. In some cases, the shared portion of the two optical path lengths may be configured to allow images with substantially the same viewing angle and the same elements / features of the scene to be formed on both image sensors.

[0103] The disclosed methods and systems can significantly reduce and potentially eliminate misalignment between images captured by visible and thermal imagers, and enable the generation of highly accurate synthetic images using integrated visible and thermal imagers. In some examples, the imaging system can form a first image and a second image of a scene on a first sensor and a second sensor, respectively. The first and second images can be formed via a first optical path and a second optical path different from the first optical path, respectively. In some cases, the first and second optical paths at least partially overlap.

[0104] In some cases, the first optical path can be configured to selectively transmit light having wavelengths within a first wavelength range (e.g., the VIS wavelength range), and the second optical path can be configured to selectively transmit light having wavelengths within a second wavelength range (e.g., the LWIR and / or MIR wavelength range). In some cases, the first optical path can be configured to provide greater optical transmittance to light having wavelengths within the first wavelength range compared to light having wavelengths within the second wavelength range. In some such cases, the second optical path can be configured to provide greater optical transmittance to light having wavelengths within the second wavelength range compared to light having wavelengths within the first wavelength range.

[0105] In some examples, the non-overlapping portions of the first and second optical paths can be generated by a dichroic beamsplitter that transmits light having wavelengths within a first wavelength range (e.g., the visible wavelength range) and redirects light having wavelengths within a second wavelength range (e.g., the LWIR or MWIR wavelength range) (e.g., rotating light having wavelengths within the second wavelength range by 90 degrees). Additionally or alternatively, one or more optical components or surfaces in the first optical path can be configured to reject light having wavelengths within the second wavelength range (e.g., the LWIR or MWIR wavelength range), and one or more optical components or surfaces in the second optical path can be configured to reject light having wavelengths within the first wavelength range.

[0106] In some cases, the first image and the second image can be substantially the same image of the same scene with the same magnification but different spectral distributions. For example, the first image may have a spectral distribution with peak wavelengths in the visible wavelength range, and the second image may have a spectral distribution with peak wavelengths in the near-infrared wavelength range.

[0107] In some implementations, the first image sensor may have high sensitivity (or peak sensitivity) in the visible (VIS) wavelength range, and the second image sensor may have high sensitivity (or peak sensitivity) in the LWIR (or MWIR) wavelength range. An image sensor with high sensitivity (or peak sensitivity) in the visible (VIS) wavelength range may be referred to as a VIS image sensor, and an image sensor with high sensitivity (or peak sensitivity) in the LWIR (or MWIR) wavelength range may be collectively referred to as an LWIR (or MWIR) image sensor. In some cases, the first and second wavelength ranges may be non-overlapping (i.e., mutually exclusive) or partially overlapping. In some examples, the first image sensor may be sensitive to light from 380 nm to 780 nm, and the second image sensor may be sensitive to light from 2.5 μm to 4 μm, from 4 μm to 7.5 μm, from 7.5 μm to 10 μm, from 10 μm to 15 μm, from 8 μm to 12 μm, from 8 μm to 15 μm, or any range or larger or smaller of these values. In some examples, the difference between the peak response wavelengths of the first and second image sensors can be from 5 micrometers to 7.5 micrometers, from 7.5 micrometers to 8.5 micrometers, from 8.5 micrometers to 9.5 micrometers, from 9.5 micrometers to 10.5 micrometers, from 10.5 micrometers to 11.5 micrometers, or any range formed by these values ​​or values ​​greater or smaller. In some cases, where the two image sensors (e.g., a VIS sensor and an LWIR sensor) have different spectral sensitivities, the first and second images can be substantially identical images of the same scene with the same magnification and spectral distribution. In some embodiments, the first image sensor can include silicon, gallium arsenide (GaAs), indium gallium arsenide (InGaAs), perovskite (e.g., CsPbBr3), graphene, or other materials.

[0108] In some embodiments, the second image sensor may include a calorimeter, a thermocouple, a thermopile, a pyroelectric microbolometer detector, or a detector including a quantum well and / or a superlattice.

[0109] In some embodiments, the second image sensor may include mercury cadmium telluride (HgCdTe or MCT), indium gallium arsenide (InGaAs), lead sulfide (PbS) and lead selenide (PbSe), indium antimonide (InSb) or other materials (e.g., heterostructures and compound semiconductors).

[0110] In some such cases, the first and second optical paths used to form the first and second images can provide similar or substantially the same spectral transmittance. For example, the two paths can provide the same amount of optical transmittance for light with wavelengths in the LWIR (and / or MWIR) wavelength range and light with wavelengths in the VIS wavelength range. In some cases, such as when the two image sensors have different spectral sensitivities, the two different optical paths can be generated by beamsplitters that do not distinguish between LWIR (or MWIR) wavelengths and VIS wavelengths (e.g., beamsplitters with the same beam splitting ratio for light with wavelengths in the LWIR (MWIR) wavelength range and light with wavelengths in the VIS wavelength range).

[0111] In some embodiments, the imaging system may fuse a first image sensor and a second image sensor to generate a fused or synthesized image that includes features captured by the VIS image sensor and the LWIR (or MWIR) image sensor. In some cases, the imaging system may process (e.g., modify) a digital image of a scene generated using the first image sensor and / or the second image sensor before fusing the VIS image and the thermal image (e.g., the LWIR or MWIR image).

[0112] Imaging and sensing systems of vehicles

[0113] Figure 5 A vehicle 500 equipped with an imaging system 502 is schematically illustrated. In some cases, in addition to the imaging system 502, a light detection and ranging (LiDAR) system 504 may also be included. In some cases, one or both of the imaging system 502 and the LiDAR system 504 may be configured to detect and image objects (both luminous and non-luminous objects) and other vehicles in the environment surrounding the vehicle 500. In some cases, the vehicle 500 may be an autonomous or semi-autonomous vehicle (e.g., a vehicle with a driving automation level of 1-6 as defined by the Society of Automotive Engineers (SAE)). The imaging system 502 may include at least one digital camera configured to generate an image of a portion of the surrounding environment. In some cases, the image generated by the imaging system 502 of the vehicle 500 may include a fused or synthesized digital image generated by processing two or more digital images or digital image signals received from two or more image sensors (e.g., sensors with different spectral responses). In some cases, the vehicle 500 may include a light source configured to illuminate the scene to enhance the image of the scene generated by the imaging system 502.

[0114] In some cases, LiDAR system 504 generates and directs a probe beam and receives reflections of the beam to detect objects in the environment. In various implementations, LiDAR system 504 can be a scanning LiDAR system or a mechanical LiDAR system. A scanning LiDAR system scans one or more probe beams across a wide field of view of the detector, while a mechanical LiDAR system emits a single probe beam (e.g., a low-divergence beam) to illuminate a narrow field of view of the detector and scans (e.g., rotates) both the detector and the probe beams.

[0115] In some embodiments, the imaging system 502 (e.g., a digital camera) can generate an image (e.g., a digital image) using image signals (digital image signals) received from one or more image sensors. The imaging system 502 may include one or more optical components that receive light from the environment and form images on two or more image sensors. The optical components may form an optical chain positioned along a light path from the environment to the image sensors. In some cases, a first optical chain that directs light from the environment to a first image sensor via a first light path can provide substantially the same optical transformation as a second optical chain that directs light from the environment to a second image sensor via a second light path. In some cases, the first and second optical chains may have different spectral transmittances but are otherwise identical (e.g., have the same image-forming properties). In some cases, the first and second optical chains may have different spectral transmittances and may be configured to form two images of the same portion of a scene on two different image sensors with different spectral responses, wherein the two images are magnified by substantially the same amount, captured through the same field of view, and include the same viewpoint of the scene.

[0116] In some cases, the two optical paths can receive light from the same scene via a common entrance aperture or optical input port. In some examples, at least a portion of the first optical path can overlap with the second optical path. As a result, the first image projected onto the first image sensor by the first optical chain can be substantially the same as the second image projected onto the second image sensor by the second optical chain (e.g., having the same magnification).

[0117] One or more image signals generated by imaging system 502 can be used by control and processing system to generate an image (e.g., a digital image) of a portion of a scene or surrounding environment. In some implementations, control and processing system may generate a digital image (e.g., a modified digital image) using at least a first image signal received from a first image sensor and a second image received from a second image sensor. In some examples, control and processing system generates a first image using the first image signal, generates a second image using the second image signal, compares the first and second images, and generates a third digital image based on the comparison. In some cases, generating a third image may include modifying the first digital image based on the second digital image (e.g., modifying a corresponding portion of the first digital image based on a portion of the second digital image). In some cases, control and processing may synchronize and / or process the first and second image signals synchronously, such that the first and second digital images correspond to images of the same scene captured simultaneously. In some implementations, control and processing may synchronize imaging system 502 with LiDAR system 504. In some such implementations, a first image signal and / or a second image signal are generated during the time interval in which the LiDAR system 504 is emitting a probe beam. In some cases, the first image signal and / or the second image signal, or two subsequent image signals received from a first image sensor and a second image sensor, are generated during the time interval in which the LiDAR system 504 is not emitting any probe beam.

[0118] Continue to refer to Figure 5 The control and processing system of imaging system 502 can generate an image 530 (digital image) of a scene in the environment. In some cases, image 530 can be generated using image signals received from the image sensor of imaging system 502 (e.g., a VIS image sensor or an LWIR image sensor). In the example shown, the scene captured by imaging system 502 includes an arriving vehicle 510, light sources and sensors carried by vehicle 510, and a human subject 506. Therefore, image 530 includes a depiction 511 of the arriving vehicle 510 and a depiction 507 of the human subject 506.

[0119] In some examples, the arriving vehicle 510 may include a second LiDAR system 514 that emits a beam of light for ranging. Therefore, image 530 includes a depiction 515 of the LiDAR system 514 and a depiction 513 of the light source 512. In some cases, one, both, or all of the LiDAR system 514 and the light source 512 may emit light with wavelengths within a wavelength range (e.g., the LWIR wavelength range) that at least partially overlaps with the spectral response of the image sensor used to generate image 530.

[0120] In some cases, imaging system 502 may include a VIS image sensor with sensitivity within the VIS range. In some such cases, in low-light environments (e.g., at night and in the absence of a sufficiently bright external light source), non-luminescent objects, humans, or animals may not be sufficiently visible in images captured by the VIS image sensor. Hereinafter, non-luminescent objects may include objects that do not emit electromagnetic radiation with wavelengths within the visible wavelength range or within the sensitivity range of the VIS image sensor.

[0121] Advantageously, in some aspects, in addition to the VIS image sensor, the imaging system may also include at least one image sensor having a spectral response in the LWIR range. In some embodiments, the imaging system may use image signals received from the LWIR image sensor to improve the visibility of non-luminescent physical objects, human subjects, or animals in a synthetic digital image generated by fusing images captured by the VIS image sensor and the LWIR image sensor.

[0122] Imaging systems including image sensors with different spectral responses

[0123] Figure 6 This is a block diagram illustrating an example imaging system 600 having a visible (VIS) image sensor 611 for capturing visible images and a long-wave infrared (LWIR) image sensor 612 for capturing LWIR or thermal images. In some cases, the thermal image may be an LWIR or MWIR image.

[0124] In some cases, the imaging system 600 can generate a synthetic or fused digital image that depicts certain non-luminescent elements in a scene, which generate electromagnetic radiation with wavelengths in the thermal wavelength range or the long-wave infrared wavelength range. In some cases, the imaging system 600 includes an optical subsystem 602, at least two image sensors 611, 612, and a control and processing system 622. In some cases, the optical subsystem 622 can be an integrated optical system comprising two partially overlapping optical paths from a single entrance aperture to two different image sensors. The imaging system 600 receives light from the scene, forms a first image on a first image sensor 611, and a second image on a second image sensor 612. In response to the formation of the first and second images, the first image sensor 611 and the second image sensor 612 can generate a first image signal 614 and a second image signal 616, and transmit the first image signal 614 and the second image signal 616 to the control and processing system 622. In some embodiments, the imaging system 600 can transmit the first image signal 614 and the second image signal 616 to another system.

[0125] In some cases, the first image signal 614 includes a first digital image associated with a first image of the scene formed on the first image sensor 611, and the second image signal 616 includes a second digital image associated with a second image of the same scene formed on the second image sensor 612. In some cases, the first image may have a different spectral intensity distribution than the second image, but be the same as the second image in other respects (e.g., having approximately the same magnification, the same features of the scene, and the same viewpoint of the scene).

[0126] In some examples, the first image signal 614 and the second image signal 616 include electrical signals carrying information (e.g., digital information) that can be used to generate the first digital image and the second digital image, respectively.

[0127] The control and processing system 622 can use the first image signal 614 and the second image signal 616 to generate a third image signal 626. In some cases, the third image signal 626 may include a synthesized or fused digital image. In some cases, the control and processing system 622 may transmit the synthesized or fused digital image to the navigation system or display of the vehicle 500 (or the autonomous vehicle computing 202f of the vehicle 200). In some embodiments, the third image signal 626 may include an electrical signal carrying information (e.g., digital information) that can be used to generate the synthesized digital image. In some cases, generating the synthesized digital image may include enhancing or adjusting the brightness of certain features that may not be sufficiently visible on the first or second image. In some cases, generating the synthesized digital image may include enhancing or adjusting the brightness and / or contrast of areas of the scene that have different brightness and / or contrast on the first image (or the corresponding first digital image) and the second image (or the corresponding second digital image), at least in part, based on one of the first or second images (e.g., an image area represented on it with higher brightness and / or contrast). In some embodiments, the third image signal 626 includes a synthesized digital image.

[0128] In some implementations, the control and processing system 622 may transmit a third image signal 626 to a display system that uses the third image signal 626 to generate an image viewable via a user interface. Additionally or alternatively, the control and processing system 622 may transmit the third image signal 626 to a navigation system that uses at least the third image signal for navigation in the environment. In some examples, the navigation system uses the third signal in combination with signals received from a LiDAR system (e.g., LiDAR 504) to navigate in the environment. The LiDAR system 504, imaging system 502, and navigation system may be mounted on a single vehicle 500 (e.g., AV), and navigation in the environment may include detecting objects and obstacles in the environment and determining their position and velocity relative to the vehicle.

[0129] In some embodiments, the first image sensor 611 may have a first spectral response, and the second image sensor may have a second spectral response different from the first spectral response. In some examples, the second peak sensitivity wavelength of the second spectral response may be at least 1 micrometer, at least 2 micrometers, at least 3 micrometers, at least 4 micrometers, at least 5 micrometers, at least 6 micrometers, at least 8 micrometers, or greater than the first peak sensitivity wavelength of the first spectral response. In some cases, the peak sensitivity wavelength of the spectral response may be the wavelength of the spectral response at other wavelengths within the response bandwidth (e.g., the operating bandwidth of the image sensor) of the spectral response. In some cases, the peak sensitivity wavelength of the spectral response may be the average or mean wavelength associated with the spectral response and may be calculated based on wavelengths with sensitivity greater than a threshold.

[0130] In some examples, the first image sensor 611 may have a peak responsivity in the visible wavelength range. In some examples, the first image sensor 611 may have a first response bandwidth that includes wavelengths from 400 nm to 700 nm, from 380 nm to 780 nm, from 400 nm to 1100 nm, or any range formed by these values ​​or greater or smaller values.

[0131] In some examples, the first image sensor 611 may have a peak responsivity in the mid-infrared or long-wave infrared wavelength range. In some examples, the second image sensor 612 may have a second response bandwidth that includes wavelengths ranging from 7.5 micrometers to 12 micrometers, from 8 micrometers to 12 micrometers, from 9 micrometers to 13 micrometers, from 12 micrometers to 15 micrometers, or any range formed by these values ​​or greater or smaller.

[0132] In some embodiments, the first image sensor 611 may include a complementary metal-oxide-semiconductor (CMOS) sensor (e.g., a silicon-based sensor) or a sensor comprising a group IIIV compound semiconductor (e.g., gallium arsenide, GaAs, or indium gallium arsenide, InGaAs, indium phosphide, InP, or combinations thereof). In some embodiments, the second image sensor 612 may include a microbolometer or an antimony-based photodetector (e.g., an antimony-based photodetector based on a type II superlattice), or another photodetector configured to generate an electrical signal in response to receiving light having a wavelength in the MWIR or LWIR wavelength range.

[0133] In some embodiments, the first image sensor 611 and the second image sensor 612 may be scalable imagers capable of generating images that can be scaled smaller or larger compared to images projected onto the corresponding sensors. In some examples, the first image sensor 611 and the second image sensor 612 may allow digital scaling of digital images.

[0134] In some embodiments, the overlap between the response or sensitivity bandwidth of the first image sensor 611 and the second image sensor 612 may be less than 30%, less than 20%, less than 10%, or a smaller value.

[0135] In some embodiments, the optical subsystem 602 may include an objective lens group 608 (also referred to as an objective optical chain), a dichroic beam splitter (e.g., a dichroic prism) 604, a first imaging lens group 609 (also referred to as a first optical chain), and a second imaging lens group 610 (also referred to as a second optical chain). However, it will be understood that the optical subsystem 602 may include fewer or more components. Therefore, the example optical subsystem 602 should not be construed as limiting.

[0136] Objective lens group 608 receives light 618A from the scene or environment via entrance aperture 624, transforms the received light, and transmits the transformed light 618B to dichroic beam splitter (e.g., dichroic prism) 604. In some cases, entrance aperture 624 may include a shutter (e.g., a mechanical or electro-optical shutter) through which light enters the optical subsystem 602 from the scene. Imaging control signals may open the shutter to allow image formation on VIS image sensor 611 and LWIR image sensor 612, or close the shutter to block light from entering the optical subsystem 602. In various embodiments, the timing at which one or both of the first image sensor 611 and the second image sensor 612 generates an image signal including an image of the scene can be controlled by controlling the shutter and / or by electronically activating the image sensor.

[0137] In some examples, the light ray 618A received via the entrance aperture 624 is intercepted by the first lens of the objective lens group 608, and the transformed light ray 618B includes the light ray exiting the last lens of the objective lens group 608. The objective lens group 608 can generate the transformed light ray 618B by redirecting one or more light rays intercepted by the first lens and outputting the redirected light ray via the last lens. However, it will be understood that the lens group 608 may include fewer lenses (e.g., one lens) or more lenses (e.g., more than two lenses). Therefore, the example lens group 608 should not be construed as limiting.

[0138] The dichroic beam splitter 604 transmits a first portion 632 of the transformed light 618B received from the objective lens group 608 toward the first imaging lens group 609, and redirects a second portion 630 of the transformed light 618B toward the second imaging lens group 610. The first portion 632 of the transformed light 618B may include light having wavelengths within a first wavelength range (or bandwidth) (e.g., the VIS wavelength range), and the second portion 630 of the transformed light 618B may include light having wavelengths within a second wavelength range (e.g., the LWIR (or MWIR) wavelength range) different from the first wavelength range. In some cases, the first and second wavelength ranges may be non-overlapping wavelength ranges. In other cases, the first and second wavelength ranges may be partially overlapping wavelength ranges. The first wavelength range may at least partially overlap with the response spectrum of the first imaging sensor 611, and the second wavelength range may at least partially overlap with the response spectrum of the second imaging sensor 612.

[0139] In some cases, the dichroic beam splitter 604 can split light from a scene into a first part and a second part with different spectral distributions. In some such cases, the first part may include wavelengths in the visible wavelength range, and the second part may include wavelengths in the mid-wave infrared (MWIR) or long-wave infrared (LWIR) wavelength range. In some cases, the spectral distribution of the first part may have an average value in the visible (VIS) wavelength range, and the spectral distribution of the second part may have an average value in the MWIR or LWIR wavelength range. Thus, in some examples, the dichroic beam splitter 604 can transmit light with wavelengths from 380 nm to 780 nm into a first imaging lens group 609, and redirect (e.g., reflect) light with wavelengths from 7.5 μm to 12 μm, from 12 μm to 15 μm, or from 8 μm to 15 μm into a second imaging lens group 610. Therefore, in these examples, the first portion 632 of the transformed light 618B received by the first imaging lens group 609 may have a wavelength distribution within the VIS wavelength range (e.g., having an average value within the VIS wavelength range), and the second portion 630 of the transformed light 618B received by the second imaging lens group 610 may have a wavelength distribution within the LWIR wavelength range (e.g., having an average value within the LWIR wavelength range).

[0140] In some examples, one or more optical surfaces of the first imaging lens group 609 may include an anti-reflective layer or coating configured to reduce or potentially remove Fresnel reflections at least in a portion of a first wavelength range (e.g., the VIS wavelength range), and one or more optical surfaces of the second imaging lens group 610 may include an anti-reflective layer or coating configured to reduce or potentially remove Fresnel reflections at least in a portion of a second wavelength range (e.g., the LWIR wavelength range).

[0141] In some examples, one or more optical surfaces of the first imaging lens group 609 may include a coating or layer configured to selectively transmit at least a portion of light having wavelengths within a first wavelength range (e.g., the VIS wavelength range) and reject / or block light having wavelengths within a second wavelength range (e.g., the LWIR wavelength range). In some examples, one or more optical surfaces of the second imaging lens group 610 may include a coating or layer configured to selectively transmit at least a portion of light having wavelengths within a second wavelength range (e.g., the LWIR wavelength range) and reject / or block light having wavelengths within a first wavelength range (e.g., the VIS wavelength range).

[0142] In some embodiments, the lens in the first imaging lens group 609 comprises or can be formed of a material with low absorption in a first wavelength range (e.g., the VIS wavelength range), and the lens in the second imaging lens group 610 comprises or can be formed of a material with low absorption in a second wavelength range (e.g., the LWIR wavelength range). For example, the lens in the first imaging lens group 609 may include silicon dioxide, quartz, or other types of glass, and the lens in the second imaging lens group 610 may include zinc selenide (ZnSe), germanium (Ge), GASIR, zinc sulfide (ZnS), calcium fluoride (CaF2), magnesium fluoride (MgF2), and sapphire (Al2O3), etc.

[0143] The first imaging lens group 609 further transforms the first portion 632 of the transformed light ray 618B to form a first image on the first image sensor 611, and the second imaging lens group 610 further transforms the second portion 630 of the transformed light ray 618B to form a second image on the second image sensor 612. The first lens group 609 and the second lens group 608 may be substantially the same lens group, or may have the same optical transformation properties (e.g., redirecting a given input light beam in the same way, having the same linear and angular magnification, etc.). Therefore, the first image and the second image formed on the first image sensor 611 and the second image sensor 612 may be images of the same part of the scene with the same magnification. In some cases, the first image and the second image formed on the first image sensor 611 and the second image sensor 612 may be images of the same part of the scene with the same magnification but (due to the spectral properties of the dichroic beam splitter 604 and the coatings that may be applied to components in one or both optical chains) have different spectral characteristics. For example, the first image may be formed by light having a wavelength distribution within the VIS spectral range (e.g., having the mean within the VIS wavelength range), and the second image may be formed by light having a wavelength distribution within the LWIR spectral range (e.g., having the mean within the LWIR wavelength range). Therefore, when the first image sensor 611 and the second image sensor 612 have similar or substantially the same spectral response, features generated by LWIR light (e.g., from a LiDAR sensor) may have a greater brightness level in the digital image of the same scene generated by the second image sensor 612 compared to the digital image of the scene generated by the first image sensor 611. The imaging system 600 may include a housing comprising an entrance aperture 624 configured to allow light from the scene or environment, and housing the optical subsystem 602, image sensors 611, 612. In some implementations, at least a portion of the control and processing system 622 may be included within the housing.

[0144] In some cases, the control and processing system 622 may include a memory and at least one processor configured to execute machine-readable instructions stored in the memory. The control and processing system 622 may include a field-programmable gate array (FPGA), memory cells, digital signal processing units, and internal wireless transceivers.

[0145] In some cases, the control and processing system 622 can control the first image sensor and the second image sensor such that they output image signals associated with the same scene (e.g., captured simultaneously), and in some cases, output these image signals simultaneously. In some implementations, the control and processing system 622 can process the VIS image signal 614 and the LWIR image signal 616, and process (e.g., compare together) such image signals associated with the same scene simultaneously.

[0146] The ability of the optical system 600 to generate two images of a scene with approximately equal magnification (e.g., using a sequence of approximately equal optical elements), combined with the synchronous generation and processing of image signals, enables the control and processing system 622 to directly compare corresponding portions of two digital images received from the VIS image sensor and the LWIR image sensor, and to generate a modified digital image. In some examples, the direct comparison may include pixel-to-pixel comparison, comparing the shape and / or size or corresponding areas on the two images.

[0147] Although shown as a dichroic beamsplitter 604, it will be understood that other prisms can be used as beamsplitters. In some cases, the beamsplitter can generate two images with similar or identical spectral properties having the same or different intensities. For example, when the peak spectral responses of the first image sensor 611 and the second image sensor 612 are sufficiently different, or when the second image sensor 612 has very low sensitivity to the VIS wavelength, a non-dichroic beamsplitter (e.g., a spectrally neutral beamsplitter) with a beam ratio between 95 / 5 and 50 / 50 in a spectral range that at least partially overlaps with the LWIR and VIS spectral ranges can replace the dichroic beamsplitter 604. In these examples, the beam ratio of the beamsplitter can be selected based on the spectral sensitivity of the first image sensor 611 and the second image sensor 612 to balance the brightness of the VIS and LWIR images formed on the VIS image sensor 611 and the LWIR image sensor 612.

[0148] It should be understood that in various implementations, one or a combination of components and / or features, including but not limited to dichroic beam splitting, coatings or filters with wavelength-selective transmission or reflection properties, and imaging sensors with different spectral responses, can be used to provide a first image signal 614 (e.g., a VIS image signal) comprising a first digital image of a scene and a second image signal 616 (e.g., an LWIR image signal) comprising a second digital image of the same scene, wherein the signature feature of LWIR light from the scene on the second digital image is brighter than the corresponding signature feature on the first digital image.

[0149] In various implementations, the optical subsystem 602 may have a narrow, medium, or wide field of view (FOV). The FOV of the optical subsystem 602 may be determined at least in part based on the characteristics of one or both of the first image sensor 611 and the second image sensor 612.

[0150] In some embodiments, the optical path from the entrance aperture 624 to the first image sensor 611 and the second image sensor 612 may include the same field of view (FOV).

[0151] In various implementations, the optical subsystem 602 may have a narrow, medium-sized, or wide field of view (FOV). In some examples, the optical subsystem 602 may be from 20 degrees to 50 degrees, from 50 degrees to 100 degrees, from 100 degrees to 150 degrees, from 150 degrees to 180 degrees, or any range formed by these values ​​or larger or smaller values.

[0152] In some embodiments, the relative alignment accuracy between the visual optical images and thermo-optical images captured by the first image sensor 611 and the second image sensor 612 of the imaging system 600 is not affected by mechanical vibration and thermal movement. Therefore, the composite image generated using the first image signal 614 and the second image signal 616 respectively (compared to a composite image generated using image signals received from separate VIS and thermal cameras) can be used to generate an accurate composite image with significantly reduced alignment error and robust image quality. In some examples, the reduced alignment error can lead to more efficient, accurate, and robust sensor fusion processing. In some examples, due to the reduced alignment error, the imaging system 600 can use digital images captured by the optical subsystem 602 to generate more accurate composite images by consuming less power and / or computational resources and in a shorter processing time. Furthermore, since the VIS and LWIR optical paths of the optical subsystem 602 at least share the objective group 608, the imaging system can generate multiple images using fewer optical components and can have a smaller form factor and lower bill of materials (BOM) compared to an imaging system with separate VIS and LWIR imaging systems.

[0153] In some embodiments, the first optical path from the entrance aperture 624 to the first image sensor 611 and the second optical path from the entrance aperture 624 to the second image sensor 612 can provide different F-values. For example, when the first image sensor is a VIS image sensor and the second image sensor is an LWIR image sensor (e.g., a thermal image sensor), the F-value of the first optical path can be greater than the F-value of the second optical path to allow more light to be provided to the LWIR image sensor.

[0154] In some embodiments, one or both of the first image signal 614 and the second image signal 616 may be transmitted to another system (e.g., the autonomous vehicle computing 202f of the autonomous system 202 in vehicle 200) and may be processed independently. In some embodiments, the control and processing system 622 may transmit one or more of the first image signal 614, the second image signal 616, and the third image signal 626 to a navigation system that uses at least the third image signal for navigation in the environment.

[0155] Figure 7 This is a block diagram illustrating an optical subsystem 702 as an example implementation of optical subsystem 602, showing an example arrangement of optical components for the objective lens, first lens group, and second lens group in optical subsystem 602.

[0156] The first imaging lens group 709 and the second imaging lens group 710 of the optical subsystem 702 may each include a plurality of lenses, including positive lenses, negative lenses, doublet lenses, and single lenses, etc., which are configured to transform the light received from the objective lens group 708 to form an image with specified properties on a respective image sensor. In some embodiments, the first imaging lens group 709 and the second imaging lens group 710 may include lens groups arranged and positioned to form a first image and a second image configured according to the characteristics of the first image sensor 611 and the second image sensor 612, respectively.

[0157] In some embodiments, the first image and the second image may include substantially the same portion of the scene captured by the objective lens group 708. In some embodiments, image features of the captured scene may include substantially the same geometric relationships relative to each other in the first and second images. In some embodiments, a first geometric relationship between features of the first image and a first boundary of the first image may be substantially the same as a second geometric relationship between corresponding features of the second image and a second boundary of the second image.

[0158] exist Figure 7 In the example shown, the objective lens group 708 of the optical subsystem 702 includes two single lenses, the first imaging lens group 709 of the optical subsystem includes three doublet lenses and three single lenses, and the second imaging lens group 710 of the optical subsystem 702 includes four single lenses. In some cases, the first and second imaging lens groups may have different spectral transmittances and may be configured to form two images of the same portion of a scene on two different image sensors with different spectral responses, wherein the two images are magnified by approximately the same amount, captured through the same field of view, and include the same angle of view of the scene.

[0159] In some examples, lenses of the first imaging lens group 709 and / or the second imaging lens group 710 may be coupled (e.g., pre-coupled) to the dichroic beam splitter 604. In some examples, using the dichroic beam splitter 604 with pre-coupled single lenses can facilitate the optical alignment of the optical subsystem 702 and improve its accuracy. In some cases, the dichroic beam splitter 604 includes at least one pre-coupled single lens (e.g., a single lens in the first imaging lens group 709). In some cases, the dichroic beam splitter 604 includes two pre-coupled single lenses, respectively, in the first imaging lens group 709 and the second imaging lens group 710. Additionally, pre-coupling a single lens to the dichroic beam splitter 604 can (e.g., by removing or reducing the air gap between the single lens and the dichroic beam splitter) reduce or potentially eliminate Fresnel reflections, thereby improving the optical transmittance from the objective group 708 to the lenses in the respective imaging lens groups.

[0160] In some embodiments, one or both of the VIS optical path from the entrance aperture 624 to the VIS imaging sensor 611 and the LWIR optical path from the entrance aperture 624 to the LWIR imaging sensor 612 may include one or more aperture stops positioned along the respective optical paths. In some cases, the entrance pupil or aperture stop 706 for VIS imaging may be located on the exit surface of the dichroic beam splitter 604 facing the first imaging lens group 709. In some cases, the entrance pupil or aperture stop 706 for LWIR imaging may be located on the entrance surface of the dichroic beam splitter 604 facing the objective lens group 708. In some examples, the first imaging lens group 709 and the corresponding aperture stop may be configured to provide an F-value of 2.4 for the VIS image formed on the first image sensor 611. In some examples, the first imaging lens group 709 and the corresponding aperture stop may be configured to provide an F-value of 1 for the LWIR image formed on the first image sensor 611. However, the embodiments are not limited thereto, and in various embodiments, the F-values ​​associated with the LWIR optical path and the VIS optical path may have other values. For example, the F-value of the LWIR image may be from 0.8 to 1, from 1 to 1.3, from 1.3 to 1.6 or other values, and the F-value of the VIS image may be from 1.2 to 1.5, from 1.5 to 3, from 3 to 8, from 8 to 16, or other values ​​depending on the lighting conditions (e.g., daytime, nighttime, darkness, and brightness).

[0161] In some embodiments, objective lens group 708 may be configured to allow transmission of light having wavelengths across a wide wavelength range spanning at least a portion of the VIS and LWIR wavelength ranges, so that both VIS and LWIR light can be transmitted to dichroic beam splitter 604. Therefore, in some examples, the lens of objective imaging lens group 708 may include a material having low absorption for light having wavelengths within at least a portion of the VIS and LWIR wavelength ranges.

[0162] For example, the lens of the objective imaging lens group 708 may include zinc selenide (ZnS), which can transmit light with wavelengths from 0.4 micrometers to 18 micrometers with relatively low absorption.

[0163] The first imaging lens group 709 can be configured to allow the transmission of light having wavelengths in the VIS wavelength range, and the second imaging lens group 710 can be configured to allow the transmission of light having wavelengths in the LWIR wavelength range. In some embodiments, the first imaging lens group 709 can also be configured to prevent the transmission of light having wavelengths in the LWIR wavelength range, and the second imaging lens group 710 can also be configured to prevent the transmission of light having wavelengths in the VIS wavelength range. In some examples, the lens of the first imaging lens group 709 may include a material having low absorption in the VIS wavelength range and, in some cases, high absorption in the LWIR range. In some examples, the lens of the second imaging lens group 709 may include a material having at least low absorption in the LWIR wavelength range and, in some cases, high absorption in the VIS range. In some examples, the lens of the first imaging lens group 709 may include a coating configured to reduce the reflection of light having wavelengths in the VIS wavelength range and increase the reflection of light having wavelengths in the LWIR (or MWIR) wavelength range. In some examples, the lens of the second imaging lens group 710 may include a coating configured to reduce the reflection of light having wavelengths in the LWIR (or MWIR) wavelength range and increase the reflection of light having wavelengths in the VIS wavelength range.

[0164] As described above, the imaging system 600 can be configured to capture images of a scene (e.g., two images) using light having wavelengths within two or more different wavelength ranges (e.g., two different wavelength ranges) via partially overlapping optical paths and using image sensors (e.g., two image sensors) with different spectral responses. In some implementations, the two different wavelength ranges may be a VIS wavelength range and an LWIR wavelength range, or a VIS wavelength range and a MWIR wavelength range. The two or more images can be digitized and then fused to generate a composite or fused image that includes at least one feature represented at different intensities (visibility or contrast) on both images and / or at least one feature represented on the second image (e.g., a mid-wave IR or long-wave R image) but not on the first image (e.g., a VIS image). In some embodiments, two images formed by the optical subsystem 602 on two image sensors 611, 612 of the imaging system 600 can be aligned such that the geometric relationships between corresponding features on the first and second images are substantially the same, thereby allowing the generation of a synthetic image without any geometric adjustments or with only very minor geometric adjustments (e.g., translation and / or transformation of the coordinates of geometric features).

[0165] In some embodiments, the control and processing system 622 may use a first image sensor 614 and a second image sensor 616 to generate a first digital image and a second digital image corresponding to a first image and a second image projected onto a first image sensor 611 and a second image sensor 612, and fuse the first digital image and the second digital image to generate a composite image. In some such embodiments, the control and processing system may process one or both of the first digital image and the second digital image before combining them into a composite image. In some examples, the control and processing system may process one or both of the first digital image and the second digital image, for example, by adjusting one or more characteristics (such as brightness, contrast, hue, saturation, sharpness, color balance, and noise, etc.) to provide a composite image that includes a balanced combination of features that can be represented on the first digital image and the second digital image with different characteristics. For example, the brightness of areas in the captured scene that appear significantly brighter on the second image compared to the brightest features of the first digital image may be reduced before combining.

[0166] In some embodiments, the optical imaging subsystem 702 may be a wide-angle imaging system configured to capture light within a large field of view (FOV) (e.g., greater than 60 degrees, greater than 80 degrees, or greater than 100 degrees). In some embodiments, the optical imaging subsystem may be an ultra-wide-angle imaging system configured to capture light within a large field of view (FOV) (e.g., greater than 120 degrees, greater than 150 degrees, or greater than 180 degrees). In some embodiments, at least one of the lenses (e.g., the first lens after the entrance aperture 624) may be a fisheye lens or a wide-angle lens.

[0167] Figure 8 An example implementation of an imaging system 600 mounted on a vehicle is illustrated schematically. Figure 6 The imaging system shown includes a first VIS image 802, a second thermal image 804, and a composite image 806 generated from a scene in front of the vehicle by the VIS image sensor and the LWIR image sensor, as well as an image generated by fusing the VIS image and the LWIR image.

[0168] In the example shown, the captured scene (or a portion of a scene captured by imaging system 600) includes oncoming vehicles and pedestrians. In some embodiments, geometric features on the first image 802 may be approximately equal to corresponding geometric features on the second image 804. In some embodiments, the geometric relationship (e.g., ratio) between two geometric features on the first image 802 may be approximately equal to the geometric relationship of corresponding geometric features on the second image 804. In some cases, the first image 802 and the second image 804 may, for example, have approximately the same magnification relative to the captured scene. In some embodiments, the coordinates of features of the captured scene on the first image 802 (e.g., relative to a first frame of the first image 802) may be approximately equal to the coordinates of the same features on the second image 804 (e.g., relative to a second frame of the second image 804). In some such embodiments, alignment / alignment of the first and second frames may automatically align the first image 802 and the second image 804 such that different features of the captured scene in the first image 802 and the second image 804 overlap on the composite image.

[0169] In some cases, objects with higher temperatures may appear brighter and / or have higher contrast in the second image 804, and cooler objects may appear brighter (or have higher contrast) in the first image 802. For example, the images of oncoming vehicles 811a, cameras 813a, and LiDAR systems 815a in the first image 802 may be more visible than the images of oncoming vehicles 811b, cameras 813b, and LiDAR systems 815b in the second image 804. As another example, the image of pedestrians 810b in the second image 804 may be more visible than the image of pedestrians 810a in the first image 802. In some cases, objects in a scene may generate both visible light and thermal radiation, and thus appear with high brightness in both the first image 802 and the second image 804. For example, the images of headlights 806a in the first image 802 and headlights 806b in the second image 804 may both have high brightness.

[0170] In some embodiments, the imaging system 600 may process the first image 802 and the second image 804 to adjust the brightness of different features before generating the composite image 806, so that they are represented with balanced brightness on the composite image 806. For example, the brightness of the pedestrian image 810b on the second image 804 may be reduced at least in part based on the brightness of the oncoming vehicle image 811a, so that the vehicle and pedestrian images 811c and 810c on the composite image provide a balanced image of the scene, which can be used by a user, driver, or computing system to take action, generate commands, or issue warnings to avoid collisions with oncoming vehicles or pedestrians.

[0171] In some embodiments, the processor of the control and processing system 622 may, in response to receiving a VIS image and an LWIR image, or generally two images comprising two different wavelength ranges, generate a composite image of the scene using a first image signal 614 and a second image signal 616 generated by a first image sensor 611 and a second image sensor 612, by registering and fusing the first and second images. In some cases, registration may include aligning the first and second images to ensure that corresponding points in the scene match, and fusing may include combining the aligned first and second images to retain the most relevant information from both the first and second images captured using different modalities (e.g., VIS imaging and IR imaging). In some imaging systems, registration of two images projected onto two different image sensors may additionally include geometric transformations to correct for differences in viewpoint and scale. Advantageously, when two images are captured by the imaging system 600 using optical subsystem 602 (or 702), the alignment step may not include any geometric transformations, or may include geometric transformations that result in significantly smaller changes in the transformed image relative to the original image, compared to the case where the two images are captured by two separate imaging systems or via two non-overlapping optical paths. In some embodiments, the processor of the control and processing system 622 may execute a fusion algorithm stored as machine-readable instructions in the non-transitory memory of the processing system 622. In various implementations, the fusion algorithm may include pixel averaging, wavelet transform, deep learning methods, high-pass filtering, intensity-hue-saturation (IHS) transform, principal component analysis (PCA), and Laplacian pyramid, etc. These algorithms can be used to transform an image into intensity, hue, and saturation components, fuse the intensity components, transform back to the original color space, reduce the dimensionality of the data, extract the most salient features from the image and combine them into a single image, decompose the image into different frequency components, fuse the different frequency components, and then reconstruct the image to preserve both spatial and spectral information. In some embodiments, fusing an IR image (e.g., mid-infrared or long-infrared) and a visible image of a scene can combine the advantages of both modalities to enhance detail and contrast and improve object detection. For example, mid-wave infrared (MWIR) and / or low-wave infrared (LWIR) imaging can capture thermal information, which is useful for detecting thermally marked features, while visible imaging provides high-resolution detail and color information; combining these images enhances overall detail and contrast, making objects easier to identify and analyze. MWIR and LWIR (thermal) imaging can detect objects based on their thermal properties, even under low-light or blurred conditions. When fused with visible images, MWIR and LWIR (thermal) imaging improves the accuracy of target detection and recognition, especially in challenging environments.In some situations, MWIR or LWIR imaging can be less affected by weather conditions (such as fog, smoke, or darkness), so fusing them with visible imaging ensures reliable performance under various environmental conditions, providing day-and-night consistent imaging capabilities. The fusion of different imaging modalities can provide a more comprehensive view of the scene, enhancing situational awareness for applications such as surveillance, security, and search and rescue operations. Furthermore, synthetic images can retain thermal information from MWIR / LWIR and detailed visual information from the visible spectrum, resulting in richer datasets for further analysis and interpretation.

[0172] In some embodiments, a portion or region of a scene may be imaged as a first image region of a first digital image 802 received from a first image sensor 611 (e.g., a VIS image sensor) and a second image region of a second digital image 804 received from a second image sensor 612 (e.g., an LWIR or MWIR image sensor). In some cases, the first normalized brightness of the first image region may differ from the second normalized brightness of the second image region. For example, the second normalized brightness may be greater than the first normalized brightness. In some implementations, the first normalized brightness may be normalized relative to the maximum brightness of the first digital image, and the second normalized brightness may be normalized relative to the maximum brightness of the second digital image. In some embodiments, a control and processing system 622 (e.g., an electronic processor of the control and processing system 622) may identify the first image region on the first digital image 802 and the second image region on the second digital image 804, and generate a composite image 806 based at least in part on the identified first and second image regions (e.g., characteristics of the first and second image regions). For example, control and processing system 622 may compare a first normalized brightness and a second normalized brightness, and generate a composite image 806 based at least in part on the result of such comparison. In some embodiments, in response to determining that the second normalized brightness is greater than the first normalized brightness, control and processing system 622 may enhance the brightness of a corresponding area in the composite image to more closely approximate the second normalized brightness. In some embodiments, in response to determining that the second normalized brightness is greater than the first normalized brightness by a threshold amount, control and processing system 622 may determine that an object or feature in the scene (e.g., an object generating thermal radiation) is not sufficiently visible in the first digital image and generate an alarm signal to indicate the presence of the object. Alternatively or additionally, in response to determining that the second normalized brightness is greater than the first normalized brightness by a threshold amount, control and processing system 622 may provide a warning control signal to the control system (e.g., safety controller 202g) of the autonomous or semi-autonomous vehicle of vehicle 200 to prevent a possible collision with the object.

[0173] In some embodiments, the features, configurations, and systems described above with respect to long-wave infrared (LWIR) image sensors, image formation via an LWIR optical path, and processing of the resulting LWIR image to generate a composite image by fusing visible (VIS) and LWIR images (e.g., in the context of imaging systems 600 and 700) can be implemented based on a mid-wave infrared (MWIR) image sensor, image formation via a MWIR optical path, and processing of the resulting MWIR image to generate a composite image by fusing VIS and MWIR images. Therefore, in some embodiments, the imaging system can be configured to generate a VIS image on a VIS image sensor via a VIS optical path, generate a MWIR image on a MWIR image sensor via a MWIR optical path that partially overlaps with the VIS optical path (e.g., including the same aperture and objective lens group), and fuse the corresponding digital VIS and MWIR images to generate a composite image capturing features of an imaging scene imaged based on VIS and MWIR light. In some such embodiments, the MWIR optical path can be formed using optical components comprising materials transparent in the MWIR wavelength range, and may include surfaces coated with an antireflective layer configured to reduce reflection of light having wavelengths in the MWIR wavelength range. In some cases, for example, lenses in the MWIR optical chain may include zinc selenide (ZnSe), germanium (Ge), GASIR, zinc sulfide (ZnS), calcium fluoride (CaF2), magnesium fluoride (MgF2), and sapphire (Al2O3), etc. In some embodiments, the MIR image sensor may be a MWIR photodetector array, including indium antimonide (InSb) photodiodes, mercury cadmium telluride (MCT) photodetectors, lead salt detectors (PbSe and PbS), and quantum cascade detectors (QCD), etc.

[0174] In some embodiments, the camera 202a of the vehicle 200 may include an imaging system 600, and the control and processing system 622 may be configured to communicate via a bus with a communication device 202e, an autonomous vehicle computing device 202f, and / or a safety controller 202g to transmit a third image signal 626 via the bus to the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g, thereby providing, for example, a composite image or information associated with or extracted from a composite image to a user, user interface, or system that can use the composite image and / or corresponding information to trigger an action. In some embodiments, the control and processing system 622 of the imaging system 600 can transmit the third image signal 626 to the vehicle's perception system (e.g., the perception system 402 in the autonomous vehicle computing 400), and the perception system can use the third image signal 626 and / or the corresponding synthetic digital image (e.g., VIS / thermal digital image) to detect and / or identify physical objects in the environment and classify the physical objects (e.g., classify them as bicycles, vehicles, traffic signs, and / or pedestrians, etc.).

[0175] In various implementations, the components of the optical imaging subsystem 602 can be contained and integrated within a single housing. In some cases, the control and processing system 622, the optical imaging subsystem 602, and the image sensors 611, 612 can be contained and integrated within a single housing. In some cases, the optical imaging subsystem 702 and the image sensors 611, 612 can be contained and integrated within a single housing. In some cases, the components of the optical imaging subsystem 702 can be contained and integrated within a single housing.

[0176] Example Implementation

[0177] The example embodiments described herein have several features, none of which are indispensable or solely responsible for their desired properties. Various example systems and methods are provided below.

[0178] Example 1. An imaging system for a vehicle, comprising:

[0179] The optical subsystem is configured as follows:

[0180] Light from the scene is received through a single entrance aperture.

[0181] The received light is divided into a first part and a second part with different spectral distributions.

[0182] The first image of the scene is formed using the first part, and

[0183] The second part is used to form a second image of the scene;

[0184] A first image sensor, having a first spectral response, is configured to generate a first digital image using the first image of the scene;

[0185] A second image sensor, having a second spectral response, is configured to generate a second digital image using a second image of the scene; and

[0186] At least one processor is configured to fuse the first digital image with the second digital image to generate a composite digital image.

[0187] Wherein, the first image and the second image of the scene include substantially the same portions of the scene, and

[0188] The second peak sensitivity wavelength of the second spectral response is at least two micrometers larger than the first peak sensitivity wavelength of the first spectral response.

[0189] Example 2. The imaging system according to Example 1, wherein the first portion includes wavelengths in the visible wavelength range, and the second portion includes wavelengths in the mid-wave infrared wavelength range, i.e., the MWIR wavelength range, or the long-wave infrared wavelength range, i.e., the LWIR wavelength range.

[0190] Example 3. An imaging system according to any one of Examples 1 to 2, wherein the spectral distribution of the first portion has a mean value in the visible wavelength range, i.e., the VIS wavelength range, and the spectral distribution of the second portion has a mean value in the MWIR wavelength range or the LWIR wavelength range.

[0191] Example 4. An imaging system according to any one of Examples 1 to 2, wherein the first peak sensitivity wavelength is in the visible wavelength range, i.e., the VIS wavelength range, and the second peak sensitivity wavelength is in the MWIR wavelength range or the LWIR wavelength range.

[0192] Example 5. The imaging system according to Example 4, wherein the second peak sensitivity wavelength is in the LWIR wavelength range.

[0193] Example 6. The imaging system according to Example 4, wherein the second peak sensitivity wavelength is in the MWIR wavelength range.

[0194] Example 7. An imaging system according to any one of Examples 1 to 7, wherein the first response bandwidth of the first spectral response includes wavelengths from 400 nm to 700 nm.

[0195] Example 8. The imaging system according to Example 5, wherein the second response bandwidth of the second spectral response includes wavelengths from 9 micrometers to 12 micrometers.

[0196] Example 9. The imaging system according to Example 5, wherein the second response bandwidth of the second spectral response includes wavelengths from 3 micrometers to 5 micrometers.

[0197] Example 10. An imaging system according to any one of Examples 1 to 9, wherein the first image sensor comprises a CMOS sensor.

[0198] Example 11. An imaging system according to any one of Examples 1 to 10, wherein the second image sensor includes a microbolometer.

[0199] Example 12. An imaging system according to any one of Examples 1 to 11, wherein the second image sensor comprises an antimony-based photodetector.

[0200] Example 13. An imaging system according to any one of Examples 1 to 12, wherein the optical subsystem is configured to form a first image of the scene via a first optical path extending from an entrance aperture to a first image sensor, and to form a second image of the scene via a second optical path extending from the entrance aperture to a second image sensor, wherein the first optical path and the second optical path partially overlap.

[0201] Example 14. The imaging system according to Example 13, wherein the first optical path and the second optical path include substantially the same field of view, i.e., FOV.

[0202] Example 15. The imaging system according to Example 13, wherein the F-value of the first optical path is greater than the F-value of the second optical path.

[0203] Example 16. The imaging system according to Example 13, wherein, for light having wavelengths in the VIS wavelength range, the optical transmittance of the first optical path is greater than the optical transmittance of the second optical path, and for light having wavelengths in one or both of the MWIR and LWIR wavelength ranges, the optical transmittance of the second optical path is greater than the optical transmittance of the first optical path.

[0204] Example 17. An imaging system according to Example 13, wherein the optical subsystem includes an objective optical chain, a first optical chain, a second optical chain, and a dichroic beam splitter, the dichroic beam splitter being used to receive light from the scene through the entrance aperture and the objective optical chain, transmit the first portion to the first optical chain, and redirect the second portion to the second optical chain.

[0205] Example 18. The imaging system according to Example 17, wherein the first optical path includes the first optical chain, the second optical path includes the second optical chain, and the overlapping portion of the first optical path and the second optical path includes the objective lens optical chain.

[0206] Example 19. An imaging system according to Example 13, wherein the first optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the VIS wavelength range compared to light having wavelengths in one or both of the MWIR and LWIR wavelength ranges.

[0207] Example 20. An imaging system according to Example 13, wherein the second optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the MWIR or LWIR wavelength range compared to light having wavelengths in the VIS wavelength range.

[0208] Example 21. An imaging system according to any one of Examples 1 to 20, wherein the imaging system includes a camera of the navigation system of the vehicle.

[0209] Example 22. An imaging system according to any one of Examples 1 to 21, wherein, in order to generate a synthetic digital image, the at least one processor is configured to compare the first digital image and the second digital image.

[0210] Example 23. An imaging system according to any one of Examples 1 to 22, wherein a first normalized brightness of at least a first image region of the first digital image is less than a second normalized brightness of a corresponding second image region of the second digital image.

[0211] Example 24. The imaging system according to Example 23, wherein each of the first normalized brightness and the second normalized brightness is normalized relative to the maximum brightness of the corresponding digital image.

[0212] Example 25. An imaging system according to Example 23, wherein the at least one processor is configured to identify a first image region on the first digital image and a second image region on the second digital image, and to generate a composite image based on the identified first and second image regions.

[0213] Example 26. An imaging system according to Example 23, wherein the at least one processor is further configured to compare the first normalized brightness and the second normalized brightness, and to generate a synthetic image based at least in part on the comparison.

[0214] Example 27. An imaging system according to any one of Examples 1 to 26, wherein the at least one processor is further configured to receive the first digital image and the second digital image substantially simultaneously.

[0215] Example 28. An imaging system according to any one of Examples 1 to 27, wherein the spectral distribution of the second portion has an average value in the MWIR wavelength range, and the at least one processor is further configured to operate the imaging system synchronously with the LiDAR system of the vehicle to receive at least the second digital image during periods when the LiDAR system does not emit a beam.

[0216] Example 29. An imaging system according to any one of Examples 1 to 28, wherein the first image and the second image include the same magnification relative to the scene.

[0217] Example 30. An imaging system according to any one of Examples 1 to 29, wherein the first image and the second image include substantially the same viewpoint of the scene.

[0218] Example 31. An imaging system according to any one of Examples 1 to 30, wherein the imaged features of the scene include substantially the same geometric relationship relative to each other in the first image and the second image.

[0219] Example 32. A method comprising:

[0220] Optical subsystems of imaging systems utilizing vehicles:

[0221] Light from the scene is received through a single entrance aperture;

[0222] The received light is divided into a first part and a second part with different spectral distributions;

[0223] A first image is formed on a first image sensor having a first spectral response using the first portion of the received light;

[0224] A second image is formed on a second image sensor having a second spectral response using the second portion of the received light, wherein the first image and the second image include the same imaging portion of the scene with the same magnification;

[0225] Using at least one processor of the imaging system, at least a first digital image from the first image sensor and at least a second digital image from the second image sensor are received; and

[0226] Using the at least one processor, a composite digital image is generated by fusing the first digital image and the second digital image.

[0227] Wherein, the first image and the second image of the scene include substantially the same portions of the scene, and

[0228] The second peak sensitivity wavelength of the second spectral response is at least 2 micrometers larger than the first peak sensitivity wavelength of the first spectral response.

[0229] Example 33. The method according to Example 32, wherein the first portion includes wavelengths in the visible wavelength range, and the second portion includes wavelengths in the mid-wave infrared wavelength range, i.e., the MWIR wavelength range, or the long-wave infrared wavelength range, i.e., the LWIR wavelength range.

[0230] Example 34. The method according to Example 32, wherein the spectral distribution of the first portion has a mean value in the visible wavelength range, i.e., the VIS wavelength range, and the spectral distribution of the second portion has a mean value in the MWIR wavelength range or the LWIR wavelength range.

[0231] Example 35. The method according to Example 32, wherein the first peak sensitivity wavelength is in the visible wavelength range, i.e., the VIS wavelength range, and the second peak sensitivity wavelength is in the MWIR wavelength range or the LWIR wavelength range.

[0232] Example 36. The method according to Example 35, wherein the second peak sensitivity wavelength is in the LWIR wavelength range.

[0233] Example 37. The method according to Example 35, wherein the second peak sensitivity wavelength is in the MWIR wavelength range.

[0234] Example 38. The method according to Example 32, wherein the first response bandwidth of the first spectral response includes wavelengths from 400 nm to 700 nm.

[0235] Example 39. The method according to Example 36, wherein the second response bandwidth of the second spectral response includes wavelengths from 9 micrometers to 12 micrometers.

[0236] Example 40. The method according to Example 36, wherein the second response bandwidth of the second spectral response includes wavelengths from 3 micrometers to 5 micrometers.

[0237] Example 41. The method according to Example 1, wherein the first image sensor comprises a CMOS sensor.

[0238] Example 42. The method according to Example 1, wherein the second image sensor includes a microbolometer.

[0239] Example 43. The method according to Example 1, wherein the second image sensor includes an antimony-based photodetector.

[0240] Example 44. The method according to Example 1, wherein the optical subsystem is configured to form a first image of the scene via a first optical path extending from the entrance aperture to the first image sensor, and to form a second image of the scene via a second optical path extending from the entrance aperture to the second image sensor, wherein the first optical path and the second optical path partially overlap.

[0241] Example 45. The method according to Example 44, wherein the first optical path and the second optical path include substantially the same field of view, i.e., FOV.

[0242] Example 46. The method according to Example 44, wherein the F-value of the first optical path is greater than the F-value of the second optical path.

[0243] Example 47. The method according to Example 44, wherein, for light having wavelengths in the VIS wavelength range, the optical transmittance of the first optical path is greater than the optical transmittance of the second optical path, and for light having wavelengths in one or both of the MWIR and LWIR wavelength ranges, the optical transmittance of the second optical path is greater than the optical transmittance of the first optical path.

[0244] Example 48. The method according to Example 44, wherein the optical subsystem includes an objective optical chain, a first optical chain, a second optical chain, and a dichroic beam splitter, the dichroic beam splitter being used to receive light from the scene through the entrance aperture and the objective optical chain, transmit the first portion to the first optical chain, and redirect the second portion to the second optical chain.

[0245] Example 49. The method according to Example 48, wherein the first optical path includes the first optical chain, the second optical path includes the second optical chain, and the overlapping portion of the first optical path and the second optical path includes the objective lens optical chain.

[0246] Example 50. The method according to Example 44, wherein the first optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the VIS wavelength range compared to light having wavelengths in one or both of the MWIR and LWIR wavelength ranges.

[0247] Example 51. The method according to Example 44, wherein the second optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the MWIR or LWIR wavelength range compared to light having wavelengths in the VIS wavelength range.

[0248] Example 52. The method according to Example 32, wherein the imaging system includes a camera of the vehicle's navigation system.

[0249] Example 53. The method according to Example 32, wherein generating the synthetic digital image includes: comparing the first digital image and the second digital image.

[0250] Example 54. The method according to Example 32, wherein a first normalized brightness of at least a first image region of the first digital image is less than a second normalized brightness of a corresponding second image region of the second digital image.

[0251] Example 55. The method according to Example 54, wherein each of the first normalized brightness and the second normalized brightness is normalized relative to the maximum brightness of the corresponding digital image.

[0252] Example 56. The method according to Example 54 further includes: using the at least one processor of the imaging system to identify a first image region on the first digital image and a second image region on the second digital image, and generating a composite image based on the identified first image region and second image region.

[0253] Example 57. The method according to Example 54 further includes: using the at least one processor to compare the first normalized brightness and the second normalized brightness, and generating a synthetic image based at least in part on the comparison.

[0254] Example 58. The method according to Example 32, wherein receiving at least a first digital image from the first image sensor and at least a second digital image from the second image sensor using at least one processor of the imaging system comprises receiving the first digital image and the second digital image substantially simultaneously.

[0255] Example 59. The method according to Example 32, wherein the spectral distribution of the second portion has a mean value within the MWIR wavelength range, and the method includes: operating the imaging system synchronously with the LiDAR system of the vehicle using the at least one processor of the imaging system to receive at least the second digital image during periods when the LiDAR system does not emit a beam.

[0256] Example 60. The method according to Example 32, wherein the first image and the second image include the same magnification relative to the scene.

[0257] Example 61. The method according to Example 32, wherein the first image and the second image include substantially the same viewpoint of the scene.

[0258] Example 62. The method according to Example 32, wherein the imaged features of the scene include substantially the same geometric relationship relative to each other on the first image and the second image.

[0259] Example 63. An imaging system for a vehicle, comprising:

[0260] The optical subsystem is configured as follows:

[0261] Receive light from the scene through a single entrance aperture.

[0262] The received light is divided into a first part and a second part with different spectral distributions.

[0263] The first image of the scene is formed using the first part, and

[0264] The second part is used to form a second image of the scene;

[0265] A first image sensor, having a first spectral response, is configured to generate a first image signal using the first image of the scene; and

[0266] A second image sensor, having a second spectral response, is configured to generate a second image signal using a second image of the scene.

[0267] Wherein, the first image and the second image of the scene include substantially the same portions of the scene, and

[0268] The second peak sensitivity wavelength of the second spectral response is at least two micrometers larger than the first peak sensitivity wavelength of the first spectral response.

[0269] Example 64. The imaging system according to Example 63 further includes at least one processor configured to receive the first image signal and the second image signal and use the first image signal and the second image signal to generate a synthetic digital image signal by fusing the first image and the second image, and to generate a third image signal including the synthetic digital image signal.

[0270] Example 65. The imaging system according to Example 63, wherein the first portion includes wavelengths in the visible wavelength range, and the second portion includes wavelengths in the mid-wave infrared wavelength range, i.e., the MWIR wavelength range, or the long-wave infrared wavelength range, i.e., the LWIR wavelength range.

[0271] Example 66. The imaging system according to Example 63, wherein the spectral distribution of the first portion has a mean value in the visible wavelength range, i.e., the VIS wavelength range, and the spectral distribution of the second portion has a mean value in the MWIR wavelength range or the LWIR wavelength range.

[0272] Example 67. The imaging system according to Example 63, wherein the first peak sensitivity wavelength is in the visible wavelength range, i.e., the VIS wavelength range, and the second peak sensitivity wavelength is in the MWIR wavelength range or the LWIR wavelength range.

[0273] Example 68. The imaging system according to Example 66, wherein the second peak sensitivity wavelength is in the LWIR wavelength range.

[0274] Example 69. The imaging system according to Example 66, wherein the second peak sensitivity wavelength is in the MWIR wavelength range.

[0275] Example 70. An imaging system according to Example 63, wherein the first response bandwidth of the first spectral response includes wavelengths from 400 nm to 700 nm.

[0276] Example 71. An imaging system according to Example 67, wherein the second response bandwidth of the second spectral response includes wavelengths from 9 micrometers to 12 micrometers.

[0277] Example 72. The imaging system according to Example 67, wherein the second response bandwidth of the second spectral response includes wavelengths from 3 micrometers to 5 micrometers.

[0278] Example 73. The imaging system according to Example 63, wherein the first image sensor includes a CMOS sensor.

[0279] Example 74. The imaging system according to Example 63, wherein the second image sensor includes a microbolometer.

[0280] Example 75. The imaging system according to Example 63, wherein the second image sensor includes an antimony-based photodetector.

[0281] Example 76. An imaging system according to Example 63, wherein the optical subsystem is configured to form a first image of the scene via a first optical path extending from an entrance aperture to a first image sensor, and to form a second image of the scene via a second optical path extending from the entrance aperture to a second image sensor, wherein the first optical path and the second optical path partially overlap.

[0282] Example 77. An imaging system according to Example 76, wherein the first optical path and the second optical path include substantially the same field of view, i.e., FOV.

[0283] Example 78. The imaging system according to Example 76, wherein the F-value of the first optical path is greater than the F-value of the second optical path.

[0284] Example 79. An imaging system according to Example 76, wherein, for light having wavelengths in the VIS wavelength range, the optical transmittance of the first optical path is greater than the optical transmittance of the second optical path, and for light having wavelengths in one or both of the MWIR and LWIR wavelength ranges, the optical transmittance of the second optical path is greater than the optical transmittance of the first optical path.

[0285] Example 80. An imaging system according to Example 76, wherein the optical subsystem includes an objective optical chain, a first optical chain, a second optical chain, and a dichroic beam splitter, the dichroic beam splitter being used to receive light from the scene through the entrance aperture and the objective optical chain, transmit the first portion to the first optical chain, and redirect the second portion to the second optical chain.

[0286] Example 81. An imaging system according to Example 80, wherein the first optical path includes the first optical chain, the second optical path includes the second optical chain, and the overlapping portion of the first optical path and the second optical path includes the objective lens optical chain.

[0287] Example 82. An imaging system according to Example 76, wherein the first optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the VIS wavelength range compared to light having wavelengths in one or both of the MWIR and LWIR wavelength ranges.

[0288] Example 83. An imaging system according to Example 76, wherein the second optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the MWIR or LWIR wavelength range compared to light having wavelengths in the VIS wavelength range.

[0289] Example 84. An imaging system according to Example 63, wherein the imaging system includes a camera of the navigation system of the vehicle.

[0290] Example 85. An imaging system according to Example 64, wherein, in order to generate a synthetic digital image, the at least one processor is configured to compare the first image and the second image.

[0291] Example 86. An imaging system according to Example 85, wherein a first normalized brightness of at least a first image region of the first image is less than a second normalized brightness of a corresponding second image region of the second image.

[0292] Example 87. An imaging system according to Example 86, wherein each of the first normalized brightness and the second normalized brightness is normalized relative to the maximum brightness of the corresponding image.

[0293] Example 88. An imaging system according to Example 86, wherein the at least one processor is configured to identify a first image region on the first image and a second image region on the second image, and to generate a synthetic digital image based on the identified first and second image regions.

[0294] Example 89. An imaging system according to Example 86, wherein the at least one processor is further configured to compare the first normalized brightness and the second normalized brightness, and to generate a synthetic image based at least in part on the comparison.

[0295] Example 90. An imaging system according to Example 64, wherein the at least one processor is further configured to receive the first image and the second image substantially simultaneously.

[0296] Example 91. An imaging system according to Example 63, wherein the spectral distribution of the second portion has an average value in the MWIR wavelength range, and the at least one processor is further configured to operate the imaging system synchronously with the LiDAR system of the vehicle to receive at least the second image during periods when the LiDAR system does not emit a beam.

[0297] Example 92. An imaging system according to Example 63, wherein the first image and the second image include the same magnification relative to the scene.

[0298] Example 93. An imaging system according to Example 63, wherein the first image and the second image include substantially the same viewpoint of the scene.

[0299] Example 94. An imaging system according to Example 63, wherein the imaged features of the scene include substantially the same geometric relationship relative to each other in the first image and the second image.

[0300] Example 95. An imaging system according to any one of the examples above, wherein the optical system and the image sensor are contained within a single housing.

[0301] the term

[0302] In this description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0303] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly stated otherwise, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly stated otherwise, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0304] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0305] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0306] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

Claims

1. An imaging system for a vehicle, comprising: The optical subsystem is configured as follows: Receive light from the scene through a single entrance aperture. The received light is divided into a first part and a second part with different spectral distributions. The first image of the scene is formed using the first part, and The second part is used to form a second image of the scene; A first image sensor, having a first spectral response, is configured to generate a first digital image using the first image of the scene; A second image sensor, having a second spectral response, is configured to generate a second digital image using the second image of the scene; as well as At least one processor is configured to fuse the first digital image with the second digital image to generate a composite digital image. Wherein, the first image and the second image of the scene include substantially the same portions of the scene, and The second peak sensitivity wavelength of the second spectral response is at least two micrometers larger than the first peak sensitivity wavelength of the first spectral response.

2. The imaging system according to claim 1, wherein, The first part includes wavelengths within the visible wavelength range, and the second part includes wavelengths within the mid-wave infrared wavelength range, i.e., the MWIR wavelength range, or the long-wave infrared wavelength range, i.e., the LWIR wavelength range.

3. The imaging system according to any one of claims 1 to 2, wherein, The spectral distribution of the first part has an average value within the visible wavelength range, i.e., the VIS wavelength range, and the spectral distribution of the second part has an average value within the MWIR wavelength range or the LWIR wavelength range.

4. The imaging system according to any one of claims 1 to 3, wherein, The first peak sensitivity wavelength is in the visible wavelength range, i.e., the VIS wavelength range, and the second peak sensitivity wavelength is in the MWIR wavelength range or the LWIR wavelength range.

5. The imaging system according to claim 4, wherein, The second peak sensitivity wavelength is within the LWIR wavelength range.

6. The imaging system according to claim 4, wherein, The second peak sensitivity wavelength is within the MWIR wavelength range.

7. The imaging system according to claim 1, wherein, The first response bandwidth of the first spectral response includes wavelengths from 400 nm to 700 nm.

8. The imaging system according to claim 5, wherein, The second response bandwidth of the second spectral response includes wavelengths from 9 micrometers to 12 micrometers.

9. The imaging system according to claim 5, wherein, The second response bandwidth of the second spectral response includes wavelengths from 3 micrometers to 5 micrometers.

10. The imaging system according to any one of claims 1 to 9, wherein, The first image sensor includes a CMOS sensor.

11. The imaging system according to any one of claims 1 to 10, wherein, The second image sensor includes a microbolometer.

12. The imaging system according to any one of claims 1 to 11, wherein, The second image sensor includes an antimony-based photodetector.

13. The imaging system according to any one of claims 1 to 12, wherein, The optical subsystem is configured to form a first image of the scene via a first optical path extending from the entrance aperture to the first image sensor, and to form a second image of the scene via a second optical path extending from the entrance aperture to the second image sensor, wherein the first optical path and the second optical path partially overlap.

14. The imaging system according to claim 13, wherein, The first optical path and the second optical path include approximately the same field of view (FOV).

15. The imaging system according to claim 13, wherein, The F-value of the first optical path is greater than the F-value of the second optical path.

16. The imaging system according to claim 13, wherein, For light having wavelengths within the VIS wavelength range, the optical transmittance of the first optical path is greater than that of the second optical path, and for light having wavelengths within one or both of the MWIR and LWIR wavelength ranges, the optical transmittance of the second optical path is greater than that of the first optical path.

17. The imaging system according to claim 13, wherein, The optical subsystem includes an objective optical chain, a first optical chain, a second optical chain, and a dichroic beam splitter. The dichroic beam splitter is used to receive light from the scene through the entrance aperture and the objective optical chain, transmit the first portion to the first optical chain, and redirect the second portion to the second optical chain.

18. The imaging system according to claim 17, wherein, The first optical path includes the first optical chain, the second optical path includes the second optical chain, and the overlapping portion of the first optical path and the second optical path includes the objective lens optical chain.

19. The imaging system according to claim 13, wherein, The first optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the VIS wavelength range compared to light having wavelengths in one or both of the MWIR and LWIR wavelength ranges.

20. The imaging system according to claim 13, wherein, The second optical path includes at least one optical surface, wherein, for light having wavelengths in the MWIR or LWIR wavelength range, the at least one optical surface is coated to provide greater optical transmittance compared to light having wavelengths in the VIS wavelength range.

21. The imaging system according to any one of claims 1 to 20, wherein, The imaging system includes a camera in the navigation system of the vehicle.

22. The imaging system according to any one of claims 1 to 21, wherein, In order to generate a synthetic digital image, the at least one processor is configured to compare the first digital image and the second digital image.

23. The imaging system according to any one of claims 1 to 22, wherein, The first normalized brightness of at least a first image region of the first digital image is less than the second normalized brightness of the corresponding second image region of the second digital image.

24. The imaging system according to claim 23, wherein, Each of the first normalized brightness and the second normalized brightness is normalized relative to the maximum brightness of the corresponding digital image.

25. The imaging system according to claim 23, wherein, The at least one processor is configured to identify the first image region on the first digital image and the second image region on the second digital image, and to generate a composite image based on the identified first and second image regions.

26. The imaging system according to claim 23, wherein, The at least one processor is also configured to compare the first normalized brightness and the second normalized brightness, and to generate a synthetic image based at least in part on the comparison.

27. The imaging system according to any one of claims 1 to 26, wherein, The at least one processor is also configured to receive the first digital image and the second digital image substantially simultaneously.

28. The imaging system according to any one of claims 1 to 27, wherein, The spectral distribution of the second portion has an average value within the MWIR wavelength range, and the at least one processor is further configured to operate the imaging system synchronously with the LiDAR system of the vehicle to receive at least the second digital image during periods when the LiDAR system does not emit a beam.

29. The imaging system according to any one of claims 1 to 28, wherein, The first image and the second image include the same magnification relative to the scene.

30. The imaging system according to any one of claims 1 to 29, wherein, The first image and the second image include approximately the same viewpoint of the scene.

31. The imaging system according to any one of claims 1 to 30, wherein, The imaged features of the scene include approximately the same geometric relationships relative to each other in the first and second images.

32. A method comprising: Optical subsystems of imaging systems utilizing vehicles: Light from the scene is received through a single entrance aperture; The received light is divided into a first part and a second part with different spectral distributions; A first image is formed on a first image sensor having a first spectral response using the first portion of the received light; A second image is formed on a second image sensor having a second spectral response using the second portion of the received light, wherein the first image and the second image include the same imaging portion of the scene with the same magnification; Using at least one processor of the imaging system, at least a first digital image from the first image sensor and at least a second digital image from the second image sensor are received; and Using the at least one processor, a composite digital image is generated by fusing the first digital image and the second digital image. Wherein, the first image and the second image of the scene include substantially the same parts of the scene. The second peak sensitivity wavelength of the second spectral response is at least 2 micrometers larger than the first peak sensitivity wavelength of the first spectral response.

33. The method according to claim 32, wherein, The first part includes wavelengths within the visible wavelength range, and the second part includes wavelengths within the mid-wave infrared wavelength range, i.e., the MWIR wavelength range, or the long-wave infrared wavelength range, i.e., the LWIR wavelength range.

34. The method according to any one of claims 32 to 33, wherein, The spectral distribution of the first part has an average value within the visible wavelength range, i.e., the VIS wavelength range, and the spectral distribution of the second part has an average value within the MWIR wavelength range or the LWIR wavelength range.

35. The method according to any one of claims 32 to 34, wherein, The first peak sensitivity wavelength is in the visible wavelength range, i.e., the VIS wavelength range, and the second peak sensitivity wavelength is in the MWIR wavelength range or the LWIR wavelength range.

36. The method according to claim 35, wherein, The second peak sensitivity wavelength is within the LWIR wavelength range.

37. The method of claim 35, wherein, The second peak sensitivity wavelength is within the MWIR wavelength range.

38. The method according to any one of claims 32 to 37, wherein, The first response bandwidth of the first spectral response includes wavelengths from 400 nm to 700 nm.

39. The method according to claim 36, wherein, The second response bandwidth of the second spectral response includes wavelengths from 9 micrometers to 12 micrometers.

40. The method of claim 36, wherein, The second response bandwidth of the second spectral response includes wavelengths from 3 micrometers to 5 micrometers.

41. The method according to any one of claims 32 to 40, wherein, The first image sensor includes a CMOS sensor.

42. The method according to any one of claims 32 to 41, wherein, The second image sensor includes a microbolometer.

43. The method according to any one of claims 32 to 42, wherein, The second image sensor includes an antimony-based photodetector.

44. The method according to any one of claims 32 to 43, wherein, The optical subsystem is configured to form a first image of the scene via a first optical path extending from the entrance aperture to the first image sensor, and to form a second image of the scene via a second optical path extending from the entrance aperture to the second image sensor, wherein the first optical path and the second optical path partially overlap.

45. The method according to claim 44, wherein, The first optical path and the second optical path include approximately the same field of view (FOV).

46. ​​The method of claim 44, wherein, The F-value of the first optical path is greater than the F-value of the second optical path.

47. The method of claim 44, wherein, For light having wavelengths within the VIS wavelength range, the optical transmittance of the first optical path is greater than that of the second optical path, and for light having wavelengths within one or both of the MWIR and LWIR wavelength ranges, the optical transmittance of the second optical path is greater than that of the first optical path.

48. The method according to claim 44, wherein, The optical subsystem includes an objective optical chain, a first optical chain, a second optical chain, and a dichroic beam splitter. The dichroic beam splitter is used to receive light from the scene through the entrance aperture and the objective optical chain, transmit the first portion to the first optical chain, and redirect the second portion to the second optical chain.

49. The method according to claim 48, wherein, The first optical path includes the first optical chain, the second optical path includes the second optical chain, and the overlapping portion of the first optical path and the second optical path includes the objective lens optical chain.

50. The method of claim 44, wherein, The first optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the VIS wavelength range compared to light having wavelengths in one or both of the MWIR and LWIR wavelength ranges.

51. The method according to claim 44, wherein, The second optical path includes at least one optical surface, wherein, for light having wavelengths in the MWIR or LWIR wavelength range, the at least one optical surface is coated to provide greater optical transmittance compared to light having wavelengths in the VIS wavelength range.

52. The method according to any one of claims 32 to 51, wherein, The imaging system includes a camera in the navigation system of the vehicle.

53. The method according to any one of claims 32 to 52, wherein, Generating the synthetic digital image includes comparing the first digital image and the second digital image.

54. The method according to any one of claims 32 to 53, wherein, The first normalized brightness of at least a first image region of the first digital image is less than the second normalized brightness of the corresponding second image region of the second digital image.

55. The method according to claim 54, wherein, Each of the first normalized brightness and the second normalized brightness is normalized relative to the maximum brightness of the corresponding digital image.

56. The method of claim 54, further comprising: Using the at least one processor of the imaging system, a first image region on the first digital image and a second image region on the second digital image are identified, and a composite image is generated based on the identified first and second image regions.

57. The method of claim 54, further comprising: Using the at least one processor, the first normalized brightness and the second normalized brightness are compared, and a synthetic image is generated based at least in part on the comparison.

58. The method according to any one of claims 32 to 57, wherein, Receiving at least a first digital image from the first image sensor and at least a second digital image from the second image sensor using at least one processor of the imaging system includes receiving the first digital image and the second digital image substantially simultaneously.

59. The method according to any one of claims 32 to 58, wherein, The spectral distribution of the second portion has an average value within the MWIR wavelength range, and the method includes: using the at least one processor of the imaging system to operate the imaging system synchronously with the LiDAR system of the vehicle to receive at least the second digital image during periods when the LiDAR system does not emit a beam.

60. The method according to any one of claims 32 to 59, wherein, The first image and the second image include the same magnification relative to the scene.

61. The method according to any one of claims 32 to 60, wherein, The first image and the second image include approximately the same viewpoint of the scene.

62. The method according to any one of claims 32 to 61, wherein, The imaged features of the scene include approximately the same geometric relationships relative to each other in the first and second images.

63. An imaging system for a vehicle, comprising: The optical subsystem is configured as follows: Receive light from the scene through a single entrance aperture; The received light is divided into a first part and a second part with different spectral distributions. The first image of the scene is formed using the first part, and The second part is used to form a second image of the scene; A first image sensor having a first spectral response and configured to generate a first image signal using the first image of the scene; as well as A second image sensor, having a second spectral response, is configured to generate a second image signal using a second image of the scene. Wherein, the first image and the second image of the scene include substantially the same portions of the scene, and The second peak sensitivity wavelength of the second spectral response is at least two micrometers larger than the first peak sensitivity wavelength of the first spectral response.

64. The imaging system of claim 63, further comprising at least one processor configured to receive the first image signal and the second image signal and use the first image signal and the second image signal to generate a composite digital image signal by fusing the first image and the second image, and to generate a third image signal including the composite digital image signal.

65. The imaging system according to any one of claims 63 to 64, wherein, The first part includes wavelengths within the visible wavelength range, and the second part includes wavelengths within the mid-wave infrared wavelength range, i.e., the MWIR wavelength range, or the long-wave infrared wavelength range, i.e., the LWIR wavelength range.

66. The imaging system according to any one of claims 63 to 65, wherein, The spectral distribution of the first part has an average value within the visible wavelength range, i.e., the VIS wavelength range, and the spectral distribution of the second part has an average value within the MWIR wavelength range or the LWIR wavelength range.

67. The imaging system according to any one of claims 63 to 66, wherein, The first peak sensitivity wavelength is in the visible wavelength range, i.e., the VIS wavelength range, and the second peak sensitivity wavelength is in the MWIR wavelength range or the LWIR wavelength range.

68. The imaging system according to any one of claims 63 to 67, wherein, The second peak sensitivity wavelength is within the LWIR wavelength range.

69. The imaging system according to any one of claims 63 to 68, wherein, The second peak sensitivity wavelength is within the MWIR wavelength range.

70. The imaging system according to any one of claims 63 to 69, wherein, The first response bandwidth of the first spectral response includes wavelengths from 400 nm to 700 nm.

71. The imaging system according to claim 67, wherein, The second response bandwidth of the second spectral response includes wavelengths from 9 micrometers to 12 micrometers.

72. The imaging system according to claim 67, wherein, The second response bandwidth of the second spectral response includes wavelengths from 3 micrometers to 5 micrometers.

73. The imaging system according to any one of claims 63 to 72, wherein, The first image sensor includes a CMOS sensor.

74. The imaging system according to any one of claims 63 to 73, wherein, The second image sensor includes a microbolometer.

75. The imaging system according to any one of claims 63 to 74, wherein, The second image sensor includes an antimony-based photodetector.

76. The imaging system according to any one of claims 63 to 75, wherein, The optical subsystem is configured to form a first image of the scene via a first optical path extending from the entrance aperture to the first image sensor, and to form a second image of the scene via a second optical path extending from the entrance aperture to the second image sensor, wherein the first optical path and the second optical path partially overlap.

77. The imaging system according to claim 76, wherein, The first optical path and the second optical path include approximately the same field of view (FOV).

78. The imaging system according to claim 76, wherein, The F-value of the first optical path is greater than the F-value of the second optical path.

79. The imaging system according to claim 76, wherein, For light having wavelengths within the VIS wavelength range, the optical transmittance of the first optical path is greater than that of the second optical path, and for light having wavelengths within one or both of the MWIR and LWIR wavelength ranges, the optical transmittance of the second optical path is greater than that of the first optical path.

80. The imaging system according to claim 76, wherein, The optical subsystem includes an objective optical chain, a first optical chain, a second optical chain, and a dichroic beam splitter. The dichroic beam splitter is used to receive light from the scene through the entrance aperture and the objective optical chain, transmit the first portion to the first optical chain, and redirect the second portion to the second optical chain.

81. The imaging system according to claim 80, wherein, The first optical path includes the first optical chain, the second optical path includes the second optical chain, and the overlapping portion of the first optical path and the second optical path includes the objective lens optical chain.

82. The imaging system according to claim 76, wherein, The first optical path includes at least one optical surface, wherein the at least one optical surface is coated to provide greater optical transmittance for light having wavelengths in the VIS wavelength range compared to light having wavelengths in one or both of the MWIR and LWIR wavelength ranges.

83. The imaging system according to claim 76, wherein, The second optical path includes at least one optical surface, wherein, for light having wavelengths in the MWIR or LWIR wavelength range, the at least one optical surface is coated to provide greater optical transmittance compared to light having wavelengths in the VIS wavelength range.

84. The imaging system according to any one of claims 63 to 83, wherein, The imaging system includes a camera in the navigation system of the vehicle.

85. The imaging system according to any one of claims 63 to 84, wherein, In order to generate a synthetic digital image, the at least one processor is configured to compare the first image and the second image.

86. The imaging system according to claim 85, wherein, The first normalized brightness of at least a first image region of the first image is less than the second normalized brightness of the corresponding second image region of the second image.

87. The imaging system according to claim 86, wherein, Each of the first normalized brightness and the second normalized brightness is normalized relative to the maximum brightness of the corresponding image.

88. The imaging system according to claim 86, wherein, The at least one processor is configured to identify the first image region on the first image and the second image region on the second image, and to generate a synthetic digital image based on the identified first and second image regions.

89. The imaging system according to claim 86, wherein, The at least one processor is also configured to compare the first normalized brightness and the second normalized brightness, and to generate a synthetic image based at least in part on the comparison.

90. The imaging system according to any one of claims 63 to 89, wherein, The at least one processor is also configured to receive the first image and the second image substantially simultaneously.

91. The imaging system according to any one of claims 63 to 90, wherein, The spectral distribution of the second portion has an average value within the MWIR wavelength range, and the at least one processor is further configured to operate the imaging system synchronously with the LiDAR system of the vehicle to receive at least the second image during periods when the LiDAR system does not emit a beam.

92. The imaging system according to any one of claims 63 to 91, wherein, The first image and the second image include the same magnification relative to the scene.

93. The imaging system according to any one of claims 63 to 92, wherein, The first image and the second image include approximately the same viewpoint of the scene.

94. The imaging system according to any one of claims 63 to 93, wherein, The imaged features of the scene include approximately the same geometric relationships relative to each other in the first and second images.

Images