Performance degradation detection in hybrid sounding and ranging systems

Abstract

A hybrid detection and ranging (HyDAR) system configured for detecting signals having a plurality of wavelengths is provided. The system comprises a laser light source which provides a laser signal; an aperture window; one or more steering mechanisms configured to guide the laser signals toward the aperture window, receive a first return light signal formed based on at least a portion of the laser signals provided by the laser light source, and receiving a second return light signal formed by light provided by one or more light sources external to the HyDAR system. The system further includes a multi-mode sensor including a light detection and ranging (LiDAR) sensor and an image sensor. The point cloud data and the image data at least partially implement temporal and spatial synchronization on the hardware level of the HyDAR system. The system further includes a controller configured to detect one or more degradation factors that affect performance of the HyDAR system.

Description

Cross-references to applications related to performance degradation detection in hybrid detection and ranging systems

[0001] This application claims priority to the following patent applications: U.S. Non-Provisional Patent Application Serial No. 18 / 818,553, filed August 28, 2024, entitled "Performance Degradation Detection in Hybrid Detection and Ranging System"; U.S. Provisional Patent Application Serial No. 63 / 537,309, filed September 8, 2023, entitled "HyDAR Sensor Blockage Detection"; U.S. Provisional Patent Application Serial No. 63 / 537,320, filed September 8, 2023, entitled "HyDAR Adaptive Laser Control for External LiDAR Interference Detection"; and U.S. Intrinsic Patent Application Serial No. 63 / 537,320, filed September 22, 2023, entitled "HyDAR Intrinsic Calibration Degradation Detection". U.S. Provisional Patent Application Serial No. 63 / 539,930 entitled “HyDAR Dynamic Calibration”; U.S. Provisional Patent Application Serial No. 63 / 539,935 entitled “HyDAR Dynamic Calibration”; U.S. Provisional Patent Application Serial No. 63 / 539,985 entitled “HyDAR Sensor Blockage Detection”; and U.S. Provisional Patent Application Serial No. 63 / 540,022 entitled “HyDAR Adaptive Laser Control for External LiDAR Interference Detection”, filed September 22, 2023. The contents of these applications are incorporated herein by reference in their entirety for all purposes. Technical Field

[0002] This disclosure generally relates to optical detection, and more specifically to the detection of performance degradation associated with hybrid detection and ranging (HyDAR) systems configured to detect signals having multiple wavelengths. Background Technology

[0003] Light detection and ranging (LiDAR) systems use light pulses to create images or point clouds of the external environment. LiDAR systems can be scanning or non-scanning. Some typical scanning LiDAR systems include a light source, a light emitter, a light steering system, and a photodetector. The light source generates a light beam, which, when emitted from the LiDAR system, is guided in a specific direction by the light steering system. When the emitted beam is scattered or reflected by an object, a portion of the scattered or reflected light returns to the LiDAR system as a returned light pulse. The photodetector detects the returned light pulse. Using the difference between the time it takes to detect the returned light pulse and the time it takes for the corresponding light pulse in the beam to be emitted, the LiDAR system can determine the distance to an object based on the speed of light. This distance determination technique is called Time-of-Flight (ToF). The light steering system can guide the light beam along different paths to allow the LiDAR system to scan the surrounding environment and generate images or point clouds. Typical non-scanning LiDAR systems illuminate the entire field of view (FOV) rather than scanning it. An example of a non-scanning LiDAR system is a flash LiDAR, which can also use ToF technology to measure the distance to objects. The LiDAR system can also use techniques other than time-of-flight and scanning to measure the surrounding environment.

[0004] Hybrid detection and ranging (HyDAR) systems can include LiDAR systems and one or more other types of sensors, such as one or more cameras, one or more radar sensors, one or more ultrasonic sensors, and / or other sensors. LiDAR systems and one or more other types of sensors can be integrated into a HyDAR system to form a compact, multi-mode sensor. Summary of the Invention

[0005] The multi-mode sensor of a HyDAR system can be configured to be compact so that it can be easily mounted on a mobile platform, such as a vehicle. When the vehicle is in operation for extended periods, one or more degradation factors may exist that affect the performance of the HyDAR system. These degradation factors may include, for example, at least partial aperture window occlusion, interference signals from one or more interfering light sources, external calibration degradation of the HyDAR system, and internal calibration degradation of the HyDAR system. Degradation factors may also affect the LiDAR sensor within the HyDAR system. Therefore, this disclosure describes techniques and methods for addressing degradation factors in HyDAR systems including LiDAR sensors. Regarding window occlusion, the techniques described herein can determine the type of occlusion at a high level in real time with limited computational requirements. One embodiment of the method uses a machine learning algorithm to detect occlusion and trigger cleaning and clearing operations on the HyDAR system or its external attachments.

[0006] Interference signals are another degradation factor for HyDAR systems. To avoid such interference signals, corresponding portions of the point cloud can be discarded, the laser source in the HyDAR system can be turned off, the laser power can be increased to enhance the light intensity, making it greater than the interfering light signal; and / or the steering mechanism of the LiDAR sensor can be adjusted towards the direction of the interfering light source to tune the FOV of the LiDAR sensor. This disclosure also provides methods relating to image sensors and LiDAR sensors in HyDAR systems to improve detection performance. The methods described herein can be applied to detect interfering light signals (e.g., direct sunlight, laser interference, high beams from oncoming vehicles, etc.) and adaptively control the HyDAR system to minimize performance degradation.

[0007] Another factor contributing to HyDAR performance degradation is related to the external calibration of the HyDAR system, which includes a LiDAR sensor. This disclosure provides techniques and methods for dynamically detecting and monitoring external calibration degradation. The HyDAR system provided for this purpose does not require synchronization or data fusion at the hardware level or above. Using the disclosed HyDAR system, point cloud data from the LiDAR sensor and image data from the image sensor are at least partially synchronized temporally and spatially at the hardware level of the HyDAR system, thus eliminating the need for extensive software-based computations, or even any software computations, transformations, or data fusion. Furthermore, if degradation is within a predefined threshold, the HyDAR system can self-adjust while continuing to operate on a mobile platform.

[0008] Another factor contributing to the performance degradation of HyDAR systems is the degradation of internal calibration over time. This disclosure provides techniques and methods for detecting and monitoring internal calibration degradation associated with misaligned internal components using LiDAR sensors and image sensors within a HyDAR system. In some examples, the HyDAR system can trigger warnings and self-correct its internal calibration. This can also help identify the root cause of performance degradation using training datasets derived from HyDAR system data. Benefiting from performance degradation detection, the methods described herein can at least monitor, correct, and / or report degradation of the following key internal calibration parameters: distance correction; oscillating mirror offset; multifaceted mirror offset; and geometric parameters.

[0009] Embodiments of the present invention are described below. In various embodiments of the present invention, a Hybrid Detection and Ranging (HyDAR) system configured to detect signals having multiple wavelengths is provided. The HyDAR system includes: a laser source providing a laser signal; an aperture window; and one or more steering mechanisms configured to: direct the laser signal toward the aperture window; receive a first returned light signal formed based on at least a portion of the laser signal provided by the laser source; and receive a second returned light signal formed by light provided by one or more sources external to the HyDAR system. The HyDAR system further includes a multi-mode sensor comprising a LiDAR sensor and an image sensor. The LiDAR sensor is configured to detect the first returned light signal to obtain one or more frames of point cloud data. The image sensor is configured to detect the second returned light signal to obtain one or more frames of image data. The point cloud data and image data are at least partially synchronized temporally and spatially at the hardware level of the HyDAR system. The HyDAR system further includes a controller configured to perform the following operations: detect one or more degradation factors affecting the performance of the HyDAR system; and in response to the detection of the one or more degradation factors, adjust the equipment configuration or operating conditions of the HyDAR system to eliminate or reduce the impact of the degradation factors. Attached Figure Description

[0010] This application can be best understood by referring to the embodiments described below in conjunction with the accompanying drawings, in which the same parts are indicated by the same reference numerals.

[0011] Figure 1 illustrates one or more exemplary LiDAR systems that are installed or included in a motor vehicle.

[0012] Figure 2 is a block diagram illustrating the interaction between an exemplary LiDAR system and several other systems, including a vehicle perception and planning system.

[0013] Figure 3 is a block diagram illustrating an exemplary LiDAR system.

[0014] Figure 4 is a block diagram illustrating a multi-mode detection system according to some embodiments.

[0015] Figure 5A is a block diagram illustrating an exemplary fiber-optic-based laser source.

[0016] Figure 5B is a block diagram illustrating an exemplary semiconductor-based laser source.

[0017] Figure 6 illustrates an exemplary light collection and distribution device according to some embodiments.

[0018] Figure 7 illustrates an exemplary signal separation device according to some embodiments.

[0019] Figure 8 illustrates the configuration of an integrated sensor for a multi-mode sensor according to various embodiments.

[0020] Figure 9 illustrates exemplary package configurations of integrated sensors for multi-mode sensors according to various embodiments.

[0021] Figures 10A to 10C illustrate exemplary LiDAR systems that use pulse signals to measure the distance to objects positioned in the field of view (FOV).

[0022] Figure 11 is a block diagram illustrating exemplary apparatus for implementing systems, devices, and methods in various embodiments.

[0023] Figure 12A is a block diagram illustrating an exemplary HyDAR system according to various embodiments.

[0024] Figure 12B is a block diagram illustrating an exemplary HyDAR system configured to detect aperture window occlusion according to various embodiments.

[0025] Figure 13 is a flowchart illustrating a method for aperture window occlusion for detecting a HyDAR system according to various embodiments.

[0026] Figure 14A is a graph illustrating a comparison of the return signal strength between a covered aperture window and an uncovered aperture window according to various embodiments.

[0027] Figure 14B is a diagram illustrating at least partial aperture window occlusion and various types of occlusion in a HyDAR system according to various embodiments.

[0028] Figures 15A to 15D are flowcharts illustrating various methods for detecting aperture window occlusion in HyDAR systems according to various embodiments.

[0029] Figure 16 is a diagram illustrating a HyDAR system receiving interference signals provided by one or more interfering light sources according to various embodiments.

[0030] Figures 17A and 17B are flowcharts illustrating various methods for detecting interference signals caused by one or more interfering light sources according to various embodiments.

[0031] Figure 17C is a diagram illustrating an exemplary process for adjusting the steering mechanism of a HyDAR system to avoid the position of one or more interfering light sources according to various embodiments.

[0032] Figure 17D is a diagram illustrating specific areas of discarded scan lines according to various embodiments to correspond to the positions of one or more interfering light sources.

[0033] Figure 17E is a diagram illustrating various processes for adjusting a HyDAR system according to various embodiments to avoid interfering with optical signals or to increase the light intensity of the HyDAR system to reduce the effects of interfering optical signals.

[0034] Figure 18 is a block diagram illustrating a mobile platform with a HyDAR system according to various embodiments, wherein the HyDAR system is externally degraded relative to the mobile platform.

[0035] Figures 19A and 19B are flowcharts illustrating methods for detecting external calibration degradation of a HyDAR system mounted on a mobile platform, according to various embodiments.

[0036] Figures 19C to 19G are diagrams illustrating exemplary methods for detecting external calibration degradation of a HyDAR system using parallel line features extending along the road surface, according to various embodiments.

[0037] Figure 19H is a diagram illustrating an exemplary method for detecting external calibration degradation of a HyDAR system using curved features extending along the road surface, according to various embodiments.

[0038] Figure 20A is a flowchart illustrating a method for detecting external calibration degradation based on both point cloud data and image data according to various embodiments.

[0039] Figure 20B is a diagram illustrating various predefined stationary targets in image data according to various embodiments, based on which external calibration degradation detection is performed.

[0040] Figures 20C to 20D are flowcharts illustrating exemplary methods for detecting external calibration degradation of a HyDAR system using feature sets based on image data and LiDAR data, according to various embodiments.

[0041] Figure 21 is a block diagram illustrating a HyDAR system that may have internal calibration degradation according to various embodiments.

[0042] Figures 22A and 22B are flowcharts illustrating exemplary methods for detecting internal calibration degradation in a HyDAR system according to various embodiments.

[0043] Figure 23 is a diagram illustrating the improved LiDAR resolution using image data according to various embodiments. Detailed Implementation

[0044] To provide a more thorough understanding of the various embodiments of the present invention, numerous specific details, such as specific configurations, parameters, and examples, are set forth in the following description. However, it should be understood that this description is not intended to limit the scope of the invention, but rather to provide a better description of exemplary embodiments.

[0045] Throughout the specification and claims, unless the context clearly indicates otherwise, the following terms have the meaning explicitly associated herein: as used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may be the same embodiment. Therefore, as described below, various embodiments of the invention can be readily combined without departing from the scope or spirit of this disclosure.

[0046] As used herein, the term “or” is an inclusive “or” operator and is equivalent to the term “and / or”, unless the context clearly indicates otherwise.

[0047] The term "based on" is not exclusive and allows for the use of additional factors not described unless explicitly stated in the context.

[0048] As used herein, unless the context otherwise requires, the term "coupled to" is intended to include both direct coupling (where two coupled elements are in contact with each other) and indirect coupling (where at least one additional element is located between the two elements). Therefore, the terms "coupled to" and "coupled with" are used synonymously. In the context of a networked environment where two or more components or devices are capable of exchanging data, the terms "coupled to" and "coupled with" are also used to indicate possible "communicable coupling" with via one or more intermediate devices. Components or devices can be optical, mechanical, and / or electrical.

[0049] Although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the examples in the various descriptions, a first optical signal may be referred to as a second optical signal, and similarly, a second optical signal may be referred to as a first optical signal. Both the first optical signal and the second optical signal can be optical signals, and in some cases, they can be separate and distinct optical signals.

[0050] Furthermore, throughout the specification, the meanings of “an,” “a,” and “the” include the plural, and the meaning of “in” can include both “in” and “on”.

[0051] While some embodiments given herein constitute a single combination of inventive elements, it should be understood that the inventive subject matter is considered to include all possible combinations of the disclosed elements. Thus, if one embodiment includes elements A, B, and C, and another embodiment includes elements B and D, the inventive subject matter is also considered to include other remaining combinations of A, B, C, or D, even if not explicitly discussed herein. Furthermore, the transitional term "comprising" means having a component or element, or those components or elements. As used herein, the transitional term "comprising" is inclusive or open-ended and does not exclude additional, unlisted elements or method steps.

[0052] As used in the description herein and throughout the claims thereafter, when a system, engine, server, device, module or other computing element is described as being configured to perform or execute functions on data in memory, the meaning of “configured to” or “programmed to” is defined as one or more processors or cores of the computing element being programmed with a set of software instructions stored in the memory of the computing element to perform that set of functions on target data or data objects stored in memory.

[0053] It should be noted that any language for computers should be understood to include any suitable combination of computing devices or network platforms, including servers, interfaces, systems, databases, agents, peers, engines, controllers, modules, or other types of computing devices, individually or in combination. It should be understood that computing devices include processors configured to execute software instructions stored on tangible, non-transitory computer-readable storage media, such as hard disk drives, FPGAs, PLAs, solid-state drives, RAM, flash memory, ROM, or any other volatile or non-volatile storage devices. These software instructions configure or program the computing device to provide roles, responsibilities, or other functions, as discussed below with respect to the disclosed apparatus. Furthermore, the disclosed technology can be embodied as a computer program product including a non-transitory computer-readable medium storing software instructions that cause a processor to perform the disclosed steps associated with the implementation of computer-based algorithms, processes, methods, or other instructions. In some embodiments, various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on HTTP, HTTPS, AES, public-key exchange, web service APIs, known financial transaction protocols, or other electronic information exchange methods. Data exchange between devices can be carried out through the following: packet-switched networks, the Internet, LAN, WAN, VPN or other types of packet-switched networks; circuit-switched networks; cell-switched networks; or other types of networks.

[0054] As described above, a HyDAR system may include an integrated LiDAR sensor and one or more other sensors to form a multimode sensor. This multimode sensor can be configured to be compact so that it can be easily mounted on a mobile platform (such as a vehicle). A HyDAR system may have one or more degradation factors that affect its performance over time. These degradation factors may include, for example, at least partial aperture window occlusion, interference signals from one or more interfering light sources, external calibration degradation of the HyDAR system, and internal calibration degradation of the HyDAR system. It should be understood that these examples are not limiting and are merely examples of common degradation factors for HyDAR systems.

[0055] Starting with window occlusion, a factor contributing to performance degradation, the aperture window of most LiDAR systems on the market today is susceptible to unintended occlusion. This is especially true for LiDAR sensors mounted on vehicles. For example, the aperture window may be exposed to raindrops, fog condensation, debris, etc., falling onto it. This disclosure describes a technique for detecting at least partial aperture window occlusion of a LiDAR sensor, image sensor, or HyDAR system using a multi-mode sensor of a HyDAR system. The technique described herein can also determine the type of occlusion at a high level in real time with limited computational requirements. One embodiment of the method uses a machine learning algorithm to detect occlusion and trigger a cleaning and clearing operation on the HyDAR system or its external accessories.

[0056] Turning to the next performance degradation factor, LiDAR sensors in a HyDAR system may be sensitive to external interfering light signals, including, for example, direct sunlight, light from other LiDAR systems, vehicle headlights, streetlights, etc. Interfering light signals from external sources can cause performance degradation in the HyDAR system. For example, interfering light signals may cause anomalous behavior and damage to the sensor itself and surrounding equipment. This disclosure provides exemplary configurations of multi-mode sensors in HyDAR systems to reduce or prevent false detections or performance degradation. For example, by using the image sensor of a multi-mode sensor in a HyDAR system, different types of interfering light sources, even malicious laser jammers in the near field, can be identified. In one example, to avoid such interfering signals, corresponding portions of the point cloud can be discarded, the laser source in the HyDAR system can be turned off, the laser power can be increased to improve the transmitted light intensity, thereby improving the signal-to-noise ratio (SNR); and / or the steering mechanism of the LiDAR sensor can be adjusted towards the direction of the interfering light source to tune the FOV of the LiDAR sensor. This disclosure also provides methods for image sensors and LiDAR sensors in HyDAR systems to improve detection performance. The method can be applied to detect interfering light signals (e.g., direct sunlight, laser interference, high beams from oncoming vehicles, etc.) and adaptively control the HyDAR system to minimize performance degradation.

[0057] Another factor contributing to HyDAR performance degradation relates to the external calibration of HyDAR systems, which include LiDAR sensors and image sensors. In the autonomous driving industry, external calibration of forward-looking cameras and LiDAR sensors can be a challenging problem, requiring precise synchronization between the two sensors and specific calibration settings. As mobile platforms (e.g., vehicles) operate under harsh environmental conditions throughout their lifespan, external calibration parameters can also degrade. External calibration measures the relationship between the HyDAR system and the mobile platform on which it is mounted. When a HyDAR system is first manufactured and installed on a mobile platform, it is calibrated to ensure correct positioning and orientation so that it can accurately detect objects within its field of view (FOV). Over time, the external calibration of a HyDAR system can change due to various factors such as environmental conditions, wear and tear, and user error. Consequently, the external calibration of the HyDAR system may degrade over time, potentially negatively impacting the system's performance.

[0058] This disclosure provides techniques and methods for dynamically detecting and monitoring external calibration degradation. The HyDAR system provided for this purpose does not require synchronization or data fusion at the hardware level or above. Using the disclosed HyDAR system, point cloud data from the LiDAR sensor and image data from the image sensor are at least partially synchronized temporally and spatially at the hardware level of the HyDAR system, thus eliminating the need for extensive software computations, conversions, or data fusion, or even software-level synchronization altogether. Furthermore, if degradation is within a predefined threshold, the HyDAR system can self-adjust while continuing to operate on a mobile platform.

[0059] Another factor contributing to the performance degradation of HyDAR systems is the degradation of internal calibration over time. The internal calibration of a HyDAR system involves calibrating the alignment of its internal components, including the LiDAR sensor and image sensor. HyDAR systems operate outdoors in all weather conditions and may encounter vibration, wear, and external damage. These harsh environmental conditions can sometimes cause deviations in the internal calibration of the HyDAR system, leading to performance degradation. This disclosure uses the LiDAR sensor and image sensor in the HyDAR system to detect and monitor internal calibration degradation associated with misaligned internal components. In some examples, the HyDAR system can trigger warnings and self-correct its internal calibration. This can also help identify the root cause of performance degradation using a training dataset from HyDAR system data. Benefiting from performance degradation detection, the methods described herein can at least monitor, correct, and report degradation of the following key internal calibration parameters: distance correction; vibrating mirror offset; polygonal mirror offset; and the geometric parameters of HyDAR system components.

[0060] Embodiments of the present invention are described below. In various embodiments of the present invention, a Hybrid Detection and Ranging (HyDAR) system configured to detect signals having multiple wavelengths is provided. The HyDAR system includes: a laser source providing a laser signal; an aperture window; and one or more steering mechanisms configured to: direct the laser signal toward the aperture window; receive a first return light signal formed based on at least a portion of the laser signal provided by the laser source; and receive a second return light signal formed by light provided by one or more sources external to the HyDAR system. The HyDAR system further includes a multi-mode sensor comprising a LiDAR sensor and an image sensor. The LiDAR sensor is configured to detect the first return light signal to obtain one or more frames of point cloud data. The image sensor is configured to detect the second return light signal to obtain one or more frames of image data. The point cloud data and image data are at least partially synchronized temporally and spatially at the hardware level of the HyDAR system. The HyDAR system further includes a controller configured to: detect one or more degradation factors affecting the performance of the HyDAR system; and, in response to the detection of the one or more degradation factors, adjust the device configuration or operating conditions of the HyDAR system to eliminate or reduce the impact of the degradation factors. The exemplary HyDAR system and various techniques for detecting performance degradation will be described in more detail below, beginning with a description of LiDAR systems typically included in HyDAR systems.

[0061] Figure 1 illustrates one or more exemplary LiDAR systems 110 and 120A-120I set up or included in a motor vehicle 100. The vehicle 100 can be a car, SUV, truck, train, van, bicycle, motorcycle, tricycle, bus, motorized scooter, tram, ship, boat, underwater vehicle, airplane, helicopter, unmanned aerial vehicle (UAV), spacecraft, etc. The motor vehicle 100 can be a vehicle with any level of automation. For example, the motor vehicle 100 can be a partially automated vehicle, a highly automated vehicle, a fully automated vehicle, or a driverless vehicle. A partially automated vehicle can perform some driving functions without human driver intervention. For example, a partially automated vehicle can perform blind spot monitoring, lane keeping and / or lane changing operations, automatic emergency braking, intelligent cruise control and / or traffic following, etc. Some operations of a partially automated vehicle may be limited to specific applications or driving scenarios (e.g., limited to highway driving). A highly automated vehicle can generally perform all the operations of a partially automated vehicle, but with fewer limitations. Highly automated vehicles can also detect their own limits while operating the vehicle and, if necessary, request the driver to take over control. Fully automated vehicles can perform all vehicle operations without driver intervention, but can also detect their own limits and, if necessary, request driver intervention. Driverless vehicles can operate autonomously without any driver intervention.

[0062] In a typical configuration, the motor vehicle 100 includes one or more LiDAR systems 110 and 120A-120I. Each of the LiDAR systems 110 and 120A-120I can be a scanning-based LiDAR system and / or a non-scanning LiDAR system (e.g., a flash LiDAR). A scanning-based LiDAR system scans one or more beams in one or more directions (e.g., horizontal and vertical) to detect objects in the field of view (FOV). A non-scanning-based LiDAR system emits a laser to illuminate the FOV without scanning. For example, a flash LiDAR is a type of non-scanning-based LiDAR system. A flash LiDAR can emit a laser, illuminating the FOV simultaneously using a single light pulse or a beam of light.

[0063] LiDAR systems are commonly used sensors in at least partially automated vehicles. In one embodiment, as shown in FIG1, a motor vehicle 100 may include a single LiDAR system 110 (e.g., without LiDAR systems 120A-120I) positioned at the highest point of the vehicle (e.g., on the top of the vehicle). Positioning the LiDAR system 110 on the top of the vehicle facilitates 360-degree scanning around the vehicle 100. In some other embodiments, the motor vehicle 100 may include multiple LiDAR systems, including two or more of systems 110 and / or 120A-120I. As shown in FIG1, in one embodiment, multiple LiDAR systems 110 and / or 120A-120I are attached to the vehicle 100 at different locations on the vehicle. For example, LiDAR system 120A is attached to vehicle 100 at the right front corner; LiDAR system 120B is attached to vehicle 100 at the front center position; LiDAR system 120C is attached to vehicle 100 at the left front corner; LiDAR system 120D is attached to vehicle 100 at the right rearview mirror; LiDAR system 120E is attached to vehicle 100 at the left rearview mirror; LiDAR system 120F is attached to vehicle 100 at the rear center position; LiDAR system 120G is attached to vehicle 100 at the right rear corner; LiDAR system 120H is attached to vehicle 100 at the left rear corner; and / or LiDAR system 120I is attached to vehicle 100 at the center towards the rear end (e.g., the rear end of the top of the vehicle). It should be understood that one or more LiDAR systems can be distributed and attached to the vehicle in any desired manner, and Figure 1 illustrates only one embodiment. As another example, LiDAR systems 120D and 120E can be attached to the B-pillar of vehicle 100 instead of the rearview mirror. As another example, LiDAR system 120B can be attached to the windshield of vehicle 100 instead of the front bumper.

[0064] In some embodiments, LiDAR systems 110 and 120A-120I are independent LiDAR systems, each with its own laser source, control electronics, transmitter, receiver, and / or steering mechanism. In other embodiments, some of LiDAR systems 110 and 120A-120I may share one or more components, thereby forming a distributed sensor system. In one example, optical fiber is used to deliver laser light from a centralized laser source to all LiDAR systems. For example, system 110 (or another system located at the center of vehicle 100 or anywhere else) includes a light source, transmitter, and photodetector, but no steering mechanism. System 110 may distribute transmitted light to each of systems 120A-120I. The transmitted light may be distributed via optical fiber. Optical connectors may be used to couple optical fiber to each of systems 110 and 120A-120I. In some examples, one or more of systems 120A-120I include a steering mechanism, but no light source, transmitter, or photodetector. The steering mechanism may include one or more movable mirrors, such as one or more polygonal mirrors, one or more single-plane mirrors, one or more multi-plane mirrors, etc. Embodiments of the light source, emitter, steering mechanism, and photodetector will be described in more detail below. Via the steering mechanism, one or more systems in systems 120A-120I scan light into one or more corresponding fields of view (FOVs) and receive the corresponding return light. The return light is formed by the scattering or reflection of transmitted light by one or more objects in the FOV. Systems 120A-120I may also include collecting lenses and / or other optics to focus and / or guide the return light into an optical fiber, which delivers the received return light to system 110. System 110 includes one or more photodetectors for detecting the received return light. In some examples, system 110 is located inside a vehicle, thus placing it in a temperature-controlled environment, while one or more systems 120A-120I may be at least partially exposed to the external environment.

[0065] Figure 2 is a block diagram 200 illustrating the interaction between an onboard LiDAR system 210 and several other systems, including a vehicle perception and planning system 220. The LiDAR system 210 can be mounted on or integrated into a vehicle. The LiDAR system 210 includes sensors that scan the surrounding environment with laser light to measure the distance, angle, and / or velocity of objects. Based on the scattered light returning to the LiDAR system 210, it can generate sensor data (e.g., image data or 3D point cloud data) representing the perceived external environment.

[0066] LiDAR system 210 may include one or more of short-range LiDAR sensors, mid-range LiDAR sensors, and long-range LiDAR sensors. Short-range LiDAR sensors measure objects up to approximately 20-50 meters away. They can be used, for example, to monitor nearby moving objects (e.g., pedestrians crossing the street in a school zone), for parking assistance applications, etc. Mid-range LiDAR sensors measure objects up to approximately 70-200 meters away. They can be used, for example, to monitor road intersections, assist merging or exiting highways, etc. Long-range LiDAR sensors measure objects located at 200 meters and above. Long-range LiDAR sensors are typically used when vehicles are traveling at high speeds (e.g., on highways), meaning the vehicle's control system may only have a few seconds (e.g., 6-8 seconds) to respond to any situation detected by the LiDAR sensors. As shown in Figure 2, in one embodiment, LiDAR sensor data can be provided to vehicle perception and planning system 220 via communication path 213 for further processing and control of vehicle operation. Communication path 213 can be any wired or wireless communication link capable of transmitting data.

[0067] Referring again to Figure 2, in some embodiments, other vehicle sensors 230 are configured to provide additional sensor data, either alone or in conjunction with the LiDAR system 210. These other vehicle sensors 230 may include, for example, one or more cameras 232, one or more radars 234, one or more ultrasonic sensors 236, and / or other sensors 238. Cameras 232 may capture images and / or video of the vehicle's external environment. Cameras 232 may capture, for example, high-definition (HD) video with millions of pixels per frame. Cameras include image sensors that facilitate the generation of monochrome or color images and videos. Color information may be important in interpreting data in certain situations (e.g., interpreting images of traffic lights). Color information may not be available from other sensors, such as LiDAR or radar sensors. Cameras 232 may include one or more of narrow-focal-length cameras, wide-focal-length cameras, side-view cameras, infrared cameras, fisheye cameras, etc. Image and / or video data generated by cameras 232 may also be provided via communication path 233 to the vehicle perception and planning system 220 for further processing and control of vehicle operation. The communication path 233 can be any wired or wireless communication link capable of transmitting data. The camera 232 can be mounted or integrated into any location on the vehicle (e.g., rearview mirror, pillar, front grille, and / or rear bumper, etc.).

[0068] Other vehicle-mounted sensors 230 may also include a radar sensor 234. The radar sensor 234 uses radio waves to determine the distance, angle, and speed of an object. The radar sensor 234 generates electromagnetic waves in the radio or microwave spectrum. These electromagnetic waves are reflected by the object, and some of the reflected waves return to the radar sensor, providing information about the object's position and speed. The radar sensor 234 may include one or more of short-range, medium-range, and long-range radars. Short-range radar measures objects at a distance of approximately 0.1–30 meters from the radar. Short-range radar is useful for detecting objects located near vehicles (such as other vehicles, buildings, walls, pedestrians, cyclists, etc.). Short-range radar can be used for blind spot detection, lane change assistance, providing rear-end collision warnings, parking assistance, and emergency braking. Medium-range radar measures objects at a distance of approximately 30–80 meters from the radar. Long-range radar measures objects located at approximately 80–200 meters. Medium-range and / or long-range radar can be used for, for example, traffic tracking, adaptive cruise control, and / or automatic braking on highways. Sensor data generated by radar sensor 234 can also be provided to vehicle perception and planning system 220 via communication path 233 for further processing and control of vehicle operation. Radar sensor 234 can be installed or integrated into any location on the vehicle (e.g., rearview mirror, pillar, front grille and / or rear bumper, etc.).

[0069] Other onboard sensors 230 may also include ultrasonic sensors 236. Ultrasonic sensors 236 use sound waves or pulses to measure objects located outside the vehicle. Sound waves generated by ultrasonic sensors 236 are emitted into the surrounding environment. At least some of the emitted waves are reflected by objects and return to ultrasonic sensors 236. Based on the returned signals, the distance to the object can be calculated. Ultrasonic sensors 236 can be used, for example, to check blind spots, identify parking spaces, and provide lane change assistance in traffic. Sensor data generated by ultrasonic sensors 236 can also be provided via communication path 233 to the vehicle perception and planning system 220 for further processing and control of vehicle operation. Ultrasonic sensors 236 can be mounted or integrated into any location on the vehicle (e.g., rearview mirrors, pillars, front grille, and / or rear bumper, etc.).

[0070] In some embodiments, one or more other sensors 238 may be attached to the vehicle and may also generate sensor data. Other sensors 238 may include, for example, a Global Positioning System (GPS), an Inertial Measurement Unit (IMU), etc. The sensor data generated by the other sensors 238 may also be provided to the vehicle perception and planning system 220 via communication path 233 for further processing and control of vehicle operation. It should be understood that communication path 233 may include one or more communication links for transmitting data between the various sensors 230 and the vehicle perception and planning system 220.

[0071] In some embodiments, as shown in FIG2, sensor data from other vehicle-mounted sensors 230 can be provided to the vehicle-mounted LiDAR system 210 via communication path 231. The LiDAR system 210 can process the sensor data from the other vehicle-mounted sensors 230. For example, sensor data from camera 232, radar sensor 234, ultrasonic sensor 236, and / or other sensors 238 can be correlated or fused with the sensor data from the LiDAR system 210, thereby at least partially offloading the sensor fusion process performed by the vehicle perception and planning system 220. It should be understood that other configurations can also be implemented to transmit and process sensor data from various sensors (e.g., data can be transmitted to a cloud or edge computing service provider for processing, and the processing results can then be transmitted back to the vehicle perception and planning system 220 and / or the LiDAR system 210).

[0072] Referring again to Figure 2, in some embodiments, sensors on other vehicles 250 are used to provide additional sensor data, either alone or in conjunction with the LiDAR system 210. For example, two or more nearby vehicles may have their own LiDAR sensors, cameras, radar sensors, ultrasonic sensors, etc. Nearby vehicles can transmit and share sensor data with each other. Communication between vehicles is also referred to as V2V (vehicle-to-vehicle) communication. For example, as shown in Figure 2, sensor data generated by other vehicles 250 can be transmitted to the vehicle perception and planning system 220 and / or the onboard LiDAR system 210 via communication paths 253 and / or 251, respectively. Communication paths 253 and 251 can be any wired or wireless communication link capable of transmitting data.

[0073] Sharing sensor data facilitates better perception of the external environment of a vehicle. For example, the first vehicle may not detect a pedestrian approaching it from behind a second vehicle. The second vehicle can share sensor data related to the pedestrian with the first vehicle, allowing the first vehicle additional reaction time to avoid a collision. In some embodiments, data generated by sensors on other vehicles 250, similar to data generated by sensor 230, can be correlated or fused with sensor data generated by LiDAR system 210 (or other LiDAR systems located in other vehicles), thereby at least partially offloading the sensor fusion process performed by vehicle perception and planning system 220.

[0074] In some embodiments, the intelligent infrastructure system 240 is used to provide sensor data, either alone or in conjunction with the LiDAR system 210. Certain infrastructure can be configured to communicate with vehicles to relay information, and vice versa. Communication between vehicles and infrastructure is generally referred to as V2I (vehicle-to-infrastructure) communication. For example, the intelligent infrastructure system 240 may include intelligent traffic lights that can communicate their status to approaching vehicles with messages such as "turns yellow in 5 seconds." The intelligent infrastructure system 240 may also include its own LiDAR system installed near an intersection, enabling it to transmit traffic monitoring information to vehicles. For example, a vehicle turning left at an intersection may not have sufficient sensing capabilities because some of its own sensors may be blocked by traffic from the opposite direction. In this case, the sensors of the intelligent infrastructure system 240 can provide useful data to the left-turning vehicle. This data may include, for example, traffic conditions, information about objects in the direction the vehicle is turning, traffic light status, and predictions. The sensor data generated by the intelligent infrastructure system 240 can be provided to the vehicle perception and planning system 220 and / or the onboard LiDAR system 210 via communication paths 243 and / or 241, respectively. Communication paths 243 and / or 241 can include any wired or wireless communication links capable of transmitting data. For example, sensor data from the intelligent infrastructure system 240 can be transmitted to the LiDAR system 210 and correlated or fused with the sensor data generated by the LiDAR system 210, thereby at least partially offloading the sensor fusion process performed by the vehicle perception and planning system 220. The above-described V2V and V2I communications are examples of vehicle-to-X (V2X) communications, where “X” represents any other device, system, sensor, infrastructure, etc., that can share data with the vehicle.

[0075] Referring again to Figure 2, the vehicle perception and planning system 220 receives sensor data from one or more of the LiDAR system 210, other onboard sensors 230, other vehicles 250, and / or intelligent infrastructure systems 240 via various communication paths. In some embodiments, different types of sensor data are correlated and / or fused by a sensor fusion subsystem 222. For example, the sensor fusion subsystem 222 can generate a 360-degree model using multiple images or videos captured by multiple cameras positioned at different locations on the vehicle. The sensor fusion subsystem 222 obtains sensor data from different types of sensors and uses the combined data to perceive the environment more accurately. For example, the onboard camera 232 may not capture a clear image because it is directly facing the sun or a light source (e.g., the headlights of another vehicle at night). The LiDAR system 210 may not be significantly affected, and therefore the sensor fusion subsystem 222 can combine the sensor data provided by the camera 232 and the LiDAR system 210, and use the sensor data provided by the LiDAR system 210 to compensate for the unclear image captured by the camera 232. As another example, in rainy or foggy weather, radar sensor 234 may perform better than camera 232 or LiDAR system 210. Accordingly, sensor fusion subsystem 222 can use sensor data provided by radar sensor 234 to compensate for sensor data provided by camera 232 or LiDAR system 210.

[0076] In other examples, sensor data generated by other onboard sensors 230 may have lower resolution (e.g., radar sensor data) and therefore may need to be correlated and verified by a LiDAR system 210, which typically has higher resolution. For example, radar sensor 234 may detect a manhole cover (also known as a maintenance hatch cover) as an object approaching a vehicle. Due to the low resolution of radar sensor 234, vehicle perception and planning system 220 may not be able to determine whether the object is an obstacle that the vehicle needs to avoid. Therefore, high-resolution sensor data generated by LiDAR system 210 can be used to correlate and verify that the object is a manhole cover and will not cause damage to the vehicle.

[0077] The vehicle perception and planning system 220 further includes an object classifier 223. Using raw sensor data and / or related / fused data provided by the sensor fusion subsystem 222, the object classifier 223 can use any computer vision technique to detect and classify objects and estimate their positions. In some embodiments, the object classifier 223 can use machine learning-based techniques to detect and classify objects. Examples of machine learning-based techniques include algorithms such as region-based convolutional neural networks (R-CNN), fast R-CNN, faster R-CNN, oriented gradient histogram (HOG), region-based fully convolutional networks (R-FCN), single-shot detectors (SSD), spatial pyramid pooling (SPP-net), and / or You Only Look Once (Yolo).

[0078] The vehicle perception and planning system 220 further includes a road detection subsystem 224. The road detection subsystem 224 locates the road and identifies objects and / or markings on the road. For example, based on raw or fused sensor data provided by radar sensor 234, camera 232, and / or LiDAR system 210, the road detection subsystem 224 can construct a 3D model of the road based on machine learning techniques (e.g., pattern recognition algorithms for lane identification). Using the 3D model of the road, the road detection subsystem 224 can identify objects (e.g., obstacles or debris) and / or markings (e.g., lane lines, turning signs, pedestrian crossing signs, etc.) on the road.

[0079] The vehicle perception and planning system 220 further includes a localization and vehicle attitude subsystem 225. Based on raw or fused sensor data, the localization and vehicle attitude subsystem 225 can determine the vehicle's position and attitude. For example, using sensor data from LiDAR system 210, camera 232, and / or GPS data, the localization and vehicle attitude subsystem 225 can determine the vehicle's precise location on the road and its six degrees of freedom (e.g., whether the vehicle is moving forward or backward, up or down, left or right). In some embodiments, a high-definition (HD) map is used for vehicle localization. The HD map can provide a very detailed three-dimensional computer map that accurately locates the vehicle's position. For example, using an HD map, the localization and vehicle attitude subsystem 225 can accurately determine the vehicle's current position (e.g., which lane the vehicle is currently in on the road, and how close it is to the curb or sidewalk) and predict the vehicle's future position.

[0080] The vehicle perception and planning system 220 further includes an obstacle predictor 226. Objects identified by the object classifier 223 can be stationary (e.g., lampposts, road signs) or dynamic (e.g., moving pedestrians, bicycles, another vehicle). For moving objects, predicting their movement paths or future positions is important for collision avoidance. The obstacle predictor 226 can predict obstacle trajectories and / or warn the driver or vehicle planning subsystem 228 of potential collisions. For example, if there is a high probability that the obstacle's trajectory will intersect with the vehicle's current movement path, the obstacle predictor 226 can generate such a warning. The obstacle predictor 226 can use various techniques to make such predictions. These techniques include, for example, constant speed or acceleration models, constant turning rate and speed / acceleration models, Kalman filter-based and extended Kalman filter-based models, recurrent neural network (RNN)-based models, long short-term memory (LSTM) neural network-based models, encoder-decoder RNN models, etc.

[0081] Referring again to Figure 2, in some embodiments, the vehicle perception and planning system 220 further includes a vehicle planning subsystem 228. The vehicle planning subsystem 228 may include one or more planners, such as a route planner, a driving behavior planner, and a motion planner. The route planner may plan the vehicle's route based on the vehicle's current location data, target location data, traffic information, etc. The driving behavior planner uses obstacle prediction results provided by obstacle predictor 226 to adjust the timing and planned movement based on how other objects might move. The motion planner determines the specific actions the vehicle needs to follow. The planning results are then transmitted to the vehicle control system 280 via vehicle interface 270. Communication can be performed via communication paths 227 and 271, which include any wired or wireless communication links capable of transmitting data.

[0082] The vehicle control system 280 controls the vehicle's steering mechanism, throttle, brakes, etc., to operate the vehicle according to a planned route and movement. In some examples, the vehicle perception and planning system 220 may further include a user interface 260 that provides access to the vehicle control system 280 to a user (e.g., a driver) to, for example, overtake or take over control of the vehicle when necessary. The user interface 260 may also be separate from the vehicle perception and planning system 220. The user interface 260 may communicate with the vehicle perception and planning system 220, for example, to acquire and display raw or fused sensor data, identified objects, the vehicle's position / attitude, etc. This displayed data can help the user better operate the vehicle. The user interface 260 may communicate with the vehicle perception and planning system 220 and / or the vehicle control system 280 via communication paths 221 and 261, respectively, which include any wired or wireless communication links capable of transmitting data. It should be understood that the various systems, sensors, communication links, and interfaces in Figure 2 can be configured in any desired manner and are not limited to the configuration shown in Figure 2.

[0083] Figure 3 is a block diagram illustrating an exemplary LiDAR system 300. The LiDAR system 300 can be used to implement the LiDAR systems 110, 120A-120I, and / or 210 shown in Figures 1 and 2. In one embodiment, the LiDAR system 300 includes a light source 310, a transmitter 320, an optical receiver and photodetector 330, a steering system 340, and a control circuitry system 350. These components are coupled together using communication paths 312, 314, 322, 332, 342, 352, 362, and 372. These communication paths include communication links (wired or wireless, bidirectional or unidirectional) between various LiDAR system components, but do not necessarily have to be the physical components themselves. While communication paths can be implemented using one or more wires, buses, or optical fibers, they can also be wireless channels or free-space optical paths, thus eliminating the need for a physical communication medium. For example, in one embodiment of the LiDAR system 300, the communication path 314 between the light source 310 and the transmitter 320 can be implemented using one or more optical fibers. Communication paths 332 and 352 may represent optical paths implemented using free-space optical components and / or optical fibers. Furthermore, communication paths 312, 322, 342, and 362 may be implemented using one or more wires carrying electrical signals. The communication paths may also include one or more communication media of the types described above (e.g., they may include optical fibers and free-space optical components, or include one or more optical fibers and one or more wires).

[0084] In some embodiments, LiDAR system 300 may be a coherent LiDAR system. Frequency-modulated continuous wave (FMCW) LiDAR is one example. Coherent LiDAR detects objects by mixing the reflected light from an object with light from a coherent laser emitter. Therefore, as shown in Figure 3, if LiDAR system 300 is a coherent LiDAR, it may include a route 372 that provides a portion of the transmitted light from emitter 320 to optical receiver and photodetector 330. Route 372 may include one or more optical components (e.g., optical fibers, lenses, mirrors, etc.) for providing light from emitter 320 to optical receiver and photodetector 330. The transmitted light provided by emitter 320 may be modulated light and may be split into two parts. One part is emitted to the FOV, while the second part is sent to optical receiver and photodetector 330 of LiDAR system 300. The second part is also referred to as light held local (LO) in LiDAR system 300. The transmitted light is scattered or reflected by various objects in the FOV, and at least a portion of it forms reflected light. The returned light is then detected and interferes with and recombines with a second portion of the transmitted light, which remains localized. Coherent LiDAR provides a mechanism for optically sensing the range of an object and its relative velocity along the line of sight (LOS).

[0085] The LiDAR system 300 may also include other components not shown in Figure 3, such as a power bus, power supply, LED indicators, switches, etc. Additionally, other communication connections between components may exist, such as a direct connection between the light source 310 and the optical receiver and photodetector 330, to provide a reference signal that allows for accurate measurement of the time from the emission of a light pulse to detection and the return of the light pulse.

[0086] Light source 310 outputs laser light to illuminate objects within the field of view (FOV). The laser light can be infrared light with wavelengths ranging from 700 nm to 1 mm. Light source 310 can be, for example, a semiconductor-based laser (e.g., a diode laser) and / or a fiber-based laser. Semiconductor-based lasers can be, for example, edge-emitting lasers (EELs), vertical-cavity surface-emitting lasers (VCSELs), external-cavity diode lasers, vertical-external-cavity surface-emitting lasers, distributed feedback (DFB) lasers, distributed Bragg reflector (DBR) lasers, interband cascade lasers, quantum cascade lasers, quantum well lasers, dual heterostructure lasers, etc. Fiber-based lasers are lasers in which the active gain medium is an optical fiber doped with rare-earth elements such as erbium, ytterbium, neodymium, dysprosium, praseodymium, thulium, and / or holmium. In some embodiments, the fiber laser is based on double-clad fiber, wherein the gain medium forms the core of the fiber surrounded by two cladding layers. Double-clad fiber allows the fiber core to be pumped with a high-power beam, thus enabling the laser source to become a high-power fiber laser source.

[0087] In some embodiments, the light source 310 includes a master oscillator (also referred to as a seed laser) and a power amplifier (MOPA). The power amplifier amplifies the output power of the seed laser. The power amplifier can be an fiber amplifier, a bulk amplifier, or a semiconductor optical amplifier. The seed laser can be a diode laser (e.g., a Fabry-Perot cavity laser, a distributed feedback laser), a solid-state bulk laser, or an external cavity tunable diode laser. In some embodiments, the light source 310 can be an optically pumped microchip laser. A microchip laser is an alignment-free monolithic solid-state laser in which the laser crystal is in direct contact with the end mirror of the laser resonator. Microchip lasers are typically pumped by laser diodes (directly or using fiber) to obtain the desired output power. Microchip lasers can be based on neodymium-doped yttrium aluminum garnet (Y3Al5O12) laser crystals (i.e., Nd:YAG) or neodymium-doped vanadate (i.e., ND:YVO4) laser crystals. In some examples, the light source 310 may have multiple amplification stages to achieve high power gain, enabling the laser output to have high power and thus allowing the LiDAR system to have a long scan range. In some examples, the power amplifier of the light source 310 can be controlled, allowing the power gain to be varied to achieve any desired laser output power. Examples of the light source 310 will be described in more detail below.

[0088] Referring to Figure 3, typical operating wavelengths of the light source 310 include, for example, approximately 850 nm, approximately 905 nm, approximately 940 nm, approximately 1064 nm, and approximately 1550 nm. For laser safety, the maximum usable laser power is capped by regulations set by the U.S. Food and Drug Administration (FDA). The optical power limit at 1550 nm is significantly higher than the power limits at the other wavelengths mentioned above. Furthermore, at 1550 nm, optical power loss in the fiber is very low. These characteristics of the 1550 nm wavelength make it more advantageous for long-range LiDAR applications. The amount of optical power output from the light source 310 can be characterized by its peak power, average power, pulse energy, and / or pulse energy density. Peak power is the ratio of pulse energy to pulse width (e.g., full width at half maximum or FWHM). Therefore, for a fixed amount of pulse energy, a smaller pulse width can provide a larger peak power. Pulse widths can range from nanoseconds to picoseconds. Average power is the product of pulse energy and pulse repetition rate (PRR). As described in more detail below, PRR represents the frequency of the pulsed laser. Generally, the smaller the time interval between pulses, the higher the PRR. PRR typically corresponds to the maximum range that the LiDAR system can measure. The light source 310 can be configured to generate pulses with a high PRR to meet the desired number of data points in the point cloud generated by the LiDAR system. The light source 310 can also be configured to generate pulses with a medium or low PRR to meet the desired maximum detection range. Wall insertion efficiency (WPE) is another factor for evaluating total power consumption and can be a useful metric for assessing laser efficiency. For example, as shown in Figure 1, multiple LiDAR systems can be attached to a vehicle, which can be an electric vehicle or a vehicle with limited fuel or battery power. Therefore, high WPE and intelligent use of laser power are often important considerations when selecting and configuring the light source 310 and / or designing a laser delivery system for vehicle-mounted LiDAR applications.

[0089] It should be understood that the above description provides a non-limiting example of light source 310. Light source 310 can be configured to include many other types of light sources (e.g., laser diodes, short-cavity fiber lasers, solid-state lasers, and / or external-cavity tunable diode lasers) configured to generate one or more optical signals of various wavelengths. In some examples, light source 310 includes amplifiers (e.g., preamplifiers and / or boost amplifiers), which can be doped fiber amplifiers, solid-state amplifiers, and / or semiconductor optical amplifiers. The amplifiers are configured to receive and amplify the optical signals at a desired gain.

[0090] Referring back to Figure 3, the LiDAR system 300 further includes a transmitter 320. A light source 310 provides laser light (e.g., in the form of a laser beam) to the transmitter 320. The laser light provided by the light source 310 may be an amplified laser having a predetermined or controlled wavelength, pulse repetition rate, and / or power level. The transmitter 320 receives the laser light from the light source 310 and transmits the laser light to a steering mechanism 340 with low divergence. In some embodiments, the transmitter 320 may include, for example, optical components (e.g., lenses, optical fibers, mirrors, etc.) for transmitting one or more laser beams directly or via the steering mechanism 340 to the field of view (FOV). Although Figure 3 illustrates the transmitter 320 and the steering mechanism 340 as separate components, in some embodiments they may be combined or integrated into a system. The steering mechanism 340 will be described in more detail below.

[0091] The laser beam supplied by light source 310 may diverge as it propagates to emitter 320. Therefore, emitter 320 typically includes a collimating lens or lens group configured to collect the diverging laser beam and produce a more parallel beam with reduced or minimal divergence. The collimated beam can then be further guided through various optics, such as mirrors and lenses. The collimating lens can be, for example, a single plano-convex lens or a lens group. The collimating lens can be configured to achieve any desired characteristics, such as beam diameter, divergence, numerical aperture, focal length, etc. The beam propagation ratio, or beam quality factor (also known as the M² factor), is used to measure the quality of the laser beam. In many LiDAR applications, good laser beam quality is important in the generated emitted laser beam. The M² factor represents the degree of variation of the beam relative to an ideal Gaussian beam. Therefore, the M² factor reflects how well a collimated laser beam can be focused on a small point, or how well a diverging laser beam can be collimated. Therefore, the light source 310 and / or emitter 320 can be configured to meet, for example, scanning resolution requirements while maintaining the desired M2 factor.

[0092] One or more beams of light provided by transmitter 320 are scanned onto the FOV by steering mechanism 340. Steering mechanism 340 scans the beams in multiple dimensions (e.g., horizontal and vertical) to allow LiDAR system 300 to map the environment by generating a 3D point cloud. The horizontal dimension may be parallel to the horizon or a surface associated with the LiDAR system or vehicle (e.g., a road surface). The vertical dimension is perpendicular to the horizontal dimension (i.e., the vertical dimension forms a 90-degree angle with the horizontal dimension). Steering mechanism 340 will be described in more detail below. The laser light scanned onto the FOV may be scattered or reflected by objects within the FOV. At least a portion of the scattered or reflected light forms return light returning to LiDAR system 300. Figure 3 further illustrates an optical receiver and photosensor 330 configured to receive the return light. The optical receiver and photosensor 330 include an optical receiver configured to collect the return light from the FOV. An optical receiver may include optical components (e.g., lenses, optical fibers, mirrors, etc.) for receiving, redirecting, focusing, amplifying, and / or filtering return light from the field of view (FOV). For example, an optical receiver typically includes a collecting lens (e.g., a single plano-convex lens or a group of lenses) to collect the return light and / or focus the collected return light onto a photodetector.

[0093] A photodetector detects the returned light focused by an optical receiver and generates a current and / or voltage signal proportional to the incident intensity of the returned light. Based on such current and / or voltage signals, depth information of the object within the field of view (FOV) can be derived. An exemplary method for deriving this depth information is based on direct time-of-flight (TOF), which will be described in more detail below. A photodetector can be characterized by its detection sensitivity, quantum efficiency, detector bandwidth, linearity, signal-to-noise ratio (SNR), overload immunity, interference immunity, etc. Depending on the application, a photodetector can be configured or customized to have any desired characteristics. For example, the optical receiver and photodetector 330 can be configured such that the photodetector has a large dynamic range while maintaining good linearity. Photodetector linearity indicates the detector's ability to maintain a linear relationship between the input optical signal power and the detector output. A detector with good linearity can maintain a linear relationship over a large dynamic range of input optical signals.

[0094] To achieve the desired detector characteristics, the structure and / or material system of the photodetector can be configured or customized. Various detector structures can be used for photodetectors. For example, a photodetector structure can be a PIN-based structure with an undoped intrinsic semiconductor region (i.e., the "I" region) between the p-type and n-type semiconductor regions. Other photodetector structures include, for example, APD (avalanche photodiode) based structures, PMT (photomultiplier tube) based structures, SiPM (silicon photomultiplier tube) based structures, SPAD (single-photon avalanche diode) based structures, and / or quantum wires. For the material system used in the photodetector, Si, InGaAs, and / or Si / Ge-based materials can be used. It should be understood that many other detector structures and / or material systems can be used in the optical receiver and photodetector 330.

[0095] Photodetectors (e.g., APD-based detectors) can have internal gain, amplifying the input signal when an output signal is generated. However, noise can also be amplified due to the photodetector's internal gain. Common noise types include signal shot noise, dark current shot noise, thermal noise, and amplifier noise. In some embodiments, the optical receiver and photodetector 330 may include a preamplifier for a low-noise amplifier (LNA). In some embodiments, the preamplifier may also include a transimpedance amplifier (TIA) that converts a current signal into a voltage signal. For linear detector systems, input equivalent noise or noise equivalent power (NEP) measures the photodetector's sensitivity to weak signals. Therefore, they can be used as indicators of overall system performance. For example, the photodetector's NEP specifies the power of the weakest signal that can be detected, and thus it specifies the maximum range of the LiDAR system. It should be understood that various photodetector optimization techniques can be used to meet the requirements of the LiDAR system 300. Such optimization techniques may include selecting different detector structures, materials, and / or implementing signal processing techniques (e.g., filtering, noise reduction, amplification, etc.). For example, coherent detection can be used in photodetectors in addition to or instead of direct detection using a returned signal (e.g., by using Time-of-Flight). Coherent detection allows the detection of the amplitude and phase information of received light by interfering the received light with a local oscillator. Coherent detection can improve detection sensitivity and noise immunity.

[0096] Figure 3 further illustrates that the LiDAR system 300 includes a steering mechanism 340. As described above, the steering mechanism 340 guides the beam from the transmitter 320 to scan the field of view (FOV) in multiple dimensions. The steering mechanism is also referred to as a grating mechanism, a scanning mechanism, or simply a light scanner. Scanning the beam in multiple directions (e.g., horizontal and vertical) facilitates the LiDAR system in mapping the environment by generating images or 3D point clouds. The steering mechanism can be based on mechanical scanning and / or solid-state scanning. Mechanical scanning uses rotating mirrors to steer the laser beam or physically rotates the LiDAR transmitter and receiver (collectively referred to as transceivers) to scan the laser beam. Solid-state scanning guides the laser beam to various locations within the FOV without mechanically moving any macroscopic components, such as transceivers. Solid-state scanning mechanisms include, for example, steering based on optical phased arrays and steering based on flash LiDAR. In some embodiments, steering performed by a solid-state scanning mechanism can be referred to as effective steering because the solid-state scanning mechanism does not physically move macroscopic components. LiDAR systems using solid-state scanning can also be referred to as non-mechanical scanning or simple non-scanning LiDAR systems (flash LiDAR systems are exemplary non-scanning LiDAR systems).

[0097] The steering mechanism 340 can be used with transceivers (e.g., transmitter 320 and optical receiver and photodetector 330) to scan the field of view (FOV) for generating images or 3D point clouds. As an example, to implement the steering mechanism 340, a 2D mechanical scanner can be used with a single-point or several single-point transceivers. The single-point transceivers transmit a single beam or a small number of beams (e.g., 2-8 beams) to the steering mechanism. 2D mechanical steering mechanisms include, for example, polygonal mirrors, oscillating mirrors, rotating prisms, rotating tilting mirrors, single-plane or multi-plane mirrors, or combinations thereof. In some embodiments, the steering mechanism 340 can include a non-mechanical steering mechanism, such as a solid-state steering mechanism. For example, the steering mechanism 340 can be based on the tuned wavelength of a laser incorporating refractive effects, and / or on a reconfigurable grating / phase array. In some embodiments, the steering mechanism 340 can implement 2D scanning using a single scanning device or by using a combination of multiple scanning devices.

[0098] As another example, to implement steering mechanism 340, a one-dimensional mechanical scanner can be used in conjunction with an array or a large number of single-point transceivers. Specifically, the transceiver array can be mounted on a rotating platform to achieve a 360-degree horizontal field of view. Alternatively, a static transceiver array can be combined with a one-dimensional mechanical scanner. One-dimensional mechanical scanners include polygonal mirrors, oscillating mirrors, rotating prisms, rotating tilting mirrors, or combinations thereof, for obtaining a forward-looking horizontal field of view. Steering mechanisms using mechanical scanners can provide robustness and reliability in mass production for automotive applications.

[0099] As another example, to implement the steering mechanism 340, a two-dimensional transceiver can be used to directly generate scanned images or 3D point clouds. In some embodiments, stitching or micro-displacement methods can be used to improve the resolution of the scanned image or the scanned field of view. For example, using a two-dimensional transceiver, signals generated in one direction (e.g., horizontal) and signals generated in another direction (e.g., vertical) can be integrated, interleaved, and / or matched to generate a higher or full-resolution image or 3D point cloud representing the scanned FOV.

[0100] Some embodiments of the redirection mechanism 340 include one or more optical redirection elements (e.g., mirrors or lenses) that redirect the returning optical signal along the receiving path (e.g., by rotation, vibration, or guidance) to direct the returning optical signal to the optical receiver and photodetector 330. The optical redirection elements that guide the optical signal along the transmission and receiving paths can be identical components (e.g., shared), separate components (e.g., dedicated), and / or a combination of shared and separate components. This means that in some cases, the transmission and receiving paths are different, although they may partially overlap (or in some cases, substantially overlap or completely overlap).

[0101] Referring again to Figure 3, the LiDAR system 300 further includes a control circuitry system 350. The control circuitry system 350 can be configured and / or programmed to control various parts of the LiDAR system 300 and / or perform signal processing. In a typical system, the control circuitry system 350 can be configured and / or programmed to perform one or more control operations, including, for example, controlling the light source 310 to obtain desired laser pulse timing, pulse repetition rate, and power; controlling the steering mechanism 340 (e.g., controlling speed, direction, and / or other parameters) to scan the field of view (FOV) and maintain pixel registration and / or alignment; controlling the optical receiver and photodetector 330 (e.g., controlling sensitivity, noise reduction, filtering, and / or other parameters) to optimize their operation; and monitoring the overall system health / functional safety status (e.g., monitoring the laser output power and / or the safety of the steering mechanism's operating status).

[0102] The control circuitry system 350 can also be configured and / or programmed to perform signal processing on the raw data generated by the optical receiver and photodetector 330 to obtain distance and reflectivity information, and to perform data packaging and communication with the vehicle perception and planning system 220 (as shown in Figure 2). For example, the control circuitry system 350 determines the time taken from the emission of a light pulse to the receipt of a corresponding return light pulse; determines when a return light pulse is not received for the emitted light pulse; determines the direction of the emitted / return light pulse (e.g., horizontal and / or vertical information); determines an estimated range in a specific direction; derives the reflectivity of objects in the field of view (FOV); and / or determines any other types of data relevant to the LiDAR system 300. The control circuitry system 350 may include digital and / or analog circuitry systems (e.g., ADCs, amplifiers, filters, etc.) for processing data representing the return light signals received by the LiDAR or HyDAR system.

[0103] The LiDAR system 300 can be incorporated into a vehicle that operates in a variety of environments, including hot or cold weather, rough road conditions that may cause severe vibrations, high or low humidity, dusty areas, etc. Therefore, in some embodiments, the optical and / or electronic components of the LiDAR system 300 (e.g., the optics, optical receivers, and photodetectors 330 in the transmitter 320, and the steering mechanism 340) are positioned and / or configured to maintain long-term mechanical and optical stability. For example, components in the LiDAR system 300 can be secured and sealed so that they can operate under all conditions the vehicle may encounter. As an example, a moisture-proof coating and / or an airtight seal can be applied to the optics, optical receivers, and photodetectors 330 of the transmitter 320, and the steering mechanism 340 (as well as other components susceptible to moisture). As another example, housings, enclosures, fairings, and / or windows can be used in the LiDAR system 300 to provide desired properties such as hardness, foreign object protection rating (IP), self-cleaning capability, chemical resistance, and impact resistance. In addition, the efficient and economical method for assembling the LiDAR system 300 can be used to meet the operational requirements of LiDAR while maintaining low cost.

[0104] Those skilled in the art will understand that Figure 3 and the above description are for illustrative purposes only, and that a LiDAR system may include other functional units, blocks, or segments, and may include variations or combinations of these functional units, blocks, or segments. For example, the LiDAR system 300 may also include other components not shown in Figure 3, such as a power bus, power supply, LED indicators, switches, etc. Additionally, other connections between components may exist, such as a direct connection between the light source 310 and the optical receiver and photodetector 330, allowing the photodetector 330 to accurately measure the time from the emission of a light pulse by the light source 310 to the detection of the returned light pulse by the photodetector 330.

[0105] The components shown in Figure 3 are coupled together using communication paths 312, 314, 322, 332, 342, 352, 362, and 372. These communication paths represent communication (bidirectional or unidirectional) between various LiDAR system components, but do not necessarily have to be the physical components themselves. While communication paths can be implemented by one or more wires, buses, or optical fibers, they can also be wireless channels or open-air optical paths, thus eliminating the need for a physical communication medium. For example, in an exemplary LiDAR system, communication path 314 includes one or more optical fibers; communication path 352 represents an optical path; and communication paths 312, 322, 342, and 362 are all wires carrying electrical signals. Communication paths can also include more than one of the communication media of the types described above (e.g., they can include optical fibers and optical paths, or one or more optical fibers and one or more wires).

[0106] Figure 4 is a block diagram illustrating an exemplary multi-mode detection system 400 with integrated sensors according to various embodiments. The multi-mode detection system 400 may be part of, or include part of, a LiDAR system (e.g., system 300). System 400 may also include one or more other sensors, such as a camera. In one example, system 400 includes a LiDAR sensor and an image sensor (or includes a LiDAR sensor and one or more other types of sensors), and this system may be referred to as a Hybrid Detection and Ranging (HyDAR) system. As shown in Figure 4, in some embodiments, on the transmitting side, system 400 may include a light source 402, a transmitter 404, and a steering mechanism 406. These components may form a transmission light path. On the receiver side, system 400 may include an optical receiver and a photosensor 430, and the system includes one or more of a light collection and distribution device 410, a signal separation device 440, and a multi-mode sensor 450. In some examples, the steering mechanism 406 is also used to receive light signals from a field of view (FOV) 470. Therefore, the steering mechanism 406 and the optical receiver and photodetector 430 can form a receiving optical path. The light source 402, the transmitter 404 and the steering mechanism 406 can be substantially the same as or similar to the light source 310, the transmitter 320 and the steering mechanism 340 described above in conjunction with FIG3.

[0107] In some examples, light source 402 is an internal light source that generates light for the multi-mode detection system 400. Examples of internal light sources include active lighting devices such as lasers (e.g., fiber lasers or semiconductor-based lasers used in one or more LiDAR transmission channels of system 400), light-emitting diodes, headlights / taillights, etc. An example of light source 402 will be described in more detail below with reference to Figures 5A and 5B. In some examples, system 400 also receives light from sources outside system 400. These external light sources may not be part of system 400. Examples of external light sources include sunlight, streetlights, and other illumination from sources outside system 400 (e.g., light from other LiDARs).

[0108] As shown in Figure 4, light generated by light source 402 (e.g., laser light from a LiDAR system) is provided to emitter 404. The light generated by light source 402 may include visible light, near-infrared (NIR) light, short-wavelength infrared (SWIR) light, mid-wavelength infrared (MWIR) light, long-wavelength infrared (LWIR) light, and / or any other wavelength. Visible light has a wavelength range of approximately 400 nm to 700 nm; near-infrared (NIR) light has a wavelength range of approximately 700 nm to 1.4 μm; short-wavelength infrared (SWIR) light has a wavelength range of approximately 1.4 μm to 3 μm; mid-wavelength infrared (MWIR) light has a wavelength range of approximately 3 μm to 8 μm; and long-wavelength infrared (LWIR) light has a wavelength range of approximately 8 μm to 15 μm.

[0109] Figure 5A is a block diagram illustrating an exemplary fiber-based laser source 500 for implementing the light source 310 depicted in Figure 3 and / or the light source 402 depicted in Figure 4. The fiber-based laser source 500 has a seed laser and one or more pumps (e.g., laser diodes) for pumping the required output power. In some embodiments, the fiber-based laser source 500 includes a seed laser 502 configured to generate initial optical pulses of one or more wavelengths (e.g., infrared wavelengths such as 1550 nm), which are provided to a wavelength division multiplexer (WDM) 504 via an optical fiber 503. The fiber-based laser source 500 further includes a pump 506 for providing laser power (e.g., different wavelengths, such as 980 nm) to the WDM 504 via an optical fiber 505. The WDM 504 multiplexes the optical pulses provided by the seed laser 502 and the laser power provided by the pump 506 onto a single optical fiber 507. The output of the WDM 504 can then be provided to one or more preamplifiers 508 via the optical fiber 507. Preamplifier 508 may be an optical amplifier that amplifies the optical signal (e.g., with a gain of approximately 10-30 dB). In some embodiments, preamplifier 508 is a low-noise amplifier. Preamplifier 508 outputs to optical combiner 510 via optical fiber 509. Combiner 510 combines the output laser of preamplifier 508 with laser power supplied by pump 512 via optical fiber 511. Combiner 510 may combine optical signals with the same or different wavelengths. An example of a combiner is WDM. Combiner 510 provides the combined optical signal to boost amplifier 514, which generates an output optical pulse via optical fiber 515. Boost amplifier 514 provides further amplification of the optical signal (e.g., another 20-40 dB). The output optical pulse can then be emitted to transmitter 320, transmitter 404, steering mechanism 340, and / or steering mechanism 406 (as shown in Figures 3 and 4). It should be understood that Figure 5A illustrates an exemplary configuration of a fiber-based laser source 500. The laser source 500 may have many other configurations using different combinations of one or more components shown in FIG. 5A and / or other components not shown in FIG. 5A (e.g., power supply, lens, filter, beam splitter, combiner, etc.).

[0110] In some variations, the fiber-based laser source 500 can be controlled (e.g., via control circuitry 350) to generate pulses of varying amplitudes based on the fiber gain distribution of the fiber used in the fiber-based laser source 500. Communication path 312 couples the fiber-based laser source 500 to control circuitry 350 (as shown in Figure 3), allowing components of the fiber-based laser source 500 to be controlled by or otherwise communicate with control circuitry 350. Alternatively, the fiber-based laser source 500 may include its own dedicated controller. Instead of control circuitry 350 communicating directly with components of the fiber-based laser source 500, a dedicated controller of the fiber-based laser source 500 communicates with and controls components of the fiber-based laser source 500 and / or communicates with it. The fiber-based laser source 500 may also include other components not shown, such as one or more power connectors, power supplies, and / or transmission lines.

[0111] Figure 5B is a block diagram illustrating an exemplary semiconductor-based laser source 540. The semiconductor-based laser source 540 is an example of the light source 310 depicted in Figure 3 and / or the light source 402 depicted in Figure 4. In the example shown in Figure 5B, the laser source 540 is a vertical-cavity surface-emitting laser (VCSEL), a type of semiconductor laser diode with a unique structure that allows it to emit light vertically from the surface of a chip, rather than through the edge of a chip as with edge-emitting laser (EEL) diodes. VCSELs offer the advantages of high-speed operation and ease of integration into semiconductor devices. Figure 5B shows a cross-sectional view of the exemplary VCSEL 540. In this example, the VCSEL 540 includes a metal contact layer 542, an upper Bragg reflector 544, an active region 546, a lower Bragg reflector 548, a substrate 550, and another metal contact layer 552. In the VCSEL 540, the metal contact layers 542 and 552 are used to form electrical contacts, thereby allowing current and / or voltage to be supplied to the VCSEL 540 to generate laser light. The substrate 550 is a semiconductor substrate, which may be, for example, a gallium arsenide (GaAs) substrate. The VCSEL 540 uses a laser resonator comprising two distributed Bragg reflectors (DBRs) (i.e., an upper Bragg reflector 544 and a lower Bragg reflector 548), with an active region 546 sandwiched between the DBR reflectors. The active region 546 includes, for example, one or more quantum wells for laser generation. The planar DBR reflector may be a mirror with alternating high and low refractive index layers. Each layer has a thickness of one-quarter of the laser wavelength in the material, producing an intensity reflectivity higher than, for example, 99%. The high-reflectivity mirror in the VCSEL can balance the short axial length of the gain region. In one example of the VCSEL 540, the upper DBR reflector 544 and the lower DBR reflector 548 may be doped with p-type and n-type materials, thereby forming a diode junction. In another example, the p-type and n-type regions may be embedded between the reflectors, requiring more complex semiconductor processes to fabricate electrical contacts with the active region, but eliminating power losses in the DBR structure. The active region 546 is sandwiched between the DBR reflectors 544 and 548 of the VCSEL 540. The active region is where the laser light is generated. The active region 546 typically has a quantum well or quantum dot structure containing the gain medium responsible for optical amplification. When a current is applied to the active region 546, it generates photons through stimulated emission. The distance between the upper DBR reflector 544 and the lower DBR reflector 548 defines the cavity length of the VCSEL 540. The cavity length, in turn, determines the wavelength of the emitted light and affects the performance characteristics of the laser.When current is applied to the VCSEL 540, it generates light that bounces between DBR reflectors 544 and 548, and exits the VCSEL 540 through, for example, the lower DBR reflector 548, thereby producing a highly coherent and vertically emitted laser beam 554. The VCSEL 540 can provide improved beam quality, low threshold current, and the ability to produce single-mode or multimode output.

[0112] In some variations, VCSEL 540 can be controlled (e.g., via control circuitry 350) to generate pulses of varying amplitudes. Communication path 312 couples VCSEL 540 to control circuitry 350 (as shown in Figure 3), allowing components of VCSEL 540 to be controlled by or otherwise communicate with control circuitry 350. Alternatively, VCSEL 540 may include its own dedicated controller. Instead of control circuitry 350 communicating directly with components of VCSEL 540, the dedicated controller of VCSEL 540 communicates with and controls components of VCSEL 540 and / or communicates with it. VCSEL 540 may also include other components not shown, such as one or more power connectors, power supplies, and / or power lines.

[0113] The VCSEL 540 can be used to generate laser pulses or continuous wave (CW) lasers. To generate laser pulses, the control circuitry 350 modulates the current supplied to the VCSEL 540. Laser pulses can be generated by rapidly switching the power supply current on and off. The pulse duration, repetition rate, and shape can be controlled by adjusting the modulation parameters. As another example, the VCSEL 540 can also be a mode-locked VCSEL, which uses a combination of current modulation and optical feedback to obtain ultrashort pulses. Mode-locked VCSELs can also be controlled to synchronize the phase of the laser mode to produce very short and high-intensity pulses. As another example, the VCSEL 540 can use Q-switching technology, which includes an optical switch in the laser cavity that temporarily blocks laser action and allows energy to accumulate in the cavity. When the switch is open, a high-intensity pulse is emitted. As another example, the VCSEL 540 can also have external modulation performed by an external modulator (not shown), such as an electro-optic or acousto-optic modulator. External modulation can be used in conjunction with the VCSEL itself to produce pulse output. The external modulator can be used to control the pulse duration and repetition rate. The type of vertical cavity surface-emitting laser (VCSEL) used as at least part of light source 310 or light source 402 depends on the application and required pulse characteristics, such as pulse duration, repetition rate and peak power.

[0114] Referring back to Figure 4, the multi-mode detection system 400 includes a transmitter 404. In some examples, the transmitter 404 may include one or more transmission channels, each carrying a light beam. The transmitter 404 may also include one or more optical components (e.g., mirrors, lenses, fiber arrays, etc.) and / or electrical components (e.g., PCB board, power supply, actuators, etc.) to form the transmission channels. The transmitter 404 may direct light from each channel to a deflection mechanism 406, which scans the light from each channel to the field of view (FOV) 470. The deflection mechanism 406 may include one or more optical or electronic scanners configured to perform at least one of point scanning or line scanning of the FOV 470.

[0115] The light source 402, emitter 404, and steering mechanism 406 may be part of a LiDAR or HyDAR system that scans light into the FOV 470. Scanning performed by the steering mechanism 406 may include, for example, line scanning and / or point scanning. For example, the steering mechanism 406 may be configured to scan all points in a line or region; scan some points in a specific line or region while skipping others; or scan some lines while skipping others. As another example, the steering mechanism 406 of the multi-mode detection system 400 may be configured to scan some points / lines at a higher resolution while scanning others at a lower resolution. For example, high-resolution scanning may be applied to a region of interest (ROI), while low-resolution scanning or no scanning may be applied to other areas of the FOV. In some embodiments, for scanning an ROI, the steering mechanism 406, which includes one or more optical or electronic scanners, may be controlled to have different characteristics than those used for scanning non-ROIs. For example, for scanning an ROI, the scanner may be controlled to have a slower scan rate and / or a smaller scan step size, thereby improving scan resolution. In addition, the light source 402 can also be configured to increase the pulse repetition rate, thereby increasing the scanning resolution.

[0116] Referring to Figure 4, in some embodiments, if the sensor in the multi-mode detection system 400 does not require active emission and / or scanning light, then for that particular sensor, one or more of the light source 402, emitter 404, and steering mechanism 406 may not be required. For example, if the system 400 includes a passive image sensor or video sensor (e.g., a camera), it may not need to actively emit light and / or scan light toward the FOV to form an image of the FOV. In this disclosure, the terms "image sensor" and "video sensor" are used interchangeably, both referring to a passive sensor capable of capturing images and / or video. As a passive sensor, an image sensor may only sense light from the FOV and use the sensed light to form an image. It may not itself emit light toward the FOV. In some other examples, an image sensor may require a light source (e.g., a flash or other illumination) to provide sufficient light conditions for sensing (e.g., capturing an image with sufficient brightness). In some examples, the image sensor may also perform point scanning or line scanning to obtain better performance, such as improved detection limits and greater dynamic range. Such image sensors may have high image resolution and complex imaging structures, and therefore may be expensive. However, as described below, integrating such an image sensor with, for example, a LiDAR sensor in a multi-mode detection system 400 can reduce the overall cost compared to two discrete sensors.

[0117] Figure 4 further illustrates that system 400 includes an optical receiver and a photosensor 430 to receive and detect light from FOV 470. As described above, the emitting side of system 400 can emit light into FOV 470. A portion of the emitted light can be reflected or scattered by objects in FOV 470 to form a return light signal. The return light signal can be received by the optical receiver and photosensor 430. Additionally, the optical receiver and photosensor 430 can also receive light signals from other external light sources, including, for example, sunlight, ambient light, streetlights, and / or other lighting sources, such as light from other LiDAR or HyDAR systems. The various light signals received by the optical receiver and photosensor 430 are collectively referred to as received light signals or collected light signals. Received or collected light signals can include the return light signal formed based on the transmitted light of system 400 and other light signals from other light sources. The received light signals can have a narrow or wide spectral range, including, for example, one or more of visible light, NIR light, SWIR light, MWIR light, and / or LWIR light. One or more of these received optical signals can be detected by different types of photodetectors, such as a LiDAR sensor for detecting IR light signals and an image sensor for detecting visible light signals. In this disclosure, one or more of these photodetectors can be integrated to form a hybrid detector. The light collection and distribution device 410, the signal separation device 440, and the multi-mode sensor 450 of the optical receiver and photodetector 430 will be described in more detail below.

[0118] Figure 6 illustrates an exemplary light collection and distribution device 410. The light collection and distribution device 410 can be configured to perform the collection of light signals from a field of view (FOV) and the distribution of light signals to at least one of a plurality of sensors (e.g., sensor 450) of a multi-mode sensor. The light signals collected and distributed by device 410 can have multiple wavelengths. At least one wavelength differs from one or more other wavelengths. As shown in Figure 6, device 410 may include light-collecting optics 602, refractive optics 610, diffractive optics 620, reflective optics 630, and / or optical fiber 640. Although Figure 4 illustrates a redirection mechanism 406 as a device separate from the light collection and distribution device 410, in some embodiments, the redirection mechanism 406 may be integrated with or part of device 410. For example, the redirection mechanism 406 may be shared between transmitter 404 and optical receiver and photodetector 430 for transmitting light signals to and receiving / redirecting light signals from the FOV. This type of configuration is also referred to as a coaxial configuration because the transmitting and receiving optical paths share some common optical components. Therefore, although not explicitly shown in Figure 6, the light-collecting optics 602 may include a steering mechanism shared between the transmitter and the receiver.

[0119] Referring to Figure 6, the light signal from the FOV can be received or collected by a light-collecting optics 602 (e.g., by a steering mechanism 406). The light-collecting optics 602 includes optics configured to collect and focus the received light signal. The light-collecting optics 602 can be optimized to maximize the amount of light signal collected from the FOV and to direct the light signal toward a specific target, such as a refractive optics, diffractive optics, reflective optics, detector, sensor, and / or imaging system. The light-collecting optics 602 can include one or more types of light-collecting optics, including one or more lenses, one or more lens groups, one or more mirrors, and one or more optical fibers. For example, a collecting lens or lens group can be used to collect light signals from a distant object in the FOV and focus the light signal onto another optical component or detector. Mirrors are another optical component that can be used in the light-collecting optics 602. They can be used to reflect light and redirect it toward a specific target. Mirrors can be used alone or in combination with lenses to form complex optical structures for collecting light signals.

[0120] In some embodiments, as shown in FIG6, a light-collecting optics 602 guides the collected light signal to one or more of a refractive optics 610, a diffractive optics 620, a reflective optics 630, and / or an optical fiber 640. In some embodiments, the light-collecting optics 602 may be optional or integrated with the refractive optics 610, diffractive optics 620, reflective optics 630, and / or optical fiber 640. For example, the collected light signal may be guided (with or without the light-collecting optics 602) to the refractive optics 610. The refractive optics 610 may include an optics device that bends the light signal when it travels from one medium (e.g., air) to another medium (e.g., glass) with a different refractive index. The refractive index is an indicator of how much a medium bends a light signal. When a light signal travels from a medium with a high refractive index to a medium with a low refractive index, the light signal deviates from the normal direction (e.g., perpendicular to the surface at the point where the light enters the second medium). When a light signal travels from a medium with a low refractive index to a medium with a high refractive index, the light bends towards the normal direction. The amount of curvature depends on the angle of incidence (the angle between the incident light signal and the normal direction of the surface of the medium) and the refractive indices of the two media. The relationship between these variables is described by Snell's law, which states that the ratio of the sine of the angle of incidence to the sine of the angle of refraction is equal to the ratio of the refractive indices of the two media.

[0121] In some embodiments, the refractive optics 610 may be implemented using a beam splitter configured to perform optical refraction such that it transmits a first portion of the incident light signal from the FOV (e.g., received directly or via a light-collecting optics) to a first sensor and reflects a second portion of the received light signal to a second sensor. The first and second sensors may be different sensors located at two different locations.

[0122] Referring again to Figure 6, the light collection and distribution device 410 may further include a diffractive optics element 620 configured to separate an incident light signal into portions with different wavelengths, intensities, or polarizations. The diffractive optics element 620 may include an optics element having a diffractive structure, such as a diffraction grating. The diffractive structure may be made of a thin layer of material containing features such as grooves, ridges, or other microstructures configured to manipulate the phase of the incident light signal. These diffractive structures can be used to manipulate properties of the light signal, such as direction, intensity, polarization, and wavelength. In some examples, the diffractive optics element 620 may include a diffraction grating, a periodic structure that separates light into its spectral components based on its wavelength. In some examples, the diffractive optics element 620 may also include a diffractive lens, a beam splitter, and a polarizer. A diffractive lens may be configured to correct chromatic aberration and other types of optical distortion and may be used to provide a lightweight and compact optical system. The diffractive optics element 620 can be used to create highly precise and complex optical elements. Therefore, they can be used in multi-mode detection systems to accurately separate optical signals with different characteristics (e.g., wavelength, intensity, polarization, etc.) and guide them to different sensors.

[0123] Figure 6 also illustrates that the light collection and distribution device 410 may include a reflective optics device 630. The reflective optics device 630 includes one or more optical components capable of reflecting incident light signals. The angle of incidence determines the angle of reflection. The characteristics of the reflective optics surface (e.g., roughness, shape, and material) may affect the reflection of the incident light signal. In one example, the reflective optics device 630 includes a Schmidt-Cassegrain-based reflective device configured to direct a portion of the incident light signal to a first sensor and another portion to a second sensor. In some examples, the reflective optics device 630 includes a Newtonian-based reflective device configured to direct a portion of the incident light signal to a first sensor and another portion to a second sensor. The first and second sensors may be different sensors located at different physical locations. They may also be different types of sensors (e.g., LiDAR sensors and image sensors).

[0124] In other embodiments, the incident light signal collected by the light-collecting optics 602 can be directed to different sensors using an optical fiber 640. The optical fiber 640 can be flexible and has any desired length. Therefore, using the optical fiber 640, the incident light signal can be directed to different sensors located at different physical locations.

[0125] As described above and as shown in Figure 4, the multi-mode detection system 400 may include a signal separation device 440. Figure 7 illustrates an example of such a signal separation device 440. The signal separation device 440 is configured to separate the incident light signal to form separate light signals with multiple different optical characteristics. The signal separation device 440 can perform various separations, including spatial separation, intensity separation, spectral separation, polarization separation, etc. Although Figure 4 illustrates that the signal separation device 440 and the light collection and distribution device 410 are two different devices, in some embodiments, the signal separation device 440 may be at least partially combined with the light collection and distribution device 410. For example, as described above, the light collection and distribution device 410 may include one or more of refractive optics, diffractive optics, reflective optics, etc., to perform spatial distribution of the incident light signal. Thus, these optical components may form part of the signal separation device 440 (e.g., as a spatial separation device) to separate the incident light signal into different portions and direct the different portions to different detectors at different physical locations.

[0126] Referring to Figure 7, the signal separation device 440 may include a spatial separation device 706, a spectral separation device 704, a polarization separation device 708, and / or other separation devices (not shown). The spatial separation device 706 is configured to separate optical signals to form separated optical signals corresponding to at least one of different spatial locations of multiple sensors or different angular directions of the optical signals. Therefore, the optical signals from the spatial separation device 706 can have different physical locations and / or different angular directions. The spectral separation device 704 is configured to separate optical signals to form separated optical signals with different wavelengths (e.g., NIR light, visible light, SWIR light, etc.). The polarization separation device 708 is configured to separate optical signals to form separated optical signals with different polarizations (e.g., horizontal or vertical).

[0127] The devices included in the signal separation device 440 can be configured and constructed in any desired manner. In one embodiment, the spatial separation device 706 may be positioned upstream to receive the incident light signal 702 and guide the spatially separated light signal to the spectral separation device 704 and / or the polarization separation device 708. In another embodiment, the spectral separation device 704 may be positioned upstream to receive the incident light signal 702 and guide the spectrally separated light signal to the spatial separation device 706 and / or the polarization separation device 708. Similarly, the polarization separation device 706 may be positioned upstream. In other words, the signal separation device 440 can be configured such that spectral separation, spatial separation, polarization separation, and / or any other separation can be performed in any desired order. In other embodiments, two or more types of separation can be performed together. For example, as described above, a prism or beam splitter can separate the light signal spectrally and spatially. Each of the devices 704, 706, and 708 will be described in more detail below.

[0128] An example of a spatial separation device 706 is an optical fiber bundle. An incident light signal 702 is coupled to the optical fiber bundle, which may include a number of optical fibers bundled together such that they are physically close to each other at one end of the bundle. The different fibers of the bundle can then be routed to different sensors located at different physical locations. Another example of a spatial separation device 706, as shown in Figure 7, includes a microlens array configured to separate the incident light signal to form a separated light signal and guide the separated light signal to the corresponding sensor. A microlens array is an optical component comprising an array of small lenses. These small lenses typically have a diameter of tens to hundreds of micrometers. Each lens in the microlens array focuses the light signal onto a specific point or sensor, and the overall effect of the array is to shape or manipulate the light signal in a specific manner. Microlens arrays can be used to improve the resolution and sensitivity of an imaging system by focusing the light signal onto a detector array or improving light collection efficiency. Microlens arrays can also be used to shape light into specific patterns or distributions for applications such as image sensing or depth sensing. Microlens arrays can also be used to couple light between optical fibers or improve the coupling efficiency between optical components. Microlens arrays can be made of materials such as glass, silicon, or plastic, and can be customized in terms of lens size, shape, and spacing to achieve desired optical performance.

[0129] Referring again to Figure 7, the signal separation device 440 may further include a spectral separation device 704 configured to separate optical signals to form separated optical signals with different wavelengths or colors. The spectral separation device 704 includes one or more of the following: a dichroic mirror, a dual-band mirror, a dual-wavelength mirror, a dichroic reflector, a red-green-blue (RGB) filter, an infrared filter, a colored glass filter, an interference filter, a diffractive optics device, a prism, a diffraction grating, a thermally etched grating, a holographic grating, and a Cezrny-Turner monochromator. For example, a prism may refract the optical signal at different angles depending on the wavelength of the light signal. Taking visible light as an example, when an incident light signal passes through a prism, the light signal can be separated into different colors for different channels, including red, green, and blue channels. As another example, diffraction gratings can also be used for spectral separation. They consist of a series of closely spaced parallel lines or slits that diffract light at different angles according to the wavelength of the light. Using a diffraction grating, the incident light signal can be similarly divided into red, green, and blue channels. The separated light signals have different wavelengths, each carrying different information that can be more easily processed by computer vision algorithms.

[0130] Figure 7 also illustrates that the signal separation device 440 may include a polarization separation device 708 configured to separate optical signals to form separated optical signals with different polarizations. In one embodiment, the polarization separation device 708 includes one or more absorptive polarizers, including crystal-based polarizers, beam-splitting polarizers, Fresnel reflection-based polarizers, birefringent polarizers, thin-film-based polarizers, wire-grating polarizers, and circular polarizers. For example, polarization separation can be achieved using a polarization filter, which is a filter that transmits only light waves with a specific polarization orientation. The polarization filter can be made of materials such as polarizing films, wire gratings, or birefringent crystals. When unpolarized light passes through a polarization filter, only the light component with the same polarization orientation as the filter is transmitted, while other polarization components are blocked. This results in polarized light having a specific polarization orientation. For example, when an optical signal passes through the polarization separation device 708, the optical signal can be separated into an optical signal with horizontal polarization, an optical signal with vertical polarization, and an optical signal with all polarizations. Image data formed from light signals with different polarizations can include different information, such as different contrasts, brightness, and colors.

[0131] By using one or more of the separation devices of the types described above and other separation / processing devices (e.g., image sensors such as CCD arrays), signal separation device 440 can process incident light signals to distinguish light intensity and / or reflectivity. Light signals reflected or received by an optical receiver at different angles may have different light intensities. Different light intensities can be sensed and represented by signal separation device 440 through, for example, different brightness / colors of captured images.

[0132] Referring back to FIG4, as described above, in some embodiments, the light collection and distribution device 410 and the signal separation device 440 may be two separate devices. For example, device 410 is configured to collect optical signals from FOV 470 and spatially distribute the received optical signals, while device 440 is configured to spectrally separate the received optical signals. In some embodiments, the light collection and distribution device 410 and the signal separation device 440 may be at least partially combined together to perform one or more of spatial separation, spectral separation, polarization separation, etc. In another embodiment, the light collection and distribution device 410 may directly distribute the optical signals to the multimode sensor 450 without using the signal separation device 440.

[0133] Referring again to Figure 4, when the received optical signals are processed by the light collection and distribution device 410 and optionally the signal separation device 440, they are passed to the multimode sensor 450. In some embodiments, the multimode sensor 450 includes a plurality of sensors corresponding to respective light emitter positions to improve light collection efficiency. For example, each of the plurality of sensors may be angularly positioned to correspond to different angular positions of multiple emitter channels that guide multiple transmitted beams to different directions. In this way, the receiving aperture for receiving the returned optical signals formed by the different transmitted beams can be maximized. Each sensor of the multimode sensor 450 may include one or more detectors or detector elements. The plurality of sensors may be of different types. For example, the plurality of sensors may include at least a first type of optical sensor and a second type of optical sensor. The first type of optical sensor may be configured to detect optical signals having a first optical characteristic, while the second type of optical sensor is configured to detect optical signals having a second optical characteristic. The first optical characteristic may be different from the second optical characteristic. For example, the first type of optical sensor may include a sensor configured to detect optical signals with NIR wavelengths for use in a LiDAR system. The second type of optical sensor may include a sensor configured to detect light signals having visible light wavelengths suitable for use with a camera. As mentioned above, LiDAR sensors can use NIR wavelength signals to generate point cloud data for distance measurement; while image sensors can use visible light to generate image data for visual computing.

[0134] In some embodiments, multiple sensors of the multi-mode sensor 450 may be combined or integrated together. FIG8 illustrates an exemplary configuration of an integrated detector of the multi-mode sensor 450 according to various embodiments of the present disclosure. As shown in FIG8, two or more sensors of the multi-mode sensor may be integrated in a single device package, detector assembly, semiconductor chip, or single printed circuit board (PCB). For example, semiconductor chip 800 may include a plurality of dies sharing a semiconductor substrate. The dies may be located in the same wafer. At least a portion of semiconductor chip 800 may be used as a sensor of multi-mode sensor 450. In the embodiment shown in FIG8, chip 800 may include four sensors 802, 804, 806, and 808. Sensors 802 and 804 may be disposed in respective dies of chip 800 (one die of chip 800 is shown as a small square). Sensors 806 and 808 may be disposed in multiple dies. For example, sensor 806 may include four detectors arranged horizontally across four dies, while sensor 808 may include four detectors arranged horizontally and vertically across four dies, forming a 2×2 array. It should be understood that sensors can be mounted on any number of dies in any desired manner. Chip 800 may also include other sensors or circuitry. For example, readout circuitry for processing signals generated by the sensors can be integrated into chip 800, thereby increasing the integration of the multi-mode sensor 450 and reducing cost.

[0135] Sensors that can be integrated into chip 800 may include photodiode-based detectors, avalanche photodiode (APD)-based detectors, charge-coupled device (CCD)-based detectors, etc. For example, photodiode-based detectors can be made of silicon or germanium; APD-based detectors can be made of silicon, germanium, indium gallium arsenide (InGaAs), or mercury cadmium telluride (MCT); and CCD-based detectors can be made of silicon, gallium arsenide (GaAs), indium phosphide (InP), and MCT. In some examples, APDs can be used to sense infrared light for LiDAR devices, and CCDs can be used to sense visible light for cameras. Therefore, multiple sensors can be integrated together on chip 800 using semiconductor chip manufacturing techniques. It should be understood that sensors included in multi-mode sensor 450 can also use other suitable semiconductor materials, such as silicon germanium (SiGe).

[0136] Referring again to Figure 8, in some embodiments, chip 800 may also integrate a photonic crystal structure, which is a type of artificial periodic structure that can manipulate optical flow in a manner similar to how crystals manipulate electron flow in solid materials. Photonic crystals are made by creating a pattern of periodic variations in the refractive index of a material. This pattern produces a photonic bandgap, which is the range of frequencies through which light cannot propagate. The photonic bandgap is generated by the interference of waves reflected from the periodic structure, resulting in destructive interference at some frequencies and constructive interference at others. The result is a range of frequencies through which light cannot propagate, similar to how an electronic bandgap blocks electron flow in a semiconductor. Photonic crystals can be made from a variety of materials, including semiconductors, metals, and polymers. Photonic crystal structures can be used to realize filters, detectors, waveguides, and laser emitters. For example, photonic bandgap can be used to fabricate filters; and the sensitivity of photonic crystals to changes in refractive index can be used to fabricate highly sensitive sensors. Therefore, by using a photonic crystal structure, chip 800 can integrate not only sensors or detectors, but also other optical components such as filters, waveguides, and light sources, thereby further improving integration. Thus, various dies or modules disposed within chip 800 can achieve different functions. Chip 800 can be bonded to other components (e.g., readout circuitry, PCB) using wire bonding, flip-chip bonding, BGA bonding, or any other suitable packaging technology.

[0137] As described above, the multi-mode sensor 450 (as shown in Figures 4 and 8) may include multiple sensors. Each sensor includes one or more detectors, one or more other optical elements (e.g., lenses, filters, etc.) and / or electrical elements (e.g., ADCs, DACs, processors, etc.). In the example shown in Figure 8, multiple sensors may be integrated or arranged together to form the multi-mode sensor 450. The multi-mode sensor 450 may be included in a detector assembly, device package, device module, or PCB. Multiple sensors are mounted to the same assembly, device package, device module, or PCB. In other embodiments, the multi-mode sensor 450 may include two or more assemblies, device packages, modules, or PCBs. Each of the multiple sensors may be mounted to a different assembly, device package, device module, or PCB. Different assemblies, device packages, modules, or PCBs may be arranged close to each other or housed to form an integrated multi-mode sensor package.

[0138] In the example shown in Figure 8, the multi-mode sensor 450 includes multiple sensors, including an imaging sensor 812, an illuminance sensor 814, a LiDAR sensor 816, and one or more other sensors 818. The imaging sensor 812 may include a detector that detects light signals and converts them into electrical signals to form an image. Therefore, the imaging sensor 812 can be used as part of a camera. The imaging sensor 812 may be a CCD sensor, a CMOS sensor, an active pixel sensor, a thermal imaging sensor, etc. The illuminance sensor 814 may include a detector that facilitates the measurement of the amount of light falling on a surface per unit area (called illuminance). For example, illuminance can be expressed in lumens per square meter. The illuminance sensor 814 may include a detector containing a photodiode, phototransistor, photovoltaic cell, photoresistor, etc. The illuminance sensor 814 can be used for lighting control, brightness control, environmental monitoring, etc.

[0139] LiDAR sensor 816 may include a detector that detects laser light (e.g., in the infrared wavelength range). The detected laser light can be used to determine the distance of an object from the LiDAR sensor. LiDAR sensor 816 can be used to generate a 3D point cloud of the surrounding area. The detector used for the LiDAR sensor may be an avalanche photodiode, a mercury cadmium telluride (HgCdTe) based infrared detector, an indium antimonide (InSb) based detector, etc. LiDAR sensor 816 can be implemented using one or more components of the LiDAR system 300 described above. Figure 8 also illustrates that multi-mode sensor 450 may include one or more other sensors 818. These other sensors 818 may facilitate temperature sensing, chemical sensing, pressure sensing, motion sensing, light sensing, proximity sensing, etc. One or more sensors 818 may include detectors such as light-emitting diodes (LEDs), photoresistors, photodiodes, phototransistors, fixed photodiodes, quantum dot photoconductors / photodiodes, silicon drift detectors, photovoltaic-based detectors, avalanche photodiodes (APDs), thermal detectors, calorimeters, microcalorimeters, cryogenic detectors, thermoelectric detectors, thermopile detectors, Golay detectors, photosensitive detectors, chemical-based detectors, polarization-sensitive photodetectors, and graphene / silicon photodetectors, etc.

[0140] Referring to Figures 4 and 8, the multiple sensors of the multi-mode sensor 450 may include various types of sensors integrated or mounted together to share, for example, semiconductor wafers, modules, printed circuit boards, and / or semiconductor packages. These sensors may also share one or more components in the transmission optical path (e.g., light source 402, emitter 404, and / or steering mechanism 406) and / or receiving optical path (e.g., light collection and distribution device 410, signal separation device 440). Therefore, the multi-mode sensor 450 can have a compact size, thereby enabling a compact multi-mode detection system. The compact multi-mode detection system can be set up or mounted in any location within a mobile platform such as a motor vehicle. For example, compared to mounting multiple discrete sensors (such as one or more cameras, one or more LiDARs, one or more thermal imaging devices, one or more ultrasonic devices, etc.), mounting a compact multi-mode detection system can significantly reduce the complexity of integrating multiple sensing capabilities into a vehicle and / or reduce costs. As shown in Figures 4 and 8, a multi-mode detection system (e.g., system 400) that includes a multi-mode sensor (e.g., sensor 450) is sometimes referred to as a hybrid detection and ranging system (HyDAR).

[0141] Referring again to Figures 4 and 8, in some embodiments, multiple detectors or sensors of the multimode sensor 450 can be configured to detect optical signals received from the same field of view (FOV). For example, Figure 4 illustrates that optical signals received from the same FOV 470 may include two or more of the following: NIR light, visible light, SWIR light, MWIR light, LWIR light, and other light. These optical signals are mixed together but can be detected by the same multimode sensor 450. For example, as described above, the mixed optical signals can be collected and distributed by device 410 and then separated by signal separation device 440 according to one or more of the optical characteristics (e.g., wavelength, polarization, angle of incidence, etc.). The separated optical signals can then be detected by corresponding optical sensors included in the multimode sensor 450. In this way, the multimode detection system 400 provides integrated multimode sensing capabilities, reducing or eliminating the need for multiple discrete or separate sensors (such as cameras, LiDAR, thermal imaging devices, etc.). This makes the sensing device more integrated and compact, thereby reducing costs and improving sensing efficiency. As an example, when discrete sensors are mounted separately on a vehicle (or another mobile platform), the data captured by different sensors (e.g., a LiDAR sensor and an image sensor such as a camera) typically requires time synchronization and / or conversion to use the same coordinate system. This data fusion process can be cumbersome, error-prone, inefficient, and power-intensive. At least some of these problems can be solved by integrating multiple sensors into the multi-mode sensor disclosed herein. For example, if a LiDAR sensor and an image sensor are integrated together (e.g., housed in a single device package, PCB, and sharing at least a portion of the transmit / receive optical path), data from the two sensors can be directly fused without prior time synchronization or coordinate transformation, or with minimal fusion work.

[0142] As described above, the multi-mode sensor 450 may include an integrated sensor array comprising multiple sensors of different types. FIG9 illustrates exemplary package configurations of integrated sensors according to various embodiments of the present disclosure. As shown in FIG9, the multi-mode sensor device 904 may include a plurality of sensors 906, each of which is disposed on a heat sink 912. The sensors 906 may be of the same type or different types. Each of the sensors 906 may be wire-bonded to an integrated circuit chip 908. The IC chip 908 may be used to process the electrical signals generated by the sensors 906 to implement a readout circuitry. The IC chip 908 may further include other signal processing circuitry, such as rendering images, performing digital signal processing functions, etc. In this configuration, the sensor array and the readout circuitry are integrated in the same device package (e.g., both the IC chip 908 and the sensor array 906 are disposed on the same PCB 914). In other embodiments, the sensor array and the readout circuitry may be packaged separately in separate modules. These two separate modules may then be mounted on a PCB so that signals can be passed between the two modules.

[0143] Figure 9 also illustrates another packaging configuration in which the readout circuitry 920 is housed in one semiconductor chip, and the integrated sensor array 924 is housed in another semiconductor chip 926. The two chips 920 and 926 are joined together via flip-chip technology, allowing electrical signals to be delivered from the sensor array 924 to the readout circuitry 920 via solder bumps 922. Once joined, the two chips 920 and 926 can be packaged together as a single device 930. It should be understood that other packaging technologies, such as through-hole packaging, surface mount packaging, ball grid array packaging, chip-scale packaging, etc., can also be used.

[0144] As described above, some LiDAR or HyDAR systems use the time-of-flight (ToF) of an optical signal (e.g., a light pulse) to determine the distance to an object in the optical path. The following description uses LiDAR system 1000 as an example. It should be understood that LiDAR devices or sensors in HyDAR systems can operate similarly. For example, referring to FIG10A, the exemplary LiDAR system 1000 includes a laser source (e.g., a fiber laser), a steering mechanism (e.g., a system of one or more moving mirrors), and a photodetector (e.g., a photodetector with one or more optics). LiDAR system 1000 can be implemented using, for example, the LiDAR system 300 described above. LiDAR system 1000 emits light pulses 1002 along an optical path 1004 defined by the steering mechanism of LiDAR system 1000. In the depicted example, the light pulses 1002 generated by the laser source are short pulses of laser light. Furthermore, the signal manipulation mechanism of LiDAR system 1000 is a pulse signal steering mechanism. However, it should be understood that LiDAR systems can operate by generating, emitting, and detecting non-pulsed light signals and using techniques other than time-of-flight to derive the distance to objects in the surrounding environment. For example, some LiDAR systems use frequency-modulated continuous wave (i.e., "FMCW"). It should also be understood that any techniques described herein for time-of-flight based systems using pulsed signals can also be applied to LiDAR systems that do not use one or both of these techniques.

[0145] Referring back to Figure 10A (e.g., illustrating a time-of-flight LiDAR system using light pulses), when light pulse 1002 reaches object 1006, light pulse 1002 is scattered or reflected to form a returning light pulse 1008. Returning light pulse 1008 can return to system 1000 along optical path 1010. The time from when emitted light pulse 1002 leaves LiDAR system 1000 to when returning light pulse 1008 returns to LiDAR system 1000 can be measured (e.g., via a processor or other electronic device within the LiDAR system, such as control circuitry system 350). This time-of-flight, combined with knowledge of the speed of light, can be used to determine the distance / range from LiDAR system 1000 to the portion of object 1006 where light pulse 1002 is scattered or reflected.

[0146] As depicted in Figure 10B, the LiDAR system 1000 scans the external environment by guiding a series of light pulses (e.g., by guiding light pulses 1002, 1022, 1026, and 1030 along optical paths 1004, 1024, 1028, and 1032, respectively). As depicted in Figure 10C, the LiDAR system 1000 receives return light pulses 1008, 1042, and 1048 (corresponding to the emitted light pulses 1002, 1022, and 1030, respectively). The return light pulses 1008, 1042, and 1048 are formed by scattering or reflecting emitted light pulses by one of objects 1006 and 1014. The return light pulses 1008, 1042, and 1048 can return to the LiDAR system 1000 along optical paths 1010, 1044, and 1046, respectively. Based on the direction of the emitted light pulse (as determined by the LiDAR system 1000) and the calculated distance from the LiDAR system 1000 to the portion of the object scattering or reflecting the light pulse (e.g., portions of objects 1006 and 1014), the external environment within the detectable range (e.g., the field of view between paths 1004 and 1032, included) can be precisely mapped or plotted (e.g., by generating a 3D point cloud or image).

[0147] If no corresponding light pulse is received for a specific emitted light pulse, the LiDAR system 1000 can determine that there is no object within its detectable range (e.g., the object is outside the maximum scan distance of the LiDAR system 1000). For example, in Figure 10B, light pulse 1026 may not have a corresponding return light pulse (as illustrated in Figure 10C) because light pulse 1026 may not generate a scattering event along its transmission path 1028 within the predetermined detectable range. The LiDAR system 1000 or an external system (e.g., a cloud system or service) communicating with the LiDAR system 1000 may interpret the lack of a return light pulse as the absence of an object positioned along the optical path 1028 within the detectable range of the LiDAR system 1000.

[0148] In Figure 10B, light pulses 1002, 1022, 1026, and 1030 can be emitted in any order, serially, in parallel, or based on other timing relative to each other. Additionally, although Figure 10B depicts the emitted light pulses as being guided in one dimension or plane (e.g., the plane of paper), the LiDAR system 1000 can also guide the emitted light pulses along other dimensions or planes. For example, the LiDAR system 1000 can also guide the emitted light pulses in a dimension or plane perpendicular to the dimension or plane shown in Figure 10B, thereby forming a 2D transmission of the light pulses. This 2D transmission of the light pulses can be point-by-point, line-by-line, one-time, or otherwise. That is, the LiDAR system 1000 can be configured to perform point scans, line scans, single scans without scanning, or combinations thereof. The point cloud or image (e.g., a single horizontal line) from the 1D transmission of the light pulses can generate 2D data (e.g., (1) data from the horizontal transmission direction and (2) the extent or distance to the object). Similarly, point clouds or images from 2D transmissions of light pulses can generate 3D data (e.g., (1) data from the horizontal transmission direction, (2) data from the vertical transmission direction, and (3) the extent or distance to the object). Typically, a LiDAR system performing an n-dimensional transmission of light pulses generates (n+1)-dimensional data. This is because the LiDAR system can measure the depth of an object or the distance to the object, which provides an additional dimension of data. Therefore, a 2D scan performed by a LiDAR system can generate a 3D point cloud for mapping the external environment of the LiDAR system.

[0149] Point cloud density refers to the number of measurements (data points) performed by a LiDAR system for each region. Point cloud density is related to the LiDAR scan resolution. Generally, at least for the region of interest (ROI), a higher point cloud density is desired, and therefore a higher resolution is required. The point density in a point cloud or image generated by a LiDAR system is equal to the number of pulses divided by the field of view. In some embodiments, the field of view can be fixed. Therefore, to increase the density of points generated by a set of transmit-receive optics (or transceiver optics), a LiDAR system may need to generate pulses more frequently. In other words, the light source in a LiDAR system can have a higher pulse repetition rate (PRR). On the other hand, by generating and transmitting pulses more frequently, the maximum distance that a LiDAR system can detect may be limited. For example, if a return signal from a distant object is received after the system transmits the next pulse, the return signal may be detected in a different order than the corresponding transmitted signal, resulting in ambiguity if the system cannot correctly correlate the return signal with the transmitted signal.

[0150] To illustrate, consider an exemplary LiDAR system capable of emitting laser pulses with repetition rates between 500 kHz and 1 MHz. Based on the time it takes for the pulse to return to the LiDAR system, and to avoid confusion between return pulses from continuous pulses in a typical LiDAR design, the maximum detection range of the LiDAR system could be 300 meters for 500 kHz and 150 meters for 1 MHz. The point density of a LiDAR system with a repetition rate of 500 kHz is half that of a 1 MHz system. Therefore, this example shows that increasing the repetition rate from 500 kHz to 1 MHz (and thus increasing the point density) may reduce the system's detection range if the system cannot properly correlate out-of-order arriving return signals. Various techniques are used to mitigate the trade-off between a higher PRR and limited detection range. For example, multiple wavelengths can be used to detect objects within different ranges. Optical and / or signal processing techniques (e.g., pulse coding techniques) are also used to correlate the emitted and returned optical signals.

[0151] The various systems, apparatuses, and methods described herein can be implemented using digital circuit systems or using one or more computers that utilize well-known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

[0152] The various systems, apparatuses, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers. Examples of client computers may include desktop computers, workstations, laptops, cellular smartphones, tablets, or other types of computing devices.

[0153] The various systems, apparatuses, and methods described herein can be implemented using computer program products tangibly contained in an information carrier, such as a non-transitory machine-readable storage device, for execution by a programmable processor; and the methods, processes, and steps described herein (including one or more steps of at least some of Figures 1 to 23) can be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a specific activity or produce a specific result. Computer programs can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0154] Figure 11 illustrates a simplified block diagram of an exemplary apparatus that can be used to implement the systems, devices, and methods described herein. Apparatus 1100 includes a processor 1110 operatively coupled to persistent storage device 1120 and main memory device 1130. Processor 1110 controls the overall operation of apparatus 1100 by executing computer program instructions that define these operations. The computer program instructions may be stored in persistent storage device 1120 or other computer-readable medium and loaded into main memory device 1130 when execution of the computer program instructions is desired. For example, processor 1110 may be used to implement one or more components and systems described herein, such as control circuitry system 350 (shown in Figure 3), vehicle perception and planning system 220 (shown in Figure 2), and vehicle control system 280 (shown in Figure 2). Therefore, at least some of the method steps in Figures 1 through 23 may be defined by computer program instructions stored in main memory device 1130 and / or persistent storage device 1120, and controlled by processor 1110 executing the computer program instructions. For example, computer program instructions can be implemented as computer-executable code programmed by a person skilled in the art to execute an algorithm defined herein by at least some of the method steps discussed in conjunction with Figures 1 to 13. Accordingly, by executing the computer program instructions, processor 1110 executes the algorithm defined by the method steps of these foregoing figures. Apparatus 1100 also includes one or more network interfaces 1180 for communicating with other devices via a network. Apparatus 1100 may also include one or more input / output devices 1190 that enable a user to interact with apparatus 1100 (e.g., a display, keyboard, mouse, speaker, buttons, etc.).

[0155] Processor 1110 may include both general-purpose microprocessors and special-purpose microprocessors, and may be the sole processor of device 1100 or one of multiple processors. Processor 1110 may include one or more central processing units (CPUs) and one or more graphics processing units (GPUs), the GPUs of which may, for example, operate independently of one or more CPUs and / or perform multitasking with one or more CPUs to accelerate processing, for example, for the various image processing applications described herein. Processor 1110, persistent storage device 1120, and / or main memory device 1130 may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), or be supplemented by one or more ASICs and / or one or more FPGAs, or incorporated into one or more ASICs and / or one or more FPGAs.

[0156] Persistent storage device 1120 and main memory device 1130 each include a tangible, non-transitory computer-readable storage medium. Persistent storage device 1120 and main memory device 1130 may each include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double-rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor storage devices (such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), optical disc read-only memory (CD-ROM), digital universal optical disc read-only memory (DVD-ROM), or other non-volatile solid-state storage devices.

[0157] Input / output device 1190 may include peripheral devices such as printers, scanners, displays, etc. For example, input / output device 1190 may include display devices for displaying information to a user (such as cathode ray tube (CRT), plasma or liquid crystal display (LCD) monitors, keyboards) and pointing devices (such as mice or trackballs) that a user can use to provide input to device 1100.

[0158] Any or all of the functions of the systems and apparatuses discussed herein may be executed by processor 1110 and / or incorporated into an apparatus or system such as LiDAR system 300. Furthermore, LiDAR system 300 and / or apparatus 1100 may utilize one or more neural networks or other deep learning techniques executed by processor 1110 or other systems or apparatuses discussed herein.

[0159] Those skilled in the art will recognize that the actual computer or computer system implementation may have other structures and may include other components, and Figure 11 is a simplified representation of some of the components of such a computer for illustrative purposes.

[0160] Figure 12A is a block diagram illustrating an exemplary HyDAR system 1200 according to various embodiments. Figure 12B is a side view illustrating an exemplary HyDAR system 1200 according to various embodiments. In some examples, as shown in Figures 12A and 12B, the HyDAR system 1200 includes a LiDAR sensor 1202 and an image sensor 1204. As described above, the LiDAR sensor 1202 and the image sensor 1204 may be integrated together (e.g., disposed in the same semiconductor die or package) or combined together to form a multimode sensor. The LiDAR sensor 1202 may be configured to detect a first returned light signal having a first wavelength (e.g., a light signal in the infrared wavelength range), and the image sensor 1204 may be configured to detect a second returned light signal having a second wavelength (e.g., a light signal in the visible light wavelength range). The LiDAR sensor 1202 may include at least a portion of, for example, the optical receiver and photodetector 330 described above in conjunction with Figure 3 and / or at least a portion of the optical receiver and photodetector 430 of the multimode detection system 400 described above in conjunction with Figure 4. The multimode sensor formed by the LiDAR sensor 1202 and the image sensor 1204 can be substantially the same as the multimode sensor 450 described above. In some examples, the image sensor 1204 may include a near-infrared (NIR) sensor, a mid-infrared (MIR) sensor, and / or a visible light sensor.

[0161] Referring again to Figure 12A, the HyDAR system 1200 also includes, for example, one or more steering mechanisms 1206, lenses or lens groups 1210 and 1212, an aperture window 1208, and a controller 1214. For simplicity, other components of the HyDAR system 1200 are omitted in Figure 12A. For example, a light source is not shown in Figure 12A, but it should be understood that the HyDAR system 1200 may include a light source, transmitter, and other components similar to those in the multi-mode detection system described above (e.g., system 400).

[0162] The LiDAR sensor 1202 and image sensor 1204 shown in Figure 12A form a multi-mode sensor and therefore share some of the transmitting and receiving optical paths. For example, as shown in Figure 12A, one or more steering mechanisms 1206 may be shared between the LiDAR sensor 1202 and the image sensor 1204. A light source (not shown in Figure 12A) sends a laser signal to the steering mechanism 1206 (which may include a polygonal mirror, an oscillating mirror, a prism, a lens, and / or any other optical component configured to redirect the light). One or more steering mechanisms 1206 redirect the laser signal to form a transmitted light signal 1201, which scans across the FOV 1220 in both horizontal and vertical directions to illuminate one or more objects in the FOV 1220. The transmitted light signal 1201 may have a wavelength in the infrared wavelength range. When the transmitted light signal 1201 is scattered or reflected by one or more objects in the FOV 1220, a first returned light signal 1203 is formed. The first returned optical signal 1203 has the same wavelength as the transmitted optical signal 1201 and is guided back to the redirection mechanism 1206. The redirection mechanism 1206 then redirects the first returned optical signal 1203 toward the LiDAR sensor 1202 via one or more receiving optical components (such as a collecting lens or lens group 1210). Thus, one or more redirection mechanisms 1206 are used to emit laser signals toward the aperture window 1208 and receive the first returned optical signal 1203 formed based on at least a portion of the laser signal provided by the laser source.

[0163] At least a portion of the optical path associated with the LiDAR sensor 1202 described above may be shared with the image sensor 1204. The image sensor 1204 may be a camera that senses the second returned light signal 1205 (e.g., NIR / MIR / visible light). The second returned light signal 1205 is formed by light provided by one or more external light sources outside the HyDAR system 1200. Such light sources may include, for example, sunlight, moonlight, vehicle headlights, and / or streetlights. These light sources may emit light directly to the HyDAR system 1200. The second returned light signal 1205 may also include reflection, transmission, and / or refraction of light emitted by the aforementioned light sources. Other visible light sources are also possible, and the second returned light signal 1205 is not limited to the aforementioned light signals. The second returned light signal 1205 is also received by one or more redirection mechanisms 1206. The redirection mechanism 1206 then redirects the second returned light signal 1205 to the image sensor 1204 via one or more receiving optical components (such as a collecting lens or lens group 1212 (FIG. 12A)). Lens or lens group 1212 may be a focusing lens (Figure 12B). In some examples, collecting lens 1210 and focusing lens 1212 may be the same lens, and other optical components (e.g., mirrors, wavelength splitters, etc.) are used to separate the first and second returned light signals and redirect them to their respective LiDAR sensor 1202 and image sensor 1204. The details of light collection and distribution, as well as signal separation, have been described above in conjunction with Figures 6 and 7, and therefore will not be repeated.

[0164] Although Figure 12A illustrates LiDAR sensor 1202 and image sensor 1204 located on opposite sides of one or more steering mechanisms 1206, they can be located on the same side. Figure 12B illustrates a side view of this configuration, where LiDAR sensor 1202 and image sensor 1204 are located on the same side of steering mechanism 1206 and aligned to receive their respective return light signals. In some examples, because the wavelengths of the return light signals received by image sensor 1204 and LiDAR sensor 1202 are different, a focusing lens 1212 is used to adjust the focal length of the optical path of the second return light signal 1205. As shown in Figure 12B, image sensor 1204 and LiDAR sensor 1202 can be placed on the same semiconductor substrate / die / package / PCB, and therefore, focusing lens 1212 can adjust the focal length of the visible light received and redirected by steering mechanism 1206.

[0165] As shown in Figure 12B, in some examples, the LiDAR sensor 1202 and the image sensor 1204 share at least a portion of the receiving optical path. Specifically, both the first returned light signal 1203 and the second returned light signal 1205 are received through the aperture window 1208 and redirected by the steering mechanism 1206 and other shared optical components (e.g., the same collecting lens). The focal length of one optical path of the second returned light signal 1205 can be adjusted using the focusing lens 1212. Therefore, when the first returned light signal 1203 and the second returned light signal 1205 are received by the LiDAR sensor 1202 and the image sensor 1204, respectively, they may already be temporally and spatially synchronized at the hardware level.

[0166] Referring again to Figures 12A and 12B, LiDAR sensor 1202 acquires one or more frames of point cloud data based on a received first returned light signal 1203 (e.g., a returned pulse in the infrared wavelength range). For example, LiDAR sensor 1202 can convert the first returned light signal 1203 into an analog electrical signal, which can then be sampled and digitized to form one or more frames of point cloud data 1218. Point cloud data 1218 represents FOV 1220 based on a scan of FOV 1220 using a transmitted light signal 1201 of a first wavelength (e.g., infrared light). Image sensor 1204 acquires one or more frames of image data 1216 based on a second returned light signal 1205 (e.g., visible light reflected from one or more objects in FOV 1220). For example, image sensor 1204 can be a camera with high resolution, enabling it to capture FOV 1220 based on a received light signal of a second wavelength (visible light / NIR / MIR). Image sensor 1204 (e.g., CMOS camera, CCD camera) converts detected light signals into electrical signals to form one or more frames of image data 1216.

[0167] As described above, when discrete sensors are mounted separately on a vehicle (or another mobile platform), the data captured by different sensors (e.g., a LiDAR sensor and an image sensor such as a camera) typically needs to be time-synchronized and / or converted to use the same coordinate system. This is because discrete sensors are not time-synchronized, so images captured by discrete sensors may be offset in time. Furthermore, discrete sensors may be mounted at different locations on the vehicle (e.g., a LiDAR sensor mounted on the top of the vehicle, while a camera is mounted on a side mirror). Therefore, they capture images from different angles. To fuse the image data provided by discrete sensors, a coordinate system transformation is required. This complex process is often referred to as data fusion. The data fusion process can involve a significant amount of computational work using vectors and matrices because both the LiDAR sensor and the image sensor can generate a large amount of data per unit of time (e.g., per second, per minute, or per hour). And if the LiDAR sensor and the image sensor operate over a long period of time (e.g., when they are used on a vehicle and operate for several hours), they can generate even larger amounts of data over time. Therefore, data fusion processes typically require systems with very high computing power, and thus often require additional hardware (e.g., many GPUs) and / or software support. Consequently, data fusion processes can be cumbersome, error-prone, inefficient, costly, and power-intensive.

[0168] At least some of the above problems can be solved by integrating multiple sensors into the multi-mode sensor disclosed herein (e.g., sensor 450). For example, in FIG12A, LiDAR sensor 1202 and image sensor 1204 are integrated together. Thus, LiDAR sensor 1202 and image sensor 1204 are disposed in a semiconductor die / device package / PCB and can share at least a portion of the transmit / receive optical path. As shown in FIG12A and FIG12B, steering mechanism 1206 may have an optical scanner (e.g., a polygonal mirror, an oscillating mirror, or a combination thereof) configured to perform: (1) scanning laser signal 1201 in the horizontal and vertical directions; receiving a first return optical signal 1203 and a second return optical signal 1205; and directing the first return optical signal 1203 and the second return optical signal 1205 to LiDAR sensor 1202 and image sensor 1204, respectively. Due to hardware-level integration, the point cloud data 1218 generated by the LiDAR sensor 1202 and the image data 1216 generated by the image sensor 1204 are synchronized in time and space. Specifically, as shown in Figure 12A, corresponding frames between the point cloud data 1218 and the image data 1216 are already synchronized in time. Furthermore, the LiDAR sensor 1202 and the image sensor 1204 capture the same angle of the field of view 1220. Therefore, no coordinate transformation is required. The data 1218 and 1216 from the two sensors 1202 and 1204, respectively, can be directly fused together by the controller 1214 without time synchronization or coordinate transformation. Therefore, the data fusion process can be achieved early through hardware configuration (e.g., using the same steering mechanism 1206 and sharing other optical components). This type of data fusion is therefore referred to as early fusion, compared to data fusion performed later in software in vehicle planning and perception systems. The latter type of data fusion requires the computer system to acquire point cloud data and image data separately, align the two sets of data in time, and perform coordinate transformations between the two sets of data to establish their correlation. Therefore, early fusion significantly improves data processing efficiency and reduces computational workload and power consumption.

[0169] As described above, the HyDAR system 1200 may include a controller 1214. The controller 1214 can control the operation of various components of the HyDAR system 1200 and can also process data generated by a multi-mode sensor, including a LiDAR sensor 1202 and an image sensor 1204. Based on this data (e.g., data already fused at the hardware level), the controller 1214 can also be configured to detect one or more degradation factors affecting the performance of the HyDAR system and adjust one or more device configurations and / or one or more operating conditions of the HyDAR system to eliminate or reduce the effects of degradation factors. In some other cases, in addition to the controller 1214, another circuitry or computer system may be connected to the HyDAR system 1200 to perform at least some of the operations described above. As described above, various external and internal factors may affect the performance of the HyDAR system, and therefore the performance of the HyDAR system may degrade over time. Degradation factors for a HyDAR system may include, for example, at least partial occlusion of the aperture window, interference signals provided by one or more interfering light sources, external calibration degradation measured by the relationship between the HyDAR system and the mobile platform on which the HyDAR system is mounted, and internal calibration degradation associated with misaligned internal components of the HyDAR system. Each of these degradation factors will be described in detail below.

[0170] Figure 13 is a flowchart 1300 according to various embodiments, illustrating a method for probing the operation of a HyDAR system (e.g., HyDAR system 1200) to detect one or more performance degradation factors. As shown in Figure 13, in some examples, a light source or emitter in the HyDAR system provides a laser signal (box 1302). One or more steering mechanisms of the HyDAR system guide the laser signal toward an aperture window (box 1304). If the aperture window is not completely blocked, the laser signal passes through the aperture window. In box 1306, the HyDAR system receives a first return light signal based on at least a portion of the laser signal. In box 1312, the HyDAR system receives a second return light signal provided by one or more light sources outside the HyDAR system. The first and second return light signals may be received by the same steering mechanism of the HyDAR system as described above.

[0171] Next, in box 1308, the LiDAR sensor in the HyDAR system detects a first return light signal to obtain one or more frames of point cloud data. In box 1314, the image sensor in the HyDAR system detects a second return light signal to obtain one or more frames of image data. As described above, the point cloud data and image data are synchronized temporally and spatially at the hardware level, eliminating the need for additional processors (e.g., GPUs) or software to synchronize them. The controller of the HyDAR system, based on one or both of the point cloud data and image data, can perform detection of one or more performance degradation factors affecting the performance of the HyDAR system (box 1320). Such detection includes detecting at least partial window occlusion of the aperture window (box 1330), detecting interference signals provided by one or more interfering light sources (box 1340), detecting external calibration degradation measured through the relationship between the HyDAR system and the mobile platform on which the HyDAR system is mounted (box 1350), and detecting internal calibration degradation associated with misaligned internal components of the HyDAR system (box 1360). In some examples, the controller of the HyDAR system can also adjust the device configuration and / or operating conditions of the HyDAR system to eliminate or reduce the negative impact of degradation factors (box 1322). Boxes 1330, 1340, 1350 and 1360 will be described in more detail below.

[0172] Beginning at box 1330, the detection of at least partial window occlusion in the HyDAR system is described in detail. Figure 14A is a histogram illustrating a comparison of the returned signal strength between an at least partially occluded aperture window and an unoccluded aperture window according to various embodiments. In Figure 14A, the vertical axis represents signal strength, and the horizontal axis represents the number of received returned optical signals falling into each signal strength interval. As shown in Figure 14A, signals 1402 and 1404 represent the first returned optical signals formed based on laser signals emitted by the HyDAR system. If the aperture window (e.g., window 1208 shown in Figures 12A and 12B) is not occluded, the returned optical signal 1402 may have normal intensity, depending on whether the returned optical signal 1402 is formed by an object located at or near the aperture window or by an object at a distance from the aperture window. Generally, the farther away the object is, the weaker the intensity of the returned optical signal. In contrast, if the aperture window is at least partially obstructed, the returned light signal 1404 has a higher signal strength compared to the returned light signal 1402 (based on unobstructed normal scattering reception). This is because, due to the obstruction, more of the laser signal from the HyDAR system is scattered towards the detector. Therefore, the returned light signal 1404 is formed by scattering from the aperture window whose emitted light signal is obstructed, the object obstructing the aperture window, and / or other internal components of the HyDAR system. Thus, the signal strength of the returned light signal 1404 can be quite high. Therefore, aperture window obstruction can be detected based on the signal strength of the returned light signal formed based on the emitted laser signal. In other words, in some cases, window obstruction can be detected using only the point cloud data generated by the LiDAR sensor. In other examples, as described in more detail below, both point cloud data generated by the LiDAR sensor and image data generated by the image sensor can be used to detect window obstruction. The addition of image data provides more information about the obstruction and helps improve the accuracy of obstruction detection and / or the classification of obstructing objects.

[0173] Many types of objects can cause occlusion of the aperture window of a HyDAR system. Figure 14B is a diagram illustrating at least partial occlusion of the aperture window of a HyDAR system due to several different types of objects according to various embodiments. For example, as shown in Figure 14B, the aperture window 1414 of the HyDAR system 1412 may be occluded (or partially occluded) by object 1418 (e.g., a leaf) or object 1422 (e.g., condensation / raindrops). Objects 1418 and / or 1422 may be located at or near the aperture window, thus occluding at least a portion of the window 1414. Other types of objects may also occlude the window 1414, such as plastic bags, paper, debris, dirt, snow, etc. If the window 1414 is at least partially occluded, the transmitted light signal emitted from the laser source of the HyDAR system 1412 may be blocked or scattered, for example, by objects 1418 and / or 1422. The return signal generated by objects 1418 and / or 1422 may correspond to signal 1404 in FIG. 14A, which has a high signal strength. FIG. 14B shows that the aperture window 1414 is partially blocked, and therefore, the transmitted light signal can pass through the window 1414 and may reach object 1416 in the FOV. Object 1416 may be an object located at a certain distance (near or far) from the aperture window 1414 and may form a first return light signal corresponding to signal 1402 in FIG. 14A. The following disclosure describes embodiments of several methods for detecting at least partial window occlusion of an aperture window and determining the location, type, and degree of occlusion.

[0174] Figures 15A to 15D are flowcharts illustrating various methods for detecting aperture window occlusion in a HyDAR system according to various embodiments. Figure 15A provides an overview of methods for detecting aperture window occlusion using point cloud data, image data, combinations thereof, and / or fused data. As shown in Figure 15A, method 1500 can be performed by a controller (e.g., the control circuitry system 350 or controller 1214 described above) or another computing device (e.g., the device shown in Figure 11). The controller may include an analog circuitry system for processing analog signals (e.g., analog voltage or current signals converted from returned light signals), a digital circuitry system for processing digital signals (e.g., digitized / sampled analog voltage or current signals), or a mixed-signal circuitry system for processing mixed signals. For simplicity, method 1500 is described below by using a controller to detect at least partial window occlusion of the aperture window of a HyDAR system. Method 1500 corresponds to block 1330 in Figure 13.

[0175] As described above, in a HyDAR system (e.g., system 1200), a LiDAR sensor (e.g., sensor 1202) can convert a first returned light signal (e.g., signal 1203) into an electrical signal, and generate point cloud data (e.g., data 1218) based on these electrical signals. An image sensor (e.g., sensor 1204) can convert a second returned light signal (e.g., signal 1205) into an electrical signal, and generate image data (e.g., data 1216) based on these electrical signals. The point cloud data may include one or more frames. One frame of point cloud data may correspond to a complete scan of the field of view (FOV). Similarly, image data may include one or more frames. One frame of image data may refer to an image captured at a predetermined resolution. The methods described below in Figures 15A to 15D include: detecting window occlusion using a serial process based on both point cloud data and image data; detecting window occlusion using a parallel process based on both point cloud data and image data; detecting window occlusion using fused point cloud data and image data; and detecting window occlusion using only image data or only point cloud data.

[0176] Figure 15A provides an overview of various methods for occlusion detection, and therefore not all boxes in Figure 15A are necessary. For example, if fused point cloud data and image data are used (box 1506), then determination boxes 1508 and 1510 may not be necessary. Referring to Figure 15A, in box 1502 of method 1500, the controller obtains a first deviation of a first returned light signal relative to a first expected value based on one or more frames of point cloud data. In box 1504 of method 1500, the controller obtains a second deviation of a second returned light signal relative to a second expected value based on one or more frames of image data. One or both of boxes 1502 and 1504 can be performed. The first returned light signal corresponds to a first region of the aperture window. The second returned light signal corresponds to a second region of the aperture window.

[0177] The first and second regions of the aperture window can be the same region or they can be different regions. Therefore, the LiDAR sensor and the image sensor can capture the same region of the aperture window simultaneously or not. In one example, the two sensors can acquire a first and a second return light signal corresponding to the same region of the aperture window at the same specific time. In this way, the point cloud data generated by the LiDAR sensor and the image data generated by the image sensor are automatically synchronized in time and space at that specific time. In other examples, the first and second regions of the aperture window can overlap at a specific time. Therefore, the point cloud data generated by the LiDAR sensor and the image data generated by the image sensor are automatically and partially synchronized in time and space at that specific time. Partially synchronized point cloud data and image data can be useful for applications that may not require strict synchronization.

[0178] As described above with reference to Figures 14A and 14B, if the aperture window of a LiDAR sensor or HyDAR system is at least partially obstructed, the returned light signal may deviate from the expected value. For LiDAR sensors, Figure 14A illustrates a comparison between the intensity of a first returned light signal 1402 without window obstruction and the intensity of a first returned light signal 1404 with at least partial window obstruction. If obstruction is present, the signal strength is much higher than the expected or range value compared to the case without obstruction. For image sensors, if window obstruction is present, the returned signal strength may be much lower than the case without window obstruction. For example, if a leaf partially covers the aperture window, an image sensor (e.g., a camera) will detect a much lower light intensity.

[0179] Signal strength is just one of many characteristics that can be used to detect deviations from expected values. Other characteristics may include, but are not limited to: average signal strength; signal strength distribution; size and / or shape associated with the time-varying behavior of one or both of the first and second regions of the aperture window; sensitivity to a predetermined wavelength range; the number of points in the point cloud data; or the distance distribution represented by the point cloud data or image data. For all these characteristics, there may be predetermined expected values ​​(e.g., normal ranges for intensity, distribution, sensitivity, number of points, and distance distribution). Based on point cloud data, the controller can detect changes in these characteristics, which represent a first deviation. Similarly, based on image data, the controller can detect changes in some of these characteristics, which represent a second deviation.

[0180] Referring again to Figure 15A, in some examples, the controller executes at least one of blocks 1508 or 1510. That is, the controller determines whether a first deviation exceeds a first threshold (block 1508), whether a second deviation exceeds a second threshold (block 1510), or both. For either the first or second deviation, if the deviation from the expected value is small (e.g., less than the corresponding first or second threshold), it may mean that aperture window occlusion is unlikely or there is no obvious occlusion. For example, the deviation may be caused by other factors such as noise, interference, external calibration degradation, internal calibration degradation, etc. (described below). Thus, further detection of window occlusion may not be performed. If the first deviation, the second deviation, or both exceed their respective thresholds, it may mean that window occlusion may exist. If any deviation does not exceed the corresponding threshold, method 1500 may loop back to obtain the next frame of point cloud data and / or the next frame of image data.

[0181] As shown in Figure 15A, the controller can perform a logical OR operation (box 1512) on the outputs of blocks 1508 and 1510, and if the result of the OR operation is positive (or high), the controller determines (box 1516) one or more locations, one or more types, and degrees of occlusion of at least part of the aperture window. For example, point cloud data may have or be used to derive location information (X, Y coordinates), reflectivity information, distance information, and other information associated with the first returned light signal (e.g., velocity, orientation, etc. derived from the first returned light signal). Image data may have a much higher resolution than point cloud data. Image data may also have information not included in point cloud data. For example, image data may include color information, brightness, contrast, etc. Therefore, if the controller determines that aperture window occlusion may exist (e.g., because a first deviation or a second deviation or both exceed their respective thresholds), the point cloud data and image data can be used to determine the location, type, and degree of window occlusion. For example, if for a specific area of ​​the aperture window, a first deviation indicates an abnormally high signal strength (because the LiDAR transmitted light signal is reflected by an obstructing object), and a second deviation indicates an abnormally low signal strength in the same area (because visible light from an external light source is blocked and cannot enter the image sensor), the controller can determine that there is window occlusion in that area.

[0182] Based on point cloud data, image data, or both, the controller can further determine the shape and size of the occluded area. Based on the values ​​of a first bias and / or a second bias, the controller can also determine the degree of window occlusion. For example, if the values ​​of the first bias and / or the second bias do not significantly exceed their respective thresholds, it may mean that the object obscuring the aperture window still allows some transmitted light signals to escape or some returned light signals to enter from outside the HyDAR system. This type of object can include, for example, condensation, raindrops, plastic bags, or other transparent or translucent objects. In contrast, if the values ​​of the first bias and / or the second bias significantly exceed their respective thresholds, it may mean that the object obscuring the aperture window is considerably opaque (e.g., a leaf).

[0183] The shape or size of an occluded object can be obtained, for example, by acquiring the deviation distribution of point cloud data and / or image data. The deviation value corresponding to the occluded area of ​​the aperture window may differ significantly from the deviation value of the unoccluded area. Thus, the shape and size of the occluded area can be derived. In some examples, utilizing image data and one or more pattern recognition algorithms (e.g., AI / ML-based pattern recognition algorithms), the controller can even identify the type of occluded object. For example, using a neural network trained to recognize objects, the controller can determine that the occluded object is a plastic bag, a leaf, a raindrop, etc.

[0184] Referring again to Figure 15A, if the controller determines that the first deviation does not exceed the first threshold (box 1508), it can process only the next frame of point cloud data (box 1509), and the process loops back to box 1502 to obtain the first deviation based on the next frame of point cloud data. Similarly, if the controller determines that the second deviation does not exceed the second threshold (box 1510), it can process only the next frame of image data (box 1511), and the process loops back to box 1504 to obtain the second deviation based on the next frame of image data.

[0185] As described above, if any one or both of the determinations at boxes 1508 and 1510 are yes, the controller determines (box 1516) that the aperture window is at least partially occluded, and further determines the location, type, and extent of the at least partial window occlusion. In method 1500 shown in Figure 15A, the controller may further determine (box 1518) whether the at least partial window occlusion lasts for at least one data collection cycle. In some scenarios, the occlusion of the aperture window may not be continuous. For example, an object occluding the window (e.g., a leaf, a plastic bag, a flower, etc.) may disappear or move its position in the next second. Therefore, the controller can be configured to detect whether the window occlusion lasts for at least one data collection cycle. The data collection cycle may be a time period of one frame for collecting point cloud data and / or image data. It may also correspond to a time period with other predetermined lengths.

[0186] In some examples, if the controller determines that the at least partial window obstruction persists for at least one data collection cycle, the controller may perform at least one of the following actions: report (box 1520) the at least partial window obstruction; or activate (box 1522) an obstruction removal mechanism for removing the at least partial window obstruction. For example, the controller may activate one or more nozzles to spray fluid (e.g., compressed air, water, etc.) in an attempt to blow away the object causing the at least partial window obstruction. It may activate windshield wipers or any other mechanism (e.g., a heater for removing condensation, a fan for blowing away plastic bags or leaves, etc.) to remove the obstructing object. If the partial window obstruction cannot be removed after activating some removal mechanism, the controller may report the at least partial window obstruction to the user or system (e.g., a vehicle planning system or a vehicle control system).

[0187] The process described above uses point cloud data and image data in a separate processing pipeline to detect at least partial window occlusion. Figure 15 also illustrates another process using fused point cloud data and image data to detect at least partial window occlusion. As mentioned above, in the multi-mode sensors of the described HyDAR system, when point cloud data is provided by a LiDAR sensor and image data by an image sensor, they are already temporally and spatially synchronized at the hardware level. Therefore, point cloud data and image data can be easily fused together (box 1506) to obtain fused data. The fusion process may not require time offset and / or coordinate transformation between point cloud data and image data, or may require significantly less computational effort. In one example, for further fusion, the outputs from both the LiDAR sensor and the image sensor are provided to a fusion processor or data merger (e.g., part of the controller, a dedicated processor, or any other computer). Based on the fused data, the controller can detect (box 1514) the fused occlusion detection results. For example, because the fused data includes both point cloud data and image data, the confidence level of the occlusion detection can be improved. For example, if the controller uses only point cloud data from a LiDAR sensor to detect occlusion, the confidence level of occlusion detection may be only 50%. If the controller uses fused data (which includes image data), the confidence level of occlusion detection (or no occlusion detected) can be increased to 90%. After occlusion detection, the process can proceed to boxes 1518, 1520, and 1522, which are the same as described above.

[0188] Figure 15A provides an overview of a method 1500 for detecting at least partial window occlusion (corresponding to box 1330 in Figure 13). Figures 15B through 15D provide several specific embodiments of method 1500. Referring first to Figure 15B, in method 1500A, the controller may begin by acquiring a frame of data including point cloud data and image data (box 1531). The point cloud data represents a first returned light signal on a first region (box 1532), and the image data represents a second returned light signal on a second region (box 1534). As mentioned above, the first and second regions may be the same region or may be different regions with overlap. In boxes 1538 and 1540, the controller determines whether there is a first deviation of the point cloud data relative to a first expected value (box 1539), and whether there is a second deviation of the image data relative to a second expected value (box 1540). The first and second deviations can represent: one or more variations in signal strength or average signal strength; variations in the distribution of signal strength; variations in the size and / or shape over time associated with one or both of the first and second regions of the aperture window; variations in sensitivity to a predetermined wavelength range; variations in the number of points in the point cloud data; and / or variations in the distance distribution represented by the point cloud data or image data.

[0189] In some examples, in boxes 1538 and 1540, the controller further determines whether a first deviation and a second deviation exceed a first threshold and a second threshold, respectively. If yes, the output of box 1538 and / or box 1540 is "yes," and vice versa. In some examples, the controller only determines whether any deviation exists relative to the expected value, and if so, the output of box 1538 and / or box 1540 is "yes," and vice versa. It should be understood that boxes 1538 and 1540 can be executed in any order. For example, the determination of deviations of point cloud data and image data relative to their respective expected values ​​can be performed in parallel time. Alternatively, the determination of deviations of point cloud data relative to its expected value can be performed before the determination of deviations of image data relative to its expected value, and vice versa. As shown in Figure 15B, an "OR" operation is performed on the determined outputs in boxes 1538 and 1540 such that if any of the determined outputs is "yes," the controller calculates the occlusion location, occlusion type, and occlusion degree. Furthermore, if the determined outputs of boxes 1538 and 1540 are both "No", the controller can continue to obtain the next frame of data (box 1549).

[0190] In some examples, if the determinations of boxes 1538 and 1540 are different, the controller can also determine priority or confidence levels based on certain rules or settings, environmental conditions, or other factors. For example, the controller can be configured to set a higher priority for the determination result of box 1538 (i.e., a determination based on point cloud data provided by a LiDAR sensor) than for the determination result of box 1540 (i.e., a determination based on image data provided by an image sensor), and vice versa. The controller can also be configured to set a higher confidence level for the determination result of box 1538 than for the determination result of box 1540, and vice versa. The controller can be further configured to set priority or confidence levels based on factors such as environmental conditions. For example, under certain weather conditions, one sensor may perform better than another, and therefore, a higher priority or confidence level can be set for the better-performing sensor. As an example, when an image sensor is directly facing the sun, it may saturate and fail to capture any useful information. In contrast, a LiDAR sensor may perform well under such conditions. In this scenario, the determination of point cloud data provided by LiDAR sensors can be set to have a higher priority or a higher confidence level. It should be understood that the controller can be configured to have other priority and / or confidence level settings associated with the deviation between the point cloud data and the image-based determination data.

[0191] In method 1500A shown in Figure 15B, the controller can increment an occlusion counter by 1 for each detected occlusion location of the aperture window (box 1547). The occlusion counter may correspond to a data collection period as described in Figure 15A. Therefore, if occlusion persists for more than one data collection period, the occlusion counter is greater than 1. In other examples, there may be multiple occlusion counters, each corresponding to a specific location within the aperture window. Therefore, the values ​​of the occlusion counters may differ in different regions of the aperture window. For example, a specific location within the aperture window may have a higher counter value, indicating that the occlusion at that location may be more severe than at another location (e.g., it may be blocked and / or allow less light to pass through over multiple data collection periods).

[0192] Referring again to Figure 15B, in box 1548, the controller can compare a counter value with a threshold (e.g., 1). If the counter value is greater than the threshold, the controller can report window occlusion with higher confidence (box 1545). If the counter value is less than the threshold, the controller can report window occlusion with lower confidence (box 1543). Although the threshold is shown as 1 in Figure 15B, it can be any other number. The controller can report window occlusion to the user and / or system (e.g., a transportation planning and perception system) and can activate one or more mechanisms to remove the occluding object, similar to those described above in conjunction with Figure 15A. The method 1500A in Figure 15B above is also referred to as a parallel process for detecting window occlusion.

[0193] Figure 15C illustrates another method 1500B, which uses only point cloud data to detect at least partial window occlusion and uses image data to determine the location, type, and extent of the occlusion. Method 1500B begins by acquiring a frame of data including point cloud data and image data (box 1551). The point cloud data represents a first return light signal on a first region (box 1552), and the image data represents a second return light signal on a second region (box 1554). The first and second regions may be the same region or may not be the same region and may overlap. In method 1550B, unlike method 1550A, the controller uses only the point cloud data to determine whether there is a first deviation between the point cloud data and a first expected value (box 1558). The controller does not use image data to determine whether there is a second deviation between the image data and a second expected value. The first deviation can represent: one or more variations in signal strength or average signal strength; variations in the distribution of signal strength; variations in the size and / or shape over time associated with one or both of the first and second regions of the aperture window; variations in sensitivity to a predetermined wavelength range; variations in the number of points in the point cloud data; and / or variations in the distance distribution represented by the point cloud data or image data. The controller can use only point cloud data for the determination in box 1558 under certain conditions, including, for example, if the point cloud data is of high quality and / or if environmental conditions do not cause the LiDAR sensor to produce many false detections. In this scenario, it may not be necessary or less necessary to use image data to enhance the confidence of the deviation determination results based solely on point cloud data. Therefore, the controller can choose to use only the point cloud data provided by the LiDAR sensor for the determination in box 1558.

[0194] In some examples, in box 1558, the controller further determines whether the first deviation exceeds a first threshold. If yes, the output of box 1558 is "Yes," and method 1500B proceeds to box 1566. In some examples, the controller only determines whether there is any deviation from the first expected value, and if so, the output of box 1558 is "Yes." If the determined output of box 1558 is "No," the controller can continue to obtain the next frame of data (box 1569).

[0195] As shown in Figure 15C, even if the controller does not use image data to determine the deviation from the expected value, it can use both point cloud data and image data to calculate (box 1566) the occlusion location, occlusion type, and occlusion degree. After performing such calculations, method 1500B can proceed to boxes 1567, 1568, 1565, and 1563. These boxes can be substantially the same as or similar to boxes 1547, 1548, 1545, and 1543 of Figure 15B, respectively, and therefore will not be described again. In some examples, the controller can calculate (box 1566) the occlusion location, the type of occluded object, and the extent of occlusion, either without determining the deviation (box 1558) or in parallel with determining the deviation. As shown in Figure 15C, if there is no deviation from the expected value and / or the deviation from the expected value is less than the expected threshold (box 1558), and / or after reporting occlusion (box 1563 or 1565), the controller can proceed to obtain the next frame (box 1569) and repeat method 1500B. The above method 1500B is also known as the LiDAR-only procedure for detecting window occlusion.

[0196] Figure 15D illustrates another method 1500C for detecting at least partial window occlusion using fused point cloud data and image data. Method 1500C begins by acquiring a frame of data comprising point cloud data and image data (box 1571). The point cloud data represents a first returned light signal on a first region (box 1572), and the image data represents a second returned light signal on a second region (box 1574). The first and second regions may be the same region or may not be the same region and may overlap with each other. As described above, in the multi-mode sensor of the described HyDAR system, when the point cloud data is provided by a LiDAR sensor and the image data is provided by an image sensor, they are already temporally and spatially synchronized at the hardware level. Therefore, the point cloud data and image data can be easily fused together by a data merger (box 1575) to obtain fused data. The fusion process may not require time offset and / or coordinate transformation between the point cloud data and the image data, or may require significantly less computational effort. In one example, the outputs from both the LiDAR sensor and the image sensor are provided to a fusion processor or data merger 1575 (e.g., part of the controller, a dedicated processor, or any other computer) for further processing. Based on the fused data, the controller can determine (box 1578) whether there is a deviation from the expected value. The expected value can also be set based on the fused expected data (e.g., fusing a first expected value based on point cloud data and a second expected value based on image data). Because the fused data combines both point cloud data and image data, the confidence of occlusion detection can be improved. For example, if the controller uses only point cloud data from the LiDAR sensor to detect occlusion, the confidence of occlusion detection may only be 50%. If the controller uses fused data (which includes image data), the confidence of occlusion detection (or no occlusion detected) can be increased to 90%. After occlusion detection, method 1500C can proceed to box 1586 to calculate the occlusion location, the type of occluding object, and the degree of occlusion. Method 1500C can then proceed to frames 1587, 1588, 1585, and 1583. These frames can be substantially the same as or similar to frames 1547, 1548, 1545, and 1543 of Figure 15B, and therefore will not be described again. In some examples, the controller can calculate (frame 1586) the occlusion location, the type of occluding object, and the degree of occlusion even with an uncertain deviation (frame 1578). As shown in Figure 15D, if there is no deviation from the expected value and / or the deviation from the expected value is less than the expected threshold (frame 1578), and / or after reporting occlusion (frames 1583 or 1585), the controller can proceed to obtain the next frame (frame 1589) and repeat method 1500C. The aforementioned method 1500C is also referred to as the fusion data flow for detecting window occlusion.

[0197] As mentioned above, obstruction of the aperture window of a HyDAR system is a degradation factor that can affect the performance of the HyDAR system. Another degradation factor is interfering light signals from one or more external interfering light sources. Figure 16 is a diagram illustrating various interference signals provided by one or more external interfering light sources according to various embodiments of a HyDAR system 1600. As shown in Figure 16, these external interfering light sources may include the sun 1606, the moon (not shown), the headlights of a vehicle 1610, streetlights (not shown), and another LiDAR or HyDAR system 1612. In other examples, light from the aforementioned interfering light sources may be reflected, transmitted, and / or refracted by other objects (e.g., object 1608, building 1604, etc.). This reflected, transmitted, and / or refracted light may be received by the HyDAR system 1600 as interfering light signals via the aperture window 1602. The examples shown in Figure 16 are not exclusive, and the interfering light sources are not limited to the examples shown in Figure 16. For example, another interfering light source may include a malicious laser jammer.

[0198] LiDAR sensors in HyDAR systems can be sensitive to external interfering light sources. Interfering light signals can introduce noise into the detection results of the HyDAR system 1600, and are generally undesirable. In some scenarios, such interfering light signals can cause malfunctions in the HyDAR system and may harm the LiDAR sensor and / or the user of the HyDAR system. Therefore, it is necessary to detect, reduce, avoid, and / or eliminate interfering light signals. The method described in this paper uses both point cloud data obtained from a LiDAR sensor and / or image data obtained from an image sensor to detect interfering light signals. In some examples, an image sensor integrated into the HyDAR system can improve the system's ability to detect and avoid interfering light signals.

[0199] Figures 17A and 17B are flowcharts illustrating various methods for detecting interfering optical signals caused by one or more interfering light sources according to various embodiments. Figure 17A illustrates an example of method 1700 for detecting interfering signals provided by one or more interfering light sources outside a HyDAR system (corresponding to block 1340 in Figure 13). In method 1700, a controller may obtain (block 1702) a first optical profile representing the characteristics of a first returned optical signal. As described above, the first returned optical signal is formed based on a transmitted light signal emitted by a laser light source. The first returned optical signal is detected by a LiDAR sensor (e.g., sensor 1202). The characteristics of the first returned optical signal may include, for example, the signal strength, wavelength, signal distribution, etc., of the first returned optical signal. The characteristics of the first returned optical signal are characteristics associated with the returned optical signal received by the LiDAR sensor. The first optical profile may be formed or extracted from point cloud data, may be formed or extracted from an electrical signal converted from the first returned optical signal, may be formed or extracted directly from the first returned optical signal, and / or may be derived from the aforementioned signal. For example, the first optical profile may include signal strength data in the form of a digital signal (sampled from an electrical signal converted from the first returned optical signal). The first light profile may also include a signal distribution, which may be a histogram derived from signal intensity data.

[0200] Similarly, the controller can obtain (box 1704) a second optical profile representing the characteristics of the second returned optical signal. As described above, the second returned optical signal is formed based on one or more light sources outside the HyDAR system. For example, the second returned optical signal can be formed by light reflected, scattered, or refracted by one or more objects in the FOV and detected by an image sensor (e.g., sensor 1204). The characteristics of the second returned optical signal can include, for example, the signal strength, wavelength, signal distribution, etc. The characteristics of the second returned optical signal are those associated with the returned optical signal received by the image sensor (e.g., a camera). The second optical profile can be formed or extracted from image data, from an electrical signal converted from the second returned optical signal, directly from the second returned optical signal, and / or derived from the aforementioned signals. For example, the second optical profile can include signal strength data in the form of a digital signal (sampled from the electrical signal converted from the second returned optical signal). The second optical profile can also include a signal distribution, which can be a histogram derived from the signal strength data.

[0201] Referring to Figure 17A, in some embodiments, the controller can determine (block 1706) whether at least one of the first or second optical profiles matches an interfering optical profile associated with an interfering signal provided by one or more interfering light sources. If a match exists, it likely means that the first and / or second returned optical signals include interfering optical signals that may affect the LiDAR sensor and / or image sensor. This could, in turn, degrade the performance of a Hybrid Light Detection and Ranging (HyDAR) system. Interfering optical profiles can be predetermined for one or more known interfering light sources. They can be formed or extracted from point cloud data or image data obtained based on the detection of known interfering light sources, from electrical signals converted from predetermined interfering optical signals of known interfering light sources, directly from predetermined interfering optical signals, and / or derived from the aforementioned signals. For example, the interfering optical profile may include signal strength data in digital signal form (sampled from electrical signals converted from predetermined interfering optical signals). The interfering optical profile may also include a signal distribution, which may be a histogram derived from the signal strength data. Several examples of interfering optical profiles are described below.

[0202] In some embodiments, to match the first / second optical profile with interfering optical profiles, the controller compares data represented by these profiles. Such data may include the measurement direction along which the HyDAR system detects the first and / or second returned optical signals; the absolute intensity detected along that measurement direction; the relative intensity detected along that measurement direction relative to different spectral ranges; the detectable size or shape of the interfering signal associated with at least one of one or more interfering light sources; and the distance distribution over multiple data collection periods.

[0203] As shown in Figure 17C, in one example, at any given time of day, a HyDAR system installed on vehicle 1751 can determine the direction of the sun (or moon). These directions of the sun (or moon) can be data included in the interfering light profile. Therefore, when the controller obtains a first light profile (based on a first returned light signal obtained by a LiDAR sensor) and / or a second light profile (based on a second returned light signal obtained by an image sensor), the controller can obtain the measurement direction along which the HyDAR system detected the first light profile and / or the second returned light signal. The controller can then compare the extracted measurement direction with the direction of the sun (or moon) included in the interfering light profile. If a match is found, it likely means that the first returned light signal and / or the second returned light signal obtained in that particular direction corresponds to an interfering light signal from the sun (or moon). In some examples, the controller can obtain more information to increase the confidence of this determination. For example, in the direction of the first returned light signal and / or the second returned light signal, in addition to the sun (or moon), there may be streetlights or some other light source (e.g., light reflected from the mirrors of tall buildings). In this scenario, the controller can compare data between the first / second light profile and the interfering light profile over several data collection cycles. This comparison can include the wavelengths contained in the first / second and interfering light profiles, and / or the absolute light intensity of the first / second returned light signal detected along the measurement direction can be compared with a predetermined interfering light signal. For example, if the interfering light source is a streetlamp, its light intensity may be much lower than sunlight but much higher than moonlight. The wavelength of the streetlamp may also differ from that of sunlight or moonlight. By comparing the various types of data included in the first / second and interfering light profiles, the controller can more confidently determine whether the first / second returned light signal contains an interfering light signal.

[0204] As another example, based on a first light profile (corresponding to a first returned light signal obtained by the LiDAR sensor) and / or a second light profile (corresponding to a second returned light signal obtained by the image sensor), the controller can extract the relative intensities of different spectral ranges detected along the measurement direction for the first light profile and / or the second returned light signal. As mentioned above, there may be reflected / scattered light signals from the object to be detected in the measurement direction. Other interfering light signals, such as streetlights, sunlight, moonlight, building reflections, etc., may also exist in the same direction. Therefore, one way to distinguish between the desired returned light signal and the unwanted interfering light signal is to analyze the relative intensities of the spectral ranges of the light signals. For example, the desired returned light signal may have the same wavelength (e.g., 905 nm) as the transmitted light signal from the laser source of the HyDAR system. Therefore, based on the first / second light profile, the controller of the HyDAR system can analyze the signal intensity of a specific returned light signal at a specific wavelength (e.g., 905 nm) and other wavelengths. If the signal strength of a specific optical signal at a specific wavelength falls within a predetermined range relative to the strengths of other returned optical signals at other wavelengths, the controller can determine that the specific returned optical signal is the desired returned optical signal. Otherwise, the controller can determine that they are interfering optical signals. In other examples, the controller can compare the relative signal strength of a specific returned optical signal with a predetermined relative signal strength of a known interfering optical signal. If a match is found, the controller determines that the specific returned optical signal is an interfering optical signal.

[0205] As another example, objects within the FOV of a HyDAR system may reflect visible light from external light sources. The reflected visible light may have a normal signal intensity range, depending on the object's reflectivity. Most objects within the FOV of a HyDAR system may not have high reflectivity like a mirror. In some scenarios, there may be buildings with mirror-like surfaces, and therefore the signal intensity of the reflected light may be very high in the visible light wavelength range. Such very high intensity can saturate the HyDAR system's image sensor, and therefore, such reflected light signals are undesirable interference signals, detrimental to generating good image data. The controller can determine that a particular returned light signal has a signal intensity above a threshold intensity in the visible light range and identify that particular returned light signal as interference. In other examples, the controller can compare the relative signal intensity of a particular returned light signal with a predetermined relative signal intensity of a known interference signal. If a match is found, the controller determines that the particular returned light signal is interference.

[0206] In some examples, based on a first light profile (corresponding to a first returned light signal obtained by a LiDAR sensor) and / or a second light profile (corresponding to a second returned light signal obtained by an image sensor), the controller can detect the size and shape associated with a specific returned light signal. The controller can then compare the detectable size or shape of the specific returned light signal with the size and shape of interfering signals associated with the same or more known interfering light sources. As an example, the image sensor of a HyDAR system can receive a specific returned light signal from the sun. This specific returned light signal, along with other returned light signals, can form a second returned light signal detected by the image sensor. The controller can obtain a second light profile representing the characteristics of the second returned light signal. Based on these characteristics (e.g., wavelength, signal strength, distribution, etc., as described above), the controller can detect the size and shape of possible interfering light sources. For example, if a specific portion of the second returned light signal has high signal strength, a specific wavelength spectrum, and is located in a certain direction in the sky, the controller can determine the size and shape of the light source that generates that specific portion of the second returned light signal. The controller can then compare the detected size and / or shape of the light source with the predetermined size and shape of known interfering light sources. If a match is found, the controller determines that the specific portion of the second returned light signal corresponds to an interfering light signal from a known interfering light source (e.g., the sun).

[0207] In some examples, based on a first light profile (corresponding to the first returned light signal obtained by the LiDAR sensor) and / or a second light profile (corresponding to the second returned light signal obtained by the image sensor), the controller can obtain a distance distribution over multiple data collection cycles. This distance distribution over multiple data collection cycles can be used in many scenarios where interference sources are not perfectly synchronized with the laser emission of the HyDAR system. An example of such an interference source could be sunlight. Sunlight entering the HyDAR system consists of randomly distributed pulses over time. These pulses can cause detection at almost any distance along the sun's direction. Therefore, the distance distribution of sunlight can cover a very large area. Thus, the HyDAR system controller can use this distance distribution to detect interference sources, including the sun and other LiDAR / HyDAR devices.

[0208] The exemplary interfering light sources described above include one or more of the following: the sun, vehicle headlights / taillights, street lighting, and light redistribution mechanisms that redirect light from another interfering source (e.g., a building with a mirror-like surface or other highly reflective object). In some examples, the external interfering light source may be another HyDAR or LiDAR system emitting laser light. Figure 17D illustrates two vehicles 1772 and 1776 traveling in opposite directions on a road. Vehicle 1772 is equipped with a HyDAR or LiDAR system that continuously emits laser signals to scan the environment. The transmitted laser signal from vehicle 1772 may be received by a HyDAR or LiDAR system installed on vehicle 1776 traveling in the opposite direction. Therefore, the laser signal from vehicle 1772 may be an interfering light signal for the HyDAR or LiDAR system installed on vehicle 1776, and vice versa. Such interfering light signals from vehicle 1772 can be detected by a HyDAR or LiDAR system installed on vehicle 1776 based on factors such as the measurement direction of the detected light signal (e.g., the direction of the light signal always comes from the opposite lane), the distance distribution over multiple data collection periods (e.g., the distance may be very large because the light signal emitted from vehicle 1772 does not correspond to any transmitted light signal from vehicle 1776), the wavelength (e.g., the two vehicles may have laser sources with different wavelengths), or other data contained in the first light profile / second light profile (corresponding to the first returned light signal detected by the LiDAR sensor and the second returned light signal detected by the image sensor).

[0209] It should be understood that the data included in the first and / or second light profiles are not limited to the examples described above. Other types of data can be extracted or derived from the first / second light profile and compared with interfering light profiles from known interfering light sources. By comparing light profiles generated based on the reflected light detected by one or both of a LiDAR sensor and an image sensor, the HyDAR system can improve its performance in detecting interfering light signals compared to using only one sensor.

[0210] Referring back to Figure 17A, in box 1706, if the controller determines that at least one of the first or second optical profiles matches the interfering optical profile (i.e., "yes"), the controller may then (box 1708) adjust at least one of the laser power, noise filter, or one or more steering mechanisms to reduce or prevent false detections by the HyDAR system. Figure 17A provides several non-limiting examples of adjustments, including: adjusting (box 1710) the steering mechanism to tune at least one of the starting or ending positions of the HyDAR system's FOV to avoid the locations of one or more interfering light sources; adjusting (box 1712) the steering mechanism to tune the duration of the data collection cycle to avoid the locations of one or more interfering light sources; adjusting (box 1714) the laser power so that the ratio of the first returned optical signal to the interfering optical signal is not less than a signal-to-noise ratio (SNR) threshold; turning off (box 1716) the laser power to avoid directing the laser signal to the locations of one or more interfering light sources; adjusting (box 1718) the controller to enhance the noise filter to remove data representing at least a portion of the interfering signal provided by one or more interfering light sources; and adjusting (box 1719) the controller to discard or ignore data representing at least a portion of the interfering optical signal provided by one or more interfering light sources. Each of boxes 1710-1719 will be described in more detail below.

[0211] As a first example of adjustment, based on the detection of interfering light signals, the controller adjusts (box 1710) the steering mechanism to tune at least one of the starting or ending positions of the HyDAR system's FOV, thereby avoiding the location of one or more interfering light sources. Figure 17C illustrates such an example, illustrating an exemplary process for adjusting the steering mechanism of a HyDAR system to avoid the location of interfering light sources such as the sun, according to various embodiments. Referring to Figures 17C and 12A, the HyDAR system 1750 can be mounted on a vehicle 1751 (e.g., mounted on the top of the vehicle). The HyDAR system 1750 can be substantially the same as or similar to the HyDAR system 1200 described above. The HyDAR system 1750 can scan the FOV 1752 using a steering mechanism (e.g., steering mechanism 1206) in its normal settings. The normal settings can be, for example, the default settings under normal vehicle operating conditions. As shown in Figure 17C, if vehicle 1751 operates during a period in the evening, HyDAR system 1750 may receive sunlight from sun 1754 at a low solar angle (i.e., the sun is near the horizon, making the angle between the direction of sunlight and the road surface potentially very small). In this scenario, the sun appears within the HyDAR system's field of view (FOV), and sunlight enters the detector directly. Therefore, more sunlight may directly enter HyDAR system 1750 (compared to the indirect case where most of the visible light received by HyDAR system 1750 is only reflected / scattered sunlight). Consequently, the sunlight received by HyDAR system 1750 may have very high intensity. Therefore, sunlight from sun 1754 at this time of day may affect (e.g., saturate) the image sensor and / or LiDAR sensor of HyDAR system 1750, and it is therefore desirable to avoid sunlight as interfering light signals.

[0212] In one embodiment, as shown in Figure 17C, to avoid interfering light sources such as the sun 1754 during a specific time period (e.g., near sunset), the controller of the HyDAR system 1750 can adjust at least one of one or more steering mechanisms to tune at least one of the start or end positions of the HyDAR system 1750's field of view, thereby avoiding the position of interfering light sources (such as the sun 1754). Thus, Figure 17C illustrates that after tuning the start and end positions of the FOV, the HyDAR system 1750 scans a new FOV 1756, which differs from the FOV 1752 in the normal setup. The new FOV 1756 does not overlap with the direction of the sun 1754, and therefore, the LiDAR sensor and / or image sensor of the HyDAR system 1750 can avoid receiving direct sunlight or reduce the amount of direct sunlight received. Accordingly, tuning the FOV to avoid interfering light sources can significantly reduce the noise or interfering light signals received by the HyDAR system 1750, and thus reduce or prevent false detections by the HyDAR system.

[0213] In some examples, the interfering source of the current HyDAR / LiDAR system may originate from another HyDAR or LiDAR system emitting laser signals, which can be referred to as the interfering HyDAR / LiDAR system (e.g., the HyDAR system installed on vehicle 1776 interferes with the HyDAR system installed on vehicle 1772 shown in Figure 17D). The interfering HyDAR / LiDAR system can emit laser signals, based on which a return light signal is formed. Such a return light signal may be an interfering light signal relative to the current HyDAR / LiDAR system. In some examples, if the current HyDAR / LiDAR system and the interfering HyDAR / LiDAR system use different light signal wavelengths, the interfering light signal from the interfering HyDAR / LiDAR system can be distinguished from the return light signal of the current HyDAR / LiDAR system by wavelength. However, in some examples, the interfering HyDAR / LiDAR systems may have the same light signal wavelength. Therefore, the current HyDAR / LiDAR system may need to adjust the steering mechanism (e.g., mechanism 1206) to tune the duration of the data collection period. As mentioned above, the data collection period corresponds to the time it takes to collect one frame of data (e.g., point cloud data). One frame of data can be acquired when the steering mechanism scans the entire FOV once. Therefore, by adjusting the steering mechanism, the data collection period can be adjusted. If the data collection period of the current HyDAR / LiDAR system differs from that of the interfering HyDAR / LiDAR system, the current HyDAR system can avoid the location of the interfering source or reduce the likelihood of receiving interfering signals at the current HyDAR / LiDAR system. In one example, adjusting the data collection period changes the frame size (e.g., frame length). Therefore, the data collection rate and / or return light direction between the current HyDAR / LiDAR system and the interfering HyDAR / LiDAR system may differ (e.g., typically, the two systems may not emit light pulses from the same angle or detect return light pulses at the same rate). Therefore, by adjusting the data collection period, the potential synchronization between the two HyDAR / LiDAR systems is reduced. Therefore, current HyDAR / LiDAR systems can reduce the likelihood of receiving interference signals generated by interfering HyDAR / LiDAR systems (e.g., system asynchrony).

[0214] Another method to reduce or eliminate false detections caused by interfering optical pulses in a current HyDAR or LiDAR system is to adjust the laser power of the light source (light source 310) in the HyDAR system (e.g., system 1200) or LiDAR system (e.g., system 300). As shown in Figure 17E, adjusting the FOV of the current HyDAR / LiDAR system as described above can be combined with adjusting the laser power. Figure 17E is a diagram illustrating exemplary scenarios of adjusting the current HyDAR / LiDAR system to avoid interfering optical signals and / or adjusting the laser power of the HyDAR system to reduce the impact of interfering optical signals according to various embodiments. Figure 17E uses a HyDAR system as an example, but the process can also be applied to a LiDAR system. Figure 17E has four schematic diagrams, named Schematic Diagrams 1 to 4. The processes shown in Schematic Diagrams 1 to 4 illustrate an exemplary sequence in which FOV adjustment and laser power adjustment can be applied. It should be understood that this sequence can be changed.

[0215] The first schematic diagram in Figure 17E (i.e., schematic #1) illustrates a scenario in which multiple HyDAR systems (e.g., systems 1780A, 1780B, and 1780C) may exist. Each of the HyDAR systems 1780A-1780C emits laser signals to scan its respective field of view (FOV). For example, HyDAR systems 1780A, 1780B, and 1780C scan FOVs 1781A, 1781B, and 1781C, respectively. In the scenario shown in the first schematic diagram of Figure 17E, FOVs 1781B and 1781C at least partially overlap with FOV 1781A. Therefore, HyDAR systems 1780B and 1780C are interfering light sources emitting interfering light signals relative to the current HyDAR 1780A. The interfering light signals emitted by HyDAR systems 1780B and 1780C can be detected using the methods described above (e.g., comparing the light profile of the returned light signal received by the LiDAR sensor and / or image sensor with the interfering light profile of a known interfering light source). In another example, HyDAR system 1780A can detect the interfering light signals emitted by interfering HyDAR systems 1780B and 1780C using image data provided by an image sensor (e.g., the image sensor of HyDAR system 1780A can capture images of HyDAR systems 1780B and 1780C (and / or the vehicles on which they are installed), and the controller can compare these images with known HyDAR / LiDAR systems to determine whether they are interfering HyDAR systems).

[0216] The second schematic diagram in Figure 17E (i.e., schematic #2) illustrates how the controller of the current HyDAR system 1780A can adjust the laser power of the laser source to reduce the impact of interfering optical signals generated by interfering HyDAR systems 1780B and 1780C. For example, the controller of HyDAR system 1780A can significantly increase the laser power of its laser source, thereby increasing the signal strength of the first returned optical signal. Consequently, when the LiDAR sensor of HyDAR system 1780A receives the first returned optical signal, the signal-to-noise ratio (SNR) increases. This is because the interfering optical signal remains constant. Therefore, if the signal strength (or signal power) of the first returned optical signal increases relative to the interfering optical signals emitted by interfering HyDAR systems 1784B and 1784C, the ratio of the signal strength of the first returned optical signal to the signal strength of the interfering optical signal increases. This ratio is referred to as the signal-to-noise ratio (SNR) associated with HyDAR system 1780A. The controller of the HyDAR system 1780A can increase the laser power to a level that ensures the signal-to-noise ratio (SNR) associated with the HyDAR system 1780A is not lower than the SNR threshold. As the laser power of the light source increases, the detection range of the HyDAR system 1780A also increases. Therefore, comparing the first and second schematic diagrams in Figure 17E, the field of view (FOV) of the HyDAR system 1780A changes from FOV 1781A to 1781A'. FOV 1781A' is greater than FOV 1781A, and the transmitted laser signal from the HyDAR system 1780A can propagate over a greater distance. In terms of detection, with the increase in SNR, performance degradation of the LiDAR sensor and / or image sensor of the HyDAR system 1780A due to interference from the light signals emitted by the HyDAR systems 1780B and 1780C can be reduced or prevented. Furthermore, this can reduce or prevent false detections by the HyDAR system 1780A.

[0217] The third schematic diagram in Figure 17E (i.e., schematic #3) illustrates that, in addition to adjusting the laser power, the controller of the current HyDAR system 1780A can also adjust the FOV via the steering mechanism to further reduce the impact of interfering optical signals generated by interfering HyDAR systems 1780B and 1780C. As described above, when the current HyDAR system 1780A detects interfering optical signals from interfering HyDAR systems 1780B and 1780C, the controller of the HyDAR system 1780A increases the laser power to increase the SNR of the returned optical signal relative to the interfering optical signal, thereby improving the detection effect. Sometimes, simply increasing the laser power may not be sufficient to improve the SNR to the desired level. In addition, excessively increasing the laser power may also pose safety issues. Therefore, in some examples, the controller of the current HyDAR system 1780A can adjust the FOV via the steering mechanism to avoid at least some of the interfering light sources. As described above, after increasing the laser power, the FOV 1781A of the HyDAR system 1780A becomes FOV 1781A'. Nevertheless, the field of view (FOV) 1781A' of HyDAR system 1780A may still significantly overlap with the FOV 1781C of interfering HyDAR system 1780C, and partially overlap with the FOV 1781B of interfering HyDAR system 1780B. Thus, the LiDAR sensor and / or image sensor of HyDAR system 1780A will still receive a significant amount of interfering optical signals from HyDAR system 1780C and some interfering optical signals from HyDAR system 1780B.

[0218] To further reduce the interfering optical signals received by the HyDAR system 1780A, the controller of the system 1780A can control a steering mechanism (e.g., steering mechanism 1206) to rotate the FOV range, as described above. For example, as shown in the third schematic diagram of FIG17E, the steering mechanism can be adjusted so that the HyDAR system 1780A changes its scanning range to cover FOV 1781A''. Compared to FOV 1781A'', FOV 1781A'' only partially overlaps with FOV 1781C from the interfering HyDAR system 1780C, and does not overlap with FOV 1781B from the interfering HyDAR system 1780B. Therefore, the LiDAR sensor and / or image sensor of the current HyDAR system 1780A receives less interfering optical signals from the interfering HyDAR system 1780C, and receives zero or very little interfering optical signals from the interfering HyDAR system 1780B. Furthermore, due to the reduced signal strength of the interfering optical signal, the SNR of the returned optical signal detected at the HyDAR system 1780A is further improved. This further improved SNR can better prevent false detections by the HyDAR system 1780A.

[0219] In some scenarios, interfering HyDAR systems 1780B and / or 1780C may be very close to the current HyDAR system 1780A. Therefore, the interfering optical signal may have very high intensity. Increasing the laser power (schematic diagram #2 in Figure 17E) may not be sufficient to achieve a sufficiently high SNR. Furthermore, due to the very close proximity of the interfering HyDAR systems 1780B and / or 1780C, adjusting the steering mechanism may not prevent significant overlap of the field of view (FOV) between the current HyDAR system 1780A and the interfering HyDAR systems 1780B and / or 1780C. In some other scenarios, the interfering optical signal may be generated by the HyDAR system 1780A itself (e.g., if the aperture window of system 1780A is obstructed, or if some internal components of system 1780A have high signal reflection, or if the internal calibration of system 1780A is degraded). In these scenarios, the controller of the HyDAR system 1780A can reduce the laser power, or even shut it off as shown in the fourth schematic diagram (i.e., schematic #4) of Figure 17E. Shutting off the laser power can prevent the laser signal from being directed to the location of interfering light sources. In some cases, the controller can report its adjustments to the user or another system (e.g., a vehicle planning and perception system). Based on this report, the user or another system can determine that no decisions should be made (e.g., vehicle planning or perception decisions) when the HyDAR system 1780A shuts off the laser power.

[0220] Referring back to Figure 17D, this figure illustrates another exemplary process for adjustment based on detected interfering light signals. Figure 17D shows an adjustment controller according to various embodiments to discard or ignore data representing at least a portion of the interfering signals provided by one or more interfering light sources. Referring to Figure 17D, in one example, a HyDAR system installed on vehicle 1776 may receive an interfering light signal 1775 from vehicle 1772. The interfering light signal 1775 may be the high beam of the headlight of vehicle 1772, or it may be a transmitted laser signal from a LiDAR or HyDAR system installed on vehicle 1772. Vehicle 1772 may transmit other light signals 1774 (beams from another headlight), which may not be received by the HyDAR system installed on vehicle 1776 and are therefore not interfering light signals.

[0221] As shown in Figure 17D, in some embodiments, the controller of the current HyDAR system (e.g., a HyDAR system installed on vehicle 1776) can be adjusted to discard or ignore data representing at least a portion of the interfering light signal 1775 provided by the interfering light source (e.g., the high beam from vehicle 1772). Figure 17D illustrates scan line 1778 generated by the controller of the current HyDAR system, and the data in scan line 1778 corresponding to region 1779 may be identified as containing excessive noise due to the interfering light signal 1775. Therefore, the controller can discard or ignore such data corresponding to region 1779. Furthermore, since such data is not used for object detection, performance degradation of the current HyDAR system can be reduced or prevented.

[0222] Although not explicitly shown in Figure 17D, in some embodiments, the controller of the current HyDAR system can apply or enhance a noise filter to remove data representing at least a portion of the interfering optical signal provided by the interfering light source. For example, a noise filter can be applied to discard all data points along a specific direction corresponding to the location of the interfering light source; discard all data points whose signal strength meets a specific threshold (high or low threshold); and / or discard data points based on the signal distribution and spacing between data points. The noise filter can be implemented in hardware and / or software. A noise filter can be applied before data is completely discarded or ignored. In some examples, a noise filter with a first-level filter can be applied to all data points in the point cloud data and / or image data obtained by the current HyDAR system; while an enhanced noise filter with a second-level filter can be applied to a portion of the data points corresponding to the interfering optical signal from a specific interfering light source. For example, referring to Figure 17D, a default level of noise filtering can be applied to all data points corresponding to scan line 1778, while an enhanced level of noise filtering can be applied to the data points corresponding to region 1779.

[0223] Figures 17A and 17C to 17E illustrate various methods for detecting interfering optical signals as a performance degradation factor of the HyDAR system and for adjusting these methods to reduce or prevent false detections. Figure 17B shows an example implementation of the above-described methods for detecting interfering optical signals and adjusting them to reduce or prevent false detections. Specifically, in an implementation of method 1730, the controller (e.g., controller 1214 or control circuitry 350) first (block 1731) acquires a frame of data (e.g., a frame of point cloud data generated by a LiDAR sensor and / or a frame of image data generated by an image sensor). The controller can extract a first optical profile (block 1732) and a second optical profile (block 1734) from the frame of data. As described above, the first optical profile can represent the characteristics of a first returned optical signal (e.g., the intensity, wavelength, distribution, etc. of the returned signal detected by the LiDAR sensor); and the second optical profile can represent the characteristics of a second returned optical signal (e.g., the intensity, wavelength, distribution, etc. of the returned signal detected by the image sensor).

[0224] In block 1736 of process 1730, the controller has a contour comparator configured to compare a first optical contour and / or a second optical contour with one or more contours of a known interference signal (block 1735). The contours of the known interference signals are stored in the HyDAR system's storage device or elsewhere (e.g., cloud storage). The contour comparator can be implemented using hardware and / or software to achieve more efficient and faster comparisons. In some cases, the comparisons and other steps in method 1730 are performed in real time so that the results of method 1730 (e.g., determining the interfering optical signal and adjusting accordingly) can be provided quickly for real-time operation of the vehicle.

[0225] In block 1737, the controller determines whether the first optical profile / second optical profile matches the profile of a known interfering signal. If so, the controller can execute a mitigation plan to reduce or prevent false detections caused by the interfering optical signal. As described above, the controller can adjust one or more of the steering mechanism, noise filter, laser power, etc., to reduce or prevent false detections caused by the interfering optical signal. After adjustment, if no match is found in block 1737, the controller can continue to acquire the next frame of data (block 1738).

[0226] Window obstruction and interference with light signals are two factors that can degrade the performance of a HyDAR system. Another performance degradation factor is external calibration degradation. Figure 18 is a block diagram illustrating a mobile platform 1800 equipped with a HyDAR system 1802 and various sensors according to various embodiments. The mobile platform 1800 can be a vehicle, robot, etc. The HyDAR system 1802 can be substantially the same as or similar to the systems 400 or 1200 described above. Sensors 1804, 1806, and 1808 can include, for example, cameras, ultrasonic sensors, radar, etc., or combinations thereof. Other components mounted to the mobile platform 1800 may also be present but are not shown. When the HyDAR system 1802 is manufactured and mounted on the mobile platform 1800, the position and orientation of the HyDAR system are calibrated so that it can operate correctly to detect objects within a desired field of view (FOV) around the mobile platform 1800. For example, the roll, pitch, and yaw settings of the HyDAR system 1802 are set and calibrated to desired values ​​so that the system operates in the desired manner. This type of calibration is called external calibration of the HyDAR system 1802 because it is performed relative to a movable platform 1800 (or its components, other sensors mounted thereon) located outside the HyDAR system 1802.

[0227] After the mobile platform 1800 has been operating for a period of time, the position and orientation of the HyDAR system 1802 relative to the vehicle may change due to factors such as vibration, shock, humidity, temperature, or other environmental, operational, or user-related factors. As shown in Figure 18, the position and orientation (e.g., roll, pitch, and yaw) of the HyDAR system 1802 relative to its original position (and therefore also relative to the mobile platform 1800) may change over time. In some examples, the position and orientation of one or more components of the HyDAR system 1802 relative to the mobile platform 1800, sensors 1804, 1806, and / or 1808, and / or other components mounted on the mobile platform 1800 may change over time. Accordingly, it is necessary to detect and adjust for external calibration degradation to compensate for such degradation.

[0228] If the LiDAR system or camera is a discrete sensor mounted on a vehicle (compared to the integrated multi-mode sensor in a HyDAR system), external calibration of the discrete LiDAR system or camera can be difficult or cumbersome because it requires precise synchronization between sensors and specific calibration settings. By using the HyDAR system described herein, which includes a multi-mode sensor integrating a LiDAR sensor and an image sensor, external calibration or monitoring of external calibration can be performed without additional synchronization. As mentioned above, in the HyDAR system described herein, point cloud data provided by the LiDAR sensor and image data provided by the image sensor are synchronized temporally and spatially at the hardware level. Therefore, no additional computational effort is required for synchronization above the hardware level. Furthermore, if the external calibration degrades within a predetermined threshold, the HyDAR system can self-adjust during operation. Therefore, it can minimize the impact on vehicle operation (e.g., the vehicle can continue to operate while the HyDAR system continues to provide detection results).

[0229] Figures 19A and 19B are flowcharts illustrating methods for detecting external calibration degradation of a HyDAR system (e.g., system 1200 or 1802) mounted on a mobile platform (e.g., platform 1800) according to various embodiments. Figures 19C and 19G are diagrams illustrating exemplary methods for detecting external calibration degradation of a HyDAR system (e.g., system 1200 or 1802) using parallel line features extending along a road surface according to various embodiments. As described above, the HyDAR system includes a LiDAR sensor and an image sensor integrated together to form a multi-mode sensor. The flowchart of Figure 19A will first be described in conjunction with the examples shown in Figures 19C to 19H. Figure 19A illustrates method 1900, which corresponds to block 1350 in Figure 13. Referring to Figure 19A, in some embodiments, the controller initiates method 1900 by acquiring point cloud data generated by the LiDAR sensor and image data generated by the image sensor. In box 1902, point cloud data representing a first returned optical signal and image data representing a second returned optical signal are combined to obtain a combined dataset. The controller can perform the combination of point cloud data and image data. Since the point cloud data and image data are synchronized in time and space at the hardware level, no additional synchronization is required. Figure 19C illustrates an exemplary image 1930 representing the combined point cloud data and image data. Image 1930 may also represent only point cloud data or only image data.

[0230] In box 1904, the controller segments the space of interest based on the combined dataset. By comparing image 1930 in Figure 19C and image 1931 in Figure 19D, the controller extracts the space of interest 1933 and removes (e.g., filters out) other features of no interest. In this example, the space of interest 1933 corresponds to features of the road surface on which vehicles equipped with the HyDAR system travel. Other features in image 1930 are removed or filtered out. Such features may include, for example, other vehicles traveling on the road surface, the sky, trees, and other roadside objects. Extracting features from the combined dataset to segment the space of interest can be performed using, for example, machine learning-based algorithms and / or other pattern recognition algorithms. For example, the image data in the combined dataset may include color and shape information of the road surface (e.g., roughly triangular if the road is straight, as shown in Figure 19C; curved if the road is curved). The point cloud data in the combined dataset may include distance and height information of objects in image 1930. For example, in image 1930, any object with a height less than a threshold (e.g., 0.1m) can be considered part of the road surface. Based on the information of these types contained in the combined dataset, the controller can accurately identify the road surface and separate it from other features (e.g., trees, vehicles, sky, etc.) in this example. In some examples, the controller may also use other data to determine operating conditions before performing segmentation to obtain the space of interest. For example, based on GPS data and / or the combined dataset, the controller can determine that the vehicle is moving forward; there is no obvious turn; the vehicle speed is within a certain threshold, etc. When all these operating conditions are met, the controller segments the space of interest 1933 (e.g., the road surface) based on the combined dataset.

[0231] Next, referring back to Figure 19A, in box 1906, the controller probes parallel line features in space of interest 1933 based on the combined dataset. This is illustrated in Figure 19E, where parallel line feature 1932 is identified from space of interest 1933. Parallel line feature 1932 may correspond to lane markings on a road surface. Lane markings can be identified using image data from the combined dataset. For example, lane markings may have high signal strength (e.g., because they contain reflective materials) and therefore can be distinguished from other features of the road surface. Parallel line feature 1932 may also correspond to other objects, such as curbs, medians, buildings, or any other objects positioned along the road surface. In one example, a Hough straight line transform can be performed to extract parallel line feature 1932. For more accurate extraction, highway lane markings may be better because they are straight and their lane curvature is more stable or changes slowly.

[0232] Referring back to Figure 19A, in box 1908, the controller identifies the intersection location of the detected parallel line feature 1932. This is illustrated in Figure 19F. For example, the parallel line feature 1932 can be extrapolated to find the intersection location 1934 at the horizon 1937. The controller can identify the intersection location 1934 using its coordinates (x, y, z). Figure 19H is a diagram illustrating another example where the lane feature is curved (compared to the straight line in Figure 19F). In this case, the controller can identify a set of intersection locations 1952A-1952C based on the curved feature. This set of intersection locations 1952A-1952C can be identified using their coordinates (e.g., A(x1, y1, z1); B(x2, y2, z2); and C(x3, y3, z3)). For example, the controller can identify a set of intersection locations 1952A-1952C based on the curvature of the lane at different locations.

[0233] Referring back to Figure 19A, the process including boxes 1902, 1904, 1906, and 1908 can be repeated multiple times to obtain multiple intersection positions. This is illustrated in Figure 19G, where multiple intersection positions 1934A-1934N at the horizon 1937A-1937N are calculated at different time points. In one example, each combined dataset corresponds to one frame of point cloud data and image data. Each combined dataset can be processed by the controller to obtain the intersection positions. Therefore, multiple combined datasets 1935 can be used to obtain multiple intersection positions 1934A-1934N.

[0234] Referring back to Figure 19A, based on multiple intersection locations, the controller can estimate whether the relationship between the HyDAR system (e.g., system 1200 or 1802) and the mobile platform (e.g., platform 1800) on which the HyDAR system is installed has deviated from its original configuration. For example, the controller can compare the multiple intersection locations with one or more corresponding stored external locations of the HyDAR system. The stored external locations might be calibration results obtained when the HyDAR system was installed on the mobile platform at the factory, and therefore might be factory calibration locations (also known as ground truth). The stored external locations could also be calibration results obtained at any other previous point in time (e.g., during subsequent vehicle maintenance services).

[0235] In some examples, since the combined dataset includes both point cloud data generated by a LiDAR sensor and image data generated by an image sensor, the controller can calculate the first intersection location using the point cloud data and the second intersection location using the image data. Similarly, the controller can acquire multiple first intersection locations using multiple frames of point cloud data and multiple second intersection locations using multiple frames of image data. To detect internal calibration degradation, the controller can compare the multiple first intersection locations with the multiple second intersection locations. If factory calibration locations (ground truth) are unavailable, this comparison may need to be performed using data from different sensors of the HyDAR system (e.g., LiDAR and image sensors). For example, as described above in conjunction with Figure 19H, if the lane features are curved, the controller can calculate multiple intersection locations 1952A-1952C. Ground truth data may be unavailable because the curvature of the road may not be accurately estimated during the manufacture of the HyDAR system or its installation on a vehicle. In this case, the controller cannot compare the intersections with the ground truth data. The controller can compare multiple first intersection locations calculated using point cloud data with multiple second intersection locations calculated using image data. In some other examples, this comparison can be performed as an additional check, even if factory calibration locations (ground truth) are available. In other examples, the calculation of intersection locations can be compared with calculations based on high-definition maps. The calculation of intersection locations can also be improved by using IMU sensors to enhance accuracy. For example, HD maps and / or IMU sensors can provide additional information about the slope of the road surface, which can be used to improve calculation accuracy.

[0236] The comparison results can be used to determine whether the relationship between the HyDAR system (e.g., system 1200 or 1802) and the mobile platform on which the HyDAR system is installed (e.g., platform 1800) has deviated from its original configuration. If the comparison results are within a predetermined threshold range, the controller can determine that the HyDAR system has not shifted and the external calibration has not degraded (or the degradation has not exceeded the threshold). If the comparison results are greater than the predetermined threshold, the controller can determine that the external calibration has degraded beyond the threshold. In this case, the HyDAR system may need to be adjusted or recalibrated.

[0237] Figure 19B illustrates a specific exemplary method 1920 for detecting external calibration degradation of a HyDAR system mounted on a vehicle. Method 1920 may correspond to block 1350 shown in Figure 13. In block 1921, the controller of the HyDAR system can detect that the vehicle is traveling through a long straight section of road at a speed exceeding a threshold (e.g., 10 m / s). In block 1922, the controller of the HyDAR system can acquire a frame of HyDAR data including a combined dataset. The combined dataset has point cloud data generated by a LiDAR sensor and image data generated by an image sensor. In block 1923, based on the frame of HyDAR data including the combined dataset, the controller segments the data with combined 2D and 3D information. As described above, the point cloud data may include 3D information such as horizontal and vertical coordinates and distance information. The image data may include 2D information. The image data may have color information and higher resolution. As described above, the controller can use the combined data to segment the space of interest. Figure 9B further illustrates some examples of segmentation that the controller of the HyDAR system can perform. In one example, the controller can segment to obtain the space of interest based on a set of criteria, which include: (1) the vertical coordinates need to be 1.5m lower than the sensor; and (2) the elevation angle is less than the azimuth angle. 0.5; (3) The elevation angle is less than the azimuth angle (-0.5); and (4) the color is yellow or white. In another example, the controller can segment the space of interest based on a set of criteria, which include: (1) the 3D coordinates need to be within 0.1 meters of the plane Ax+By+Cz+d=0; (2) the elevation and azimuth angles need to satisfy the condition "K Elevation Angle + M Azimuth + n > 0”; and (3) the color is yellow or white. In the above equations, A, B, C, d, K, M and n are configurable constants.

[0238] In box 1924, the controller identifies parallel line features within the segmented space of interest. For example, line features could correspond to lane markings on a highway. In box 1925, the controller calculates the locations of intersections (also known as vanishing points). In box 1926, the controller accumulates the locations of intersections (also known as vanishing points) (e.g., by repeating boxes 1922-1925) to create a distribution. In box 1927, the controller determines whether the distribution of vanishing point locations matches expectations. This is achieved by comparing the distribution of vanishing point locations with a distribution obtained through factory calibration (ground truth). If the controller determines that the distribution matches expectations (e.g., its deviation from the ground truth is within a threshold range), it reports (box 1929) that the extrinsic parameters are consistent. That is, the extrinsic calibration has not degraded (or has degraded slightly but is still within tolerance). Otherwise, the controller reports (box 1928) that the extrinsic calibration has deviated from the expected value.

[0239] The methods 1900 and 1920 described above for detecting external calibration degradation can be performed using line features of the road surface during vehicle operation and are referred to as dynamic external calibration, or simply dynamic calibration. In some other embodiments, external calibration detection can be performed using one or more feature sets of a predefined stationary target. These feature sets are also referred to as key features or key points. Figure 20A is a flowchart illustrating another method 2000 according to various embodiments, which is used to provide information related to key features of a predefined stationary target for detecting external calibration degradation obtained from point cloud data and image data. Method 2000 may be part of box 1350 in Figure 13.

[0240] Referring to Figure 20A, in block 2002, the controller identifies one or more feature sets associated with one or more predefined stationary targets based on point cloud data representing a first returned light signal and image data representing a second returned light signal. Examples of predefined stationary targets are shown in Figure 20B. For example, stationary target 2012 may have an array of circles with a predefined pattern (e.g., as shown in Figure 20B, two black circles in the center surrounded by white circles). Another exemplary stationary target 2014 may have a checkerboard pattern. Another stationary target 2016 may have an alternating pattern of black and white blocks as shown in Figure 20B. It should be understood that predefined stationary targets are not limited to the examples shown in Figure 20B. Key features (or key points) identified in these predefined stationary targets may include certain specific points, scale-invariant features, and / or features unaffected by viewpoint, FOV size, etc. These features can be used to detect external calibration degradation with greater accuracy.

[0241] Referring again to Figure 20A, key features identified from a predefined stationary target may possess specific properties designed for accurate external calibration. These specific properties may be related to the intensity of the key feature or environmental conditions. When detecting external calibration degradation, the HyDAR system is activated to capture images of the predefined stationary target using both a LiDAR sensor and an image sensor. Therefore, key features can be extracted from both point cloud data and image data.

[0242] Figure 20A further illustrates that, in block 2004, the controller can provide one or more of the following information based on one or more identified feature sets. For example, the controller can provide: the position of one or more feature sets in three-dimensional (3D) space; the position of one or more feature sets in elevation and azimuth angles; point cloud data representing a first returned light signal at a region associated with one or more feature sets; image data representing a second returned light signal at one or more feature sets; gradients associated with the point cloud data representing the first returned light signal at one or more feature sets; gradients associated with the image data representing the second returned light signal at one or more feature sets; gradient histograms (HOGs) associated with the point cloud data representing the first returned light signal at one or more feature sets; and gradient histograms (HOGs) associated with the image data representing the second returned light signal at one or more feature sets. At least some of the above information is derived (e.g., calculated, transformed, determined, etc.) from point cloud data and image data captured by the HyDAR system for a predefined stationary target.

[0243] Figure 20C is a flowchart illustrating a method 2020 for detecting external calibration degradation of a HyDAR system using a feature set based on image data and LiDAR point cloud data, according to various embodiments. Method 2020 corresponds to block 1350 in Figure 13. Referring to Figure 20C, the controller obtains image data from the HyDAR system and optionally point cloud data. In block 2022, the controller identifies a first feature set associated with a predefined stationary target, based at least on the image data representing a second returned light signal. If the controller also obtains point cloud data as part of a combined dataset, the first feature set can also be identified based on both the image data and the point cloud data. The controller can provide various information (e.g., position, gradient, histogram, etc.) based on the first feature set, as described above in conjunction with Figure 20A.

[0244] In box 2024, the controller obtains a second set of features from a sensor external to the HyDAR system. This sensor may include a camera, an infrared camera, and one or more components of the LiDAR or HyDAR system. This external sensor is not part of the HyDAR system on which it is being externally calibrated. The external sensor may be mounted on the same mobile platform (e.g., a vehicle) as the HyDAR system. Both the first and second set of features can be obtained using the same predefined stationary target. The HyDAR system and the external sensor may be positioned to face the same predefined stationary target. Similarly, based on the second set of features, the controller can provide various information (e.g., position, gradient, histogram, etc.) as described above in conjunction with Figure 20A.

[0245] Next, in box 2026, the controller establishes the correspondence between the first feature set and the second feature set. For example, the controller can identify corresponding features / gradients / positions / histograms / data in the two feature sets. In box 2028, the controller calculates the affine transformation between the first and second feature sets. An affine transformation is a geometric transformation that preserves points, lines, and planes. It includes transformations such as translation, rotation, scaling (resize), and shearing (distortion). In an affine transformation, for example, any straight line before the transformation remains a straight line after the transformation; parallel lines remain parallel after the transformation; and the proportion of distances between points on a straight line remains unchanged after the transformation. Affine transformations can be represented using matrix multiplication and addition, where coordinates (x, y) are transformed into new coordinates (x', y') through a matrix and a translation vector. Therefore, an affine transformation can be used to transform the first feature set into the second feature set, and vice versa. Using the first and second feature sets, the controller can calculate the affine transformation between them.

[0246] Referring again to Figure 20C, in box 2029, the controller can estimate, based on the calculated transformation, whether the relationship between the HyDAR system and the mobile platform on which the HyDAR system is installed has deviated from its original configuration. For example, the affine transformation between the first and second feature sets can be compared with the factory-calibrated affine transformation (ground truth). If their difference exceeds a threshold, it means that the external calibration of the HyDAR system has degraded (changed relative to external sensors).

[0247] In another example, within boxes 2028-2029, the controller can calculate the relationship between two coordinate systems associated with two feature sets. After coordinate system transformation, the first feature set becomes a first-transformed feature set. If the first-transformed feature set matches the second feature set (e.g., within a threshold range), the position of the HyDAR system has not shifted. This, in turn, means that the external calibration of the HyDAR system has not degraded. In other examples, the two feature sets can also be transformed to a common coordinate system for comparison.

[0248] As mentioned above, in one example, detecting external calibration degradation based on a predefined stationary target can use only image data because it has a higher resolution than point cloud data. However, the controller can also use a combined dataset that includes both point cloud data and image data to identify key features. Also as mentioned above, Method 2020 requires the use of external sensors to determine whether the external calibration of the HyDAR system has degraded. Although a HyDAR system includes a LiDAR sensor and an image sensor, they are integrated together. Therefore, the internal image sensor or LiDAR sensor cannot be used to perform external calibration degradation detection because their positions and orientations may shift together. Therefore, external sensors are typically required. However, if the LiDAR sensor and image sensor in the same HyDAR system are mounted separately and their positions and orientations do not shift together, they can also be used for external calibration degradation detection.

[0249] Figure 20D is a flowchart illustrating a specific exemplary method 2030 for detecting external calibration degradation of a HyDAR system mounted on a vehicle. Method 2030 corresponds to block 1350 of Figure 13. As shown in Figure 20D, in block 2031, the HyDAR system (e.g., system 1200 or 1802) and a second sensor external to the HyDAR system are positioned facing a predefined stationary target with detectable key points (e.g., key features). In block 2032, the HyDAR system sends a laser emission, including a laser signal, toward the predefined stationary target. The HyDAR system detects a first returned light signal via its LiDAR sensor and a second returned light signal via its image sensor. In block 2033, the second sensor may also detect the returned light signal. The second sensor is located external to the HyDAR system and may be a LiDAR sensor, a camera, or any other sensor. The HyDAR system has an internal or external controller. The second sensor may also have an internal or external controller.

[0250] In block 2034, the controller of the HyDAR system (or another computer or control circuit) identifies a first set of keypoints based on the first and second returned light signals detected by the HyDAR system. In block 2035, for example, an internal or external controller of a second sensor outside the HyDAR system also identifies a second set of keypoints. In block 2036, the controller of the HyDAR system establishes a correspondence between the first and second sets of keypoints, similar to the above. In block 2037, the controller of the HyDAR system calculates a transformation (e.g., an affine transformation or any other transformation that converts one coordinate system to another). In block 2038, the controller can determine whether the transformation is as expected (e.g., by comparison with ground truth). If yes, the controller reports (block 2040) that the extrinsic parameters are consistent, meaning that no or minimal external calibration is required. If no, the controller reports (block 2039) that the external calibration has deviated from the expected value. As mentioned above, the controller can also compare the transformed first set of keypoints with the second set of keypoints, and vice versa. Alternatively, the controller can transform both the first and second keypoint sets to a common coordinate system and then compare the transformed sets. Based on this comparison, the controller can determine whether the HyDAR system suffers from external calibration degradation.

[0251] In addition to external calibration degradation, HyDAR systems (or LiDAR systems) may also have internal calibration degradation. Figure 21 is a block diagram illustrating a HyDAR system 1200 that may have internal calibration degradation according to various embodiments. When a HyDAR system is manufactured, the positional and orientational relationships between its internal components are carefully configured and set. HyDAR systems can be mounted on mobile platforms such as vehicles. Therefore, HyDAR systems may operate outdoors in all-weather conditions, in which case they may be exposed to vibration, wear, shock, and / or other external damage or influences. Therefore, the relationships between the internal components of a HyDAR system may change when it operates for extended periods. For example, as shown in Figure 21, some surfaces of one or more steering mechanisms 1206 may have different flatness; steering mechanism 1206 may be offset horizontally or vertically relative to its original position; its pitch, yaw, and roll relative to other components may also change; and / or the distance between the collection lens 1210 and the LiDAR sensor 1202 may also change, causing the first returned light signal to be poorly focused on the LiDAR sensor 1202. These types of changes in relationships between internal components of a HyDAR system are referred to as internal calibration degradation. It should be understood that internal calibration degradation is not limited to the example shown in Figure 21.

[0252] Figure 22A is a flowchart illustrating an exemplary method 2200 for detecting internal calibration degradation of a HyDAR system according to various embodiments. Method 2200 corresponds to block 1360 of Figure 13. As shown in Figure 22A, in block 2202 of method 2200, the controller obtains at least image data representing a second returned light signal under different internal configurations of the HyDAR system. Optionally, the controller may obtain point cloud data or a combined dataset including point cloud data and image data. As described above, the configuration of the HyDAR system described herein has an integrated multi-mode sensor including a LiDAR sensor and an image sensor. Therefore, the point cloud data and image data are synchronized in time and space at the hardware level. Method 2200 can be performed using only image data or a combined dataset, and the following description uses image data for illustration.

[0253] Image data may include data representing second returned light signals detected in two or more internal configurations of the HyDAR system. For example, referring to FIG21, image data may be generated based on second returned light signals received from two different surfaces of the steering mechanism 1206, or based on second returned light signals formed by laser signals emitted from two different surfaces. Specifically, in one configuration, surface 1206A is used to emit laser signals and receive second returned light signals; while in another configuration, surface 1206B is used to emit laser signals and receive second returned light signals. When the HyDAR system 1200 is manufactured, surfaces 1206A and 1206B may be identical at the factory; however, over time, one or both surfaces may become tilted / bent / deformed, etc., as shown in FIG21.

[0254] As another example, multiple different configurations could include using different levels of laser power (e.g., high or low), different laser signal repetition rates, different data collection cycles, and different LiDAR sensor and / or image sensor sensitivities. In other words, multiple internal configurations of a HyDAR system have different operating parameters or hardware configurations.

[0255] Referring back to Figure 22A, in block 2204, the controller identifies multiple feature sets from the same predefined target set for each of the different internal configurations. The predefined targets can be stationary targets as described above, or they can be moving targets. As long as the same target set is used in each of the multiple different internal configurations, the resulting feature sets can be used to detect internal calibration degradation.

[0256] In block 2206 of method 2200, the controller associates each of the multiple feature sets with the physical settings of a predefined target to calculate the relative position and orientation of the corresponding internal configurations in different internal configurations. For example, referring to Figure 21, suppose surfaces 1206A and 1206C are to measure the same portion of the FOV, and as shown in Figure 21, surface 1206A is slightly off-center by 0.5 degrees in the horizontal direction (but the controller is unaware of this). The controller detects the same feature in the FOV at azimuth = 0 degrees when using surface 1206C (configuration 1), and again at azimuth = 1 degree when using surface 1206A (configuration 2). By using the difference in detecting the same feature across different configurations, the controller can infer the internal offset between surfaces 1206A and 1206C.

[0257] In block 2208, the controller can verify whether the calculated relative positions and orientations for different internal configurations are within the acceptable tolerance range of the target value. For example, when the HyDAR system operates using a first internal configuration with a first relative position and orientation, a first feature set can be obtained; and when the HyDAR system operates using a second internal configuration with a second relative position and orientation, a second feature set can be obtained. The first and second relative positions, along with the orientation, can be used to calculate transformations (e.g., affine transformations). Alternatively, the first relative position and orientation can be transformed to a coordinate system under the second internal configuration, and vice versa.

[0258] Similar to the description of detecting external calibration degradation above, internal calibration degradation can be detected by comparing the transformations of relative positions and orientations under two different internal configurations with the ground truth (or simply put, they should have no difference or only minimal difference). Internal calibration degradation can also be detected by transforming the relative positions and orientations from the coordinate system of the first configuration to the coordinate system of the second configuration (and vice versa). In other words, internal calibration degradation can be detected by comparing the relative positions and orientations obtained under two different internal configurations after appropriate coordinate transformations. Essentially, it uses the relative position / orientation under one configuration as a benchmark to determine whether the change in the relative position / orientation under another configuration relative to the target value exceeds an acceptable tolerance.

[0259] Figure 22B illustrates a flowchart of an exemplary method 2230 for detecting internal calibration degradation using two configurations of a steering mechanism (e.g., mechanism 1206 in Figure 21). In block 2231 of method 2230, the HyDAR system is positioned facing a predefined target with detectable key points. The predefined target may be those shown in Figure 20B. In block 2232, the HyDAR system operates to emit a laser signal using the steering mechanism in the first configuration and receive a first return light signal and a second return light signal (e.g., receiving the return light signal using the first surface 1206A). In block 2233, the HyDAR system operates to emit a laser signal using the steering mechanism in the second configuration and receive the first return light signal and the second return light signal (e.g., receiving the return light signal using the second surface 1206B).

[0260] In block 2234, a first set of key points is obtained based on the first and second returned optical signals received by the HyDAR system under the first configuration. In block 2235, a second set of key points is obtained based on the first and second returned optical signals received by the HyDAR system under the second configuration.

[0261] In box 2236, the controller associates the first set of keypoints with the second set of keypoints. In box 2237, the controller calculates the transformation between the two sets of keypoints. In box 2238, the controller determines whether the transformation is as expected. If yes, the controller reports (box 2240) that the internal parameters are consistent, meaning that no internal calibration degradation was detected or minimal internal calibration degradation was detected. If no, the controller reports (box 2239) that the internal calibration has deviated from the expected value. Boxes 2236-2240 are substantially the same as or similar to boxes 2036-2040 in method 2030 for detecting external calibration degradation shown in Figure 20D, and therefore will not be described again.

[0262] As described above, the HyDAR system described herein can perform early fusion of point cloud data and image data to detect one or more degradation factors, including window occlusion, interfering light signals, external calibration degradation, and internal calibration degradation. Additionally, early fusion can be used to improve the resolution of the point cloud data. Figure 23 is a diagram illustrating the improved point cloud data resolution using image data according to various embodiments. As shown in Figure 23, point cloud data 2310 is generated by a LiDAR sensor; and image data 2320 is generated by an image sensor. Typically, point cloud data 2310 has a lower resolution than image data 2320. This is because modern image sensors have very large pixel arrays (e.g., tens of millions or hundreds of millions of pixels), while the resolution of LiDAR sensors is limited by scan speed, scan beam count, and laser pulse trigger rate.

[0263] Therefore, when the HyDAR system has an integrated multi-mode sensor including both a LiDAR sensor and an image sensor, at least a portion of the image data representing the second returned light signal has corresponding point cloud data representing the first returned light signal. On the other hand, a portion of the image data representing the second returned light signal may not have corresponding point cloud data representing the first returned light signal. For example, as shown in FIG23, when the LiDAR sensor and the image sensor of the HyDAR system sense returned light from the same FOV, data points 2312A, 2312B, and 2312C in point cloud data 2310 correspond to data points 2322A, 2322C, and 2322E in image data 2320, respectively. However, data points 2322B, 2322D, and 2322F in image data 2320 do not have corresponding data points in point cloud data 2310 because point cloud data 2310 has a lower point cloud resolution.

[0264] In some examples, the controller of the HyDAR system can infer distance information using portions of image data representing the second returned light signal that do not contain corresponding point cloud data representing the first returned light signal, based on at least a portion of image data representing the second returned light signal having corresponding point cloud data representing the first returned light signal. For example, as shown in Figure 23, in point cloud data 2330, data points 2312A, 2312B, and 2312C are data points acquired based on the first returned light signal detected by the LiDAR sensor of the HyDAR system. They are the same as the data points in point cloud data 2310. The controller can find their corresponding data points 2322A, 2322C, and 2322E in image data 2320. Because image data 2320 has a high resolution, there may be one or more additional data points between these data points 2322A, 2322C, and 2322E. As shown, there is an additional data point 2322B between data points 2322A and 2322C in image data 2320. The controller can determine whether data point 2322B has any changes compared to its neighboring data points 2232A and 2232C, and the extent of those changes. These changes can be in brightness, contrast, color, etc. Small changes indicate a smaller physical distance within the field of view (FOV) represented by neighboring data points 2232A, 2232B, and 2232C. Otherwise, the physical distance may be larger.

[0265] Therefore, using information obtained from the degree of change between data points 2322A, 2322B, and 2322C, the controller can infer the distance information of the inferred data point 2332 between the actual data points 2312A and 2312B in the point cloud data 2330. The inferred data point 2332 is not actually obtained by probing the first returned light signal. Instead, it is an inferred data point. For example, if the controller determines that the changes from data points 2322B to 2322A and from data points 2322B to 2322C are small, this means that the changes in the physical distances from the inferred data points 2332A to 2312A and from the inferred data points 2332A to 2312B are likely small. These physical distance changes can be calculated or inferred based on the changes calculated between data points 2322A, 2322B, and 2322C. Therefore, the distance at the inferred data point 2332A can be inferred or estimated with relatively high confidence.

[0266] The foregoing description should be understood as illustrative and exemplary in all respects, not restrictive, and the scope of the invention disclosed herein is not determined by the description, but by the claims as interpreted in the fullest extent permitted by patent law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A hybrid detection and ranging (HyDAR) system configured for detecting signals having multiple wavelengths, the system comprising: A laser source that provides a laser signal; An aperture window; one or more steering mechanisms configured to perform the following operations: directing a laser signal toward the aperture window; receiving a first return light signal formed based on at least a portion of the laser signal provided by the laser source; receiving a second return light signal formed by light provided by one or more sources outside the HyDAR system; a multi-mode sensor including a light detection and ranging (LiDAR) sensor and an image sensor, the LiDAR sensor being configured to detect the first return light signal to obtain one or more frames of point cloud data, and the image sensor being configured to detect the second return light signal to obtain one or more frames of image data, wherein the point cloud data and the image data are at least partially temporally and spatially synchronized at the hardware level of the HyDAR system; and a controller configured to perform the following operations: detecting one or more degradation factors affecting the performance of the HyDAR system; and adjusting the device configuration or operating conditions of the HyDAR system in response to detecting the one or more degradation factors to eliminate or reduce the impact of the degradation factors.

2. The system according to claim 1, wherein the image sensor comprises at least one of a near-infrared (NIR) sensor, a mid-infrared (MIR) sensor, or a visible light sensor.

3. The system of claim 1, wherein the one or more steering mechanisms include an optical scanner configured to perform the following operations: scanning the laser signal in a horizontal and vertical direction; receiving the first return light signal and the second return light signal; and directing the first return light signal and the second return light signal toward the LiDAR sensor and the image sensor, respectively.

4. The system of claim 1, wherein the first returned optical signal is formed by scattering or reflecting at least a portion of the laser signal through at least one of the following: an object located at or near the aperture window, the object obscuring at least a portion of the aperture window, or an object located at a distance from the aperture window.

5. The system of claim 1, wherein the second returned optical signal is formed by scattering or reflecting at least one of NIR, MIR, or visible light from one or more light sources outside the HyDAR system.

6. The system of claim 5, wherein the visible light from the one or more light sources outside the HyDAR system comprises one or more of the following: (a) sunlight; (b) moonlight; (c) light emitted from vehicle headlights; (d) streetlights; (e) reflected, transmitted, and / or refracted light of (a)-(d); and (f) other visible light sources.

7. The system of claim 1, further comprising a focusing lens, wherein one or more steering mechanisms are configured to direct the second returned light signal toward the image sensor via the focusing lens.

8. The system of claim 1, wherein detecting the one or more degradation factors affecting the performance of the HyDAR system comprises detecting one or more of the following: at least partial window occlusion of the aperture window; interference signals provided by one or more interfering light sources; external calibration degradation measured by the relationship between the HyDAR system and a movable platform on which the HyDAR system is mounted; and internal calibration degradation associated with misaligned internal components of the HyDAR system.

9. The system of claim 8, wherein detecting at least partial window occlusion of the aperture window comprises: Based on the one or more frames of the point cloud data, a first deviation of the first returned optical signal relative to a first expected value is obtained, wherein the first returned optical signal corresponds to a first region of the aperture window; Based on the one or more frames of the image data, a second deviation of the second returned light signal relative to the second expected value is obtained, the second returned light signal corresponding to a second region of the aperture window, the second region at least partially overlapping the first region; And at least one of the following operations: determining whether the first deviation exceeds a first threshold, or determining whether the second deviation exceeds a second threshold.

10. The system of claim 9, wherein the at least partial window occlusion of the aperture window further comprises: Based on at least one of the following: determining that the first deviation exceeds the first threshold, or determining that the second deviation exceeds the second threshold, one or more locations, one or more types, and degrees of blockage of the at least part of the aperture window are determined.

11. The system of claim 9, wherein the at least partial window occlusion of the aperture window further comprises: Determine whether the at least partial window occlusion lasts for at least one data collection cycle; Furthermore, the adjustment of the equipment or operating conditions of the HyDAR system includes: based on determining that the at least partial window occlusion has lasted for at least one data collection cycle, performing at least one of the following operations: reporting the at least partial window occlusion; or activating an occlusion removal mechanism to remove the at least partial window occlusion.

12. The system of claim 9, wherein the first deviation or the second deviation represents one or more of the following changes associated with the first or second returned optical signal: a change in signal strength or average signal strength; a change in the distribution of the signal strength; a change in the size and / or shape over time associated with one or both of the first and second regions of the aperture window; a change in sensitivity to a predetermined wavelength range; a change in the number of points in the point cloud data; or a change in the distance distribution represented by the point cloud data or the image data.

13. The system of claim 8, wherein the at least partial window occlusion of the aperture window being detected comprises: Using one or more machine learning networks, the point cloud data representing the first returned light signal and the image data representing the second returned light signal are fused to obtain fused data; and fused occlusion detection results are detected based on the fused data.

14. The system of claim 13, wherein the at least partial window occlusion of the aperture window includes: Based on the fused occlusion detection results, determine whether the occlusion of at least part of the window lasts for less than one data collection cycle; Furthermore, adjusting the device configuration or operating conditions of the HyDAR system includes, based on determining that the at least partial window occlusion lasts for at least one data collection cycle, performing at least one of the following operations: reporting the at least partial window occlusion; or activating an occlusion removal mechanism to remove the at least partial window occlusion.

15. The system of claim 8, wherein detecting interference signals caused by the one or more interfering light sources comprises: A first optical profile representing the characteristics of the first returned optical signal is obtained; a second optical profile representing the characteristics of the second returned optical signal is obtained. Determining whether at least one of the first or second light profile matches an interfering light profile associated with an interfering signal provided by the one or more interfering light sources; and wherein adjusting the equipment configuration or operating conditions of the HyDAR system includes: adjusting the laser power, noise filter, or at least one of the one or more steering mechanisms to reduce or prevent false detections by the HyDAR system based on the determination that at least one of the first or second light profile matches the interfering light profile.

16. The system of claim 15, wherein at least one of the first light profile, the second light profile, or the interfering light profile includes data representing one or more of the following: the measurement direction along which the HyDAR system detects the first returned light signal and / or the second returned light signal; the absolute intensity detected along the measurement direction; the relative intensity of different spectral ranges detected along the measurement direction; the detectable size or shape of at least one of the one or more interfering light sources; and the distance distribution over multiple data collection periods.

17. The system of claim 15, wherein the one or more interfering light sources include one or more of the sun, vehicle headlights / taillights, street lighting, HyDAR or LiDAR systems, and light redistribution mechanisms that redirect light from another interfering source.

18. The system of claim 15, wherein adjusting at least one of the laser power, the noise filter, or the one or more steering mechanisms to reduce or prevent false detections by the HyDAR system comprises one or more of the following operations: adjusting at least one of the one or more steering mechanisms to tune at least one of the start or end positions of the field of view of the HyDAR system to avoid the positions of the one or more interfering light sources; and adjusting at least one of the one or more steering mechanisms to tune the duration of the data collection cycle to avoid the positions of the one or more interfering light sources or reduce the likelihood of receiving the interfering signal at the HyDAR system.

19. The system of claim 15, wherein adjusting at least one of the laser power, the noise filter, or the one or more steering mechanisms to reduce or prevent false detections by the HyDAR system comprises one or more of the following operations: adjusting the laser power to increase the signal intensity ratio of the first returned optical signal to the interfering optical signal to at least a signal-to-noise ratio (SNR) threshold; and turning off the laser power to avoid directing the laser signal to the location of the one or more interfering light sources.

20. The system of claim 15, wherein adjusting at least one of the laser power, the noise filter, or the one or more steering mechanisms to reduce or prevent false detections by the HyDAR system comprises one or more of the following operations: adjusting the controller to enhance the noise filter to remove data representing at least a portion of the interfering optical signal provided by the one or more interfering light sources; and adjusting the controller to discard or ignore data representing at least a portion of the interfering optical signal provided by the one or more interfering light sources.

21. The system of claim 8, wherein detecting the external calibration degradation measured by the relationship between the HyDAR system and the mobile platform on which the HyDAR system is mounted comprises: (i) The point cloud data representing the first returned light signal and the image data representing the second returned light signal are combined to obtain a combined dataset; (ii) Segment the space of interest based on the combined dataset; (iii) Detect parallel line features in the space of interest based on the combined dataset; (iv) Identify the intersection points of the detected parallel line features.

22. The system of claim 21, wherein detecting the external calibration degradation measured by the relationship between the HyDAR system and the mobile platform on which the HyDAR system is mounted further comprises: Repeat at least some of the steps (i)-(iv) according to claim 21 to obtain a plurality of intersection locations; And based on the multiple intersection points, estimate whether the relationship between the HyDAR system and the mobile platform on which the HyDAR system is installed has deviated from the original configuration.

23. The system of claim 8, wherein detecting the external calibration degradation measured by the relationship between the HyDAR system and the mobile platform on which the HyDAR system is mounted comprises: Based on the point cloud data representing the first returned light signal and the image data representing the second returned light signal, identify one or more feature sets associated with one or more predefined stationary targets; and based on the identified one or more feature sets, provide one or more of the following information: the position of the one or more feature sets in three-dimensional (3D) space; the position of the one or more feature sets in elevation and azimuth angles; the point cloud data representing the first returned light signal at the region associated with the one or more feature sets; the image data representing the second returned light signal at the one or more feature sets; the gradient associated with the point cloud data representing the first returned light signal at the one or more feature sets; the gradient associated with the image data representing the second returned light signal at the one or more feature sets; the gradient histogram (HOG) associated with the point cloud data representing the first returned light signal at the one or more feature sets; and the gradient histogram (HOG) associated with the image data representing the second returned light signal at the one or more feature sets.

24. The system of claim 8, wherein detecting the external calibration degradation measured by the relationship between the HyDAR system and the mobile platform on which the HyDAR system is installed comprises: Based at least on the image data representing the second returned light signal, a first set of features associated with the predefined stationary target is identified; A second feature set is obtained from sensors outside the HyDAR system, including cameras, infrared cameras, and one or more of the LiDAR or HyDAR systems; a correspondence is established between the first feature set and the second feature set; an affine transformation between the first feature set and the second feature set is calculated; and based on the calculated transformation, it is estimated whether the relationship between the HyDAR system and the mobile platform on which the HyDAR system is installed has deviated from the original configuration.

25. The system of claim 8, wherein detecting internal calibration degradation associated with misaligned internal components of the HyDAR system comprises: Obtain at least the image data representing the second returned light signal under different internal configurations of the HyDAR system; Based on the image data, for each of the different internal configurations, multiple feature sets are identified from the same predefined target set; each feature set is associated with the physical settings of the predefined target to calculate the relative position and orientation of the corresponding internal configuration in the different internal configurations; and it is verified whether the calculated relative position and orientation of the different internal configurations are within the acceptable tolerance range of the target value.

26. The system of claim 1, wherein the point cloud data representing the first returned optical signal and the image data representing the second returned optical signal are time- and space-synchronized through the configuration of the one or more steering mechanisms.

27. The system according to claim 26, wherein: At least a portion of the image data representing the second returned light signal has corresponding point cloud data representing the first returned light signal; or a portion of the image data representing the second returned light signal does not have corresponding point cloud data representing the first returned light signal.

28. The system of claim 27, wherein the controller is further configured to perform: inferring distance information using the portion of the image data representing the second returned light signal that does not contain corresponding point cloud data representing the first returned light signal, based on the at least a portion of the image data representing the second returned light signal having corresponding point cloud data representing the first returned light signal.

Images