System and method for updating point clouds in lidar systems

Abstract

The LIDAR system includes at least one laser light source, at least one photosensitive detector including a plurality of pixels, and at least one processor configured to cause the at least one laser light source to project laser light toward a field of view of the LIDAR system, a point cloud including distance information relative to an object in a field of view of the LIDAR system is generated based on an output signal generated by at least one photosensitive detector in response to a received laser return signal reflected from the object in the field of view, and generating an image representing at least a portion of a field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector. The processor is configured to compare the point cloud to the image to detect whether there are inconsistencies between the representation in the point cloud and the corresponding representation in the image of the object from the field of view, and, in response to detecting one or more inconsistencies between the representation in the point cloud and the corresponding representation in the image of the object from the field of view, determine whether there is a difference between the representation in the point cloud and the corresponding representation in the image of the object from the field of view. One or more aspects of the generated point cloud are adjusted to provide an updated point cloud.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 373,056, filed on August 21, 2022, the disclosure of which is expressly incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to techniques for scanning a surrounding environment and, for example, to systems and methods for detecting objects in a surrounding environment using LIDAR technology. Background Art

[0004] With the advent of driver assistance systems and autonomous vehicles, motor vehicles need to be equipped with systems that can reliably sense and interpret their surroundings, including identifying obstacles, hazards, objects, and other physical parameters that may affect vehicle navigation. To this end, many different technologies have been proposed, including radar, LIDAR, and camera-based systems operating alone or in a redundant manner.

[0005] One consideration for driver assistance systems and autonomous vehicles is the ability of the system to determine the surrounding environment under different conditions including rain, fog, darkness, strong light and snow. Light detection and ranging systems (LIDAR, also known as LADAR) are examples of technologies that can work well under different conditions by illuminating an object with light and measuring the reflected pulses using a sensor to measure the distance to the object. A laser is an example of a light source that can be used in a LIDAR system. An electro-optical system (such as a LIDAR system) may include a light deflector, which is used to project light emitted by a light source into the environment of the electro-optical system. The light deflector can be controlled to pivot around at least one axis for projecting light into a desired position in the field of view of the electro-optical system. It may be desirable to design an improved system and method for determining the position and / or orientation of a light deflector for accurately controlling and / or monitoring the movement of the light deflector.

[0006] The systems and methods of the present disclosure are directed to improving the performance of monitoring the position and / or orientation of light deflectors used in electro-optical systems. Summary of the invention

[0007] One aspect of the present disclosure relates to a LIDAR system. The LIDAR system may include at least one laser light source; at least one photosensitive detector, the at least one photosensitive detector including a plurality of pixels; and at least one processor configured to perform operations. The processor may be configured to cause the at least one laser light source to project laser light toward the field of view of the LIDAR system; generate a point cloud including distance information relative to an object in the field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; and generate an image representing at least a portion of the field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector. The processor may also be configured to compare the point cloud with the image to detect whether there is an inconsistency between the representation in the point cloud and the corresponding representation in the image of the object from the field of view; and in response to detecting one or more inconsistencies between the representation in the point cloud and the corresponding representation in the image of the object from the field of view, adjust one or more aspects of the generated point cloud to provide an updated point cloud.

[0008] Another aspect of the present disclosure relates to a method for detecting an object using a LIDAR system. The method can perform operations including: causing at least one laser light source to project laser light toward a field of view of the LIDAR system; generating a point cloud including distance information relative to an object in the field of view of the LIDAR system based on an output signal generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; generating an image representing at least a portion of the field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud with the image to determine whether there is an inconsistency between the representation in the point cloud and a corresponding representation in the image of an object from the field of view; and adjusting one or more aspects of the generated point cloud to provide an updated point cloud based on detecting one or more inconsistencies between the representation in the point cloud and the corresponding representation in the image of an object from the field of view.

[0009] Another aspect of the present disclosure relates to a non-transitory computer-readable storage medium including stored instructions, which when executed by at least one processor, causes the at least one processor to perform a method for detecting an object using a LIDAR system. The method may include: causing at least one laser light source to project laser light toward a field of view of the LIDAR system; generating a point cloud including distance information relative to an object in the field of view of the LIDAR system based on an output signal generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; generating an image representing at least a portion of the field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud with the image to determine whether there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of an object from the field of view; and adjusting one or more aspects of the generated point cloud to provide an updated point cloud based on detecting one or more inconsistencies between a representation in the point cloud and a corresponding representation in the image of an object from the field of view.

[0010] Yet another aspect relates to a method for detecting an object. The method may include: receiving a point cloud including distance information relative to an object in a field of view based on an output signal generated by at least one photosensitive detector in response to a received laser return signal reflected from the object in the field of view; receiving an image representing at least a portion of the field of view based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud with the image to detect whether there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of the object from the field of view; and in response to detecting one or more inconsistencies between a representation in the point cloud and a corresponding representation in the image of the object from the field of view, adjusting one or more aspects of the generated point cloud to provide an updated point cloud.

[0011] Yet another aspect relates to a non-transitory computer-readable storage medium comprising stored instructions that, when executed by at least one processor, cause the at least one processor to perform a method. The method may include: receiving a point cloud including distance information relative to an object in a field of view based on an output signal generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; receiving an image representing at least a portion of the field of view based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to detect whether there is an inconsistency between representations in the point cloud and corresponding representations in the image of an object from the field of view; and in response to detecting one or more inconsistencies between representations in the point cloud and corresponding representations in the image of an object from the field of view, adjusting one or more aspects of the generated point cloud to provide an updated point cloud.

[0012] Yet another aspect relates to a method for detecting an object. The method may include: receiving a point cloud including distance information relative to an object in a field of view based on output signals generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; determining a first object in the point cloud by identifying a point group associated with the first object in the field of view; receiving an image representing at least a portion of the field of view based on output signals generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud with the image to determine whether there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of the first object from the field of view; and adjusting one or more aspects of the first object to provide an updated object map based on detecting one or more inconsistencies between a representation in the point group associated with the first object and a corresponding representation in the image of the first object from the field of view.

[0013] Yet another aspect relates to a non-transitory computer-readable storage medium comprising stored instructions that, when executed by at least one processor, cause the at least one processor to perform a method. The method may include: receiving a point cloud including distance information relative to objects in a field of view based on output signals generated by at least one photosensitive detector in response to received laser return signals reflected from objects in the field of view; determining a first object in the field of view in the point cloud by identifying a point group associated with the first object; receiving an image representing at least a portion of the field of view based on output signals generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to determine whether there is an inconsistency between representations in the point cloud and corresponding representations in the image of the first object from the field of view; and adjusting one or more aspects of the first object to provide an updated object map based on detecting one or more inconsistencies between representations in the point group associated with the first object and corresponding representations in the image of the first object from the field of view.

[0014] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

[0016] Figure 1A is a diagram illustrating an exemplary LIDAR system consistent with the disclosed embodiments.

[0017] Figure 1B is an image showing an exemplary output of a single scan cycle of a LIDAR system mounted on a vehicle consistent with the disclosed embodiments.

[0018] Figure 1C is another image showing a representation of a point cloud model determined from the output of a LIDAR system consistent with the disclosed embodiments.

[0019] Figure 2A-2G is a diagram showing different configurations of a projection unit according to some embodiments of the present disclosure.

[0020] Figures 3A-3D are diagrams illustrating different configurations of a scanning unit according to some embodiments of the present disclosure.

[0021] FIG. 4A to FIG. 4E are diagrams illustrating different configurations of a sensing unit according to some embodiments of the present disclosure.

[0022] Figure 5AIncluded are four example graphs showing emission patterns in a single frame time for a single portion of the field of view.

[0023] Figure 5B Included are three example graphs showing emission schemes in a single frame time for the entire field of view.

[0024] Figure 5C is a graph showing the actual light emissions projected toward the entire field of view and the reflections received during a single frame time.

[0025] Figures 6A-6C is a diagram illustrating a first example implementation consistent with some embodiments of the present disclosure.

[0026] Fig.6D is a diagram illustrating a second example implementation consistent with some embodiments of the present disclosure.

[0027] Figure 7 is a flow chart of an example method for resolving object detection ambiguity consistent with some embodiments of the present disclosure.

[0028] Fig. 8A and 8B are images showing representations of a point cloud and a camera image, respectively, generated from the output of a LIDAR system consistent with the disclosed embodiments.

[0029] Fig.9A and 9B is a graph illustrating detection of a signal consistent with some embodiments of the present disclosure.

[0030] Fig. 10A and 10B is a diagram illustrating exemplary optical paths of signals consistent with some embodiments of the present disclosure.

[0031] Fig.11 is a block diagram representation of various components of a LIDAR system consistent with some embodiments of the present disclosure.

[0032] Fig.12 A table showing classification of objects consistent with some embodiments of the present disclosure is presented.

[0033] Fig.13A and 13B A point cloud and an ambient light image with pixel saturation are shown, respectively, consistent with some embodiments of the present disclosure.

[0034] Fig.14A and 14B are example images with high resolution and low resolution, respectively, consistent with some embodiments of the present disclosure.

[0035] Fig.15is a flow chart of another example method for resolving object detection ambiguity consistent with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] The following detailed description refers to the accompanying drawings. Where possible, the same reference numerals are used in the drawings and the following description to represent the same or similar parts. Although several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, the components shown in the drawings may be replaced, added, or modified, and the illustrative methods described herein may be modified by replacing, reordering, removing, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the proper scope is defined by the appended claims.

[0037] Definition of terms

[0038] The disclosed embodiments may relate to optical systems. As used herein, the term "optical system" broadly includes any system for generating, detecting and / or manipulating light. By way of example only, an optical system may include one or more optical components for generating, detecting and / or manipulating light. For example, a light source, a lens, a mirror, a prism, a beam splitter, a collimator, a polarization optical device, an optical modulator, an optical switch, an optical amplifier, an optical detector, an optical sensor, an optical fiber, a semiconductor optical component, although each is not necessarily required, each can be a part of an optical system. In addition to one or more optical components, an optical system may also include other non-optical components, such as electrical components, mechanical components, chemical reaction components, and semiconductor components. Non-optical components may cooperate with optical components of an optical system. For example, an optical system may include at least one processor for analyzing detected light.

[0039] Consistent with the present disclosure, the optical system may be a LIDAR system. As used herein, the term "LIDAR system" broadly includes any system capable of determining the value of a parameter indicating the distance between a pair of tangible objects based on reflected light. In one embodiment, the LIDAR system may determine the distance between a pair of tangible objects based on reflections of light emitted by the LIDAR system. As used herein, the term "determining the distance" broadly includes generating an output indicating the distance between a pair of tangible objects. The determined distance may represent a physical dimension between a pair of tangible objects. By way of example only, the determined distance may include a flight line distance between a LIDAR system and another tangible object in the field of view of the LIDAR system. In another embodiment, the LIDAR system may determine the relative speed between a pair of tangible objects based on reflections of light emitted by the LIDAR system. Examples of output indicating the distance between a pair of tangible objects include: a number of standard units of length between the tangible objects (e.g., meters, inches, kilometers, millimeters), a number of arbitrary units of length (e.g., a number of LIDAR system lengths), a ratio between the distances to another length (e.g., a ratio to the lengths of objects detected in the field of view of the LIDAR system), an amount of time (e.g., given as standard units, arbitrary units, or a ratio, such as the time it takes light to travel between the tangible objects), one or more positions (e.g., specified using an agreed upon coordinate system, specified relative to a known position), etc.

[0040] The LIDAR system can determine the distance between a pair of tangible objects based on reflected light. In one embodiment, the LIDAR system can process the detection results of the sensor, which generates time information indicating the time period between the emission of the light signal and the time when the sensor detects the light signal. This time period is occasionally referred to as the "time offlight" of the light signal. In one example, the light signal can be a short pulse whose rise and / or fall time can be detected in reception. Using known information about the speed of light in the relevant medium (usually air), information about the time of flight of the light signal can be processed to provide the distance traveled by the light signal between emission and detection. In another embodiment, the LIDAR system can determine the distance based on frequency phase shift (or multi-frequency phase shift). Specifically, the LIDAR system can process information indicating one or more modulation phase shifts of the light signal (e.g., by solving some simultaneous equations to give the final measurement). For example, the emitted light signal can be modulated with one or more constant frequencies. At least one phase shift of the modulation between the emitted signal and the detected reflection can indicate the distance traveled by the light between emission and detection. Modulation can be applied to continuous wave light signals, quasi-continuous wave light signals, or other types of emitted light signals. Note that additional information may be used by the LIDAR system to determine distance, such as the projection location, position information (e.g., relative position) between the detection locations of the signal (especially if far from each other), etc.

[0041] In some embodiments, the LIDAR system can be used to detect multiple objects in the environment of the LIDAR system. The term "detecting an object in the environment of the LIDAR system" broadly includes generating information indicating an object that reflects light toward a detector associated with the LIDAR system. If the LIDAR system detects more than one object, the information generated about the different objects can be interconnected, such as a car driving on a road, a bird sitting on a tree, a person touching a bicycle, and a truck moving toward a building. The size of the environment in which the LIDAR system detects objects can vary relative to the implementation. For example, the LIDAR system can be used to detect multiple objects in the environment of a vehicle equipped with a LIDAR system, up to a horizontal distance of 100m (or 200m, 300m, etc.), and up to a vertical distance of 10m (or 25m, 50m, etc.). In another example, a LIDAR system can be used to detect multiple objects in the vehicle's environment or within a predetermined horizontal range (e.g., 25°, 50°, 100°, 180°, etc.) and up to a predetermined vertical elevation angle (e.g., ±10°, ±20°, +40°-20°, ±90°, or 0°-90°).

[0042] As used herein, the term "detecting an object" may broadly refer to determining the presence of an object (e.g., an object may exist in a certain direction relative to a LIDAR system and / or another reference location, or an object may exist in a certain volume of space). Additionally or alternatively, the term "detecting an object" may refer to determining the distance between an object and another location (e.g., the location of a LIDAR system, a location on the earth, or the location of another object). Additionally or alternatively, the term "detecting an object" may refer to identifying an object (e.g., classifying a type of object (such as a car, plant, tree, road); recognizing a specific object (e.g., the Washington Monument); determining a license plate number; determining the composition of an object (e.g., solid, liquid, transparent, translucent); determining kinematic parameters of an object (e.g., whether it is moving, its speed, its direction of movement, expansion of the object). Additionally or alternatively, the term "detecting an object" may refer to generating a point cloud map, wherein each point of the one or more points of the point cloud map corresponds to a location in the object or a location on a face thereof. In one embodiment, a data resolution associated with a point cloud map representation of a field of view may be associated with 0.1°×0.1° or 0.3°×0.3° of the field of view.

[0043] Consistent with the present disclosure, the term "object" broadly includes a finite composition of matter from which at least a portion of light can be reflected. For example, an object can be at least partially solid (e.g., a car, a tree); at least partially liquid (e.g., a puddle on the road, rain); at least partially gaseous (e.g., smoke, clouds); composed of a large number of different particles (e.g., a dust storm, fog, spray); and can have one or more magnitudes, such as ~1 millimeter (mm), ~5mm, ~10mm, ~50mm, ~100mm, ~500mm, ~1 meter (m), ~5m, ~10m, ~50m, ~100m, etc. Smaller or larger objects and any size between those examples can also be detected. Note that for various reasons, a LIDAR system may detect only a portion of an object. For example, in some cases, light may be reflected from only some sides of an object (e.g., only the side opposite the LIDAR system will be detected); in other cases, light may be projected onto only a portion of an object (e.g., a laser beam projected onto a road or building); in other cases, an object may be partially blocked by another object between the LIDAR system and the detected object; in other cases, the LIDAR's sensor may only detect light reflected from a portion of an object (e.g., because of ambient light or other interference that interferes with detection of some portions of the object).

[0044] Consistent with the present disclosure, a LIDAR system may be configured to detect an object by scanning the environment of the LIDAR system. The term "scanning the environment of the LIDAR system" broadly includes illuminating the field of view or a portion of the field of view of the LIDAR system. In one example, scanning the environment of the LIDAR system may be achieved by moving or pivoting a light deflector to deflect light in different directions toward different portions of the field of view. In another example, scanning the environment of the LIDAR system may be achieved by changing the positioning (i.e., position and / or orientation) of a sensor relative to the field of view. In another example, scanning the environment of the LIDAR system may be achieved by changing the positioning (i.e., position and / or orientation) of a light source relative to the field of view. In yet another example, scanning the environment of the LIDAR system may be achieved by changing the position of at least one light source and at least one sensor to move rigidly relative to the field of view (i.e., the relative distance and orientation of at least one sensor and at least one light source remain unchanged).

[0045] As used herein, the term "field of view of a LIDAR system" may broadly include the range of the observable environment of the LIDAR system in which objects may be detected. Note that the field of view (FOV) of a LIDAR system may be affected by various conditions, such as, but not limited to: the orientation of the LIDAR system (e.g., the direction of the optical axis of the LIDAR system); the positioning of the LIDAR system relative to the environment (e.g., the distance above the ground and adjacent terrain and obstacles); the operating parameters of the LIDAR system (e.g., transmit power, computing settings, defined operating angles), etc. For example, the field of view of the LIDAR system may be defined by a solid angle (e.g., defined using φ, θ angles, where φ and θ are angles defined in a vertical plane (e.g., relative to an axis of symmetry of the LIDAR system and / or its FOV)). In one example, the field of view may also be limited to a certain range (e.g., up to 200m).

[0046] Similarly, the term "instantaneous field of view" can broadly include the range of the observable environment in which the LIDAR system can detect objects at any given moment. For example, for a scanning LIDAR system, the instantaneous field of view is narrower than the entire FOV of the LIDAR system, and it can be moved within the FOV of the LIDAR system to enable detection in other parts of the FOV of the LIDAR system. Movement of the instantaneous field of view within the FOV of the LIDAR system can be achieved by moving a light deflector of the LIDAR system (or external to the LIDAR system) to deflect light beams to and / or from the LIDAR system in different directions. In one embodiment, the LIDAR system can be configured to scan a scene in the environment in which the LIDAR system operates. As used herein, the term "scene" can broadly include some or all objects within the field of view of the LIDAR system that are in their relative positions and their current state during the operating duration of the LIDAR system. For example, a scene may include ground elements (e.g., earth, roads, grass, sidewalks, road surface markings), sky, man-made objects (e.g., vehicles, buildings, signs), vegetation, people, animals, light projecting elements (e.g., flashlights, the sun, other LIDAR systems), and the like.

[0047] The disclosed embodiments may involve obtaining information for generating a reconstructed three-dimensional model. Examples of the types of reconstructed three-dimensional models that can be used include point cloud models and polygonal meshes (e.g., triangle meshes). The terms "point cloud" and "point cloud model" are well known in the art and should be interpreted as including a collection of data points that are spatially located in a certain coordinate system (i.e., have an identifiable location in the space described by the corresponding coordinate system). The term "point cloud point" refers to a point in space (which can be dimensionless, or a small unit of space, for example, 1 cm 3), and its position can be described by a point cloud model using a set of coordinates (e.g., (X, Y, Z), (r, ϕ, θ)). By way of example only, a point cloud model can store additional information for some or all of its points (e.g., color information for the points generated from a camera image). Similarly, any other type of reconstructed three-dimensional model can store additional information for some or all of its objects. Similarly, the terms "polygonal mesh" and "triangle mesh" are well known in the art and should be interpreted as including a collection of vertices, edges, faces, etc. that define the shape of one or more 3D objects (such as polyhedral objects). Faces can include one or more of the following: triangles (triangle meshes), quadrilaterals, or other simple convex polygons, as this can simplify rendering. Faces can also include more general concave polygons or polygons with holes. Polygonal meshes can be represented using different techniques, such as: vertex-vertex meshes, face-vertex meshes, wing-edge meshes, and rendering dynamic meshes. Different parts of the polygonal mesh (e.g., vertices, faces, edges) are spatially located in some coordinate system directly and / or relative to each other (i.e., have identifiable locations in a space described by a corresponding coordinate system). Generation of the reconstructed three-dimensional model can be accomplished using any standard, specialized, and / or novel photogrammetric techniques, many of which are known in the art. Note that the LIDAR system can generate other types of environment models.

[0048] Consistent with the disclosed embodiments, the LIDAR system may include at least one projection unit having a light source configured to project light. As used herein, the term "light source" refers broadly to any device configured to emit light. In one embodiment, the light source may be a laser (such as a solid-state laser, a laser diode, a high-power laser) or an alternative light source (such as a light source based on a light emitting diode (LED)). In addition, the light source 112 shown throughout the accompanying drawings may emit light in different formats, such as light pulses, continuous waves (CW), quasi-CW, and the like. For example, one type of light source that can be used is a vertical cavity surface emitting laser (VCSEL). Another type of light source that can be used is an external cavity diode laser (ECDL). In some examples, the light source may include a laser diode configured to emit light at a wavelength between about 650 nm and 1150 nm. Alternatively, the light source may include a laser diode configured to emit light having a wavelength between about 800 nm and about 1000 nm, between about 850 nm and about 950 nm, or between about 1300 nm and about 1600 nm. Unless otherwise specified, the term "about" with respect to a numerical value is defined as a variance of up to 5% relative to the stated value. Figure 2A-2C Additional details are described regarding the projection unit and the at least one light source.

[0049] Consistent with the disclosed embodiments, the LIDAR system may include at least one scanning unit having at least one light deflector configured to deflect light from a light source so as to scan a field of view. The term "light deflector" broadly includes any mechanism or module configured to deflect light from its original path; for example, a mirror, a prism, a controllable lens, a mechanical mirror, a mechanical scanning polygon, active diffraction (e.g., a controllable LCD), a Risley prism, non-mechanical electro-optical beam steering (such as manufactured by Vscent), a polarization grating (such as provided by Boulder Non-Linear Systems), an optical phased array (OPA), etc. In one embodiment, the light deflector may include multiple optical components, such as at least one reflective element (e.g., a mirror), at least one refractive element (e.g., a prism, a lens), etc. In one example, the light deflector may be movable to deflect light to different degrees (e.g., discrete degrees, or over a continuous span of degrees). The light deflector may optionally be controllable in different ways (e.g., deflect to an angle α, change the deflection angle Δα, move a component of the light deflector by M millimeters, change the speed at which the deflection angle is changed). In addition, the light deflector may optionally be operable to change the deflection angle within a single plane (e.g., a θ coordinate). The light deflector may optionally be operable to change the deflection angle within two non-parallel planes (e.g., θ and ϕ coordinates). Alternatively or additionally, the light deflector may optionally be operable to change the deflection angle between predetermined settings (e.g., along a predefined scanning route) or in other ways. With respect to the use of light deflectors in LIDAR systems, it should be noted that the light deflectors may be used in an outbound direction (also referred to as a transmit direction or TX) to deflect light from a light source to at least a portion of a field of view. However, the light deflectors may also be used in an inbound direction (also referred to as a receive direction or RX) to deflect light from at least a portion of the field of view to one or more light sensors. Reference is made below to Figure 3A-3C Additional details are described regarding the scanning unit and at least one light deflector.

[0050] The disclosed embodiments may involve pivoting an optical deflector in order to scan a field of view. As used herein, the term "pivot" broadly includes the rotation of an object (particularly a solid object) about one or more rotational axes while substantially keeping the center of rotation fixed. In one embodiment, the pivoting of the optical deflector may include the rotation of the optical deflector about a fixed axis (e.g., an axis), but not necessarily. For example, in some MEMS mirror embodiments, the MEMS mirror may be moved by actuating a plurality of benders connected to the mirror, and the mirror may undergo some spatial translation in addition to rotation. However, such a mirror may be designed to rotate about a substantially fixed axis, and therefore consistent with the present disclosure, it is considered to be pivoted. In other embodiments, some types of optical deflectors (e.g., non-mechanical electro-optical beam steering OPAs) do not require any moving parts or internal movement to change the deflection angle of the deflected light. Note that any discussion relating to moving or pivoting an optical deflector also applies, with necessary modifications, to controlling the optical deflector so that it changes the deflection behavior of the optical deflector. For example, controlling the optical deflector may cause a change in the deflection angle of a light beam arriving from at least one direction.

[0051] The disclosed embodiments may involve receiving a reflection associated with a portion of a field of view corresponding to a single instantaneous position of an optical deflector. As used herein, the term "instantaneous position of an optical deflector" (also referred to as "state of an optical deflector") broadly refers to a location or position in space where at least one controlled component of the optical deflector is located at an instantaneous point in time or within a short time span. In one embodiment, the instantaneous position of the optical deflector may be measured relative to a reference frame. The reference frame may relate to at least one fixed point in the LIDAR system. Or, for example, the reference frame may relate to at least one fixed point in the scene. In some embodiments, the instantaneous position of the optical deflector may include some movement of one or more components (e.g., mirrors, prisms) of the optical deflector, typically to a limited extent relative to a maximum degree of change during a scan of the field of view. For example, a scan of the entire field of view of the LIDAR system may include changing the deflection of the light over a span of 30°, and the instantaneous position of at least one optical deflector may include an angular displacement of the optical deflector within 0.05°. In other embodiments, the term "instantaneous position of a light deflector" may refer to the position of a light deflector during acquisition of light that is processed to provide data for a single point of a point cloud (or another type of 3D model) generated by the LIDAR system. In some embodiments, the instantaneous position of a light deflector may correspond to a fixed position or orientation in which the deflector is paused for a short time during illumination of a particular sub-region of the LIDAR field of view. In other cases, the instantaneous position of a light deflector may correspond to a certain position / orientation along a scanning range of positions / orientations of the light deflector that the light deflector traverses as part of a continuous or semi-continuous scan of the LIDAR field of view. In some embodiments, the light deflector may be moved so that during a scanning cycle of the LIDAR's FOV, the light deflector is at a plurality of different instantaneous positions. In other words, the deflector may move through a series of different instantaneous positions / orientations during the time period in which the scanning cycle occurs, and the deflector may reach each different instantaneous position / orientation at a different time during the scanning cycle.

[0052] Consistent with the disclosed embodiments, the LIDAR system may include at least one sensing unit having at least one sensor configured to detect reflections from an object in the field of view. The term "sensor" broadly includes any device, element, or system capable of measuring a characteristic of an electromagnetic wave (e.g., power, frequency, phase, pulse timing, pulse duration) and generating an output related to the measured characteristic. In some embodiments, the at least one sensor may include a plurality of detectors consisting of a plurality of detection elements. The at least one sensor may include one or more types of light sensors. Note that the at least one sensor may include a plurality of sensors of the same type, which may differ in other characteristics (e.g., sensitivity, size). Other types of sensors may also be used. Combinations of several types of sensors may be used for different reasons, such as improving detection over a range span (particularly over a small range); improving the dynamic range of the sensor; improving the temporal response of the sensor; and improving detection under changing environmental conditions (e.g., atmospheric temperature, rain, etc.).

[0053] In one embodiment, the at least one sensor includes a SiPM (silicon photomultiplier), which is a solid-state single-photon sensitive device built from an array of avalanche photodiodes (APDs), single-photon avalanche diodes (SPADs), used as detection elements on a common silicon substrate. In one example, the typical distance between SPADs can be between about 10 microns and about 50 microns, where each SPAD can have a recovery time between about 20ns and about 100ns. Similar photomultipliers from other non-silicon materials can also be used. Although the SiPM device operates in a digital / switching mode, the SiPM is an analog device because all microcells can be read in parallel, making it possible to generate signals over a dynamic range from a single photon to hundreds and thousands of photons detected by different SPADs. Note that the outputs from different types of sensors (e.g., SPADs, APDs, SiPMs, PIN diodes, photodetectors) can be combined together into a single output that can be processed by the processor of the LIDAR system. Refer to the following Figure 4A-4C Additional details regarding the sensing unit and the at least one sensor are described.

[0054] Consistent with the disclosed embodiments, the LIDAR system may include or communicate with at least one processor configured to perform different functions. The at least one processor may constitute any physical device having circuits that perform logical operations on one or more inputs. For example, the at least one processor may include one or more integrated circuits (ICs), including application specific integrated circuits (ASICs), microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or other circuits suitable for executing instructions or performing logical operations. The instructions executed by the at least one processor may be preloaded into a memory integrated with or embedded in the controller, for example, or may be stored in a separate memory. The memory may include a random access memory (RAM), a read-only memory (ROM), a hard disk, an optical disk, a magnetic medium, a flash memory, other permanent, fixed or volatile memory, or any other mechanism capable of storing instructions. In some embodiments, the memory is configured to store information representing data about an object in the environment of the LIDAR system. In some embodiments, the at least one processor may include more than one processor. Each processor may have a similar configuration, or the processors may have different configurations that are electrically connected or disconnected from each other. For example, the processors may be separate circuits or integrated in a single circuit. When more than one processor is used, the processors may be configured to operate independently or collaboratively. The processors may be coupled electrically, magnetically, optically, acoustically, mechanically, or by other means that allow them to interact. Figure 5A-Figure 5C Additional details are described regarding a processing unit and at least one processor.

[0055] System Overview

[0056] Figure 1AA LIDAR system 100 is shown that includes a projection unit 102, a scanning unit 104, a sensing unit 106, and a processing unit 108. The LIDAR system 100 can be mounted on a vehicle 110. Consistent with embodiments of the present disclosure, the projection unit 102 can include at least one light source 112, the scanning unit 104 can include at least one light deflector 114, the sensing unit 106 can include at least one sensor 116, and the processing unit 108 can include at least one processor 118. In one embodiment, the at least one processor 118 can be configured to coordinate the operation of the at least one light source 112 with the movement of the at least one light deflector 114 in order to scan a field of view 120. During a scanning cycle, each instantaneous position of the at least one light deflector 114 can be associated with a particular portion 122 of the field of view 120. Additionally, the LIDAR system 100 can include at least one optional optical window 124 for directing light projected toward the field of view 120 and / or receiving light reflected from objects in the field of view 120. The optional optical window 124 can serve different purposes, such as collimating the projected light and focusing the reflected light. In one embodiment, the optional optical window 124 can be an opening, a flat window, a lens, or any other type of optical window.

[0057] Consistent with the present disclosure, the LIDAR system 100 can be used in autonomous or semi-autonomous road vehicles (e.g., cars, buses, vans, trucks, and any other land vehicles). An autonomous road vehicle with a LIDAR system 100 can scan its environment and drive to a destination vehicle without human input. Similarly, the LIDAR system 100 can also be used in autonomous / semi-autonomous aircraft (e.g., UAVs, drones, quadcopters, and any other aerial vehicles or devices); or in autonomous or semi-autonomous watercraft (e.g., boats, ships, submarines, or any other vessels). Autonomous aircraft and vessels with a LIDAR system 100 can scan their environment and navigate to a destination autonomously or using a remote human operator. According to one embodiment, a vehicle 110 (road vehicle, aircraft, or vessel) can use a LIDAR system 100 to help detect and scan the environment in which the vehicle 110 is operating.

[0058] It should be noted that the LIDAR system 100 or any of its components can be used with any of the example embodiments and methods disclosed herein. In addition, although some aspects of the LIDAR system 100 are described with respect to an exemplary vehicle-based LIDAR platform, the LIDAR system 100, any of its components, or any of the processes described herein can be applicable to LIDAR systems of other platform types.

[0059] In some embodiments, the LIDAR system 100 may include one or more scanning units 104 to scan the environment around the vehicle 110. The LIDAR system 100 may be attached or mounted to any portion of the vehicle 110. The sensing unit 106 may receive reflections from the surroundings of the vehicle 110 and transmit a reflection signal indicating light reflected from an object in the field of view 120 to the processing unit 108. Consistent with the present disclosure, the scanning unit 104 may be mounted or incorporated into a bumper, fender, side panel, spoiler, roof, headlight assembly, taillight assembly, rearview mirror assembly, hood, cargo box, or any other suitable portion of the vehicle 110 that is capable of accommodating at least a portion of the LIDAR system. In some cases, the LIDAR system 100 may capture a complete surround view of the environment of the vehicle 110. Thus, the LIDAR system 100 may have a 360 degree horizontal field of view. In one example, as Figure 1A As shown, the LIDAR system 100 may include a single scanning unit 104 mounted on a rooftop vehicle 110. Alternatively, the LIDAR system 100 may include multiple scanning units (e.g., two, three, four, or more scanning units 104), each having a field of view such that the overall horizontal field of view is covered by a 360-degree scan around the vehicle 110. Those skilled in the art will appreciate that the LIDAR system 100 may include any number of scanning units 104 arranged in any manner, each scanning unit 104 having a field of view of 80° to 120° or less, depending on the number of units employed. In addition, a 360-degree horizontal field of view may also be obtained by mounting multiple LIDAR systems 100 on the vehicle 110, each LIDAR system 100 having a single scanning unit 104. However, it should be noted that one or more LIDAR systems 100 do not have to provide a full 360-degree field of view, and in some cases, a narrower field of view may be useful. For example, a vehicle 110 may require a first LIDAR system 100 with a 75° field of view looking forward of the vehicle, and may require a second LIDAR system 100 with a similar FOV looking backward (optionally with a lower detection range). Note also that different vertical field of view angles may also be implemented.

[0060] Figure 1B1 is an image showing an exemplary output from a single scanning cycle of a LIDAR system 100 mounted on a vehicle 110 consistent with the disclosed embodiments. In this example, the scanning unit 104 is incorporated into the right headlight assembly of the vehicle 110. Each gray point in the image corresponds to a location in the environment surrounding the vehicle 110 determined from reflections detected by the sensing unit 106. In addition to location, each gray point may also be associated with different types of information, such as intensity (e.g., how much light is returning from that location), reflectivity, proximity to other points, etc. In one embodiment, the LIDAR system 100 may generate multiple point cloud data entries from the detected reflections for multiple scanning cycles of the field of view to enable, for example, determination of a point cloud model of the environment surrounding the vehicle 110.

[0061] Figure 1C is an image showing a representation of a point cloud model determined from the output of the LIDAR system 100. Consistent with the disclosed embodiments, a surround view image can be generated from the point cloud model by processing the generated point cloud data entries of the environment surrounding the vehicle 110. In one embodiment, the point cloud model can be provided to a feature extraction module, which processes the point cloud information to identify a plurality of features. Each feature can include data about different aspects of objects in the point cloud and / or the environment surrounding the vehicle 110 (e.g., cars, trees, people, and roads). The features can have the same resolution as the point cloud model (i.e., have the same number of data points, optionally arranged as a 2D array of similar size), or can have a different resolution. The features can be stored in any kind of data structure (e.g., raster, vector, 2D array, 1D array). In addition, virtual features such as representations of the vehicle 110, boundary lines, or bounding boxes that separate regions or objects in the image (e.g., as Figure 1B ) and icons representing one or more recognized objects) can be superimposed on the representation of the point cloud model to form a final surround view image. For example, a symbol of vehicle 110 can be superimposed in the center of the surround view image.

[0062] Projection unit

[0063] Figure 2A-2G Various configurations of the projection unit 102 and its role in the LIDAR system 100 are depicted. Specifically, Figure 2A is a diagram showing a projection unit 102 having a single light source; Figure 2B is a diagram showing a plurality of projection units 102 having a plurality of light sources directed toward a common light deflector 114; Figure 2C is a diagram showing a projection unit 102 having a primary light source and a secondary light source 112; Figure 2D is a diagram showing an asymmetric deflector used in some configurations of the projection unit 102; Figure 2Eis a diagram showing a first configuration of a non-scanning LIDAR system; Figure 2F is a diagram showing a second configuration of a non-scanning LIDAR system; Figure 2G is a diagram showing a LIDAR system that scans in an outbound direction and does not scan in an inbound direction. Those skilled in the art will appreciate that the depicted configuration of projection unit 102 may have many variations and modifications.

[0064] Figure 2A An example of a bistatic configuration of the LIDAR system 100 is illustrated, wherein the projection unit 102 includes a single light source 112. The term "bistatic configuration" broadly refers to a LIDAR system configuration in which projected light exiting the LIDAR system and reflected light entering the LIDAR system traverse substantially different optical paths. In some embodiments, the bistatic configuration of the LIDAR system 100 can include separating the optical paths by using completely different optical components, by using parallel but not completely separate optical components, or by using the same optical components for only a portion of the optical paths (the optical components can include, for example, windows, lenses, mirrors, beam splitters, etc.). Figure 2A In the example shown, the bistatic configuration includes a configuration in which outgoing light and incident light pass through a single optical window 124, but the scanning unit 104 includes two light deflectors, a first light deflector 114A for outgoing light and a second light deflector 114B for incident light (incident light in a LIDAR system includes emitted light reflected from objects in a scene, and may also include ambient light arriving from other sources). Figure 2E and 2G In the example depicted in , the bistatic configuration includes a configuration in which the outgoing light passes through the first optical window 124A and the incoming light passes through the second optical window 124B. In all of the above example configurations, the incoming optical path and the outgoing optical path are different from each other.

[0065] In this embodiment, all components of the LIDAR system 100 may be contained within a single housing 200, or may be divided among multiple housings. As shown, the projection unit 102 is associated with a single light source 112, which includes a laser diode 202A (or one or more laser diodes coupled together) configured to emit light (projected light 204). In a non-limiting example, the light projected by the light source 112 may be at a wavelength between about 800nm ​​and 950nm, have an average power between about 50mW and about 500mW, have a peak power between about 50W and about 200W, and have a pulse width between about 2ns and about 100ns. In addition, the light source 112 may optionally be associated with an optical component 202B for manipulating the light emitted by the laser diode 202A (e.g., for collimation, focusing, etc.). Note that other types of light sources 112 may be used, and the present disclosure is not limited to laser diodes. In addition, the light source 112 can emit its light in different formats, such as light pulses, frequency modulation, continuous wave (CW), quasi-CW, or any other form corresponding to the specific light source employed. The projection format and other parameters can be changed from time to time by the light source based on different factors, such as instructions from the processing unit 108. The projected light is projected toward the exit deflector 114A, which acts as a steering element for directing the projected light in the field of view 120. In this example, the scanning unit 104 also includes a pivotable return deflector 114B, which directs photons (reflected light 206) reflected back from an object 208 within the field of view 120 toward the sensor 116. The reflected light is detected by the sensor 116, and information about the object (e.g., the distance to the object 212) is determined by the processing unit 108.

[0066] In this figure, the LIDAR system 100 is connected to a host 210. Consistent with the present disclosure, the term "host" refers to any computing environment that can interface with the LIDAR system 100, which can be a vehicle system (e.g., part of the vehicle 110), a test system, a security system, a surveillance system, a traffic control system, a city modeling system, or any system that monitors its surroundings. Such a computing environment may include at least one processor and / or may be connected to the LIDAR system 100 via a cloud. In some embodiments, the host 210 may also include an interface to external devices such as cameras and sensors configured to measure different characteristics of the host 210 (e.g., acceleration, steering wheel deflection, backdrive, etc.). Consistent with the present disclosure, the LIDAR system 100 may be fixed to a stationary object (e.g., a building, a tripod) associated with the host 210 or a portable system (e.g., a portable computer, a movie camera) associated with the host 210. Consistent with the present disclosure, the LIDAR system 100 may be connected to the host 210 to provide the output of the LIDAR system 100 (e.g., a 3D model, a reflectivity image) to the host 210. Specifically, the host 210 can use the LIDAR system 100 to help detect and scan the environment of the host 210 or any other environment. In addition, the host 210 can integrate, synchronize, or otherwise use the output of the LIDAR system 100 with the output of other sensing systems (e.g., cameras, microphones, radar systems). In one example, the LIDAR system 100 can be used by a security system. Figure 7 This embodiment is described in more detail.

[0067] The LIDAR system 100 may also include a bus 212 (or other communication mechanism) that interconnects subsystems and components for transmitting information within the LIDAR system 100. Optionally, the bus 212 (or another communication mechanism) may be used to interconnect the LIDAR system 100 with a host computer 210. Figure 2A In the example of , the processing unit 108 includes two processors 118 to adjust the operation of the projection unit 102, the scanning unit 104, and the sensing unit 106 in a coordinated manner based at least in part on information received from internal feedback of the LIDAR system 100. In other words, the processing unit 108 can be configured to dynamically operate the LIDAR system 100 in a loop. The loop system is characterized by having feedback from at least one element and updating one or more parameters based on the received feedback. In addition, the loop system can receive feedback and update its own operation based at least in part on the feedback. A dynamic system or element is a system or element that can be updated during operation.

[0068] According to some embodiments, scanning the environment around the LIDAR system 100 may include illuminating the field of view 120 with light pulses. The light pulses may have parameters such as pulse duration, pulse angular dispersion, wavelength, instantaneous power, photon density at different distances from the light source 112, average power, pulse power intensity, pulse width, pulse repetition rate, pulse sequence, pulse duty cycle, wavelength, phase, polarization, etc. Scanning the environment around the LIDAR system 100 may also include detecting and characterizing various aspects of the reflected light. The characteristics of the reflected light may include, for example: flight time (i.e., time from emission until detection), instantaneous power (e.g., power signature), average power over the entire return pulse, and photon distribution / signal over the return pulse period. By comparing the characteristics of the light pulses with the characteristics of the corresponding reflections, the distance and possibly physical characteristics (such as the reflection intensity of the object 212) may be estimated. By repeating this process over multiple adjacent portions 122, the entire scan of the field of view 120 may be achieved in a predetermined pattern (e.g., a grating, Lissajous, or other pattern). As discussed in more detail below, in some cases, LIDAR system 100 may direct light to only some portions 122 of field of view 120 during each scanning cycle. These portions may be adjacent to each other, but need not be.

[0069] In another embodiment, the LIDAR system 100 may include a network interface 214 for communicating with a host computer 210 (e.g., a vehicle controller). Communication between the LIDAR system 100 and the host computer 210 is represented by a dashed arrow. In one embodiment, the network interface 214 may include an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the network interface 214 may include a local area network (LAN) card to provide a data communication connection to a compatible LAN. In another embodiment, the network interface 214 may include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 214 depends on the communication network(s) through which the LIDAR system 100 and the host computer 210 operate. For example, the network interface 214 may be used to provide the output of the LIDAR system 100 (such as a 3D model, operating parameters of the LIDAR system 100, etc.) to an external system. In other embodiments, the communication unit may be used to, for example, receive instructions from an external system, receive information about the inspected environment, receive information from another sensor, and the like.

[0070] Figure 2BAn example of a monostatic configuration of a LIDAR system 100 including multiple projection units 102 is shown. The term "monostatic configuration" refers generally to a LIDAR system configuration in which the projected light exiting the LIDAR system and the reflected light entering the LIDAR system traverse substantially similar optical paths. In one example, the outgoing light beam and the incoming light beam can share at least one optical component through which both the outgoing light beam and the incoming light beam pass. In another example, the outgoing light can pass through an optical window (not shown) and the incoming light radiation can pass through the same optical window. The monostatic configuration can include a configuration in which the scanning unit 104 includes a single light deflector 114 that directs the projected light toward the field of view 120 and directs the reflected light toward the sensor 116. As shown, both the projected light 204 and the reflected light 206 hit the asymmetric deflector 216. The term "asymmetric deflector" refers to any optical device having two sides that is capable of deflecting a light beam that hits it from one side in a direction different from the direction in which it deflects a light beam that hits it from the second side. In one example, the asymmetric deflector does not deflect the projected light 204 and deflects the reflected light 206 toward the sensor 116. One example of an asymmetric deflector may include a polarizing beam splitter. In another example, the asymmetric 216 may include an optical isolator that allows light to pass in only one direction. A graphical representation of the asymmetric deflector 216 is shown in FIG. Figure 2D Consistent with the present disclosure, a monostatic configuration of LIDAR system 100 may include an asymmetric deflector to prevent reflected light from hitting light source 112 and direct all reflected light toward sensor 116, thereby increasing detection sensitivity.

[0071] exist Figure 2B In an embodiment of the present invention, the LIDAR system 100 includes three projection units 102, each projection unit 102 having a single light source 112 directed toward a common light deflector 114. In one embodiment, multiple light sources 112 (including two or more light sources) can project light having substantially the same wavelength, and each light source 112 is generally associated with a different region of the field of view (represented in the figure as 120A, 120B, and 120C). This enables scanning of a wider field of view than can be achieved using light sources 112. In another embodiment, multiple light sources 102 can project light having different wavelengths, and all light sources 112 can be directed to the same portion (or overlapping portions) of the field of view 120.

[0072] Figure 2CAn example of a LIDAR system 100 is shown, in which the projection unit 102 includes a main light source 112A and an auxiliary light source 112B. The main light source 112A can project light having a longer wavelength than the wavelength to which the human eye is sensitive, so as to optimize the SNR and the detection range. For example, the main light source 112A can project light having a wavelength between about 750nm and 1100nm. In contrast, the auxiliary light source 112B can project light having a wavelength visible to the human eye. For example, the auxiliary light source 112B can project light having a wavelength between about 400nm and 700nm. In one embodiment, the auxiliary light source 112B can project light along an optical path substantially the same as the light projected by the main light source 112A. The two light sources can be time-synchronized and can project light emissions together or in an interlaced pattern. The interlaced pattern means that the light sources are not active at the same time, which can mitigate mutual interference. Those skilled in the art will readily see that other combinations of wavelength ranges and activation schedules can also be implemented.

[0073] In accordance with some embodiments, the auxiliary light source 112B may cause a person's eye to blink when the auxiliary light source 112B is too close to the LIDAR optical output port. This can ensure an eye safety mechanism that is not feasible for a typical laser source utilizing a near infrared spectrum. In another embodiment, the auxiliary light source 112B can be used for calibration and reliability at a point of service, in a manner somewhat similar to the calibration of headlights with special reflectors / patterns at a certain height from the ground relative to the vehicle 110. An operator at the service point can detect the calibration of the LIDAR by simply visually inspecting the scan pattern on a feature target such as a test pattern plate at a specified distance from the LIDAR system 100. In addition, the auxiliary light source 112B can provide an operational confidence component for the end user that the LIDAR is working. For example, the system can be configured to allow a person to place a hand in front of the light deflector 114 to test its operation.

[0074] The auxiliary light source 112B may also have invisible elements that may double as a backup system in the event that the primary light source 112A fails. This feature may be useful for fail-safe devices with an increased level of functional safety. Considering that the auxiliary light source 112B may be visible and also due to cost and complexity reasons, the auxiliary light source 112B may be associated with less power than the primary light source 112A. Therefore, in the event that the primary light source 112A fails, the system functionality will fall back to the functionality and capabilities set by the auxiliary light source 112B. Although the capabilities of the auxiliary light source 112B may not be as good as those of the primary light source 112A, the LIDAR system 100 system may be designed in such a way that the vehicle 110 can safely reach its destination.

[0075] Figure 2DAn asymmetric deflector 216 is illustrated that may be part of the LIDAR system 100. In the illustrated example, the asymmetric deflector 216 includes a reflective surface 218 (such as a mirror) and a unidirectional deflector 220. Although not necessarily so, the asymmetric deflector 216 may optionally be a static deflector. The asymmetric deflector 216 may be used in a monostatic configuration of the LIDAR system 100 to allow for a common optical path for sending and receiving light via at least one deflector 114 (e.g., as Figure 2B and 2C However, typical asymmetric deflectors such as beam splitters are characterized by energy losses (especially in the receive path), which may be more sensitive to power losses than the transmit path.

[0076] like Figure 2D As depicted in , the LIDAR system 100 may include an asymmetric deflector 216 positioned in a transmission path, which includes a unidirectional deflector 220 for separating between a transmitted light signal and a received light signal. Optionally, the unidirectional deflector 220 may be substantially transparent to the transmitted light and substantially reflective to the received light. The transmitted light is generated by the projection unit 102 and may travel through the unidirectional deflector 220 to the scanning unit 104, which deflects it toward the optical exit. The received light reaches at least one deflection element 114 through the optical entrance, which deflects the reflected signal into a separate path away from the light source and toward the sensing unit 106. Optionally, the asymmetric deflector 216 may be combined with a polarized light source 112 that is linearly polarized using the same polarization axis as the unidirectional deflector 220. It is noteworthy that the cross-section of the outgoing light beam is much smaller than the cross-section of the reflected signal. Therefore, the LIDAR system 100 may include one or more optical components (e.g., lenses, collimators) for focusing or otherwise manipulating the emitted polarized light beam to the size of the asymmetric deflector 216. In one embodiment, the one-way deflector 220 may be a polarization beam splitter that is nearly transparent to the polarized light beam.

[0077] Consistent with some embodiments, the LIDAR system 100 may also include an optical device 222 (e.g., a quarter wave plate retarder) for modifying the polarization of the emitted light. For example, the optical device 222 may modify the linear polarization of the emitted light beam to a circular polarization. Light reflected from the field of view back to the system 100 will return to the optical device 222 through the deflector 114, carrying a circular polarization with a reverse handedness relative to the transmitted light. The optical device 222 then converts the received reverse handedness polarized light to a linear polarization that is not on the same axis as the linear polarization of the polarizing beam splitter 216. As noted above, the received light-patch is larger than the transmitted light-patch due to the optical dispersion of the beam traversing the distance to the target.

[0078] Some of the received light will hit the one-way deflector 220, which will reflect the light toward the sensor 106 with some power loss. However, another portion of the received light spot will fall on the reflective surface 218 (e.g., a polarizing beam splitter slit) around the one-way deflector 220. The reflective surface 218 will reflect the light toward the sensing unit 106 with essentially zero power loss. The one-way deflector 220 will reflect light composed of various polarization axes and directions that will eventually reach the detector. Optionally, the sensing unit 106 may include a sensor 116 that is agnostic to laser polarization and is primarily sensitive to the amount of incident photons in a certain wavelength range.

[0079] Note that the proposed asymmetric deflector 216 provides much superior performance when compared to a simple reflector with a channel hole therein. In a reflector with a hole, all reflected light reaching the hole is lost by the detector. However, in the deflector 216, the unidirectional deflector 220 deflects a large portion (e.g., about 50%) of this light toward the corresponding sensor 116. In a LIDAR system, the number of photons reaching the LIDAR from a long distance is very limited, so the improvement of the photon capture rate is important.

[0080] According to some embodiments, a device for beam splitting and steering is described. A polarized light beam can be emitted from a light source having a first polarization. The emission light beam can be directed through a polarizing beam splitter assembly. The polarizing beam splitter assembly includes a unidirectional slit on a first side and a reflector on an opposite side. The unidirectional slit enables the polarized emission light beam to travel toward a quarter wave plate / wave retarder, which changes the emission signal from a polarized signal to a linear signal (or vice versa), thereby preventing a subsequent reflected light beam from traveling through the unidirectional slit.

[0081] Figure 2EAn example of a bistatic configuration of the LIDAR system 100 without the scanning unit 104 is shown. In order to illuminate the entire field of view (or substantially the entire field of view) without the deflector 114, the projection unit 102 may optionally include an array of light sources (e.g., 112A-112F). In one embodiment, the light source array may include a linear array of light sources controlled by the processor 118. For example, the processor 118 may cause the linear array of light sources to sequentially project collimated laser beams toward the first optional optical window 124A. The first optional optical window 124A may include a diffuser lens for diffusing the projected light and sequentially forming a wide horizontal beam and a narrow vertical beam. Optionally, some or all of the at least one light source 112 of the system 100 may project light simultaneously. For example, the processor 118 may cause the light source array to project beams from multiple non-adjacent light sources 112 simultaneously. In the depicted example, light sources 112A, 112D, and 112F simultaneously project laser beams toward the first optional optical window 124A, thereby illuminating the field of view with three narrow vertical beams. The light beam from the fourth light source 112D can reach an object in the field of view. The light reflected from the object can be captured by the second optical window 124B and can be redirected to the sensor 116. Figure 2E The configuration depicted in is considered a bistatic configuration because the optical paths of the projected light and the reflected light are substantially different. Note that the projection unit 102 may also include a plurality of light sources 112 arranged in a non-linear configuration, such as a two-dimensional array, hexagonal tiling, or in any other manner.

[0082] Figure 2F An example of a monostatic configuration of the LIDAR system 100 without the scanning unit 104 is illustrated. Figure 2E Similar to the example embodiment shown in , in order to illuminate the entire field of view without the deflector 114, the projection unit 102 may include an array of light sources (e.g., 112A-112F). Figure 2E In contrast, this configuration of the LIDAR system 100 may include a single optical window 124 for both projected light and reflected light. Using the asymmetric deflector 216, the reflected light may be redirected to the sensor 116. Figure 2E The configuration shown in is considered a monostatic configuration because the optical paths of the projected light and the reflected light are substantially similar to each other. The term "substantially similar" in the context of the optical paths of the projected light and the reflected light means that the overlap between the two optical paths can be greater than 80%, greater than 85%, greater than 90%, or greater than 95%.

[0083] Figure 2G An example of a bistatic configuration of a LIDAR system 100 is shown. The configuration of the LIDAR system 100 in this figure is similar to Figure 2AFor example, both configurations include a scanning unit 104 for directing the projection light in an exit direction toward the field of view. Figure 2A In contrast to the embodiment of FIG. 1 , in this configuration, the scanning unit 104 does not redirect the reflected light in the incident direction. Instead, the reflected light passes through the second optical window 124B and enters the sensor 116. Figure 2G The configuration depicted in is considered a bistatic configuration because the optical paths of the projected light and the reflected light are substantially different from each other. The term "substantially different" in the context of the optical paths of the projected light and the reflected light means that the overlap between the two optical paths may be less than 10%, less than 5%, less than 1%, or less than 0.25%.

[0084] Scanning unit

[0085] Figures 3A-3D Various configurations of scanning unit 104 and their role in LIDAR system 100 are depicted. Specifically, Figure 3A is a diagram showing a scanning unit 104 having a MEMS mirror (eg, square), Figure 3B is a diagram showing another scanning unit 104 having a MEMS mirror (eg, circular), Figure 3C is a diagram showing a scanning unit 104 having a reflector array for a monostatic scanning LIDAR system, and Figure 3D is a diagram showing an example LIDAR system 100 that mechanically scans the environment surrounding the LIDAR system 100. Those skilled in the art will appreciate that the depicted configuration of the scanning unit 104 is merely exemplary and that there may be numerous variations and modifications within the scope of the present disclosure.

[0086] Figure 3AAn example scanning unit 104 having a single-axis square MEMS mirror 300 is shown. In this example, the MEMS mirror 300 is used as at least one deflector 114. As shown, the scanning unit 104 may include one or more actuators 302 (specifically, 302A and 302B). In one embodiment, the actuator 302 may be made of a semiconductor (e.g., silicon) and include a piezoelectric layer (e.g., PZT, lead zirconate titanate, aluminum nitride) that changes its size in response to an electrical signal applied by an actuation controller, a semiconductor layer, and a base layer. In one embodiment, the physical properties of the actuator 302 may determine the mechanical stress experienced by the actuator 302 when an electric current passes through it. When the piezoelectric material is activated, it exerts a force on the actuator 302 and causes it to bend. In one embodiment, when the mirror 300 is deflected at a certain angular position, the resistivity (Ractive) of one or more actuators 302 in the active state may be measured and compared to the resistivity (Rrest) in the static state. Feedback including Ractive can provide information to determine the actual mirror deflection angle compared to the expected angle, and if necessary, the deflection of the mirror 300 can be corrected. The difference between Rrest and Ractive can be correlated with the angle deflection value through the mirror drive, which can be used to close the loop. This embodiment can be used for dynamic tracking of the actual mirror position, and can optimize the response, amplitude, deflection efficiency and frequency of linear mode and resonant mode MEMS mirror solutions. This embodiment is described in more detail below with reference to Figures 32-34.

[0087] During scanning, current (represented as a dotted line in the figure) can flow from contact 304A to contact 304B (through actuator 302A, spring 306A, reflector 300, spring 306B and actuator 302B). Isolation gaps (such as isolation gap 310) in semiconductor frame 308 can make actuators 302A and 302B two separate islands electrically connected through spring 306 and frame 308. The current or any associated electrical parameters (voltage, current frequency, capacitance, relative dielectric constant, etc.) can be monitored by associated position feedback. In the event of a mechanical failure (one of the components is damaged), the current flowing through the structure will change and become different from its functional calibration value. In extreme cases (for example, when a spring is damaged), the current will stop completely because the faulty element interrupts the circuit in the electrical chain.

[0088] Figure 3BAnother example scanning unit 104 having a dual-axis circular MEMS mirror 300 is shown. In this example, the MEMS mirror 300 is used as at least one deflector 114. In one embodiment, the MEMS mirror 300 may have a diameter between about 1 mm and about 5 mm. As shown, the scanning unit 104 may include four actuators 302 (302A, 302B, 302C, and 302D), each of which may have a different length. In the example shown, the current (represented as a dashed line in the figure) flows from contact 304A to contact 304D, but in other cases, the current may flow from contact 304A to contact 304B, from contact 304A to contact 304C, from contact 304B to contact 304C, from contact 304B to contact 304D, or from contact 304C to contact 304D. Consistent with some embodiments, the dual-axis MEMS mirror can be configured to deflect light in both the horizontal and vertical directions. For example, the deflection angle of the dual-axis MEMS mirror can be between about 0° and 30° in the vertical direction and between about 0° and 50° in the horizontal direction. Those skilled in the art will appreciate that the configuration of the depicted mirror 300 can have many variations and modifications. In one example, at least the deflector 114 can have a dual-axis square mirror or a single-axis circular mirror. Depicted in Figure 3A and 3B The examples of circular and square mirrors are provided as examples only. Any shape may be used depending on the system specifications. In one embodiment, the actuator 302 may be incorporated as an integral part of at least the deflector 114 so that the power to move the MEMS mirror 300 is applied directly toward it. Additionally, the MEMS mirror 300 may be connected to the frame 308 by one or more rigid support elements. In another embodiment, at least the deflector 114 may include an electrostatic or electromagnetic MEMS mirror.

[0089] As described above, the monostatic scanning LIDAR system utilizes at least a portion of the same optical path to transmit the projected light 204 and to receive the reflected light 206. The light beam in the outgoing path can be collimated and focused into a narrow beam, while the reflection in the return path is spread into a larger spot due to dispersion. In one embodiment, the scanning unit 104 can have a large reflection area in the return path and an asymmetric deflector 216 that redirects the reflection (i.e., the reflected light 206) to the sensor 116. In one embodiment, the scanning unit 104 can include a MEMS mirror that has a large reflection area and a negligible impact on the field of view and frame rate performance. Refer to the following Figure 2D Additional details regarding the asymmetric deflector 216 are provided.

[0090] In some embodiments (e.g., Figure 3CIn one embodiment, implementing the light deflectors 114 as a group of smaller individual light deflectors operating in synchronization can allow the light deflectors 114 to perform at high scanning rates with larger deflection angles. In terms of effective area, the deflector array can essentially act as a large light deflector (e.g., a large mirror). The deflector array can be operated using a shared steering component configuration that allows the sensor 116 to collect reflected photons from substantially the same portion of the field of view 120 that is simultaneously illuminated by the light source 112. The term "simultaneously" means that two selected functions occur during coincident or overlapping time periods, where one function begins and ends during the duration of the other function, or where the latter function begins before the other function is completed.

[0091] Figure 3C An example of a scanning unit 104 having a reflector array 312 having small reflective mirrors is shown. In this embodiment, the reflector array 312 acts as at least one deflector 114. The reflector array 312 may include a plurality of reflector units 314 that are configured to pivot (individually or together) to steer light pulses toward the field of view 120. For example, the reflector array 312 may be part of an exit path of light projected from the light source 112. Specifically, the reflector array 312 may direct the projected light 204 toward a portion of the field of view 120. The reflector array 312 may also be part of a return path of light reflected from a surface of an object within the illuminated portion of the field of view 120. Specifically, the reflector array 312 may direct the reflected light 206 toward the sensor 116 or toward the asymmetric deflector 216. In one example, the area of ​​the reflector array 312 can be between about 75 mm2 and about 150 mm2, where each reflector element 314 can have a width of about 10 microns, and the support structure can be less than 100 microns.

[0092] According to some embodiments, the reflector array 312 may include one or more subsets of steerable deflectors. Each subset of electrically steerable deflectors may include one or more deflector units, such as reflector unit 314. For example, each steerable deflector unit 314 may include at least one of a MEMS mirror, a reflective surface assembly, and an electromechanical actuator. In one embodiment, each reflector unit 314 may be independently controlled by a separate processor (not shown) so that it can be tilted toward a specific angle along each of one or two separate axes. Alternatively, the reflector array 312 may be associated with a common controller (e.g., processor 118) that is configured to synchronously manage the movement of the reflector units 314 so that at least a portion of them will pivot simultaneously and point in approximately the same direction.

[0093] In addition, at least one processor 118 may select at least one reflector unit 314 (hereinafter referred to as a "TX reflector") for the outgoing path and a set of reflector units 314 (hereinafter referred to as "RX reflectors") for the return path. In accordance with the present disclosure, increasing the number of TX reflectors can increase the reflected photon beam spread. In addition, reducing the number of RX reflectors can narrow the receiving field and compensate for ambient light conditions (such as clouds, rain, fog, extreme heat, and other environmental conditions), and improve the signal-to-noise ratio. Moreover, as described above, the transmitted beam is generally narrower than the reflected spot, and therefore can be completely deflected by a small portion of the deflection array. In addition, light reflected from the transmitting portion of the deflection array (e.g., the TX reflector) can be blocked from reaching the sensor 116, thereby reducing the impact of internal reflections of the LIDAR system 100 on the operation of the system. In addition, at least one processor 118 can pivot one or more reflector units 314 to overcome mechanical damage and drift (e.g., mechanical damage and drift due to thermal and gain effects). In one example, one or more reflector units 314 may move differently than expected (frequency, rate, speed, etc.), and their movement may be compensated by appropriately electrically controlling the deflectors.

[0094] Figure 3DAn exemplary LIDAR system 100 is shown that mechanically scans the environment of the LIDAR system 100. In this example, the LIDAR system 100 may include a motor or other mechanism for rotating the housing 200 around the axis of the LIDAR system 100. Alternatively, the motor (or other mechanism) may mechanically rotate a rigid structure of the LIDAR system 100 on which one or more light sources 112 and one or more sensors 116 are mounted, thereby scanning the environment. As described above, the projection unit 102 may include at least one light source 112 that is configured to project light emissions. The projected light emissions may travel along an exit path toward the field of view 120. Specifically, when the projected light 204 travels toward the optional optical window 124, the projected light emissions may be reflected by the deflector 114A through the exit aperture 314. The reflected light emissions may travel along a return path from the object 208 toward the sensing unit 106. For example, the reflected light 206 may be reflected by the deflector 114B when traveling toward the sensing unit 106. Those skilled in the art will appreciate that a LIDAR system having a rotation mechanism for synchronously rotating one or more light sources or one or more sensors may use this synchronous rotation instead of (or in addition to) turning the internal light deflector.

[0095] In embodiments where the scanning of the field of view 120 is mechanical, the projected light emissions may be directed to an exit aperture 314, which is a portion of a wall 316 that separates the projection unit 102 from the rest of the LIDAR system 100. In some examples, the wall 316 may be formed of a transparent material (e.g., glass) coated with a reflective material to form the deflector 114B. In this example, the exit aperture 314 may correspond to a portion of the wall 316 that is not coated with a reflective material. Additionally or alternatively, the exit aperture 314 may include a hole or cut-away in the wall 316. The reflected light 206 may be reflected by the deflector 114B and directed toward an entrance aperture 318 of the sensing unit 106. In some examples, the entrance aperture 318 may include a filter window configured to allow wavelengths within a specific wavelength range to enter the sensing unit 106 and attenuate other wavelengths. Reflections from the object 208 of the field of view 120 may be reflected by the deflector 114B and hit the sensor 116. By comparing the reflected light 206 to several properties of the projected light 204, at least one aspect of the object 208 may be determined. For example, by comparing the time when the light source 112 emitted the projected light 204 and the time when the sensor 116 received the reflected light 206, the distance between the object 208 and the LIDAR system 100 may be determined. In some examples, other aspects of the object 208 (such as shape, color, material, etc.) may also be determined.

[0096] In some examples, LIDAR system 100 (or a portion thereof, including at least one light source 112 and at least one sensor 116) can rotate about at least one axis to determine a three-dimensional map of the surrounding environment of LIDAR system 100. For example, LIDAR system 100 can rotate about a substantially vertical axis (as shown by arrow 320) to scan field 120. Although Figure 3D The LIDAR system 100 is shown rotating clockwise around an axis (as indicated by arrow 320), but additionally or alternatively, the LIDAR system 100 may rotate in a counterclockwise direction. In some examples, the LIDAR system 100 may rotate 360 ​​degrees around a vertical axis. In other examples, the LIDAR system 100 may rotate back and forth along a sector of the LIDAR system 100 that is less than 360 degrees. For example, the LIDAR system 100 may be mounted on a platform that swings back and forth around an axis without making a full rotation.

[0097] Sensing unit

[0098] Figures 4A-4E Various configurations of the sensing unit 106 and their role in the LIDAR system 100 are depicted. Specifically, Figure 4A is a diagram showing an example sensing unit 106 having a detector array, Figure 4B is a diagram showing single-base scanning using a two-dimensional sensor, Figure 4C is a diagram showing an example of the two-dimensional sensor 116, Figure 4D is a diagram showing a lens array associated with sensor 116, and Figure 4E A diagram including three lens structures. It will be appreciated by those skilled in the art that the depicted configuration of the sensing unit 106 is merely exemplary and that many alternative variations and modifications are possible consistent with the principles of the present disclosure.

[0099] Figure 4AAn example of a sensing unit 106 having a detector array 400 is shown. In this example, at least one sensor 116 includes the detector array 400. The LIDAR system 100 is configured to detect objects (e.g., bicycle 208A and cloud 208B) in the field of view 120 located at different distances from the LIDAR system 100 (which may be several meters or more). The object 208 may be a solid object (e.g., a road, a tree, a car, a person), a fluid object (e.g., fog, water, atmospheric particles), or another type of object (e.g., dust or powdered lighting objects). When photons emitted from the light source 112 hit the object 208, they are reflected, refracted, or absorbed. Typically, as shown, only a portion of the photons reflected from the object 208A enter the optional optical window 124. Since each ~15cm distance change results in a 1ns travel time difference (because the photons travel to the object 208 at the speed of light and travel from the object 208 at the speed of light), the time difference between the travel times of different photons hitting different objects can be detected by a time-of-flight sensor with a sufficiently fast response.

[0100] The sensor 116 includes a plurality of detection elements 402 for detecting photons of a photon pulse reflected back from the field of view 120. The detection elements may all be included in a detector array 400, which may have a rectangular arrangement (e.g., as shown) or any other arrangement. The detection elements 402 may operate simultaneously or partially simultaneously with each other. Specifically, each detection element 402 may emit detection information for each sampling duration (e.g., every 1 nanosecond). In one example, the detector array 400 may be a SiPM (silicon photomultiplier), which is a solid-state single-photon sensitive device constructed from an array of single-photon avalanche diodes (SPADs, used as detection elements 402) on a common silicon substrate. Similar photomultipliers from other non-silicon materials may also be used. Although the SiPM device operates in a digital / switching mode, the SiPM is an analog device because all microcells are read in parallel, making it possible to generate signals in a dynamic range from a single photon to hundreds and thousands of photons detected by different SPADs. As mentioned above, more than one type of sensor (e.g., SiPM and APD) may be implemented. Possibly, the sensing unit 106 may include at least one APD integrated into a SiPM array and / or at least one APD detector located next to a SiPM on a separate or common silicon substrate.

[0101] In one embodiment, the detection elements 402 may be grouped into a plurality of regions 404. A region is a geometric location or environment within the sensor 116 (e.g., within the detector array 400) and may be shaped into different shapes (e.g., rectangular, square, circular, etc. as shown, or any other shape). While not all individual detectors included within the geometric area of ​​a region 404 must belong to that region, in most cases they will not belong to other regions 404 that cover other regions of the sensor 310 - unless some overlap is desired in the seams between regions. Figure 4A As shown, the regions may be non-overlapping regions 404, but alternatively, they may overlap. Each region may be associated with a regional output circuit 406 associated with the region. The regional output circuit 406 may provide a regional output signal for a corresponding group of detection elements 402. For example, the region of output circuit 406 may be a summing circuit, but other forms that combine the outputs of individual detectors into a single output (whether scalar, vector, or any other format) may be used. Optionally, each region 404 is a single SiPM, but this is not required, and the regions may be sub-portions of a single SiPM, groups of several SiPMs, or even combinations of different types of detectors.

[0102] In the example shown, the processing unit 108 is located at a separate housing 200B (inside or outside the host 210) of the host 210 (e.g., within the vehicle 110), and the sensing unit 106 may include a dedicated processor 408 for analyzing the reflected light. Alternatively, the processing unit 108 may be used to analyze the reflected light 206. Note that the LIDAR system 100 may implement multiple housings in other ways other than the example shown. For example, the light deflector 114 may be located in a different housing than the projection unit 102 and / or the sensing module 106. In one embodiment, the LIDAR system 100 may include multiple housings connected to each other in different ways (e.g., wired connections, wireless connections (e.g., RF connections), fiber optic cables, and any combination of the above).

[0103] In one embodiment, analyzing the reflected light 206 may include determining the time of flight of the reflected light 206 based on the outputs of the various detectors of the different regions. Optionally, the processor 408 may be configured to determine the time of flight of the reflected light 206 based on multiple regions of the output signal. In addition to the time of flight, the processing unit 108 may analyze the reflected light 206 to determine the average power over the entire return pulse, and may determine the photon distribution / signal over the return pulse period (“pulse shape”). In the example shown, the output of any detection element 402 may not be sent directly to the processor 408, but may be combined (e.g., summed) with the signals of other detectors of the region 404 before being passed to the processor 408. However, this is merely an example, and the circuitry of the sensor 116 may send information from the detection element 402 to the processor 408 via other routes (not via the regional output circuit 406).

[0104] Figure 4B is a diagram showing a LIDAR system 100 configured to scan an environment of the LIDAR system 100 using a two-dimensional sensor 116. Figure 4B In the example of , the sensor 116 is a matrix of 4×6 detectors 410 (also referred to as “pixels”). In one embodiment, the pixel size may be approximately 1×1 mm. The sensor 116 is two-dimensional in the sense that it has more than one set (e.g., rows, columns) of detectors 410 on two non-parallel axes (e.g., orthogonal axes, as illustrated in the example shown). The number of detectors 410 in the sensor 116 may vary between different implementations, e.g., depending on the desired resolution, signal-to-noise ratio (SNR), desired detection distance, etc. For example, the sensor 116 may have anywhere between 5 and 5,000 pixels. In another example (not shown in the figure), the sensor 116 may also be a one-dimensional matrix (e.g., 1×8 pixels).

[0105] Note that each detector 410 may include multiple detection elements 402, such as avalanche photodiodes (APDs), single photon avalanche diodes (SPADs), a combination of avalanche photodiodes (APDs) and single photon avalanche diodes (SPADs), or detection elements that measure the flight time from the laser pulse transmission event to the reception event and the intensity of the received photons. For example, each detector 410 may include any value between 20 and 5,000 SPADs. The outputs of the detection elements 402 in each detector 410 may be summed, averaged, or otherwise combined to provide a unified pixel output.

[0106] In the example shown, the sensing unit 106 may include a two-dimensional sensor 116 (or multiple two-dimensional sensors 116) whose field of view is smaller than the field of view 120 of the LIDAR system 100. In this discussion, the field of view 120 (the entire field of view that can be scanned by the LIDAR system 100 without moving, rotating, or rolling in any direction) is represented as the "first FOV 412", and the smaller FOV of the sensor 116 is represented as the "second FOV 412" (interchangeably referred to as the "instantaneous FOV"). The coverage area of ​​the second FOV 414 relative to the first FOV 412 can vary, depending on the specific use of the LIDAR system 100, and can be, for example, between 0.5% and 50%. In one example, the second FOV 412 can be extended in the vertical dimension by between about 0.05° and 1°. Even if the LIDAR system 100 includes more than one two-dimensional sensor 116 , the combined field of view of the sensor array may still be smaller than the first FOV 412 (eg, at least 5 times, at least 10 times, at least 20 times, or at least 50 times smaller).

[0107] To cover the first FOV 412, the scanning unit 106 can direct photons arriving from different parts of the environment to the sensor 116 at different times. In the illustrated monostatic configuration, while directing the projected light 204 toward the field of view 120 and when the at least one light deflector 114 is in the instantaneous position, the scanning unit 106 can also direct the reflected light 206 to the sensor 116. Typically, at each moment during scanning of the first FOV 412, the light beam emitted by the LIDAR system 100 covers a portion of the environment that is larger than the second FOV 414 (in the angular opening) and includes the portion of the environment from which the scanning unit 104 and the sensor 116 collect light.

[0108] Figure 4C is a diagram illustrating an example of a two-dimensional sensor 116. In this embodiment, the sensor 116 is a matrix of 8×5 detectors 410, and each detector 410 includes a plurality of detection elements 402. In one example, detector 410A is located in the second row (denoted as “R2”) and the third column (denoted as “C3”) of the sensor 116, which includes a matrix of 4×3 detection elements 402. In another example, detector 410B located in the fourth row (denoted as “R4”) and the sixth column (denoted as “C6”) of the sensor 116 includes a matrix of 3X3 detection elements 402. Thus, the number of detection elements 402 in each detector 410 may be constant, or may vary, and different detectors 410 in a common array may have different numbers of detection elements 402. The outputs of all of the detection elements 402 in each detector 410 may be summed, averaged, or otherwise combined to provide a single pixel output value. Note that while Figure 4CThe detectors 410 in the example of FIG. 5 are arranged in a rectangular matrix (straight rows and columns), but other arrangements may also be used, such as a circular arrangement or a honeycomb arrangement.

[0109] According to some embodiments, measurements from each detector 410 may enable determination of the time of flight from a light pulse emission event to a reception event and the intensity of the received photons. The reception event may be the result of a light pulse reflecting from an object 208. The time of flight may be a timestamp value representing the distance of the reflecting object to the optional optical window 124. The time of flight value may be achieved by photon detection and counting methods such as a time-correlated single photon counter (TCSPC), analog methods for photon detection such as signal integration and quantization (via an analog-to-digital converter or a common comparator), or other methods.

[0110] In some embodiments and with reference to Figure 4B , during a scanning cycle, each instantaneous position of at least one light deflector 114 can be associated with a particular portion 122 of the field of view 120. The design of the sensor 116 enables association between reflected light from a single portion of the field of view 120 and a plurality of detectors 410. Thus, the scanning resolution of the LIDAR system can be represented by the number of instantaneous positions (per scanning cycle) multiplied by the number of detectors 410 in the sensor 116. The information from each detector 410 (i.e., each pixel) represents a basic data element for constructing a captured field of view in three-dimensional space. For example, this can include a basic element of a point cloud representation, which has a spatial position and an associated reflection intensity value. In one embodiment, reflections from a single portion of the field of view 120 detected by the plurality of detectors 410 can be returned from different objects located in the single portion of the field of view 120. For example, a single portion of the field of view 120 can be larger than 50×50 cm at the far field, which can easily include two, three, or more objects that partially cover each other.

[0111] Figure 4D is a cross-sectional view of a portion of a sensor 116 according to an example of the presently disclosed subject matter. The illustrated portion of the sensor 116 includes a portion of a detector array 400 that includes four detection elements 402 (e.g., four SPADs, four APDs). The detector array 400 can be a photodetector sensor implemented in a complementary metal oxide semiconductor (CMOS). Each of the detection elements 402 has a sensitive area that is positioned within the substrate surroundings. Although not necessarily the case, the sensor 116 can be used in a monostatic LiDAR system with a narrow field of view (e.g., because the scanning unit 104 scans different portions of the field of view at different times). The narrow field of view of the incoming light beam (if implemented) eliminates the problem of out-of-focus imaging. As Figure 4DAs illustrated in , the sensor 116 may include multiple lenses 422 (e.g., microlenses), each of which may direct incident light toward a different detection element 402 (e.g., toward an active area of ​​the detection element 402), which may be useful when out-of-focus imaging is not an issue. The lenses 422 may be used to increase the optical fill factor and sensitivity of the detector array 400, because most of the light reaching the sensor 116 may be deflected toward the active area of ​​the detection element 402.

[0112] like Figure 4D The detector array 400 illustrated in the example may include several layers built into the silicon substrate by various methods (e.g., implantation) to produce sensitive regions, contact elements to the metal layers, and isolation elements (e.g., shallow trench implantation STI, guard rings, optical trenches, etc.). The sensitive region may be a volume element in a CMOS detector that enables optical conversion of incident photons into current as long as the appropriate voltage bias is applied to the device. In the case of APD / SPAD, the sensitive region will be a combination of electric fields that pull electrons generated by photon absorption toward a multiplication region where the photon-induced electrons are amplified, producing a breakdown avalanche of multiplied electrons.

[0113] Front-side illumination detectors (e.g. Figure 4D The optical port through which photons impinge on the detector sensitive area is formed by a channel through the metal layer. It should be noted that the channel through which light from some directions may be blocked by one or more metal layers (e.g., metal layer ML6, as shown). Figure 4D The optical light absorption efficiency of the detector is reduced by the fact that the detector is blocked by the leftmost detector element 402 in the figure. This blockage reduces the overall optical light absorption efficiency of the detector.

[0114] Figure 4E Three detection elements 402 are shown, each with an associated lens 422, according to an example of the disclosed subject matter. Figure 4E Each of the three detection elements denoted 402(1), 402(2), and 402(3) illustrates a lens configuration that may be implemented in association with one or more of the detection elements 402 of the sensor 116. Note that combinations of these lens configurations may also be implemented.

[0115] In the lens configuration shown with respect to detection element 402 (1), the focal point of the associated lens 422 can be located above the semiconductor surface. Optionally, openings in different metal layers of the detection element can have different sizes aligned with the focused light cone generated by the associated lens 422. Such a structure can improve the signal-to-noise ratio and resolution of the array 400 as a whole device. Large metal layers can be important for power delivery and ground shielding. This approach can be useful, for example, for monostatic LiDAR designs with a narrow field of view, where the incident beam consists of parallel rays and the imaging focus has no effect on the detected signal.

[0116] In the lens configuration shown with respect to detection element 402 (2), the efficiency of photon detection by detection element 402 can be improved by identifying a sweet point. Specifically, a photodetector implemented in CMOS can have a sweet point in a sensitive volume region where the probability of photons producing an avalanche effect is highest. Therefore, the focus of lens 422 can be positioned at the sweet point location inside the sensitive volume region, as shown by detection element 402 (2). The lens shape and distance from the focus can take into account the refractive index of all elements that the laser beam passes through along the path from the lens to the sensitive sweet point location buried in the semiconductor material.

[0117] In About Figure 4E In the lens configuration shown on the right side of the detection element, diffusers and reflective elements can be used to improve the efficiency of photon absorption in the semiconductor material. Specifically, near-IR wavelengths require the silicon material to travel a very long path in order to achieve a high probability of absorbing the photons that travel through. In a typical lens configuration, a photon may pass through the sensitive area and may not be absorbed into a detectable electron. For CMOS devices manufactured with typical foundry processes, the long absorption path that increases the probability of a photon producing an electron will cause the size of the sensitive area to become impractical dimensions (e.g., tens of μm). Figure 4EThe rightmost detector element in the figure illustrates a technique for processing incident photons. An associated lens 422 focuses the incident light onto a diffuser element 424. In one embodiment, the light sensor 116 may further include a diffuser located in a gap away from the outer surface of at least some of the detectors. For example, the diffuser 424 may deflect the light beam laterally (e.g., as vertically as possible) toward the sensitive area and the reflective optical groove 426. The diffuser is located at, above, or below the focus. In this embodiment, the incident light can be focused on a specific location where the diffuser element is located. Optionally, the detector element 422 is designed to optically avoid an inactive region where electrons excited by the photons may be lost and reduce the effective detection efficiency. The reflective optical groove 426 (or other form of optical reflective structure) causes the photons to bounce back and forth on the sensitive area, thereby increasing the probability of detection. Ideally, the photon will be trapped indefinitely in the cavity composed of the sensitive area and the reflective groove until the photon is absorbed and an electron / hole pair is generated.

[0118] Consistent with the present disclosure, long paths are created to allow the impinging photons to be absorbed to facilitate a higher probability of detection. Optical grooves may also be implemented in the detection element 422 to reduce cross talk effects of parasitic photons generated during avalanches, which may leak to other detectors and cause false detection events. According to some embodiments, the photodetector array may be optimized so that a higher yield of received signals is utilized, meaning that as much received signal as possible is received and less signal is lost due to internal degradation of the signal. The photodetector array may be improved by: (a) moving the focus to a location above the semiconductor surface, optionally by appropriately designing the metal layer above the substrate; (b) by steering the focus to the most responsive / sensitive area (or "sweet spot") of the substrate, and (c) adding a diffuser above the substrate to steer the signal toward the "sweet spot" and / or adding reflective material to the grooves so that the deflected signal is reflected back to the "sweet spot".

[0119] While in some lens configurations, the lens 422 may be positioned so that its focal point is over the center of the corresponding detection element 402, note that this is not required. In other lens configurations, the positioning of the focal point of the lens 422 relative to the center of the corresponding detection element 402 is shifted based on the distance of the corresponding detection element 402 from the center of the detection array 400. This may be useful in relatively large detection arrays 400, where detector elements further from the center receive light at increasingly off-axis angles. Shifting the position of the focal point (e.g., toward the center of the detection array 400) allows correction of the angle of incidence. Specifically, shifting the position of the focal point (e.g., toward the center of the detection array 400) allows correction of the angle of incidence while using substantially the same lens 422 for all detection elements, which is positioned at the same angle relative to the surface of the detector.

[0120] When using a relatively small sensor 116 that covers only a small portion of the field of view, it may be useful to add an array of lenses 422 to the array of detecting elements 402, because in this case, the reflected signals from the scene arrive at the detector array 400 from substantially the same angle, and it is therefore easy to focus all light onto the individual detectors. It should also be noted that in one embodiment, the lens 422 can be used in the LIDAR system 100 to facilitate increasing the overall probability of detection of the entire array 400 (preventing photons from being "wasted" in dead zones between detectors / sub-detectors), at the expense of spatial distinctiveness. This embodiment is in contrast to prior art implementations (such as CMOS RGB cameras), which prioritize spatial distinctiveness (i.e., light propagating in the direction of detecting element A is not allowed to be directed by the lens towards detecting element B, i.e., "bleed" to another detecting element of the array). Optionally, the sensor 116 includes an array of lenses 422, each lens 422 being associated with a corresponding detection element 402, and at least one of the lenses 422 deflects light propagating to a first detection element 402 toward a second detection element 402 (thereby increasing the overall probability of detection for the entire array).

[0121] Specifically, consistent with some embodiments of the present disclosure, the light sensor 116 may include an array of light detectors (e.g., detector array 400), each light detector (e.g., detector 410) being configured to cause an electric current to flow when light passes through an outer surface of the corresponding detector. In addition, the light sensor 116 may include at least one microlens configured to direct light toward the light detector array, the at least one microlens having a focal point. The light sensor 116 may also include at least one layer of conductive material inserted between the at least one microlens and the light detector array, and having a gap therein that allows light to pass from the at least one microlens to the array, the at least one layer being sized to maintain a space between the at least one microlens and the array so that the focal point (e.g., the focal point may be a plane) is located in the gap, at a position spaced apart from the detection surface of the light detector array.

[0122] In a related embodiment, each detector may include a plurality of single photon avalanche diodes (SPADs) or a plurality of avalanche photodiodes (APDs). The conductive material may be a multilayer metal structure, and at least one layer of conductive material may be electrically connected to the detectors in the array. In one example, at least one conductive material layer includes a plurality of layers. In addition, the gap may be shaped to converge from at least one microlens toward the focus and diverge from the area of ​​the focus toward the array. In other embodiments, the light sensor 116 may also include at least one reflector adjacent to each photodetector. In one embodiment, a plurality of microlenses may be arranged in a lens array, and a plurality of detectors may be arranged in a detector array. In another embodiment, a plurality of microlenses may include a single lens configured to project light to a plurality of detectors in the array.

[0123] refer to Figure 2E , Figure 2F and Figure 2GAs a non-limiting example, note that one or more sensors 116 of the system 100 may receive light from the scanning deflector 114 or directly from the FOV without scanning. Even if light from the entire FOV reaches at least one sensor 116 simultaneously, in some embodiments, one or more sensors 116 may sample only a portion of the FOV at any given time for detection output. For example, if the illumination of the projection unit 102 illuminates different portions of the FOV at different times (whether using the deflector 114 and / or by activating different light sources 112 at different times), then light may reach all pixels or sensors 116 of the sensing unit 106, and only pixels / sensors that are intended to detect LIDAR illumination may actively collect data for detection output. In this way, the remaining pixels / sensors are not unnecessarily collecting ambient noise. With respect to scanning - either in the outgoing direction or in the incoming direction - it should be noted that substantially different scanning ranges may be achieved. For example, in some embodiments, the scanning area may cover 1‰ or 0.1‰ of the FOV, while in other embodiments, the scanning area may cover 10% or 25% of the FOV. Of course, all other relative portions of the FOV value may also be implemented.

[0124] Processing Unit

[0125] Figures 5A-5C Describes different functions of the processing unit 108 according to some embodiments of the present disclosure. Specifically, Figure 5A is a diagram illustrating the emission pattern in a single frame time for a single portion of the field of view, Figure 5B is a diagram showing the emission scheme in a single frame time for the entire field of view. Figure 5C is a graph showing the actual light emission projected towards the field of view during a single scan cycle.

[0126] Figure 5A Four examples of emission patterns in a single frame time of a single portion 122 of a field of view 120 associated with an instantaneous position of at least one light deflector 114 are illustrated. Consistent with embodiments of the present disclosure, the processing unit 108 may control (or coordinate the operation of) at least one light source 112 and the light deflector 114 in a manner that enables light flux to vary while scanning the field of view 120. Consistent with other embodiments, the processing unit 108 may control only the at least one light source 112, and the light deflector 114 may move or pivot in a fixed, predefined pattern.

[0127] Figure 5AGraphs AD in FIG. depict the power of light emitted toward a single portion 122 of the field of view 120 over time. In FIG. A, the processor 118 may control the operation of the light source 112 in such a manner that an initial light emission is projected toward a portion 122 of the field of view 120 during scanning of the field of view 120. When the projection unit 102 includes a pulsed light source, the initial light emission may include one or more initial pulses (also referred to as “pilot pulses”). The processing unit 108 may receive pilot information about reflections associated with the initial light emission from the sensor 116. In one embodiment, the pilot information may be represented as a single signal based on the output of one or more detectors (e.g., one or more SPADs, one or more APDs, one or more SiPMs, etc.), or as multiple signals based on the output of multiple detectors. In one example, the pilot information may include analog and / or digital information. In another example, the pilot information may include a single value and / or multiple values ​​(e.g., for different times and / or portions of a segment).

[0128] Based on the information about the reflection associated with the initial light emission, the processing unit 108 can be configured to determine the type of subsequent light emission to be projected toward the portion 122 of the field of view 120. The subsequent light emission determined for the particular portion of the field of view 120 can be made during the same scanning cycle (i.e., in the same frame) or in a subsequent scanning cycle (i.e., in a subsequent frame). This embodiment is described in more detail below with reference to Figures 23-25.

[0129] In FIG. B, the processor 118 may control the operation of the light source 112 in such a manner that light pulses of different intensities are projected toward a single portion 122 of the field of view 120 during scanning of the field of view 120. In one embodiment, the LIDAR system 100 may be operable to generate one or more different types of depth maps, such as any one or more of the following types: a point cloud model, a polygonal mesh, a depth image (maintaining depth information for each pixel of an image or a 2D array), or any other type of 3D model of a scene. The sequence of depth maps may be a time sequence, where different depth maps are generated at different times. Each depth map of a sequence associated with a scanning cycle (interchangeably referred to as a "frame") may be generated for a duration corresponding to a subsequent frame time. In one example, a typical frame time may last less than one second. In some embodiments, the LIDAR system 100 may have a fixed frame rate (e.g., 10 frames per second, 25 frames per second, 50 frames per second), or the frame rate may be dynamic. In other embodiments, the frame times of different frames may be different across the sequence. For example, LIDAR system 100 may implement a rate of 10 frames per second, which includes generating a first depth map at 100 milliseconds (on average), a second frame at 92 milliseconds, a third frame at 142 milliseconds, and so on.

[0130] In FIG. C, the processor 118 may control the operation of the light source 112 in such a manner that light pulses associated with different durations are projected toward a single portion 122 of the field of view 120 during scanning of the field of view 120. In one embodiment, the LIDAR system 100 may be operable to generate a different number of pulses in each frame. The number of pulses may vary between 0 and 32 pulses (e.g., 1, 5, 12, 28, or more pulses) and may be based on information derived from previous emissions. The time between light pulses may depend on the desired detection range and may be between 500ns and 5000ns. In one example, the processing unit 108 may receive information about the reflection associated with each light pulse from the sensor 116. Based on this information (or lack of information), the processing unit 108 may determine whether additional light pulses are needed. Note that the durations of the processing time and the emission time in FIG. AD are not to scale. Specifically, the processing time may be substantially longer than the emission time. In FIG. D, the projection unit 102 may include a continuous wave light source. In one embodiment, the initial light emission may include a time period for emitting light, and the subsequent emission may be a continuation of the initial emission, or there may be a discontinuity. In one embodiment, the intensity of the continuous emission may vary over time.

[0131] Consistent with some embodiments of the present disclosure, an emission pattern may be determined for each portion of the field of view 120. In other words, the processor 118 may control the emission of light to allow for differentiated illumination of different portions of the field of view 120. In one example, the processor 118 may determine an emission pattern for a single portion 122 of the field of view 120 based on detection of reflected light from the same scanning cycle (e.g., initial emission), which makes the LIDAR system 100 extremely dynamic. In another example, the processor 118 may determine an emission pattern for a single portion 122 of the field of view 120 based on detection of reflected light from a previous scanning cycle. Differences in patterns for subsequent emissions may result from determining different values ​​for light source parameters for subsequent emissions, such as any of the following.

[0132] a. The total energy of subsequent emissions.

[0133] b. Energy profile of subsequent emissions.

[0134] c. The number of light pulse repetitions per frame.

[0135] d. Optical modulation characteristics such as duration, rate, peak, average power and pulse shape.

[0136] e. The characteristics of the subsequently emitted wave, such as polarization, wavelength, etc.

[0137] Consistent with the present disclosure, differentiation in subsequent emissions may be used for different purposes. In one example, it may be possible to limit the emission power level in one portion of the field of view 120 where safety is a consideration, while emitting higher power levels for other portions of the field of view 120 (thereby improving signal-to-noise ratio and detection range). This is related to eye safety, but may also be related to skin safety, safety of optical systems, safety of sensitive materials, etc. In another example, it may be possible to direct more energy to portions of the field of view 120 that will have greater use (e.g., areas of interest, targets at longer distances, low-reflective targets, etc.), while limiting illumination energy to other portions of the field of view 120 based on detection results from the same frame or previous frames. Note that the processing unit 108 may process detected signals from a single instantaneous field of view several times within a single scan frame time; for example, subsequent emissions may be determined after each emitted pulse or after multiple emitted pulses.

[0138] Figure 5BThree examples of emission schemes in a single frame time for field of view 120 are shown. Consistent with embodiments of the present disclosure, at least one processing unit 108 may use the obtained information to dynamically adjust an operating mode of LIDAR system 100 and / or determine values ​​of parameters of specific components of LIDAR system 100. The obtained information may be determined from processing data captured in field of view 120, or received (directly or indirectly) from host computer 210. Processing unit 108 may use the obtained information to determine a scanning scheme for scanning different portions of field of view 120. The obtained information may include current light conditions, current weather conditions, a current driving environment of a host vehicle, a current position of a host vehicle, a current trajectory of a host vehicle, a current topography of a roadway surrounding the host vehicle, or any other condition or object detectable by light reflections. In some embodiments, the determined scanning scheme may include at least one of (a) specifying a portion within the field of view 120 to be actively scanned as part of a scanning cycle, (b) a projection plan for the projection unit 102, which defines a light emission profile at different portions of the field of view 120; (c) a deflection plan for the scanning unit 104, which defines, for example, a deflection direction, a frequency, and specifies idle elements within a reflector array; and (d) a detection plan for the sensing unit 106, which defines a detector sensitivity or responsivity pattern.

[0139] In addition, the processing unit 108 may determine the scanning scheme at least in part by obtaining an identification of at least one region of interest within the field of view 120 and at least one region of no interest within the field of view 120. In some embodiments, the processing unit 108 may determine the scanning scheme at least in part by obtaining an identification of at least one region of high interest within the field of view 120 and at least one region of lower interest within the field of view 120. The identification of at least one region of interest within the field of view 120 may be determined, for example, by processing data captured in the field of view 120, data based on another sensor (e.g., camera, GPS), data received from the host 210 (directly or indirectly), or any combination of the above. In some embodiments, the identification of at least one region of interest may include an identification of a portion, area, segment, pixel, or object within the field of view 120 that is important to monitor. Examples of areas that may be identified as regions of interest may include crosswalks, moving objects, people, nearby vehicles, or any other environmental conditions or objects that may assist in vehicle navigation. Examples of areas that may be identified as regions of no interest (or lower interest) may be static (non-moving) distant buildings, skylines, areas above the horizon, and objects in the field of view. Upon obtaining an identification of at least one region of interest within the field of view 120, the processing unit 108 may determine a scanning scheme or change an existing scanning scheme. Further determining or changing light source parameters (as described above), the processing unit 108 may allocate detector resources based on the identification of at least one region of interest. In one example, to reduce noise, the processing unit 108 may activate detectors 410 for expected regions of interest and disable detectors 410 for expected regions of no interest. In another example, the processing unit 108 may change the detector sensitivity, for example, increasing the sensor sensitivity for long-range detection of low reflected power.

[0140] Figure 5BFigures AC in depict examples of different scanning schemes for scanning the field of view 120. Each square in the field of view 120 represents a different portion 122 associated with an instantaneous position of at least one light deflector 114. The legend 500 details the level of light flux represented by the fill pattern of the squares. Figure A depicts a first scanning scheme in which all portions have the same importance / priority and are assigned a default light flux. The first scanning scheme can be used during a startup phase or interleaved periodically with another scanning scheme to monitor the entire field of view for unexpected / new objects. In one example, the light source parameters in the first scanning scheme can be configured to generate light pulses with a constant amplitude. Figure B depicts a second scanning scheme in which a portion of the field of view 120 is assigned a high light flux, while the rest of the field of view 120 is assigned a default light flux and a low light flux. The least interesting portion of the field of view 120 can be assigned a low light flux. Figure C depicts a third scanning scheme in which compact vehicles and buses are identified in the field of view 120 (see outline). In this scanning scheme, the edges of vehicles and buses can be tracked with high power, and less light flux (or no light flux) can be allocated to the central masses of vehicles and buses. This light flux allocation enables more of the optical budget to be focused on the edges of the identified objects and less on their centers which are of lower importance.

[0141] Figure 5C The emission of light toward a field of view 120 during a single scanning cycle is shown. In the depicted example, the field of view 120 is represented by an 8×9 matrix, where each of the 72 cells corresponds to a discrete portion 122 associated with a different instantaneous position of at least one light deflector 114. In this exemplary scanning cycle, each portion includes one or more white dots representing the number of light pulses projected toward the portion, and some portions include black dots representing reflected light from the portion detected by the sensor 116. As shown, the field of view 120 is divided into three sectors: sector I on the right side of the field of view 120, sector II in the middle of the field of view 120, and sector III on the left side of the field of view 120. In this exemplary scanning cycle, sector I is initially assigned a single light pulse per portion; sector II, previously identified as a region of interest, is initially assigned three light pulses per portion; and sector III is initially assigned two light pulses per portion. As also shown, a scan of the field of view 120 reveals four objects 208: two free-form objects in the near field (e.g., between 5 meters and 50 meters), a rounded square object in the mid-field (e.g., between 50 meters and 150 meters), and a triangular object in the far field (e.g., between 150 meters and 500 meters). Figure 5CThe discussion uses the number of pulses as an example of light flux distribution, but it should be noted that the light flux distribution to different parts of the field of view can also be implemented in other ways, such as: pulse duration, pulse angle dispersion, wavelength, instantaneous power, photon density at different distances from the light source 112, average power, pulse power intensity, pulse width, pulse repetition rate, pulse sequence, pulse duty cycle, wavelength, phase, polarization, etc. Figure 5C The illustration of light emission as a single scanning cycle in FIG. 1 illustrates different capabilities of LIDAR system 100. In a first embodiment, processor 118 is configured to use two light pulses to detect a first object (e.g., a rounded square object) at a first distance, and to use three light pulses to detect a second object (e.g., a triangular object) at a second distance greater than the first distance. Fig.11 -13 describe this embodiment in more detail. In a second embodiment, the processor 118 is configured to allocate more light to the portion of the field of view where the region of interest is identified. Specifically, in this example, sector II is identified as the region of interest, and therefore it is allocated three light pulses, while the rest of the field of view 120 is allocated two or fewer light pulses. This embodiment is described in more detail below with reference to Figures 20-22. In a third embodiment, the processor 118 is configured so that only a single light pulse is directed toward Figure 5C The light source 112 is controlled in such a manner that the portions B1, B2, and C1 in the field of view 120 are projected (even though they are part of sector III that was originally allocated two light pulses per portion). This occurs because the processing unit 108 detects an object in the near field based on the first light pulse. This embodiment is described in more detail below with reference to Figures 23-25. The allocation of less than the maximum pulse amount may also be the result of other considerations. For example, in at least some areas, the detection of an object at a first distance (e.g., a near-field object) can result in a reduction in the total amount of light emitted to that portion of the field of view 120. This embodiment is described in more detail below with reference to Figures 14-16. Other reasons for determining the power allocation to different portions are discussed below with respect to Figures 29-31, Figures 53-55, and Figures 50-52.

[0142] Additional details and examples regarding the different components of the LIDAR system 100 and their associated functionality are included in U.S. patent application No. 15 / 391,916 filed by applicant on December 28, 2016; U.S. patent application No. 15 / 393,749 filed by applicant on December 29, 2016; U.S. patent application No. 15 / 393,285 filed by applicant on December 29, 2016; and U.S. patent application No. 15 / 393,593 filed by applicant on December 29, 2016, the entire contents of which are incorporated herein by reference.

[0143] Example Implementation: Vehicle

[0144] Figure 6A-6C An implementation of a LIDAR system 100 in a vehicle (e.g., vehicle 110) is shown. Any aspects of the LIDAR system 100 described above or below may be incorporated into the vehicle 110 to provide a range-sensing vehicle. Specifically, in this example, the LIDAR system 100 integrates multiple scanning units 104 and potentially multiple projection units 102 into a single vehicle. In one embodiment, a vehicle may utilize such a LIDAR system to improve power, range, and accuracy in and outside of overlapping regions, as well as redundancy in sensitive portions of the FOV (e.g., the vehicle's forward moving direction). Fig. 6A As shown, the vehicle 110 may include a first processor 118A for controlling the scanning of the field of view 120A, a second processor 118B for controlling the scanning of the field of view 120B, and a third processor 118C for controlling the synchronization of scanning the two fields of view. In one example, the processor 118C may be a vehicle controller and may have a shared interface between the first processor 118A and the second processor 118B. The shared interface may enable data exchange at an intermediate processing level and synchronization of scanning of the combined fields of view to form an overlap in space in time and / or space. In one embodiment, the data exchanged using the shared interface may be: (a) the flight time of a received signal associated with pixels in and / or near the overlapping fields of view; (b) the laser steering position status; (c) the detection status of an object in the field of view.

[0145] Figure 6B An overlap region 600 between field of view 120A and field of view 120B is shown. In the depicted example, the overlap region is associated with 24 portions 122 from field of view 120A and 24 portions 122 from field of view 120B. Assuming the overlap region is defined and known by processors 118A and 118B, each processor can be designed to limit the amount of light emitted in the overlap region 600 in order to comply with eye safety restrictions across multiple source lights, or for other reasons such as maintaining an optical budget. In addition, processors 118A and 118B can avoid interference between light emitted by the two light sources through loose synchronization between scanning unit 104A and scanning unit 104B and / or by controlling laser transmission timing and / or detection circuit enable timing.

[0146] Figure 6C1 shows how the overlap region 600 between the field of view 120A and the field of view 120B can be used to increase the detection range of the vehicle 110. Consistent with the present disclosure, two or more light sources 112 that project their nominal light emissions into the overlap region can be used to increase the effective detection range. The term "detection range" can include the approximate distance from the vehicle 110 at which the LIDAR system 100 can clearly detect an object. In one embodiment, the maximum detection range of the LIDAR system 100 is about 300 meters, about 400 meters, or about 500 meters. For example, for a detection range of 200 meters, the LIDAR system 100 can detect an object located 200 meters (or less) from the vehicle 110 more than 95%, more than 99%, more than 99.5% of the time. Even when the reflectivity of the object may be less than 50% (e.g., less than 20%, less than 10%, or less than 5%). In addition, the LIDAR system 100 can have a false alarm rate of less than 1%. In one embodiment, light projected from two light sources that are juxtaposed in space in time and space can be utilized to improve SNR, and thus increase the service range and / or quality of objects located in the overlapping area. Processor 118C can extract high-level information from the reflected light in fields of view 120A and 120B. The term "extracting information" may include any process of identifying information associated with an object, individual, location, event, etc. in captured image data by any means known to a person of ordinary skill in the art. In addition, processors 118A and 118B can share high-level information, such as objects (road separators, backgrounds, pedestrians, vehicles, etc.) and motion vectors to enable each processor to alert peripheral areas that are about to become areas of interest. For example, a moving object in field of view 120A can be determined to be about to enter field of view 120B.

[0147] Example Implementation: Surveillance System

[0148] Fig.6D An embodiment of a LIDAR system 100 in a monitoring system is shown. As described above, the LIDAR system 100 may be fixed to a stationary object 650, which may include a motor or other mechanism for rotating the housing of the LIDAR system 100 to obtain a wider field of view. Alternatively, the monitoring system may include multiple LIDAR units. Fig.6D In the example depicted in , a monitoring system may use a single rotatable LIDAR system 100 to obtain 3D data representing field of view 120 and process the 3D data to detect people 652 , vehicles 654 , environmental changes, or any other form of safety-critical data.

[0149] Consistent with some embodiments of the present disclosure, 3D data may be analyzed to monitor retail business processes. In one embodiment, 3D data may be used for retail business processes involving physical security (e.g., detection of: intrusions within a retail facility, vandalism within or around a retail facility, unauthorized access to secure areas, and suspicious behavior around cars in parking lots). In another embodiment, 3D data may be used for public safety (e.g., detection of: people slipping and falling onto store property, hazardous liquid spills or obstructions on store floors, assaults or kidnappings in store parking lots, obstructions to fire exits, and crowding in store areas or outside stores). In another embodiment, 3D data may be used for business intelligence data collection (e.g., tracking people passing through store areas to determine, for example, how many people pass by, where they stay, how long they stay, and how their shopping habits compare to their purchasing habits).

[0150] Consistent with other embodiments of the present disclosure, 3D data can be analyzed and used for traffic enforcement. Specifically, 3D data can be used to identify vehicles that travel over the legal speed limit or some other road legal requirements. In one example, the LIDAR system 100 can be used to detect vehicles that cross a stop line or a designated parking spot when a red traffic light is displayed. In another example, the LIDAR system 100 can be used to identify vehicles traveling in lanes reserved for public transportation. In yet another example, the LIDAR system 100 can be used to identify vehicles turning in an intersection where a specific turn is prohibited when the light is red.

[0151] Note that while examples of various disclosed embodiments have been described above and below with respect to a control unit that controls the scanning of a deflector, the various features of the disclosed embodiments are not limited to such a system. Rather, the techniques for distributing light to various portions of the LIDAR FOV may be applicable to types of light-based sensing systems (LIDAR or otherwise) that may desire or need to direct different amounts of light to different portions of the field of view. In some cases, such light distribution techniques may positively impact detection capabilities, as described herein, but may also yield other advantages.

[0152] It should also be noted that various portions of the present disclosure and claims may use terms such as "first," "second," "third," etc. to refer to various components or portions of components (e.g., light sources, sensors, sensor pixels, field of view portions, field of view pixels, etc.). These terms are used solely to facilitate the description of the various disclosed embodiments and are not intended to be limiting or to indicate any necessary correlation to similarly named elements or components in other embodiments. For example, a characteristic described as being associated with a "first sensor" in one embodiment described in one portion of the present disclosure may or may not be associated with the "first sensor" of a different embodiment described in a different portion of the present disclosure.

[0153] Note that the LIDAR system 100 or any of its components may be used with any of the specific embodiments and methods disclosed below. However, the specific embodiments and methods disclosed below are not necessarily limited to the LIDAR system 100, and may potentially be implemented in or by other systems (such as, but not limited to, other LIDAR systems, other electro-optical systems, other optical systems, etc., whichever is applicable). Moreover, while the system 100 is described with respect to an exemplary vehicle-based LIDAR platform, the system 100, any of its components, and any of the processes described herein may be applicable to LIDAR systems disposed on other platform types. Likewise, the embodiments and processes disclosed below may be implemented on or by a LIDAR system (or other systems such as other electro-optical systems), the LIDAR system being mounted on a system disposed on a platform other than a vehicle, or even regardless of any particular platform.

[0154] Example implementation: Using a LIDAR sensor as a camera to update a point cloud

[0155] Object detection ambiguity is a problem for LIDAR systems and other systems related to driver assistance and autonomous vehicles. The ability to accurately sense and interpret the surrounding environment (including identifying obstacles, hazards, objects, and other physical parameters that may affect vehicle navigation) is important for the safe operation of the vehicle platform. Challenges in object detection and identification in LIDAR systems may be caused by effects such as crosstalk, blooming, etc. For example, blooming can be caused by reflections of high-intensity signals (e.g., from highly reflective and / or nearby objects). Such reflected signals can distort the apparent contours or boundary edges of the detected objects (based on the sensor output), which can result in determining that the detected objects are larger than their actual size or have a shape different from their actual shape. Therefore, solutions that provide enhanced detection capabilities are needed in LIDAR systems, especially in the presence of conditions that cause crosstalk, blooming, or other conditions that may adversely affect the ability of the LIDAR system to accurately detect and / or identify / characterize objects.

[0156] Various types of LIDAR systems may be used in conjunction with the presently disclosed embodiments. Details related to at least some such LIDAR systems are provided in other portions of this disclosure. In one embodiment, a LIDAR system such as 100 ( Figure 1A ) includes a projection unit 102 and at least one sensing unit 106, the projection unit 102 is used to project laser light into a field of view 120, and the at least one sensing unit 106 receives laser return signals reflected from objects in the field of view 120. In response to an output signal provided by the sensing unit 106 in response to receiving the laser return signal, the processing unit 108 of the LIDAR system 100 can generate a point cloud, where each point in the point cloud represents a data point associated with a particular spatial location in the field of view of the LIDAR system and a range (distance) value associated with the spatial location. However, in some cases, the point cloud generated based on the sensor output signal may include certain inaccuracies or distortions with respect to the detected objects. For example, a highly reflective object can saturate one or more detector pixels, and the charge from the saturated pixels that is transferred to surrounding pixels can make objects in the point cloud appear larger than they are.

[0157] To address these issues, the processing unit 108 may be configured to acquire at least one image representing the environment of the LIDAR system. For example, the acquired image may constitute an image generated based on the acquisition of radiation other than the laser projected by the LIDAR system (e.g., an ambient light image). The acquired image may be generated by various sensors included in the LIDAR system. In some embodiments, the LIDAR time-of-flight (ToF) sensor within the sensing unit 106 (i.e., the same sensor (s) used to detect laser reflections from objects in the environment of the LIDAR system) may be used not only to generate a LIDAR point cloud, but also to generate an image based on ambient light (or other radiation sources other than the LIDAR laser). Based on the analysis of the acquired image, the processing unit may determine whether there is an inaccuracy relative to one or more generated point clouds. In addition, the processing unit may be configured to adjust or modify certain aspects of the generated point cloud, the area of ​​the generated point cloud, etc. based on the analysis of the acquired image.

[0158] The processing unit 108 may use one or more sensors associated with the LIDAR system to acquire an image representing the environment of the LIDAR system. For example, in some embodiments, the processing unit 108 may generate an image of at least a portion 122 of the field of view 120 using non-laser light incident on the sensing unit 106. The non-laser light may include ambient light reflected from objects in the environment of the LIDAR system 100 (e.g., from light sources in the environment and external to the LIDAR system 100, such as sunlight, vehicle headlights, street lighting, etc.). In some cases, the non-laser light may be emitted by a light source included as a component of the LIDAR system 100 (e.g., an infrared light source, a flash light source, an infrared flash light source, etc.).

[0159] Although the sensing unit 106 can be configured to primarily detect reflected laser signals of a particular wavelength, the sensing unit 106 can also include at least some sensitivity to light of other wavelengths. As a result, the sensing unit 106 can be used to generate images based on light acquired from the environment of the LIDAR system. The processing unit 108 (as described above) can analyze at least one of those images relative to one or more generated point clouds to determine, for example, an inconsistency between an object representation in one or more point clouds and a corresponding object representation in at least one acquired image. Through this comparison, the LIDAR system 100 can detect artifacts, distortions, or other types of inconsistencies in the point cloud information. In addition, in response to determining that the point cloud representation of the detected object may include at least one inconsistency, the generated point cloud can be corrected (based on information associated with (one or more) generated images) to resolve inconsistencies or even ambiguities related to the representation of the object in the detected point cloud. In other cases, ghost objects or other object representations included in the point cloud that do not correspond to objects actually present in the LIDAR system environment can be detected, and appropriate corrections to (one or more) point clouds can be generated. Such corrections may increase the accuracy of the point cloud generated by the LIDAR system, and in turn may improve the performance of systems that rely on the generated point cloud for operation (e.g., autonomous driving systems, navigation systems, collision avoidance systems, advanced driver assistance systems, etc.).

[0160] Figure 7 An example method 700 for resolving object detection ambiguities or defects / artifacts consistent with the present disclosure is depicted. The method 700 may be performed by at least one processor (such as a processor of the LIDAR system 100 (e.g., Figure 1A At least one processor 118 of the processing unit 108 in the example of Figure 2A Note that the method 700 may be implemented by any type of LIDAR system, not necessarily by the LIDAR system 100.

[0161] At step 702, at least one processor causes a laser light source to project laser light toward a field of view of a LIDAR system. Figure 2E As depicted in , the processor 118 of the processing unit 108 can be configured to cause the projection unit 102 to project light toward the field of view. The processing unit 108 can be configured to determine a projection plan for the projection unit 102, the projection plan defining a light emission distribution at different portions of the field of view. The light source can be a laser (such as a solid-state laser, a laser diode, a high-power laser) or an alternative light source (such as a light source based on a light emitting diode (LED)). In some embodiments, the at least one laser light source can include a single laser source. For example, as Figure 2AAs shown, the light source (e.g., projection unit 102) may include a single light source 112. In other cases, the light source may generate multiple laser beams for projection toward the field of view of the LIDAR system. Multiple laser sources (e.g., laser arrays) may be included in the LIDAR system for generating multiple laser beams.

[0162] Figure 2E One such example is shown in FIG. 1 where the LIDAR system includes an array of laser sources. Specifically, Figure 2E As shown, the light source (e.g., projection unit 102) may include an array of light sources 112A-112F. In some embodiments, the light source array may include a linear array of light sources controlled by processor 118 to sequentially project collimated laser beams. In some embodiments, some or all of at least one light source 112 may project light simultaneously. For example, processor 118 may cause the light source array to project light beams from multiple light sources 112 simultaneously.

[0163] In some embodiments, the LIDAR system may further include at least one deflector configured to project laser light into the field of view of the LIDAR system. Figure 2B As depicted in , the deflector 114 is configured to direct laser light 204 projected by the light source 112 of the projection unit 102 toward the field of view 120 for scanning. In some aspects, the scanning scheme may include at least one of: specifying a portion within the field of view 120 to be actively scanned as part of a scanning cycle, and / or a deflection plan of the scanning unit 104 that defines, for example, a deflection direction, a frequency, and a specified idle element within the reflector array. A sensor such as the sensor 116 is configured to receive light reflected from an object in the field of view 120 to generate an output related to the received light reflections. In some embodiments, as Figure 2A As depicted in FIG. 1 , at least one deflector may include a first light deflector 114A for outgoing light and a second light deflector 114B for incoming light. The deflectors may be part of a bistatic configuration of the LIDAR system, where projected light leaving the LIDAR system and reflected light entering the LIDAR system traverse substantially different optical paths.

[0164] In some embodiments, at least one deflector may also be configured to receive a laser return signal reflected from an object in the field of view. Figure 2BAs depicted in , the deflector 114 can be configured to direct reflected light from objects in the field of view toward the sensor 116. The deflector can be part of a monostatic configuration of the LIDAR system, in which projected light exiting the LIDAR system and reflected light entering the LIDAR system traverse substantially similar optical paths. In step 704, at least one processor can receive or generate a point cloud including distance information relative to objects in the field of view of the LIDAR system based on an output signal generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view of the LIDAR system. The point cloud can be associated with multiple data points, and each data point can include a spatial location relative to the LIDAR field of view and a distance or range value to a surface at the spatial location from which at least one reflection signal was received. As an example, Fig. 8A An exemplary representation of a point cloud of a scene determined from the output of the LIDAR system 100 is shown. The point cloud is generated based on output signals generated by at least one photosensitive detector, which may include a sensing unit 106 including a sensor 116 including a plurality of detectors 410 (also referred to as "sensor pixels"), such as Figure 4B As shown. The measurements from each detector 410 enable the flight time from the light pulse emission event to the reception event to be determined and the intensity of the received photons to be determined. In addition, the spatial position of each point on the point cloud is determined based on the instantaneous position of the scanning unit 104 at the time between the light pulse emission and reception. Each measurement signal is added to the point cloud according to its determined position and provides information about the objects in the field of view (such as the distance to the various parts of the detected objects and the direction relative to the LIDAR field of view). The LIDAR system can generate a point cloud model of the environment around the vehicle 110 by generating multiple point cloud entries from the detected reflections of multiple scanning cycles of the field of view.

[0165] At step 706, at least one processor may receive, or cause acquisition or generation of, an image representing at least a portion of a field of view of the LIDAR system based on an output signal generated by at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector. For example, using non-laser light (e.g., sunlight) incident on the at least one photosensitive detector, the processor may be configured to receive or generate an ambient light image (e.g., a noise image) or a sequence of continuous ambient light images (i.e., a video) of at least a portion of the field of view of the LIDAR system. Figure 8B An exemplary image of a scene determined from the output of the LIDAR system 100 is shown, where the scene is Fig. 8AIt should be noted that in some cases, at least one dedicated imaging sensor (e.g., a near infrared (nIR) sensor, a single wavelength sensor, etc.) may be included and aligned with the sensing unit 106 of the LIDAR system to provide such as Figure 8B The image shown. In this case, the sensing unit 106 may include a wavelength sensitive beam splitter or appropriate filters on each sensor to filter out unwanted radiation (i.e., noise) and direct the unwanted radiation to at least one dedicated imaging sensor. Using the output(s) of the sensing unit of the LIDAR system to generate a point cloud representation of the environment and an ambient light image representing the environment may reduce the cost and complexity associated with the LIDAR system because few or no additional components would be required.

[0166] In other cases, the host vehicle or the host vehicle's environment may include a camera, from which captured images may be acquired and compared to the point cloud generated by the LIDAR system. However, such a system may increase complexity and cost. For example, adding components such as a camera may increase the cost of the LIDAR system. In addition, synchronizing the images acquired from the camera device with the point cloud generated by the LIDAR system in time may increase complexity.

[0167] Where the signals acquired for generating both the ambient light image and the point cloud data are acquired from the same sensor, the point cloud data and the ambient light image are registered without requiring transformation or synchronization between multiple sensors.

[0168] The ambient light image may be generated or acquired at any suitable time relative to the generation of the point cloud by the LIDAR system and / or relative to the capture of the LIDAR light reflections. In some embodiments, at least one processor of the LIDAR system may be configured to generate the ambient light image after a predetermined number of laser scans of the field of view. For example, in some cases, at least one processor may acquire the ambient light image based on the output of sensor 116 generated after each laser scan of the LIDAR system field of view. That is, at least one processor may be configured such that the ambient light image is generated between the capture of laser frames of the field of view (e.g., a full scan of the LIDAR field of view).

[0169] However, in other cases, the ambient light image may be received or generated based on sensed radiation received at any time other than during the time interval when the LIDAR sensor receives reflected LIDAR laser light. Fig.9A and 9B A graph showing the detection of a tracking signal including laser reflections (e.g., sensed by a light sensor) and ambient light readings (i.e., noise) is shown. The light sensor may include a SiPM detector. Fig.9A As shown, a sequence of light pulses P1, P2, P3, and P4 may be emitted from at least one laser light source, and laser reflections P1, P2, P3, and P4 may be detected by at least one sensor of the LIDAR system at different times (e.g., time t1), depending on the distance to the surface from which pulses P1, P2, P3, and P4 were reflected. Although a sequence of four light pulses is shown, LIDAR operation may involve a trace that exhibits thousands of reflected laser pulses. During any time period when the at least one sensor does not detect a LIDAR laser reflection, ambient light readings (i.e., noise signals), such as N1, N2, N3, N4, and N5, may be acquired and processed, as shown in FIG. Fig. 9B As shown (e.g., by sampling the output of at least one sensor). For example, the processor can process the tracking signal to remove the reflected signals R1, R2, R3, and R4 (e.g., based on signal thresholds, pulse width determinations, determination of reflective patterns, etc.). The non-reflected light signals (e.g., N1, N2, N3, N4, and N5) can be summed or integrated to generate an ambient / noise signal image. For example, each ambient / noise signal acquisition (e.g., N1, N2, N3, etc.) can represent a single ambient light field of view pixel. Sampling and processing of multiple ambient / noise signals can be used to generate an ambient light image. In some cases, an ambient light / noise image (or portion of such an image) can be generated on a running basis as a LIDAR point cloud representing a field of view capture (e.g., a full scan of the LIDAR field of view) is generated. Thus, the generated ambient light / noise image can correspond in time to the corresponding generated point cloud, which can eliminate the need for a separate time correlation task, which may be required when a dedicated ambient light sensor separate from the LIDAR laser detector is used to generate the ambient light / noise image.

[0170] In some cases, noise capture time periods N1, N2, N3, etc. can be established based on the emission times of pulses P1, P2, P3, etc. and based on the expected flight times associated with those pulses and their associated reflections. For example, during any time period or time window when no laser pulse reflection is expected at the detector, the detector collects ambient light signals representing the environment of the LIDAR system. In another example, a single continuous time period (e.g., N1) before or after the expected laser pulse reflection is received can be a noise capture time period. Since the ambient light readings are captured during the pixel time of the TOF readings, rather than during a dedicated time period, there is no time loss to the detector availability. During the scanning process of the FOV, the ambient light pixel values ​​can be saved. The collection of ambient light pixels can be saved to an ambient light image in an array format (e.g., a two-dimensional array) or an image format. For example, the position of the ambient light pixel in the collection can be determined by the scanning position when the ambient light pixel is generated.

[0171] In some embodiments, a LIDAR system may include at least two photosensitive detectors that share at least one optical path segment. Fig. 10A and 10B is an exemplary diagram showing the optical path of a received optical signal. Fig. 10A and 10BAs depicted in FIG. 1 , the LIDAR system may include sensor 116a, sensor 116b, filter 1010, filter 1012, window 1014, and filter 1016. Sensor 116a may include a silicon photomultiplier (SiPM) infrared sensor configured to detect light of a specific wavelength (e.g., 905nm (infrared)), and sensor 116b may be a complementary metal oxide semiconductor (CMOS) sensor or a single wavelength sensor configured to detect light of a wavelength different from the SiPM wavelength (e.g., 690nm (red)), or vice versa. In some embodiments, when sensor 116a is configured as a time of flight (ToF) sensor, filter 1010 may be configured to filter out all wavelengths except signals within a target wavelength range (e.g., 905+ / -20nm) from the received reflected laser signal. Additionally or alternatively, when sensor 116b is configured as an ambient light sensor, light deflected by filter 1010 toward sensor 116b, such as red light (e.g., 690+ / -20nm), may be further filtered by filter 1016 to pass only light of the wavelength that sensor 116b is configured to detect. Additionally or alternatively, window 1014, which may be a LIDAR window, and filter 1012 may be configured to filter light entering the system and passing through scanning unit 104 to selectively remove certain wavelengths of light and, thus, minimize stray light in the LIDAR system. By implementing detectors positioned on the same optical path within the LIDAR system, the sensors may detect light of different wavelengths separately but simultaneously. These different sensors do not require complex synchronization or transformation, thereby reducing the complexity of spatial and temporal alignment that may be required to synchronize with sensors external to the LIDAR system.

[0172] Using sensors 116a and 116b that are each sensitive to different wavelengths or wavelength ranges can provide several potential advantages. For example, as described above, sensor 116a can be configured as a time-of-flight sensor that is sensitive to the laser wavelength (e.g., 905nm) emitted by the LIDAR light source. Therefore, the output from sensor 116a can be used to generate a LIDAR point cloud that includes distance information relative to objects at various positions within the LIDAR field of view. Sensor 116b can be sensitive to another wavelength (e.g., a red wavelength of 690nm, another wavelength in the visible spectrum, etc.). The output of sensor 116b can be used to identify various types of objects in the LIDAR field of view. For example, the output from sensor 116b can indicate whether an emitter or reflector of red visible light exists within or overlaps a certain field of view position / pixel. This information can be used to identify certain objects of interest associated with the emission or reflection of red light (e.g., traffic lights, traffic signs, vehicle brake lights, etc.).

[0173] This information can be layered with the point cloud information generated by the LIDAR system. For example, (e.g., based on the output of sensor 116a), for each pixel of the LIDAR field of view, the LIDAR system can generate a point cloud point indicating the distance to the object or portion of the object located at the FOV pixel location. Similarly, (e.g., based on the output of sensor 116b), for each pixel of the LIDAR field of view, the LIDAR system can generate an indicator of whether the LIDAR FOV pixel is associated with emission or reflection of red light. The measurements from sensor 116b can also be used to generate an ambient light image.

[0174] The processor 118 may also be configured to control various aspects of the acquired image based on the output signal generated by the sensor 116. For example, in some embodiments, at least one processor may be configured to generate an image using an exposure time determined or varied according to the field of view pixels. The exposure time should be less than the total pixel time defined by the correlation between the scanner speed and the instantaneous field of view. In some embodiments, the processor may be configured to control the light integration time associated with each field of view pixel individually. This may enable the system, for example, to devote imaging and processing resources to certain regions of interest in the field of view. Such regions of interest may include regions where certain detected objects are located, particularly those detected objects for which inconsistencies, distortions, etc. are detected relative to their point cloud representations, or objects detected with low certainty. In this manner, the exposure time of a first pixel in a plurality of field of view pixels may be greater than the exposure time of a second field of view pixel in a plurality of FOV pixels. For example, the exposure time of the first pixel may be 1.25 times, 1.5 times, 2 times, 5 times, or 10 times greater than the exposure time associated with a second pixel different from the first pixel. In this case, the processor can be configured to normalize the output signal generated for each pixel based on the exposure time. The exposure time can be adaptive depending on the pixel ambient light level. For example, in low ambient light conditions, a longer exposure time may be advantageous. Depending on the measured ambient light level of the area of ​​the FOV, the exposure time can vary between pixels.

[0175] Other aspects associated with the acquisition of ambient light images may also be controlled. For example, in some embodiments, at least one processor may also be configured to selectively generate an ambient light image (representing at least a portion of the field of view of the LIDAR system) based on one or more sensed environmental conditions. Environmental conditions (such as rain, fog, glare, snow, etc.) may hinder the ability of the LIDAR system to accurately detect objects in the field of view. For example, these conditions may cause scattering of the projected laser and / or reflected laser beams, which may interfere with the ability of the LIDAR system to detect objects or certain object features. Such conditions may also cause distortion of the point cloud representation of the detected objects. Therefore, in the event that such environmental conditions are detected, the processor 118 may acquire one or more ambient light images for use in coordinating the point cloud representation of at least some of the detected objects. When such environmental conditions are not detected, the processor may abandon the generation of an ambient light / noise image.

[0176] Other environmental conditions may also be used as control inputs for capturing ambient light images. For example, at least one processor may be configured to switch to a specific mode (such as night mode) depending on the ambient light conditions. This may include widening the wavelength band of the tunable filter to let more light into the sensing unit, or triggering a threshold light level below which the ambient light image is not captured. In some embodiments, the night mode may involve adjusting the exposure time for collecting the ambient light signal for each pixel. In particular, in low ambient light scenes, the exposure time may be increased, while in high ambient light scenes, the exposure time may be reduced. In some embodiments, at least one processor may be configured to abandon generating an ambient light image when the sensed ambient light level is below a predetermined threshold. The typical lux illuminance of ambient daylight is 10,000 to 25,000. However, the typical lux illuminance of light with moonlight at night is 1, and is less than 0.01 in the absence of moonlight. Therefore, when the sensed ambient light level is less than a predetermined threshold, at least one processor may abandon generating an ambient light image. Of course, the sensor 116 may be sensitive to light outside the visible range. For example, sensor 116 may be sensitive to IR ambient light and may generate an output signal in response to incident IR ambient light sufficient to generate an IR ambient light image. However, similar to the response to visible light, if the IR ambient light level drops below a predetermined threshold, processor 118 may abandon generating an ambient light image.

[0177] The generated ambient light image(s) may be compared to the generated point cloud(s), and based on the comparison, detected inconsistencies between the ambient light image(s) and the point cloud(s) may be corrected. Figure 7As described in the example process of FIG. 7 , at step 708 , at least one processor can compare the point cloud to the image to determine whether there is an inconsistency between the representation in the point cloud and the corresponding representation in the image of the object from the field of view.

[0178] In some embodiments utilizing two separate sensors to generate a point cloud and an ambient light image, the point cloud and corresponding representations of detected objects in the acquired image may not correspond in time despite being associated with a common object in the environment of the LIDAR system. That is, a point cloud (or data used to generate a point cloud) representing a certain detected object may be generated before or after an image including a representation of the same detected object is acquired. Given the difference in timestamps between the generated point cloud and the acquired image and potential relative motion between the detected object and the LIDAR system, a detected object may occupy a larger or smaller portion of the generated point cloud compared to the acquired image.

[0179] In order to implement two separate sensors, synchronization is required to compare the generated point cloud and ambient light image. Because the timestamps associated with the point cloud generation and image acquisition are known, and because the self-motion of the LIDAR system (e.g., the self-motion of the vehicle on which the LIDAR system is deployed) can also be determined, the processor 118 can take into account the self-motion of the LIDAR system when comparing the generated point cloud and the acquired image. For example, the processor 118 or any other processor associated with the LIDAR system or the host vehicle can determine the self-motion of the host vehicle / LIDAR system based on outputs from one or more sensors (such as a GPS system, a speedometer, an inertial sensor, etc.). Based on such sensor outputs, the self-motion of the host vehicle / LIDAR system can be determined between the time when the point cloud data frame is collected and the time when the ambient light image is created. The effect of the determined self-motion can be subtracted from the point cloud or the ambient light image (whichever is later) so as to transform the ambient light image and the point cloud to a common timestamp. This transformation can facilitate comparison of representations of objects detected in the point cloud and the ambient light image.

[0180] Various types of comparisons may be performed with respect to representations of detected objects in a point cloud and representations of detected objects in an ambient light image. As an initial comparison, in some embodiments, comparison of a point cloud to an image may include correlating at least one object representation in the point cloud with a corresponding at least one object representation in the image. For example, in an approach utilizing two separate sensors, a point cloud and an image that have been transformed to a common timestamp may include representations of detected objects in the same or similar spatial locations relative to world coordinates. Alternatively, in some cases, such as where a LIDAR sensor is used to generate both a point cloud and an ambient light image, representations of detected objects in the point cloud and the ambient light image may be in the same or similar spatial locations relative to the LIDAR FOV, and thus no transformations similar to those required when using two separate sensors are required. In either case, knowing where the representations of detected objects should appear in the corresponding point cloud and ambient image can facilitate comparison of the detected object representations in each.

[0181] One or more image recognition techniques may be used to further implement the association of object representations detected in the point cloud and ambient light image. For example, in some cases, the processor 118 may be configured to access object-related classification information stored in one or more memories and, based on the stored information and the point cloud information, identify and classify specific objects (e.g., vehicles, pedestrians, buildings, traffic lights, traffic signs, etc.) in the field of view of the LIDAR system 100. Similar steps may be implemented to identify and classify objects represented by multiple pixels in one or more ambient light images. Similar identifications and / or classifications resulting from analysis of the point cloud and corresponding ambient light image may further increase the confidence level of accurate object identification / classification, particularly where those identifications / classifications are developed relative to areas where the point cloud / image is expected to represent commonly detected objects.

[0182] Once the processor 118 has correlated the representations of the detected object in both the point cloud and at least one ambient light image, those correlated representations may be analyzed to detect potential inconsistencies. For example, the comparison of the point cloud to the image may include a comparison of the size of the detected object representation in the point cloud with the size of the corresponding representation of the detected object in the ambient light image. In one example, where the point cloud and the ambient light image have been transformed to a common timestamp and separate sensors are used in generating the point cloud and the ambient light image, the detected object representations in the point cloud and the ambient light image will be expected to have the same or similar size. Differences in the size of the detected object representations may indicate the presence of a halo effect, etc., which may result in an artificially enlarged object representation in the generated point cloud. Additionally or alternatively, the comparison of the point cloud to the image may also include a comparison of the shape of the detected object representation in the point cloud with the shape of the corresponding representation of the detected object in the image. For example, the processor may be configured to perform edge detection techniques to find the edges of the object and compare the edges and shapes with object-related information stored in one or more memories. The processor may also be configured to perform a feature-based approach to find a match between object features and image features. If disparity in size and / or shape is detected between the point cloud and the detected object representation in the ambient light image, information related to the detected disparity may be used to correct one or more of the generated point clouds, as described below.

[0183] The point cloud data may be affected by the angle of incidence of the illumination beam and the object on which the beam is incident. The ambient light image is the sum of all light reflected from the object toward the sensor. The contours of the edges of the 3D objects in the point cloud data can be obtained (e.g., by obtaining a projection of the 3D object detected in the point cloud to a plane and using an edge detection algorithm (such as a contrast-based edge detection algorithm)). The contours in the image can be obtained (e.g., using an edge detection algorithm). The contours of the object representation obtained from the point cloud and from the image can be compared. Under certain light conditions, or for certain illumination angles, or for certain objects (e.g., objects with high reflectivity, such as retroreflectors), the noisy image can have better quality than the point cloud for shape determination, and vice versa. If a disparity in size and / or shape is detected between the point cloud and the contour of the detected object representation in the ambient light image, information related to the detected disparity can be used to correct one or more generated point clouds, as described below.

[0184] Other comparisons may also be employed. In some embodiments, the comparison of the point cloud with the ambient light image may include a comparison of the reflectivity of the detected object representation in the point cloud with the reflectivity of the corresponding representation of the detected object in the image. For example, the LIDAR system 100 may determine the reflectivity of a portion of the field of view, including the area of ​​the generated point cloud occupied by the detected object. The intensity or other characteristics of the reflected laser pulses in the area may be determined and normalized based on the determined distance to determine a reflectivity value associated with the detected object (or portion of the detected object). Similarly, the LIDAR system 100 may determine the reflectivity value of the detected object based on the ambient light image including the representation of the detected object. For example, the ambient light image may be analyzed to determine the brightness associated with the object detected in the image. The brightness may be normalized based on the exposure time or ambient light level at which the ambient light image was captured. In combination with range information, for example, the brightness may be used to estimate the surface reflectivity of an object or a particular portion of an object. The surface reflectivity information derived from the ambient light image may then be used to help detect the presence of artifacts (e.g., artifacts based on reflectivity) associated with the generated point cloud.

[0185] Various techniques can be used to compare the ambient light image and the detected object representations in the generated point cloud. For example, in some cases, a pixel-by-pixel analysis is performed relative to the ambient image or relative to the field of view pixels associated with the point cloud to identify detected object candidates, classify the detected object candidates based on size, shape, etc., and segment the image / point cloud based on detected objects of interest, etc. Various other image processing / computer vision techniques can also be used depending on the requirements of a particular application. In some embodiments, the comparison of the point cloud to the image can be performed by a trained neural network. For example, the trained neural network can include a convolutional neural network designed to perform tasks related to image analysis. Fig.11A block diagram 1100 is shown depicting an example embodiment including a trained neural network 1110 included in the processing unit 108. However, such a trained neural network may be implemented using one or more processors external to the processing unit 108 (e.g., a dedicated AI processor). Once the sensing unit 106 receives the return signal, the processing unit 108 uses the laser return signal and the non-laser return signal to generate a point cloud and an ambient light image, respectively. The generated image is input into the trained neural network 1110, which may be trained to identify multiple categories of objects, for example, based on a data set of images. The network may be trained to associate detected objects represented in the generated point cloud with corresponding detected objects represented in the ambient light image. In addition, the network may also be trained to identify differences in size, shape, reflectivity, etc. between the associated detected object representations in the point cloud and the ambient image. Thus, the training data may include the generated point cloud and one or more acquired ambient images provided as input to the trained neural network. The image may also include annotations to facilitate training, including data related to inconsistencies in the image or point cloud associated with the point cloud. Based on these inputs, the network may generate outputs including one or more of: identification of detected objects in the point cloud and / or ambient light image, classification of detected objects in the point cloud and / or ambient light image, identification or classification of individual pixels or points in the point cloud, information associating detected object representations or individual points in the point cloud with corresponding detected object representations in the ambient light image, and information indicating whether there is an inconsistency between the detected object representations in the point cloud and the ambient light image. The inconsistency may be identified with respect to at least one parameter associated with the detected object representations (size, shape, reflectivity, etc.).

[0186] For example, in some embodiments, the training data may include a collection of point clouds and images. The disclosed system may use one or more of the analyzed images to identify the shape of a representation of an object in the image. The disclosed system may also compare the results of the analysis (e.g., comparing the identified shape) to a distorted shape of the object included in the point cloud). Thus, using such techniques, the disclosed system may identify one or more inconsistencies between the point cloud and the image. For example, a point cloud may not include data for a particular object (e.g., a black car), however, an image may include an identifiable representation of the particular object.

[0187] Return to Figure 7 At step 710, at least one processor of the LIDAR system may adjust one or more aspects of the generated point cloud to provide an updated point cloud. Figure 7In an example of, based on one or more inconsistencies between representations in the point cloud and corresponding representations in an ambient light image of a detected object from a field of view, the detection is adjusted. In some embodiments, the adjustment of one or more aspects of the point cloud may include reconstructing the point cloud information to account for the inconsistencies between the representations in the detected point cloud and corresponding representations in the image of the object from the field of view. Various adjustments may be made to the point cloud (or a portion of the point cloud) based on the detected inconsistencies relative to the ambient light image. For example, the adjustment of one or more aspects of the generated point cloud may include removing points from or adding points to the generated point cloud. Additionally or alternatively, the adjustment of one or more aspects of the generated point cloud may include reducing the size of at least one object representation in the generated point cloud. Additionally or alternatively, the adjustment of one or more aspects of the generated point cloud may include changing the shape of at least one object representation in the generated point cloud. For example, such an adjustment may be made in response to an analysis of the detected object representation in the ambient light image indicating that the detected object representation in the generated point cloud is distorted, too large, too small, etc. In this case, the detected object representation in the point cloud may be adjusted so that it is consistent with the corresponding detected object representation in at least one ambient light image.

[0188] In another example, the adjustment of one or more aspects of the generated point cloud includes updating the confidence level of the points from the generated point cloud. The processor can be configured to determine at least one confidence level during a scanning cycle of collecting point cloud information. The processor can be configured to output a point cloud including confidence level values ​​of the points of the generated point cloud. The confidence level assigned to each of the one or more points of the point cloud can indicate the possibility that the point cloud model corresponds to a real-world object located at a corresponding position identified by the point cloud data point instead of a false detection. The determined confidence level can be used to detect objects in the scene, to classify objects in the scene, to understand the scene, to estimate the state of the LIDAR system 100, to estimate the environmental conditions of the LIDAR system, etc. The confidence level can be provided in various formats, such as high / low; high / medium / low, percentage, integer, fraction, etc. For example, due to various conditions (such as fog, rain, etc.) that affect the performance of the LIDAR system, the initial confidence level of the points of the generated point cloud can be assigned as "low". However, based on a comparison of the collected point cloud data (and, for example, detected object representations therein) with the detected object representations in the ambient light image, the confidence levels of points in the generated point cloud may be updated. Specifically, if, based on an analysis of the ambient light image, the processor determines that a point cloud data point that was initially classified as having a low confidence level is associated with a detected object represented in the ambient light image, the confidence level of that data point may be increased.

[0189] Comparison between the generated point cloud and the ambient light image can also be used to classify detected objects. For example, in some embodiments, adjustment of one or more aspects of the generated point cloud can include changing the classification represented by at least one object in the generated point cloud. Classifying an object can include identifying the object as a type of object, such as a car, a plant, a tree, a road, a pedestrian, a traffic sign, a road hazard, a building, a road, etc. The type of object classification can be determined based on the shape of the object, the determined spatial relationship, and / or the determined reflectivity, as represented by the generated point cloud data. However, analysis of one or more corresponding ambient light images may indicate that the initial object classification determined based on the point cloud data needs to be updated. Fig.12 A table showing the object classification process is shown. Fig.12 As shown in Table 1210 of the LIDAR system, objects 1202a, 1202b, 1202c, 1202d, and 1202e can be identified in the field of view of the LIDAR system and assigned initial classifications based on the generated point cloud data. For example, object 1202a can initially be classified as a tree, objects 1202b, 1202c, and 1202e are classified as cars, and 1202d is classified as a traffic sign. The initial classifications stored in Table 1210 can be stored in a database in one or more memories. After comparing the point cloud data with the acquired ambient light image (step 708), the comparison can reveal that object 1202a is actually a person. In response, the processor can change the classification of object 1202a from a tree to a person, as shown in Table 1220. The adjusted classifications stored in Table 1220 can also be stored in a database in one or more memories.

[0190] In some embodiments, when the LIDAR system records light associated with a first pixel of a plurality of pixels of a photosensitive detector at a second pixel (or pixels) of the photosensitive detector, an inconsistency between a detected object representation in a point cloud and a corresponding detected object representation in an ambient light image may arise. For example, a charge from the first pixel is generated in response to at least one laser return signal reflected from one or more objects in the field of view and incident on the first pixel. In one example, receipt of the at least one laser return signal causes saturation of the first pixel. As an example, Fig.13A Conceptual representation of the point cloud generated based on pixel saturation at the sensor. When highly reflective objects reflect high-intensity signals toward the LIDAR system, this reflection can produce effects such as haloing, where the highly reflective object appears distorted or stretched and appears to expand into its surroundings, such as Fig.13ABlooming occurs when the charge in a detector pixel exceeds a saturation level and the excess charge may be transferred to one or more adjacent detector pixels. In some embodiments, the second pixel (and potentially other pixels) is adjacent to the first pixel in at least one photosensitive detector. That is, those pixels adjacent to the saturated first pixel may be those pixels most likely to experience charge transfer associated with blooming. In this case, the size of the object representation can be adjusted based on the corresponding ambient light image (e.g., a noise image). For example, the LIDAR system can adjust the size of one or more object representations in the point cloud based on the size of one or more object representations in the ambient light image. Fig.13A and Fig. 13B In this case, object representations 1310b and 1312b correspond to 1310a and 1312a. The LIDAR system can determine Fig. 13B The size of the objects in the ambient light image 1310b and 1312b is smaller than Fig.13A The corresponding object representations in the point cloud of and thus reducing the size of object representations 1310a and 1312a.

[0191] Lost object (e.g. black vehicle)

[0192] In some embodiments, the object representation may be at least partially missing or obscured from the point cloud, but visible in the ambient image. For example, certain materials of the object may absorb light emitted by the LIDAR system. Black or dark objects (e.g., black vehicles) may absorb light at wavelengths such as 905nm, 940nm, etc. When such an object is in the FOV of the LIDAR system, there may be a "hole" or missing object in the point cloud where the black or dark object is located. This may be particularly difficult to detect if the background of the dark object is the sky (i.e., no background objects are partially detected but partially obscured by the object, indicating its presence). In contrast, due to the difference in ambient light levels between highly absorbing objects and other objects or the surrounding environment, black or dark objects can be easily distinguished in the ambient light image, resulting in high contrast in the ambient light image. The representation of a highly absorbing object in the ambient light image will have a low ambient light level. The LIDAR system can determine the presence of an object and other object parameters (such as shape) based on the presence of the object in the image and the absence of the corresponding object in the point cloud. For example, at step 708 of method 700, when the at least one processor compares the point cloud to the image to determine inconsistencies, missing objects (e.g., black vehicles, etc.) that are not identified in the point cloud may be identified in the ambient light image. At step 710, the at least one process may adjust one or more aspects of the generated point cloud based on the detection of the missing objects to provide an updated point cloud.

[0193] Ghost

[0194] Embodiments described herein include the following recognition: when repeated light emissions or pulses are used to illuminate the FOV portion 122, reflections of different light emissions may not be distinguishable from each other. Therefore, a reflection of a first emission from an object beyond the detection range of the LIDAR system 100 may return to the sensing unit 106 in the TOF associated with a second emission after the first emission (e.g., within a predefined time window in which the reflection of the second emission is expected to return to the sensing unit), and thus appear as a valid target at a closer distance than the actual distance. This representation of a long-distance target is also referred to as a ghost below, and may be caused, for example, by highly reflective targets (such as street signs, mirrors, and retroreflectors) located outside the maximum detection range of the LIDAR system. The maximum detection range of the LIDAR system can be limited by the maximum time window in which the LIDAR system can wait for a return pulse, and may depend on various factors (such as frame rate, computing power, illumination modulation (if any), etc.). In addition, in an example, in a scanning LIDAR system, the sensor can only stay on each pixel for a very short time before moving on to check another pixel. Therefore, the time to wait for each emission pulse to return as a reflection (if any) is limited. For example, reflections of a transmitted pulse from a target in the field of view that is up to about 300m away from the LIDAR system (taking into account the speed of light entering and exiting an object) can be detected within a 2μs time window from the start of the emission. Depending on the light emission intensity, light reflected from most targets beyond the preset maximum detection range may be reduced by the time the reflection reaches the LIDAR system. However, highly (very) reflective targets (e.g., street signs, license plates, etc.) reflect light pulses even at greater distances. Reflections from farther targets detected in a detection time slot corresponding to another pixel (or another pulse of the same pulse train) may result in false positive object detections ("ghosts").

[0195] When such objects are not identified as ghosts, but are treated like real targets, such ghosts may cause false detections by the LIDAR system, which in turn may cause errors in the client and / or the platform hosting the LIDAR system. For example, a falsely identified ghost may cause the vehicle 110 hosting the LIDAR system 100 to make incorrect decisions on the road.

[0196] In the case of suspected ghosting (i.e., an object may be detected and an incorrect distance is assigned to the detection, resulting in the object representation being incorrectly positioned in the point cloud), there may be a mechanism to determine from the point cloud that the object is a suspected ghost. The suspicion can be confirmed by correlating the object representation position in the point cloud with the object representation position in the ambient light image. If conflicting information about the position is derived from the comparison, the object (or part of the object, or a single point) position can be marked with low confidence. In response, the processor can change the position of certain points in the point cloud (suspected to be ghost points) to be consistent with the correct physical position of the object in the FOV. The ambient light image can be used to determine the position. The position can be further determined by using a longer distance position determined by the maximum detection range and a redundant time difference of the TOF measurement that causes a false "ghost" detection. In addition, after the object is detected (i.e., the point group in the point cloud is detected as an object), the object representation position can be changed to be consistent with the physical position of the object in the FOV.

[0197] Additionally, the LIDAR system or processor can determine suspected "ghost" detections by comparing the point cloud data and the ambient light image and finding inconsistencies in the position information between them. The LIDAR system can determine the corrected position of the object based on the inconsistency between the position of the object in the point cloud and the position of the object in the image, as well as other object parameters such as shape, size, or reflectivity.

[0198] Correct identification of “ghost” objects or detections can increase the range of a LIDAR system by using the object’s identified position in the ambient light image to confirm its position, thereby enabling identification of remote objects located beyond the maximum range (defined by TOF to maximum range).

[0199] Of course, while the point cloud can be adjusted in response to analysis of one or more corresponding ambient light images, such adjustments need not always be made when an inconsistency is detected between the detected object representations in the point cloud and the detected object representations in the ambient light image. For example, in some cases, the ambient light image may not be of sufficient quality (e.g., due to low light, glare, bright ambient sources, etc.) to warrant such an adjustment. Therefore, in some embodiments, at least one processor is further configured to forgo adjustments to one or more aspects of the generated point cloud comparison based on the sensed quality of the image. For example, the point cloud may be adjusted only when the image quality exceeds a predetermined threshold. In one example, in Fig.14A and Fig. 14B In, by Fig.14A The first ambient light image represented by Fig. 14B The second ambient light image shows a lower noise level. Fig.14A There may be sufficient image quality to determine inconsistencies and adjust the point cloud representation of an object (such as sign 1410), but Fig. 14B The image may not be of sufficient quality to detect such an object and therefore may not provide the information needed to determine inconsistencies and make adjustments to the point cloud. To account for noise, in some embodiments, a noise removal filter may be applied to the ambient light image before comparing the image to the point cloud.

[0200] It should be noted that while the above embodiments involve adjusting a point cloud representing an object in the FOV of a LIDAR system, objects identified within a point cloud or any data derived from a point cloud may be adjusted in a similar manner. Objects may be detected within a point cloud using any available technique. Detecting objects in a point cloud involves identifying groups of points in the point cloud that belong to the same discrete object in the FOV. Objects may be identified and classified. For example, an object may be classified as a car, truck, person, or traffic sign. In this way, an object map of all relevant objects in the point cloud may be generated. In the event of an inconsistency between an object in the point cloud and the data from the ambient light image, at least one parameter of the object may be adjusted to compensate for the inconsistency. For example, the position, size, shape, reflectivity, or presence of the object may be adjusted to compensate for these inconsistencies.

[0201] Fig.15 Another example method 1500 for resolving object detection ambiguities or defects / artifacts consistent with the present disclosure is depicted. Similar to method 700, method 1400 may be performed by at least one processor (such as a processor of LIDAR system 100 (e.g., Figure 1A At least one processor 118 of the processing unit 108 in the example of Figure 2A The method 1400 may be performed by the two processors 118 of the processing unit 108, etc.). Note that the method 1400 may be implemented by any type of LIDAR system, not necessarily by the LIDAR system 100. In a manner similar to steps 702 and 704 described above, at step 1502, at least one processor may receive or generate a point cloud including distance information relative to an object in the field of view of the LIDAR system based on an output signal generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view. The point cloud may be associated with a plurality of data points, and each data point may include a spatial position relative to the LIDAR field of view and a distance or range value to a surface at the spatial position from which at least one reflection signal was received.

[0202] At step 1504, the at least one processor may determine a first object in the point cloud by identifying a group of points in the field of view associated with the first object. For example, as described above, a feature extraction module may process the point cloud information to identify and classify a plurality of features representing a first object (e.g., a car, a tree, a person, or a road) in the environment. Features may be stored in any kind of data structure (e.g., raster, vector, 2D array, 1D array).

[0203] At step 1506, as described with respect to step 706, at least one processor may receive or generate an image representing at least a portion of the field of view based on an output signal generated by at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector. The at least one photosensitive detector may include a single photosensitive detector (e.g., a single photosensitive detector sensitive to infrared laser light and ambient infrared light). The single photosensitive detector may include a multi-channel detector or a SiPM detector. Alternatively, the at least one photosensitive detector may include a dedicated sensor for ambient light measurement that is sensitive to infrared wavelengths or visible light. (e.g., sensor 116b). Using non-laser light (e.g., sunlight) incident on the at least one photosensitive detector, the processor may also be configured to receive or generate an ambient light image (e.g., a noise image) or a sequence of continuous ambient light images (i.e., a video) of at least a portion of the field of view of the LIDAR system.

[0204] At step 1508, similar to step 708, at least one processor may compare the point cloud to the image to determine if there is an inconsistency between the representation in the point cloud and the corresponding representation in the image of the first object from the field of view. As described above, various types of comparisons may be performed with respect to the representation of the detected object in the point cloud and the representation of the detected object in the ambient light image. As an initial comparison, in some embodiments, the comparison of the point cloud to the image may include correlating at least one object representation in the point cloud with a corresponding at least one object representation in the image. For example, in an approach utilizing two separate sensors, the point cloud and the image may include representations of detected objects in the same or similar spatial locations relative to world coordinates. Alternatively, in some cases, such as where a LIDAR sensor is used to generate both the point cloud and the ambient light image, the representations of the detected objects in the point cloud and the ambient light image may be in the same or similar spatial locations relative to the LIDAR FOV, and thus no transformation is required as would be required when two separate sensors are used. In either case, knowing where the representation of the detected object should appear in the corresponding point cloud and ambient image may facilitate comparison of the representation of the detected object in each. In some cases, at least one processor may identify missing objects in the ambient light image that are not identified in the point cloud (eg, black vehicles, etc.).

[0205] At step 1510, at least one processor may adjust one or more aspects of the first object to provide an updated object map based on detecting one or more inconsistencies between representations in a point group associated with the first object and corresponding representations in an image of the first object from the field of view. As described above, for example, an object may be classified as a car, a truck, a person, or a traffic sign. In this way, an object map of all relevant objects in a point cloud may be generated, and then the object map of all relevant objects in the point cloud may be adjusted based on detecting one or more inconsistencies between representations in a point group associated with the object and corresponding representations in an image of the same object from the field of view. For example, the position, size, shape, reflectivity, or presence of the object may be adjusted to compensate for these inconsistencies. As another example, if a missing object (e.g., a black vehicle, etc.) is detected in an ambient light image but the missing object is not identified in the point cloud, at least one processor may provide an updated object map including a representation of the missing object.

[0206] The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art in view of the specification and practice of the disclosed embodiments. In addition, although various aspects of the disclosed embodiments are described as being stored in a memory, those skilled in the art will appreciate that these aspects may also be stored on other types of computer-readable media, such as secondary storage devices, e.g., a hard disk or CD ROM, or other forms of RAM or ROM, USB media, DVD, Blu-ray or other optical drive media.

[0207] Computer programs based on the written description and disclosed methods are within the skill of experienced developers. Various programs or program modules may be created using any technology known to those skilled in the art, or may be designed in conjunction with existing software. For example, program segments or program modules may be designed in or with the aid of .Net Framework, .Net Compact Framework (and related languages, such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML / AJAX combinations, XML, or HTML including Java applets.

[0208] In addition, although illustrative embodiments have been described herein, the scope of any and all embodiments with equivalent elements, modifications, omissions, combinations (e.g., combinations across aspects of various embodiments), adaptations and / or changes is understood by those skilled in the art based on this disclosure. The limitations in the claims are to be interpreted broadly based on the language employed in the claims and are not limited to the examples described in this specification or during the prosecution of this application. These examples should be interpreted as non-exclusive. In addition, the steps of the disclosed methods may be modified in any way, including by reordering steps and / or inserting or deleting steps. Therefore, the description and embodiments are to be considered illustrative only, with the true scope and spirit being indicated by the appended claims and the full scope of their equivalents.

Claims

1. A LIDAR system, comprising: at least one laser light source; at least one photosensitive detector; and At least one processor configured to: causing the at least one laser light source to project laser light toward the field of view of the LIDAR system; generating a point cloud including distance information relative to objects in the field of view of the LIDAR system based on output signals generated by the at least one photosensitive detector in response to received laser return signals reflected from objects in the field of view; generating an image representing at least a portion of the field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to detect if there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of an object from the field of view; as well as In response to detecting one or more inconsistencies between representations in the point cloud and corresponding representations in the image of the object from the field of view, one or more aspects of the generated point cloud are adjusted to provide an updated point cloud.

2. The LIDAR system according to claim 1, wherein: Adjusting one or more aspects of the generated point cloud may include removing points from the generated point cloud or adding points to the generated point cloud.

3. The LIDAR system according to claim 1, wherein: Adjusting one or more aspects of the generated point cloud includes updating confidence levels of points from the generated point cloud.

4. The LIDAR system according to claim 1, wherein: Adjusting one or more aspects of the generated point cloud includes changing a classification of at least one object representation in the generated point cloud.

5. The LIDAR system according to claim 1, wherein: Adjusting one or more aspects of the generated point cloud includes reducing a size of at least one object representation in the generated point cloud.

6. The LIDAR system of claim 1, wherein: Adjusting one or more aspects of the generated point cloud includes changing a shape of at least one object representation in the generated point cloud.

7. The LIDAR system of claim 1, wherein: Comparing the point cloud to the image includes correlating at least one object representation in the point cloud with a corresponding at least one object representation in the image.

8. The LIDAR system according to claim 7, wherein: The comparison of the point cloud and the image further includes a comparison of a size of the at least one object representation in the point cloud with a size of a corresponding representation of the at least one object in the image.

9. The LIDAR system according to claim 7, wherein: The comparing of the point cloud and the image further includes comparing a reflectivity of the at least one object representation in the point cloud with a reflectivity of a corresponding representation of the at least one object in the image.

10. The LIDAR system according to claim 7, wherein: The comparison of the point cloud and the image also includes a comparison of a shape of at least one representation of an object in the point cloud with a shape of a corresponding representation of the at least one object in the image.

11. The LIDAR system of claim 1, wherein: Comparing the point cloud to the image identifies objects in the image that were not identified in the point cloud.

12. The LIDAR system of claim 1, wherein: The comparison of the point cloud and the image is performed by a trained neural network.

13. The LIDAR system of claim 11, wherein: The neural network is trained based on a dataset of captured images and associated point clouds.

14. The LIDAR system of claim 1, wherein: When the LIDAR system records light associated with a first pixel of the plurality of pixels of the at least one photosensitive detector through at least a second pixel of the plurality of pixels of the at least one photosensitive detector, an inconsistency between a representation in the point cloud and a corresponding representation in the image of the object from the field of view is caused.

15. The LIDAR system of claim 14, wherein: The charge from the first pixel is generated by the first pixel in response to one or more of the laser return signals incident on the first pixel that are reflected from an object in the field of view.

16. The LIDAR system of claim 15, wherein: Receiving one or more of the laser return signals saturates the first pixel.

17. The LIDAR system of claim 15, wherein: The at least second pixel is adjacent to the first pixel in the at least one photosensitive detector.

18. The LIDAR system of claim 1, wherein: Adjusting one or more aspects of the point cloud includes reconstructing the point cloud information to account for inconsistencies between the detected representations in the point cloud and corresponding representations in the image of the object from the field of view.

19. The LIDAR system of claim 1, wherein: The at least one laser light source comprises an array of laser sources.

20. The LIDAR system of claim 1, wherein: The at least one processor is configured to cause the image to be generated between laser frame captures of the field of view.

21. The LIDAR system of claim 1 , further comprising at least one deflector configured to scan the field of view with the projected laser light, and wherein: The at least one processor is configured to generate the image after a predetermined number of scans of the field of view.

22. The LIDAR system of claim 1, wherein: The at least one processor is configured to generate the image using an exposure time determined per pixel of the field of view.

23. The LIDAR system of claim 22, wherein: The exposure time of the first field of view pixels is greater than the exposure time of the second field of view pixels.

24. The LIDAR system of claim 1, wherein the at least one processor is further configured to: The image representing at least a portion of the field of view of the LIDAR system is selectively generated based on one or more sensed environmental conditions.

25. The LIDAR system of claim 4, wherein: The at least one processor is further configured to When the sensed ambient light level is less than a predetermined threshold, generating the image is abandoned.

26. The LIDAR system of claim 1, wherein: The at least one processor is further configured to Adjusting one or more aspects of the generated point cloud comparison based on the sensed quality of the image is foregone.

27. The LIDAR system of claim 1, wherein: The at least one photosensitive detector includes a first detector sensitive to infrared light and a second detector sensitive to a wavelength of visible light, wherein an output from the first detector is used to generate the point cloud and an output from the second detector is used to indicate whether a point in the point cloud is associated with at least one object that emits or reflects the wavelength of visible light.

28. The LIDAR system of claim 27, wherein: The wavelength of the visible light is the wavelength of red light.

29. The LIDAR system of claim 27, wherein: The output of the second detector is used to identify a traffic light, a traffic sign, or a vehicle brake light in a portion of the field of view.

30. The LIDAR system of claim 1, wherein: The at least one photosensitive detector is a single photosensitive detector.

31. The LIDAR system of claim 1, wherein: An output signal responsive to a received laser return signal reflected from the object in the field of view and an output signal responsive to non-laser light incident on the at least one photosensitive detector are generated by a single photosensitive detector.

32. A method for detecting an object using a LIDAR system, the method comprising: causing at least one laser light source to project laser light toward the field of view of the LIDAR system; generating a point cloud including distance information relative to objects in the field of view of the LIDAR system based on output signals generated by at least one photosensitive detector in response to received laser return signals reflected from objects in the field of view; generating an image representing at least a portion of the field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to determine if there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of an object from the field of view; and Based on detecting one or more inconsistencies between representations in the point cloud and corresponding representations in the image of objects from the field of view, one or more aspects of the generated point cloud are adjusted to provide an updated point cloud.

33. A non-transitory computer-readable storage medium comprising stored instructions that, when executed by at least one processor, cause the at least one processor to perform a method for detecting an object using a LIDAR system, the method comprising: causing at least one laser light source to project laser light toward the field of view of the LIDAR system; generating a point cloud including distance information relative to objects in the field of view of the LIDAR system based on output signals generated by at least one photosensitive detector in response to received laser return signals reflected from objects in the field of view; generating an image representing at least a portion of the field of view of the LIDAR system based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to determine if there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of an object from the field of view; as well as Based on detecting one or more inconsistencies between representations in the point cloud and corresponding representations in the image of objects from the field of view, one or more aspects of the generated point cloud are adjusted to provide an updated point cloud.

34. A method for detecting an object, comprising: receiving a point cloud including distance information relative to an object in a field of view based on output signals generated by at least one photosensitive detector in response to a received laser return signal reflected from the object in the field of view; receiving an image representative of at least a portion of the field of view based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to detect if there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of an object from the field of view; as well as In response to detecting one or more inconsistencies between representations in the point cloud and corresponding representations in the image of the object from the field of view, one or more aspects of the generated point cloud are adjusted to provide an updated point cloud.

35. A non-transitory computer-readable storage medium comprising stored instructions which, when executed by at least one processor, cause the at least one processor to perform a method comprising: receiving a point cloud including distance information relative to an object in the field of view based on output signals generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; receiving an image representative of at least a portion of the field of view based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to detect if there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of an object from the field of view; as well as In response to detecting one or more inconsistencies between representations in the point cloud and corresponding representations in the image of the object from the field of view, one or more aspects of the generated point cloud are adjusted to provide an updated point cloud.

36. A method for detecting an object, the method comprising: receiving a point cloud including distance information relative to objects in the field of view based on output signals generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; determining a first object in the point cloud by identifying a group of points associated with the first object in the field of view; receiving an image representative of at least a portion of the field of view based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to determine if there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of the first object from the field of view; as well as Based on detecting one or more inconsistencies between representations in the point group associated with the first object and corresponding representations in the image of the first object from the field of view, one or more aspects of the first object are adjusted to provide an updated object map.

37. The method of claim 36, further comprising: The first object is classified as one of a car, a truck, a person, or a traffic sign.

38. The method of claim 36, wherein: The comparing of the point cloud to the image also includes comparing a size of the first object in the point cloud to a size of a corresponding representation of the first object in the image.

39. The method of claim 36, wherein: The comparing of the point cloud to the image further includes comparing a reflectivity of the first object in the point cloud to a reflectivity of a corresponding representation of the first object in the image.

40. The method of claim 36, wherein: The comparing of the point cloud to the image also includes comparing a shape of the first object in the point cloud to a shape of a corresponding representation of the first object in the image.

41. The method of claim 36, wherein: The comparison of the point cloud and the image is performed by a trained neural network.

42. The method according to claim 41, wherein: The neural network is trained on a dataset of labeled images to identify various categories of objects.

43. A non-transitory computer-readable storage medium comprising stored instructions which, when executed by at least one processor, cause the at least one processor to perform a method comprising: receiving a point cloud including distance information relative to objects in the field of view based on output signals generated by at least one photosensitive detector in response to a received laser return signal reflected from an object in the field of view; determining the first object in the point cloud by identifying a group of points associated with the first object in the field of view; receiving an image representative of at least a portion of the field of view based on an output signal generated by the at least one photosensitive detector in response to non-laser light incident on the at least one photosensitive detector; comparing the point cloud to the image to determine if there is an inconsistency between a representation in the point cloud and a corresponding representation in the image of the first object from the field of view; as well as Based on detecting one or more inconsistencies between representations in the point group associated with the first object and corresponding representations in the image of the first object from the field of view, one or more aspects of the first object are adjusted to provide an updated object map.

Images