Sensor device, lidar device, and control method thereof

Abstract

The LIDAR device disclosed in the embodiments may include a data collection unit that collects raw data; a pre-processing unit that removes noise from the collected data; a histogram circuit that converts an output of the pre-processing unit into a histogram; a buffer that buffers the output of the pre-processing unit; a sub-histogram extraction unit that receives the converted histogram data and the buffered data and detects at least one peak from the histogram; and a waveform analyzer that generates a target waveform based on a correlation between the at least one peak data and the buffered data.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to a sensor device and a control method thereof. FIELD OF THE INVENTION The present invention relates to a rotating sensor device and a LIDAR device. [Background technology]

[0002] Autonomous vehicles (AVs) use multiple sensors for situational awareness. Sensors that are part of an AV's self-driving system (SDS) may include one or more cameras, light detection and ranging (LIDAR), inertial measurement units (IMUs), etc. Sensors such as cameras and LIDAR are used to capture and analyze the scene around the vehicle. The captured scene is then used to detect objects, including static objects such as fixed structures and dynamic objects such as pedestrians and other vehicles. Data collected by the sensors can also be used to detect conditions such as road markings, lane curvature, traffic lights, and signs. Furthermore, a representation of the scene, such as a 3D point cloud captured by the vehicle's LIDAR, may be combined with one or more images captured from a camera to gain additional insight into the scene or situation around the vehicle.

[0003] A lidar transceiver may also include a transmitter that transmits light in the ultraviolet (UV), visible, and infrared spectral regions, and one or more photodetectors that convert other electromagnetic radiation into electrical signals. Photodetectors may be used in a variety of applications, including fiber optic communication systems, process control, environmental sensing, safety and security, and other imaging applications such as optical sensing and distance measurement applications. The high sensitivity of photodetectors allows them to detect faint signals returned from distant objects. However, sensitivity to optical signals requires high degree of alignment between components and alignment of the laser emission. Summary of the Invention [Problem to be solved by the invention]

[0004] Embodiments of the invention may provide sensor and lidar devices that include a transceiver device having a transmitter assembly for transmitting optical signals and a receiver assembly for receiving and processing the optical signals.

[0005] Embodiments of the invention can provide sensor and lidar devices that use a fixed pixel size or angular resolution with an integrated amount of raw data per point to improve the probability of detection and data quality (accurate and precise range and intensity).

[0006] Embodiments of the invention may provide a sensor device and a lidar device having a Geiger mode avalanche photodiode lidar system (GmAPD).

[0007] Embodiments of the invention provide a sensor device and a method for controlling a LIDAR device that generates and processes LIDAR data by spatially and temporally integrating data in a digital signal processing (DSP) circuit / module / processor.

[0008] Embodiments of the invention provide methods for controlling sensor devices and LIDAR devices that transform a multi-stage histogram so that a portion of a high-resolution histogram can be reconstructed in a time domain of interest.

[0009] Embodiments of the invention can provide sensor and lidar devices to improve driving performance without reducing data quality.

[0010] Embodiments of the invention can provide a method for synchronizing the rotational movement of an imaging device with the same system clock as the Precision Time Protocol (PTP) and controlling the capture of an optical impression using a rotating imaging device such as a sensor device and a lidar device.

[0011] An embodiment of the invention can provide a method of controlling acquisition of an optical impression using a rotating imaging device, in which the angle of view of the sensor of the rotating imaging device is controlled relative to an azimuth angle that is constant or kept constant during image acquisition.

[0012] Embodiments of the invention may provide an imaging device and a method for controlling the same for acquiring an optical impression via rotational scanning, where the rotational scanning speed and / or angular position of the rotational imaging device minimizes positional drift within the field of view of the device.

[0013] Embodiments of the invention can provide an apparatus and method for processing image data acquired via a rotating imager by dividing and grouping the image data into two or more portions according to angular position or scanning technique.

[0014] Embodiments of the invention can provide an apparatus and method for controlling a rotating imaging device, capturing image data from sensors within the rotating imaging device, and controlling the rotational movement of the rotating imaging device such that the sensors of the rotating imaging device are synchronized with the capture of image data via the same system clock. [Means for solving the problem]

[0015] A LIDAR device according to an embodiment of the invention may include a data collection unit that collects raw data; a pre-processing unit that removes noise from the collected data; a histogram circuit that converts an output of the pre-processing unit into a histogram; a buffer that buffers the output of the pre-processing unit; a sub-histogram extraction unit that receives the converted histogram data and the buffered data and detects at least one peak from the histogram; and a waveform analyzer that generates a target waveform based on a correlation between the at least one peak data and the buffered data.

[0016] According to an embodiment of the invention, the data stored in the buffer may be deleted once it has been recorded by the sub-histogram extractor.

[0017] According to an embodiment of the invention, a filter may be included to extract peaks through the transformed histogram data.

[0018] According to an embodiment of the invention, the transformed histogram data may include a window circuit for identifying a window for performing filtering within the filter.

[0019] According to an embodiment of the invention, the histogram circuit and the sub-histogram extractor can record, read, and delete histogram data via internal memory and dual ports.

[0020] According to an embodiment of the invention, the buffering by the buffer and the pre-processing by the pre-processing unit can oversample adjacent super-pixels.

[0021] According to embodiments of the invention, the LIDAR device may include a Geiger-mode APD, and the LIDAR device may be a rotating imaging device.

[0022] According to an embodiment of the invention, it may include a plurality of vertically arranged light source arrays and a transmission optical system disposed on the emission side of the light source array.

[0023] According to an embodiment of the invention, the rotating video device can rotate in synchronization with the master clock of the system in the vehicle.

[0024] The lidar device according to an embodiment of the invention includes a light source array that generates an optical pulse having a plurality of light sources arranged in at least two rows, a transmission optical system disposed on the emission side of the light source array that refracts light toward an object, a sensor array having a plurality of photodetectors that sense the optical pulse reflected from the scan region of the object, a reception optical system disposed on the incident side of the sensor array, and two a *2 b (a = 2 to 3m, b < a) It may include a main processor that captures in a three-dimensional point cloud through oversampling in which pixels are acquired and at least one adjacent row of the acquired superpixels is superimposed.

[0025] According to an embodiment of the invention, the lidar device has an angular field of view of 30 degrees or less.

[0026] According to an embodiment of the invention, the superpixel has 12 pixels arranged in two rows, and the lidar device can rotate at an angle of 2 mrad per frame.

[0027] The control method of the lidar device according to an embodiment of the invention includes the steps of receiving and preprocessing raw lidar data, converting the preprocessed output into a histogram, buffering the preprocessed output, detecting peaks from the output converted into the histogram, and generating a target waveform based on the correlation between the detected peaks and the time data points corresponding to the buffered output.

[0028] According to an embodiment of the invention, the buffered data may be removed after being read.

[0029] According to an embodiment of the invention, the pre-processing step may oversample at least one row of superpixels acquired in the rotational direction.

[0030] The lidar device can determine position via multiple line scans with time lag in the rotational direction and scans interpolated by a non-linear function in the vertical direction.

[0031] Further scope of applicability of the present invention will become apparent from the following detailed description. However, the detailed description and specific examples, while indicating preferred embodiments of the present invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art from such detailed description. The present invention will be more fully understood from the detailed description given below and the accompanying drawings, which are given by way of example only and therefore do not limit the invention. [Effects of the Invention]

[0032] The sensor device and LIDAR device according to embodiments of the invention can improve performance without degrading data quality, thereby providing an optimized autonomous driving device. Also, the performance or size of a processor (e.g., FPGA) can be reduced to reduce costs and power consumption, thereby providing an optimized digital signal processor (DSP).

[0033] Embodiments of the invention can more effectively capture optical impressions through rotating imaging devices, such as sensor devices and LIDAR devices, by adjusting the rotational scan speed and angular position of the rotating imaging device so that the field of view of the rotating imaging device is minimized. [Brief explanation of the drawings]

[0034] [Figure 1] FIG. 1 is a perspective view of a vehicle having a lidar device according to an embodiment of the invention.

[0035] [Figure 2] FIG. 2 is a block diagram illustrating an example of a vehicle system including the LIDAR device of FIG. 1.

[0036] [Figure 3] FIG. 3 is a conceptual diagram illustrating the operation of the LIDAR device of FIG. 2.

[0037] [Figure 4] FIG. 3 is a perspective view showing an example of the arrangement of a transmission module and a sensing module of the LIDAR device of FIG.

[0038] [Figure 5] 5 is a diagram showing an example of a line scan of the scan area of ​​the LIDAR device of FIGS. 3 and 4.

[0039] [Figure 6] 5 is a diagram illustrating the operation of a transmission module that transmits pulses for line scanning in the LIDAR device of FIGS. 3 and 4. FIG.

[0040] [Figure 7] 5 is a diagram illustrating the operation of a transmitting module and a sensing module that transmit and receive line scans in the LIDAR device of FIGS.

[0041] [Figure 8] 10 is a diagram illustrating an area scan and a laser spot by a lidar device of a comparative example.

[0042] [Figure 9] As an example of a transmission module of a LIDAR device of the invention, a target area is scanned with light output from a light source array.

[0043] [Figure 10]An example of a transmission module of the inventive LIDAR device is scanning a target area with light generated from a light source array having a faulty light source.

[0044] [Figure 11] 1 is a diagram illustrating an example of a pixel array and superpixels for capturing a point cloud in a LIDAR device of the invention.

[0045] [Figure 12] 1 is a diagram illustrating a superpixel oversampling method for capturing point clouds in a lidar device of the invention.

[0046] [Figure 13] 13 is an example of a histogram of the oversampled data of FIG. 12.

[0047] [Figure 14] FIG. 1 is a conceptual diagram illustrating how a rotational scan area is divided and extracted according to priority in a LIDAR device according to an embodiment of the invention.

[0048] [Figure 15] FIG. 1 is a block diagram illustrating an example of data packet processing according to priority by a processor of an inventive lidar device.

[0049] [Figure 16] FIG. 1 is a block diagram illustrating an example of division by a flow control module according to priority regions in a lidar device of the invention.

[0050] [Figure 17] 1 is a flowchart illustrating a method for processing data according to areas of priority in a LIDAR device of the invention.

[0051] [Figure 18] FIG. 10 is a perspective view of a multi-axis movable transmission module in another example of the LIDAR device of the invention.

[0052] [Figure 19] Figures 19(A) and (B) are examples of scan profile images using a linear merit function with the horizontal and vertical axes (A, B) as references in a lidar device of the comparative example.

[0053] [Figure 20] Figures 20(A) and (B) are examples of scan profile images using a nonlinear merit function with the horizontal and vertical axes (A, B) as references in the LIDAR device of the invention.

[0054] [Figure 21] 1 is a diagram showing an image of an area scan profile on the horizontal and vertical axes (A, B) in the LIDAR device of the invention.

[0055] [Figure 22] FIG. 10 is a conceptual diagram illustrating decoupling of the horizontal and vertical axes by linear interpolation in another example of the LIDAR device of the invention.

[0056] [Figure 23] 10 is a flowchart for adjusting a multi-stage scan range in another example of the LIDAR device of the invention.

[0057] [Figure 24] 10 is a diagram illustrating a third scanning process for a peak position in a scan area in another example of the LIDAR device of the present invention.

[0058] [Figure 25] FIG. 2 is a detailed block diagram of a main processor of a LIDAR device according to an embodiment of the invention.

[0059] [Figure 26] 26 is a flowchart illustrating the processing of lidar data by the main processor of FIG. 25.

[0060] [Figure 27] 26 is a flowchart illustrating a data processing process by the main processor of FIG. 25.

[0061] [Figure 28] 4 is a flow chart illustrating the operation of a histogram circuit according to an embodiment of the invention;

[0062] [Figure 29] 10 is an example of a timing diagram for reading and clearing a histogram in a memory according to an embodiment of the invention.

[0063] [Figure 30] 1 is a diagram illustrating DSP pipeline resources sharing the same resource according to an embodiment of the invention.

[0064] [Figure 31] FIG. 10 is a timing diagram illustrating sub-histogrammer output to a streaming interface according to an embodiment of the invention.

[0065] [Figure 32] 10 is an example output of a histogram filter according to an embodiment of the invention. DETAILED DESCRIPTION OF THE INVENTION

[0066] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings, in which like reference numerals indicate like components. However, the present invention may be embodied in a variety of different forms and is not limited to the embodiments merely exemplified herein. Rather, such embodiments are provided as examples to ensure that this disclosure is thorough and complete, and to fully convey the features and functions of the present invention to those of ordinary skill in the art. Therefore, processes, elements, and techniques that are not necessary for a person of ordinary skill in the art to fully understand the features and functions of the present invention may not be described. Unless otherwise specified, like reference numerals indicate like components in the accompanying drawings and written description, and therefore, descriptions thereof will not be repeated.

[0067] Lidar systems are sometimes called depth-sensing systems, laser distance measurement systems, laser radar systems, LIDAR systems, or laser / light detection and ranging (LADAR) systems. Lidar is a type of distance measurement sensor characterized by long sensing range, high resolution, and low environmental interference. Lidar has been widely applied in the fields of intelligent robots, unmanned aerial vehicles, and autonomous or self-driving vehicles. The operating principle of Lidar is to estimate distance based on the round-trip time (e.g., flight time or delay time) of electromagnetic waves between a source and a target.

[0068] Generally, a lidar system, such as a direct time-of-flight (D-TOF) lidar system, measures the distance (e.g., depth) of an object by emitting a light pulse (e.g., a laser pulse) toward the object and measuring the time it takes for the light pulse to reflect off the object and be sensed by a sensor in the lidar system. For example, to reduce noise from ambient light, repeated measurements can be performed to generate a relative time-of-flight (TOF) individual histogram based on the repeated measurements, and peaks in the individual histograms can be calculated to detect events (e.g., to detect the depth of a point or region of the object that reflects the light pulse again).

[0069] The above-mentioned aspects and features of embodiments of the present invention will be explained in more detail with reference to the drawings.

[0070] FIG. 1 is a perspective view of a vehicle having a lidar device according to an embodiment of the invention.

[0071] Referring to FIG. 1, a moving object such as a vehicle 500 may include a lidar device 100, a camera unit 101, vehicle recognition sensors 102 and 104, a GPS (Global Positioning System) sensor 103, a vehicle control module 212, and an ultrasonic sensor 105.

[0072] The LIDAR device 100 is a rotating imaging device or sensor device that is attached to a part of the vehicle 500 and rotates 360 degrees, sensing the distance between the vehicle and objects (static objects, dynamic objects), the surrounding environment, and their shapes, and controlling driving using the measured data. Using this sensing technology, it is possible to collect and analyze the objects and environment around the vehicle as a 3D point cloud, and generate sensing data that provides information on objects located within an appropriate proximity range.

[0073] The lidar device 100 can communicate with a vehicle control module 212 to send / receive information related to vehicle driving. The vehicle control module 212 can communicate with various systems or sensors within the vehicle and perform various controls. The vehicle control module 212 may include a control device such as an electronic control unit (ECU) that controls and monitors various systems of the vehicle. The vehicle control module 212 can communicate with an external mobile device and may be electrically connected to a removable storage device.

[0074] The camera unit 101 may be installed in one or more locations inside and / or outside the vehicle, and may capture images of the front and / or rear of the vehicle and provide or store the images through a display device (not shown). The captured image data may optionally include audio data. As another example, the camera unit 101 may be installed at the front, rear, corners, or sides of the vehicle 500 to capture images of the vehicle's surroundings and provide the images through a display device (not shown). The vehicle control module 212 or another processor may identify traffic lights, vehicles, pedestrians, etc. based on the data captured by the camera unit 101 and provide the acquired information to the driver. Such a camera unit 101 may be used as a driving assistance device.

[0075] A plurality of forward radars 102 are installed in front of the vehicle 500 to detect the distance between the vehicle 500 and a forward object. A plurality of rearward radars 104 are installed behind the vehicle 500 to detect the distance between the vehicle 500 and a rearward object. When object information is detected through these radars 102 and 104, the driver is notified of surrounding objects or obstacles by an alarm or warning message.

[0076] The GPS sensor 103 can receive signals from satellites and provide them to devices such as the vehicle control module 212, the lidar device 100, and the camera unit 101, which can provide or calculate information such as the vehicle's position, speed, time, etc. based on the GPS position signals.

[0077] The ultrasonic sensor 105 can sense the distance to nearby vehicles or obstacles to provide convenience for safely parking a vehicle in a parking space. In addition, the ultrasonic sensor 105 can prevent accidents that may occur while driving. Such ultrasonic sensors 105 may be installed on the rear, side, or tires of the vehicle.

[0078] 2, a vehicle system 200 having a lidar device 100 and a vehicle control module 212 receives input from a user or driver and provides information to the user or driver via a user interface 211. The user interface 211 may include a display device, a touch panel, buttons, voice recognition, and a wired or wireless input device, and is connected by wire or wireless to enable communication between the driver and various devices.

[0079] The vehicle system 200 communicates with a remote device 213, which can remotely communicate with a user or the outside and receive external control signals. The communication unit 215 can support wired or wireless communication, and may be, for example, a wired or wireless module.

[0080] The storage unit 220 may include one or more sub-memories 221 therein. The storage unit 220 may also include a portable or removable storage device 222. The LIDAR device 100 can communicate with a user interface 211 and a camera unit 101.

[0081] The LIDAR device 100 includes a drive unit 150, such as a motor, that can rotate a portion of or the entire LIDAR device 100 360 degrees in response to a control signal. The drive unit 150 communicates with internal components of the LIDAR device 100, such as the measurement system 110, to enable the LIDAR device 100 to rotate around an axis.

[0082] The LIDAR device 100 may include a measurement system 110, a transmission module 120, and a sensing module 130. The driver 150 may transmit a driving force to rotate the measurement system 110, the transmission module 120, and the sensing module 130.

[0083] The measurement system 110 may include a main processor 111 and a main memory 112. The main processor 111 may be embodied as a general-purpose processor, an application-specific integrated circuit (ASIC), one or more field-programmable gate arrays (FPGAs), a group of processing components, or other suitable electronic processing components. The main memory (e.g., memory, memory unit, storage device, etc.) 112 may include one or more devices (e.g., RAM, ROM, flash memory, hard disk storage, etc.) for storing data and / or computer code for completing or facilitating the various processes described herein. The main memory 112 may be or include volatile or non-volatile memory. The main memory 112 may include database components, object code components, script components, or any other type of information structure to support the various activities and information structures described herein. According to an embodiment, the main memory 112 may be communicatively coupled to the main processor 111.

[0084] The measurement system 110 may include one or more processors (also referred to as central processing units or CPUs). The one or more processors may be coupled to a communications infrastructure or bus. Each of the one or more processors may be a graphics processing unit (GPU). In some examples, a GPU (graphics processing unit) may include a processor, which is a specialized electronic circuit designed to process mathematically intensive application programs. A GPU may have a parallel architecture that is efficient for parallel processing of large blocks of data, such as the mathematically intensive data commonly used in computer graphics applications, images, video, etc.

[0085] The measurement system 110 may be a computer system coupled to one or more user input / output devices, such as a monitor, keyboard, and pointing device, that communicate with a communications infrastructure via a user input / output interface.

[0086] The transmitting module 120 may include a light source array 121 and a transmitting optical system 123. The transmitting module 120 may include a processor or control module, such as a general-purpose processor, ASIC, or FPGA, that can control the driving and transmission of the light source array 121 and the transmission of optical signals, and may also include an internal memory in which code for controlling laser beam generation is stored.

[0087] The light source array 121 may include a one-dimensional or two-dimensional array and may be individually addressable or controllable. The light source array 121 may include a plurality of light sources that generate laser beams or light pulses. The light sources may include, but are not limited to, laser diodes (LDs), edge emitting lasers, vertical-cavity surface emitting lasers (VCSELs), distributed feedback lasers, light emitting diodes (LEDs), super luminescent diodes (SLDs), and the like.

[0088] The light source array 121 may include a plurality of electrically coupled surface emitting laser diodes, such as a VCSEL array, where each emitter may be individually addressable or controllable. The light source array 121 may be embodied as a one-dimensional (Q*P) VCSEL array having Q rows and P columns, or a two-dimensional array, where Q and P (columns, rows) may be 2 or greater (Q>P). Furthermore, a plurality of VCSEL arrays may be grouped to form respective light sources.

[0089] The optical signal emitted from the light source array 121 may be irradiated toward an object via the transmission optical system 123. The transmission optical system 123 may include one or more lenses, or one or more lenses with a microlens array in front of them. The transmission optical system 123 may include one or more optical lens elements so that the laser beam is shaped in a desired manner. That is, the transmission module 120 may set the irradiation direction or irradiation angle of the light generated from the light source array 121 under the control of the main process 111. The LIDAR device 100 may also include a beam splitter (not shown) therein for superimposing or separating the transmitted laser beam L1 and the received laser beam L2.

[0090] The transmitting module 120 can be irradiated with pulsed or continuous light and can transmit it multiple times (plurality of time) toward the object to be scanned. The processor 111 can generate a start signal at the time of light transmission and provide it to a time to digital converter (TDC). The start signal can be used to calculate the time of flight (TOF) of the light.

[0091] The sensing module 130 may include a sensor array 131 and receiving optics 133. The sensing module 130 may include a processor that performs at least one of the following functions: converting a primitive histogram, including a matching filter, a peak detection circuit, a SPAD saturation and quenching circuitry, and compensating for pulse shape distortion. Such a processor may be embodied as a general-purpose processor, an application-specific integrated circuit (ASIC), one or more field-programmable gate arrays (FPGAs), a group of processing components, or other suitable electronic processing components. The sensing module 130 may include a memory (not shown) having one or more devices (e.g., RAM, ROM, flash memory, hard disk storage, etc.) for storing detected optical signals therein.

[0092] The sensor array 131 can receive the laser beam L2 reflected or scattered from an object via a receiving optical system 133. Such a sensor array 131 may include a detector divided into a plurality of pixels, and a light detecting element may be disposed in each of the plurality of pixels. The receiving optical system 133 may be an optical element for focusing reflected light onto a predetermined pixel.

[0093] When reflected light is received by the sensor array 131, the sensing module 130 can convert the reflected light into a stop signal. The stop signal, together with a start signal, may be used to calculate the time of flight of light. The sensor array 131 or the sensing module 130 may include a TDC for measuring the time of flight of light detected by each of a plurality of photodetectors. The photodetectors may be light-receiving elements that generate an electrical signal in response to detected optical energy, and may be, for example, single photon avalanche diodes (SPADs) with high sensing sensitivity.

[0094] The sensor array 131 may be implemented as a one-dimensional or two-dimensional array and may be a collection of photodetectors, such as single avalanche photodiodes (SPADs), or single photon detectors (APDs). An embodiment of the invention may be implemented using a single photon photodetector. The sensor array 131 measures light pulses, i.e., light corresponding to image pixels, through a receiving optical system 133. The sensor array may be arranged as a two-dimensional SPAD array having M rows and N columns, where M and N may be 2 or greater. A plurality of SPAD subarrays may be grouped to form a photo sensor. The sensor array may be a Geiger mode, i.e., a guided mode APD (GmAPD).

[0095] The main processor 111 processes signals to obtain information about an object using light detected by the sensing module 120. The main processor 111 determines the distance to the object based on the time of flight of light reflected from the object and processes data to analyze the position and shape of the object. Information analyzed by the processor 111, i.e., information about the shape and position of the object, may be transmitted to another device.

[0096] The transmitting optical system 123 refracts the light pulses generated through the light source array 121 and irradiates them toward the object, and the light pulses are incident on and reflected from the surface of the object, and the reflected light pulses can be sensed by the sensor array 131 via the receiving optical system 133. The distance or depth to the object can be determined based on the time elapsed from the emission of the light pulse to the detection of the reflected light pulse (TOF: Time of Flight).

[0097] If the light detection element includes a SPAD, sensing sensitivity is high but noise also increases. To calculate a reliable time-of-flight (TOF) of light using a SPAD, the main processor 111 irradiates light toward an object several times, generates a histogram of the light reflected from the object, and statistically analyzes the histogram. The main processor 111 can calculate the distance from the TOF of the light to points on the object surface and generate a point cloud based on this distance data. The generated point cloud data may be stored in a database in the main memory 111. The point cloud data stored in this database may be converted into a three-dimensional shape and stored. The point cloud data may be preprocessed by the preprocessing unit 11B (FIG. 25) to remove noise. The noise may be automatically detected and removed due to distortion caused by light reflected through special environments such as glass.

[0098] The LIDAR device 100 can combine one or more image data acquired by the camera unit 101 to obtain an image representation, such as a point cloud, to gain additional insight into the scene or situation around the vehicle.

[0099] As shown in FIG. 3, the LIDAR device 100 can determine the position of an object using the angle of an emitted laser beam. For example, if the LIDAR device 100 knows the irradiation angle of a laser beam irradiated toward a scan area T0, the LIDAR device 100 can determine the position of the object based on the irradiation angle of the emitted laser beam by detecting the laser beam reflected from an object present in the scan area T0 in the sensor array 131. The LIDAR device 100 can also determine the position of an object using the angle of the received laser beam. For example, if a first object and a second object are at the same distance from the LIDAR device 100 but at different positions relative to the LIDAR device 100, the laser beam reflected from the first object and the laser beam reflected from the second object may be detected at different points on the sensor array 131. That is, the LIDAR device 100 can determine the position of the object based on the point at which the reflected laser beam is detected by the sensor array 131.

[0100] The LIDAR device 100 may have a scan area T0 including an object to detect the position of any object in the vicinity. Here, the scan area T0 represents a detectable area on a single screen and may refer to a set of points, lines, or surfaces forming a single screen during one frame. The scan area may also refer to an area irradiated by a laser beam emitted from the LIDAR device 100, and the irradiated area may refer to a set of points, lines, or surfaces tangent to a spherical surface at the same distance from the laser irradiated during one frame. The field of view (FOV) refers to a detectable area and may be defined as the angular range of each scan area of ​​the rotating LIDAR device when the LIDAR device 100 is viewed as the origin.

[0101] To expand the scan area T0, the light source array 131 can be driven in a line or a plane. For example, when a plurality of light sources are arrayed in a line or a plane, the direction and size of the laser beams emitted by the light sources can be changed through drive control, thereby expanding the scan area T0 of the LIDAR device 100 in a line or a plane.

[0102] As another example, the transmission module 120 or the transmission optical system 123 may include a scanning unit (not shown). The scanning unit may change the irradiation direction and / or size of the laser beam in the light source array 121, thereby expanding the scan area of ​​the LIDAR device 100 or changing the scan direction. The scanning unit may expand or change the scan area of ​​the point-like laser beam into a line or a plane. The number of the scanning units may be one or more, but is not limited thereto.

[0103] The LIDAR device 100 may provide a single-photon detector (SAPD) or Geiger-mode LIDAR having a light source array 121 and a sensor array 131. The Geiger-mode LIDAR is sensitive to single photons and can transmit space-filling (i.e., gapless) images. That is, a space-filling image has a recovery time in which the detector must return to a photon detection state after detecting one photon to detect the next photon. This recovery time appears as a time difference or gap between detected photons.

[0104] APDs are defined as photodetectors that have gain within the device by applying a reverse voltage to cause an avalanche event. APDs have a high signal-to-noise ratio due to current amplification caused by the avalanche effect in diodes, and have higher quantum efficiency than detectors with other characteristics. APDs can be classified into Geiger-mode and linear-mode. In linear mode, a reverse voltage lower than the breakdown voltage is applied, and the output current is linearly proportional to the input light intensity, resulting in a gain characteristic proportional to the reverse voltage. In Geiger mode, a reverse voltage higher than the breakdown voltage is applied to detect single photons.

[0105] In addition, the Geiger mode of the invention can operate in short-time operation Geiger mode, can generate a digital output signal for single-photon level input, and has a gain range 100 times larger than that of linear mode. The receiving sensitivity is 100 photons / 1 nsec in the advance mode and 1 photon / 1 nsec in the Geiger mode, enabling single-photon detection (SAPD).

[0106] In addition, the laser beam used in the Geiger mode of the present invention uses infrared wavelengths above 1000 nm, for example, above 1400 nm or in the 1550 nm wavelength band, which is higher than linear modes that use wavelengths below 900 nm. The Geiger mode also drives the light source array and sensor array to transmit uniform, space-filling imaging, i.e., gapless imaging. Such wavelengths above 1400 nm are visually safe, reducing problems caused by beam divergence and distance limitations.

[0107] The Geiger mode according to an embodiment of the invention may be insensitive to malfunctions or defects of individual light sources due to the light source array 121 having a plurality of light sources (emitters) and the sensor array 131 having a plurality of photodetectors. As shown in FIGS. 4 to 6, the LIDAR device 100 may include a transmitting module 120 and a sensing module 130 integrated together. The LIDAR device 100 may include a transmitting module 120 vertically arranged in front of one surface of a fixed frame 300, and a transmitting optical system 123 may be vertically disposed on the output side of the transmitting module 120. The transmitting module 120 may be connected to a transceiver interface board 125 at its bottom. A driver circuit 120A may be disposed on one side of the transmitting module 120.

[0108] The LIDAR device 100 includes a sensing module 130 at the rear of one side of a fixed frame 300, and a portion of the sensing module 130, i.e., a portion of a receiving optical system 133, may be supported by penetrating an inner wall frame of the fixed frame 300. A circuit board 135 may be coupled to the rear of the sensing module 130 and electrically connected to the sensor array 131.

[0109] The fixed frame 300 of the LIDAR device 100 is provided with a heat sink 301 on part or the entire other surface, and the heat sink 301 has a number of heat sink pins to dissipate the heat generated by the transmitting module 120 and the sensing module 130.

[0110] The LIDAR device 100 may be provided with a transparent protective plate 129 to protect the transmitting module 120 and a portion of the sensing module 130. The protective plate 129 may be disposed on one side of the transmitting module 120 and may be spaced apart from the front of the sensing module 130, i.e., the light incident side region.

[0111] The LIDAR device 100 has a linear shape and includes a vertical light source array 121 and a transmission optical system 123. The transmission module 120 and the sensing module 130 are arranged to face the object in the same direction. The LIDAR device 100 can rotate 360 ​​degrees around the rotation shaft of a drive motor connected to the bottom.

[0112] The light source array 121 may have a predetermined number of light sources arranged in one or more vertical rows, for example, 48 light sources arranged in two rows. Such an array of light sources may be arranged on a one-dimensional plane and transmit light toward the side of the LIDAR device 100. The laser beam emitted by the light source array 121 passes through a transmission optical system 123, which may be configured to focus the beam on a scan area using a pattern such as a microlens array and lock a desired wavelength. The driver circuit 120A may use high-current, short pulses (e.g., 3 to 6 ns). In this case, the transmitted pulse energy may be in the range of ~2uj for a ~5 ns pulse.

[0113] As shown in Figure 6, the profile of the beam emitted by the transmitting module 120 may have a line-shaped pattern SR1 as shown in Figure 5, with a beam divergence angle R1 of 1 mrad or less and a vertical elevation H1 of 30 degrees or less. The output of such a light source may be adjusted so that the emitted beam provides uniformly distributed illumination toward the object by the light source array 121. The point cloud acquired using such a light source array 121 and sensor array 131 may be classified as vertical space-filling.

[0114] Additionally, the transmission optical system 123 may be provided as a monolithic optical element within the light-emitting field, separate or spaced apart from the light source array 121, to affect the shape of the light emitted from the light source array 121. The transmission optical system 123 may be aligned to project the output of individual light sources onto the entire field.

[0115] 7, the one-dimensional light source array 121 of the transmitting module 120 and the one-dimensional sensor array 131 of the sensing module 130 may be aligned to correspond to each other, allowing the LIDAR device 100 to use the data acquired by the scan region SR1 as superpixels.

[0116] The lidar points form a point cloud, and larger meanings and information can be obtained by grouping some points within the point cloud into superpixels. Such point clouds contain object shape and distance information. Superpixels group the lidar point cloud into larger units of meaning, and at this time, group adjacent points to divide it into larger areas, clarifying the boundaries of specific objects or structures, thereby enabling more precise object recognition and classification.

[0117] As shown in Figures 8A, 8B, and 8C, the linear mode of the comparative example uses a dispersed laser spot and detector, resulting in gaps between imaging points. That is, the linear mode results in large non-illuminated (or non-imaged) areas (e.g., gaps) of the scene that are not captured by the sensor due to the dispersed laser beam. Dispersed laser spots may also be more sensitive to malfunctions or defects of individual light sources (emitters). In this linear mode, the divergence angle is 2 mrad, and at a distance of 200 m, the dispersed laser spot is formed over a scan area of ​​40 cm, with each pixel size being detected as a divergence angle of 2 mrad. The linear mode of the comparative example cannot provide uniform illumination because gaps or non-uniformities occur in the output of the laser diode array due to malfunction or failure of individual emitters.

[0118] As shown in FIG. 9, the transmission optical system 123 according to an embodiment of the present invention can distribute each laser beam with uniform illumination over the entire target area. That is, all areas of the target may be illuminated with uniform output light. As shown in FIG. 10, if some light sources #4 malfunction or are inoperable (or deactivated), laser beams emitted by different light sources may be irradiated onto the entire target area via the transmission optical system 123. As a result, the optical sensitivity received over the target area may have a substantially uniform distribution. With this configuration, even if some light sources are deactivated, the overall optical sensitivity is uniform, there is only a negligible power reduction, and there is no lost gap in the optical field, so there may be only a negligible decrease in the sensitivity of the received light. This LIDAR device improves data reliability because the failure of some light sources does not significantly affect data. Furthermore, there is no need to request service for the LIDAR device even if some light sources are faulty.

[0119] The invention also uses superpixels to optimize the tradeoff between signal and resolution, i.e., superpixels can be used to ensure sufficient signal strength while maintaining high resolution, allowing for more accurate measurements and imaging.

[0120] The LIDAR device 100 can implement horizontal space-filling by overlapping measured point cloud data in the rotational direction using a transmitting module 120 having a vertical light source array 121 and a transmitting optical system 123, and a sensing module 130 having a receiving optical system 133 and a sensor array 131.

[0121] The light source array 121 may include a plurality of light sources having at least two rows. For example, the light source array 121 may be arranged in a Q*P number of rows (Q and P are integers equal to or greater than 2), for example, 2 m *2 n The light sources may be arranged in an array, where m is 6 or more, for example, in the range of 6 to 10, and is preferably 9, and n is smaller than m and is 2 or less, for example, 1. Here, Q is the number of light sources arranged in the vertical direction, and P is the number of light sources arranged in the horizontal direction.

[0122] The light source array 121 is illuminated with one-dimensional illumination and can cover the entire scene through rotation. A predetermined area in the illuminated scan area can be defined as a superpixel, for example, 2 a *2 b A pixel can be defined as a superpixel, where a is the number of pixels in the vertical direction, which may be less than m and in the range of 2 to 3, and preferably 2.60 (excluding decimal points), and b is the number of pixels in the horizontal direction, which may be less than a or equal to n, and preferably 1.

[0123] Here, the vertical elevation of the superpixel may be 2 mrad (e.g., 0.11 degrees), the divergence angle R1 of the superpixel and the scan area SR1 may be the same, and the distance measurement error or magnitude at the target distance (e.g., 200 m) may be 16 cm or less (i.e., 16 cm @ 200 m) or 14 cm or less (i.e., 14 cm @ 200 m).

[0124] Horizontal space filling can be implemented by oversampling in the horizontal direction (or rotational direction) at a high laser pulse rate. Here, oversampling refers to sampled data in which adjacent superpixels overlap at least one column. Ten or more pulses are applied per frame, e.g., 13 pulses, to obtain 156 pixel samples. That is, one frame is obtained by 12 pixels x 13 pulses, and these 12 pixels can be used as one point cloud. Furthermore, if the superpixels are arranged in one column, they rotate 85 degrees, which is 512 / 6. However, if they are arranged in two columns, 12 pixels and 13 measurements are used to measure 156 samples (=12 x 13). This measured sample data can be defined as one point cloud.

[0125] The LIDAR device 100 has a transmission optical system 123 and / or a reception optical system 133 with a field of view of 30 degrees or less, and can rotate to cover the entire scene. For example, when emitting a single laser pulse, it can measure with a rotational accuracy of up to 3 cm at a distance of 200 m. In addition, it can capture a 3D point cloud of the surrounding environment using oversampling technology. That is, it can effectively detect objects around short-range and long-range vehicles for safe exploration.

[0126] As shown in Figure 13(A), 12 pixels are measured 13 times, resulting in a rotational scan of less than 0.12 degrees per frame, i.e., an angle of 2 mrad, and the acquired data can be displayed in histogram bins as shown in (B).

[0127] In another example of the invention, referring to FIG. 14, a moving body such as a vehicle 500 can separate priorities according to the rotational scan area of ​​the LIDAR device 100 to reduce processing resources and power consumption when generating large amounts of data.

[0128] The LIDAR device 100 may divide the field of view into at least two regions A1 and A2 and allocate separate resources between the two regions. The division may be performed during acquisition, for example, by dividing data into multiple groups or swaths according to the location or range of acquisition. That is, the division may be performed on data acquired via a sensor array. For example, the first region A1 may be a high-priority region or a region of high relevance that requires more in-depth analysis in terms of processing. At least one of the regions other than the first region A1, for example, the second region A2, may be a low-priority region to which a single processing step is applied relatively infrequently.

[0129] The second area A2, which has a lower priority, may be used for data logging purposes. Conversely, the first area A1, which has a higher priority, may be monitored by a sub-backup controller that does not need to monitor all areas. This has the advantage of more efficiently utilizing computational resources for monitoring the first area A1. During normal operation, the first area A1, which has a higher priority, may be the area in front of the vehicle. However, when dividing the field of view (FOV) for the second area A2, which has a different priority, it may be adjusted so that the highly relevant area faces a meaningful direction. In other words, it may be important for the highly relevant area to have the correct pointing angle.

[0130] Lidar data packets acquired from the first area A1, which has a high priority, are routed through a multicast communication port. Lidar data packets that fall within the multicast angle range defined by the start angle Rst and stop angle Rsp of the azimuth angle R can be routed through a multicast UDP / IP endpoint instead of a unicast UDP / IP endpoint, as shown in Figure 15.

[0131] The invention provides a method for ensuring and stabilizing the pointing angle as follows. For example, as shown in FIG. 15A, the main process 111 can transmit and receive lidar data packets, i.e., 3D data, via data communication with the network via a first communication port (UDP / IP Port 1) 33A and a second communication port (UDP / IP Port 2) 33B. That is, the division or grouping of sensor data can be implemented as a multicast function. A multicast function generally allows a subset of unicast lidar data packets (e.g., UDP packets) to be routed to alternate targets (e.g., IP addresses and UDP ports). Furthermore, lidar data packets received from a second area A2 off the azimuth angle can be unicast along with other data, such as Geiger-mode avalanche photodiode ("GmAPD") and status packets. As an additional example, the different areas can be sector-shaped, which is a portion of a circle.

[0132] As shown in FIG. 16 , data division may be implemented via a flow control module 136 that generates groups or swaths from raw data generated by the lidar device 100. The flow control module 136 may form at least two data groups from raw data received from the lidar device using criteria such as azimuth angle. The raw data may represent data from a readout interface of the imaging device, such as a ROIC (Readout Interface Circuit) 137. The range for a given group or sample may be specified by setting two azimuth angle values ​​that specify start and stop limits and a limit for the defined region. Another method is to define a single azimuth angle value and then specify the number of sensor frames calculated at that azimuth angle value. The flow control module 136 and ROIC 137 may be included in a processor.

[0133] The rotational scanning movement of the lidar device may be identical to the master clock of a mobile or vehicle system. For example, the rotational scanning movement of the lidar device 100 may be synchronized with the master clock of the system, such as the Precision Time Protocol ("PTP") grandmaster of a self-driving system (SDS).

[0134] Alternatively or additionally, the present invention may synchronize shutter control of at least one camera of a vehicle, e.g., an autonomous vehicle (AV), with a system master clock or with the rotational scanning movement of a lidar device. For example, the camera unit 101 (FIG. 2) or sensor may have a global shutter, which can be synchronized with the lidar scanning movement. Thus, the camera unit 101 can be controlled so that its shutter is synchronized with the rotation of the lidar device 100. To this end, for example, the timing or clock of the lidar device 100 can be synchronized with the PTP. Similarly, the shutter signal of the camera unit 101 can be synchronized with the PTP of the SDS. This allows images captured via the camera unit 101 to be synchronized with the capture representation (form or format) of the lidar device 100, thereby improving the combination of lidar data and images captured by the camera unit 101. This improves synchronization between the data output stack (e.g., 3D map) of the lidar device 100 and the vision stack (video generated by the camera unit) of the autonomous vehicle, thereby improving the autonomous driving system's (SDS) ability to sense the vehicle's surrounding environment and improving computer bandwidth by aligning the two output stacks.

[0135] Additionally or alternatively, the present invention may operate in an azimuth lock mode in which the pointing angle of the camera unit 101 and / or the LIDAR device 100 is controlled over time. The azimuth lock mode is a mode in which data is collected while fixed at a specific azimuth angle or angle. In this mode, the LIDAR device rotates to match the specified azimuth angle and collects data, thereby enabling more accurate acquisition and tracking of information in a specific direction. In the azimuth lock mode, the field of view of the LIDAR device can be fixed in azimuth relative to a fixed reference plane. That is, the azimuth angle of the LIDAR sensor can be controlled to a predetermined value or range. The present invention may suppress or prevent position drift in the pointing angle of a LIDAR device (e.g., a mechanical optical sensor such as a LIDAR) through control in the azimuth lock mode. Position drift refers to a deviation from a target position or direction due to a change in position or direction. Therefore, without the azimuth lock mode, position drift may cause unwanted rotation and movement to an undesired pointing angle. Therefore, the invention allows for more consistent and precise adjustment and alignment of the pointing angle of a rotational imaging device, such as a LIDAR device, allowing the LIDAR device to more accurately capture a desired object or area, while maintaining system stability and accuracy.

[0136] More specifically, the measurement system 110 (FIG. 2) or main processor 111 of the lidar device 100 can control the rotation rate and / or angular position to minimize position drift within the angle at which the pointing angle is stabilized. Additionally, the measurement system 110 (FIG. 2) or main processor 111 can synchronize the rotation by the drive unit 150 (FIG. 2) to the PTP time. For example, the azimuth angle can be controlled to within ±5° during operation. Preferably, the azimuth phase angle can be controlled to within ±2°, which is a range that can accurately capture the position and orientation relative to the front of the vehicle.

[0137] 17, a method for improving image data acquired by a LIDAR device includes measuring the angular position of a rotatable part (S121), acquiring the measured data as an optical impression through an image sensor of a sensor array (S123), dividing the acquired imaging data according to priority (S125), and capturing images of prioritized areas among the divided areas based on PTP or field of view (S127). The data division may involve grouping and dividing the imaging data according to the angular position or scan range at which the imaging data was acquired. Furthermore, when capturing an optical impression through the image sensor, the capture is performed in response to a trigger signal, and the trigger signal may be generated in response to an angular position aligned with a reference point.

[0138] In addition, as a method for synchronizing the capture of acquired imaging data with the rotational movement of a rotating imaging device, i.e., a LIDAR device, the rotational movement and the capture of imaging data can be coordinated or synchronized using the same system clock. In addition, as a method for acquiring optical impressions using a LIDAR device, which is a rotating imaging device, the rotational movement can be coordinated or synchronized with a system clock, such as a precision time protocol. Furthermore, the field of view of the LIDAR device can be controlled relative to the azimuth angle; for example, the azimuth angle during image data acquisition can be set constant or maintained approximately constant. In this case, the rotational scan speed and / or angular position of the LIDAR device can be controlled to minimize positional drift within the field of view range.

[0139] 18, a transmitter module 120 according to an embodiment of the invention includes a light source unit 121A having a light source array 121, a collimating lens 123A, and a transmission optical system 123 having a VBG (Volume Bragg Grating) lens 123B, which are disposed on a substrate 102A. The collimating lens 123A has a convex curved surface on the output side and can extend long in the longitudinal direction Y of the light source array 121. The collimating lens 123A refracts incident light to be parallel, and the VBG lens can directly irradiate or focus light of a specific wavelength, and can particularly adjust the optical path to a desired direction.

[0140] The high sensitivity of the sensor array 131 allows it to sense faint signals returned from distant objects, while its high sensitivity to optical signals can provide high-precision alignment between components on the emission path of the laser pulse within the transmit module 120. Therefore, the proposed lidar device 100 and its control method of the invention provides active optical alignment of the transmit module 120 and the sensing module 130 to improve production tolerances and increase manufacturing yields.

[0141] To this end, the light source unit 121A, collimating lens 123A, and VBG lens 123B of the transmitter module 120, which include the light source array 121, can adjust multiple linear and rotational axes (x / y / z / Rx / Ry / Rz) to find the optimal area scan or angular position. The collimating lens 123A and / or VBG lens 123B can be coupled to move in three-dimensional axial directions (X, Y, Z) or rotate (Rx, Ry, Rz) based on each axial direction by providing motors on both sides of each axial direction. Therefore, to find the optimal position for each axis, each component can be independently driven and controlled to perform area scan. That is, to find the optimal position of the components of the transmitter optical system, active alignment is required. Two issues that can arise in connection with this alignment are as follows: 1) how to define the optimal position using non-independent parameters, and 2) how to effectively reach the optimal position of a coupled axial system. Here, for active alignment, multiple axes (x / y / z / Rx / Ry / Rz, etc.) must be adjusted to find the optimal position, but in actual production conditions, there is a strong axis coupling effect, which may occur due to alignment equipment limitations, process variations, or the internal physical mechanism of the optical system.

[0142] The invention provides at least two main approaches to solving this problem: 1) a new algorithm for identifying the optimal position; and 2) an efficient scanning method, both of which can be used to achieve high yields and fast process times.

[0143] Referring to FIGS. 19 through 24, an area scanning method for finding an optimal position in a LIDAR device according to an embodiment of the present invention is described. FIGS. 19A and 19B show an example of an area scanning algorithm using a non-linear merit function for calculating an optimal position according to an embodiment of the present invention, and FIGS. 20A and 20B show an example of an area scanning algorithm using a linear merit function. In FIGS. 19A and 19B, the first axis (A-axis) parameter and the second axis (B-axis) parameter can perform within a specific specification. For example, performance can be achieved under the condition: first axis parameter > 0.88, second axis parameter > 30 dB. Furthermore, when external noise is present (FIG. 19B), the direction of the driving force can further enhance resistance to the external noise.

[0144] As shown in Figures 20A and 20B, when applying an area scan algorithm using a linear merit function, it can be seen that the positions of the vertical first axis (A-axis) parameter and the horizontal second axis (B-axis) parameter in the scan area deviate from the specified specifications. For example, the scan position deviates from the specifications of the first axis parameter > 0.88 and the second axis parameter > 30 dB, and falls within the first axis parameter < 0.88 and / or the second axis parameter < 30 dB. In other words, the linear merit function makes it difficult to perform active alignment of individual parameters because various parameters for quantifying optical performance are not provided independently of each other. That is, when a weight is assigned to one parameter, other parameters may deviate from the specifications. Furthermore, when an area scan is performed using a gradient descent method to find the optimal position, resistance to external noise may be reduced. These issues make it difficult to find the optimal parameters for active alignment.

[0145] The present invention can provide better performance when a nonlinear merit function is used for area scanning. That is, it is possible to perform area scanning so that a selected position performs within specifications without assigning a very high weighting to one specific parameter. Furthermore, since area scanning is performed in an adapted direction to which a driving force is applied, it is robust to external noise and can enable optimal position calculation.

[0146] 21 is a diagram showing an example of fine line scanning according to an embodiment of the invention. That is, after an area scan, a fine line scan is performed to find the final position. This area scan method involves a trade-off between time and accuracy. When the scan steps required to achieve high accuracy are fine, the scanning process is very time-consuming, and attempting to improve the slow scan time by increasing the scan steps results in lower accuracy.

[0147] As shown in Figure 22(A), an area scanning method that scans a position along a second axis (B-axis) relative to each first axis (A-axis) can increase the number of scanning steps and time, resulting in reduced scanning efficiency. As shown in Figure 22(B), the present invention can search for a position by scanning the first and second axes (A-axis, B-axis) that are actually combined using a linear interpolation method. Linear interpolation can generate a grid-structured digital surface model by shaping the surface using a grid structure, thereby collecting more accurate distance and spatial information. Here, the first axis can be a vertical axis direction relative to the scan area, and the second axis can be a horizontal axis direction perpendicular to the first axis.

[0148] As shown in Figures 23 and 22(B), the first step is to search for the optimal position by scanning the second axis parameter when the first axis parameter is at the reference position (A = 0); the second step is to search for the optimal position by scanning the second axis parameter when the first axis parameter is at a position to the left of the reference position (A = -11,000 Arcsec); the third step is to search for the optimal position by scanning the second axis parameter when the first axis parameter is at a position to the right of the reference position (A = +11,000 Arcsec); the fourth step is to interpolate the optimal positions found in steps 1 to 3 using a pre-interpolation method and perform a line scan along the A axis parameter to search for the optimal position; and the fifth step is to search for the optimal position by fine line scanning the optimal position found in step 4 along the second axis parameter, which is perpendicular to the first axis. That is, the fine line scan method using linear interpolation involves scanning the second axis parameter at a position that is a reference zero for the first axis parameter, scanning the second axis parameter at a position 3 degrees (-11,000 Arcsec) before the reference position for the first axis parameter, and scanning the second axis parameter at a position 3 degrees (+11,000 Arcsec) behind the reference position for the second axis parameter. The scan positions are set to two positions on opposite sides of the reference position, but they can also be set to three or four or more scan positions. Alternatively, the scan positions are set to three degrees before or after the reference position, but they can also be set selectively within a range of 1 to 7 degrees, or within a range of 3 to 7 degrees. Once at least three scan positions are determined, these three positions can be line-scanned by interpolating them in the first and second axis directions using line interpolation. This linear interpolation scan method can find the optimal position by extracting the first axis parameter and the second axis parameter separately and then combining them.

[0149] The present invention also discloses a system and method that uses a function-based line scan to perform active optical alignment and replace area scans. As shown in the example of FIG. 23, two initial scans can be performed using a linear interpolation method, with a third scan performed to improve accuracy. If a quadratic function occurs at the peak position, a scan for the third position can be performed to establish a non-linear scan path. That is, a quadratic function-based line scan, i.e., a line scan using a non-linear function, can be performed by interpolating three initial line scans to determine the alignment position according to the scan path. Here, the function-based scan method can be generated based on the initial line scan and a physical mechanism. The number of line scans can be determined based on fitting parameters required to describe the peak position. The transmission module and main processor of such a LIDAR device perform a line scan with a time difference in the horizontal direction (i.e., the rotational direction) and a scan interpolated using a non-linear function in the vertical direction, and the sensing module and main processor can determine the optimal position using the scanned data.

[0150] A scanning method using such a nonlinear function algorithm can improve yield, reduce production losses and damages, improve processing time and accuracy for finding the optimum position, and provide a fast scanning method for the optimum position.

[0151] As shown in FIG. 25, the main processor 111 may include a data collection unit 11A, a preprocessor 11B, a histogram circuit 11C, a filter 11D, a timing circuit 11E, a window circuit 11F, a sub-histogram extraction unit 11G, and a waveform analyzer 11H.

[0152] 25 and 26, the data collection unit 11A collects LIDAR raw data, i.e., Geiger-mode APD data, detected by the sensor array 131 of the sensing module 130 (S101). That is, the acquired 3D point cloud data can be collected spatially and temporally using superpixels. The LIDAR device can use a fixed pixel size (angular resolution) and a fixed amount of raw data per point, acquire a wide range of 3D information in the form of a point cloud, and group some points using superpixels.

[0153] The pre-processing unit 11B removes distortions from the collected original data for each pixel. That is, the pre-processing unit 11B can pre-process sampled data to remove noise from the point cloud or to match data acquired at different locations in the field (S102).

[0154] The pre-processing unit 11B classifies and bins events for the raw data of each pixel. Binning is a process of bundling data into groups, dividing the data into manageable sizes, and arranging them in chronological order.

[0155] Here, TOF_bin is calculated using the following formula:

[0156] TOF_bin = (gate_delay - signal_delay) + raw_bin gate_delay: refers to the delay between the time a signal is generated and the time it is received.

[0157] signal_delay: means the delay that occurs while a signal is being transmitted.

[0158] raw_bin: A value obtained by binning the initial time measurement results.

[0159] That is, TOF_bin is a value obtained by correcting the initial time measurement result obtained by taking into account the speed and delay of the signal, and can be processed as a value that enables accurate distance and position measurement. The pre-processing unit 11B outputs TOF bin data aligned with the laser pulse.

[0160] The histogram circuit 11C classifies the raw data into various types of events. The previously calculated TOF bin data is then used to convert the various events into histograms (S103). The entire or downsampled histogram data is then stored in memory. The histogram data stored in memory can be erased immediately after being read.

[0161] The TOF data for each optical pulse output to the pre-processing unit 11B is stored in a buffer or internal memory 41 (FIG. 28) in a FIFO (First-in, first-out) manner (S106), and the buffer or internal memory outputs the TOF bin data in the order they are input. Once the original data is converted into a histogram, a copy of the original data is buffered in the buffer.

[0162] The timing circuit 11E controls the timing of emitting and controlling the laser beam so that the pulse transmission and reception times can be detected, i.e., the timing circuit 11E controls the timing for calculating the distance to an object.

[0163] The window circuit 11F can identify a specific window for performing filtering through the histogram data (S104). The window circuit 11F can accumulate counts of detected photons over a time window corresponding to a pulse subset from a multiple pulse sequence. The window circuit can detect a time interval for sensing pulses and detect data having a specific time length.

[0164] The filter 11D can remove noise from the histogram data or extract meaningful peaks (S105). That is, the filter 11D separates noise and peaks according to the histogram distribution and extracts the signal with the highest peak. The filter 11D can be applied to extract targets from the histogram of avalanche events and the histogram of target events. In this case, the returned target is a window of histogram bins in which the target extraction range is considered to be within the span of the window. That is, the filter can search for and estimate specific targets from given data.

[0165] The sub-histogram extraction unit 11G visualizes the data distribution using a span histogram (S107). That is, the sub-histogram extraction unit 11G divides the histogram data into certain bins using peak data and TOF data, and extracts a graph showing the frequency of data belonging to each bin. This allows the distribution of the entire data as well as the distribution of a specific data bin to be grasped through the histogram data. That is, the sub-histogram extraction unit 11G receives the histogram peak data via the filter 11D and the histogram data copied via the buffer, and extracts more information about the estimation target using high-resolution histogram data in the area containing the peak. The sub-histogram extraction unit 11G can also reconstruct a portion of the high-resolution histogram based on the peak. The sub-histogram extraction unit 11G operates repeatedly to output LIDAR data.

[0166] Once each statistically significant peak is found by the filter, the sub-histogram extraction unit 11G can reconstruct a sub-histogram from the stored raw data at the highest possible time resolution, based on the global maximum position and radial time range defined by the local peak search radius. That is, the sub-histogram extraction unit 11G identifies the sub-histogram of the span in which the target position is estimated. However, because the memory (BRAM) 42 must be cleared to process the next group of histogram data, the histogram cannot be stored until a peak is found. To this end, the preprocessing unit 11B broadcasts the TOF values ​​output by the buffer 41 (FIG. 28) to both the histogram circuit 11C and the buffer 42 so that the sub-histogram extraction unit 11G can use them.

[0167] The waveform analyzer 11C will determine the exact range of targets within the data extracted by the sub-histogram extractor 11G, i.e., the waveform analyzer 11C will output the lidar data with the highest peaks in the streaming interface.

[0168] Here, the sub-histogram extraction unit 11G defines the signal processed by the pre-processing unit 11B and histogram circuit 11C as a first pre-processed output, and the signal processed and buffered via the pre-processing unit 11B and internal memory as a second pre-processed output. The sub-histogram extraction unit 11G detects at least one peak having statistical significance in the first pre-processed output, and a waveform analyzer generates a target waveform based on the correlation between the detected at least one peak and time data points corresponding to the buffered second pre-processed output. Then, the buffered second pre-processed output data is deleted.

[0169] 27 and 28, the histogram data recording and reading operations are as follows: new TOF data is received in a standby state (S111), and it is determined whether the TOF data is valid (S112). If the TOF data is valid, the histogram data is recorded in the internal memory through the first port (S113). That is, the histogram value of the time bin in the internal memory 41 is incremented by 1, read, and recorded, using the same port. If the TOF data is invalid, it is checked whether new TOF data has been received.

[0170] It is determined whether the histogram data recording operation is complete (S114). If complete, the histogram data recorded in the internal memory is read (S115). If not complete, the process moves on to determining whether new TOF data is received. At this time, the histogram read operation reads the histogram values ​​for the bins recorded in the internal memory through the first port (Port A). Then, the read histogram data is erased through the second port (Part B). This operation is repeated until the histogram read operation is completed (S116).

[0171] 28, when TOF data is input from buffer 41, it communicates with internal memory (BRAM) 42 via dual ports (Port A, Port B), reading data through one port and erasing the read data through the other port. In this case, it may take one clock cycle to read the histogram values, rather than two clock cycles. Here, internal memory 42 may be implemented as block RAM, or may be implemented inside main processor 111, i.e., inside FPGA, and histogram data may be read and erased in parallel.

[0172] FIG. 29 is a timing diagram showing the flow of reading and clearing histogram data from the internal memory 42 by the histogram circuit of FIG.

[0173] In FIG. 29, dout_a indicates the frequency at which histogram data belongs to a specific interval, and the frequency of each interval can be expressed as the height of a bar to visualize the data distribution. wren_a indicates the width of each interval, and the histogram divides continuous data into intervals to calculate the frequency, indicating the width of each interval. din_a indicates which interval within the histogram the data belongs to. Such histogram data (hist0 to hist6) may be stored and removed according to the values ​​of bins (bin0 to bin7) based on the clock (clock) and address (addr_a).

[0174] One or more peaks may be determined from the histogram circuit to calculate the TOF of a major peak or the TOF of a minor peak, which may be detected based on the number of event counts in a single bin or after smoothing the collected histogram.

[0175] 30 is a diagram illustrating an oversampling operation for a superpixel according to an embodiment of the present invention. As shown in FIG. 30 and FIG. 25, adjacent superpixels share operating resources. For example, the first through fourth superpixels may undergo buffering by a buffer, preprocessing by a preprocessing unit 11B, histogram processing by a histogram circuit 11C, histogram filtering by a filter 11D, secondary subhistogram extraction by a subhistogram extraction unit 11G, and high-resolution LIDAR data extraction by a waveform analyzer 11H, depending on time.

[0176] The sub-histogram output by the sub-histogram extraction unit 11G can be visualized through a streaming interface as shown in Figure 31. For example, the sub-histogram can be output as a span size (hist(b)-hist(b+11)) for 12 data items. As shown in Figure 32, the histogram filter outputs filtered data along each start bin and 12 histogram data items (hist_word 0-11) according to an alternative approach.

[0177] The main processor 111 spatially and temporally integrates data to generate lidar data, i.e., GmAPD sensing data. The raw lidar data is converted into a histogram before processing. To distinguish between signal and noise in the lidar data, a histogram filter 11D can be applied to the histogram for statistically significant peaks to distinguish between signal and noise. Once the histogram peaks are identified, high-resolution histogram data from the regions contained within the histogram peaks can be used to extract additional information about any target. Due to the high speed of GmAPD data and FPGA resource limitations, optimization is necessary for real-time processing. Therefore, embodiments of the invention include a system and DSP optimization methodology that increases performance without degrading data quality. According to some aspects, a multi-stage histogram method is disclosed that can reconstruct a portion of a high-resolution histogram only in the time region of interest.

[0178] The features, structures, and effects described in the above embodiments are included in at least one embodiment of the present invention and are not necessarily limited to only one embodiment. Furthermore, the features, structures, and effects illustrated in each embodiment may be combined or modified in other embodiments by a person skilled in the art to which the embodiments belong. Therefore, such combinations and modifications should be construed as falling within the scope of the present invention. Furthermore, while the above description focuses on the embodiments, this is merely illustrative and does not limit the present invention. Those skilled in the art will recognize that various modifications and applications not exemplified above are possible within the scope of the present invention without departing from the essential characteristics of the present embodiments. For example, each component specifically illustrated in the embodiments can be modified. Differences related to such modifications and applications should be construed as falling within the scope of the present invention as defined by the appended claims.

Claims

1. a data collection unit that collects raw data generated based on optical signals reflected from the object; a pre-processing unit that removes noise from the collected data; a histogram circuit that converts the output of the preprocessing unit into histogram data; a buffer for buffering an output of the preprocessing unit; a sub-histogram extracting unit that receives the converted histogram data and the buffered data, detects at least one peak from the histogram data, and generates peak data; a waveform analyzer that generates a target waveform based on a correlation between the at least one peak data and the buffered data.

2. The LIDAR device according to claim 1 , wherein the data stored in the buffer is deleted after being recorded by the sub-histogram extractor.

3. The lidar device of claim 1 , further comprising a filter that extracts peaks through the transformed histogram data.

4. 4. The lidar device of claim 3, further comprising a window circuit that identifies a window for performing filtering within the filter on the transformed histogram data.

5. 5. The LIDAR device according to claim 1, wherein the histogram circuit and the sub-histogram extractor record, read, and delete histogram data via an internal memory and a dual port.

6. The LIDAR device of claim 1 , wherein the buffering by the buffer and the preprocessing by the preprocessing unit oversamples adjacent superpixels.

7. The LIDAR device according to claim 1 , wherein the LIDAR device includes a Geiger-mode APD that generates the raw data.

8. The LIDAR device of claim 7 , wherein the LIDAR device is a rotating imaging device.

9. The lidar device of claim 8 , comprising a transmission module having a plurality of vertically arranged light source arrays and a transmission optical system disposed on the output side of the light source arrays.

10. 9. The lidar device of claim 8, wherein the rotating imaging device rotates in synchronization with a master clock of a system within a vehicle.

11. a light source array having a plurality of light sources arranged in at least two rows and generating light pulses; a transmission optical system disposed on the output side of the light source array to refract light toward an object; a sensor array having a plurality of photodetectors for sensing light pulses reflected from a scan area of ​​the object; a receiving optical system disposed on the incident side of the sensor array; 2 of the scan area along the rotation direction ａ *2 ｂ A lidar device including a main processor that captures into a 3D point cloud through oversampling in which (a = 2 to 3, b < a) pixels are acquired and at least one adjacent column of the acquired superpixel is superimposed.

12. The LIDAR device of claim 11 , wherein the LIDAR device has an angle of view of 30 degrees or less.

13. The superpixel is arranged in two rows of 12 pixels, The LIDAR device of claim 11 , wherein the LIDAR device rotates at an angle of 2 mrad per frame.

14. receiving and preprocessing raw lidar data; converting the preprocessed output into a histogram; buffering the preprocessed output; detecting peaks from the histogram-converted output; generating a target waveform based on a correlation between the detected peaks and corresponding time data points in the buffered output.

15. 15. The method of claim 14, further comprising removing the buffered data after reading it.

16. 15. The method for controlling a LIDAR device according to claim 14, wherein the pre-processing step oversamples at least one row of the superpixels acquired in the rotational direction.

17. 15. The method for controlling a LIDAR device according to claim 14, wherein the LIDAR device determines its position through multiple line scans with a time difference in the rotational direction and a scan interpolated by a non-linear function in the vertical direction.

Images