Sensor device, lidar device, and method for controlling the same

Abstract

A LIDAR apparatus disclosed in an embodiment may include: a data collection unit that collects raw data; a preprocessing unit that removes noise from the collected data; a histogram circuit that converts an output of the pre-processing unit into a histogram; a buffer that buffers an output of the pre-processing unit; a sub-histogram extraction unit that receives the converted histogram data and the buffered data, and detects at least one peak value from the histogram; and a waveform analyzer that generates a target waveform based on a correlation between the at least one peak data and the buffered data.

Description

Technical Field

[0001] The present invention relates to a sensor device and a control method thereof. More specifically, the present invention relates to a LiDAR device and a control method thereof. In addition, the present invention relates to a rotational sensor device and a LiDAR device. Background Art

[0002] Autonomous vehicles (AVs) use multiple sensors for situational awareness. Sensors as part of an autonomous driving system (SDS) of an AV can include one or more of cameras, light detection and ranging (LiDAR) devices, and inertial measurement units (IMUs). Sensors such as cameras and LiDAR are used to capture and analyze the surrounding scene of the vehicle. These captured scenes are then used to detect objects, including static objects such as fixed structures and dynamic objects such as pedestrians and other vehicles. In addition, data collected from the sensors can be used to detect conditions such as road markings, lane curvature, traffic signals, and signs. Further, scene representations (such as three-dimensional point clouds obtained from a vehicle's LiDAR) can be combined with one or more images obtained from a camera to provide additional insights into the scene or environment around the vehicle.

[0003] In addition, a LiDAR transceiver can include a transmitter that emits light in the ultraviolet (UV), visible, and infrared spectral regions, and / or one or more photodetectors that convert other electromagnetic radiation into electrical signals. For example, photodetectors can be used in various applications such as fiber optic communication systems, process control, environmental sensing, safety and security, optical sensing, and other imaging applications including ranging applications. Since photodetectors have high sensitivity, they can detect weak signals returned from distant objects. However, the sensitivity to optical signals requires precise alignment between components and alignment of laser emission. Summary of the Invention Technical Problem

[0004] The present invention provides a sensor device and a LiDAR device, comprising: a transceiver device having a transmitter component that emits an optical signal; and a receiver component that receives and processes the optical signal.

[0005] The present invention also provides a sensor device and a LiDAR device that use a fixed pixel size or angular resolution together with the integrated raw data amount per point and are used to improve detection capabilities and data quality (accurate and precise range and intensity).

[0006] The present invention also provides a sensor device and a LiDAR device having a Geiger mode avalanche photodiode (GmAPD) LiDAR system.

[0007] The present invention also provides a method for controlling a sensor device and a LiDAR device, wherein a digital signal processing (DSP) circuit / module / processor integrates data spatially and temporally to generate and process LiDAR data.

[0008] The present invention also provides a method for controlling a sensor device and a LiDAR device, the method transforming a multi-stage histogram to reconstruct a part of a high-resolution histogram in a time domain of interest.

[0009] The present invention also provides a sensor device and a LiDAR device for improving driving performance without degrading data quality.

[0010] The present invention also provides a method for controlling a rotational imaging device such as a sensor device and a LiDAR device to obtain an optical impression by synchronizing the rotational movement of the imaging device with a system clock such as a precision time protocol (PTP).

[0011] The present invention also provides a method for controlling a rotational imaging device such as a sensor device and a LiDAR device to obtain an optical impression using the rotational imaging device, wherein the field of view of the sensor of the rotational imaging device is constant or is controlled for an azimuth angle that remains constant during image acquisition.

[0012] The present invention also provides an imaging device and a method for controlling the imaging device to obtain an optical impression by rotational scanning, wherein the rotational scanning speed and / or angular position of the rotational imaging device minimizes the position drift within the field of view of the device.

[0013] The present invention also provides a device and a method for processing image data acquired by a rotational imaging device, wherein the image data is divided and grouped into two or more segments according to the angular position or the scanning method.

[0014] The present invention also provides a device and a method for controlling a rotational imaging device, capturing image data from a sensor in the rotational imaging device, and controlling the rotational movement of the rotational imaging device to synchronize the sensor in the rotational imaging device with the capture of the image data by a common system clock. Technical solution

[0015] An embodiment of the present invention provides a LiDAR device, including: a data collection unit that collects raw data generated based on light signals reflected from an object; a preprocessing unit that removes noise from the collected data; a histogram circuit that converts the output of the preprocessing unit into histogram data; a buffer that buffers the output of the preprocessing unit; a sub-histogram extraction unit that receives the converted histogram data and the buffered data, and detects at least one peak from the histogram data to generate peak data; and a waveform analyzer that generates a target waveform based on the correlation between at least one peak data and the buffered data.

[0016] In one embodiment, the data stored in the buffer can be recorded by the sub-histogram extraction unit and then deleted.

[0017] In one embodiment, the LiDAR device may further include: a filter that extracts peaks from the converted histogram data.

[0018] In one embodiment, the LiDAR device may further include: a window circuit that identifies a window for performing filtering on the converted histogram data within the filter.

[0019] In one embodiment, the histogram circuit and the sub-histogram extraction unit record, read, and delete histogram data through an internal memory and a dual port.

[0020] In one embodiment, the buffering performed by the buffer and the preprocessing performed by the preprocessing unit oversample a plurality of adjacent superpixels.

[0021] In one embodiment, the LiDAR device may further include: a Geiger mode APD that generates raw data and is a rotating imaging device.

[0022] In one embodiment, the LiDAR device may further include: a plurality of light source arrays arranged vertically; and a transmission module having a transmission optical system disposed on the emission side of the light source arrays.

[0023] In one embodiment, the rotating imaging device rotates synchronously with the master clock of the system in the vehicle.

[0024] In one embodiment of the present invention, a LiDAR device includes: a light source array that generates optical pulses and has a plurality of light sources arranged in at least two rows; a transmitting optical system that is disposed on the emission side of the light source array and refracts light toward an object; a sensor array that senses the optical pulses reflected from the scanned area of the object and has a plurality of photodetectors; a receiving optical system that is disposed on the incident side of the sensor array; and a main processor that acquires 2 a x 2 b (where a = 2 to 3 and b < a) pixels of the superpixels along the rotation direction, and captures a three-dimensional point cloud by oversampling at least one adjacent row in the acquired superpixels.

[0025] In one embodiment, the LiDAR device has a field of view of 30 degrees or less.

[0026] In one embodiment, the superpixels may include 12 pixels arranged in two rows, and the LiDAR device may rotate at an angle of 2 mrad per frame.

[0027] In an embodiment of the present invention, a method for controlling a LiDAR device includes: receiving and preprocessing raw LiDAR data; converting the output of the preprocessing into a histogram; buffering the output of the preprocessing; detecting peaks from the output converted into a histogram; and generating a target waveform based on the correlation between the detected peaks and the time data points corresponding to the buffered output.

[0028] In one embodiment, the method may further include: reading and then deleting the buffered data.

[0029] In one embodiment, the preprocessing may include oversampling at least one row in a plurality of superpixels acquired along the rotation direction.

[0030] In one embodiment, the LiDAR device may determine its position by multiple line scans having a time difference along the rotation direction and scans interpolated by a non-linear function along the vertical direction.

[0031] Based on the following detailed description, the additional scope of application of the present invention will become apparent. However, the detailed description and specific examples represent the preferred embodiments of the present invention and are provided for illustration only. This is because various changes and modifications within the spirit and scope of the present invention will be apparent to those skilled in the art based on the detailed description. The present invention will be more fully understood from the following detailed description and the accompanying drawings, which are provided for illustrative purposes only and are not to be construed as limiting the present invention. Effects of the Invention

[0032] The sensor device and the LiDAR device according to an embodiment of the present invention can improve performance without degrading data quality and provide an optimized autonomous driving device. In addition, by reducing the performance or size of a processor (e.g., FPGA), cost and power consumption can be reduced, and an optimized digital signal processor (DSP) can be provided.

[0033] According to an embodiment of the present invention, an optical impression can be acquired more effectively by a rotational imaging device such as a sensor device and a LiDAR device. The rotational scan speed and the angular position can be adjusted to minimize the field of view of the rotational imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a perspective view of a vehicle including a LiDAR device according to an embodiment of the present invention.

[0035] Figure 2 is a block diagram showing an example of a vehicle system including a Figure 1 LiDAR device.

[0036] Figure 3 is a conceptual diagram for explaining the Figure 2 operation of a LiDAR device.

[0037] Figure 4 is a perspective view showing an example of the arrangement of a transmission module and a sensing module of a Figure 2 LiDAR device.

[0038] Figure 5 is a view showing an example of line scanning in a scanning area of a Figure 3 and Figure 4 LiDAR device shown.

[0039] Figure 6 is a view showing the operation of a transmission module that transmits a pulse for line scanning in a Figure 3 and Figure 4 LiDAR device shown.

[0040] Figure 7 is a view showing the operation of a transmission module and a sensing module that transmit and receive line scanning in a Figure 3 and Figure 4 LiDAR device shown.

[0041] Figure 8 is a view showing a scanning area and a laser spot of a LiDAR device according to a comparative example.

[0042] Figure 9 is a view showing an example of a target area scanned by light emitted from a light source array as an example of a transmission module of a LiDAR device according to the present invention.

[0043] Figure 10 This is an example of the transmission module of the LiDAR device according to the present invention, showing an example of a target area scanned by light emitted from a light source array with defective light sources.

[0044] Figure 11 This is a view showing an example of a pixel array and a superpixel for capturing point clouds in the LiDAR device according to the present invention.

[0045] Figure 12 This is a view for showing a method of oversampling a superpixel for capturing point clouds in the LiDAR device according to the present invention.

[0046] Figure 13 This is a view showing Figure 12 an example of a histogram of oversampled data in

[0047] Figure 14 This is a conceptual diagram showing the division and extraction of a rotational scanning area based on priority in the LiDAR device according to an embodiment of the present invention.

[0048] Figure 15 This is a block diagram showing an example of priority-based data packet processing of the processor of the LiDAR device according to the present invention.

[0049] Figure 16 This is a block diagram showing an example of division of a priority area by a flow control module in the LiDAR device according to the present invention.

[0050] Figure 17 This is a flowchart showing a method of data processing for a priority area in the LiDAR device according to the present invention.

[0051] Figure 18 This is a perspective view showing a multi-axis movable transmission module in another example of the LiDAR device according to the present invention.

[0052] Figure 19 (A) and (B) of are views showing examples of images of scanning profiles based on a linear merit function with respect to the horizontal axis and the vertical axis (A, B) in a comparative example of the LiDAR device.

[0053] Figure 20 (A) and (B) of are views showing examples of image scanning profiles based on a non-linear merit function with respect to the horizontal axis and the vertical axis (A, B) in the LiDAR device according to the present invention.

[0054] Figure 21It is a view showing an image of the area scan profile on the horizontal axis A and the vertical axis B in the LiDAR device according to the present invention.

[0055] Figure 22 It is a conceptual diagram for showing the decoupling of the horizontal axis and the vertical axis by linear interpolation in another example of the LiDAR device according to the present invention.

[0056] Figure 23 It is a flowchart for adjusting the multi-stage scan range in another example of the LiDAR device according to the present invention.

[0057] Figure 24 It is a view for explaining the 3D scan process for the peak position in the scan area in another example of the LiDAR device according to the present invention.

[0058] Figure 25 It is a detailed block diagram of the main processor of the LiDAR device according to an embodiment of the present invention.

[0059] Figure 26 It shows Figure 25 The flowchart of the processing of LiDAR data by the shown main processor.

[0060] Figure 27 It shows Figure 25 The flowchart of the data processing process of the shown main processor.

[0061] Figure 28 It is a flowchart showing the operation of the histogram circuit according to an embodiment of the present invention.

[0062] Figure 29 It is a timing diagram showing an example of the timing diagram for reading and clearing the histogram in the memory according to an embodiment of the present invention.

[0063] Figure 30 It is a view showing the DSP pipeline resources sharing the same resources according to an embodiment of the present invention.

[0064] Figure 31 It is a timing diagram showing the output of the sub-histogrammer for the stream interface according to an embodiment of the present invention.

[0065] Figure 32 It is a timing diagram showing an example of the output of the histogram filter according to an embodiment of the present invention. Detailed Description of the Invention

[0066] In the following, embodiments will be described in detail with reference to the accompanying drawings, where like reference numerals represent like components. However, the present invention can be implemented in various forms and should not be limited to the embodiments illustrated herein only. Instead, these embodiments are provided as examples to ensure that the present disclosure is thorough and complete, and will fully convey the features and functions of the present invention to those skilled in the art. Therefore, processes, elements, and technologies that are not necessary for a full understanding of the features and functions of the present invention may not be described. Unless otherwise stated, like reference numerals in the drawings and the written description represent like components, and thus their descriptions will not be repeated.

[0067] A LiDAR system can be referred to as a depth detection system, a laser ranging system, a lidar system, a LIDAR system, or a laser / optical detection and ranging (LADAR or radar) system. LiDAR is a type of ranging sensor, characterized by its long detection range, high resolution, and low environmental interference. LiDAR is widely used in fields such as intelligent robots, unmanned aerial vehicles, and autonomous or self-driving technologies. The operating principle of LiDAR is based on the round-trip time of electromagnetic waves (e.g., flight time or delay time) between a source and a target to estimate the distance.

[0068] Generally, a LiDAR system (such as a direct time-of-flight (D-TOF) LiDAR system) measures the distance (e.g., depth) to an object by emitting light pulses (e.g., laser pulses) towards the object and measuring the time it takes for the light pulses to be reflected from the object and detected by the sensor of the LiDAR system. For example, to reduce noise from ambient light, repeated measurements can be performed to generate respective histograms of relative time-of-flight (TOF) values based on the repeated measurements, and the peaks of the respective histograms can be calculated to detect events (e.g., to determine the depth of a point or region of the object that reflects the light pulse).

[0069] The above aspects and features of the embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

[0070] Figure 1 is a perspective view of a vehicle including a LiDAR device according to an embodiment of the present invention.

[0071] Reference Figure 1 , a moving object such as vehicle 500 may include a LiDAR device 100, a camera unit 101, vehicle identification sensors 102 and 104, a global positioning system (GPS) sensor 103, a vehicle control module 212, and an ultrasonic sensor 105.

[0072] The LiDAR device 100, as a rotational imaging device or a sensor device, can be combined on a part of the vehicle 500, rotate 360 degrees, perform sensing of the distance, surrounding environment, and shape between the vehicle and an object (static object or dynamic object), and control driving using the measurement data. The objects or environment around the vehicle can be collected and analyzed, and detection data providing information about the objects positioned within an appropriate proximity range can be generated through the three-dimensional point cloud using this sensing technology.

[0073] The LiDAR device 100 can communicate with the vehicle control module 212 and can send and receive information related to vehicle driving. The vehicle control module 212 can communicate with various systems or sensors within the vehicle and can perform various controls. The vehicle control module 212 is a device that controls and monitors various systems of the vehicle and can include a control device such as an electronic control unit (ECU). The vehicle control module 212 can communicate with an external mobile device and can be electrically connected to a removable storage device.

[0075] The camera unit 101 can be installed at one or more positions inside and / or outside the vehicle and can capture images of the front and / or rear of the moving vehicle and can provide or store the images via a display device (not shown). The captured image data can optionally include audio data. In another example, the camera unit 101 can be installed at the front, rear, each corner, or each side of the vehicle 500 to capture the surrounding images of the vehicle and provide the images through a display device (not shown). The vehicle control module 212 or another processor can identify traffic signals, vehicles, and pedestrians based on the data captured by the camera unit 101 to provide the acquired information to the driver. The above camera unit 101 can be used as a driving assistance device.

[0076] The front radar 102 is installed in multiple units at the front of the vehicle 500 and detects the distance between the vehicle 500 and the front object. The rear radar 104 is installed in multiple units at the rear of the vehicle 500 and detects the distance between the vehicle 500 and the rear object. When there is object information detected by these radars 102 and 104, a nearby object or obstacle is detected and the driver is notified through an alarm or warning message.

[0077] The GPS sensor 103 can receive signals from satellites and provide the received signals to devices such as the vehicle control module 212, the LiDAR device 100, and the camera unit 101, and the devices can provide or calculate information such as the position, speed, and time of the vehicle based on the GPS position signals.

[0078] The ultrasonic sensor 105 can perform sensing of the distance to nearby vehicles or obstacles to provide convenience for safely parking the vehicle in a parking space. In addition, the ultrasonic sensor 105 can prevent accidents that may occur during driving. The ultrasonic sensor 105 can be installed at the rear, side, or wheels of the vehicle.

[0080] As Figure 2 shown, a vehicle system 200 including a LiDAR device 100 and a vehicle control module 212 receives input from a user or driver or provides information to the user or driver through a user interface 211. The user interface 211 can include a display device, a touch panel, buttons, voice recognition, a wired or wireless input device, and is connected via a wired or wireless communication to enable communication between the driver and various devices.

[0082] The vehicle system 200 communicates with a remote device 213, which can communicate remotely with a user or an external system or receive an external control signal. The communication unit 215 can support wired or wireless communication and can be, for example, a wired or wireless module.

[0083] The storage unit 220 can include one or more sub-memories 221 therein. In addition, the storage unit 220 can include a portable or removable storage device 222. The LiDAR device 100 can communicate with the user interface 211 and the camera unit 101.

[0085] The LiDAR device 100 can include a drive unit 150 such as a motor, and the drive unit 150 can rotate a part or all of the LiDAR device 100 360 degrees based on a control signal. The drive unit 150 communicates with internal components of the LiDAR device 100 such as a measurement system 110 and enables the LiDAR device 100 to rotate and move around its axis.

[0086] The LiDAR device 100 can include a measurement system 110, a transmission module 120, and a sensing module 130. The drive unit 150 can transmit a driving force to enable the measurement system 110, the transmission module 120, and the sensing module 130 to rotate.

[0087] The measurement system 110 may include a main processor 111 and a main memory 112. The main processor 111 may be implemented as a general-purpose processor, an ASIC (Application Specific Integrated Circuit), one or more FPGAs (Field Programmable Gate Arrays), a set of processing components, or other suitable electronic processing components. The main memory 112 (e.g., memory, memory unit, and storage device) may include one or more devices (e.g., RAM, ROM, flash memory, hard disk memory, etc.) for storing data and / or computer code to complete or facilitate the various processes described in the present invention. The main memory 112 may be or include volatile memory or non-volatile memory. The main memory 112 may include database components, object code components, script components, or any other type of information structure to support the various activities and information structures described in the present invention. According to an embodiment, the main memory 112 may be communicatively connected to the main processor 111.

[0088] The measurement system 110 may include one or more processors (also referred to as central processing units or CPUs). One or more processors may be connected to a communication infrastructure or bus. Additionally, each of the one or more processors may be a graphics processing unit (GPU). In some examples, the GPU may include dedicated electronic circuits designed to process math-intensive applications. The GPU may have an efficient parallel architecture for parallel processing of large chunks of data, such as math-intensive data commonly used in computer graphics applications, images, and videos.

[0089] The measurement system 110, as a computer system, may be connected to one or more user input / output devices, such as a monitor, keyboard, or pointing device, which communicate with the communication infrastructure through a user input / output interface.

[0091] The transmission module 120 may include a light source array 121 and a transmission optical system 123. The transmission module 120 may include a processor or control module, such as a general-purpose processor, ASIC, or FPGA, capable of controlling the operation of the light source array 121 and the transmission of optical signals, and may also include an internal memory storing code for controlling the generation of laser beams.

[0093] The light source array 121 may include a one-dimensional or two-dimensional array and may be individually addressable or controllable. The light source array 121 may include multiple light sources that generate laser beams or light pulses. The light sources may include light sources such as laser diodes (LDs), edge-emitting lasers, vertical-cavity surface-emitting lasers (VCSELs), distributed feedback lasers, light-emitting diodes (LEDs), superluminescent diodes (SLDs), etc. However, it is not limited thereto.

[0094] The light source array 121 may include a plurality of surface-emitting laser diodes of an electrically connected type, such as a VCSEL array, and each transmitter may be individually addressable or controllable. The light source array 121 may be implemented as a one-dimensional (Q*P) VCSEL array or a two-dimensional array having Q rows and P columns. Here, Q and P (rows and columns) are integers greater than or equal to 2 (Q > P). In addition, each VCSEL array may be grouped into a plurality of units to form individual light sources.

[0095] The optical signal emitted from the light source array 121 may be projected toward an object through the transmitting optical system 123. The transmitting optical system 123 may include one or more lenses, or one or more lenses together with a microlens array positioned in front of them. The transmitting optical system 123 may include one or more optical lens elements to shape the laser beam in a desired manner. In other words, the transmitting module 120 may set the irradiation direction or angle of the light generated by the light source array 121 under the control of the main processor 111. In addition, the LiDAR device 100 may include a beam splitter (not shown) to overlap or separate the transmitted laser beam L1 and the received laser beam L2 in the device.

[0097] The transmitting module 120 may irradiate pulsed light or continuous light, and may transmit it toward the target object to be scanned multiple times. The processor 111 generates a start signal during light transmission, and this start signal may be provided to a time-to-digital converter (TDC). The start signal may be used to calculate the time of flight (TOF) of the light.

[0099] The sensing module 130 may include a sensor array 131 and a receiving optical system 133. The sensing module 130 may perform the conversion of an original histogram, including a matched filter, a peak detection circuit, an SPAD saturation and quenching circuit, and also includes a processor that performs at least one of compensating for pulse shape distortion. The processor may be implemented as a general-purpose processor, an ASIC (application-specific integrated circuit), one or more FPGAs (field-programmable gate arrays), a set of processing components, or other suitable electronic processing elements. The sensing module 130 may include a memory (not shown) having one or more devices (e.g., RAM, ROM, flash memory, hard disk memory, etc.) for internally storing the detected optical signals.

[0101] The sensor array 131 may receive the laser beam L2 reflected or scattered from the object through the receiving optical system 133. The sensor array 131 may include a detector divided into a plurality of pixels, and light detection elements may be provided on each of the plurality of pixels. The receiving optical system 133 may be an optical element for focusing the reflected light onto a predetermined pixel.

[0102] When the reflected light is received by the sensor array 131, the sensing module 130 may convert the reflected light into a stop signal. The stop signal, together with the start signal, may be used to calculate the TOF of the light. The sensor array 131 or the sensing module 130 may include a time-to-digital converter (TDC) to measure the TOF of the light detected by each of the plurality of photodetectors. The photodetector may be a light receiving element that generates an electrical signal based on the detected light energy, and may be, for example, a single photon avalanche diode (SPAD) having high sensing sensitivity.

[0103] The sensor array 131 may be implemented as a one-dimensional or two-dimensional array, and may be a set of photodetectors such as SPADs (single photon avalanche diodes) or single photon detectors (APDs, avalanche photodiodes). An embodiment of the present invention may be implemented using single photon photodetectors. The sensor array 131 measures light pulses, i.e., the light corresponding to the image pixels passing through the receiving optical system 133. In addition, the sensor array as an SPAD array may be arranged as a two-dimensional SPAD array having M rows and N columns. Here, M and N are integers of 2 or greater. In addition, each of the SPAD sub-arrays may be grouped into a plurality to form a light sensor. The sensor array may operate in Geiger mode, i.e., gated-mode APD (GmAPD).

[0105] The main processor 111 processes the signal using the light detected by the sensing module 120 to obtain information about the object. Based on the TOF of the light reflected from the object, the main processor 111 determines the distance to the object and performs data processing for analyzing the position and shape of the object. The information analyzed by the processor 111, such as the shape and position of the object, may be transmitted to other devices.

[0106] The transmitting optical system 123 refracts the light pulses generated by the light source array 121 to project them towards the object, and the light pulses are incident on the surface of the object and reflected from the surface of the object, and the reflected light pulses are sensed by the sensor array 131 through the receiving optical system 133. Based on the time (TOF: Time of Flight) elapsed from the emission of the light pulse to the detection of the reflected light pulse, the distance or depth to the object can be determined.

[0108] When the light detection element includes an SPAD, the sensing sensitivity is high, but the noise also increases. To calculate the reliable TOF of light using the SPAD, the main processor 111 emits light toward the object multiple times, generates a histogram of the reflected light from the object, and then performs statistical analysis on the histogram. The main processor 111 can calculate the distance to a point on the object surface based on the TOF of the light and can generate a point cloud based on the distance data. The generated point cloud data can be stored in a database within the main memory 111. The point cloud data stored in the database can be converted into a three-dimensional shape and stored as a three-dimensional shape. The point cloud data can be processed by the preprocessing unit 11B (refer to Figure 25 ) to preprocess the data suitable for the purpose and remove noise. Noise such as distortion caused by reflection in a special environment (such as glass) can be automatically detected and eliminated.

[0110] The LiDAR device 100 can combine one or more image data obtained from the camera unit 101 with an image representation such as the acquired point cloud to obtain an additional view of the scene or situation around the vehicle.

[0112] As Figure 3 shown, the LiDAR device 100 can use the angle of the emitted laser beam to determine the position of the object. For example, when the angle of the laser beam emitted from the LiDAR device 100 toward the scanning area T0 is known, the LiDAR device 100 can use the sensor array 131 to detect the laser beam reflected from the object in the scanning area T0 and can determine the position of the object based on the emission angle of the laser beam. In addition, the LiDAR device 100 can use the angle of the received laser beam to determine the position of the object. For example, when the distances of the first object and the second object from the LiDAR device 100 are the same but they are located at different positions from the LiDAR device 100, the laser beam reflected from the first object and the laser beam reflected from the second object can be detected at different points on the sensor array 131. In other words, the LiDAR device 100 can determine the position of the object based on the point on the sensor array 131 where the reflected laser beam is detected.

[0113] The LiDAR device 100 may have a scanning area T0 including an object for detecting the positions of any surrounding objects. The scanning area T0 represents the detectable area as a single frame, which may be described as a set of points, lines, or planes forming a screen during one frame. In addition, the scanning area may indicate the irradiation area of the laser beams emitted from the LiDAR device 100, and the irradiation area may indicate a set of points, lines, or planes where the laser beams emitted during one frame intersect the spherical surface at the same distance. In addition, the field of view (FOV) may indicate the detectable area (field) and may be defined as the angular range of each scanning area of the LiDAR device 100 when the LiDAR device is regarded as the origin.

[0115] To expand the scanning area T0, the light source array 131 may operate in a line or plane configuration. For example, when multiple light sources are arranged in a line or plane configuration, the directions and sizes of the laser beams emitted by these light sources can be changed through drive control, thereby allowing the scanning area T0 of the LiDAR device 100 to be expanded in a line or plane configuration.

[0116] In another example, the transmitting module 120 or the transmitting optical system 123 may include a scanning unit (not shown). The scanning unit may change the directions and / or sizes of the laser beams in the light source array 121, thereby allowing the LiDAR device 100 to expand its scanning area or change the scanning direction. The scanning unit may expand or change the scanning area of the dot-shaped laser beam into a line or plane configuration. Although one or more scanning units may be provided, embodiments of the present invention are not limited thereto.

[0118] The LiDAR device 100 may provide a single-photon avalanche photodetector (SAPD) or Geiger-mode LiDAR including a light source array 121 and a sensor array 131. The Geiger mode may be highly sensitive to single photons and may transmit a space-filling (gapless) image. In other words, a space-filling image refers to an image in which the detector has a recovery time (the time required for the detector to return to the photon detection state to detect the next photon after detecting one photon), and this recovery time is represented as the time difference or gap between the detected photons.

[0119] The photodetector is defined as an APD, where a reverse voltage is applied to induce avalanche multiplication, thereby providing internal gain within the component, and due to the current amplification effect caused by the avalanche effect in the diode, the APD provides a high signal-to-noise ratio and a higher quantum efficiency compared to other detectors. The APD can be classified into Geiger mode and linear mode. In the linear mode, the applied reverse voltage is less than the breakdown voltage, and the magnitude of the output current is linearly proportional to the intensity of the input light, and has a gain characteristic proportional to the magnitude of the reverse voltage. The Geiger mode detects single photons by applying a reverse voltage greater than the breakdown voltage.

[0121] In addition, the Geiger mode according to the present invention can be operated in a short-time operation Geiger mode, generate a digital output signal in response to a single-photon-level input, and has a gain area more than 100 times larger than that of the linear mode. The reception sensitivity in the linear mode is 100 photons / 1nsec, while the reception sensitivity in the Geiger mode is 1 photon / 1nsec, enabling single-photon detection (SAPD).

[0122] In addition, the laser beam used in the Geiger mode according to the present invention utilizes an infrared wavelength exceeding 1000 nm, such as in the wavelength band above 1400 nm or 1550 nm, which is higher than the wavelength below 900 nm commonly used in the linear mode. The Geiger mode operates a light source array and a sensor array to send a uniform space-filling imaging, i.e., gapless imaging. The wavelength exceeding 1400 nm is considered eye-safe, helping to alleviate problems related to beam divergence or distance limitation.

[0124] Due to the light source array 121 having multiple emitters and the sensor array 131 having multiple photodetectors, the Geiger mode according to an embodiment of the present invention is insensitive to faults or defects of individual light sources. As Figures 4 to 6 shown, the LiDAR device 100 can integrally include a transmission module 120 and a sensing module 130. The LiDAR device 100 includes the transmission module 120 vertically arranged on the front side of one surface of the fixed frame 300, and on the emission side of the transmission module 120, the transmission optical system 123 can be arranged in the vertical direction. The transmission module 120 can be connected to the transceiver interface board 125 on its lower side. The driver circuit 120A can be arranged on one side of the transmission module 120.

[0125] The LiDAR device 100 includes the sensing module 130 at the rear side of one surface of the fixed frame 300, and a part of the sensing module 130 (i.e., a part of the receiving optical system 133) can penetrate the inner wall frame of the fixed frame 300 and be supported by it. The circuit board 135 is coupled to the rear of the sensing module 130 and can be electrically connected to the sensor array 131.

[0126] The fixed frame 300 of the LiDAR device 100 includes a heat dissipation plate 301 on a part or the whole of another surface, and the heat dissipation plate 301 includes a plurality of heat dissipation fins for dissipating the heat generated by the transmission module 120 and the sensing module 130.

[0127] The LiDAR device 100 may include a transparent protection plate 129 to protect a part of the transmission module 120 and the sensing module 130. The protection plate 129 may be disposed on one side of the transmission module 120 and may be spaced apart from the front portion (i.e., the light incident area) of the sensing module 130.

[0128] The LiDAR device 100 has a line configuration of the vertical light source array 121 and the transmission optical system 123, and the transmission module 120 and the sensing module 130 are arranged to face an object in the same direction. The LiDAR device 100 rotates 360 degrees about the rotation axis of the drive motor coupled to its lower side.

[0130] In the light source array 121, a predetermined number of light sources may be arranged in one or more rows in the vertical direction, such as 48 light sources arranged in two rows. These arranged light sources are set on a one-dimensional plane and can emit light laterally from the LiDAR device 100. The laser beam emitted by the light source array 121 passes through the transmission optical system 123, which is configured to focus the light beam on the scanning area through a pattern such as a microlens array and lock in the desired wavelength. The driver circuit 120A may utilize short high-current pulses (e.g., 3 - 6 ns). Here, for a pulse of about 5 ns, the transmitted pulse energy range may be about 2 μJ.

[0132] As Figure 6 shown, the profile of the light beam emitted by the transmission module 120 forms a linear pattern SR1, as Figure 5 shown, and the light beam divergence angle R1 may be less than or equal to 1 mrad, and the vertical elevation angle H1 may be less than or equal to 30 degrees. The output of these light sources can be adjusted so that the emitted light beam can be provided as uniformly distributed illumination towards the object through the light source array 121. The point cloud obtained using such a light source array 121 and the sensor array 131 can be classified as vertically space-filling.

[0133] In addition, the transmission optical system 123 is separated or spaced apart from the light source array 121 to affect the shape of the light emitted from the light source array 121, and may be set as a monolithic optical element in the emission field. The transmission optical system 123 can be aligned to project the output of each light source onto the entire field.

[0135] AsFigure 7 As shown, the one-dimensional light source array 121 of the transmitting module 120 and the one-dimensional sensor array 131 of the sensing module 130 can be correspondingly aligned. Therefore, the LiDAR device 100 can use the data obtained from the scanning area SR1 as superpixels.

[0136] LiDAR points form a point cloud, and superpixels created by grouping some points within the point cloud can provide greater meaning or information. These point clouds include information about the shape and distance of objects. Superpixels group the LiDAR point cloud into more meaningful units, clearly defining the boundaries of specific objects or structures by grouping adjacent points and dividing them into larger regions, thereby enabling more accurate object recognition and classification.

[0138] As Figure 8 shown in (A), (B), and (C), the linear mode of the comparative example uses scattered laser spots and detectors, creating gaps between the imaging points. In other words, the linear mode has large non-illuminated (or non-imaged) regions (e.g., gaps) in the scene that are not captured by the sensor due to the scattered laser beam. Additionally, the scattered laser spots may be more sensitive to faults or defects in individual emitters. In the linear mode, the divergence angle is 2mrad, and the scattered laser spots at a distance of 200m form a scanning area of 40cm, and each pixel can be detected at a divergence angle of 2mrad. The linear mode of the comparative example may not provide uniform illumination because gaps or inconsistencies occur in the output of the laser diode array due to faults or failures of individual emitters.

[0140] As Figure 9 shown, the transmitting optical system 123 in an example of the present invention can disperse each laser beam into uniform illumination across the entire target area. That is, all areas of the target can be illuminated with uniform output light. As Figure 10 shown, when some light sources #4 fail or are disabled (not activated), the laser beams emitted by different light sources can be projected onto the entire target area through the transmitting optical system 123. Therefore, the photosensitivity received across the target area can maintain a substantially uniform distribution. This configuration can ensure that even when some light sources are not activated, the overall photosensitivity remains uniform, the power reduction is minimized, and there is no gap loss in the light field, resulting in a negligible deterioration in the sensitivity of the received light. This LiDAR device can minimize the impact of individual light source failures on the data to improve data reliability. Additionally, even if some light sources are defective, immediate repair may not be required.

[0141] In addition, the present invention can use superpixels to optimize the trade-off between signal and resolution. In other words, by adopting superpixels, sufficient signal strength can be ensured while maintaining high resolution, enabling more accurate measurement and imaging.

[0143] The LiDAR device 100 can use a transmission module 120 having a vertical light source array 121 and a transmission optical system 123, and a sensing module 130 having a receiving optical system 133 and a sensor array 131 to achieve horizontal space filling by overlapping the measured point cloud data in the rotation direction.

[0144] The light source array 121 can include a plurality of light sources arranged in at least two rows. For example, the light source array 121 can be arranged in a QxP arrangement (Q and P are integers greater than or equal to 2), and for example, can be arranged as 2 m x2 n array, where m is 6 or greater, for example, in the range of 6 to 10, preferably 9, and n is less than m and 2 or less, for example, 1. Here, Q represents the number of vertically arranged light sources, and P represents the number of horizontally arranged light sources.

[0145] The light source array 121 emits one-dimensional illumination and can cover the entire scene by rotation. In the illumination scan area, a predetermined area can be defined as a superpixel, such as 2 a x2 b pixel configuration, where a represents the number of vertical pixels, which can be less than m and in the range of 2 to 3 (preferably approximately 2.60, excluding decimals), and b represents the number of horizontal pixels, which can be less than or equal to a or equal to n (preferably 1).

[0146] Here, the vertical elevation angle of the superpixel can be 2 mrad (e.g., 0.11 degrees). The divergence angle R1 of the superpixel and the scan area SR1 can be the same, and at the target distance (e.g., 200 m), the measurement error or size of the superpixel can be less than or equal to 16 cm (i.e., ∼16 cm@200 m) or at most 14 cm (∼14 cm@200 m).

[0148] Horizontal spatial filling can be achieved by oversampling in the horizontal (or rotational) direction at a high laser pulse rate. Here, oversampling refers to sampled data where adjacent superpixels overlap in at least one row. 10 or more pulses are emitted per frame, such as 13 pulses, resulting in 156 pixel samples. In other words, a single frame can be derived from 12 pixels * 13 pulses, where the 12 pixels can be used as a point cloud. Additionally, when the superpixels are arranged in a single row, the measurement is performed 85 times (512 / 6), but when in two rows, 12 pixels and 13 measurements result in 156 samples (= 12 * 13). This measured sample data can be defined as a single point cloud.

[0149] The LiDAR device 100 can have a transmitting optical system 123 and / or a receiving optical system 133 with a FOV of 30 degrees or less, thereby achieving full-scene coverage through rotation. For example, with a single laser pulse, the measurement can achieve a rotational accuracy of up to 3 cm at a distance of 200 m. Additionally, using the oversampling technique, the LiDAR device can capture a three-dimensional point cloud of the surrounding environment. In other words, the LiDAR device can effectively detect nearby objects around vehicles at short and long distances for safety detection.

[0150] As Figure 13 shown in (A) of Figure 13 , 12 pixels can be measured 13 times, where each frame is rotationally scanned at an angle of less than 0.12 degrees (i.e., 2 mrad), and the obtained data can be represented as histogram bins, as

[0152] shown in (B) of Figure 14 .

[0153] The LiDAR device 100 can divide the FOV into at least two regions A1 and A2 and allocate separate resources between the two regions. This division can be performed during acquisition, such as grouping or splitting the data into multiple groups or strips based on the position or range of the acquired data. In other words, the first region A1 can be a high-priority region or a more relevant region that requires more detailed analysis in terms of processing. At least one other region, such as the second region A2, can be a low-priority region, and a relatively less frequent processing is applied to this low-priority region.

[0154] The second area A2 with low priority can be used for data recording purposes. In contrast, the first area A1 with high priority can be monitored by a sub-standby controller that does not need to monitor all areas. That is, this allows for more efficient use of computing resources to monitor the first area A1. Under normal operation, the first area A1 with high priority can correspond to the front area of the vehicle. However, when dividing the FOV in the second area A2 with different priorities, the high-correlation area can be adjusted to face a meaningful direction, that is, it is important for the high-correlation area to maintain the correct pointing angle.

[0155] The LiDAR data packets obtained from the first area A1 with high priority are routed to the multicast communication port. Among the LiDAR data packets within the multicast angle range defined by the start angle R st and the stop angle R sp can be routed to the multicast UDP / IP endpoint instead of the unicast UDP / IP endpoint, as Figure 15 shown.

[0156] The present invention can provide a method for ensuring and stabilizing the pointing angle as follows. For example, as shown in (A) of Figure 15 , the main processor 111 can communicate the LiDAR data packets with the network via the first communication port (UDP / IP port 1) 33A and the second communication port (UDP / IP port 2) 33B, and can send or receive LiDAR data packets, that is, 3D data. That is, the multicast function can be used to achieve sensor data partitioning or grouping. The multicast function generally allows a subset of unicast LiDAR data packets (e.g., UDP packets) to be routed to alternative entities (e.g., IP addresses and UDP ports). In addition, the LiDAR data packets received from the second area A2 outside the azimuth angle range can be unicast together with other data (such as Geiger-mode avalanche photodiode (“GmAPD”) data and status packets). In another example, different areas can be sector segments of a circular area.

[0158] As Figure 16As shown, data partitioning can be implemented by the flow control module 136, which generates groups or strips from the raw data generated by the LiDAR device 100. The flow control module 136 can use criteria such as azimuth to form at least two data groups from the raw data received by the LiDAR device. The raw data can refer to data from the readout interface of the imaging device (such as the readout interface circuit (ROIC) 137). The range of a given group or sample can be specified by setting limits for the corresponding region, which are defined by two azimuth values specifying a start limit and a stop limit. Alternatively, a single azimuth value can be defined, and then the number of sensor frames calculated from the azimuth value can be specified. The flow control module 136 and the ROIC 137 can be integrated into a processor.

[0160] The rotational scanning movement of the LiDAR device can be synchronized with the master clock of a mobile object system or a vehicle system. For example, the rotational scanning movement of the LiDAR device 100 can be synchronized with the master clock of a system such as the Precision Time Protocol (“PTP”) grandmaster in an SDS (Autonomous Driving System).

[0161] Alternatively or additionally, the present invention allows at least one shutter control between the cameras of a vehicle (e.g., an autonomous vehicle (AV)) to be synchronized with the master clock of the system or the rotational scanning movement of the LiDAR device. For example, in the camera unit 101 (refer to Figure 2 ) or the sensor, there may be a global shutter, and the global shutter can be synchronized with the LiDAR scan movement. Thus, the camera unit 101 can be controlled to ensure that the shutter of the camera unit 101 is synchronized with the rotation of the LiDAR device 100. For this purpose, the timing or clock of the LiDAR device 100 can be synchronized with PTP. Similarly, the shutter signal of the camera unit 101 can be synchronized with the PTP of the SDS. This ensures that the images captured by the camera unit 101 are synchronized with the capture representation (form or method) of the LiDAR device 100, thereby improving the combination of LiDAR data and camera images. Therefore, the synchronization between the data output stack (e.g., 3D map) of the LiDAR device and the vision stack (video from the camera unit) of the autonomous vehicle can be improved, thereby improving the environmental detection function of the autonomous driving system (SDS), and the computational bandwidth can be improved by aligning the two output stacks.

[0162] Additionally or alternatively, in the present invention, the pointing angle of the camera unit 101 and / or the LiDAR device 100 can operate in an azimuth lock mode that is controlled over time. The azimuth lock mode is a mode of fixing at a specific azimuth angle or angle to collect data. In this mode, the LiDAR device can rotate to a specified azimuth angle to collect data, thereby more precisely acquiring or tracking information in a specific direction. The field of view of the LiDAR device in the azimuth lock mode can be fixed in azimuth relative to a fixed reference plane. That is, the azimuth angle of the LiDAR sensor can be controlled within a predetermined value or range. The present invention can suppress or prevent position drift at the pointing angle of the LiDAR device (e.g., a mechanical optical sensor such as LIDAR) by controlling the azimuth lock mode. Position drift refers to a change in position or direction, indicating a deviation from the target position or direction. Therefore, without the azimuth lock mode, position drift may cause an undesired rotation and may cause movement to an undesired pointing angle. Therefore, the present invention allows for more accurate adjustment of the consistent pointing angle of the LiDAR, which is a rotational imaging device. In addition, the LiDAR device can more accurately capture a desired target or area, thereby maintaining the stability and accuracy of the system.

[0164] More specifically, the measurement system 110 (refer to Figure 2 ) or the main processor 111 of the LiDAR device 100 can control the rotation speed and / or the angular position to minimize position drift within an angle range where the pointing angle is stable. Additionally, the measurement system 110 (refer to Figure 2 ) or the main processor 111 can synchronize the rotation caused by the drive unit 150 (refer to Figure 2 ) with the PTP time. For example, during operation, the azimuth angle can be controlled within ±5°. Preferably, the azimuth phase angle can be controlled within ±2°, and the azimuth angle is within a range suitable for accurately capturing the position and direction of the front area of the vehicle.

[0165] As Figure 17 shown, a method for improving the image data obtained by the LiDAR device includes: measuring the angular position of the rotatable part in operation S121, acquiring an optical impression through the image sensor of the sensor array based on the measured data in operation S123, dividing the acquired imaging data according to priorities in operation S125, and capturing an image of the priority area in the divided area based on PTP or FOV in operation S127. The division of the acquired imaging data can group and divide the imaging data based on the angular position or the scanning range of the acquired data. When capturing the optical impression using the image sensor, the capture is performed in response to a trigger signal, and the trigger signal can be generated in response to the angular position being aligned with a reference point.

[0166] In addition, as a method for synchronizing the capture of acquired imaging data with the rotational movement of a rotational imaging device (i.e., a LiDAR device), the rotational movement and the capture of imaging data can be adjusted or synchronized by the same system clock. Additionally, as a method for obtaining an optical impression using a LiDAR as a rotational imaging device, the rotational movement can be adjusted or synchronized with a system clock such as the Precision Time Protocol. The FOV of the LiDAR device can be controlled with respect to the azimuth angle, and for example, when acquiring imaging data, the azimuth angle can be set or maintained consistently. In this case, the rotational scan speed of the LiDAR device and / or the angular position of the device can be controlled to minimize position drift within the FOV.

[0168] As Figure 18 As shown, the transmission module 120 according to an embodiment of the present invention includes: a light source unit 121A having a light source array 121; and a transmission optical system 123 having a collimating lens 123A and a volume Bragg grating (VBG) lens 123B, all of which are arranged on a substrate 120A. The collimating lens 123A may have a convex curved surface shape on the emission side and extend longitudinally along the long axis Y of the light source array 121. The collimating lens 123A may refract incident light in parallel, and the VBG lens may directly emit or focus light of a specific wavelength and, in particular, may adjust the optical path in a desired direction.

[0169] The high sensitivity of the sensor array 131 allows the sensor array 131 to detect weak signals returned from distant objects, but the high sensitivity to optical signals can provide precise alignment of components within the laser pulse emission path of the transmission module 120. Therefore, the LiDAR device 100 and its control method proposed according to the present invention provide active optical alignment of the transmission module 120 and the sensing module 130 to improve production tolerances and increase manufacturing yields.

[0170] To this end, the light source unit 121A having the light source array 121, the collimating lens 123A, and the VBG lens 123B of the transmitting module 120 can adjust multiple linear and rotational axes x / y / z / Rx / Ry / Rz to find the optimal area scan or angular position. The collimating lens 123A and / or the VBG lens 123B can include motors on either side of each axis to move along the three-dimensional axes X, Y, and Z, or can be rotatably connected about the respective axes X, Y, and Z by Rx, Ry, and Rz. Therefore, independent drive control of each component can be performed to find the optimal position of each axis, enabling area scanning to be achieved. That is, active alignment is required to find the optimal position of the components in the transmitting optical system, and the two main challenges associated with alignment may be as follows. 1) Defining the optimal position using non-independent parameters, and 2) effectively reaching the optimal position in a coupled axis system. Here, multiple axes need to be adjusted to find the optimal position for active alignment, and there is a strong axis coupling effect under actual production conditions, which may be caused by limitations of the alignment equipment, process deviations, or internal physical mechanisms of the optical system.

[0171] The present invention provides at least two main methods for solving these technical problems: 1) a new algorithm for identifying the optimal position; 2) an effective scanning method. These methods contribute to achieving high yield and fast processing time.

[0173] Regarding the area scanning method for finding the optimal position in a LiDAR device according to an embodiment of the present invention, reference can be made to Figures 19 to 24 . Figure 19 (A) and (B) of show examples of the application of an area scanning algorithm based on a non-linear figure of merit function, and Figure 20 (A) and (B) of show examples of an area scanning algorithm based on a linear figure of merit function. In Figure 19 (A) and (B) of, the parameters of the first axis (A axis) and the second axis (B axis) achieve performance within specific specifications. For example, performance can be achieved under the condition of the specification that the first axis parameter > 0.88 and the second axis parameter > 30 dB. In addition, in the presence of external noise (refer to (B) of Figure 19 ), due to the direction of the driving force, the impedance force to noise can be increased.

[0174] As Figure 20As shown in (A) and (B), when applying a scanning algorithm based on a linear merit function, it can be known that in the scanning area, within the parameters of the vertical first axis (A axis) and the horizontal second axis (B axis), the position deviates from the specific specifications. For example, the scanning position deviates from the specifications (A axis parameter > 0.88, B axis parameter > 30 dB), but instead falls within A axis parameter < 0.88 and / or B axis parameter < 30 dB. That is, the linear merit function does not provide independent configuration for various parameters required to quantify the optical performance, making it difficult to independently and actively align the parameters. In other words, when assigning weights to one parameter, other parameters may deviate from the specifications, and when using the gradient descent method to perform area scanning to find the optimal position, the impedance force against external noise may be reduced. These problems make it challenging to determine the optimal parameters for active alignment.

[0175] When performing area scanning using a non - linear merit function, the present invention can provide better performance. Specifically, the present invention avoids applying excessive weight to a single specific parameter, ensures that area scanning can be performed while achieving performance within the specifications at the selected position, and performs area scanning in the appropriate direction of applying the driving force, thereby being robust to external noise and enabling the calculation of the optimal position.

[0177] Figure 21 is a view showing an example of thin - line scanning according to an embodiment of the present invention. In other words, thin - line scanning is performed after area scanning to find the final position. This method for area scanning involves a trade - off between time and accuracy. When a fine scanning step is required to achieve high accuracy, the scanning process becomes very time - consuming, and improving the slow scanning time by increasing the step size reduces the accuracy.

[0178] As Figure 22 shown in (A), the area scanning method of scanning the positions along the second axis (B axis) for each first axis (A axis) may increase the scanning step size or time, thereby reducing the scanning efficiency. As Figure 22 shown in (B), the present invention identifies the position by scanning along the combined direction of the first axis and the second axis (A axis and B axis) through a linear interpolation method. Linear interpolation can construct the surface shape into a grid structure and can generate a digital surface model of the grid, thereby enabling the collection of more accurate spatial and distance information. Here, the first axis can represent the vertical direction of the scanning area, and the second axis can be the horizontal direction orthogonal to the first axis.

[0179] As Figure 23 and Figure 22As shown in (B), in the first step, the second axis parameter can be scanned to identify the optimal position when the first axis parameter is at the reference position (A = 0). In the second step, the second axis parameter can be scanned to identify the optimal position when the first axis parameter is to the left of the reference position (A = -11,000 arcseconds). In the third step, the second axis parameter can be scanned to identify the optimal position when the first axis parameter is to the right of the reference position (A = +11,000 arcseconds). In the fourth step, linear interpolation can be used to interpolate the optimal positions obtained from the first to the third steps, scanning along the A-axis parameter to identify the optimal position. And in the fifth step, a fine line scan can be performed on the optimal position obtained in the fourth step along the second axis parameter perpendicular to the first axis to identify the optimal position. In other words, the fine line scanning method using linear interpolation scans the second axis parameter at the reference position (0) of the first axis parameter, and scans the second axis parameter at a position 3 degrees before the reference position (-11,000 arcseconds), and scans the second axis parameter at a position 3 degrees after the reference position (+11,000 arcseconds). Although in this case the scanning positions are set on two opposite sides of the reference position, three or more scanning positions can be set. In addition, the scanning positions are set as angles before or after the reference position, but the scanning positions can be selectively set within the range of 1 to 7 degrees, or selectively performed within the range of 3 to 7 degrees. After obtaining at least three scanning positions, linear interpolation is applied to these three positions to perform interpolation along the directions of the first axis and the second axis for line scanning. This scanning method using linear interpolation extracts the first parameter and the second axis parameter respectively, and combines the parameters to find the optimal position.

[0181] In addition, the present invention discloses a system and method that perform active optical alignment and use a function-based line scan operation instead of area scanning. As Figure 23 shown, similar to the linear interpolation method, initial scans are performed at two positions, and a third scan can be performed to improve accuracy. When there is a quadratic function at the peak position, a scan for the third position can be performed, and a non-linear scan path can be set. That is, a quadratic function-based line scan interpolated within the initial 3-line scan (i.e., a line scan using a non-linear function) can determine the alignment position along the scan path. Here, a function-based scan method can be generated based on the initial line scan and physical mechanisms. The number of line scans can be determined by the fitting parameters required to describe the peak position. The transmission module and the main processor of the LiDAR device perform time-differentiated line scans and vertical interpolation scans in the horizontal (i.e., rotational) direction using a non-linear function, and can use the scan data to determine the optimal position.

[0182] The scanning method using the non - linear function algorithm can increase the yield, reduce manufacturing losses and damages, and enhance the processing time and the accuracy for determining the optimal position. In addition, the scanning method can provide a fast scan for optimal positioning.

[0184] As Figure 25 shown, the main processor 111 may include a data acquisition unit 11A, a pre - processor 11B, a histogram circuit 11C, a filter 11D, a timing circuit 11E, a window circuit 11F, a sub - histogram extraction unit 11G, and a waveform analyzer 11H.

[0185] Referring Figure 25 and Figure 26 , in operation S101, the data acquisition unit 11A collects the raw LiDAR data detected by the sensor array 131 of the sensing module 130, that is, Geiger - mode APD data. In other words, the acquired 3D point - cloud data can be collected as super - pixels both spatially and temporally. The LiDAR device can use a fixed pixel size (angular resolution) and a fixed amount of raw data per point to acquire a wide range of 3D information in the form of a point cloud and group some points into super - pixels.

[0186] The pre - processor 11B removes the distortion in the raw data of each collected pixel. Specifically, in operation S102, the pre - processor 11B can remove noise from the point cloud or can pre - process the sampled data for the intended purpose to align the data acquired from different positions.

[0187] The pre - processor 11B classifies the events in the raw data of each pixel and performs binning. Binning refers to the process of grouping data into clusters, which means dividing the data into manageable sizes and sorting the data in chronological order.

[0188] Here, TOF_bin is calculated using the following formula:

[0189] TOF_bin=(gate_delay - signal_delay)+raw_bin

[0190] gate_delay: Indicates the delay between the time when the signal is generated and the time when the signal is received.

[0191] signal_delay: Indicates the delay that occurs when the signal is transmitted.

[0192] raw_bin: The value obtained by binning the initial time - measurement results.

[0193] In other words, the TOF_bin is a value obtained by correcting the initial time measurement result by considering the speed and delay of the signal, and can be processed into a value capable of precise distance or position measurement. The pre-processor 11B outputs TOF bin data aligned with the laser pulse.

[0195] The histogram circuit 11C classifies the raw data into various types of events. Here, using the previously calculated TOF bin data, the raw data is converted into a histogram of different events in operation S103. The entire histogram data or the downsampled histogram data is stored in the memory. In addition, once the histogram data stored in the memory is read, the histogram data can be cleared.

[0196] In operation S106, the TOF data of each optical pulse output from the pre-processor 11B is stored in a buffer or the internal memory 41 (refer to Figure 28 ) by the FIFO (First In First Out) method, and the buffer or the internal memory outputs the TOF bin data in its input order. When the raw data is converted into a histogram, a copy of the raw data is cached.

[0197] The timing circuit 11E controls the timing of the emission and the timing of the control of the laser beam to detect the transmission and reception times of the pulses. In other words, the timing circuit 11E controls the timing for calculating the distance to the object.

[0198] In operation S104, the window circuit 11F identifies a specific window for performing filtering through the histogram data. The window circuit 11F can accumulate the count of photons detected across a time window corresponding to a subset of pulses from a multi-pulse sequence. The window circuit can detect the time interval for pulse detection and can detect data with a specific time length.

[0199] In operation S105, the filter 11D can remove noise from the histogram data or extract meaningful peaks. Specifically, the filter 11D distinguishes noise or peaks based on the histogram distribution to extract the signal with the highest peak. The filter 11D can be applied to extract a target from the histograms of avalanche events and desired events. Here, the returned target is a histogram bin window whose extraction range considered to be the target is within the span of the window. That is, the filter finds and estimates a specific target from the given data.

[0201] In operation S107, the sub-histogram extraction unit 11G uses a span histogram to visualize the data distribution. That is, the sub-histogram extraction unit 11G divides the histogram peak data and the TOF data into specific bins and extracts a graph representing the frequency of the data within each bin. This allows for simultaneous understanding of the distribution shape in a specific section of the histogram data and the overall data distribution. That is, the sub-histogram extraction unit 11G receives the peak data from the filter 11D and the histogram data copied via the buffer, uses the high-resolution histogram data in the region including the peak, and extracts more information about the estimation target. Additionally, the sub-histogram extraction unit 11G can reconstruct a part of the high-resolution histogram based on the peak. The sub-histogram extraction unit 11G can repeat the operation to output LiDAR data.

[0202] The sub-histogram extractor 11G can reconstruct the sub-histogram within the global maximum position and the radial time range, where when the filter detects each peak with statistical significance, the stored raw data is defined by the local peak detection radius at the highest possible time resolution. That is, the sub-histogram extraction unit 11G identifies the sub-histogram of the span where the target position is estimated. However, in order to process the next set of histogram data, it is necessary to clear the memory (BRAM) 42, so the histogram may not be stored until a peak is found. To solve this problem, the pre-processor 11B broadcasts the TOF value, which is the output of the buffer 41 (refer to Figure 28 ) to both the histogram circuit 11C and the buffer 42, enabling the sub-histogram extraction unit 11G to use the TOF value.

[0205] The waveform analyzer 11C determines the exact range of the target within the data range extracted by the sub-histogram extraction unit 11G. In other words, the waveform analyzer 11C outputs the LiDAR data with the highest peak through the stream interface.

[0207] Here, regarding the sub-histogram extraction unit 11G, the signal processing passing through the pre-processor 11B and the histogram circuit 11C can also be defined as the first pre-processing output, and the signal processing buffered and output via the pre-processor 11B and the internal memory can be defined as the second pre-processing output. The sub-histogram extraction unit 11G can detect at least one peak with statistical significance within the first pre-processing output, and the waveform analyzer can generate a target waveform based on the correlation between the at least one detected peak and the time data points corresponding to the buffered second pre-processing output. Subsequently, the buffered second pre-processing output data is deleted.

[0209] As Figure 27 and Figure 28As shown, for the recording and reading operations of histogram data, in operation S111, new TOF data is received in the standby state, and in operation S112, it is determined whether the TOF data is valid. When the TOF data is valid, in operation S113, the histogram data is recorded in the internal memory through the first port. Specifically, the histogram value of the time bin in the internal memory 41 is read and recorded while incrementing by 1, and the same port is used for both operations. When the TOF data is invalid, it is checked whether new TOF data is received.

[0210] In operation S114, it is checked whether the histogram data recording operation is completed, and when it is completed, in operation S115, the histogram data recorded in the internal memory is read, and when it is not completed, it is checked whether new TOF data is received. Here, during the histogram reading operation, the histogram value recorded in the internal memory is read through the first port (port A). Moreover, the read histogram data is deleted through the second port (port B). In operation S116, this process is repeated to complete the histogram reading operation.

[0211] As Figure 28 shown, when TOF data is input from the buffer 41, communication with the internal memory (BRAM) 42 can be performed through a dual port (port A and port B) to read data through one port and delete the read data through the other port. In this case, the reading of the histogram value can occur within one clock cycle without two clock delays. Here, the internal memory 42 can be implemented as a block RAM (BRAM), and can be implemented within the main processor 111 (i.e., FPGA), and the histogram data can be processed in parallel for reading and deletion.

[0212] Figure 29 is a timing diagram showing the process of reading and deleting the histogram data in the internal memory 42 via the Figure 28 histogram circuit in.

[0213] In Figure 29 , dout_a can represent the frequency of the histogram data belonging to a specific interval, and the frequency of each interval is represented as the bar height to visualize the data distribution. wren_a represents the width of each interval, and the histogram calculates the frequency by dividing into intervals of continuous data and represents the width of each interval. din_a represents which interval within the histogram the data belongs to. The histogram data hist0 to hist6 can be stored and deleted based on the clock cycle and the address addr_a through the values of the bins bin0 to bin7.

[0214] One or more peaks can be determined from the histogram circuit, and one or more peaks can be determined to calculate the TOF of the main peak or the TOF of the secondary peak. Peaks can be detected based on the number of event counts in a single bin, or after smoothing the collected histogram.

[0216] Figure 30 is a view showing an oversampling operation for superpixels according to an embodiment of the present invention. As Figure 30 and Figure 25 shown, multiple adjacent superpixels share operation resources. For example, the first to fourth superpixels can be performed by buffering over time through a buffer, preprocessing through a preprocessor 11B, histogram operation through a histogram circuit 11C, histogram filtering through a filter 11D, secondary sub-histogram extraction through a sub-histogram extraction unit 11G, and high-resolution LiDAR data extraction through a waveform analyzer 11H.

[0217] The sub-histogram output from the sub-histogram extraction unit 11G can be visualized through a stream interface, as Figure 31 shown, and the interval width of 12 data can be output, that is, the span size (hist(b) to hist(b + 11)). As Figure 32 shown, the histogram filter follows an alternative method, outputting filtered data and 12 histogram data points (hist_word0 - 11) along each starting bin.

[0219] The main processor 111 integrates spatial and temporal data to generate LiDAR data, that is, the sensed data of the GmAPD. The original LiDAR data is converted into a histogram before processing. To distinguish the signal from the noise in the LiDAR data, the histogram filter 11D is applied to the histogram of statistically significant peaks, thereby allowing the separation of the signal and the noise. In addition, once the histogram peaks are distinguished, high-resolution histogram data of the regions included in the histogram peaks can be used to extract additional information about any target for estimation. Due to the high speed of the GmAPD data and the FPGA resource limitations, real-time processing requires optimization. Therefore, embodiments of the present invention include systems and DSP optimization methods that improve performance without degrading data quality. In some aspects, a multi-step histogram method is disclosed that allows partial reconstruction of a high-resolution histogram only in the region of interest.

[0220] The features, structures, and effects described in the above embodiments are included in at least one embodiment of the present invention and are not necessarily limited to a single embodiment. In addition, the features, structures, and effects illustrated in each embodiment can be combined or modified by those skilled in the art for other embodiments to achieve. Therefore, such combinations and modifications should be construed as being included within the scope of the present invention. In addition, although the embodiments have been mainly described above, they are merely examples and do not limit the present invention, and those skilled in the art can recognize that various modifications and applications not illustrated above are possible without departing from the essential features of the present embodiment. For example, each component specifically described in the embodiment can be implemented in a modified form. And the differences related to these modifications and applications should be construed as being included within the scope of the present invention defined by the appended claims.

Claims

1. A LiDAR device, comprising: a data collection unit that collects raw data generated based on light signals reflected from an object; a preprocessing unit that removes noise from the collected data; a histogram circuit that converts the output of the preprocessing unit into histogram data; a buffer that buffers the output of the preprocessing unit; a sub-histogram extraction unit that receives the converted histogram data and the buffered data, and detects at least one peak from the histogram data to generate peak data; and a waveform analyzer that generates a target waveform based on the correlation between the at least one peak data and the buffered data.

2. The LiDAR device according to claim 1, wherein the data stored in the buffer is recorded by the sub-histogram extraction unit and then deleted.

3. The LiDAR device according to claim 1, comprising a filter that extracts peaks through the converted histogram data.

4. The LiDAR device according to claim 3, comprising a window circuit that identifies a window for performing filtering on the converted histogram data within the filter.

5. The LiDAR device according to any one of claims 1 to 4, wherein the histogram circuit and the sub-histogram extraction unit record, read, and delete histogram data through an internal memory and a dual port.

6. The LiDAR device according to any one of claims 1 to 4, wherein the buffering performed by the buffer and the preprocessing performed by the preprocessing unit oversample adjacent superpixels.

7. The LiDAR device according to any one of claims 1 to 4, comprising a Geiger mode APD that generates the raw data.

8. The LiDAR device according to claim 7, wherein the LiDAR device is a rotating imaging device.

9. The LiDAR device according to claim 8, further comprising: a plurality of light source arrays arranged vertically; and a transmission module having a transmission optical system disposed on the emission side of the light source array.

10. The LiDAR device according to claim 8, wherein the rotating imaging device rotates synchronously with the master clock of the system in the vehicle.

11. A LiDAR device, comprising: a light source array that generates light pulses and has a plurality of light sources arranged in at least two columns; a transmission optical system disposed on the emission side of the light source array to refract light toward an object; a sensor array that senses light pulses reflected from a scanned area of the object and has a plurality of photodetectors; a reception optical system disposed on the incident side of the sensor array; and A main processor, the main processor acquiring 2 a x 2 b (where a = 2 to 3, b < a) pixels along the rotation direction, and capturing a three-dimensional point cloud by performing overlapping oversampling on at least one adjacent column among the obtained superpixels.

12. The LiDAR device according to claim 11, wherein the LiDAR device has a field of view of 30 degrees or less.

13. The LiDAR device according to claim 11, wherein, the superpixel is arranged by 12 pixels in two columns, and the LiDAR device rotates at an angle of 2 mrad per frame.

14. A control method for a LiDAR device, comprising the following steps: receiving and preprocessing raw LiDAR data; converting the output of the preprocessing into a histogram; buffering the output of the preprocessing; detecting peaks from the output converted into the histogram; and generating a target waveform based on the correlation between the detected peaks and the time data points corresponding to the buffered output.

15. The method according to claim 14, including reading and then deleting the buffered data.

16. The method according to claim 14, wherein, the preprocessing step includes oversampling at least one column of a plurality of superpixels obtained along the rotation direction.

17. The method according to claim 14, wherein, the LiDAR device determines the position by a plurality of line scans having a time difference along the rotation direction and scans interpolated by a non-linear function along the vertical direction.

Images