Method for generating color image by using lidar device, and lidar device using same

Abstract

A method for generating a color image by using a LiDAR device, according to the present invention, may comprise: acquiring LiDAR data; estimating the direction of an external light source by using ambient values of a plurality of pixels; estimating surface information for each of the plurality of pixels by using pixel position coordinates and distance values of the plurality of pixels; generating a geometric intensity value for each of the plurality of pixels on the basis of the direction of the external light source and the surface information for each of the plurality of pixels; generating a corrected intensity value for each of the plurality of pixels on the basis of an intensity value for each of the plurality of pixels and the geometric intensity value for each of the plurality of pixels; and generating a color image by inputting, to a colorization model, LiDAR data composed of a plurality of pixels of which pixel values are the corrected intensity values.

Description

Method for generating color images using a LiDAR device and a LiDAR device using the same

[0001] The present invention relates to a method for generating a color image, and more specifically, to a method for generating a color image using a lidar device including a laser detecting array and a lidar device using the same.

[0002]

[0003] Recently, LiDAR (Light Detection and Ranging) has been gaining prominence alongside interest in autonomous and driverless vehicles. LiDAR is a device that uses lasers to acquire surrounding distance information; thanks to its superior precision and resolution, as well as its ability to perceive objects in three dimensions, it is increasingly being applied in various fields, including not only automobiles but also drones and aircraft.

[0004] Meanwhile, a solid-state LiDAR device is a device capable of acquiring distance information about a three-dimensional surrounding space without mechanical moving components, and a laser output array can be used to implement the solid-state LiDAR device.

[0005] However, since LiDAR devices are designed to detect the distance to an object using lasers in a wavelength band where sunlight noise is minimized, it is physically difficult to acquire light in a wavelength band corresponding to the visible light band.

[0006] Consequently, it is not easy to generate color images using data acquired from LiDAR devices.

[0007] Furthermore, in fields requiring higher safety standards, such as autonomous vehicles, various sensors capable of acquiring data with different characteristics are currently deployed together and used complementarily.

[0008]

[0009] One objective of the present invention is to provide a method for generating a color image using a lidar device including a laser detecting array.

[0010] The problems to be solved by the present invention are not limited to those described above, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the attached drawings.

[0011]

[0012] According to one embodiment of the present invention, a method for generating a color image using a LiDAR device comprising a detecting element array comprises: acquiring LiDAR data—wherein the LiDAR data is composed of a plurality of pixels, wherein each of the plurality of pixels includes pixel position coordinates and at least one pixel value, and the at least one pixel value includes a distance value, an intensity value, and an ambient value—estimating the direction of an external light source using the ambient values ​​of the plurality of pixels—wherein the direction of the external light source includes a light source direction vector—estimating surface information for each of the plurality of pixels using the pixel position coordinates and distance values ​​of the plurality of pixels—wherein the surface information for each of the plurality of pixels includes a normal vector—generating a geometric intensity value for each of the plurality of pixels based on the direction of the external light source and the surface information for each of the plurality of pixels, generating a corrected intensity value for each of the plurality of pixels based on the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels—and a LiDAR composed of a plurality of pixels in which the pixel value is the corrected intensity value. A method for generating a color image may be provided, comprising inputting data into a colorization model to generate a color image—wherein the color image is composed of a plurality of color pixels, and each of the plurality of color pixels includes pixel location coordinates and at least one color channel value.

[0013] According to another embodiment of the present invention, a method for generating a color image using a LiDAR device comprising a detecting element array comprises: acquiring LiDAR data—wherein the LiDAR data is composed of a plurality of pixels, each of the plurality of pixels includes pixel position coordinates and at least one pixel value, and the at least one pixel value includes a distance value, an intensity value, and an ambient value—estimating the direction of an external light source using the ambient values ​​of the plurality of pixels—wherein the direction of the external light source includes a light source direction vector—estimating surface information for each of the plurality of pixels using the pixel position coordinates and distance values ​​of the plurality of pixels—wherein the surface information for each of the plurality of pixels includes a normal vector—generating a geometric intensity value for each of the plurality of pixels based on the direction of the external light source and the surface information for each of the plurality of pixels; generating a corrected intensity value for each of the plurality of pixels based on the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels; and based on the ambient value of each of the plurality of pixels A method for generating a color image may be provided, comprising: generating a weighted intensity value for each of the plurality of pixels by adjusting each corrected intensity value; and inputting LiDAR data composed of a plurality of pixels, wherein the pixel value is the weighted intensity value, into a colorization model to generate a color image—wherein the color image is composed of a plurality of color pixels, and each of the plurality of color pixels includes pixel position coordinates and at least one color channel value.

[0014] The means for solving the problem of the present invention are not limited to the means for solving the problem described above, and unmentioned means for solving the problem will be clearly understood by those skilled in the art from this specification and the attached drawings.

[0015]

[0016] According to one embodiment of the present invention, a method for generating a color image using a lidar device including a laser detecting array may be provided.

[0017] The effects of the present invention are not limited to those described above, and unmentioned effects will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

[0018]

[0019] FIG. 1 is a drawing for explaining a lidar device disclosed by the present application.

[0020] FIG. 2 is a drawing for explaining the VICSel disclosed by the present application.

[0021] Figure 3 is a diagram illustrating the limitations of SPAD.

[0022] Figure 4 is a diagram illustrating an approach to determining the flight time of a laser using electrical signals output from a SPAD.

[0023] Figure 5 is a diagram illustrating the number of electrical signals output after the same amount of time has elapsed from each laser output point when a laser is output multiple times from a lidar device.

[0024] FIG. 6 is a drawing for explaining the histogram disclosed in the present application.

[0025] FIG. 7 is a drawing for explaining the generation of a histogram in a lidar device disclosed in the present application.

[0026] FIG. 8 is a diagram illustrating the determination of an echo signal based on a histogram in a lidar device.

[0027] FIG. 9 is a diagram illustrating various examples for expanding the area where a laser output from a lidar device is irradiated.

[0028] FIG. 10 is a drawing for explaining a lidar device disclosed by the present application.

[0029] Figure 11 is a diagram illustrating the laser irradiation area of ​​a lidar device.

[0030] Figure 12 is a diagram illustrating the horizontal and vertical angles of view of the laser irradiation area of ​​a fixed lidar device.

[0031] Figure 13 is a diagram illustrating the light detection area of ​​a lidar device.

[0032] Figure 14 is a diagram illustrating the horizontal and vertical angles of view of the light detection area of ​​a fixed lidar device.

[0033] FIG. 15 is a drawing for explaining lidar data disclosed through the present application.

[0034] FIG. 16 is a drawing for explaining the point cloud disclosed in the present application.

[0035] FIG. 17 is a drawing for explaining the enhanced point cloud disclosed in the present application.

[0036] FIG. 18 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0037] FIG. 19 is a drawing for explaining a method for estimating the direction of an external light source according to one embodiment.

[0038] FIG. 20 is a diagram illustrating a method for estimating surface information for each of a plurality of pixels according to one embodiment.

[0039] FIG. 21 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0040] FIG. 22 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0041] FIG. 23 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0042] FIG. 24 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0043] FIG. 25 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0044] FIG. 26 is a drawing for illustrating result images according to another modified embodiment.

[0045]

[0046] The embodiments described in this specification are intended to clearly explain the concept of the invention to those skilled in the art to which the invention pertains, and therefore the invention is not limited to the embodiments described in this specification, and the scope of the invention should be interpreted to include modifications or variations that do not deviate from the concept of the invention.

[0047] The terms used in this specification have been selected to be as widely used as possible, taking into account their functions in the present invention; however, these terms may vary depending on the intent of those skilled in the art to which the present invention pertains, case law, or the emergence of new technologies. However, if a specific term is defined and used with an arbitrary meaning, the meaning of that term will be described separately. Accordingly, the terms used in this specification should be interpreted based on their actual meaning and the content throughout this specification, rather than merely their names.

[0048] The drawings attached to this specification are intended to facilitate the explanation of the present invention. The shapes depicted in the drawings may be exaggerated as necessary to aid in understanding the present invention, and therefore the present invention is not limited by the drawings.

[0049] When an element or layer described in this specification is referred to as being "on" or "on" another element or layer, it may include not only being directly on top of another element or layer, but also cases where another layer or other component is interposed in between.

[0050] Throughout this specification, the same reference numbers may, in principle, represent the same components.

[0051] Numbers used in the description of this specification (e.g., first, second, etc.) may be understood as identification symbols to distinguish one component from another component.

[0052] The suffixes "module" and "part" for components used in the description of this specification are used or interchangeably for ease of drafting the specification and may not have distinct meanings or roles in themselves.

[0053] In this specification, if it is determined that a specific description of known configurations or functions related to the present invention may obscure the essence of the present invention, such detailed description will be omitted as necessary.

[0054] According to one embodiment of the present invention, a method for generating a color image using a LiDAR device comprising a detecting element array comprises: acquiring LiDAR data—wherein the LiDAR data is composed of a plurality of pixels, wherein each of the plurality of pixels includes pixel position coordinates and at least one pixel value, and the at least one pixel value includes a distance value, an intensity value, and an ambient value—estimating the direction of an external light source using the ambient values ​​of the plurality of pixels—wherein the direction of the external light source includes a light source direction vector—estimating surface information for each of the plurality of pixels using the pixel position coordinates and distance values ​​of the plurality of pixels—wherein the surface information for each of the plurality of pixels includes a normal vector—generating a geometric intensity value for each of the plurality of pixels based on the direction of the external light source and the surface information for each of the plurality of pixels, generating a corrected intensity value for each of the plurality of pixels based on the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels—and a LiDAR composed of a plurality of pixels in which the pixel value is the corrected intensity value. A method for generating a color image may be provided, comprising inputting data into a colorization model to generate a color image—wherein the color image is composed of a plurality of color pixels, and each of the plurality of color pixels includes pixel location coordinates and at least one color channel value.

[0055] Here, estimating the direction of the external light source may include: identifying at least one object using ambient values ​​of the plurality of pixels; identifying at least one shadow associated with the at least one object using ambient values ​​of the plurality of pixels; and obtaining the light source direction vector for the external light source based on the at least one object and the at least one shadow associated with the at least one object.

[0056] Here, estimating surface information for each of the plurality of pixels may include: obtaining point data corresponding to each of the plurality of pixels; selecting a pixel group for each of the plurality of pixels; estimating a plane for each of the plurality of pixels based on the point data of the pixel group selected for each of the plurality of pixels, and obtaining a normal vector of the estimated plane.

[0057] Here, the maximum value range of the geometric intensity value may be the same as the maximum value range of the intensity value.

[0058] Here, generating the geometric intensity value includes generating the geometric intensity value of (M,N) pixels, wherein the light source direction vector is L_light, the normal vector of the (M,N) pixels is N_(M,N), and the geometric intensity value of the (M,N) pixels is I_geo(M,N), the following relationship may be satisfied.

[0059] [Relationship]

[0060] I_geo(M,N) = L_light N_(M,N) * Maximum value of intensity

[0061] Here, generating the corrected intensity value may include generating the corrected intensity value by assigning weights to the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels.

[0062] Here, generating the corrected intensity value includes generating the corrected intensity value of (M,N) pixels, wherein if the intensity value of (M,N) pixels is I_raw(M,N) and the corrected intensity value of (M,N) pixels is I_amd(M,N), the following relationship may be satisfied.

[0063] [Relationship]

[0064] I_amd(M,N) = a * I_raw(M,N) + (1-a) * I_geo(M,N)

[0065] Here, the colorization model includes at least one artificial neural network layer, wherein the at least one artificial neural network layer may include at least one of a feedforward neural network, a radial basis function network or a Kohonen self-organizing network, a convolutional neural network (CNN), a recurrent neural network (RNN), a Long Short Term Memory Network (LSTM), or Gated Recurrent Units (GRUs).

[0066] Here, the colorization model may be a model trained using a training dataset that takes black and white images as input data and color images as output data.

[0067] According to another embodiment of the present invention, a method for generating a color image using a LiDAR device comprising a detecting element array comprises: acquiring LiDAR data—wherein the LiDAR data is composed of a plurality of pixels, each of the plurality of pixels includes pixel position coordinates and at least one pixel value, and the at least one pixel value includes a distance value, an intensity value, and an ambient value—estimating the direction of an external light source using the ambient values ​​of the plurality of pixels—wherein the direction of the external light source includes a light source direction vector—estimating surface information for each of the plurality of pixels using the pixel position coordinates and distance values ​​of the plurality of pixels—wherein the surface information for each of the plurality of pixels includes a normal vector—generating a geometric intensity value for each of the plurality of pixels based on the direction of the external light source and the surface information for each of the plurality of pixels; generating a corrected intensity value for each of the plurality of pixels based on the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels; and based on the ambient value of each of the plurality of pixels A method for generating a color image may be provided, comprising: generating a weighted intensity value for each of the plurality of pixels by adjusting each corrected intensity value; and inputting LiDAR data composed of a plurality of pixels, wherein the pixel value is the weighted intensity value, into a colorization model to generate a color image—wherein the color image is composed of a plurality of color pixels, and each of the plurality of color pixels includes pixel position coordinates and at least one color channel value.

[0068]

[0069] I. LiDAR Device

[0070] [Overview of LiDAR Devices]

[0071] The disclosure of the present application relates to a solid-state LiDAR.

[0072] A LiDAR (Light Detection and Ranging) device is a device that uses lasers to measure the distance between the device and an object.

[0073] More specifically, a lidar device is a device for measuring the distance between the lidar device and an object by emitting a laser and detecting the laser reflected from the object.

[0074] Generally, to measure the distance from a lidar device to an object, the time of flight (TOF) for the round-trip flight path of the laser output from the lidar device between the lidar device and the object is used.

[0075] Accordingly, the lidar device includes a laser emitting element for outputting a laser and a detecting element for detecting a received laser, and includes at least one processor for determining the time interval between the time when the laser is output and the time when the laser is detected.

[0076] Below, the components included in the lidar device will be described in more detail.

[0077]

[0078] FIG. 1 is a drawing for explaining a lidar device disclosed by the present application.

[0079] Referring to FIG. 1, the lidar device (1000) disclosed by the present application may include a laser emitting element (1010).

[0080] [Laser light-emitting device]

[0081] Function of laser light-emitting devices

[0082] A laser emitting element (1010) is configured to generate and emit a laser.

[0083] At this time, the laser generated and output from the laser emitting element (1010) may be light of a specific wavelength that has monochromaticity.

[0084] In addition, when the LiDAR device (1000) disclosed by the present application determines the distance to an object using a Direct TOF (Time-of-Flight) method that determines the flight time based on the flight start time and flight end time of the laser, the laser emitting element (1010) is configured to output a pulsed laser. At this time, the pulsed laser refers to a laser that is emitted for a short period of time, and typically, the emission duration of the pulsed laser can be designed to be about 1 to 20 ns, but is not limited thereto.

[0085] Types of laser light-emitting devices

[0086] The types of laser light-emitting elements (1010) can vary.

[0087] For example, the laser emitting element (1010) may be an edge-emitting laser (EEL) and a vertical cavity surface emitting laser (VCSEL), but is not limited thereto, and may be various types of elements configured to generate and output a laser.

[0088] Typically, in the field of technology for solid-state LiDAR devices, a VCSEL is used as a laser-emitting element (1010).

[0089] Therefore, the following explains the Vixel in more detail.

[0090] FIG. 2 is a drawing for explaining the VICSel disclosed by the present application.

[0091] VCSEL stacking structure

[0092] The VIXEL (1100) may include an upper reflective layer (1110), a lower reflective layer (1120), an active layer (1130) interposed between the upper reflective layer (1110) and the lower reflective layer (1120), an upper electrode (1140) in contact with the upper reflective layer (1110), and a lower electrode (1150) in electrical contact with the lower reflective layer (1120).

[0093] Structural features of the upper reflection layer

[0094] The upper reflective layer (1110) may be a distributed Bragg reflector (DBR). That is, the upper reflective layer (1110) may be a reflective layer having a multilayer structure in which two materials with different refractive indices are alternately stacked.

[0095] For example, the upper reflective layer (1110) may be a reflective layer having a multi-layer structure in which a first material layer (1111) having a first refractive index and a second material layer (1112) having a second refractive index are alternately stacked.

[0096] In this case, the fact that a multilayer structure, in which two materials with different refractive indices are alternately stacked, can function as a reflective layer for light in a specific wavelength band can be physically explained by Fresnel reflection and constructive and destructive interference of light, and the reflectivity of the multilayer structure increases as the number of alternating stacks of two materials with different refractive indices increases.

[0097] In addition, the upper reflective layer (1110) can be doped with a specific type.

[0098] For example, the upper reflective layer (1110) can be doped with P-type or N-type.

[0099] At this time, the fact that the upper reflective layer (1110) is doped with a specific type means that all of the multiple layers included in the multilayer structure of the upper reflective layer (1110) are doped with a specific type.

[0100] For example, if the doping type of the upper reflective layer (1110) is P type, both the first material layer (1111) and the second material layer (1112) included in the upper reflective layer (1110) are doped with P type, and if the doping type of the upper reflective layer (1110) is N type, both the first material layer (1111) and the second material layer (1112) included in the upper reflective layer (1110) are doped with N type.

[0101] Structural features of the lower reflection layer

[0102] The lower reflective layer (1120) may be a dispersed Bragg reflector. That is, the lower reflective layer (1120) may be a reflective layer having a multilayer structure in which two materials with different refractive indices are alternately stacked.

[0103] For example, the lower reflective layer (1120) may be a reflective layer having a multi-layer structure in which a third material layer (1121) having a third refractive index and a fourth material layer (1122) having a fourth refractive index are alternately stacked.

[0104] In addition, the lower reflective layer (1120) can be doped with a specific type.

[0105] For example, the lower reflective layer (1120) can be doped with P-type or N-type.

[0106] At this time, the fact that the lower reflective layer (1120) is doped with a specific type means that all of the multiple layers included in the multilayer structure of the lower reflective layer (1120) are doped with a specific type.

[0107] For example, if the doping type of the lower reflective layer (1120) is P type, both the third material layer (1121) and the fourth material layer (1122) included in the lower reflective layer (1120) are doped with P type, and if the doping type of the lower reflective layer (1120) is N type, both the third material layer (1121) and the fourth material layer (1122) included in the lower reflective layer (1120) are doped with N type.

[0108] The relationship between the upper reflective layer and the lower reflective layer

[0109] The upper reflective layer (1110) and the lower reflective layer (1120) of the VIXEL (1100) are doped with different types.

[0110] For example, if the doping type of the upper reflective layer (1110) of the VXEL (1100) is P type, the doping type of the lower reflective layer (1120) of the VXEL (1100) is N type, and if the doping type of the upper reflective layer (1110) of the VXEL (1100) is N type, the doping type of the lower reflective layer (1120) of the VXEL (1100) is P type.

[0111] In addition, the reflectance of the upper reflective layer (1110) and the lower reflective layer (1120) of the Vixel (1100) is different from each other.

[0112] For example, the reflectance of the upper reflective layer (1110) of the VIXEL (1100) may be higher than the reflectance of the lower reflective layer (1120), and the reflectance of the upper reflective layer (1110) may be lower than the reflectance of the lower reflective layer (1120).

[0113]

[0114] WXEL's operating principle

[0115] When a predetermined voltage is applied between the upper electrode (1140) and the lower electrode (1150) of the VCSEL (1100), electrons and holes in the upper reflective layer (1110) and the lower reflective layer (1120) move and combine in the active layer (1130), and accordingly, light is generated in the active layer (1130).

[0116] For example, the doping type of the upper reflective layer (1110) of the VCSEL (1100) is P type and the doping type of the lower reflective layer (1120) is N type, and a predetermined voltage is applied between the upper electrode (1140) and the lower electrode (1150), and when the voltage applied to the upper electrode (1140) is greater than the voltage applied to the lower electrode (1150), holes in the upper reflective layer (1110) move to the active layer (1130), and electrons in the lower reflective layer (1120) move to the active layer (1130), and light is generated by the recombination of holes and electrons in the active layer (1130).

[0117] Additionally, for example, the doping type of the upper reflective layer (1110) of the VCSEL (1100) is N type and the doping type of the lower reflective layer (1120) is P type, and a predetermined voltage is applied between the upper electrode (1140) and the lower electrode (1150), and when the voltage applied to the lower electrode (1150) is greater than the voltage applied to the upper electrode (1140), electrons in the upper reflective layer (1110) move to the active layer (1130), and holes in the lower reflective layer (1120) move to the active layer (1130), and light is generated by the recombination of holes and electrons in the active layer (1130).

[0118] Additionally, the light generated in the VXEL (1100) is output in the direction where the reflective layer with the lower reflectivity is located between the upper reflective layer (1110) and the lower reflective layer (1120).

[0119] For example, light generated in the active layer (1130) of the VIXEL (1100) is incident on the upper reflective layer (1110) or the lower reflective layer (1120), is reflected alternately in the upper reflective layer (1110) and the lower reflective layer (1120), and then passes through the reflective layer with lower reflectivity among the upper reflective layer (1110) and the lower reflective layer (1120) and is output.

[0120]

[0121] Referring again to FIG. 1, the lidar device (1000) disclosed by the present application may include a detecting element (1020).

[0122] [Detecting element]

[0123] Function of the detecting element

[0124] The detecting element (1020) is configured to generate an electrical signal in response to the received light when light is received.

[0125] At this time, the electrical signal output from the detecting element (1020) may be an analog signal having a value corresponding to the intensity of the received light, or a digital signal corresponding to whether or not the light is received.

[0126] Types of detecting elements

[0127] In addition, the types of the detecting element (1020) can be varied.

[0128] For example, the detecting element (1020) may be a PD (Photo detector), an APD (Avalanche Photo Diode), a SAPD (Single Photon Avalanche Diode), or a SiPM (Silicon Photomultiplier), but is not limited thereto, and may be various types of elements configured to output an electrical signal in response to received light.

[0129] Typically, a SPAD is used as a detecting element (1020) in the technical field of a solid-state LiDAR device.

[0130] Therefore, SPAD will be explained in more detail below.

[0131] Operation of outputting an electrical signal in response to the received light of the SPAD

[0132] SPAD is a semiconductor-based photodetector that is a device in which current flows even when a small amount of photons are absorbed by applying a reverse bias voltage greater than the breakdown voltage, causing an avalanche breakdown phenomenon, and is also called a Geiger mode APD.

[0133] In this case, since the SPAD undergoes avalanche breakdown even when absorbing a small amount of photons, it is necessary to suppress excess current. Therefore, to suppress excess current, the SPAD requires a quenching circuit that inhibits the continuous flow of current.

[0134] The Quenching Circuit is configured to instantaneously reduce the magnitude of the voltage applied to the SPAD after the avalanche breakdown phenomenon occurs in the SPAD.

[0135] At this time, when the magnitude of the voltage applied to the SPAD by the Quenching Circuit is reduced, the flow of current in the SPAD is cut off.

[0136] After this, even if photons reach the SPAD, no current flows in the SPAD until the magnitude of the voltage applied to the SPAD is restored.

[0137] That is, the SPAD operates through an avalanche breakdown stage, a quenching stage, and a charging stage by absorbing a small amount of photons while a reverse voltage greater than the breakdown voltage is applied. In the avalanche breakdown stage, a current of a certain magnitude flows, and in the quenching stage and charging stage, no current flows even if photons are absorbed.

[0138] Therefore, during the quenching and charging phases, even if light is received, no electrical signal is output in response to the received light, and this is typically referred to as dead time.

[0139] Such dead time can range from several hundred ps to several us depending on the material of the SPAD or the configuration of the quenching circuit.

[0140] Advantages and Limitations of SPAD

[0141] SPAD has the advantage of being able to detect even when only a small amount of photons are returned to the lidar device as the laser output from the lidar device is reflected from an object located at a distance, because current flows even when a small amount of photons are absorbed.

[0142] However, SPADs have a limitation in that the intensity of the received light cannot be determined because they output an electrical signal of a constant magnitude regardless of the number of absorbed photons when the avalanche breakdown phenomenon occurs.

[0143] Figure 3 is a diagram illustrating the limitations of SPAD.

[0144] More specifically, FIG. 3(a) is a diagram showing the electrical signal output from SPAD over time for a certain period from the time of laser output after a laser is output from a lidar device in an ideal situation where there is no external light such as sunlight, and FIG. 3(b) is a diagram showing the electrical signal output from SPAD over time for a certain period from the time of laser output after a laser is output from a lidar device in a situation where there is external light such as sunlight.

[0145] First, since Figure 3(a) assumes an ideal situation where there is no external light such as sunlight, in the situation of Figure 3(a), not a single photon is received by the SPAD until the laser output from the lidar device is reflected from the target and received by the lidar device.

[0146] Accordingly, the state in which a reverse voltage greater than the breakdown voltage is applied to the SPAD is maintained until the laser output from the lidar device is reflected from the target and received by the lidar device, and when a photon included in the laser output from the lidar device is received by the SPAD in this state, an avalanche breakdown phenomenon occurs in the SPAD and a first electrical signal (1211) is output from the SPAD.

[0147] At this time, the time interval between the time when the laser is output from the lidar device and the time when the first electrical signal (1211) is output may correspond to the flight time of the laser output from the lidar device between the lidar device and the target object.

[0148] Therefore, as shown in Figure 3 (a), it is not difficult to measure the distance between the LiDAR device and the target using a SPAD in an ideal situation where there is no external light, such as sunlight.

[0149] However, as shown in Figure 3 (b), the limitations of SPAD are revealed in situations where there is external light such as sunlight.

[0150] Since Figure 3(b) assumes a situation where there is external light such as sunlight, in the situation of Figure 3(b), photons caused by external light such as sunlight can be received by the SPAD before and after the laser output from the lidar device is reflected from the target and received by the lidar device.

[0151] To explain this more specifically, photons caused by external light, such as sunlight, are received by the SPAD even before the laser output from the lidar device is reflected from the target and returns to the lidar device, and accordingly, an avalanche breakdown phenomenon occurs in the SPAD, and a second electrical signal (1221) is output from the SPAD. After that, the magnitude of the voltage applied to the SPAD is momentarily reduced due to the quenching circuit of the SPAD, the flow of current in the SPAD is blocked, and no electrical signal is output from the SPAD until the voltage applied to the SPAD is restored through the charging stage. After the voltage applied to the SPAD is restored through the charging stage, photons caused by external light, such as sunlight, are received by the SPAD before the laser output from the lidar device is reflected from the target and returns to the lidar device, and accordingly, a third electrical signal (1222) is output from the SPAD. After passing through the aforementioned Quenching and Charging steps, and after the voltage applied to the SPAD is restored, the laser output from the LiDAR device is reflected from the target and returns to the LiDAR device, and at least one photon included in the laser output from the LiDAR device is received by the SPAD, and accordingly, a fourth electrical signal (1223) is output from the SPAD. Afterwards, photons caused by external light, such as sunlight, may be further received by the SPAD, and accordingly, a fifth electrical signal (1224) and a sixth electrical signal (1225) are generated.

[0152] At this time, even though the number of laser photons output from the lidar device, reflected from the target object, and received by the SPAD is greater than the number of photons received by the SPAD due to external light such as sunlight, the magnitude of the electrical signal output from the SPAD is the same as described above, so the magnitudes of the second to sixth electrical signals (1221 to 1225) are equal to each other.

[0153] Therefore, the lidar device cannot determine which of the second to sixth electrical signals (1221 to 1225) is an electrical signal caused by a photon of a laser that is output from the lidar device, reflected from the object, and received by the SPAD.

[0154] Accordingly, even if the time interval between the time when the laser is output from the lidar device and the time when the fourth electrical signal (1223) is output corresponds to the flight time of the laser output from the lidar device between the lidar device and the target, it is difficult to measure the distance between the lidar device and the target using the SPAD because the fourth electrical signal (1223) cannot be determined to be an electrical signal caused by a photon of the laser output from the lidar device, reflected from the target, and received by the SPAD.

[0155] Although only five electrical signals are briefly shown in Figure 3(b), in reality, hundreds of electrical signals can be output over a certain period after the laser is emitted from the lidar device, so it can be seen that measuring the distance between the lidar device and the target using SPAD is more difficult.

[0156] Therefore, a more advanced method may be required to measure the distance between a LiDAR device and an object using a SPAD.

[0157]

[0158] [Approach and Histogram for Determining Laser Flight Time Using Electrical Signals Output from SPAD]

[0159] Explanation of the approach

[0160] Figure 4 is a diagram illustrating an approach to determining the flight time of a laser using electrical signals output from a SPAD.

[0161] More specifically, FIG. 4(a) is a diagram showing the electrical signal output from SPAD over time for a certain period of time from the point of laser output after a laser is output from a lidar device when an object is located at a first distance from the lidar device, and FIG. 4(b) is a diagram showing the electrical signal output from SPAD over time for a certain period of time from the point of laser output after a laser is output from a lidar device at a different time from FIG. 4(a) when an object is located at the first distance from the lidar device.

[0162] At this time, both (a) and (b) of Fig. 4 assume a situation where there is external light such as sunlight.

[0163] First, referring to FIG. 4(a), a plurality of electrical signals including a first electrical signal (1231), a second electrical signal (1232), and a third electrical signal (1235) are output from the SPAD for a certain period of time from the time of laser output when a laser is output from the lidar device.

[0164] In this case, since the aforementioned details apply to the output of electrical signals from the SPAD, redundant descriptions will be omitted.

[0165] In addition, at this time, the first electrical signal (1231) is an electrical signal generated as a laser output from a LiDAR device is received by a SPAD after being reflected from an object, and the second electrical signal (1232) and the third electrical signal (1235) are electrical signals generated as photons caused by external light, such as sunlight, are received by a SPAD.

[0166] Referring again to FIG. 4(b), a plurality of electrical signals including a fourth electrical signal (1241), a fifth electrical signal (1242), and a sixth electrical signal (1245) are output from the SPAD for a certain period of time from the time of laser output when the laser is output from the lidar device.

[0167] In this case, since the aforementioned details apply to the output of electrical signals from the SPAD, redundant descriptions will be omitted.

[0168] In addition, at this time, the fourth electrical signal (1241) is an electrical signal generated as a laser output from a LiDAR device is received by a SPAD after being reflected from an object, and the fifth electrical signal (1242) and the sixth electrical signal (1245) are electrical signals generated as photons caused by external light, such as sunlight, are received by a SPAD.

[0169] Referring again to FIG. 4 (a) and (b), in FIG. 4 (a), the first time interval (1233), which is the time interval between the first electrical signal (1231) and the laser output time, is the same as the second time interval (1243), which is the time interval between the fourth electrical signal (1241) and the laser output time, in FIG. 4 (b).

[0170] This is because in FIG. 4 (a), the object is located at the first distance from the lidar device, and in FIG. 4 (b), the object is also located at the first distance from the lidar device, and the first time interval (1233) corresponds to the round-trip flight time of the laser output from the lidar device between the lidar device and the object, and the second time interval (1243) also corresponds to the round-trip flight time of the laser output from the lidar device between the lidar device and the object.

[0171] On the other hand, the third time interval (1232), which is the time interval between the second electrical signal (1232) and the laser output time in FIG. 4 (a), is different from the fourth time interval (1242), which is the time interval between the fifth electrical signal (1242) and the laser output time in FIG. 4 (b).

[0172] This is because both the second electrical signal (1232) and the fifth electrical signal (1242) are electrical signals generated as photons from external light, such as sunlight, are received by the SPAD, but photons from external light, such as sunlight, are received by the SPAD without any regularity and are arbitrary.

[0173] Accordingly, when N lasers are emitted from the lidar device in a situation similar to the situation described in FIG. 4 (a) and (b) where the target is located at the first distance, the number of electrical signals emitted after a time interval equal to the first time interval (1233) and the second time interval (1243) from each laser output point may be N. On the other hand, the number of electrical signals emitted after a specific time interval different from the first time interval (1233) and the second time interval (1243) from each laser output point may be less than N, which is explained more specifically through FIG. 5.

[0174] Figure 5 is a diagram illustrating the number of electrical signals output after the same amount of time has elapsed from each laser output point when a laser is output multiple times from a lidar device.

[0175] Before explaining Fig. 5, Fig. 5 assumes that N lasers are output from the lidar device in a situation where the object is located at the first distance, similar to the situation described in Fig. 4 (a) and (b).

[0176] Accordingly, as described in FIG. 4 (a) and (b), when the object is located at the first distance from the lidar device, the laser output from the lidar device is received by the SPAD after the same time interval as the first time interval (1233) and the second time interval (1243) from the time of laser output.

[0177] Accordingly, when N lasers are emitted from the lidar device while the target is located at the first distance from the lidar device, the electrical signal generated from the SPAD as the laser emitted from the lidar device is reflected from the target and returns to the lidar device occurs after a time interval equal to the first time interval (1233) and the second time interval (1243) from the time of output of each of the N lasers emitted. In FIG. 5, the time interval equal to the first time interval (1233) and the second time interval (1243) is described as the fifth time interval (1250).

[0178] That is, referring to FIG. 5, when N lasers are output from the lidar device, the number of electrical signals generated after the 5th time interval (1250) from each laser output time is N.

[0179] On the other hand, as described in Figures 4 (a) and (b), photons caused by external light such as sunlight are received by the SPAD without any regularity and are random. Therefore, when N lasers are output from the lidar device, the number of electrical signals that occur after a specific time interval different from the fifth time interval (1250) from each laser output time is less than N.

[0180] Therefore, as explained through Fig. 5, when the number of electrical signals output after the same amount of time has elapsed from each laser output point is compared, the time interval in which the largest number of electrical signals occurred can be determined by outputting a laser multiple times from the lidar device and comparing the number of electrical signals output.

[0181] At this time, the time interval during which the largest number of electrical signals occurred corresponds to the round-trip flight time of the laser output from the lidar device between the lidar device and the target.

[0182] Therefore, if the method described above is used through Fig. 5, it becomes possible to measure the distance to an object using a SPAD in a LiDAR device.

[0183] Below, we will explain in more detail the histogram used to utilize the approach described above.

[0184]

[0185] Histogram

[0186] FIG. 6 is a drawing for explaining the histogram disclosed in the present application.

[0187] More specifically, FIG. 6(a) is a diagram for explaining the data structure of the histogram disclosed in the present application, and FIG. 6(b) is a diagram briefly schematically representing the histogram disclosed in the present application.

[0188] Referring to Figures 6 (a) and (b), the histogram is data composed of counting values ​​corresponding to a preset number of time bins and a preset number of time bins.

[0189] That is, the histogram is data composed of N time bins including a first time bin (TB1) to an Nth time bin (TBn) and N counting values ​​including a first counting value (C1) corresponding to the first time bin (TB1) to an Nth counting value (Cn) corresponding to the Nth time bin (TBn).

[0190] In this case, each time bin of the histogram represents a time interval after a specific amount of time has elapsed since the laser output point.

[0191] For example, the first time bin (TB1) of the histogram represents a time interval having a time length of 2ns after 0 seconds have passed from the reference time corresponding to the laser output time, and the second time bin (TB2) represents a time interval having a time length of 2ns after 2ns have passed from the reference time corresponding to the laser output time.

[0192] Additionally, for example, when a histogram is generated based on electrical signals output from a SPAD for a certain period of time from each of the m laser output times, the first counting value (C1) corresponding to the first time bin (TB1) of the histogram corresponds to the number of electrical signals generated from the SPAD within a time interval having a time length of 2ns from 0 seconds after each of the m laser output times, and the second counting value (C2) corresponding to the second time bin (TB2) of the histogram corresponds to the number of electrical signals generated from the SPAD within a time interval having a time length of 2ns from 2ns after each of the m laser output times.

[0193] Below, we will explain in more detail how to generate the aforementioned histogram in a LiDAR device.

[0194]

[0195] [Creation of Histograms and Explanation of Histograms]

[0196] FIG. 7 is a drawing for explaining the generation of a histogram in a lidar device disclosed in the present application.

[0197] More specifically, Fig. 7 is a diagram for specifically explaining the process of generating a histogram through M sampling cycles in a lidar device.

[0198] Definition of the sampling cycle

[0199] As described above through Fig. 3, after a laser is emitted once from the lidar device, it is difficult to distinguish whether the electrical signals emitted from the SPAD for a certain period of time from the time of laser emission are electrical signals caused by photons of the laser reflected from the target and received by the SPAD, or electrical signals caused by photons received by the SPAD due to external light such as sunlight.

[0200] Accordingly, as described above through FIGS. 4 and 5, in order to measure the distance between the lidar device and the target object using a SPAD, the lidar device outputs a laser multiple times, and it is necessary to detect the electrical signal output from the SPAD for a certain period of time from the time of each laser output.

[0201] In other words, to measure the distance between a lidar device and an object using a SPAD, the lidar device needs to emit a laser and repeat a series of operations multiple times, including detecting electrical signals output from a detecting element for a preset time starting from a point in time corresponding to the laser output time.

[0202] Accordingly, a series of operations including outputting a laser from a LiDAR device and detecting an electrical signal output from a detecting element for a preset time starting from a point corresponding to the time of laser output can be defined as a unit operation cycle, and for convenience of explanation, the above-described unit operation cycle is defined and described as a sampling cycle.

[0203] That is, in this specification, a section in which a series of operations are performed to output a laser using a laser-emitting element and to detect an electrical signal output from a detecting element for a preset time starting from a point in time corresponding to the time of laser output is defined and described as a sampling cycle.

[0204] Additionally, for convenience of explanation, the preset time interval for detecting an electrical signal output from a detecting element may be described in this specification using the term "detecting window."

[0205] Operation of the LiDAR device in the sampling cycle

[0206] In one sampling cycle, the lidar device can generate a laser emission trigger signal to operate a laser emitter, and the laser emitter of the lidar device outputs a laser in response to the laser emission trigger signal.

[0207] For example, referring to FIG. 7, in the first sampling cycle (1310), the lidar device can generate a first laser emission trigger signal (1311), and the laser emitting element outputs a first laser (1312) in response to the first laser emission trigger signal (1311).

[0208] Additionally, for example, in a second sampling cycle (1320), the lidar device may generate a second laser emission trigger signal (1321), and the laser emitting element outputs a second laser (1322) in response to the second laser emission trigger signal (1321).

[0209] Additionally, for example, in the M sampling cycle (1330), the lidar device may generate an M laser emission trigger signal (1331), and the laser emitting element outputs an M laser (1332) in response to the M laser emission trigger signal (1331).

[0210] Additionally, in one sampling cycle, the lidar device can generate a detecting trigger signal to set a sampling reference time of the detecting element, and an electrical signal output from the detecting element of the lidar device during the detecting window from the sampling reference time set in response to the detecting trigger signal is detected.

[0211] For example, referring to FIG. 7, in the first sampling cycle (1310), the lidar device may generate a first detecting trigger signal (1313), and an electrical signal output from the detecting element of the lidar device is detected during the first detecting window (1315) from the first sampling reference time (1314) set in response to the first detecting trigger signal (1313).

[0212] Additionally, for example, in a second sampling cycle (1320), the lidar device may generate a second detecting trigger signal (1323), and an electrical signal output from a detecting element of the lidar device is detected during a second detecting window (1325) from a second sampling reference time (1324) set in response to the second detecting trigger signal (1323).

[0213] Additionally, for example, in the M sampling cycle (1330), the lidar device may generate an M detecting trigger signal (1333), and an electrical signal output from a detecting element of the lidar device is detected during a third detecting window (1335) from the M sampling reference time (1334) set in response to the M detecting trigger signal (1333).

[0214] At this time, the laser emission trigger signal and the detecting trigger signal may be mutually synchronized, and the meaning of the laser emission trigger signal and the detecting trigger signal being mutually synchronized is that the laser emission trigger signal and the detecting trigger signal may be generated at the same time, the time interval between the generation times of the laser emission trigger signal and the detecting trigger signal may be maintained constant, and the time interval between the generation times of the laser emission trigger signal and the detecting trigger signal may be maintained within a preset time interval.

[0215] Therefore, in one sampling cycle, as the laser emission trigger signal and the detecting trigger signal are mutually synchronized, the laser output time and the sampling reference time can be mutually synchronized.

[0216] For example, in the first sampling cycle (1310), the first laser emission trigger signal (1311) and the first detecting trigger signal (1313) may be synchronized with each other, in the second sampling cycle (1320), the second laser emission trigger signal (1321) and the second detecting trigger signal (1323) may be synchronized with each other, and in the M sampling cycle (1330), the M laser emission trigger signal (1331) and the M detecting trigger signal (1333) may be synchronized with each other.

[0217] Detection of electrical signals output from a detecting element and generation of histograms in a sampling cycle

[0218] As described above, in one sampling cycle, an electrical signal output from a detecting element of the lidar device is detected during the detecting window from a sampling reference time set in response to a detecting trigger signal.

[0219] Hereinafter, an electrical signal output from a detection element of a lidar device during a detection window is detected through exemplary situations during the first detection window (1315) and the second detection window (1325).

[0220] Referring again to FIG. 7, for example, a first electrical signal (1316) and a second electrical signal (1317) are output from a detecting element of the lidar device during a first detecting window (1315), and a third electrical signal (1326) and a fourth electrical signal (1327) are output from a detecting element of the lidar device during a second detecting window (1325).

[0221] At this time, in one sampling cycle, an electrical signal output from a detecting element of the LiDAR device is detected during the detecting window from the sampling reference point, and at each judgment point corresponding to a preset clock, it is determined whether an electrical signal has been output from the detecting element, and a counting value is generated based on the judgment result. At this time, the judgment point corresponding to the preset clock may be the rising edge of the preset clock.

[0222] For example, during the first detecting window (1315), it is determined whether an electrical signal is output from the detecting element at a first determination time corresponding to the first clock (1410) from the first sampling reference time (1314), and since no electrical signal is detected at the first determination time, the counting value is generated as 0.

[0223] Additionally, for example, during the first detecting window (1315), it is determined whether an electrical signal is output from the detecting element at a second judgment time corresponding to the second clock (1420) from the first sampling reference time (1314), and since an electrical signal is detected at the second judgment time, a counting value is generated as 1.

[0224] Additionally, for example, during the first detecting window (1315), it is determined whether an electrical signal is output from the detecting element at a third determination time corresponding to the third clock (1430) from the first sampling reference time (1314), and since no electrical signal is detected at the third determination time, the counting value is generated as 0.

[0225] Additionally, for example, during the second detecting window (1315), it is determined whether an electrical signal is output from the detecting element at a fourth determination time corresponding to the Nth clock (1440) from the first sampling reference time (1314), and since an electrical signal is detected at the fourth determination time, a counting value is generated as 1.

[0226] Additionally, for example, during the second detecting window (1325), it is determined whether an electrical signal is output from the detecting element at a fifth determination time corresponding to the first clock (1450) from the second sampling reference time (1324), and since no electrical signal is detected at the fifth determination time, the counting value is generated as 0. At this time, the first clock (1450) from the second sampling reference time (1324) may be the N+1th clock from the first sampling reference time (1324), or it may be the N+1+kth clock. (At this time, k may correspond to the number of clocks corresponding to the time interval between the first sampling cycle (1310) and the second sampling cycle (1310).)

[0227] Additionally, for example, during the second detecting window (1325), it is determined whether an electrical signal is output from the detecting element at the sixth determination time corresponding to the second clock (1460) from the second sampling reference time (1324), and since an electrical signal is detected at the sixth determination time, a counting value is generated as 1.

[0228] Additionally, for example, during the second detecting window (1325), it is determined whether an electrical signal is output from the detecting element at the seventh determination time corresponding to the third clock (1470) from the second sampling reference time (1324), and since an electrical signal is detected at the seventh determination time, a counting value is generated as 1.

[0229] Additionally, for example, during the second detecting window (1325), it is determined whether an electrical signal is output from the detecting element at the eighth judgment time corresponding to the Nth clock (1480) from the second sampling reference time (1324), and since an electrical signal is detected at the eighth judgment time, the counting value is generated as 0.

[0230] Additionally, in one sampling cycle, the counting value generated at each judgment point is assigned to a time bin set according to the time interval between the sampling reference point and the judgment point. At this time, each time bin may represent a time interval after a specific amount of time has elapsed from the sampling reference point. For example, the first time bin (TB1) may represent a time interval with a length of 2ns after 0 seconds have passed from the sampling reference point, the second time bin (TB2) may represent a time interval with a length of 2ns after 2ns have passed from the sampling reference point, the third time bin (TB3) may represent a time interval with a length of 2ns after 4ns have passed from the sampling reference point, and the Nth time bin (TBn) may represent a time interval with a length of 2ns after 2*(N-1)ns have passed from the sampling reference point.

[0231] For example, a counting value generated at a first judgment time corresponding to the first clock (1410) from the first sampling reference time (1314) during the first detection window (1315) is assigned to a first time bin corresponding to the first clock (1410) from the first sampling reference time (1314). This can be understood as the counting value generated at a first judgment time corresponding to the first clock (1410) from the first sampling reference time (1314) during the first detection window (1315) being assigned to a first time bin (TB1) representing a time interval after the time interval between the first sampling reference time (1314) and the first clock (1410) has elapsed.

[0232] Additionally, for example, a counting value generated at a second judgment time corresponding to the second clock (1420) from the first sampling reference time (1314) during the first detecting window (1315) is assigned to a second time bin (TB2) corresponding to the second clock (1420) from the first sampling reference time (1314).

[0233] Additionally, for example, a counting value generated at a third judgment time corresponding to the third clock (1430) from the first sampling reference time (1314) during the first detecting window (1315) is assigned to a third time bin (TB3) corresponding to the third clock (1430) from the first sampling reference time (1314).

[0234] Additionally, for example, a counting value generated at a fourth judgment time corresponding to the Nth clock (1440) from the first sampling reference time (1314) during the first detecting window (1315) is assigned to the Nth time bin (TBn) corresponding to the Nth clock (1440) from the first sampling reference time (1314).

[0235] At this time, the first data (1510) shows data generated as the counting value generated at each judgment point during the first detection window (1315) of the first sampling cycle (1310) is assigned to the time bin corresponding to each judgment point.

[0236] As shown in FIG. 7, the first time bin (TB1) of the first data (1510) corresponds to a counting value of 0, the second time bin (TB2) corresponds to a counting value of 1, the third time bin (TB3) corresponds to a counting value of 0, and the Nth time bin (TBn) corresponds to a counting value of 1.

[0237] Again, I will explain by returning to an example where, in a single sampling cycle, the counting value generated at each decision point is assigned to a time bin set according to the time interval between the sampling reference point and the decision point.

[0238] For example, during the second detection window (1325), the counting value generated at the fifth judgment time corresponding to the first clock (1450) from the second sampling reference time (1324) is assigned or accumulated in the first time bin corresponding to the first clock (1450) from the second sampling reference time (1324). At this time, the first clock (1450) from the second sampling reference time (1324) is a clock at a physically different time from the first clock (1410) from the first sampling reference time (1314) described above. However, since the time interval between the second sampling reference time (1324) and the first clock (1450) and the time interval between the first sampling reference time (1314) and the first clock (1410) are the same, the time bin corresponding to the first clock (1450) from the second sampling reference time (1324) and the time bin corresponding to the first clock (1410) from the first sampling reference time (1314) are the same as the first time bin (TB1).

[0239] That is, the counting value 0 generated through the second sampling cycle (1320) is accumulated in the first time bin (TB1) to which the counting value 0 is assigned through the first sampling cycle (1310).

[0240] Accordingly, the first time bin (TB1) of the second data (1520) obtained through the first sampling cycle (1310) and the second sampling cycle (1320) corresponds to a counting value of 0.

[0241] Additionally, for example, a counting value generated at a sixth judgment time corresponding to the second clock (1460) from the second sampling reference time (1324) during the second detecting window (1325) is assigned to or accumulated in a second time bin corresponding to the second clock (1460) from the second sampling reference time (1324).

[0242] That is, the counting value 1 generated through the second sampling cycle (1320) is accumulated in the second time bin (TB2) to which the counting value 1 is assigned through the first sampling cycle (1310).

[0243] Accordingly, a counting value of 2 corresponds to the second time bin (TB2) of the second data (1520) obtained through the first sampling cycle (1310) and the second sampling cycle (1320).

[0244] Additionally, for example, a counting value generated at a seventh judgment time corresponding to a third clock (1470) from a second sampling reference time (1324) during a second detecting window (1325) is assigned to or accumulated in a third time bin corresponding to a third clock (1470) from a second sampling reference time (1324).

[0245] That is, a counting value 1 generated through the second sampling cycle (1320) is assigned or accumulated in the third time bin (TB3), where a counting value 0 is assigned through the first sampling cycle (1310).

[0246] Accordingly, a counting value of 1 corresponds to the third time bin (TB3) of the second data (1520) obtained through the first sampling cycle (1310) and the second sampling cycle (1320).

[0247] Additionally, for example, the counting value generated at the 8th judgment time corresponding to the Nth clock (1480) from the 2nd sampling reference time (1324) during the 2nd detection window (1325) is assigned or accumulated in the Nth time bin corresponding to the Nth clock (1480) from the 2nd sampling reference time (1324).

[0248] That is, the counting value 0 generated through the second sampling cycle (1320) is accumulated in the Nth time bin (TBn), to which the counting value 1 is assigned through the first sampling cycle (1310).

[0249] Accordingly, the Nth time bin (TBn) of the second data (1520) obtained through the first sampling cycle (1310) and the second sampling cycle (1320) corresponds to a counting value of 1.

[0250] In the above description, allocating a counting value may involve storing the counting value in memory corresponding to the time bin, and accumulating a counting value may involve adding the generated counting value to the counting value stored in memory corresponding to the time bin.

[0251] Although the above descriptions were explained through the first sampling cycle (1310) and the second sampling cycle (1320), the above operations may be performed for all of the first to M sampling cycles (1310 to 1330) in order to generate a histogram through M sampling cycles, and accordingly, a histogram (1530) generated through the first to M sampling cycles (1310 to 1330) may be generated.

[0252] At this time, the histogram (1530) includes N time bins including the first to Nth time bins (TB1 to TBn) and the first to Nth counting values ​​(C1 to Cn) corresponding to each of the N time bins.

[0253] In addition, at this time, each counting value corresponding to each time bin may correspond to the number of judgment points in which an electrical signal output from a detecting element is detected among the judgment points corresponding to each time bin through M sampling cycles, and may correspond to the sum of the counting values ​​generated at the judgment points corresponding to each time bin.

[0254] Relationship between time bin length and LiDAR device distance resolution

[0255] When measuring the distance between a LiDAR device and an object using the histogram generated as described above, the length of the time bin, which is the length of the time interval represented by the time bin, affects the distance resolution of the LiDAR device.

[0256] This is because electrical signals generated during the length of the time interval corresponding to the length of the time bin are determined to have occurred after the same time interval from the laser output point.

[0257] To explain with a specific example, assuming that a specific time bin represents a time interval with a length of 2ns starting from 300ns after a reference time corresponding to the laser output time, the electrical signal output from the SPAD is treated as having been output after the same time interval from the reference time when it is output 300ns after the reference time, 301ns after the reference time, and 302ns after the reference time.

[0258] This is explained more specifically using the situation where a first object is located at a distance of 15m from the lidar device, a situation where a second object is located at a distance of 15.05m, and a situation where a third object is located at a distance of 15.10m.

[0259] In a situation where a first object is located at a distance of 15m from the lidar device, when a laser is emitted from the lidar device, an electrical signal is generated 300ns after a reference time corresponding to the time of laser emission.

[0260] In addition, in a situation where a second object is located at a distance of 15.05 m from the lidar device, when a laser is emitted from the lidar device, an electrical signal is generated 301 ns after the reference time corresponding to the time of laser output.

[0261] In addition, in a situation where a third object is located at a distance of 15.10 m from the lidar device, when a laser is emitted from the lidar device, an electrical signal is generated 302 ns after the reference time corresponding to the time of laser emission.

[0262] At this time, if the time length of the time bin is 2ns, the first to third objects can all be judged to be at the same distance.

[0263] Therefore, when the time length of the time bin is 2ns, the distance resolution of the LiDAR device becomes 0.1m.

[0264] In contrast, when the time length of the time bin is 10ns, it is determined that objects located at a distance of 15m from the lidar device and objects located at a distance of 15.5m are located at the same distance.

[0265] Therefore, when the time length of the time bin is 10ns, the distance resolution of the LiDAR device becomes 0.5m.

[0266]

[0267] Exemplary combinations of time bin length, number of time bins, and number of sampling cycles

[0268] For the sake of understanding, exemplary combinations of the time bin length, the number of time bins, and the number of sampling cycles are described.

[0269] In an exemplary lidar device, the length of the time bin is set to 2ns, and the number of time bins can be set to 500.

[0270] Therefore, in such an exemplary lidar device, the time taken for one sampling cycle to be performed can be 1000ns.

[0271] In addition, in an exemplary lidar device, the number of sampling cycles can be set to 357.

[0272] In this case, the time required to generate one histogram through 357 sampling cycles may be at least 357,000 ns.

[0273] Of course, the exemplary figures mentioned above can be designed differently as needed.

[0274] [Measuring the distance between a LiDAR device and an object using a histogram]

[0275] Determination of the echo signal

[0276] FIG. 8 is a diagram illustrating the determination of an echo signal based on a histogram in a lidar device.

[0277] In order to measure the distance between the lidar device and the target object, an echo signal (1620) can be determined based on a histogram (1600).

[0278] At this time, the echo signal (1620) includes a time bin group (1621) and a counting value group (1622) corresponding to the time bin group (1621) as part of a histogram satisfying a preset standard.

[0279] At this time, various algorithms may be used to determine the echo signal (1620).

[0280] For example, the above echo signal (1620) can be determined as a counting value group greater than or equal to a preset threshold value (1610) and a corresponding time bin group.

[0281] Additionally, for example, the echo signal (1620) may be determined as the largest counting value (Ck) among the counting values ​​included in the histogram (1600), the time bin (TBk) to which the largest counting (Ck) is assigned, and adjacent time bins (TBk-2, TBk-1, TBk+1, TBk+2 and TBk+3) and the counting values ​​(Ck-2, Ck-1, Ck+1, Ck+2 and Ck+3) assigned thereto.

[0282]

[0283] Measuring the distance to an object based on the echo signal

[0284] In order to measure the distance between a LiDAR device and an object based on an echo signal, a time value corresponding to the echo signal is determined, and this can be determined by various methods.

[0285] For example, the time value corresponding to the echo signal may be determined as the time value corresponding to the time bin to which the largest counting value among the counting values ​​included in the echo signal is assigned, may be determined as the time value corresponding to the median value of the time bins to which the counting values ​​included in the echo signal are assigned, and may be determined as the time value corresponding to the average value of the time bins to which the counting values ​​included in the echo signal are assigned, but is not limited thereto and may be determined by various methods.

[0286] When the time value corresponding to the echo signal is determined, the distance between the lidar device and the target can be calculated through the following Equation 1.

[0287] [Relationship 1]

[0288] Distance = (c * time value corresponding to the echo signal) / 2

[0289]

[0290] Estimation of laser reflection intensity on an object based on echo signal

[0291] The reflection intensity of the laser on the target can be estimated based on the echo signal.

[0292] In this case, the laser reflection intensity on the target refers to the intensity of the laser that is emitted from the lidar device, reflected from the target, and returns to the lidar device.

[0293] In addition, at this time, the intensity of the laser that is output from the lidar device, reflected from the target, and returned to the lidar device depends on the angle of incidence of the laser on the surface of the target and the physical properties of the target (surface characteristics, color, reflectance, etc.).

[0294] The reason the reflection intensity of a laser on an object can be estimated based on echo signals is that as the intensity of the laser reflected from the object and returning to the lidar device increases, the probability of detection by the lidar device's detection element increases, and consequently, the counting value of the echo signal can increase.

[0295] At this time, in order to estimate the reflection intensity of the laser on the target object based on the echo signal, counting values ​​included in the echo signal can be used in various ways.

[0296] For example, the reflection intensity of a laser on an object can be determined by the largest counting value among the counting values ​​included in the echo signal, the sum of the counting values ​​included in the echo signal, the width of the echo signal, the area of ​​the echo signal, etc., but is not limited thereto and can be determined by various methods.

[0297] At this time, the reflection intensity of the laser on the estimated object may be expressed using terms such as reflection intensity, intensity, etc. in this specification.

[0298] [Processor]

[0299] Processor functions

[0300] Referring again to FIG. 1, the processor (1030) is configured to control the laser emitting element (1010) and the detecting element (1020) and to perform functions such as generating the above-described histogram, determining the echo signal, measuring the distance between the lidar device and the target based on the echo signal, and estimating the reflection intensity of the laser on the target based on the echo signal.

[0301] At this time, the processor (1030) may be implemented as a single processor, but is not limited thereto, and may be implemented through multiple processors embedded in each component of the lidar device according to function.

[0302] For example, the processor (1030) may be configured such that a first processor embedded in the laser emitting element (1010) controls the laser emitting element, a second processor embedded in the detecting element (1020) controls the detecting element, generates a histogram and determines an echo signal, and a third processor provided separately from the laser emitting element (1010) and the detecting element (1020) performs the function of measuring the distance between the lidar device and the target object based on the echo signal and estimating the reflection intensity of the laser on the target object.

[0303]

[0304] [Necessity of configuring the LiDAR device to include a laser-emitting element array, a transmitting optical system, a detecting element array, and a receiving optical system]

[0305] Necessity of Laser Emitting Element Array and Transmitting Optic Assembly

[0306] As described above, since a lidar device is a device for measuring the distance between the lidar device and an object using a laser, the measurable area of ​​the lidar device is related to the area irradiated by the laser output from the lidar device.

[0307] In other words, to expand the measurable range of a lidar device, it is necessary to expand the area irradiated by the laser output from the lidar device.

[0308] FIG. 9 is a diagram illustrating various examples for expanding the area where a laser output from a lidar device is irradiated.

[0309] FIG. 9(a) is a diagram illustrating an exemplary lidar device composed of a laser emitting element and a diffuser.

[0310] Referring to FIG. 9 (a), the laser (1711) output from the laser emitting element (1710) is diffused through the diffuser (1712), thereby expanding the area where the laser is irradiated.

[0311] At this time, as the laser (1711) output from the laser emitting element (1710) passes through the diffuser (1712) and diffuses, the density of photons per unit area decreases rapidly as the distance increases.

[0312] Therefore, when the lidar device is configured as in (a) of Fig. 9, the amount of photons reflected from an object located at a distance from the lidar device is reduced, and accordingly, the amount of photons returning to the lidar device is also reduced.

[0313] Ultimately, configuring the lidar device as in Fig. 9 (a) reduces the measurable distance of the lidar device.

[0314] Figure 9(b) is a diagram illustrating an exemplary lidar device composed of a laser emitting element and a rotating mirror.

[0315] Referring to FIG. 9(b), a laser (1721) output from a laser emitting element (1720) is irradiated outside the lidar device through a first rotating mirror (1722) that rotates about a first rotation axis and a second rotating mirror (1723) that rotates about a second rotation axis perpendicular to the first rotation axis.

[0316] At this time, as the rotation angle of the first rotating mirror (1722) and the second rotating mirror (1723) changes, the direction in which the laser (1721) output from the laser emitting element (1720) is irradiated outward changes.

[0317] That is, the direction in which the laser (1721) is irradiated is determined according to the rotation angles of the first rotating mirror (1722) and the second rotating mirror (1723) at the time when the laser (1721) is output.

[0318] Accordingly, when the first rotating mirror (1722) and the second rotating mirror (1723) rotate while the laser (1721) is emitted multiple times from the laser emitting element (1720) over time, the direction in which the laser (1721) emitted from the laser emitting element (1720) is irradiated outward changes over time, and the area where the laser is irradiated is expanded.

[0319] However, when configuring the lidar device as shown in Fig. 9 (b), the size of the lidar device increases due to the volume of the rotating mirrors, power is consumed for the rotational operation of the rotating mirrors, and a problem of reduced durability occurs due to the rotation of the rotating mirrors.

[0320] That is, as described in FIG. 9 (a) and (b), there is a clear limitation to expanding the area where the laser is irradiated using a single laser emitting element (1720).

[0321] Therefore, using a laser-emitting element array to expand the area where the laser is irradiated can be a useful solution compared to the configuration of the lidar device described in (a) and (b) of FIG. 9.

[0322] Figure 9 (c) is a diagram illustrating, in an exemplary manner, the area where a laser is irradiated by a lidar device composed of a laser-emitting element array.

[0323] Referring to FIG. 9 (c), the first laser (1733) output from the first laser emitting element (1731) included in the laser emitting element array (1730) and the second laser (1734) output from the second laser emitting element (1732) are output in parallel in the same direction while separated by a first distance (1735).

[0324] At this time, the first distance (1734) is a distance corresponding to the size (length or width) of the laser light-emitting element array (1730).

[0325] Accordingly, when configuring the lidar device as in (c) of FIG. 9, the area where the laser is irradiated is expanded by an amount corresponding to the size of the laser-emitting element array (1730).

[0326] However, in this case, since the directions of travel of the first laser (1733) and the second laser (1734) are parallel, the extent to which the area irradiated by the laser is expanded is minimal.

[0327] Therefore, in order to further expand the area where the laser is irradiated using the laser emitting element array (1730), it is necessary to configure the lasers output from the plurality of laser emitting elements included in the laser emitting element array (1730) to be output in different directions.

[0328] Hereinafter, an exemplary lidar device configured such that lasers output from a plurality of laser-emitting elements included in a laser-emitting element array (1730) are output in different directions is described through (d) of FIG. 9.

[0329] In addition, (d) of FIG. 9 is a diagram illustrating an exemplary lidar device composed of an array of laser-emitting elements arranged on a curved surface.

[0330] Referring to FIG. 9 (d), the first laser emitting element (1741) included in the laser emitting element (1740) faces the first direction and outputs the first laser (1743) in the first direction, and the second laser emitting element (1742) faces the second direction and outputs the second laser (1744) in the second direction.

[0331] That is, when a lidar device is configured as in (d) of FIG. 9, the multiple laser-emitting elements of the laser-emitting element array (1740) each emit lasers in different directions depending on the direction they are facing, and accordingly, the area where lasers are irradiated from the lidar device is expanded.

[0332] However, when configuring a lidar device as in (d) of Fig. 9, it is necessary to align the directions of the laser-emitting elements one by one so that the laser is emitted in the desired direction, which results in excessive time and cost.

[0333] In addition, in this case, it is difficult to collimate the lasers output from the multiple laser light-emitting elements of the laser light-emitting element array (1740), so the density of photons per unit area decreases as the distance increases.

[0334] Therefore, unlike the lidar device described in (c) and (d) of FIG. 9, a laser emitting element array is used, and a transmitting optic assembly is required to steer and collimate the lasers output from each of the plurality of laser emitting elements included in the laser emitting element array.

[0335]

[0336] Necessity of Detecting Element Array and Receiving Optic Assembly

[0337] As described above, since a lidar device is a device designed to measure the distance between the lidar device and an object using a laser, the measurable range of the lidar device is related to the range in which light can be detected.

[0338] In addition, since the lidar device utilizes the laser output from the lidar device and reflected from the target to be received by the detection element, a receiving optic assembly is required to focus the light received by the lidar device in order to increase the reception efficiency.

[0339] At this time, since light incident on the receiving optic assembly in different directions can be focused at different points, it is necessary to configure a detector element array so that detector elements are arranged at the points where light incident in different directions is focused in order to expand the area where light can be detected.

[0340]

[0341] [Lidar device comprising a laser emitting element array, a transmitting optical system, a detecting element array, and a receiving optical system]

[0342] FIG. 10 is a drawing for explaining a lidar device disclosed by the present application.

[0343] Referring to FIG. 10, the lidar device (1800) disclosed by the present application may include a laser emitting element array (1810), a transmitting optic assembly (1820), a detecting element array (1850), and a receiving optic assembly (1860).

[0344] Laser light-emitting element array

[0345] Definition of a laser light-emitting element array

[0346] A laser light-emitting element array (1810) is defined as having a plurality of laser light-emitting elements arranged in an array form.

[0347] At this time, a plurality of laser light-emitting elements included in the laser light-emitting element array (1810) can be arranged in an array form on a single plane.

[0348] In addition, at this time, a plurality of laser light-emitting elements included in the laser light-emitting element array (1810) may be implemented to share at least one substrate.

[0349] Types of laser light-emitting element arrays

[0350] As described above, there can be various types of laser light-emitting devices.

[0351] Accordingly, the types of laser light-emitting element arrays (1810) in which multiple laser light-emitting elements are arranged in an array form can also vary.

[0352] However, in the technical field of a fixed-type lidar device, the laser-emitting element array (1810) is typically implemented as a VCSEL array.

[0353] This is due to the fact that, as mentioned above, in the case of VCSELs, they have a multilayer structure and emit lasers in the direction in which multiple layers are stacked, so arranging them in an array form on a single plane can be more advantageous.

[0354] Transmitting Optic Assembly

[0355] Function of the transmitting optic assembly

[0356] The transmitting optic assembly (1820) is configured to collimate and steer the laser output from the laser light-emitting element included in the laser light-emitting element array (1810) by utilizing phenomena such as refraction, diffraction, and reflection of light.

[0357] At this time, collimating the laser output from the laser emitting element can be defined as reducing the divergence angle of the laser output from the laser emitting element, but is not limited thereto, and includes concepts understood by a person skilled in the art as a function of collimating the laser.

[0358] In addition, steering the laser output from the laser emitting element may be defined as changing the path of the laser output from the laser emitting element, but is not limited thereto, and includes concepts understood by a person skilled in the art as a function of steering the laser.

[0359] When a transmitting optic assembly (1820) is used in a lidar device (1800), the laser output from the laser emitting element is collimated by the transmitting optic assembly (1820), thereby reducing energy loss due to the long-distance flight of the laser, and thereby enabling the lidar device to measure the distance to an object located at a greater distance.

[0360] In addition, when a transmitting optic assembly (1820) is used in the lidar device (1800), the laser output from the laser emitting element is steered by the transmitting optic assembly (1820), thereby enabling the laser's path of travel to be changed in a direction different from the output direction of the laser output from the laser emitting element.

[0361] Types of optics constituting the transmitting optic assembly

[0362] Additionally, the transmitting optic assembly (1820) is composed of a combination of one or more optics, and the types of optics constituting the transmitting optic assembly (1820) may vary.

[0363] For example, the types of optics constituting the transmitting optic assembly (1820) may be lenses, prisms, micro lenses, and meta lenses, but are not limited thereto, and may be various types of optics.

[0364] In addition, the types of lenses constituting the transmitting optic assembly (1820) may vary.

[0365] For example, the type of lens constituting the transmitting optic assembly (1820) may be a convex lens, a concave lens, a biconvex lens, a plano-convex lens, a convex meniscus lens, a biconcave lens, a plano-concave lens, a concave meniscus lens, an equi-convex lens, or an equi-concave lens.

[0366] In addition, for example, the type of lens constituting the transmitting optic assembly (1820) may be a spherical lens, an aspherical lens, or a cylindrical lens.

[0367] In addition, for example, the type of lens constituting the transmitting optic assembly (1820) may be a symmetry lens or an asymmetry lens.

[0368] Structure of the transmitting optic assembly - various combinations

[0369] The transmitting optic assembly (1820) may be composed of a combination of one or more optics, and in this case, may be implemented as a combination of various types of optics.

[0370] The transmitting optic assembly (1820) can be implemented as a single lens.

[0371] For example, the transmitting optic assembly (1820) can be implemented as a single convex lens or a single concave lens.

[0372] Additionally, the transmitting optic assembly (1820) can be implemented as a composite lens composed of multiple lenses.

[0373] For example, the transmitting optic assembly (1820) may be implemented as a composite lens composed of a combination of a plurality of convex lenses and a plurality of concave lenses.

[0374] Additionally, the transmitting optic assembly (1820) can be implemented as a combination of multiple composite lenses.

[0375] For example, the transmitting optic assembly (1820) can be implemented as a combination of a first composite lens which is a symmetric lens and a second composite lens which is an asymmetric lens.

[0376] Additionally, the transmitting optic assembly (1820) can be implemented with various combinations of various optics for collimating and steering the laser output from the laser emitting element, in addition to the examples described above.

[0377] Structure of the transmitting optic assembly - Lens layer structure

[0378] When the transmitting optic assembly (1820) is configured to include a composite lens, the composite lens may include a plurality of lens layers stacked along a common axis.

[0379] At this time, a plurality of lens layers may be aligned and arranged such that the optical axis of each of the plurality of lens layers coincides with the common axis, and the optical axis of each of the plurality of lens layers refers to a virtual axis perpendicular to the surface that passes through the center of each of the plurality of lens layers.

[0380] In addition, at this time, the optical axis of the composite lens may correspond to a virtual axis in which the optical axes of each of the plurality of lens layers included in the composite lens are aligned.

[0381] Laser irradiation direction according to the relative positional relationship between the transmitting optic assembly and the laser emitter

[0382] Placement relationship between the transmitting optic assembly and the laser emitter

[0383] When a laser emitting element and a transmitting optic assembly (1820) are used in a lidar device (1800), the laser emitting element is positioned to output a laser toward the transmitting optic assembly (1820).

[0384] For example, a first laser emitting element (1811) and a second laser emitting element (1812) included in a laser emitting element array (1810) are arranged to output a laser toward a transmitting optic assembly (1820).

[0385] At this time, the laser emitting element can be positioned so that the laser output from the laser emitting element travels parallel to the optical axis (1821) of the transmitting optic assembly (1820).

[0386] For example, a first laser emitting element (1811) included in a laser emitting element array (1810) may be arranged so that a first laser (1831) output from the first laser emitting element (1811) travels parallel to the optical axis (1821) of the transmitting optic assembly (1820), and a second laser emitting element (1812) may be arranged so that a second laser (1832) output from the second laser emitting element (1812) travels parallel to the optical axis (1821) of the transmitting optic assembly (1820).

[0387] Of course, the fact that the laser output from the laser emitting element travels parallel to the optical axis (1821) of the transmitting optic assembly (1820) does not mean that all light rays of the laser output from the laser emitting element travel parallel to the optical axis (1821) of the transmitting optic assembly (1820), but includes that at least some of the light rays of the laser output from the laser emitting element travel parallel to the optical axis (1821) of the transmitting optic assembly (1820).

[0388] Laser irradiation angle and direction according to the relative positional relationship between the transmitting optic assembly and the laser emitter

[0389] The laser output from the laser emitting element can be collimated and steered as it passes through the transmitting optic assembly (1820) so that it can be irradiated in a specific direction.

[0390] For example, a first laser (1831) output from a first laser emitting element (1811) included in a laser emitting element array (1810) is collimated and steered as it passes through a transmitting optic assembly (1820) and irradiated in a first direction, and a second laser (1832) output from a second laser emitting element (1812) is collimated and steered as it passes through a transmitting optic assembly (1820) and irradiated in a second direction.

[0391] At this time, the direction in which the laser output from the laser emitting element is irradiated by steering by the transmitting optic assembly (1820) may vary depending on the relative positional relationship between the transmitting optic assembly (1820) and the laser emitting element.

[0392] For example, if the transmitting optic assembly (1820) is implemented as a symmetric lens, the angle between the direction of propagation of the laser passing through the transmitting optic assembly (1820) and the optical axis (1821) of the transmitting optic assembly (1820) may vary depending on the distance between the laser emitting element located within the focal plane of the transmitting optic assembly (1820) and the optical axis (1821) of the transmitting optic assembly (1820).

[0393] For a more specific example, if the transmitting optic assembly (1820) is implemented as a symmetric lens, as the distance between the laser emitting element located within the focal plane of the transmitting optic assembly (1820) and the optical axis (1821) of the transmitting optic assembly (1820) increases, the angle between the direction of propagation of the laser passing through the transmitting optic assembly (1820) and the optical axis (1821) of the transmitting optic assembly (1820) may increase.

[0394] Additionally, for example, if the transmitting optic assembly (1820) is implemented as a symmetric lens, the direction of travel of the laser passing through the transmitting optic assembly (1820) and steering may vary depending on the direction in which the laser emitting element is positioned within the focal plane of the transmitting optic assembly (1820) from the optical axis (1821) of the transmitting lens assembly (1820).

[0395] Additionally, for example, if the transmitting optic assembly (1820) is implemented as a cylindrical lens that steers the laser in the first axial direction, the angle between the direction of travel of the laser passing through the transmitting optic assembly (1820) and the optical axis (1821) of the transmitting optic assembly (1820) and the transmitted optic assembly (1820) may vary depending on the distance in the first axial direction between the laser emitting element and the optical axis (1821) of the transmitting optic assembly (1820).

[0396] As described above, since the direction in which a laser emitted from a laser emitter travels through the transmitting optic assembly (1820) can be determined according to the relative positional relationship between the laser emitter and the transmitting optic assembly (1820), when using multiple laser emitters with different relative positional relationships with the transmitting optic assembly (1820) in a lidar device, it is possible to output multiple lasers traveling through the transmitting optic assembly (1820) in different directions.

[0397] Therefore, using a laser emitting element array (1810) and a transmitting optic assembly (1820) makes it possible to expand the area where the laser is irradiated.

[0398]

[0399] Relationship between a laser emitter array and a transmitting optic assembly

[0400] Placement relationship between the transmitting optic assembly and the laser emitting element array

[0401] The laser emitting element array (1810) can be positioned on the focal plane of the transmitting optic assembly (1820).

[0402] Of course, if necessary, the laser emitting element array (1810) may be positioned to have a preset offset from the focal plane of the transmitting optic assembly (1820), but for convenience of explanation, the description is based on the assumption that the laser emitting element array (1810) is positioned on the focal plane of the transmitting optic assembly (1820).

[0403] Additionally, the laser emitting element array (1810) may be arranged so that a plurality of lasers output from a plurality of laser emitting elements of the laser emitting element array (1810) travel parallel to the optical axis (1821) of the transmitting optic assembly (1820).

[0404] Irradiation direction of lasers output from the laser-emitting element array and passed through the transmitting optic assembly

[0405] Each of the multiple lasers output from the multiple laser-emitting elements of the laser-emitting element array (1810) can be collimated and steered as it passes through the transmitting optic assembly (1820) so as to be irradiated in different directions.

[0406] At this time, the direction in which the lasers output from the plurality of laser emitting elements of the laser emitting element array (1810) are irradiated by steering by the transmitting optic assembly (1820) depends on the relative positional relationship between the transmitting optic assembly (1820) and each of the plurality of laser emitting elements.

[0407] That is, the direction of irradiation of the laser output from each of the plurality of laser emitting elements can be determined according to the relative position of each of the transmitting optic assembly (1820) and each of the plurality of laser emitting elements.

[0408] For example, the first direction in which the first laser (1831) output from the first laser emitting element (1811) included in the laser emitting element array (1810) is irradiated by steering by the transmitting optic assembly (1820) is determined according to the relative position between the optical axis (1821) of the transmitting optic assembly (1820) and the first laser emitting element (1811).

[0409] Additionally, for example, the second direction in which the second laser (1832) output from the second laser emitting element (1812) included in the laser emitting element array (1810) is irradiated by steering by the transmitting optic assembly (1820) is determined according to the relative position between the optical axis (1821) of the transmitting optic assembly (1820) and the second laser emitting element (1812).

[0410] This can be explained as the emission angle or steering angle of the laser output from each of the plurality of laser emitting elements being determined according to the relative position of each of the transmitting optic assembly (1820) and each of the plurality of laser emitting elements.

[0411] Laser irradiation area of ​​the LiDAR device

[0412] Laser irradiation area of ​​the LiDAR device

[0413] In this specification, the laser irradiation area of ​​the lidar device (1800) is a concept for comprehensively describing the space where a laser output from the lidar device (1800) can be irradiated.

[0414] At this time, the laser irradiation area of ​​the lidar device (1800) may include not only a space where a laser output from the lidar device (1800) is irradiated at a certain time, but also a space where a laser output from the lidar device (1800) can be irradiated for a certain period of time.

[0415] Refer to Fig. 11 to explain the laser irradiation area of ​​the lidar device in more detail.

[0416] Figure 11 is a diagram illustrating the laser irradiation area of ​​a lidar device.

[0417] More specifically, FIG. 11 is a diagram briefly illustrating, for convenience of explanation, that each of the multiple lasers output from multiple laser-emitting elements of a laser-emitting element array is steered and irradiated in different directions as it passes through a transmitting optic assembly, and is used to explain the laser irradiation area of ​​a lidar device based on this.

[0418] Referring to FIG. 11, each of the multiple lasers output from the multiple laser-emitting elements of the laser-emitting element array of the lidar device (1840) is steered as it passes through the transmitting optic assembly and irradiated in different directions.

[0419] That is, the first laser (1841) output from the lidar device (1840) is output from the first laser emitting element and is irradiated in the first direction as it passes through the transmitting optic assembly, the second laser (1842) is output from the second laser emitting element and is irradiated in the second direction as it passes through the transmitting optic assembly, the third laser (1843) is output from the third laser emitting element and is irradiated in the third direction as it passes through the transmitting optic assembly, and the fourth laser (1844) is output from the fourth laser emitting element and is irradiated in the fourth direction as it passes through the transmitting optic assembly.

[0420] At this time, referring again to FIG. 11, it can be seen that a plurality of lasers output from the lidar device (1840) are irradiated within a specific space, and to express this further, the set of spaces where a plurality of lasers output from the lidar device (1840) are irradiated can be expressed as being defined as a specific space.

[0421] Accordingly, in this specification, a specific space in which a plurality of lasers output from a lidar device (1840) are irradiated is described as a laser irradiation area (1845).

[0422] That is, in the example illustrated in FIG. 11, a square pyramid space with the lidar device (1840) as a vertex can be the laser irradiation area (1845).

[0423] At this time, referring again to FIG. 11, a specific space where a plurality of lasers output from the lidar device (1840) are irradiated can be a space with a shape that spreads out from the lidar device (1840) as the origin.

[0424] That is, multiple lasers output from the lidar device (1840) can be irradiated in different directions with the lidar device (1840) as a reference point, and accordingly, the specific space where the multiple lasers output from the lidar device (1840) are irradiated can be a space with a shape that spreads out with the lidar device (1840) as the origin.

[0425] Accordingly, a coordinate system with the lidar device (1840) as the origin can be effectively used to mathematically describe the laser irradiation area (1845) of the lidar device described above.

[0426] For example, to mathematically describe the size of the laser irradiation area (1845) of the above-described lidar device, the angle between the lasers irradiated to the outermost edge in a coordinate system with the lidar device (1840) as the origin may be used.

[0427] More specifically, to mathematically describe the size of the laser irradiation area (1845) in the horizontal direction, the angle between the direction in which the first laser (1841) is irradiated and the direction in which the second laser (1842) is irradiated may be used in a coordinate system with the lidar device (1840) as the origin, and to mathematically describe the size of the laser irradiation area (1845) in the vertical direction, the angle between the direction in which the third laser (1843) is irradiated and the direction in which the fourth laser (1844) is irradiated may be used in a coordinate system with the lidar device (1840) as the origin.

[0428] At this time, in this specification, the size of the laser irradiation area (1845) of the lidar device is expressed in terms of an angle and is described as the angle of view of the laser irradiation area (1845). Below, the horizontal angle of view and the vertical angle of view of the laser irradiation area of ​​the fixed lidar device will be described in more detail.

[0429] Horizontal and vertical angles of view of the laser irradiation area of ​​a fixed LiDAR device

[0430] Figure 12 is a diagram illustrating the horizontal and vertical angles of view of the laser irradiation area of ​​a fixed lidar device.

[0431] FIG. 12 shows only the laser emitting element array (1900) and the transmitting optic assembly (1950) of the configuration of the fixed lidar device for convenience of explanation.

[0432] As described above, the laser irradiation area (1960) of the lidar device is defined as an area where a laser output from the lidar device can be irradiated.

[0433] Accordingly, the horizontal and vertical angles of view of the laser irradiation area (1960) of the lidar device represent the range of angles at which a laser output from the lidar device can be irradiated, and can be defined by the lasers irradiated at the outermost edge.

[0434] At this time, in a fixed lidar device comprising a laser emitting element array (1900) and a transmitting optic assembly (1950), the area where a laser output from each of the plurality of laser emitting elements can be irradiated may be defined by the relative positional relationship with the transmitting optic assembly (1950).

[0435] Accordingly, in a fixed lidar device comprising a laser emitting element array (1900) and a transmitting optic assembly (1950), the horizontal angle of view of the laser irradiation area (1960) can be defined by a first angle (1961), which is the angle between the irradiation direction after passing through the transmitting optic assembly (1950) of the first laser (1911) output from the first laser emitting element (1910) positioned at the first end among a plurality of laser emitting elements arranged in a center row, and the irradiation direction after passing through the transmitting optic assembly (1950) of the second laser (1921) output from the second laser emitting element (1920) positioned at the second end.

[0436] Additionally, in a fixed lidar device comprising a laser emitting element array (1900) and a transmitting optic assembly (1950), the vertical angle of view of the laser irradiation area (1960) can be defined by a second angle (1962), which is the angle between the irradiation direction after passing through the transmitting optic assembly (1950) of the third laser (1931) output from the third laser emitting element (1930) positioned at the third end among a plurality of laser emitting elements arranged in a central column, and the irradiation direction after passing through the transmitting optic assembly (1950) of the fourth laser (1941) output from the fourth laser emitting element (1940) positioned at the fourth end.

[0437] Detecting element array

[0438] Definition of a Detecting Element Array

[0439] The detecting element array (1850) is defined as a plurality of detecting elements arranged in an array form.

[0440] At this time, a plurality of detecting elements included in the detecting element array (1850) can be arranged in an array form on a single plane.

[0441] In addition, at this time, a plurality of detecting elements included in the detecting element array (1850) may be implemented to share at least one substrate.

[0442] Types of Detecting Element Arrays

[0443] As described above, there can be various types of detecting elements.

[0444] Accordingly, the types of the detection element array (1850), in which multiple detection elements are arranged in an array form, can also vary.

[0445] Receiving Optic Assembly

[0446] Functions of the receiving optic assembly

[0447] The receiving optic assembly (1860) is configured to focus light incident on the receiving optic assembly (1860) to a detecting element by utilizing phenomena such as refraction, diffraction, and reflection of light.

[0448] Types of optics that make up the receiving optic assembly

[0449] Additionally, the receiving optic assembly (1860) is composed of a combination of one or more optics, and the types of optics constituting the receiving optic assembly (1860) may vary.

[0450] For example, the types of optics constituting the receiving optic assembly (1860) may be lenses, prisms, micro lenses, and meta lenses, but are not limited thereto, and may be various types of optics.

[0451] In addition, the types of lenses constituting the receiving optic assembly (1860) may vary.

[0452] For example, the type of lens constituting the receiving optic assembly (1860) may be a convex lens, a concave lens, a biconvex lens, a plano-convex lens, a convex meniscus lens, a biconcave lens, a plano-concave lens, a concave meniscus lens, an equi-convex lens, or an equi-concave lens.

[0453] In addition, for example, the type of lens constituting the receiving optic assembly (1860) may be a spherical lens, an aspherical lens, or a cylindrical lens.

[0454] In addition, for example, the type of lens constituting the receiving optic assembly (1860) may be a symmetry lens or an asymmetry lens.

[0455] Structure of the Receiving Optic Assembly - Various Combinations

[0456] The receiving optic assembly (1860) may be composed of a combination of one or more optics, and in this case, may be implemented as a combination of various types of optics.

[0457] The receiving optic assembly (1860) can be implemented as a single lens.

[0458] For example, the receiving optic assembly (1860) can be implemented as a single convex lens or a single concave lens.

[0459] Additionally, the receiving optic assembly (1860) can be implemented as a composite lens composed of multiple lenses.

[0460] For example, the receiving optic assembly (1860) may be implemented as a composite lens composed of a combination of a plurality of convex lenses and a plurality of concave lenses.

[0461] Additionally, the receiving optic assembly (1860) can be implemented as a combination of multiple composite lenses.

[0462] For example, the receiving optic assembly (1860) can be implemented as a combination of a first composite lens which is a symmetric lens and a second composite lens which is an asymmetric lens.

[0463] In addition, the receiving optic assembly (1860) can be implemented with various combinations of various optics to focus light incident on the receiving optic assembly to the detecting element, in addition to the examples described above.

[0464] Additionally, the receiving optic assembly (1860) may be implemented to include a band pass filter that passes only light of a specific wavelength band.

[0465] At this time, the transmission band of the bandpass filter included in the receiving optic assembly (1860) is provided to include the wavelength band of the laser output from the laser emitting element.

[0466] In particular, the receiving optic assembly (1860) used in the lidar device is implemented to include a bandpass filter, thereby blocking light in a wavelength band other than the wavelength band of the laser output from the laser emitting element and selectively transmitting only the light in the wavelength band of the laser output from the laser emitting element, so as to significantly reduce the external light reaching the detecting element.

[0467] Structure of the receiving optic assembly - Lens layer structure

[0468] When the receiving optic assembly (1860) is configured to include a composite lens, the composite lens may include a plurality of lens layers stacked along a common axis.

[0469] At this time, a plurality of lens layers may be aligned and arranged such that the optical axis of each of the plurality of lens layers coincides with the common axis, and the optical axis of each of the plurality of lens layers refers to a virtual axis perpendicular to the surface that passes through the center of each of the plurality of lens layers.

[0470] In addition, at this time, the optical axis of the composite lens may correspond to a virtual axis in which the optical axes of each of the plurality of lens layers included in the composite lens are aligned.

[0471] Light detection direction according to the relative positional relationship between the receiving optic assembly and the detecting element

[0472] Placement relationship between the receiving optic assembly and the detecting element

[0473] When a detecting element and a receiving optic assembly (1860) are used in a lidar device (1800), the detecting element is located in the focal plane of the receiving optic assembly (1860) and is positioned in a direction facing the receiving optic assembly (1860).

[0474] Light detection angle and direction according to the relative positional relationship between the receiving optic assembly and the detecting element

[0475] Light incident from a specific direction on the receiving optic assembly (1860) can be focused by the detecting element.

[0476] At this time, the point where the light incident on the receiving optic assembly (1860) is focused by the receiving optic assembly (1860) may vary depending on the direction of the light incident on the receiving optic assembly (1860).

[0477] For example, light (1871) incident in a first direction to the receiving optic assembly (1860) is focused by the receiving optic assembly (1860) to a point where the first detecting element (1851) is located, and light (1872) incident in a second direction to the receiving optic assembly (1860) is focused by the receiving optic assembly (1860) to a point where the second detecting element (1852) is located.

[0478] That is, depending on the relative positional relationship between the receiving optic assembly (1860) and the detecting element, the incident direction or angle of light that can be focused on the detecting element among the light incident on the receiving optic assembly (1860) can be determined.

[0479] For example, if the receiving optic assembly (1860) is implemented as a symmetric lens, the angle between the optical axis (1861) of the receiving optic assembly and the direction of incidence of light that can be focused to the detecting element after passing through the receiving optic assembly (1860) may vary depending on the distance between the detecting element located within the focal plane of the receiving optic assembly (1860) and the optical axis (1861) of the receiving optic assembly (1860).

[0480] For a more specific example, when the receiving optic assembly (1860) is implemented as a symmetric lens, as the distance between the detecting element located within the focal plane of the receiving optic assembly (1860) and the optical axis (1861) of the receiving optic assembly (1860) increases, the angle between the incident direction of light that can be focused to the detecting element after passing through the receiving optic assembly (1860) and the optical axis (1861) of the receiving optic assembly (1860) may increase.

[0481] Additionally, for example, if the receiving optic assembly (1860) is implemented as a symmetric lens, the direction of incidence of light that can be focused to the detecting element after passing through the receiving optic assembly (1860) may vary depending on the direction in which the detecting element is positioned within the focal plane of the receiving optic assembly (1860) from the optical axis (1861) of the receiving lens assembly (1860).

[0482] As described above, the incident direction of light that can be focused to the detecting element after passing through the receiving lens assembly (1860) can be determined according to the relative positional relationship between the detecting element and the receiving optic assembly (1860). Therefore, when multiple detecting elements with different relative positional relationships with the receiving optic assembly (1860) are used in the lidar device, light incident in different incident directions can pass through the receiving optic assembly (1860) and be focused to each of the multiple detecting elements.

[0483] Therefore, using the detecting element array (1850) and the receiving optic assembly (1860) makes it possible to expand the area in which light can be detected.

[0484] Relationship between the detecting element array and the receiving optic assembly

[0485] Placement relationship between the receiving optic assembly and the detecting element array

[0486] The detecting element array (1850) can be positioned on the focal plane of the receiving optic assembly (1860).

[0487] Of course, if necessary, the detecting array (1850) may be positioned to have a preset offset from the focal plane of the receiving optic assembly (1860), but for convenience of explanation, the description is based on the assumption that the detecting element array (1850) is positioned on the focal plane of the receiving optic assembly (1860).

[0488] The direction of incidence of light on the receiving optic assembly that can be focused onto the detecting element array after passing through the receiving optic assembly

[0489] Light incident on the receiving optic assembly (1860) in different directions can pass through the receiving optic assembly (1860) and be focused into each of the multiple detecting elements of the detecting element array (1850).

[0490] At this time, the incident direction of the light receiving optic assembly (1860) that can be focused on each of the plurality of detecting elements varies depending on the relative positional relationship between the receiving optic assembly (1860) and each of the plurality of detecting elements.

[0491] That is, the direction of light that each of the multiple detecting elements can detect (incident direction with respect to the receiving optic assembly) can be determined according to the relative position of each of the receiving optic assembly (1860) and each of the multiple detecting elements.

[0492] For example, the incident direction of light to the receiving optic assembly (1860) that can be detected by the first detecting element (1851) included in the detecting element array (1850) is determined according to the relative position between the optical axis (1861) of the receiving optic assembly (1860) and the first detecting element (1851).

[0493] Additionally, for example, the incident direction of light to the receiving optic assembly (1860) that can be detected by the second detecting element (1852) included in the detecting element array (1850) is determined according to the relative position between the optical axis (1861) of the receiving optic assembly (1860) and the second detecting element (1852).

[0494] In other words, light incident on the receiving optic assembly (1860) in a specific direction is focused on a specific detecting element among a plurality of detecting elements, and whether it is focused on which detecting element is due to the direction of the light incident on the receiving optic assembly (1860).

[0495] This can be explained as the incident direction or incident angle of light that each of the plurality of detecting elements can detect is determined according to the relative position of each of the receiving optic assembly (1860) and each of the plurality of detecting elements.

[0496] Light detection area of ​​the LiDAR device

[0497] Light detection area of ​​the LiDAR device

[0498] In this specification, the light-sensing area of ​​the lidar device (1800) is a concept for comprehensively describing the space in which light can be detected by the lidar device (1800).

[0499] At this time, the light detection area of ​​the lidar device (1800) may include not only a space where light can be detected from the lidar device (1800) at a certain time, but also a space where light can be detected from the lidar device (1800) for a certain period of time.

[0500] Refer to Fig. 13 to describe the light detection area of ​​the lidar device in more detail.

[0501] Figure 13 is a diagram illustrating the light detection area of ​​a lidar device.

[0502] More specifically, FIG. 13 is a diagram for explaining the light detection area of ​​a lidar device based on the following, which briefly illustrates, for convenience of explanation, that light incident in different directions on a receiving optic assembly passes through the receiving optic assembly and is focused on each of the multiple detecting elements of the detecting element array.

[0503] Referring to FIG. 13, light incident in different directions on the receiving optic assembly of the lidar device (1880) passes through the receiving optic assembly and is focused on each of the plurality of detecting elements of the lidar device (1880).

[0504] That is, the first light (1881) incident in the first direction on the lidar device (1880) is focused to the first detecting element as it passes through the receiving optic assembly, the second light (1882) incident in the second direction is focused to the second detecting element as it passes through the receiving optic assembly, the third light (1883) incident in the third direction is focused to the third detecting element as it passes through the receiving optic assembly, and the fourth light (1884) incident in the fourth direction is focused to the fourth detecting element as it passes through the receiving optic assembly.

[0505] At this time, referring again to FIG. 13, it can be seen that the light focused by the detection element array of the lidar device (1880) is incident on the lidar device (1880) within a specific space, and to express this further, the set of light focused by the detection element array of the lidar device (1880) can be expressed as being defined as a specific space.

[0506] Accordingly, in this specification, a specific space into which light focused by the array of detecting elements of the lidar device (1880) is incident is described as a light detection area (1885).

[0507] That is, in the example illustrated in FIG. 13, a square pyramid space with the lidar device (1880) as a vertex can be the light sensing area (1845).

[0508] At this time, referring again to FIG. 13, the specific space into which light focused to the detection element array of the lidar device (1880) is incident can be a space with a shape that converges to the lidar device (1880) as the origin.

[0509] That is, the light focused into the detection element array of the lidar device (1880) can be incident from different directions toward the lidar device (1880), and accordingly, the specific space into which the light focused into the detection element array of the lidar device (1880) is incident can be a space with a shape that converges to the lidar device (1880) as the origin.

[0510] Accordingly, a coordinate system with the lidar device (1880) as the origin can be effectively used to mathematically describe the light detection area (1885) of the lidar device described above.

[0511] For example, to mathematically describe the size of the light detection area (1885) of the above-described lidar device, the angle between the lights incident at the outermost edge and focused into the detection element array in a coordinate system with the lidar device (1880) as the origin may be used.

[0512] More specifically, to mathematically describe the size of the light detection area (1885) in the horizontal direction, the angle between the direction in which the first light (1881) is incident and the direction in which the second light (1882) is incident may be used in a coordinate system with the lidar device (1880) as the origin, and to mathematically describe the size of the light detection area (1885) in the vertical direction, the angle between the direction in which the third light (1883) is incident and the direction in which the fourth light (1884) is incident may be used in a coordinate system with the lidar device (1880) as the origin.

[0513] At this time, in this specification, the size of the light detection area (1885) of the LiDAR device is expressed in terms of an angle and is described as the angle of view of the light detection area (1885). Below, the horizontal angle of view and the vertical angle of view of the light detection area of ​​the fixed LiDAR device will be described in more detail.

[0514]

[0515] Horizontal and vertical angles of view of the light detection area of ​​a fixed LiDAR device

[0516] Figure 14 is a diagram illustrating the horizontal and vertical angles of view of the light detection area of ​​a fixed lidar device.

[0517] FIG. 14 illustrates only the detecting element array (2000) and the receiving optic assembly (2050) of the configuration of the fixed lidar device for convenience of explanation.

[0518] As described above, the light detection area (2060) of the lidar device is defined as an area in which the lidar device can detect light.

[0519] Accordingly, the horizontal and vertical angles of view of the light detection area (2060) of the lidar device represent the range of angles at which the lidar device can detect light, and can be defined by the outermost incident directions among the incident directions to the receiving optic assembly of the light that can be detected by the lidar device.

[0520] At this time, in a fixed lidar device comprising a detecting element array (2000) and a receiving optic assembly (2050), the incident direction or incident angle of light that each of the plurality of detecting elements can detect with respect to the receiving optic assembly can be determined by the relative positional relationship between the receiving optic assembly (2050) and each of the plurality of detecting elements.

[0521] Accordingly, in a fixed lidar device comprising a detecting element array (2000) and a receiving optic assembly (2050), the horizontal angle of view of the light detection area (2060) can be defined by a third angle (2061), which is the angle between the first incident direction of light (2011) focused by a first detecting element (2010) positioned at the first end among a plurality of detecting elements arranged in a center row and the second incident direction of light (2021) focused by a second detecting element (2020) positioned at the second end and the receiving optic assembly (2050).

[0522] Additionally, in a fixed lidar device comprising a detecting element array (2000) and a receiving optic assembly (2050), the vertical angle of view of the light detection area (2060) may be defined by a fourth angle (2062), which is the angle between the third incident direction of light (2031) focused on the receiving optic assembly (2050) by the third detecting element (2030) positioned at the third end among a plurality of detecting elements arranged in a central column, and the fourth incident direction of light focused on the receiving optic assembly (2050) by the fourth detecting element (2040) positioned at the fourth end.

[0523] Angle of view of the LiDAR device

[0524] As described above, since the lidar device is a device for measuring the distance between the lidar device and an object using a laser, the lidar device can measure the distance between the lidar device and an object only when the following conditions are satisfied.

[0525] i) An object is located within the area where the laser output from the lidar device is irradiated.

[0526] ii) A laser reflected from an object is received by a detecting element of the lidar device.

[0527] Therefore, the measurable area of ​​the lidar device becomes the area where the laser irradiation area of ​​the lidar device described above and the light detection area of ​​the lidar device described above overlap.

[0528] In this context, the field of view of a lidar device is a concept that expresses the device's measurable range as an angle relative to a specific origin.

[0529] In addition, when manufacturing a lidar device, the laser irradiation area and the light detection area of ​​the lidar device are generally manufactured to be aligned with each other at a specific distance.

[0530] Accordingly, the field of view of the lidar device can be defined as the vertical and horizontal fields of view of the laser irradiation area described above, and can also be defined as the vertical and horizontal fields of view of the light detection area described above.

[0531]

[0532] Although omitted in FIG. 10, the lidar device (1800) disclosed by the present application may further include a bandpass filter.

[0533] Band pass filter

[0534] A bandpass filter is configured to allow only light of a specific wavelength band to pass through.

[0535] At this time, the transmission band of the bandpass filter is configured to include the wavelength band of the laser output from the laser emitting element.

[0536] In addition, at this time, the bandpass filter may be located inside the receiving optic assembly, and more specifically, may be located between a plurality of lenses constituting the receiving optic assembly.

[0537] In addition, at this time, the bandpass filter may be located outside the receiving optic assembly, or it may be located between the receiving optic assembly and the detecting element array.

[0538]

[0539] [Optical connection between laser emitting element array and detecting element array]

[0540] Definition of optical connection

[0541] As described above, each of the multiple lasers output from the multiple laser emitting elements of the laser emitting element array can be collimated and steered to be irradiated in different directions as it passes through the transmitting optic assembly, and the light incident in different directions on the receiving optic assembly can pass through the receiving optic assembly and be focused on each of the multiple detecting elements of the detecting element array.

[0542] At this time, the lidar device may be configured such that a laser output from a specific laser emitter of a laser emitter array passes through a transmitting optic assembly and is irradiated in a specific direction, and when the laser is reflected from an object located at a preset distance from the lidar device, the laser is focused through a receiving optic assembly to a specific detecting element of a detecting element array.

[0543] In this specification, the relationship between the specific laser emitting element and the specific detecting element described above is described as a relationship in which the specific laser emitting element and the specific detecting element are optically connected to each other.

[0544] That is, in this specification, a laser emitting element and a detecting element are defined as being optically connected to each other in a relationship where the laser irradiation direction and the light detection direction, defined according to the relationship between the optical configurations of the lidar device, are matched with each other.

[0545] Various optical connections between laser emitting element arrays and detecting element arrays

[0546] A lidar device is configured such that a plurality of laser-emitting elements of a laser-emitting element array and a plurality of detecting elements of a detecting element array are optically connected to each other, and the optical connection relationship between the plurality of laser-emitting elements and the plurality of detecting elements can be varied.

[0547] For example, a lidar device can be configured so that one laser-emitting element is optically connected to one detecting element.

[0548] That is, the lidar device can be configured so that each of the plurality of laser-emitting elements of the laser-emitting element array is optically connected to each of the plurality of detecting elements of the detecting element array.

[0549] For example, in a more specific configuration of a lidar device, when a plurality of laser-emitting elements of a laser-emitting element array are arranged in a two-dimensional array consisting of M rows and N columns, and a plurality of detecting elements of a detecting element array are arranged in a two-dimensional array consisting of M rows and N columns, a laser-emitting element located at (X,Y) can be optically connected to a detecting element located at (X,Y).

[0550] In addition, for example, a lidar device may be configured such that a single laser-emitting element is optically connected to a plurality of detecting elements.

[0551] That is, the lidar device can be configured such that each of the plurality of laser-emitting elements of the laser-emitting element array is optically connected to each of the portion groups of the plurality of detecting elements of the detecting element array.

[0552] For example, in a more specific configuration of a lidar device, when multiple laser-emitting elements of a laser-emitting element array are arranged in a two-dimensional array consisting of M rows and N columns, and multiple detecting elements of a detecting element array are arranged in a two-dimensional array consisting of 3M rows and 3N columns, one laser-emitting element can be optically connected to nine detecting elements.

[0553] In addition, for example, a lidar device may be configured such that a plurality of laser-emitting elements are optically connected to a single detecting element.

[0554] That is, the lidar device can be configured such that each of a group of multiple laser-emitting elements in a laser-emitting element array is optically connected to each of a plurality of detecting elements in a detecting element array.

[0555] For example, in a more specific configuration of a lidar device, when multiple laser-emitting elements of a laser-emitting element array are arranged in a two-dimensional array consisting of 3M rows and 3N columns, and multiple detecting elements of a detecting element array are arranged in a two-dimensional array consisting of M rows and N columns, nine laser-emitting elements can be optically connected to one detecting element.

[0556] In addition, for example, a lidar device may be configured such that a plurality of laser-emitting elements are optically connected to a plurality of detecting elements.

[0557] That is, the lidar device can be configured such that each of the groups of multiple laser-emitting elements in a laser-emitting element array is optically connected to each of the groups of multiple detecting elements in a detecting element array.

[0558] For example, in a more specific configuration of a lidar device, when a plurality of laser-emitting elements of a laser-emitting element array are arranged in a two-dimensional array consisting of A*M rows and B*N columns, and a plurality of detecting elements of a detecting element array are arranged in a two-dimensional array consisting of C*M rows and D*N columns, A*B laser-emitting elements can be optically connected to C*D detecting elements.

[0559]

[0560] [Lidar Data]

[0561] Definition of LiDAR data

[0562] Through FIGS. 1 to 8, the process of generating a histogram through a series of operations of a laser emitting element and a detecting element that are optically connected to each other was explained, and furthermore, through the generated histogram, how the echo signal is determined, how the distance to the target is estimated, and how the reflection intensity of the laser on the target is estimated were explained in detail.

[0563] Meanwhile, it was explained that laser emitting elements can be provided in the form of an array and detecting elements can also be provided in the form of an array, wherein each laser emitting element has its own unique orientation and furthermore, each detecting element also has its own unique orientation, and it was explained in detail that at least one laser emitting element and at least one detecting element can be optically connected.

[0564] Ultimately, in a lidar in which both laser-emitting elements and detecting elements are implemented in an array form, the optically connected laser-emitting elements and detecting elements can be conceptually defined as a single laser-detector pair or a single laser-detector set.

[0565] In this specification, LiDAR data is defined as a set of histograms, echo signals, distances, and / or reflection intensities acquired by each laser-detector pair over a predetermined very short period of time.

[0566] At this time, by interpreting the lidar data acquired by all laser-detector pairs, information about a specific scene within the field of view of the lidar device (hereinafter, information about a specific scene obtained through the interpretation of lidar data, etc., can be referred to as frame information) can be acquired, and by controlling the period for acquiring lidar data, the resolution on the time axis of the information about scenes within the field of view of the lidar device (e.g., frames per second (fps)) can be controlled.

[0567] Below, LiDAR data is explained in more detail.

[0568]

[0569] Composition of LiDAR data

[0570] FIG. 15 is a drawing for explaining lidar data disclosed through the present application.

[0571] Referring to FIG. 15, the lidar data (2100) disclosed in this application consists of a plurality of pixel position coordinates (2110) and at least one pixel value (2120) corresponding to the plurality of pixel position coordinates (2110).

[0572] At this time, the LiDAR data (2100) disclosed through the present application may be generated or output in frame units, and the following descriptions will be explained based on one frame of LiDAR data (2100).

[0573] Definition of a pixel and the relationship between a pixel and a detection element array

[0574] As described above, in this specification, LiDAR data (2100) is described as a set of pixel values ​​corresponding to each of a plurality of pixels. Generally, a pixel is a concept used in the display field, and similar to the concept used in the display field, it can be distinguished from one another by coordinate values ​​corresponding to each pixel, and in this specification, the coordinates of the pixels are coordinates that can be defined within an array of detecting elements used to obtain the pixel values ​​corresponding to those pixels.

[0575] For example, the first coordinate of the first detecting element is different from the second coordinate of the second detecting element.

[0576] In this case, a single pixel value may be obtained from a single detecting device, or a single pixel value may be obtained from multiple detecting devices. For the sake of convenience of explanation, it will not be clearly distinguished below whether the pixel value was obtained from a single detecting device or from multiple detecting devices.

[0577] Pixel location coordinates

[0578] Generally, a detection element array can be implemented in the form of a grid where the coordinates of the detection elements have rows and columns, and in this case, the coordinates of a single detection element can be defined by the row number of the row to which the detection element belongs and the column number of the column to which the detection element belongs.

[0579] For example, in a grid-shaped array of detecting elements consisting of M rows and N columns, if a specific detecting element is placed in the x-th row and y-th column, the coordinates of the specific detecting element can be defined as (x,y), and the pixel coordinates corresponding to the pixel value obtained from the specific detecting element can be used as (x,y).

[0580] Of course, in this case, coordinate values ​​transformed based on the coordinates of the detecting element may be used as pixel coordinates. For example, if the position of the detecting element and the direction in which light received by the detecting element is incident on the receiving optic assembly are reversed from each other by the receiving optic assembly described above, the position coordinates of the first pixel corresponding to the first detecting element located at (1,1) within the detecting element array may be used as (M,N), and the position coordinates of the M*N pixel corresponding to the M*N detecting element located at (M,N) within the detecting element array may be used as (1,1).

[0581] pixel value

[0582] Referring again to FIG. 15, at least one pixel value (2120) constituting the lidar data (2100) may include any one of a histogram (2121), an echo signal (2122), a distance (2123), a reflection intensity (2124), or a representative counting value (2125).

[0583] At this time, the histogram (2121) of a specific pixel of the lidar data (2100) can be generated based on an electrical signal output from a detecting element corresponding to the specific pixel, and since the above-described details can be applied to this, redundant descriptions will be omitted.

[0584] In addition, at this time, the echo signal (2122) of a specific pixel of the LiDAR data (2100) can be generated based on a histogram of the detecting element corresponding to the specific pixel, and since the above-described details can be applied to this, redundant descriptions will be omitted.

[0585] In addition, at this time, the distance (2123) of a specific pixel of the LiDAR data (2100) can be measured based on an echo signal generated based on a histogram of a detecting element corresponding to the specific pixel, and since the above-described details can be applied to this, redundant descriptions will be omitted.

[0586] In addition, at this time, the reflection intensity (2124) of a specific pixel of the LiDAR data (2100) can be estimated based on an echo signal generated based on a histogram of a detecting element corresponding to the specific pixel, and since the above-described details can be applied to this, redundant descriptions will be omitted.

[0587] In addition, at this time, the representative counting value (2125) of a specific pixel of the lidar data (2100) can be generated based on a histogram of the detecting element corresponding to the specific pixel, and can be estimated based on an echo signal generated based on the histogram.

[0588] For example, the representative counting value (2125) can be obtained as the maximum counting value, average counting value, etc. among the counting values ​​included in the histogram or echo signal.

[0589] Specific examples of LiDAR data (types of LiDAR data)

[0590] Lidar data (2100) can be distinguished based on what the aforementioned pixel values ​​are.

[0591] For example, if the pixel value (2120) is a distance, the LiDAR data may be referred to as a distance map or a depth map.

[0592] In another example, if the pixel value (2120) is the reflection intensity, the LiDAR data may be referred to as a reflection intensity map or an intensity map.

[0593] In another example, if the pixel value (2120) is an echo signal, the LiDAR data may be referred to as an echo signal map.

[0594] In another example, if the pixel value (2120) is a counting value, the LiDAR data may be referred to as a count value map.

[0595] In another example, if the pixel value (2120) is an ambient value, the LiDAR data can be referred to as an ambient value map.

[0596] At this time, the ambient value may refer to a value obtained by the detection array described above operating while the laser output array described above is not operating, and may be obtained by accumulating a counting value obtained by the detection array described above operating during a certain period of time when the laser output array described above is not operating.

[0597] In addition, the LiDAR data (2100) can be distinguished in various ways according to the pixel values ​​described above, in addition to the examples described above.

[0598]

[0599] Below, the frame information obtainable from LiDAR data is explained in more detail.

[0600]

[0601] [Frame Information]

[0602] Definition of frame information

[0603] As described above, the LiDAR device emits a laser around the device and detects the laser reflected from an object to measure the distance to the object located near the LiDAR device. Therefore, the LiDAR device can measure the distance to points where the object is located.

[0604] Accordingly, in this specification, frame information is described as a set of data regarding points where an object whose distance is measured by a lidar device is located, and as information about a specific scene within the field of view of the lidar device obtained through the interpretation of lidar data, etc.

[0605] Types of frame information

[0606] Frame information includes a point cloud, which is a set of location coordinates for points where objects whose distances are measured by the LiDAR device are located, and an enhanced point cloud that includes additional information related to the points along with the location coordinates.

[0607] Of course, point clouds and enhanced point clouds are concepts explained separately for the sake of convenience of explanation, and in this specification, data including additional information related to points along with location coordinates may be represented as a point cloud.

[0608] Below, point clouds and enhanced point clouds are explained in more detail.

[0609]

[0610] Point cloud

[0611] Point data of a point cloud

[0612] FIG. 16 is a drawing for explaining the point cloud disclosed in the present application.

[0613] As described above, the point cloud (2200) is one of the frame information and refers to a set of location coordinates for points where the distance of an object measured from the LiDAR device is located.

[0614] Accordingly, in this specification, data for each point of the point cloud (2200) is described as point data.

[0615] According to this, the point cloud (2200) illustrated in FIG. 16 may be described as including first point data (2211) for a first point and k-th point data (2212) for a k-th point.

[0616]

[0617] Location coordinates of point data

[0618] As illustrated in FIG. 16, each point data of the point cloud (2200) includes location coordinates (2220).

[0619] At this time, the position coordinates (2200) of the above point data are generally expressed using a coordinate system based on the optical origin of the lidar device.

[0620] For example, in FIG. 16, the position coordinates (2220) of each point data of the point cloud (2200) are expressed using an orthogonal coordinate system based on the optical origin of the lidar device.

[0621] Of course, the coordinate system for expressing the position coordinates (2200) of the above point data can be a cylindrical coordinate system, a spherical coordinate system, etc., in addition to the orthogonal coordinate system shown in FIG. 16, and can be various other coordinate systems.

[0622]

[0623] The relationship between point clouds and LiDAR data

[0624] As described above, the point cloud (2200) is one of the frame information, and the frame information is information about a specific scene within the field of view of the lidar device obtained through the interpretation of lidar data, etc.

[0625] Therefore, the point cloud (2200) can be obtained based on LiDAR data.

[0626] More specifically, the point cloud (2200) can be obtained based on pixels in which an object is detected among the pixels included in the lidar data.

[0627] In this case, the pixels where the object is detected may refer to pixels that have a distance value as their pixel value, and having a distance value may mean that the distance to the object has been measured.

[0628] In addition, at this time, the location coordinates (2220) of each point data included in the point cloud (2200) can be obtained based on the pixel coordinates of the pixels included in the LiDAR data and the corresponding distance values.

[0629] More specifically, the point cloud (2200) can be obtained by identifying the pixels in which an object is detected among the pixels included in the LiDAR data, and using the pixel coordinates and corresponding distance values ​​of each of the identified pixels.

[0630] Below, we will explain in more detail how to obtain the position coordinates (2220) of the point data based on the LiDAR data.

[0631] First, as described above, a single laser detector set has a unique orientation direction, and accordingly, the distance value of a pixel corresponding to a single laser detector set may represent the distance to an object located in the unique orientation direction.

[0632] Therefore, the LiDAR device can store information about the direction corresponding to each pixel position coordinate of the LiDAR data.

[0633] For example, the first pixel position coordinates (1,1) and the first direction vector ( _1, _1) This can be matched and stored, and K pixel position coordinates (M,N) and the k-th direction vector ( _M, _N) can be matched and stored.

[0634] Accordingly, the location coordinates (2220) of the point data included in the point cloud (2200) obtain direction information stored by matching each pixel coordinate based on the pixel coordinates of the pixels included in the LiDAR data, and can be obtained based on the distance value corresponding to the obtained direction information.

[0635] For example, the first position coordinates (X_1, Y_1, Z_1) of the first point data (2211) are matched to the first direction vector (1,1) which is the position coordinate of the first pixel corresponding to the first point data and stored. _1, It can be obtained based on _1) and a first distance value (R_1) corresponding to the first pixel.

[0636] In addition, when a point cloud is acquired based on pixels in which an object is detected among the pixels included in the LiDAR data, the number of pixels included in the LiDAR data and the number of point data included in the point cloud may differ from each other.

[0637] In other words, this is because the number of pixels included in the LiDAR data may correspond to the number of detector elements within the detector element array regardless of whether pixel values ​​are acquired or distances are measured, but the number of point data included in the point cloud may be acquired based on the pixels among the pixels included in the LiDAR data for which distance values ​​have been acquired.

[0638] Enhanced point cloud

[0639] Enhanced point cloud and enhanced point data

[0640] FIG. 17 is a drawing for explaining the enhanced point cloud disclosed in the present application.

[0641] As described above, the enhanced point cloud (2300) is one of the frame information and means a set of location coordinates for points where an object whose distance is measured in the lidar device is located, and information related to the points.

[0642] Accordingly, in this specification, data including location coordinates for each point of the enhanced point cloud (2300) and information related to the point matching thereto is to be described as enhanced point data.

[0643] In this case, since the details regarding the location coordinates of the point data described above can be applied to the location coordinates of the enhanced point data, redundant descriptions will be omitted.

[0644] In addition, at this time, information related to the branch is described as a point value (2330).

[0645] Accordingly, each enhanced point data included in the enhanced point cloud (2300) includes location coordinates and a point value (2330) that matches the location coordinates.

[0646] At this time, the point value (2330) included in the enhanced point data may be one as shown in FIG. 17, but is not limited thereto and may be multiple.

[0647] Below, the point value (2330) included in the enhanced point data will be explained in more detail.

[0648] Point value of enhanced point data

[0649] The point value (2330) of the enhanced point data can be obtained based on the lidar data.

[0650] For example, the point value (2330) of the enhanced point data can be used as the pixel value of multiple pixels of the lidar data.

[0651] For a more specific example, the point value (2330) of the enhanced point data may be any one of the histogram, echo signal, distance, reflection intensity, or representative counting value of the corresponding pixel.

[0652] In addition, for example, the point value (2330) of the enhanced point data may be a value obtained by processing the pixel values ​​of multiple pixels of the LiDAR data.

[0653] For a more specific example, the point value (2330) of the enhanced point data may be a value obtained by processing, such as by assigning a weight to any one of the histogram, echo signal, distance, reflection intensity, or representative counting value of the corresponding pixel.

[0654] In another more specific example, the point value (2330) of the enhanced point data may be a value adjusted by taking into account the pixel value of the surrounding pixels, any one of the histogram, echo signal, distance, reflection intensity, or representative counting value of the corresponding pixel.

[0655] Additionally, the point value (2330) of the enhanced point data can be obtained based on the point cloud.

[0656] For example, the point value (2330) of the enhanced point data may be a normal vector value for a virtual plane calculated by considering the location coordinates of the corresponding point data and the location coordinates of the surrounding point data.

[0657] Additionally, for example, the point value (2330) of the enhanced point data may be a value that is adjusted by taking into account the location coordinates of the surrounding point data for the location coordinates of the corresponding point data.

[0658]

[0659] [Operation timing of the LiDAR device for generating LiDAR data]

[0660] Histogram acquisition interval

[0661] It has already been described above that the pixel value of a corresponding pixel is obtained by the operation of a laser-detector pair (multiple sampling cycles).

[0662] In this specification, a section for acquiring a histogram by performing a plurality of sampling cycles for a single laser-detector pair is described as a histogram acquisition section.

[0663] Of course, the above histogram acquisition interval could also be expressed as a distance acquisition interval, a pixel value acquisition interval, etc., but for the sake of clarity in the following explanation, it will be referred to as a histogram acquisition interval.

[0664] At this time, the length of the histogram acquisition interval for all laser-detector pairs may be the same, but is not limited thereto, and the length of the histogram acquisition interval for some laser-detector pairs may differ.

[0665] Necessity of controlling the operation timing of LiDAR devices

[0666] As described above, LiDAR data is a set of pixel values ​​acquired by each laser-detector pair during a predetermined very short period of time, and frame information is information about a specific scene obtained through the interpretation of LiDAR data, etc.

[0667] Therefore, LiDAR data or frame information is acquired by passing through the histogram acquisition section for all laser-detector pairs. Of course, LiDAR data or frame information that defines only a specific part of the entire scene using pixels from some laser-detector pairs may also be acquired; however, for the sake of convenience, the explanation is based on the premise that LiDAR data or frame information is acquired by passing through the histogram acquisition section for all laser-detector pairs.

[0668] At this time, the operation timing of all laser-detector pairs can be related to frame rate, interference, measurable distance, eye safety, etc., and can be a strategic design consideration for the lidar device.

[0669] Various examples of LiDAR device operation timing

[0670] According to the first example of the operation timing of the lidar device, the histogram acquisition intervals for all laser-detector pairs are different from each other.

[0671] This means that the operation for acquiring the histogram for another laser-detector pair is performed after the histogram acquisition period for one laser-detector pair has passed.

[0672] However, in this case, the histogram acquisition intervals for all laser-detector pairs may partially overlap with each other.

[0673] In addition, according to the first example of the operation timing of the lidar device, the arrangement of histogram acquisition intervals for all laser-detector pairs may follow a preset order.

[0674] Of course, the array of histogram acquisition intervals for all laser-detector pairs may also be determined randomly.

[0675] In addition, according to the first example of the operation timing of the lidar device, the arrangement of histogram acquisition intervals for all laser-detector pairs for acquiring lidar data of each frame may be identical to each other.

[0676] For example, the array of histogram acquisition intervals for all laser-detector pairs for acquiring the first lidar data of the first frame may be the same as the array of histogram acquisition intervals for all laser-detector pairs for acquiring the second lidar data of the second frame.

[0677] Of course, the arrangement of histogram acquisition intervals for all laser-detector pairs to acquire LiDAR data for each frame may differ from one another.

[0678] For example, the arrangement of histogram acquisition intervals for all laser-detector pairs for acquiring the first lidar data of the first frame may be different from the arrangement of histogram acquisition intervals for all laser-detector pairs for acquiring the second lidar data of the second frame.

[0679] In addition, according to the second example of the operation timing of the lidar device, the histogram acquisition intervals for laser-detector pairs of some groups among all laser-detector pairs are identical to each other, but differ from the histogram acquisition intervals for laser-detector pairs of other groups.

[0680] This means that after the histogram acquisition interval for some groups of laser-detector pairs has passed, the operation for acquiring histograms for another group of laser-detector pairs is performed.

[0681] However, in this case, the fact that the histogram acquisition intervals for some groups of laser-detector pairs are identical includes not only that some groups of laser-detector pairs operate at physically completely identical timings, but also that they operate at physically slightly different timings but substantially at the same timing.

[0682] In this case, the laser-detector pairs of some groups may vary.

[0683] For example, some groups of laser-detector pairs may include laser-detector pairs arranged in a single row.

[0684] Additionally, for example, some groups of laser-detector pairs may include laser-detector pairs arranged in a single column.

[0685] Additionally, for example, some groups of laser-detector pairs may include laser-detector pairs arranged in at least two rows.

[0686] Additionally, for example, some groups of laser-detector pairs may include laser-detector pairs arranged in at least two columns.

[0687] In addition to the examples described above, some groups of laser-detector pairs can be grouped in various ways.

[0688] However, for the sake of convenience of explanation, the following description assumes that some groups of laser-detector pairs include laser-detector pairs arranged in a single row.

[0689] In addition, according to the second example of the operation timing of the lidar device, the arrangement of histogram acquisition intervals for all groups of laser-detector pairs may follow a preset order.

[0690] Of course, the arrangement of histogram acquisition intervals for all groups of laser-detector pairs may also be determined randomly.

[0691] In addition, according to the second example of the operation timing of the lidar device, the arrangement of histogram acquisition intervals for all groups of laser-detector pairs for acquiring lidar data of each frame may be identical to each other.

[0692] For example, the arrangement of histogram acquisition intervals for all groups of laser-detector pairs for acquiring first lidar data of the first frame may be the same as the arrangement of histogram acquisition intervals for all groups of laser-detector pairs for acquiring second lidar data of the second frame.

[0693] Of course, the arrangement of histogram acquisition intervals for all groups of laser-detector pairs to acquire LiDAR data for each frame may differ from one another.

[0694] For example, the arrangement of histogram acquisition intervals for all groups of laser-detector pairs for acquiring first lidar data of the first frame may be different from the arrangement of histogram acquisition intervals for all groups of laser-detector pairs for acquiring second lidar data of the second frame.

[0695] In addition, according to the third example of the operation timing of the lidar device, the histogram acquisition intervals for all laser-detector pairs are identical to each other.

[0696] This means that the operation for acquiring histograms for all laser-detector pairs is performed simultaneously.

[0697] However, in this case, the operation for acquiring histograms for all laser-detector pairs being performed simultaneously includes not only that all laser-detector pairs operate at physically completely identical timings, but also that they operate at physically slightly different timings but substantially at the same timing.

[0698]

[0699] II. Method for generating a color image using LiDAR data according to one embodiment

[0700] [Difficulties in generating color images using LiDAR data]

[0701] To implement an artificial intelligence model that generates color images using LiDAR data, it is necessary to train the model using a training dataset in which the input data is LiDAR data and the output data is color images.

[0702] However, since LiDAR data is acquired from a LiDAR device while color images are acquired from cameras, generating a training dataset with LiDAR data as input and color images as output requires installing both a LiDAR device and a camera simultaneously to acquire both data and color images, and also necessitates a matching process between the two datasets.

[0703] Therefore, generating a training dataset with LiDAR data as input and color images as output is not simple, and it is also difficult to obtain a large number of training datasets.

[0704] On the other hand, regarding colorization models that generate color images using grayscale images, much prior research and development has been conducted, and many training datasets have been secured.

[0705] Therefore, to generate color images using LiDAR data, one might consider adopting a model that converts black-and-white images into color images; however, in this case, a natural color image may not be generated due to the differences between the black-and-white images and the LiDAR data.

[0706] More specifically, as described above, LiDAR data in which pixel values ​​are reflection intensity can be referred to as a reflection intensity map or intensity map. Since the reflection intensity is a value related to the degree to which a laser output from a LiDAR device is reflected from an object and received by the LiDAR device, the reflection intensity map or intensity map can be viewed as an image generated using the laser output array of the LiDAR device as a point light source.

[0707] On the other hand, since black-and-white and color images can be viewed as images generated using sunlight as a parallel light source, LiDAR data and black-and-white images differ in that they are images generated by different light sources. Due to this difference, when LiDAR data is input into an AI model that converts pre-generated black-and-white images into color images, a natural-looking color image may not be generated.

[0708] Below, a method for generating more natural color images using LiDAR data is explained in more detail.

[0709] FIG. 18 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0710] Referring to FIG. 18, a method (3000) for generating a color image using lidar data according to one embodiment includes acquiring lidar data (S3010), estimating the direction of an external light source using ambient values ​​of a plurality of pixels (S3020), estimating surface information for each of a plurality of pixels using pixel position coordinates and distance values ​​of a plurality of pixels (S3030), generating a geometric intensity value for each of a plurality of pixels based on the direction of the external light source and the surface information for each of a plurality of pixels (S3040), generating a corrected intensity value for each of a plurality of pixels based on the intensity value of each of a plurality of pixels and the geometric intensity value of each of a plurality of pixels (S3050), and inputting the corrected intensity value of each of a plurality of pixels into a colorization model to acquire a color channel value for each of a plurality of pixels (S3060).

[0711]

[0712] [Acquired LiDAR data (S3010)]

[0713] LiDAR data according to one embodiment is composed of a plurality of pixels, and each of the plurality of pixels is composed of a pixel location coordinate and at least one pixel value corresponding to the pixel location coordinate.

[0714] At this time, the above at least one pixel value includes a distance value, a reflection intensity value, and an ambient value.

[0715] In addition, the above reflection intensity value may be described as an intensity value below, and since the aforementioned details may apply to the distance value, reflection intensity value, and ambient value, redundant descriptions will be omitted.

[0716] In addition, the above intensity value can be expressed as I_raw below, and the intensity value for the pixel located at (M,N) can be expressed as I_raw(M,N).

[0717] In addition, since the aforementioned details can be applied to the above LiDAR data, redundant descriptions will be omitted.

[0718]

[0719] [Estimate the direction of an external light source using ambient values ​​of multiple pixels (S3020)]

[0720] Estimating the direction of an external light source using ambient values ​​of a plurality of pixels according to one embodiment (S3020) can be performed by various algorithms.

[0721] For example, estimating the direction of an external light source using ambient values ​​of a plurality of pixels according to one embodiment (S3020) can be performed using a pre-trained artificial intelligence model.

[0722] At this time, the above-mentioned pre-trained artificial intelligence model can be trained using a training data set that takes LiDAR data composed of multiple pixels as input data and a light source direction vector as output data.

[0723] In addition, at this time, the above-mentioned pre-trained artificial intelligence model can be trained using a training data set that takes an image composed of multiple pixels as input data and a light source direction vector as output data.

[0724] Additionally, for example, estimating the direction of an external light source using ambient values ​​of a plurality of pixels according to one embodiment (S3020) can be performed by an algorithm that obtains a light source direction vector based on at least one object and a shadow associated with at least one object.

[0725] Below, an algorithm according to an example for estimating the direction of the external light source is explained in more detail using FIG. 19.

[0726] FIG. 19 is a drawing for explaining a method for estimating the direction of an external light source according to one embodiment.

[0727] Referring to FIG. 19, a method (3100) for estimating the direction of an external light source according to one embodiment includes identifying at least one object using ambient values ​​of a plurality of pixels (S3110), identifying at least one shadow associated with at least one object using ambient values ​​of a plurality of pixels (S3120), and obtaining a light source direction vector for an external light source based on at least one object and at least one shadow associated with at least one object (S3130).

[0728] Identifying at least one object using the ambient values ​​of multiple pixels (S3110)

[0729] Identifying at least one object using ambient values ​​of a plurality of pixels according to one embodiment (S3110) may include applying a plurality of pixels, in which the ambient values ​​are pixel values, to a pre-trained artificial intelligence model to classify or cluster at least one object.

[0730] At this time, the aforementioned pre-trained artificial intelligence model may be applied to various artificial intelligence models for identifying at least one object from an image.

[0731] For example, the above-mentioned pre-trained artificial intelligence model may be an artificial intelligence model trained based on a training data set that includes images for which at least one object has been labeled.

[0732] At this time, the image in which labeling has been performed for at least one object may be lidar data in which labeling has been performed for at least one object, and in this case, the lidar data may be composed of multiple pixels in which ambient values ​​are pixel values.

[0733] Identifying at least one shadow associated with at least one object using the ambient values ​​of multiple pixels (S3120)

[0734] Identifying at least one shadow associated with at least one object using the ambient values ​​of a plurality of pixels according to one embodiment (S3120) may include applying a plurality of pixels, in which the ambient values ​​are pixel values, to a pre-trained artificial intelligence model to classify or cluster at least one shadow.

[0735] At this time, the aforementioned pre-trained artificial intelligence model may be applied to various artificial intelligence models for identifying at least one shadow from an image.

[0736] For example, the above-mentioned pre-trained artificial intelligence model may be an artificial intelligence model trained based on a training dataset that includes images for which at least one shadow has been labeled.

[0737] At this time, the image in which at least one shadow has been labeled can be lidar data in which at least one object has been labeled, and in this case, the lidar data can be composed of multiple pixels in which the ambient value is the pixel value.

[0738] In addition, at this time, a shadow associated with at least one object may mean a shadow extended from the object, and may mean a shadow determined to be associated with at least one object by a pre-stored algorithm.

[0739] Obtain a light source direction vector for an external light source based on at least one object and at least one shadow associated with at least one object (S3130)

[0740] When an external light source is located in a specific direction, the position, direction, and size of the shadow cast by an object may vary depending on the direction of the external light source.

[0741] Therefore, the location of an external light source can be estimated by considering at least one object and the position, direction, and size of the shadow associated with at least one object.

[0742] At this time, the position of the external light source can be obtained as a light source direction vector, and the light source direction vector can be expressed as a three-dimensional vector.

[0743] Below, the light source direction vector for an external light source can be represented as L_light.

[0744]

[0745] [Surface information for each of the multiple pixels is estimated using the pixel location coordinates and distance values ​​of the multiple pixels (S3030)]

[0746] When a pixel and pixels adjacent to that pixel include distance values ​​as pixel values, a 3D position corresponding to each pixel can be generated.

[0747] At this time, the 3D positions corresponding to one pixel and the pixels adjacent to that one pixel correspond to the surface positions of an object located around the LiDAR device.

[0748] Therefore, by using the 3D positions corresponding to a single pixel and the pixels adjacent to that single pixel, geometric position information of the object's surface can be estimated.

[0749] That is, the direction in which the surface of an object is tilted can be estimated, and this can be determined by the normal vector of a virtual plane formed by a single pixel and the 3D positions corresponding to the pixels adjacent to that single pixel.

[0750] At this time, estimating surface information for each of the plurality of pixels using pixel position coordinates and distance values ​​of the plurality of pixels according to one embodiment (S3030) can be performed using various algorithms.

[0751] For example, estimating surface information for each of the plurality of pixels using pixel location coordinates and distance values ​​of the plurality of pixels according to one embodiment (S3030) can be implemented using various algorithms included in a pre-stored PCL (Point Cloud Library).

[0752] Hereinafter, an algorithm according to an example for estimating surface information for each of a plurality of pixels will be explained in more detail using FIG. 20.

[0753] FIG. 20 is a diagram illustrating a method for estimating surface information for each of a plurality of pixels according to one embodiment.

[0754] Referring to FIG. 20, a method (3200) for estimating surface information for each of a plurality of pixels according to one embodiment may include obtaining point data for each of a plurality of pixels (S3210), selecting a pixel group for each of a plurality of pixels (S3220), and estimating a plane for each of a plurality of pixels based on the point data of the selected pixel group for each of a plurality of pixels and obtaining a normal vector of the estimated plane (S3230).

[0755] Acquire point data for each of the multiple pixels (S3210)

[0756] Regarding the acquisition of point data for each of a plurality of pixels according to one embodiment (S3210), the above-described details may apply, so redundant descriptions are omitted, and regarding the point data, the above-described details may also apply, so redundant descriptions are omitted.

[0757] Select a pixel group for each of the multiple pixels (S3220)

[0758] In selecting a pixel group for each of a plurality of pixels according to one embodiment (S3220), the pixel group includes each of the plurality of pixels and at least one pixel adjacent to each of the plurality of pixels.

[0759] For example, a first pixel group for a (M,N) pixel may include a pixel located at (M,N), a pixel located at (M-1,N), a pixel located at (M+1,N), a pixel located at (M,N-1), a pixel located at (M,N+1), a pixel located at (M-1,N-1), a pixel located at (M-1,N+1), a pixel located at (M+1,N-1), and a pixel located at (M+1,N+1).

[0760] For each of the multiple pixels, a plane for each of the multiple pixels is estimated based on the point data of the selected pixel group, and a normal vector of the estimated plane is obtained (S3230)

[0761] Estimating a plane for each of the plurality of pixels based on point data of a selected pixel group for each of the plurality of pixels according to one embodiment and obtaining a normal vector of the estimated plane (S3230) can be implemented using various algorithms included in a pre-stored PCL (Point Cloud Library).

[0762] For ease of understanding regarding the method (3200) for estimating surface information for each of a plurality of pixels according to the above-described embodiment, the method (3200) for estimating surface information for each of a plurality of pixels according to the above-described embodiment is described below by way of example.

[0763] According to a method (3200) for estimating surface information for each of a plurality of pixels according to one embodiment,

[0764] First to N point data for each of the first to N pixels are obtained.

[0765] And, from the first pixel group for the first pixel to the Nth pixel group for the Nth pixel, are selected.

[0766] After that, for the first pixel, the first plane formed by the first pixel group is estimated based on the point data of the first pixel group, and the first normal vector of the estimated first plane is obtained.

[0767] Then, the first normal vector is stored as the pixel value for the first pixel.

[0768] In addition, for the second pixel, the second plane formed by the second pixel group is estimated based on the point data of the second pixel group, and the second normal vector of the estimated second plane is obtained.

[0769] And, the second normal vector is stored as the pixel value for the second pixel.

[0770] In this way, for the Nth pixel, the Nth plane formed by the Nth pixel group is estimated based on the point data of the Nth pixel group, and the Nth normal vector of the estimated Nth plane is obtained.

[0771] And the Nth normal vector is stored as the pixel value for the Nth pixel.

[0772] Ultimately, in accordance with the above-described details, the normal vector for each pixel from the first pixel to the Nth pixel is obtained and stored.

[0773] Of course, the order in which normal vectors are obtained from the 1st to the Nth pixel is explained sequentially from the 1st to the Nth pixel merely for the sake of convenience of understanding and may be changed.

[0774] In this case, the normal vector can be expressed as a 3-dimensional vector.

[0775] Below, the normal vector of a pixel located at (M,N) can be expressed as N_(M,N).

[0776]

[0777] [Generate geometric intensity values ​​for each of the multiple pixels based on the direction of the external light source and surface information for each of the multiple pixels (S3040)]

[0778] The above geometric intensity value is generated such that the maximum value is equal to the maximum value of the above intensity value.

[0779] This may be intended to match the size range between the two values ​​when correcting the intensity value using the geometric intensity value.

[0780] For example, if the maximum value of the intensity value is 255, the geometric intensity value can also be generated so that the maximum value is 255.

[0781] For a more specific example, the geometric intensity value can be generated by multiplying the maximum value of the intensity value by the inner product of the light source vector and the normal vector.

[0782] At this time, when the geometric intensity value of the pixel located at (M,N) is expressed as I_geo(M,N), the geometric intensity value may satisfy the following relationship.

[0783] [Relationship]

[0784] I_geo(M,N) = L_light N_(M,N) * Maximum value of intensity

[0785]

[0786] [Generates a corrected intensity value for each of the multiple pixels based on the intensity value of each of the multiple pixels and the geometric intensity value of each of the multiple pixels (S3050)]

[0787] Generating a corrected intensity value according to one embodiment may include generating a corrected intensity value by assigning weights to the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels.

[0788] For example, generating a corrected intensity value according to one embodiment may satisfy the following relationship, wherein the corrected intensity value of a pixel located at (M,N) is expressed as I_amd(M,N).

[0789] [Relationship]

[0790] I_amd(M,N) = a * I_raw(M,N) + (1-a) * I_geo(M,N)

[0791]

[0792] [Input the corrected intensity values ​​of each of the multiple pixels into a colorization model to obtain the color channel values ​​of each of the multiple pixels (S3060)]

[0793] A colorization model according to one embodiment may be implemented using a machine learning method. For example, a colorization model according to one embodiment may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, reinforcement learning, etc.

[0794] In addition, a colorization model according to one embodiment may be implemented with at least one artificial neural network (ANN). For example, a colorization model according to one embodiment may include at least one artificial neural network layer among various artificial neural network layers such as a feedforward neural network, a radial basis function network or a Kohonen self-organizing network, a convolutional neural network (CNN), a recurrent neural network (RNN), a Long Short Term Memory Network (LSTM), or Gated Recurrent Units (GRUs), but is not limited thereto.

[0795] In addition, at least one artificial neural network layer included in the colorization model according to one embodiment may use the same or different activation function.

[0796] At this time, the activation function may include, but is not limited to, a sigmoid function, a hyperbolic tangent function, a ReLU function (Rectified Linear unit function), a leaky ReLU function, an ELU function (Exponential Linear unit function), a softmax function, etc., and may include various activation functions (including custom activation functions) for outputting a result or transmitting it to another artificial neural network layer.

[0797] In addition, a colorization model according to one embodiment can be trained using at least one loss function.

[0798] At this time, the above at least one loss function may include, but is not limited to, MSE (Mean Squared Error), RMSE (Root Mean Squared Error), Binary Crossentropy, Categorical Crossentropy, Sparse Categorical Crossentropy, etc., and may include various functions (including custom loss functions) for calculating the difference between the predicted result value and the actual result value.

[0799] In addition, at least one optimizer may be used for learning the colorization model according to one embodiment.

[0800] At this time, the at least one optimizer may include, but is not limited to, Gradient descent, Batch Gradient Descent, Stochastic Gradient Descent, Mini-batch Gradient Descent, Momentum, AdaGrad, RMSProp, AdaDelta, Adam, NAG, NAdam, RAdam, AdamW, etc.

[0801] In addition, a colorization model according to one embodiment can be trained by various training data sets.

[0802] For example, a colorization model may be a model trained using a training dataset that uses black and white images as input data and color images as output data.

[0803] In addition, for example, the colorization model may be a model generated by further training a model that has been trained using a training dataset in which a black-and-white image is input data and a color image is output data, using at least one LiDAR data as input data and a color image as output data.

[0804] In addition, inputting the corrected intensity values ​​of each of the multiple pixels into the colorization model includes inputting LiDAR data composed of multiple pixels, where the pixel values ​​are the corrected intensity values, into the colorization model.

[0805] In addition, at this time, obtaining the color channel value of each of the multiple pixels may include obtaining a color image composed of multiple pixels in which the color channel value is the pixel value.

[0806] At this time, the color channel value may be a color channel value according to the RGB color plane, a color channel value according to the YCrCb color plane, and a color channel value according to the HSV color plane, but is not limited thereto, and may be a color channel value according to various color planes or color spaces.

[0807] [Principles of Problem Solving]

[0808] According to the present invention, in order to convert LiDAR data composed of a plurality of pixels whose pixel values ​​are intensity values ​​into a color image, an artificial intelligence model that converts a black-and-white image into a color image is employed, and by correcting the LiDAR data so that the shooting environment of the black-and-white image is simulated through the estimated position of a parallel light source and the estimated geometric information of an object, the difference between the black-and-white image and the LiDAR data is reduced, thereby solving the problem that the converted color image may appear awkward.

[0809]

[0810] III. Method for generating a color image using LiDAR data according to another embodiment

[0811] [Difficulties in generating color images using LiDAR data]

[0812] As described above, using LiDAR data composed of multiple pixels with pixel values ​​of intensity values ​​to correct LiDAR data composed of multiple pixels with pixel values ​​of ambient values ​​reduces the difference between the grayscale image and the LiDAR data, thereby making the color image conversion natural.

[0813] However, the above ambient value cannot be generated in the absence of an external light source; consequently, there is a problem in that the aforementioned algorithm is difficult to apply in indoor or nighttime environments where there is no external light source.

[0814] Below, a method for solving the aforementioned problem is explained in more detail.

[0815]

[0816] FIG. 21 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0817] Referring to FIG. 21, a method (3300) for generating a color image using lidar data according to one embodiment includes acquiring lidar data (S3310), selecting a colorization algorithm based on a preset standard (S3320), acquiring color channel values ​​of each of a plurality of pixels using a first colorization algorithm (S3330), and acquiring color channel values ​​of each of a plurality of pixels using a second colorization algorithm (S3340).

[0818] At this time, regarding the acquisition of the above-mentioned LiDAR data (S3310), since the above-mentioned details may apply, redundant descriptions will be omitted.

[0819]

[0820] [Select a colorization algorithm based on pre-set criteria (S3320)]

[0821] At this time, the above-mentioned preset standard may be a standard using ambient values ​​among pixel values.

[0822] For example, the above-mentioned preset criterion may be a criterion using the average, maximum value, median value, or total sum of the ambient values ​​of all pixels in LiDAR data composed of multiple pixels where the pixel value is an ambient value.

[0823] Additionally, selecting a colorization algorithm based on the above-mentioned preset criteria (S3320) may include determining whether the first preset criteria are satisfied and whether the second preset criteria are satisfied, and selecting a colorization algorithm based on the result of the determination.

[0824] At this time, the first preset criterion may be whether the average, maximum value, median value, or total sum of the ambient values ​​of all pixels exceeds the first criterion value.

[0825] In addition, at this time, the second preset criterion may be whether the average, maximum value, median value, or total sum of the ambient values ​​of all pixels is less than or equal to the second criterion value.

[0826] At this time, the first preset criterion may be a criterion for determining whether the ambient value of the lidar data is sufficient to determine the location of an external light source, and the second preset criterion may be a criterion for determining whether the ambient value of the lidar data is insufficient to determine the location of an external light source.

[0827] In addition, for the convenience of explanation, the first preset criterion and the second preset criterion were used in the explanation, but the colorization algorithm may also use a single preset criterion value for selection.

[0828] For example, whether the above-described first preset criterion is satisfied can be determined by whether the average, maximum value, median value, or total sum of the ambient values ​​of the lidar data exceeds the preset criterion value, and whether the above-described second preset criterion is satisfied can be determined by whether the average, maximum value, median value, or total sum of the ambient values ​​of the lidar data is less than or equal to the preset criterion value.

[0829] In addition, at this time, the above-mentioned preset standard may be set based on time information, measurement values ​​from an external light intensity sensor, etc.

[0830]

[0831] [Obtain the color channel values ​​of each of the multiple pixels using the first colorization algorithm (S3330)]

[0832] Based on the above-described judgment result, if the first preset criterion is satisfied, obtaining the color channel value of each of the multiple pixels using the first colorization algorithm (S3330) can be performed.

[0833] At this time, the first colorization algorithm may be the color image acquisition method described through FIGS. 18 to 20 of the present specification. Accordingly, redundant descriptions are omitted.

[0834]

[0835] [Obtain the color channel values ​​of each of the multiple pixels using the second colorization algorithm (S3340)]

[0836] Based on the above-described judgment result, if the second preset criterion is satisfied, obtaining the color channel value of each of the multiple pixels using the second colorization algorithm (S3340) may be performed.

[0837] This will be explained in more detail below with reference to Fig. 22.

[0838]

[0839] [Second Colorization Algorithm]

[0840] FIG. 22 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0841] Referring to FIG. 22, a method (3400) for generating a color image using LiDAR data according to one embodiment includes obtaining a pixel direction vector corresponding to each of a plurality of pixels (S3410), estimating surface information for each of a plurality of pixels using pixel position coordinates and distance values ​​of a plurality of pixels (S3420), generating a geometric intensity value for each of a plurality of pixels based on the pixel direction vector corresponding to each of a plurality of pixels and the surface information for each of a plurality of pixels (S3430), generating a corrected intensity value for each of a plurality of pixels based on the intensity value of each of a plurality of pixels and the geometric intensity value of each of a plurality of pixels (S3440), and inputting the corrected intensity value of each of a plurality of pixels into a colorization model to obtain a color channel value for each of a plurality of pixels (S3450).

[0842]

[0843] Obtain a pixel direction vector corresponding to each of the multiple pixels (S3410)

[0844] As described above, the direction corresponding to each of the plurality of pixels is determined by the configuration of the laser output element array, the detecting element array, the transmitting optic assembly, and the receiving optic assembly of the lidar device.

[0845] Therefore, the pixel direction vector corresponding to each of the multiple pixels can be stored in advance.

[0846] At this time, the pixel direction vector corresponding to each of the multiple pixels can be represented as L_laser, and the pixel direction vector of the pixel located at (M,N) can be represented as L_laser(M,N).

[0847]

[0848] Surface information for each of the multiple pixels is estimated using the pixel location coordinates and distance values ​​of the multiple pixels (S3420)

[0849] Regarding the estimation of surface information for each of the plurality of pixels using the pixel position coordinates and distance values ​​of the plurality of pixels (S3420), the above-described details may be applied, so redundant descriptions are omitted.

[0850]

[0851] A geometric intensity value for each of the multiple pixels is generated based on a pixel direction vector corresponding to each of the multiple pixels and surface information for each of the multiple pixels (S3430)

[0852] The above geometric intensity value is generated such that the maximum value is equal to the maximum value of the above intensity value.

[0853] This may be intended to match the size range between the two values ​​when correcting the intensity value using the geometric intensity value.

[0854] For example, if the maximum value of the intensity value is 255, the geometric intensity value can also be generated so that the maximum value is 255.

[0855] For a more specific example, the geometric intensity value can be generated by multiplying the maximum value of the intensity value by the inner product of the light source vector and the normal vector.

[0856] At this time, when the geometric intensity value of the pixel located at (M,N) is expressed as I_geo(M,N), the geometric intensity value may satisfy the following relationship.

[0857] [Relationship]

[0858] I_geo(M,N) = L_laser(M,N) N_(M,N) * Maximum value of intensity

[0859]

[0860] Generates a corrected intensity value for each of the multiple pixels based on the intensity value of each of the multiple pixels and the geometric intensity value of each of the multiple pixels (S3440)

[0861] Generating a corrected intensity value according to one embodiment may include generating a corrected intensity value by assigning weights to the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels.

[0862] For example, generating a corrected intensity value according to one embodiment may satisfy the following relationship, wherein the corrected intensity value of a pixel located at (M,N) is expressed as I_amd(M,N).

[0863] [Relationship]

[0864] I_amd(M,N) = a * I_raw(M,N) + (1-a) * I_geo(M,N)

[0865]

[0866] The corrected intensity values ​​of each of the multiple pixels are input into a colorization model to obtain the color channel values ​​of each of the multiple pixels (S3450)

[0867] The process of obtaining the color channel values ​​of each of the plurality of pixels by inputting the corrected intensity values ​​of each of the plurality of pixels into a colorization model (S3450) can be applied as described above, so redundant descriptions are omitted.

[0868]

[0869] [Principles of Problem Solving]

[0870] As described in FIGS. 21 and 22, a color image is generated by the first colorization algorithm when the first preset criterion is satisfied, and a color image is generated by the second colorization algorithm when the second preset criterion is satisfied.

[0871] At this time, the first preset criterion is a criterion for determining whether the ambient value of the lidar data is sufficient to determine the location of an external light source, and the second preset criterion is a criterion for determining whether the ambient value of the lidar data is insufficient to determine the location of an external light source.

[0872] That is, in situations where the location of an external light source can be determined, a color image is generated by the first colorization algorithm, and in situations where the location of an external light source cannot be determined, a color image is generated by the second colorization algorithm.

[0873] In this case, for both the first and second colorization algorithms, the LiDAR data input to the colorization model is LiDAR data composed of multiple pixels whose pixel values ​​are intensity values ​​corrected.

[0874] In addition, at this time, the corrected intensity value satisfies the following relationship in both the first colorization algorithm and the second colorization algorithm.

[0875] [Relationship]

[0876] I_amd(M,N) = a * I_raw(M,N) + (1-a) * I_geo(M,N)

[0877]

[0878] However, in the case of the first colorization algorithm and the second colorization algorithm, the geometric intensity values ​​each satisfy the following relationship separately.

[0879] [Relationship] - In the case of the first colorization algorithm

[0880] I_geo(M,N) = L_light N_(M,N) * Maximum value of intensity

[0881] [Relationship] - In the case of the second colorization algorithm

[0882] I_geo(M,N) = L_laser(M,N) N_(M,N) * Maximum value of intensity

[0883]

[0884] That is, referring to the two relationship equations described above, the first colorization algorithm derives a geometric intensity value using the direction of the estimated external light source, whereas the second colorization algorithm derives a geometric intensity value using the pixel direction stored in advance.

[0885] Therefore, the second colorization algorithm becomes an algorithm that can operate even in situations where the position of an external light source cannot be estimated.

[0886] In addition, in order to acquire an image through a camera in a situation where there is no external light source, light output from a point light source such as a flash must be used. Since the pixel direction used in the second colorization algorithm corresponds to the direction of the laser output from the laser output array of the LiDAR device, the second colorization algorithm further simulates the shooting environment for acquiring a camera image in a situation where there is insufficient external light source.

[0887] Accordingly, according to the method described in FIGS. 21 and 22, the LiDAR data simulates the camera's shooting environment in both cases where the external light source is sufficient and where the external light source is insufficient, thereby enabling the generation of more natural color images.

[0888]

[0889] IV. Method for generating a color image using LiDAR data according to another embodiment

[0890] [Overview]

[0891] Among the pixel values ​​of the LiDAR data, the ambient value is a value acquired when the laser output element array is not operating, and is a value generated by an external parallel light source.

[0892] Therefore, LiDAR data composed of multiple pixels whose pixel values ​​are ambient values ​​is a value generated in an environment similar to the camera's shooting environment.

[0893] Accordingly, by adjusting the corrected intensity values ​​generated in the above-described embodiments using ambient values, the shooting environment of the camera can be simulated more accurately.

[0894] Below, we will explain in more detail how to adjust the corrected intensity value using the ambient value and how to generate a color image using it.

[0895]

[0896] FIG. 23 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0897] Referring to FIG. 23, a method (3500) for generating a color image using LiDAR data according to one embodiment comprises acquiring LiDAR data (S3510), estimating the direction of an external light source using ambient values ​​of a plurality of pixels (S3520), estimating surface information for each of a plurality of pixels using pixel position coordinates and distance values ​​of a plurality of pixels (S3530), generating a geometric intensity value for each of a plurality of pixels based on the direction of the external light source and the surface information for each of a plurality of pixels (S3540), generating a corrected intensity value for each of a plurality of pixels based on the intensity value of each of a plurality of pixels and the geometric intensity value of each of a plurality of pixels (S3550), adjusting the corrected intensity value of each of a plurality of pixels based on the ambient value of each of a plurality of pixels to generate a weighted intensity value for each of a plurality of pixels (S3560), and inputting the weighted intensity value of each of a plurality of pixels into a colorization model to the color channel of each of a plurality of pixels Includes obtaining a value (S3570).

[0898] At this time, regarding acquiring the above-mentioned LiDAR data (S3510), estimating the direction of an external light source using the ambient values ​​of multiple pixels (S3520), estimating surface information for each of the multiple pixels using the pixel position coordinates and distance values ​​of the multiple pixels (S3530), generating a geometric intensity value for each of the multiple pixels based on the direction of the external light source and the surface information for each of the multiple pixels (S3540), and generating a corrected intensity value for each of the multiple pixels based on the intensity value of each of the multiple pixels and the geometric intensity value of each of the multiple pixels (S3550), the above-mentioned details may be applied, so redundant descriptions are omitted.

[0899]

[0900] A weighted intensity value is generated for each of the multiple pixels by adjusting the corrected intensity value of each of the multiple pixels based on the ambient value of each of the multiple pixels (S3560)

[0901] Generating a weighted intensity value according to one embodiment may include generating a weighted intensity value by assigning weights to the ambient value of each of the plurality of pixels and the corrected intensity value of each of the plurality of pixels.

[0902] For example, generating a weighted intensity value according to one embodiment may satisfy the following relationship, wherein the ambient value of a pixel located at (M,N) is represented as A_(M,N) and the weighted intensity value is represented as I_weight(M,N).

[0903] [Relationship]

[0904] I_weight(M,N) = b * I_amd(M,N) + (1-b) * A_(M,N)

[0905]

[0906] The weighted intensity values ​​of each of the multiple pixels are input into a colorization model to obtain the color channel values ​​of each of the multiple pixels (S3570)

[0907] Regarding the acquisition of color channel values ​​for each of the plurality of pixels by inputting the weighted intensity values ​​of each of the plurality of pixels into a colorization model (S3570), the details regarding the acquisition of color channel values ​​for each of the plurality of pixels by inputting the corrected intensity values ​​of each of the plurality of pixels into a colorization model (S3060), as described above through FIG. 18, may be applied, and since the input values ​​have merely been changed to weighted intensity values, redundant descriptions will be omitted.

[0908]

[0909] V. Variations of the method for generating color images using LiDAR data according to various embodiments

[0910] [Reasons why variations of the embodiments are possible]

[0911] The embodiments described above through FIGS. 18 to 23 explain correcting or adjusting the pixel values ​​of LiDAR data before they are input into a colorization model.

[0912] However, estimating the direction of external light sources and geometric information to simulate the camera's shooting environment enables the generation of more natural color images, even when used to correct or adjust the output image of a colorization model.

[0913] This will be explained in more detail below.

[0914]

[0915] FIG. 24 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0916] Referring to FIG. 24, a method (3600) for generating a color image using lidar data according to one embodiment includes acquiring lidar data (S3610), inputting intensity values ​​of a plurality of pixels into a colorization model to acquire color channel values ​​of each of the plurality of pixels (S3620), estimating the direction of an external light source using ambient values ​​of a plurality of pixels (S3630), estimating surface information for each of the plurality of pixels using pixel position coordinates and distance values ​​of a plurality of pixels (S3640), generating geometric intensity values ​​for each of the plurality of pixels based on the direction of the external light source and surface information for each of the plurality of pixels (S3650), and correcting the color channel values ​​of each of the plurality of pixels based on the geometric intensity values ​​of the plurality of pixels to generate corrected color channel values ​​for each of the plurality of pixels (S3660).

[0917]

[0918] At this time, regarding acquiring the above-mentioned LiDAR data (S3610), estimating the direction of an external light source using the ambient values ​​of multiple pixels (S3630), estimating surface information for each of the multiple pixels using the pixel position coordinates and distance values ​​of the multiple pixels (S3640), and generating geometric intensity values ​​for each of the multiple pixels based on the direction of the external light source and the surface information for each of the multiple pixels (S3650), the above-mentioned details may be applied, so redundant descriptions will be omitted.

[0919] In addition, regarding the acquisition of color channel values ​​of each of the multiple pixels by inputting the intensity values ​​of the multiple pixels into a colorization model (S3620), the details regarding the acquisition of color channel values ​​of each of the multiple pixels by inputting the corrected intensity values ​​of each of the multiple pixels into a colorization model (S3060), as described above through FIG. 18, may be applied, and since the input values ​​have merely been changed to intensity values, redundant descriptions will be omitted.

[0920]

[0921] Based on the geometric intensity values ​​of multiple pixels, the color channel values ​​of each of the multiple pixels are corrected to generate the corrected color channel values ​​of each of the multiple pixels (S3660)

[0922] Generating a corrected color channel value according to one embodiment may include generating a corrected color channel value by assigning weights to the color channel value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels.

[0923] For example, generating a corrected color channel value according to one embodiment may satisfy the following relationship, wherein the geometric intensity value of the pixel located at (M,N) is expressed as I_geo(M,N), the color channel value of the pixel located at (M,N) can be expressed as R_(M,N), G_(M,N), B(M,N), and the corrected color channel value can be expressed as R_amd(M,N), G_amd(M,N), B_amd(M,N).

[0924] [Relationship]

[0925] R_amd(M,N) = c * R_(M,N) + (1-c) * I_geo(M,N)

[0926] G_amd(M,N) = c * G_(M,N) + (1-c) * I_geo(M,N)

[0927] B_amd(M,N) = c * B(M,N) + (1-c) * I_geo(M,N)

[0928] At this time, generating the aforementioned corrected color channel value can be understood as generating a corrected color image by using alpha blending of LiDAR data composed of multiple pixels whose pixel values ​​are geometric intensity values ​​and a color image composed of multiple pixels whose pixel values ​​are color channel values, and the alpha blending may be a technique for synthesizing the two images by applying weights to them.

[0929] In this case, alpha blending can be represented by the following relationship, which can be understood as the aforementioned relationship for each of the R, G, and B color channels.

[0930] Corrected color image = c * color image + (1-c) * LiDAR data composed of multiple pixels where the pixel value is the geometric intensity value

[0931]

[0932] [Another modified embodiment]

[0933] FIG. 25 is a drawing for explaining a method of generating a color image using LiDAR data according to one embodiment.

[0934] Referring to FIG. 25, a method (3700) for generating a color image using LiDAR data according to one embodiment comprises acquiring LiDAR data (S7010), estimating the direction of an external light source using ambient values ​​of a plurality of pixels (S3720), estimating surface information for each of a plurality of pixels using pixel position coordinates and distance values ​​of a plurality of pixels (S3730), generating a geometric intensity value for each of a plurality of pixels based on the direction of the external light source and the surface information for each of a plurality of pixels (S3740), generating a corrected intensity value for each of a plurality of pixels based on the intensity value of each of a plurality of pixels and the geometric intensity value of each of a plurality of pixels (S3750), inputting the corrected intensity value of each of a plurality of pixels into a colorization model to acquire a color channel value for each of a plurality of pixels (S3760), and correcting the color channel value of each of a plurality of pixels based on the ambient values ​​of a plurality of pixels to generate a corrected color channel value for each of a plurality of pixels (S3770). Includes.

[0935] At this time, regarding the acquisition of the above LiDAR data (S7010), the estimation of the direction of an external light source using the ambient values ​​of multiple pixels (S3720), the estimation of surface information for each of the multiple pixels using the pixel position coordinates and distance values ​​of the multiple pixels (S3730), the generation of a geometric intensity value for each of the multiple pixels based on the direction of the external light source and the surface information for each of the multiple pixels (S3740), the generation of a corrected intensity value for each of the multiple pixels based on the intensity value of each of the multiple pixels and the geometric intensity value of each of the multiple pixels (S3750), and the acquisition of a color channel value for each of the multiple pixels by inputting the corrected intensity value of each of the multiple pixels into a colorization model (S3760), the above-described details may be applied, so redundant descriptions will be omitted.

[0936]

[0937] Based on the ambient values ​​of multiple pixels, the color channel values ​​of each of the multiple pixels are corrected to generate the corrected color channel values ​​of each of the multiple pixels (S3770)

[0938] Generating a corrected color channel value according to one embodiment may include generating a corrected color channel value by assigning weights to the color channel value of each of the plurality of pixels and the ambient value of each of the plurality of pixels.

[0939] For example, generating a corrected color channel value according to one embodiment may satisfy the following relationship, wherein the ambient value of a pixel located at (M,N) is expressed as A_(M,N), the color channel value of a pixel located at (M,N) can be expressed as R_(M,N), G_(M,N), B(M,N), and the corrected color channel value can be expressed as R_amd(M,N), G_amd(M,N), B_amd(M,N).

[0940] [Relationship]

[0941] R_amd(M,N) = d * R_(M,N) + (1-d) * A_(M,N)

[0942] G_amd(M,N) = d * G_(M,N) + (1-d) * A_(M,N)

[0943] B_amd(M,N) = d * B(M,N) + (1-d) * A_(M,N)

[0944] At this time, generating the aforementioned corrected color channel value can be understood as generating a corrected color image by using alpha blending of LiDAR data composed of multiple pixels whose pixel values ​​are ambient values ​​and a color image composed of multiple pixels whose pixel values ​​are color channel values, and the alpha blending may be a technique for synthesizing the two images by applying weights to them.

[0945] In this case, alpha blending can be represented by the following relationship, which can be understood as the aforementioned relationship for each of the R, G, and B color channels.

[0946] Corrected color image = d * color image + (1-d) * LiDAR data composed of multiple pixels where the pixel value is the ambient value

[0947] [Result image according to another modified embodiment above]

[0948] FIG. 26 is a drawing for illustrating result images according to another modified embodiment.

[0949] FIG. 26 illustrates a color image (3810) acquired through a camera, lidar data (3820) composed of multiple pixels whose pixel value is an intensity value, lidar data (3830) composed of multiple pixels whose pixel value is the aforementioned corrected intensity value, a color image (3840) generated by inputting the lidar data composed of multiple pixels whose pixel value is the corrected intensity value into a colorization model, and a color image (3850) corrected using lidar data composed of multiple pixels whose pixel value is an ambient value.

[0950] At this time, the LiDAR data (3830) composed of a plurality of pixels in which the pixel value is the corrected intensity value described above is generated by generating the corrected intensity value of each of the plurality of pixels based on the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels (S3750).

[0951] In addition, at this time, the color image (3840) generated by inputting LiDAR data composed of multiple pixels, the pixel values ​​being corrected intensity values, into a colorization model is generated by inputting the corrected intensity values ​​of each of the multiple pixels into a colorization model to obtain the color channel values ​​of each of the multiple pixels (S3760).

[0952] In addition, at this time, a color image (3850) corrected using LiDAR data composed of multiple pixels whose pixel values ​​are ambient values ​​is generated by correcting the color channel values ​​of each of the multiple pixels based on the ambient values ​​of the multiple pixels, thereby generating the corrected color channel values ​​of each of the multiple pixels (S3770).

[0953] Referring again to FIG. 26, the LiDAR data (3830) composed of multiple pixels whose pixel values ​​are the corrected intensity values ​​described above is similar to the brightness distribution of the color image (3810) obtained through the camera than the LiDAR data (3820) composed of multiple pixels whose pixel values ​​are intensity values.

[0954] For example, referring to the first region (3821) of the color image (3810), the second region (3822) of the lidar data (3820) composed of multiple pixels in which the pixel value is an intensity value, and the third region (3823) of the lidar data (3830) composed of multiple pixels in which the pixel value is the corrected intensity value described above, the brightness distribution of the third region (3823) is more similar to the brightness distribution of the first region (3821) than the brightness distribution of the second region (3822).

[0955] Additionally, referring again to FIG. 26, the color channel value of each of the plurality of pixels is corrected based on the ambient values ​​of the plurality of pixels to generate the corrected color channel value of each of the plurality of pixels (S3770), which further enhances the characteristics of the object, such as gloss and shadow, in the generated color image (3840).

[0956]

[0957] The above-described embodiments and variations may be used separately or in combination.

[0958] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0959] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0960] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0961]

[0962] The form for carrying out the invention may substitute for the best form for carrying out the invention discussed above.

[0963]

[0964] -

[0965] -

Claims

1. A method for generating a color image using a LiDAR device including a detecting element array, wherein LiDAR data is acquired - wherein the LiDAR data is composed of a plurality of pixels, each of the plurality of pixels includes pixel location coordinates and at least one pixel value, and the at least one pixel value includes a distance value, an intensity value, and an ambient value; The direction of an external light source is estimated using the ambient values ​​of the plurality of pixels above—wherein, the direction of the external light source includes a light source direction vector—; Surface information for each of the plurality of pixels is estimated using the pixel position coordinates and distance values ​​of the plurality of pixels—wherein, the surface information for each of the plurality of pixels includes a normal vector—; Generating a geometric intensity value for each of the plurality of pixels based on the direction of the external light source and surface information for each of the plurality of pixels; Generating a corrected intensity value for each of the plurality of pixels based on the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels; and LiDAR data composed of a plurality of pixels, wherein the pixel value is the corrected intensity value, is input into a colorization model to generate a color image—wherein, the color image is composed of a plurality of color pixels, and each of the plurality of color pixels includes pixel position coordinates and at least one color channel value—; comprising Method for generating color images using a LiDAR device.

2. In Paragraph 1, Estimating the direction of the above external light source is, Identifying at least one object using the ambient values ​​of the plurality of pixels above; Identifying at least one shadow associated with at least one object using the ambient values ​​of the plurality of pixels; and Acquiring the light source direction vector for an external light source based on the at least one object and the at least one shadow associated with the at least one object; Method for generating color images using a LiDAR device.

3. In Paragraph 1, Estimating surface information for each of the above plurality of pixels is, Acquiring point data corresponding to each of the above plurality of pixels; Selecting a pixel group for each of the above plurality of pixels; Estimating a plane for each of the plurality of pixels based on point data of a selected pixel group for each of the plurality of pixels, and obtaining a normal vector of the estimated plane; Method for generating color images using a LiDAR device.

4. In Paragraph 1, The maximum value range of the above geometric intensity value is the same as the maximum value range of the above intensity value. Method for generating color images using a LiDAR device.

5. In Paragraph 4, Generating the above geometric intensity value is, Includes generating geometric intensity values ​​of (M,N) pixels, If the light source direction vector is L_light, the normal vector of the pixel (M,N) is N_(M,N), and the geometric intensity value of the pixel (M,N) is I_geo(M,N), then the following relationship is satisfied [Relationship] I_geo(M,N) = L_light N_(M,N) * Maximum value of intensity Method for generating color images using a LiDAR device.

6. In Paragraph 5, Generating the above corrected intensity value is, Generating the corrected intensity value by assigning weights to the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels. Method for generating color images using a LiDAR device.

7. In Paragraph 6, Generating the above corrected intensity value is, Includes generating corrected intensity values ​​for (M,N) pixels, If the intensity value of the above (M,N) pixel is I_raw(M,N) and the corrected intensity value of the above (M,N) pixel is I_amd(M,N), then the following relationship is satisfied [Relationship] I_amd(M,N) = a * I_raw(M,N) + (1-a) * I_geo(M,N) Method for generating color images using a LiDAR device.

8. In Paragraph 1, The above colorization model includes at least one artificial neural network layer, wherein The at least one artificial neural network layer comprises at least one of a feedforward neural network, a radial basis function network or a Kohonen self-organizing network, a convolutional neural network (CNN), a recurrent neural network (RNN), a Long Short Term Memory Network (LSTM), or Gated Recurrent Units (GRUs). Method for generating color images using a LiDAR device.

9. In Paragraph 8, The above colorization model is a model trained using a training dataset that takes black-and-white images as input data and color images as output data. Method for generating color images using a LiDAR device.

10. A method for generating a color image using a LiDAR device including a detecting element array, wherein LiDAR data is acquired - wherein the LiDAR data is composed of a plurality of pixels, each of the plurality of pixels includes pixel location coordinates and at least one pixel value, and the at least one pixel value includes a distance value, an intensity value, and an ambient value; The direction of an external light source is estimated using the ambient values ​​of the plurality of pixels above—wherein, the direction of the external light source includes a light source direction vector—; Surface information for each of the plurality of pixels is estimated using the pixel position coordinates and distance values ​​of the plurality of pixels—wherein, the surface information for each of the plurality of pixels includes a normal vector—; Generating a geometric intensity value for each of the plurality of pixels based on the direction of the external light source and surface information for each of the plurality of pixels; Generating a corrected intensity value for each of the plurality of pixels based on the intensity value of each of the plurality of pixels and the geometric intensity value of each of the plurality of pixels; Generating a weighted intensity value for each of the plurality of pixels by adjusting the corrected intensity value of each of the plurality of pixels based on the ambient value of each of the plurality of pixels; and LiDAR data composed of a plurality of pixels, wherein the pixel value is the weighted intensity value, is input into a colorization model to generate a color image—wherein, the color image is composed of a plurality of color pixels, and each of the plurality of color pixels includes pixel location coordinates and at least one color channel value—; comprising Method for generating color images using a LiDAR device.

Images