Method and apparatus for noise modeling based on lidar data

Abstract

One embodiment of the present disclosure provides a method for generating a noise model. The method comprises the steps of: extracting lidar data of the same region of interest from a plurality of lidar frames containing a measurement target; merging the extracted lidar data and generating a noise distribution based on distance deviations and patterns between the lidar data; and generating a noise model from the noise distribution based on a gamma function.

Description

Technology Field

[0001] The present invention relates to a LiDAR data-based noise modeling method and apparatus, and more specifically, to a technology in which a LiDAR sensor collects data of a specific object and its surrounding environment and generates a noise model. Background Technology

[0002] Existing studies often used data generated in virtual simulation environments as is, or utilized existing noise distributions.

[0003] However, mathematically calculated virtual noise distributions struggle to reflect reality and thus exhibit noise distributions different from actual LiDAR data. To address this, a noise model with enhanced realism is required.

[0004] To this end, the distribution of real-world noise data can be analyzed and additionally rendered onto the noise data generated in the virtual environment to impart realism.

[0005] Conventional techniques have limitations in infusing realism into data generated within simulations. Most simulation tools rely heavily on embedded modeling, which suffers from the inherent problem of being unable to accurately reflect texture information and the multifaceted noise occurring in real-world scenarios.

[0006] Accordingly, the need for methods to quantify noise models is increasing. Prior art literature

[0007] Korean Registered Patent 10-2420585 (Apparatus and method for determining point cloud information considering the operating environment of a LiDAR system (Publication date: July 13, 2022)) The problem to be solved

[0008] The present invention aims to solve the problems of the aforementioned prior art by providing a technology in which a LiDAR sensor collects data of a specific object and its surrounding environment and generates a noise model.

[0009] However, the technical problem that this embodiment aims to solve is not limited to the technical problem described above, and other technical problems may exist. means of solving the problem

[0010] As a technical means for achieving the technical problem described above, an embodiment according to the first aspect of the present disclosure provides a method for generating a noise model. The method comprises the steps of: extracting lidar data of the same region of interest from a plurality of lidar frames containing a measurement target; merging the extracted lidar data and generating a noise distribution based on distance deviations and patterns between the lidar data; and generating a noise model from the noise distribution based on a gamma function.

[0011] Additionally, an embodiment according to a second aspect of the present disclosure provides a noise modeling generation device. The device comprises a communication module, at least one processor, and a memory electrically connected to the processor and storing at least one code executed by the processor, wherein the memory stores a code that, when executed through the processor, causes the processor to extract sensor data of the same region of interest from a plurality of lidar frames containing a measurement target, merge the extracted lidar data, generate a noise distribution based on distance deviations and patterns between the lidar data, and generate noise modeling from the noise distribution based on a gamma function. Effects of the invention

[0012] The present invention can enhance the realism of LiDAR data generated in a virtual environment by reducing the gap between synthetic data and actual sensor data to increase the realism of the generated LiDAR data.

[0013] In addition, the present invention can be utilized as a tool to generate a large amount of high-fidelity and realistic training data at a low cost, thereby ensuring economic efficiency.

[0014] In addition, the present invention can be utilized as a data generation algorithm for training a deep learning model specialized in object recognition of a driving vehicle.

[0015] In addition, the present invention can generate virtual data that can more closely mimic actual sensor characteristics.

[0016] In addition, the present invention more realistically represents the way an actual lidar sensor operates and, when handling near and far distance data, can simulate reality more closely using a gamma distribution model that operates in actual lidar rather than the consistent noise profile of existing Gaussian models. Brief explanation of the drawing

[0017] FIG. 1 is a drawing illustrating a noise modeling generation device connected to a lidar sensor in communication according to one embodiment of the present invention. Figure 2 is a diagram illustrating the detailed configuration of the noise modeling generation device shown in Figure 1. FIG. 3 is a diagram showing the noise distribution and gamma distribution function of lidar data according to one embodiment of the present invention. Figure 4 is a diagram showing data measured by a lidar according to an embodiment of the present invention as 9-axis freeway and lidar data. FIG. 5 is a diagram showing data to which a gamma noise distribution is applied and data to which a Gaussian noise distribution is applied according to one embodiment of the present invention. FIG. 6 is a flowchart illustrating the sequence of a noise modeling generation method according to another embodiment of the present invention. Specific details for implementing the invention

[0018] The present disclosure will be described in detail below with reference to the attached drawings. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed herein, and the technical concept disclosed herein is not limited by the attached drawings. All terms used herein, including technical and scientific terms, should be interpreted in the sense generally understood by those skilled in the art to which the present disclosure pertains. Terms defined in advance should be interpreted as having additional meanings consistent with relevant technical literature and the present disclosure, and should not be interpreted in a highly ideal or restrictive sense unless otherwise defined.

[0019] In order to clearly explain the invention in the drawings, parts unrelated to the explanation have been omitted, and the size, form, and shape of each component shown in the drawings may be varied. Throughout the specification, identical or similar parts are denoted by identical or similar reference numerals.

[0020] Throughout the specification, when it is stated that a part is "connected (connected, contacted, or coupled)" to another part, this includes not only cases where they are "directly connected (connected, contacted, or coupled)," but also cases where they are "indirectly connected (connected, contacted, or coupled)" with other members interposed therebetween. Furthermore, when it is stated that a part "includes (provides, or provides)" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for additional "included (provided, or provided)" of other components.

[0021] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, "part" is not limited to software or hardware, and "part" may be configured to reside in an addressable storage medium or configured to run on one or more processors. Accordingly, as an example, "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." In addition, the components and '~parts' may be implemented to play one or more CPUs within the device or secure multimedia card.

[0022] In the following description, suffixes such as "module" and "part" for components are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art have been omitted where it is determined that such detailed descriptions could obscure the essence of the embodiments disclosed in this specification.

[0023] Terms indicating ordinal numbers, such as first, second, etc., used herein are used solely for the purpose of distinguishing one component from another and do not limit the order or relationship of the components. For example, the first component of the present disclosure may be named the second component, and similarly, the second component may be named the first component. Singular forms used herein should be interpreted to include plural forms unless explicitly to the contrary.

[0024] The "user terminal" mentioned below may be implemented as a computer or portable terminal capable of connecting to a server or other terminal via a network. Here, the computer may include, for example, a laptop, desktop, or notebook equipped with a web browser, and VR HMDs (e.g., HTC VIVE, Oculus Rift, GearVR, DayDream, PSVR, etc.). Here, the VR HMD includes PC models (e.g., HTC VIVE, Oculus Rift, FOVE, Deepon, etc.), mobile models (e.g., GearVR, DayDream, Storm Mirror, Google Cardboard, etc.), console models (PSVR), and stand-alone models implemented independently (e.g., Deepon, PICO, etc.). Portable terminals are wireless communication devices that ensure portability and mobility, and may include, for example, smartphones, tablet PCs, and wearable devices, as well as various devices equipped with communication modules such as Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, Ultrasonic, Infrared, WiFi, and LiFi. Additionally, "network" refers to a connection structure capable of exchanging information between each node, such as terminals and servers, and includes Local Area Networks (LAN), Wide Area Networks (WAN), the World Wide Web (WWW), wired and wireless data communication networks, telephone networks, wired and wireless television communication networks, etc.Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, Visible Light Communication (VLC), and LiFi.

[0025] FIG. 1 is a drawing illustrating a noise modeling generation device connected to a lidar sensor in communication according to one embodiment of the present invention.

[0026] Referring to FIG. 1, the noise modeling generation device of the present invention and the LiDAR sensor can be connected via communication.

[0027] The noise modeling generation device (100) may be in the form of a server. The server may be formed as a cloud computing server such as SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). Additionally, the management server may be built in the form of a private cloud, a public cloud, or a hybrid cloud system, but the scope of the present invention is not limited thereto.

[0028] The noise modeling generation device (100) receives a plurality of LiDAR frames containing a measurement target from a LiDAR sensor (200) and extracts LiDAR data of the same region of interest. Then, the extracted LiDAR data are merged, and a noise distribution can be generated based on the distance deviation and pattern between the LiDAR data. Then, a noise model can be generated from the noise distribution based on a gamma function. Then, the noise model can be applied to a preset simulation environment to extract LiDAR data above a preset threshold. Then, the LiDAR data above a preset threshold can be applied to an object recognition model to verify the LiDAR data above a preset threshold.

[0029] The lidar sensor (200) can collect multiple lidar frames containing a measurement target. At this time, the multiple lidar frames can be collected according to the distance recognized by the lidar sensor (200).

[0030] Figure 2 is a diagram illustrating the detailed configuration of the noise modeling generation device shown in Figure 1.

[0031] Referring to FIG. 2, the noise modeling generation device (100) of the present invention includes a communication module (110), a processor (120), and a memory (130).

[0032] The communication module (110) may include a device comprising hardware and software necessary to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device.

[0033] The communication module (110) can receive point cloud data from the lidar sensor (200). However, the role of the communication module (110) is not limited to this.

[0034] The processor (120) may include various types of devices for controlling and processing data. The processor (120) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program.

[0035] In one example, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the invention is not limited thereto.

[0036] The processor (120) performs operations according to the code stored in memory (130).

[0037] The memory (130) can store at least one of the information and data input to the communication module (110), the information and data required for the function performed by the processor (120), and the data generated according to the execution of the processor (120).

[0038] The term "memory" (130) should be interpreted as a general term for a non-volatile storage device that retains stored information even when power is not supplied, and a volatile storage device that requires power to retain stored information. In addition to a volatile storage device that requires power to retain stored information, the memory (130) may include a magnetic storage medium or a flash storage medium, but the scope of the present invention is not limited thereto.

[0039] Memory (130) is electrically connected to the processor (120) and stores at least one code that is executed by the processor (120). Memory (130) stores code that causes the processor (120) to perform the following functions and procedures when executed through the processor (120).

[0040] Memory (130) stores code that causes the extraction of LiDAR data of the same region of interest from multiple LiDAR frames containing the measurement target.

[0041] The memory (130) stores code that merges the extracted lidar data and causes a noise distribution to be generated based on the distance deviation and pattern between the lidar data.

[0042] Memory (130) stores code that causes noise modeling to be generated from a noise distribution based on a gamma function. For example, the gamma function can generate multiple parameter distributions using a pre-set formula.

[0043] Memory (130) stores code that causes the noise model to be applied to a preset simulation environment to extract LiDAR data above a preset threshold.

[0044] Memory (130) stores code that causes the LiDAR data above a preset threshold to be applied to an object recognition model and to verify the LiDAR data above the preset threshold. For example, the object recognition model may be YOLO v4 (You Only Look Once v4).

[0045] FIG. 3 is a diagram showing the noise distribution and gamma distribution function of lidar data according to one embodiment of the present invention.

[0046] Referring to Fig. 3, the analysis of LiDAR data can be performed by designating a specific region of interest in multiple LiDAR frames capturing the same space. At this time, since system noise exists even when measuring the same point, the noise distribution at the same point can be calculated. Furthermore, to analyze the distribution of data for a specific point, the uncertainty of the point extracted from the frame exhibits a pattern appearing as a deviation along a linear line. This implies that the noise is distance-oriented rather than direction-oriented.

[0047] Based on this, the extracted data was analyzed, and as shown in Figure 6, it can be confirmed that the noise distribution is not a symmetrical Gaussian distribution seen in most simulations, but rather a distribution skewed in one direction.

[0048] Furthermore, LiDAR noise modeling can be approximated using a gamma distribution that includes shape and velocity to represent the analyzed noise distribution. The equation for this is as follows.

[0049]

[0050] At this time, It can be represented by the following gamma function.

[0051]

[0052] The simulation environment for LiDAR data generation utilizes a simulator equipped with various functions customized for the ongoing test, and virtual simulation data can be generated through this simulator.

[0053] FIG. 4 is a diagram showing data measured by a lidar sensor according to an embodiment of the present invention as 9-axis freeway and lidar data.

[0054] Referring to FIG. 4, the LiDAR measurement data is obtained by installing a LiDAR sensor (200) and a camera in a pre-set physical environment, and a noise modeling generation device (100) receives the surrounding GT (Ground Truth, measurement data) from the LiDAR sensor (200) and the camera. At this time, GT is generated at fixed distances while moving along a pre-set path, and GT generation can be stopped when stationary or in a similar surrounding environment. The GT collected in this way can be utilized together with position and velocity data collected by the device.

[0055] And GT can be represented as Label data. Label data has 9 degrees of freedom and has the position and orientation of the data along 9 axes. First, if the length of the GT in the x-axis direction, the width of the GT in the y-axis direction, and the height of the GT in the z-axis direction are known based on the x, y, and z of the center point of the GT, the object can be represented as a rectangular prism. In addition, the direction the rectangular prism faces can be represented by adding a yaw (z-axis) rotation. By inputting data with 7 degrees of freedom and 0 for the x and y-axis rotations, the GT can be represented with 9 degrees of freedom as shown in Fig. 4 (a).

[0056] And when the label is shown in Fig. 4(a) and the LiDAR data is displayed on a single screen, it is as shown in Fig. 4(b).

[0057] FIG. 5 is a diagram showing data to which a gamma noise distribution is applied and data to which a Gaussian noise distribution is applied according to one embodiment of the present invention.

[0058] Referring to FIG. 5, gamma and Gaussian distribution data can be generated by applying a gamma distribution or a Gaussian distribution to noise-free data extracted through a simulator. At this time, if a noise model with a gamma distribution applied to the noise-free data is applied, it can be represented as in FIG. 5 (a). And data with a Gaussian noise model applied to the noise-free data can be represented as in FIG. 5 (b).

[0059] FIG. 6 is a flowchart illustrating the sequence of a noise modeling generation method according to another embodiment of the present invention.

[0060] The noise modeling generation method described below can be performed by the noise modeling generation device described above with reference to FIGS. 1 to 5. Accordingly, the content of the embodiment of the present disclosure described above with reference to FIGS. 1 to 5 can be applied in the same way to the embodiment described below, and content that overlaps with the description above will be omitted. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps may be performed almost simultaneously.

[0061] Referring to FIG. 6, the noise modeling generation method includes a LiDAR data extraction step (S10), a noise distribution generation step (S20), a noise modeling generation step (S30), a LiDAR data extraction step (S40) above a preset threshold, and a LiDAR data verification step (S50) above a preset threshold.

[0062] The lidar data extraction step (S10) is a step of extracting lidar data of the same region of interest from multiple lidar frames containing a measurement target.

[0063] The noise distribution generation step (S20) is a step of merging the extracted lidar data and generating a noise distribution based on the distance deviation and pattern between the lidar data.

[0064] The noise modeling generation step (S30) is a step of generating a noise model from a noise distribution based on a gamma function. For example, the gamma function can generate multiple parameter distributions using a pre-set formula.

[0065] The step of extracting lidar data above a preset threshold (S40) is a step of applying a noise model to a preset simulation environment to extract lidar data above a preset threshold.

[0066] The step of verifying LiDAR data above a preset threshold (S50) is a step of verifying LiDAR data above a preset threshold by applying LiDAR data above a preset threshold to an object recognition model. For example, the object recognition model may be YOLO v4 (You Only Look Once v4).

[0067] One embodiment of the present invention may also be implemented in the form of a recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include all computer storage media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0068] Although the method and system of the present invention have been described in relation to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0069] A person skilled in the art to which this disclosure pertains will understand that, based on the foregoing description, other specific forms can be easily modified without altering the technical spirit or essential features of this disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of this disclosure is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalents thereof should be interpreted as being included within the scope of this disclosure. The scope of this application is defined by the claims set forth below rather than by the foregoing detailed description, and all modifications or variations derived from the meaning and scope of the claims and equivalents thereof should be interpreted as being included within the scope of this application. Explanation of the symbols

[0070] 100: Noise modeling generation device 110: Communication module 120: Processor 130: Memory 200: LiDAR sensor

Claims

Claim 1 A method for generating noise modeling, performed by a noise modeling generation device, comprising: a) extracting LiDAR data of the same region of interest from a plurality of LiDAR frames containing a measurement target; b) merging the extracted LiDAR data and generating a noise distribution based on distance deviations and patterns between the LiDAR data; and c) generating a noise model from the noise distribution based on a gamma function, wherein the gamma function generates a plurality of parameter distributions using a pre-set formula, and step b) includes designating a specific region of interest in a plurality of LiDAR frames capturing the same space, repeatedly measuring LiDAR data of the same point in the region of interest to calculate a noise distribution of the same point according to system noise, and analyzing a pattern in which the uncertainty of the same point extracted from the plurality of LiDAR frames appears as a deviation on a linear line. Claim 2 delete Claim 3 A noise modeling generation method according to claim 1, further comprising the step of applying the noise model to a preset simulation environment to extract LiDAR data above a preset threshold. Claim 4 A noise modeling generation method according to claim 3, further comprising the step of applying lidar data above a preset threshold to an object recognition model to verify lidar data above the preset threshold. Claim 5 A noise modeling generation method according to claim 4, wherein the object recognition model is YOLO v4 (You Only Look Once v4). Claim 6 A noise modeling generation device comprising: a communication module; at least one processor; and a memory electrically connected to the processor and storing at least one code executed by the processor, wherein the memory stores a code that, when executed through the processor, causes the processor to extract LiDAR data of the same region of interest from a plurality of LiDAR frames containing a measurement target, merge the extracted LiDAR data, generate a noise distribution based on distance deviations and patterns between the LiDAR data, and generate a noise model from the noise distribution based on a gamma function, wherein the gamma function generates a plurality of parameter distributions using a preset formula, and generating the noise distribution involves designating a specific region of interest in a plurality of LiDAR frames capturing the same space, repeatedly measuring LiDAR data of the same point in the region of interest to calculate the noise distribution of the same point according to system noise, and analyzing a pattern in which the uncertainty of the same point extracted from the plurality of LiDAR frames appears as a deviation on a linear line. Claim 7 delete Claim 8 A noise modeling generation device according to claim 6, which further stores code that causes the extraction of LiDAR data above a preset threshold by applying the noise model to a preset simulation environment. Claim 9 A noise modeling generating device according to claim 8, which further stores code that causes the LiDAR data above a preset threshold to be applied to an object recognition model and to verify the LiDAR data above the preset threshold. Claim 10 A noise modeling generation device according to claim 9, wherein the object recognition model is YOLO v4 (You Only Look Once v4).

Images