Electronic device and method for matching point clouds to three-dimensional space

Abstract

The electronic device of the present invention is configured to acquire a first point cloud for three-dimensional space from a first sensing device among the plurality of sensing devices, acquire a second point cloud for three-dimensional space from a second sensing device among the plurality of sensing devices, select at least two first reference points from the first point cloud based on user input, select at least two second reference points from the second point cloud based on user input, the second reference points corresponding to the at least two first reference points selected from the first point cloud, and align the first point cloud and the second point cloud based on the at least two first reference points and the at least two second reference points.

Description

[Technical Field]

[0001] The present invention relates to an electronic device and method for aligning a point cloud to a three-dimensional space. [Background technology]

[0002] Recently, technology that uses 3D sensors to sense three-dimensional space and generate data about the three-dimensional space based on this sensing has been used in various industrial technology fields. One type of 3D sensor, the Light Detection and Ranging (LiDAR) sensor, emits light toward an object in three-dimensional space and receives the reflected light to obtain information about the three-dimensional space. For example, a LiDAR sensor can sense the distance to an object in three-dimensional space and various physical properties, making it useful for autonomous driving technology.

[0003] Since there is a physical limit to the sensing area that one sensing device can detect, multiple sensing devices must be appropriately positioned to cover a large space. Information about a large space can be obtained based on data received from multiple sensing devices. When multiple lidar sensors are used to obtain multiple pieces of information about a 3D space, it is important to accurately match the multiple pieces of information. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0004] The objective of the present invention is to perform calibration between 3D point clouds without a pre-mapping process.

[0005] Another object of the present invention is to reduce the waste of computing resources and the delay in computing time by performing matching between three-dimensional point clouds without using a three-dimensional map (e.g., HD Map).

[0006] Yet another object of the present invention is to perform alignment between three-dimensional point clouds with simple operations, without the user having to set parameters such as a Cartesian coordinate system (e.g., x, y, z axes) or three-axis rotational motion (e.g., pitch, yaw, roll). [Means for solving the problem]

[0007] An electronic device according to the present invention includes a communication circuit communicatively connected with a plurality of sensing devices, an input device, one or more processors, and one or more memories storing instructions that, when executed, cause the one or more processors to acquire a first point cloud for a three-dimensional space from a first sensing device of the plurality of sensing devices, acquire a second point cloud for the three-dimensional space from a second sensing device of the plurality of sensing devices, select at least two first reference points from the first point cloud based on user input, select at least two second reference points from the second point cloud based on user input, the second reference points corresponding to the at least two first reference points selected from the first point cloud, and align the first point cloud with the second point cloud based on the at least two first reference points and the at least two second reference points.

[0008] The instructions are configured to cause the one or more processors to align the at least two first reference points and the at least two second reference points based on a predetermined time point, and to align the first point cloud and the second point cloud based on the aligned at least two first reference points and the at least two second reference points.

[0009] The instructions are configured to cause the one or more processors to determine, based on user input, the first sensing device as a reference sensing device and the second sensing device as a sensing device to be corrected, and to align the at least two first reference points and the at least two second reference points based on a predetermined point in time by rotating or translating the second point cloud around three mutually perpendicular rotation axes with respect to the first point cloud.

[0010] The instructions are configured to cause the one or more processors to: identify an overlap region between the first point cloud and the aligned second point cloud; determine, in the overlap region, a first point of the first point cloud and a second point of the second point cloud corresponding to the first point, where the distance between the first point and the second point is less than a predetermined distance; calculate a position vector between the first point and the second point; and perform a fine alignment between the first point cloud and the aligned second point cloud based on the position vector.

[0011] The instructions are configured to cause the one or more processors to calculate a position vector between the first point and the second point based on position coordinates of the first point and position coordinates of the second point on a three-dimensional spatial coordinate system.

[0012] At least two landmarks are positioned in the three-dimensional space, and the instructions are configured to cause the one or more processors to select positions of the at least two landmarks included in the first point cloud as the at least two first reference points and to select positions of the at least two landmarks included in the second point cloud as the at least two second reference points.

[0013] Each of the at least two landmarks is an object or a portion of an object.

[0014] Each of the plurality of sensing devices includes a lidar sensor that acquires a point cloud for the three-dimensional space.

[0015] Further comprising a display, the instructions are configured to cause the one or more processors to display the first point cloud and the second point cloud on the display.

[0016] The system further includes a display, and the instructions are configured to cause the one or more processors to display the aligned first and second point clouds on the display in response to completion of alignment of the first and second point clouds.

[0017] A method for aligning point clouds for a three-dimensional space of an electronic device according to the present invention includes: an operation of acquiring a first point cloud for the three-dimensional space from a first sensing device among a plurality of sensing devices; an operation of acquiring a second point cloud for the three-dimensional space from a second sensing device among the plurality of sensing devices; an operation of selecting at least two first reference points from the first point cloud based on user input; an operation of selecting at least two second reference points from the second point cloud based on user input, the second reference points corresponding to the at least two first reference points selected from the first point cloud; and an operation of aligning the first point cloud with the second point cloud based on the at least two first reference points and the at least two second reference points.

[0018] The matching operation includes an operation of aligning the at least two first reference points and the at least two second reference points based on a predetermined point in time, and an operation of matching the first point cloud and the second point cloud based on the aligned at least two first reference points and the at least two second reference points.

[0019] After the operation of acquiring the second point cloud, the method further includes an operation of determining the first sensing device as a reference sensing device and the second sensing device as a sensing device to be corrected based on user input, and the aligning operation further includes an operation of aligning the at least two first reference points and the at least two second reference points based on a predetermined point in time by rotating or translating the second point cloud around three mutually perpendicular rotation axes based on the first point cloud.

[0020] The matching operation further includes the operations of: identifying an overlap region between the first point cloud and the aligned second point cloud; selecting, in the overlap region, a first point of the first point cloud and a second point of the second point cloud corresponding to the first point, wherein the distance between the first point and the second point is less than a predetermined distance; calculating a position vector between the first point and the second point; and performing a fine alignment between the first point cloud and the aligned second point cloud based on the position vector.

[0021] The operation of calculating the position vector is an operation of calculating a position vector between the first point and the second point based on the position coordinates of the first point and the position coordinates of the second point on a three-dimensional spatial coordinate system.

[0022] At least two landmarks are arranged in the three-dimensional space, and the operation of selecting the at least two first reference points is an operation of selecting positions of the at least two landmarks included in the first point cloud as the at least two first reference points, and the operation of selecting the at least two second reference points is an operation of selecting positions of the at least two landmarks included in the second point cloud as the at least two second reference points.

[0023] Each of the at least two landmarks is an object or a portion of an object.

[0024] Each of the plurality of sensing devices includes a lidar sensor that acquires a point cloud for the three-dimensional space.

[0025] The method further includes the act of displaying the first point cloud and the second point cloud with a display.

[0026] The method further includes, in response to completion of the alignment of the first point cloud and the second point cloud, displaying the aligned first point cloud and the aligned second point cloud on a display. [Effects of the Invention]

[0027] According to the present invention, calibration between 3D point clouds can be performed without a pre-mapping process.

[0028] Furthermore, according to the present invention, matching between 3D point clouds can be performed without using a 3D map (e.g., HD Map), thereby reducing the waste of computing resources and the delay in computing time.

[0029] Furthermore, according to the present invention, the user can perform alignment between three-dimensional point clouds with simple operations, without having to set parameters such as a Cartesian coordinate system (e.g., x, y, z axes) or three-axis rotational movement (e.g., pitch, yaw, roll). [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a diagram illustrating a system according to one embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram of an electronic device and a sensing device according to one embodiment of the present invention. [Figure 3] FIG. 3 is a diagram illustrating a method for sensing a three-dimensional space using multiple sensing devices according to one embodiment of the present invention. [Figure 4] FIG. 4 is an operational flowchart of an electronic device according to an embodiment of the present invention. [Figure 5] FIG. 5 is a diagram illustrating a method for sensing a three-dimensional space using multiple sensing devices according to one embodiment of the present invention. [Figure 6] FIG. 6 is an operational flowchart of an electronic device according to an embodiment of the present invention. [Figure 7] FIG. 7 is a screen shot of a program that provides various information about a three-dimensional space according to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing a first point cloud acquired from a first sensing device determined by a reference sensing device according to one embodiment of the present invention, and a second point cloud acquired from a second sensing device determined by a sensing device to be corrected. [Figure 9] FIG. 9 is a diagram showing a first point cloud acquired from a first sensing device determined by a reference sensing device according to one embodiment of the present invention, and a second point cloud acquired from a second sensing device determined by a sensing device to be corrected. [Figure 10] FIG. 10 is a diagram illustrating aligned first and second point clouds according to one embodiment of the present invention. [Figure 11] FIG. 11 is an operational flowchart of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] The embodiments of the present invention are illustrated for the purpose of explaining the technical concept of the present invention, and the scope of the invention is not limited to the embodiments presented below or the specific description of these embodiments.

[0032] Unless otherwise defined, all technical and scientific terms used in the present invention have the meanings commonly understood by those of ordinary skill in the art to which the present invention belongs. All terms used in the present invention are selected for the purpose of more clearly describing the present invention, and are not selected to limit the scope of the rights of the present invention.

[0033] As used herein, expressions such as "including," "comprising," "having," etc. should be understood as open-ended terms that include the possibility of including other embodiments, unless otherwise stated in the phrase or sentence in which the expression is included.

[0034] Unless otherwise specified, singular expressions described in the present invention also include plural meanings, and this also applies to singular expressions described in the claims. The expressions "first," "second," etc. used in the present invention are used to distinguish multiple elements from one another, and do not limit the order or importance of the elements.

[0035] The term "module" as used herein refers to software or hardware components such as field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs). However, "modules" are not limited to hardware and software. A "module" may also be configured to reside on an addressable storage medium or to execute on one or more processors. Thus, by way of example, "modules" include components such as software components, object-oriented software components, class components, and task components, as well as processors, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within components and "modules" may be combined into fewer components and "modules" or further separated into additional components and "modules."

[0036] As used herein, the phrase "based on" is used to describe one or more factors that influence the decision, act of judgment, or behavior described in the phrase or sentence containing the phrase, and does not exclude additional factors that influence the decision, act of judgment, or behavior.

[0037] In the present invention, when a component is referred to as being "coupled" or "connected" to another component, it should be understood that the component can be directly coupled or connected to the other component, or can be coupled or connected via yet another component.

[0038] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, identical or corresponding components are designated by the same reference numerals. Furthermore, in the following description of the embodiments, duplicated descriptions of identical or corresponding components may be omitted. However, omission of a description of a component does not mean that such a component is not included in a certain embodiment.

[0039] FIG. 1 illustrates a system 10 according to an embodiment of the present invention. The system 10 includes an electronic device 110 and a plurality of sensing devices 120. The electronic device 110 is communicatively connected to the plurality of sensing devices 120 to transmit and receive various data. While the figure illustrates an example in which the plurality of sensing devices 120 is three (e.g., a first sensing device 120a, a second sensing device 120b, and a third sensing device 120c), the number of the plurality of sensing devices 120 is not limited thereto.

[0040] The sensing device 120 according to one embodiment is a device that acquires a point cloud as spatial information for a three-dimensional space. The sensing device 120 acquires the point cloud for the three-dimensional space by emitting light into the three-dimensional space and receiving the light reflected by an object. The sensing device 120 includes at least one sensor. A point cloud refers to a set cloud of multiple points spread in a three-dimensional space. A point cloud is also referred to as a set of points, a point group, or point cloud data. Unlike a two-dimensional image, a point cloud contains depth (z-axis) information, and is therefore data that enables three-dimensional modeling.

[0041] According to one embodiment, the sensing device 120 acquires a point cloud for a three-dimensional space. For example, the sensing device 120 acquires a point cloud for a three-dimensional space by time or at predetermined time intervals (e.g., 0.1 seconds), thereby acquiring a time-series point cloud.

[0042] According to an embodiment, the sensing device 120 is installed indoors or outdoors at a location where it can sense a three-dimensional space. To monitor a wide three-dimensional space, a plurality of sensing devices 120 are installed in consideration of a sensing area corresponding to the range of sensing by sensors included in the sensing device. For example, the plurality of sensing devices 120 may be installed at regular intervals. For example, the plurality of sensing devices 120 may be distributed at positions where they sense the three-dimensional space in different directions. For example, the plurality of sensing devices 120 may be distributed at positions where they sense one area of ​​the three-dimensional space.

[0043] The sensing device 120 includes a Light Detection and Ranging (LiDAR) sensor as a 3D sensor that senses a three-dimensional space. A sensing device including a LiDAR sensor acquires a volumetric point cloud for a three-dimensional space. The LiDAR sensor can sense the shape, size, and position of an object contained in the three-dimensional space. A multi-channel LiDAR sensor that collects information for a three-dimensional space is suitable for applications that utilize the approximate shape, size, and volume of an object.

[0044] The sensing device 120 may further include various types of sensors, such as a radar sensor, an infrared sensor, and a camera (image sensor). The sensing device 120 may include multiple sensors of the same type or may use a combination of different types of sensors. However, the types of sensors are not limited thereto.

[0045] According to an embodiment, the electronic device 110 is a server device that operates a service that provides 3D information about a 3D space. The electronic device 110 may be implemented using cloud computing technology. The electronic device 110 is communicatively connected to a plurality of sensing devices 120 and can acquire a point cloud about the 3D space from the plurality of sensing devices 120. The electronic device 110 can provide a customer with 3D information about the 3D space using the point cloud acquired from the plurality of sensing devices 120. The customer can use the service using their terminal device (not shown). For example, a customer can install a plurality of sensing devices 120 at locations where they want to monitor a 3D space. The plurality of sensing devices 120 transmit the point cloud about the 3D space to the electronic device 110. The electronic device 110 models the 3D information about the 3D space using the point cloud acquired from the plurality of sensing devices 120. The electronic device 110 transmits the 3D information about the 3D space to the customer's terminal device. The customer can receive various services provided by the electronic device 110 via the customer's terminal device.

[0046] Because the sensing area that one sensing device 120 can sense is limited, in order to monitor a large three-dimensional space, multiple sensing devices 120 must sense each area included in the three-dimensional space. In this case, in order to obtain three-dimensional information for the entire three-dimensional space, it is important to accurately align (calibrate) the point clouds acquired by each of the multiple sensing devices 120 into a single data set. For example, if the point clouds obtained by the multiple sensing devices 120 are not accurately aligned, each of the multiple sensing devices 120 may mistakenly recognize an object as a different object.

[0047] Generally, to align multiple point clouds, Cartesian coordinate system information (three-axis (e.g., x, y, z) coordinate information) and three-axis rotational movement information (e.g., pitch, yaw, and roll information) must be known. For example, since point clouds contain three-dimensional information, the pitch, yaw, or roll values ​​of the multiple point clouds differ depending on the direction from which each of the multiple sensing devices 120 views the three-dimensional space. Therefore, it is very difficult to align point clouds unless one is an expert with a deep understanding of point clouds.

[0048] Furthermore, to align multiple point clouds, each of the multiple point clouds can be mapped to 3D map information based on 3D map information. For example, multiple point clouds can be aligned by acquiring HD map information for 3D space (e.g., satellite information for 3D space) in advance and mapping each point cloud to the corresponding map information. In this case, since 3D map information must be used for high-precision alignment, the mapping process requires a lot of computational resources. Furthermore, when 3D map information is used, it takes a lot of time to process multiple calculations.

[0049] The present invention discloses a method for performing calibration between 3D point clouds without a pre-mapping process, the specific calibration method of which will be described later.

[0050] FIG. 2 is a block diagram of an electronic device 110 and a sensing device 120 according to one embodiment of the present invention.

[0051] As shown in FIG. 2 , an electronic device 110 according to one embodiment includes one or more processors 111, one or more memories 113, a communication circuit 115, an input device 117, and a display 119. Some components of the electronic device 110 may be omitted or replaced. Some additional or simplified components may be integrated or implemented as a single or multiple individual components. The term "processor 111" refers to a collection of one or more processors 111 unless otherwise clearly indicated in the context. The term "memory 113" refers to a collection of one or more memories 113 unless otherwise clearly indicated in the context. At least some components within the electronic device 110 may be connected to each other via a bus, a general purpose input / output (GPIO), a serial peripheral interface (SPI), a mobile industry processor interface (MIPI), or the like to exchange data and / or signals.

[0052] According to an embodiment, the processor 111 of the electronic device 110 may perform calculations and data processing related to control and / or communication with each component of the electronic device 110. The processor 111 may be operatively connected to, for example, the components of the electronic device 110. The processor 111 may store instructions or data received from other components of the electronic device 110 in the memory 113 of the electronic device 110, load and process the instructions or data stored in the memory 113, and store the resulting data back in the memory 113. The memory 113 stores instructions for the operation of the processor 111.

[0053] According to an embodiment, the memory 113 of the electronic device 110 can store various information. The memory 113 stores basic information about the plurality of sensing devices 120. The memory 113 stores a plurality of point clouds acquired from the plurality of sensing devices 120.

[0054] According to one embodiment, the communication circuitry 115 of the electronic device 110 can establish a wired or wireless communication channel with an external device (e.g., multiple sensing devices 120) and transmit and receive various data to and from the external device. According to one embodiment, the communication circuitry 115 includes at least one port for connecting to an external device via a wired cable for wired communication with the external device. In this case, the communication circuitry 115 can communicate with the wired external device via the at least one port. According to one embodiment, the communication circuitry 115 includes a cellular communication module and can be configured to connect to a cellular network (e.g., 3G, LTE, 5G, Wibro, or Wimax). According to another embodiment, the communication circuitry 115 includes a short-range communication module and can transmit and receive data to and from the external device using short-range communication (e.g., but not limited to, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), and UWB).

[0055] In one embodiment, input device 117 of electronic device 110 can receive instructions or data from outside (e.g., a user) of electronic device 110 for use by components (e.g., processor 111) of electronic device 110. Input device 117 may include, for example, a mouse, a microphone, or a keyboard.

[0056] According to an embodiment, the display 119 of the electronic device 110 displays various screens under the control of the processor 111. The display 119 is, for example, a monitor. The display 119 can be implemented in various ways, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display 119, an active-matrix organic light-emitting diode (AM-OLED), and a plasma display panel (PDP). According to an embodiment, the display 119 is implemented in the form of a touch sensor panel (TSP) that recognizes contact or proximity (e.g., hovering) of various external objects.

[0057] The sensing device 120 according to one embodiment includes a controller 121, a memory 123, a communication circuit 125, and at least one sensor 127. Some components of the sensing device 120 may be omitted or replaced. Some components may be integrated or implemented as a single or multiple individual components. The term "sensor 127" refers to a collection of one or more sensors 127 unless the context clearly indicates otherwise.

[0058] According to an embodiment, the controller 121 of the sensing device 120 performs calculations and data processing related to control and / or communication of each component of the sensing device 120. The controller 121 is operatively connected to, for example, the components of the sensing device 120. The controller 121 stores commands or data received from other components of the sensing device 120 in the memory 123 of the sensing device 120, loads and processes the commands or data stored in the memory 123, and stores the resulting data back in the memory 123. The memory 123 stores instructions related to the operation of the sensing device 120. The controller 121 of the sensing device 120 executes a program installed in the sensing device 120. The controller 121 controls a processing module that executes a program for sensing a three-dimensional space.

[0059] The sensor 127 of the sensing device 120 according to one embodiment is a sensor 127 for sensing a three-dimensional space. The sensor 127 includes a light-emitting unit that emits light into the three-dimensional space and a light-receiving unit that receives light reflected from an object, and further includes a dedicated controller 121 that acquires a point cloud for the three-dimensional space based on the intensity of the light received by the light-receiving unit. The sensor 127 can acquire a time-series (or time-dependent) point cloud for the three-dimensional space to track an object located in the three-dimensional space within the sensing area. The sensor 127 is a lidar sensor, including a three-dimensional lidar sensor, and can acquire data for a specific range of space. The sensor 127 can further include various types of sensors, such as a radar sensor, an infrared sensor, an ultrasonic sensor, or a camera, depending on the environment.

[0060] According to one embodiment, the communication circuitry 125 of the sensing device 120 can establish a wired or wireless communication channel with an external device (e.g., the electronic device 110) to transmit and receive various data to and from the external device. According to one embodiment, the communication circuitry 125 includes at least one port for connecting to an external device via a wired cable for wired communication with the external device. In this case, the communication circuitry 125 can communicate with the wired external device via the at least one port. According to one embodiment, the communication circuitry 125 includes a cellular communication module and is configured to connect to a cellular network (e.g., 3G, LTE, 5G, Wibro, or Wimax). According to another embodiment, the communication circuitry 125 includes a short-range communication module and can transmit and receive data to and from the external device using short-range communication (e.g., but not limited to, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), and UWB).

[0061] According to an embodiment, the sensor 127 of the sensing device 120 further includes a position sensor (e.g., a GPS sensor) in addition to the LIDAR sensor described above. The sensing device 120 may also include a configuration for improving sensing performance depending on the installation environment of the sensing device 120. According to an embodiment, the controller 121 of the sensing device 120 operates the sensor 127 to acquire a point cloud for a three-dimensional space by executing one or more instructions stored in the memory 123, and transmits the point cloud for the three-dimensional space to the electronic device 110 via the communication circuit 125. The sensing device 120 may transmit information identifying the sensing device 120 (e.g., ID information) along with the point cloud for the three-dimensional space.

[0062] According to an embodiment, the processor 111 of the electronic device 110 generates 3D information for the 3D space by three-dimensionally modeling a structure of the 3D space based on the point clouds received from the plurality of sensing devices 120. The processor 111 may perform a series of operations to detect objects in the 3D space and monitor the 3D space based on the point cloud for the 3D space. The electronic device 110 receives the point clouds for the 3D space from the plurality of sensing devices 120, and performs an operation of aligning the plurality of point clouds for the 3D space.

[0063] FIG. 3 is a diagram illustrating a method for sensing a three-dimensional space using multiple sensing devices 120 according to one embodiment of the present invention.

[0064] As shown in FIG. 3 , according to an embodiment, multiple sensing devices 120 are arranged at a certain distance apart to sense a three-dimensional space. While the present drawing assumes that there are two multiple sensing devices 120, the number of multiple sensing devices 120 is not limited thereto. The first sensing device 120a acquires a first point cloud for the three-dimensional space within the first sensing area 310a, and the second sensing device 120b acquires a second point cloud for the three-dimensional space within the second sensing area 310b. The first sensing device 120a transmits the acquired first point cloud to the electronic device 110, and the second sensing device 120b transmits the acquired second point cloud to the electronic device 110. Depending on the type of sensor, the sensing devices 120 have a predetermined field of view angle and a sensing limit distance, which determine a sensing area for detecting objects in the three-dimensional space.

[0065] Because the first sensing device 120a and the second sensing device 120b sense the three-dimensional space in different directions, the first point cloud and the second point cloud are not aligned with each other. For example, for an object located in the three-dimensional space, the shape of the object included in the first point cloud and the shape of the object included in the second point cloud may differ due to the difference in the sensing directions. That is, even if multiple sensing devices 120 sense the same three-dimensional space, different point clouds may be obtained depending on the locations of the sensing devices 120. If the first point cloud and the second point cloud do not match with each other, the electronic device 110 may mistakenly recognize the same object as different objects. Therefore, it is important for the electronic device 110 to accurately align the first point cloud and the second point cloud with each other.

[0066] 4 is an operational flowchart of the electronic device 110 according to an embodiment of the present invention. Specifically, FIG. 4 is an operational flowchart of the reference point tuning (or landmark tuning) among the two tuning steps included in the 3D point cloud matching method performed by the electronic device 110.

[0067] As shown in operational flowchart 400, in operation 410, the processor 111 of the electronic device 110 according to one embodiment acquires a first point cloud for a specific time point in three-dimensional space from a first sensing device 120a among the plurality of sensing devices 120. The processor 111 receives the first point cloud for the three-dimensional space from the first sensing device 120a via the communication circuit 115. The first sensing device 120a continuously senses the three-dimensional space using a sensor to acquire a time-series (time-based) point cloud including the first point cloud for the three-dimensional space at a specific time point. The first sensing device 120a transfers the first point cloud for the three-dimensional space to the electronic device 110, and the electronic device 110 acquires the first point cloud for the three-dimensional space from the first sensing device 120a.

[0068] In one embodiment, in operation 420, the processor 111 acquires a second point cloud for a specific time in three-dimensional space from a second sensing device 120b of the plurality of sensing devices 120. Here, the first point cloud and the second point cloud are acquired at the same specific time. The processor 111 receives the second point cloud for three-dimensional space from the second sensing device 120b via the communication circuit 115. The second sensing device 120b continuously senses the three-dimensional space using a sensor to acquire time-series point clouds including the second point cloud for the three-dimensional space at the specific time. The second sensing device 120b transfers the second point cloud for three-dimensional space to the electronic device 110, and the electronic device 110 acquires the second point cloud for three-dimensional space from the second sensing device 120b. The second sensing device 120b is located at a position a certain distance away from the position where the first sensing device 120a is located.

[0069] In one embodiment, processor 111 selects at least two first reference points from the first point cloud based on user input in operation 430. Processor 111 receives user input for selecting at least two first reference points from the first point cloud via input device 117. For example, the user clicks at least two first reference points in the first point cloud displayed on display 119 via input device 117 (e.g., a mouse).

[0070] In one embodiment, in operation 440, the processor 111 selects, based on user input, at least two second reference points in the second point cloud that correspond to the at least two first reference points selected from the first point cloud. In the present invention, the term "corresponding" means that points in different point clouds indicate the same point in the real world. In other words, a second reference point corresponding to a first reference point means that the point indicated by the first reference point in the real world and the point indicated by the second reference point in the real world are the same. The processor 111 receives user input for selecting at least two second reference points from the second point cloud via the input device 117. For example, the user clicks at least two second reference points in the second point cloud displayed by the display 119 via the input device 117.

[0071] In one embodiment, the processor 111 aligns the first point cloud and the second point cloud based on at least two first reference points and at least two second reference points in operation 450. The processor 111 can align the at least two first reference points and the at least two second reference points based on a predetermined time point. The processor 111 aligns the first point cloud and the second point cloud based on the aligned at least two first reference points and at least two second reference points.

[0072] 5 is a diagram illustrating a method for sensing a three-dimensional space using a plurality of sensing devices 120 according to an embodiment of the present invention. Content that overlaps with that described in FIG. 3 will be omitted.

[0073] 5, in one embodiment, multiple sensing devices 120 are positioned a certain distance apart to sense a three-dimensional space. A first sensing device 120a acquires a first point cloud for the three-dimensional space within a first sensing area 310a, and a second sensing device 120b acquires a second point cloud for the three-dimensional space within a second sensing area 310b. The first sensing device 120a transfers the acquired first point cloud to the electronic device 110, and the second sensing device 120b transfers the acquired second point cloud to the electronic device 110. Depending on the type of sensor, the sensing devices have a predetermined field of view angle and sensing limit distance, which determine a sensing area for detecting objects in the three-dimensional space.

[0074] To accurately align the first point cloud and the second point cloud, it is important for the user to accurately select at least two first reference points and at least two second reference points. To this end, at least two landmarks 510 and 520 are positioned in three-dimensional space, particularly in the first sensing area 310a and the second sensing area 310b. For example, each of the at least two landmarks 510 and 520 is an object or a portion of an object. In this case, the shapes of the at least two landmarks 510 and 520 are included in the first point cloud and the second point cloud. The landmarks 510 and 520 are distinctive objects with shapes that are easily identifiable by the user. For example, the landmarks 510 and 520 are objects shaped like a tricorn hat. For example, the landmarks are the corners of an object. This allows the user to accurately select at least two second reference points corresponding to the at least two first reference points.

[0075] 6 is an operational flowchart of the electronic device 110 according to an embodiment of the present invention. Description of the same content as that described in FIG. 4 will be omitted. Specifically, FIG. 6 is an operational flowchart of the electronic device 110 for adjusting a reference point using an actual landmark.

[0076] As shown in the operational flowchart 600, in one embodiment, the processor 111 of the electronic device 110 acquires a first point cloud from a first sensing device 120a of the plurality of sensing devices 120 in Operation 610. In one embodiment, the processor 111 acquires a second point cloud from a second sensing device 120b in Operation 620. The processor 111 displays the first point cloud and the second point cloud on the display 119. FIG. 7 shows a screen 700 of a program that provides various information about a three-dimensional space. The screen 700 displays a first point cloud 710a and a second point cloud 710b for the three-dimensional space. The first point cloud 710a and the second point cloud 710b are expressed differently so that the user can easily distinguish them. For example, the processor 111 can display the first point cloud 710a and the second point cloud 710b in different colors on the display 119. FIG. 7 shows a screen 700 in which a first point cloud 710a and a second point cloud 710b are not aligned with each other.

[0077] According to an embodiment, in Operation 630, the processor 111 determines the second sensing device 120b as the calibration target sensing device based on user input. The user determines the second sensing device 120b as the calibration target sensing device by selecting the second point cloud from the first point cloud and the second point cloud displayed on the display 119 using the input device 117. In FIG. 7 , the user determines the second sensing device 120b as the calibration target sensing device by selecting the second point cloud 710b acquired from the second sensing device 120b as the calibration target data. In this case, the second point cloud 710b selected as the calibration target data may be changed to another color to allow the user to easily recognize that the selection of the calibration target data has been completed.

[0078] According to one embodiment, the processor 111 determines the first sensing device 120a as a reference sensing device based on user input in operation 640. The user determines the first sensing device 120a as the reference sensing device by selecting the first point cloud 710a acquired from the first sensing device 120a as reference data. The order of operations 630 and 640 may be reversed to achieve this embodiment.

[0079] In one embodiment, in operation 650, the processor 111 selects the positions of at least two landmarks included in the first point cloud as at least two first reference points based on user input. In one embodiment, as described above, at least two landmarks can also be placed in three-dimensional space. In this case, the user selects the positions of at least two landmarks included in the first point cloud as at least two first reference points using the input device 117. Placing landmarks is not required but is optional; the user can select at least two first reference points in the first point cloud even if there are no landmarks. For example, the user can select at least two unique topographical features that are easy for the user to identify in the first point cloud as at least two first reference points.

[0080] 8 illustrates a first point cloud 710a acquired from a first sensing device 120a, which is determined as a reference sensing device, and a second point cloud 710b acquired from a second sensing device 120b, which is determined as a correction target sensing device. A user selects at least two first reference points 801a, 803a from the first point cloud 710a using the input device 117. For example, the user may select the positions of at least two landmarks from the first point cloud 710a as the at least two first reference points 801a, 803a, or may select the at least two first reference points 801a, 803a based on terrain features having unique shapes in the first point cloud 710a. That is, the processor 111 may select the at least two first reference points 801a, 803a from the first point cloud 710a based on user input. That is, the user can select at least two first reference points 801a, 803a from the first point cloud 710a for matching without the need for a free mapping process.

[0081] 6 , in one embodiment, the processor 111 selects the positions of at least two landmarks included in the second point cloud as at least two second reference points based on user input in Operation 660. The user can select the positions of at least two landmarks included in the second point cloud as at least two second reference points using the input device 117. Because the at least two landmarks exist at specific positions in real three-dimensional space, using the at least two landmarks allows for easy and accurate selection of second reference points corresponding to the first reference points.

[0082] 9 is a diagram showing a first point cloud 710a acquired from a first sensing device 120a, which is determined as a reference sensing device, and a second point cloud 710b acquired from a second sensing device 120b, which is determined as a correction target sensing device. A user can select at least two second reference points 901b, 903b in the second point cloud 710b using the input device 117. For example, the user can select the positions of at least two landmarks in the second point cloud 710b as the at least two second reference points 901b, 903b, or can select the at least two second reference points 901b, 903b based on terrain features having unique shapes in the second point cloud 710b. That is, the processor 111 can select the at least two second reference points 901b, 903b from the second point cloud 710b based on user input. Here, the at least two second reference points 901b, 903b are points corresponding to the at least two first reference points 801a, 803a selected from the first point cloud.

[0083] Returning to FIG. 6 again, in operation 670, the processor 111 according to one embodiment can align at least two first reference points and at least two second reference points based on a predetermined time point. The processor 111 can align at least two first reference points and at least two second reference points based on a predetermined time point by rotating or translating the second point cloud around three mutually orthogonal rotation axes with respect to the first point cloud. Specifically, the processor 111 rotates the second point cloud, which is data to be corrected, at a predetermined time point so that the at least two first reference points and the at least two second reference points are arranged on the same plane. The processor 111 can perform pitch, yaw, and / or roll rotation on the second point cloud with respect to the first point cloud. Thereafter, the processor 111 can translate the second point cloud so that the at least two first reference points and the at least two second reference points arranged on the same plane are aligned.

[0084] In one embodiment, the processor 111 aligns the first point cloud and the second point cloud in operation 680. The processor 111 aligns the first point cloud and the second point cloud based on the aligned at least two first reference points and at least two second reference points. FIG. 10 is a diagram showing the aligned first point cloud 710a and second point cloud 710b. Comparing the first point cloud 710a and the second point cloud 710b with FIGS. 7 to 9, it can be seen that the at least two first reference points 801a, 803a and the at least two second reference points 901b, 903b of the first point cloud 710a and the second point cloud 710b are accurately aligned (see reference points 1001 and 1003).

[0085] According to the above method, a user simply selects at least two corresponding points in each of the first point cloud 710a and the second point cloud 710b to align the two point clouds.

[0086] Figure 11 is an operational flowchart of the electronic device 110 according to an embodiment of the present invention. Specifically, Figure 11 is an operational flowchart related to fine tuning, one of two adjustments included in the 3D point cloud matching method performed by the electronic device 110. The fine tuning in Figure 11 may be performed after the reference point adjustment in Figures 4 and 6, or may be performed without the reference point adjustment.

[0087] According to one embodiment, the processor 111 of the electronic device 110 may identify an overlap region between the first point cloud and the second point cloud in operation 1110. If reference point alignment between the first point cloud and the second point cloud has been performed, the processor 111 may identify an overlap region between the first point cloud and the aligned second point cloud.

[0088] In one embodiment, the processor 111 determines a first point of a first point cloud in the overlap region in Operation 1120. In one embodiment, the processor 111 determines a second point of a second point cloud corresponding to the first point in Operation 1130, where the distance between the first point and the second point is shorter than a predetermined distance. That is, two points having a distance shorter than the predetermined distance are used for fine alignment between the two point clouds, and points having a distance greater than the predetermined distance are not used for fine alignment between the two point clouds. In one embodiment, the processor 111 can calculate a position vector between the first point and the second point in Operation 1140. For example, the processor 111 can calculate a position vector from the first point of the first point cloud to the second point of the first point cloud. The processor 111 can calculate the position vector between the first point and the second point based on the position coordinates of the first point and the position coordinates of the second point in a three-dimensional spatial coordinate system. The processor 111 can determine that the two point clouds are separated by the distance corresponding to the position vector in the direction of the position vector.

[0089] In one embodiment, the processor 111 performs fine alignment between a first point cloud and a second point cloud based on the position vector in operation 1150. The processor 111 moves the first point cloud in the direction of the position vector by the size of the position vector to reduce the distance between the two point clouds. Alternatively, the processor 111 moves the second point cloud in the opposite direction of the position vector by the size of the position vector. In other words, two points that are closer than a predetermined distance are paired together to achieve fine alignment between the two point clouds. The above process can also reduce the fine distance difference between the two point clouds.

[0090] As described above, the user can easily and quickly align two point clouds using reference point alignment and / or fine alignment. Furthermore, since alignment can be performed between point clouds without a free mapping process, it is possible to reduce the waste of computing resources and the computation time.

[0091] Although process steps, method steps, algorithms, etc., are described sequentially in the flowcharts illustrated in the figures, such processes, methods, and algorithms may be configured to operate in any suitable order. That is, the process, method, and algorithm steps described in various embodiments of the present invention need not be performed in the order described in the present invention. Also, although some steps may be described as occurring asynchronously, in other embodiments, such steps may occur simultaneously. Furthermore, illustration of a process by depiction in a figure does not imply that the illustrated process is exclusive of other variations and modifications thereto, nor does it imply that the illustrated process or any of its steps is essential to one or more of the various embodiments of the present invention, or that the illustrated process is preferred.

[0092] Although the technical features of the present invention have been described above with reference to some embodiments and examples shown in the accompanying drawings, it should be understood that various substitutions, modifications, and alterations can be made without departing from the technical scope of the present invention, as understood by those skilled in the art. It is understood that such substitutions, modifications, and alterations fall within the scope of the appended claims. Although the above-described method has been described with reference to specific embodiments, the method can also be embodied as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of storage devices that store data readable by a computer system. Examples of computer-readable recording media include ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and the like. Furthermore, computer-readable recording media can be distributed among computer systems connected via a network, so that computer-readable code can be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the above-described embodiments can be easily construed by programmers skilled in the art.

Claims

1. a communication circuit communicatively coupled to the plurality of sensing devices; An input device; one or more processors; one or more memories; The one or more memories, when executed, cause the one or more processors to: acquiring a first point cloud for a three-dimensional space from a first sensing device among the plurality of sensing devices; acquiring a second point cloud for the three-dimensional space from a second sensing device among the plurality of sensing devices; selecting at least two first reference points from the first point cloud based on user input; selecting, based on user input, at least two second reference points from the second point cloud that correspond to at least two first reference points selected from the first point cloud; an electronic device storing instructions configured to align the first point cloud and the second point cloud based on the at least two first reference points and the at least two second reference points.

2. The instructions may cause the one or more processors to: aligning the at least two first reference points and the at least two second reference points based on a predetermined time point; 2. The electronic device of claim 1, further comprising: a first point cloud and a second point cloud configured to be aligned based on the at least two first reference points and the at least two second reference points.

3. The instructions may cause the one or more processors to: determining the first sensing device as a reference sensing device and the second sensing device as a sensing device to be corrected based on a user input; 3. The electronic device according to claim 2, wherein the electronic device is configured to align the at least two first reference points and the at least two second reference points based on a predetermined point in time by rotating or translating the second point cloud around three mutually orthogonal rotation axes with respect to the first point cloud.

4. The instructions may cause the one or more processors to: determining an overlap region between the first point cloud and the aligned second point cloud; determining a first point of the first point cloud and a second point of the second point cloud corresponding to the first point in the overlap region, wherein the distance between the first point and the second point is less than a predetermined distance; calculating a position vector between the first point and the second point; The electronic device of claim 3 , configured to perform a fine alignment between the first point cloud and the aligned second point cloud based on the position vector.

5. The instructions may cause the one or more processors to:

5. The electronic device according to claim 4, configured to calculate a position vector between the first point and the second point based on position coordinates of the first point and position coordinates of the second point on a three-dimensional spatial coordinate system.

6. At least two landmarks are arranged in the three-dimensional space; The instructions may cause the one or more processors to: selecting the positions of the at least two landmarks included in the first point cloud as the at least two first reference points; The electronic device of claim 1 , further configured to select the positions of the at least two landmarks included in the second point cloud as the at least two second reference points.

7. 7. The electronic device of claim 6, wherein each of the at least two landmarks is an object or a portion of an object.

8. The electronic device of claim 1 , wherein each of the plurality of sensing devices comprises a lidar sensor that acquires a point cloud for the three-dimensional space.

9. further comprising a display; The instructions may cause the one or more processors to:

2. The electronic device of claim 1, wherein the display is configured to display the first point cloud and the second point cloud.

10. further comprising a display; The instructions may cause the one or more processors to:

7. The electronic device of claim 6, further configured to display the aligned first and second point clouds on the display in response to completion of alignment of the first and second point clouds.

11. 1. A method for aligning a point cloud to a three-dimensional space of an electronic device, comprising: acquiring a first point cloud for the three-dimensional space from a first sensing device among a plurality of sensing devices; acquiring a second point cloud for the three-dimensional space from a second sensing device among the plurality of sensing devices; selecting at least two first reference points from the first point cloud based on user input; selecting, based on user input, at least two second reference points from the second point cloud that correspond to at least two first reference points selected from the first point cloud; and matching the first point cloud and the second point cloud based on the at least two first reference points and the at least two second reference points.

12. The matching action is aligning the at least two first reference points and the at least two second reference points based on a predetermined point in time; and matching the first point cloud and the second point cloud based on the aligned at least two first reference points and the aligned at least two second reference points.

13. After the operation of acquiring the second point cloud, the method further includes an operation of determining, based on a user input, the first sensing device as a reference sensing device and the second sensing device as a sensing device to be corrected; The matching action is The method for matching according to claim 12, further comprising an operation of aligning the at least two first reference points and the at least two second reference points based on a predetermined point in time by rotating or translating the second point cloud around three mutually orthogonal rotation axes with respect to the first point cloud.

14. The matching action is determining an overlap region between the first point cloud and the aligned second point cloud; selecting, in the overlap region, a first point of the first point cloud and a second point of the second point cloud corresponding to the first point, wherein the distance between the first point and the second point is less than a predetermined distance; calculating a position vector between the first point and the second point; 14. The method of claim 13, further comprising: performing a fine alignment between the first point cloud and the aligned second point cloud based on the position vector.

15. The operation of calculating the position vector includes:

15. The matching method according to claim 14, further comprising the step of calculating a position vector between the first point and the second point based on the position coordinates of the first point and the position coordinates of the second point on a three-dimensional spatial coordinate system.

16. At least two landmarks are arranged in the three-dimensional space; the operation of selecting the at least two first reference points is an operation of selecting positions of the at least two landmarks included in the first point cloud as the at least two first reference points; The matching method of claim 11, wherein the operation of selecting at least two second reference points is an operation of selecting the positions of the at least two landmarks included in the second point cloud as the at least two second reference points.

17. 17. The method of claim 16, wherein each of the at least two landmarks is an object or a portion of an object.

18. The method of claim 11 , wherein each of the plurality of sensing devices comprises a lidar sensor that acquires a point cloud for the three-dimensional space.

19. The method of claim 11 further comprising the act of displaying the first point cloud and the second point cloud on a display.

20. 12. The method of claim 11, further comprising the operation of displaying the aligned first and second point clouds on a display in response to completion of alignment of the first and second point clouds.

Images