Method for generating spatial map using captured image of target area and electronic device for performing same

Abstract

A method for generating a spatial map according to the present invention may include the steps of: obtaining a captured image for a target area in a space; obtaining LiDAR scan data by scanning the depth of the target area with respect to a first height; performing object detection on the captured image; and generating a spatial map for the target area on the basis of the LiDAR scan data and the captured image according to the object detection.

Description

Method for generating a spatial map using a photographed image of a target area and an electronic device for performing the same

[0001] The present disclosure relates to a method for generating a spatial map, and to a method for increasing the accuracy of the spatial map by using a photographed image of a target area.

[0002] Electronic devices such as robot vacuum cleaners can scan a space using built-in LiDAR sensors as they move through it, and create a map of the space (spatial map) based on the scan data.

[0003] However, if a spatial map is created using only data (lidar scan data) scanned using a lidar sensor, the created spatial map may differ from the structure of the actual space.

[0004] This is because, while various objects (e.g., furniture, home appliances, etc.) may exist in a space, electronic devices cannot determine whether the point where the signal (light) transmitted from the lidar sensor is reflected is a wall of the space or an object existing within the space. In reality, electronic devices can generate a spatial map by recognizing the point where an object exists as a wall of the space, and therefore the boundaries of the spatial map may differ from the actual space.

[0005] Accordingly, an electronic device is provided that generates a more accurate spatial map of the surrounding environment based on lidar scan data and camera images. Additionally, a method for generating this map is also provided.

[0006] Additional embodiments are set forth in the following description, some of which will become apparent from the description or may be learned by practice of the examples presented.

[0007] According to one aspect of the present disclosure, a method for generating a spatial map disclosed as a technical means for achieving a technical problem may include a step of acquiring a captured image for a target area in space, a step of acquiring LiDAR scan data that scans a depth of the target area with respect to a first height, a step of performing object detection on the captured image, and a step of generating a spatial map for the target area based on the LiDAR scan data and the captured image according to the object detection.

[0008] According to one aspect of the present disclosure, an electronic device for generating a spatial map, disclosed as a technical means for achieving a technical problem, includes a memory storing a program for generating a spatial map and at least one processor, wherein the at least one processor executes the program to obtain a captured image for a target area in space, obtain LiDAR scan data obtained by scanning a depth of the target area with respect to a first height set in advance, perform object detection on the captured image, and then generate a spatial map for the target area based on both the LiDAR scan data and the captured image according to the object detection.

[0009] According to one aspect of the present disclosure, a non-transitory computer-readable recording medium disclosed as a technical means for achieving a technical task may have stored thereon a program for executing at least one of the embodiments of the disclosed method on a computer.

[0010] According to one embodiment of the present disclosure, a computer program may be stored on a non-transitory medium for performing at least one of the embodiments of the disclosed method on a computer.

[0011] The features, advantages and other aspects of the above and other specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0012] FIG. 1 is a diagram illustrating a system for generating a spatial map according to one embodiment of the present disclosure.

[0013] FIG. 2 is a diagram for comparing a spatial map generated according to one embodiment of the present disclosure with a spatial map generated using only existing lidar scan data.

[0014] FIG. 3 is a diagram for explaining the configuration of a server included in a system for generating a spatial map according to one embodiment of the present disclosure.

[0015] FIG. 4 is a drawing for explaining the configuration of a robot cleaner included in a system for generating a spatial map according to one embodiment of the present disclosure.

[0016] FIG. 5 is a drawing for explaining the purpose and main features of a method for generating a spatial map according to one embodiment of the present disclosure.

[0017] FIG. 6 is a flowchart illustrating a method for generating a spatial map according to one embodiment of the present disclosure.

[0018] FIG. 7 is a flowchart illustrating detailed steps included in step 605 of FIG. 6 according to one embodiment of the present disclosure.

[0019] FIG. 8 is a flowchart illustrating detailed steps included in step 701 of FIG. 7 according to one embodiment of the present disclosure.

[0020] FIG. 9 is a flowchart illustrating detailed steps included in step 802 of FIG. 8 according to one embodiment of the present disclosure.

[0021] FIG. 10 is a flowchart for explaining detailed steps included in step 803 of FIG. 8 according to one embodiment of the present disclosure.

[0022] FIG. 11 is a flowchart illustrating detailed steps included in step 803 of FIG. 8 according to one embodiment of the present disclosure.

[0023] FIG. 12 is a flowchart illustrating detailed steps included in step 803 of FIG. 8 according to one embodiment of the present disclosure.

[0024] FIG. 13 is a flowchart illustrating detailed steps included in step 902 of FIG. 9 according to one embodiment of the present disclosure.

[0025] FIG. 14 is a flowchart illustrating detailed steps included in step 902 of FIG. 9 according to one embodiment of the present disclosure.

[0026] FIG. 15 is a flowchart for explaining detailed steps included in step 902 of FIG. 9 according to one embodiment of the present disclosure.

[0027] FIG. 16 is a diagram for explaining a step of acquiring lidar scan data for a target area in a process of generating a spatial map according to one embodiment of the present disclosure.

[0028] FIG. 17 is a drawing for explaining a step of detecting an object in a captured image of a target area in a process of generating a spatial map according to one embodiment of the present disclosure.

[0029] FIG. 18 is a diagram for explaining a step of obtaining a depth image from a photographed image and obtaining depth values ​​from the depth image in a process of generating a spatial map according to one embodiment of the present disclosure.

[0030] FIG. 19 is a diagram for explaining a step of performing depth calibration using depth values ​​acquired from lidar scan data and depth images in the process of generating a spatial map according to one embodiment of the present disclosure.

[0031] FIG. 20 is a diagram for explaining a step of obtaining depth values ​​of points located at an optimal height in a process of generating a spatial map according to one embodiment of the present disclosure.

[0032] FIG. 21 is a diagram for explaining an operation of obtaining depth values ​​corresponding to lidar scan data from a depth image using a transformation matrix in a process of generating a spatial map according to one embodiment of the present disclosure.

[0033] FIG. 22 is a diagram for explaining an operation of obtaining depth values ​​corresponding to lidar scan data from a depth image through semantic segmentation in a process of generating a spatial map according to one embodiment of the present disclosure.

[0034] FIG. 23 is a diagram for explaining an operation of acquiring depth values ​​corresponding to lidar scan data from a depth image through elbow point detection in a process of generating a spatial map according to one embodiment of the present disclosure.

[0035] FIG. 24 is a drawing for explaining an operation of determining an optimal height in a process of generating a spatial map according to one embodiment of the present disclosure.

[0036] FIG. 25 is a diagram for explaining three conditions used to determine an optimal height in the process of generating a spatial map according to one embodiment of the present disclosure.

[0037] In describing this disclosure, descriptions of technical details that are well-known in the technical field to which this disclosure pertains and are not directly related to this disclosure will be omitted. This is to avoid obscuring the gist of this disclosure by omitting unnecessary explanations and to convey it more clearly. Furthermore, the terms described below are defined based on their functions in this disclosure and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the contents of this specification as a whole.

[0038] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.

[0039] As used herein, the expressions "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", "at least one of A, B, or C" are intended to include one or all possible combinations of the listed items. It should be understood that the terms "includes", "comprises", "has", "having", "including", and / or "comprising" as used herein specify the presence of stated features, values, steps, operations, components, elements, or combinations thereof, but do not preclude the presence or addition of one or more other features, values, steps, operations, components, elements, or combinations thereof.

[0040] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure of the present disclosure is complete and to fully inform those skilled in the art of the present disclosure of the scope of the disclosure. An embodiment of the present disclosure may be defined according to the claims. Like reference numerals denote like elements throughout the specification. In addition, when describing an embodiment of the present disclosure, if a detailed description of a related function or configuration is determined to unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, the terms described below are terms defined in consideration of the functions of the present disclosure and may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

[0041] In one embodiment, each block of the flowchart diagrams and combinations of the flowchart diagrams can be performed by computer program instructions. The computer program instructions can be installed on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, and the instructions, when executed by the processor of the computer or other programmable data processing apparatus, can create means for performing the functions described in the flowchart block(s). The computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing apparatus to implement the functions in a particular manner, and the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes instruction means for performing the functions described in the flowchart block(s). The computer program instructions can also be installed on a computer or other programmable data processing apparatus.

[0042] Additionally, each block in the flowchart diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). In one embodiment, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may be executed substantially simultaneously or, depending on the function, may be executed in reverse order.

[0043] The term '~ unit' used in one embodiment of the present disclosure may represent software or a hardware component such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the '~ unit' may perform a specific role. Meanwhile, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. In one embodiment, the '~ unit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided through a specific component or a specific '~ unit' may be combined to reduce the number of components or separated into additional components. In addition, in one embodiment, the '~ unit' may include one or more processors.

[0044] Below, the meanings of terms used in this disclosure are explained.

[0045] A "spatial map" may refer to a map representing the structure or shape of a space. According to one embodiment of the present disclosure, a spatial map representing the structure of a house in which a user resides may be generated using an electronic device such as a robot vacuum cleaner. The "spatial map" may be in the form of a general floor plan representing the structure of a house, etc., with various information (e.g., appliances or furniture placed within the space) added. Terms such as "floor plan" or "spatial layout" may be used instead of "spatial map."

[0046] The term "target area" may refer to an area that an electronic device photographs or scans in the process of generating a spatial map. According to one embodiment of the present disclosure, the electronic device may photograph a plurality of target areas within a space using a camera and scan them using a lidar sensor to generate a spatial map. The electronic device may generate a spatial map using the photographed images and lidar scan data acquired for the plurality of target areas within the space. Instead of the term "target area," terms such as "area of ​​interest," "photographed area," or "scanned area" may also be used.

[0047] 'LiDAR scan data' may refer to data acquired by scanning a target area using a LiDAR sensor equipped in an electronic device. According to one embodiment of the present disclosure, the electronic device may acquire LiDAR scan data including depth values ​​(distances from the LiDAR sensor to the plurality of points) of a plurality of spots located at a predetermined specific height using the LiDAR sensor. According to one embodiment of the present disclosure, the electronic device may acquire ToF scan data including depth values ​​of a plurality of points located at a specific height using a ToF sensor instead of the LiDAR sensor, and may perform embodiments of the present disclosure using the ToF scan data instead of the LiDAR scan data. Alternatively, according to one embodiment of the present disclosure, the electronic device may acquire stereo vision scan data including depth values ​​of a plurality of points located at a specific height using stereo vision technology using two cameras instead of the LiDAR sensor, and may perform embodiments of the present disclosure using the stereo vision scan data instead of the LiDAR scan data. Instead of 'lidar scan data', the terms '1D scan data', 'lidar data', 'depth scan data' or 'scan data' may also be used.

[0048] The term "captured image" may refer to an image of a target area captured using a camera equipped in an electronic device. The captured image may be an RGB image, and may be converted into a depth image through depth estimation as described in the embodiments of the present disclosure.

[0049] A 'depth image' may refer to an image including depth information for a specific space. Each of a plurality of points included in the depth image may have a corresponding depth value. As described above, a depth image may be acquired by performing depth estimation on a captured image. In the present disclosure, the depth information included in the depth image may refer to relative depth information expressing the relative depth between a plurality of points on the depth image. Accordingly, in embodiments of the present disclosure, an electronic device may perform depth calibration using lidar scan data to obtain absolute depth information (depth values ​​expressed in standard measurement units) for each point in the depth image. Instead of 'depth image', terms such as 'depth map' or 'depth estimation image' may also be used.

[0050] 'Depth calibration' may refer to an operation for matching the scales between lidar scan data and depth images. According to one embodiment of the present disclosure, lidar scan data may include absolute depth information (depth values ​​of points measured in standard measurement units), depth images may include relative depth information (values ​​that relatively represent the depths of points), and an electronic device may obtain a scale factor for converting relative depth values ​​for points on the depth image into absolute depth values ​​by performing depth calibration on the lidar scan data and the depth images. Instead of 'depth calibration', terms such as 'depth scaling', 'depth scale transformation', 'calibration', 'scaling', or 'scale transformation' may be used.

[0051] The term "scale factor" may refer to a value used to convert relative depth values ​​included in a depth image into absolute depth values. As previously explained, an electronic device can obtain a scale factor by performing depth calibration. The electronic device can obtain absolute depth values ​​by multiplying the relative depth values ​​for points included in the depth image by the scale factor. Instead of "scale factor," terms such as "scale value," "transformation factor," "global depth scale," or "depth scale" may also be used.

[0052] A 'depth value' may refer to a distance from a reference location for measurement (e.g., the location of an electronic device performing a photograph or scan) to a specific point in space. In embodiments of the present disclosure, the depth value may include a relative depth value and an absolute depth value. The relative depth value may refer to a value that relatively represents the depth of points, and the absolute depth value may refer to a value that measures the depth of points using a standard measurement unit. As a specific example, assuming that a first point and a second point are included in space, if the relative depth values ​​for the first point and the second point are 3 and 5, respectively, and the absolute depth value for the first point is 60 cm, the absolute depth value for the second point becomes 100 cm.

[0053] A 'dynamic height scan' may refer to an operation of selecting an optimal height in a target area to determine the boundary of a spatial map. According to one embodiment of the present disclosure, an electronic device may determine the boundary of a spatial map based on depth values ​​of points included in the optimal height for the target area. Instead of 'dynamic height scan', terms such as 'optimal height scan', 'dynamic height selection', or 'optimal height selection' may be used. In addition, instead of 'optimal height', terms such as 'optimal scan height', 'correction height', 'representative height', or 'reference height' may be used.

[0054] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0055] Embodiments of the present disclosure relate to a method for generating a spatial map using LiDAR scan data obtained by scanning a target area using a LiDAR sensor and an image obtained by capturing a target area using a camera, and an electronic apparatus for performing the same.

[0056] Electronic devices implementing the embodiments described herein may be implemented in various forms. According to one embodiment, an electronic device (e.g., a robot vacuum cleaner) equipped with both a lidar sensor and a camera may directly scan and capture a target space, thereby acquiring lidar scan data and captured images, and generate a spatial map based on the acquired lidar scan data and captured images.

[0057] Alternatively, according to one embodiment, an electronic device having only one of a lidar sensor or a camera may directly acquire only one of the lidar scan data or the captured image, acquire the other from another external device, and then generate a spatial map based on the acquired lidar scan data and captured image.

[0058] Alternatively, according to one embodiment, the electronic device may acquire both lidar scan data and captured images from outside, and generate a spatial map based on the acquired lidar scan data and captured images.

[0059] This disclosure describes embodiments in an environment where the electronic device is a robot vacuum cleaner. However, as previously described, the embodiments described in this disclosure may also be performed by other types of electronic devices other than a robot vacuum cleaner (e.g., a robotic mobile device that performs various operations while moving automatically or according to a user's command). For example, an electronic device such as a butler robot or a pet robot may perform the embodiments of this disclosure. If the embodiments of this disclosure are performed by an electronic device that does not include at least one of a lidar sensor and a camera, the operation of the robot vacuum cleaner directly acquiring lidar scan data or a photographed image described below may be replaced with an operation of acquiring lidar scan data or a photographed image from an external source.

[0060] In embodiments of the present disclosure, the robot cleaner determines whether a spatial map needs to be corrected (or whether the captured image needs to be used when generating a spatial map) based on a captured image from a camera, and can correct or generate a spatial map using a depth image acquired from the captured image and lidar scan data acquired via a lidar sensor. A detailed description is provided below with reference to the drawings.

[0061] FIG. 1 is a diagram illustrating a system for generating a spatial map according to one embodiment of the present disclosure. Referring to FIG. 1, the system for generating a spatial map according to one embodiment of the present disclosure may include a server (100), a robot cleaner (200), and a mobile terminal (300). As previously described, while embodiments using a robot cleaner (200) are described in the present disclosure, various other types of electronic devices may be used in place of the robot cleaner (200).

[0062] According to one embodiment of the present disclosure, a robot cleaner (200) can obtain information (captured images and LiDAR scan data) for generating a spatial map (10). The robot cleaner (200) can measure the depth of a plurality of points located at a specific height in a target area using a LiDAR sensor (250), and can photograph the target area using a camera (260).

[0063] The robot cleaner (200) can use the acquired information (information for generating a spatial map) to generate a spatial map (10) for a space (e.g., a user's house). Alternatively, a server (100) or a mobile terminal (300) can use the information acquired by the robot cleaner (200) to generate a spatial map (10). Alternatively, at least two or more devices among the robot cleaner (200), the server (100), or the mobile terminal (300) can cooperate to generate a spatial map (10). In the embodiments of the present disclosure, the robot cleaner (200) is described as performing all processes for generating a spatial map (10), but some or all of the processes may be performed by another device (the server (100) or the mobile terminal (300)).

[0064] According to one embodiment of the present disclosure, the robot cleaner (200) can perform operations based on the generated spatial map. For example, the robot cleaner (200) can perform cleaning tasks based on the generated spatial map.

[0065] The space map (10) generated by the robot cleaner (200) can be transmitted to the user's mobile terminal (300) via the server (100) and displayed on the screen of the mobile terminal (300). According to one embodiment of the present disclosure, an application for using and controlling the robot cleaner (200) can be installed on the mobile terminal (300), and when the user executes the application on the mobile terminal (300), the mobile terminal (300) can access the server (100), receive the space map (10), and display it on the screen.

[0066] According to embodiments of the present disclosure, the robot cleaner (200) can set the borders of the spatial map (10) to be as close as possible to the structure of an actual space, and for this purpose, not only the lidar scan data but also an image captured of the target area can be used. In the past, when the robot cleaner (200) created a spatial map using only the lidar scan data, spatial distortion could occur, which will be described in detail with reference to FIG. 2.

[0067] FIG. 2 is a diagram for comparing a spatial map generated according to one embodiment of the present disclosure with a spatial map generated using only existing lidar scan data. FIG. 2 illustrates spatial maps (10, 20) generated for the same space. The spatial map (10) on the left is generated according to one embodiment of the present disclosure, and the old spatial map (20) on the right is generated using only existing lidar scan data.

[0068] When a robot cleaner (200) generates a spherical space map (20) using only lidar scan data, distortion of the space may occur. This is because various objects (e.g., furniture, home appliances, etc.) may exist in the space, and the robot cleaner (200) cannot distinguish whether the point where a laser pulse is reflected is a wall of the space or an object using only the lidar scan data, and thus recognizes the object as a wall of the space and generates the spherical space map (20). In addition, according to one embodiment of the present disclosure, the robot cleaner (200) performs a lidar scan at a low height to acquire lidar scan data, and there is a high possibility that an object exists at the height at which the lidar scan is performed (e.g., a height close to the floor). Therefore, the spherical space map (200) generated based only on the lidar scan data may differ from the actual structure (floor plan) of the space.

[0069] Referring to FIG. 2, the spherical space map (20) has its border set so as not to include the first area (21) to the sixth area (26) where furniture or home appliances are located. On the other hand, the space map (10) generated according to one embodiment of the present disclosure has its border set so as to include all of the first area (21) to the sixth area (26) where furniture or home appliances are located, thereby reflecting the structure of the actual space as accurately as possible.

[0070] According to one embodiment of the present disclosure, the robot cleaner (200) can use images captured of target areas within a space (particularly, the first area (21) to the sixth area (26)) to create a space map (10). The robot cleaner (200) can first create a spherical space map (20) using only lidar scan data, and then secondarily correct the spherical space map (20) using captured images of the first area (21) to the sixth area (26) where an object is located, or can create the space map (10) from the beginning using both lidar scan data and captured images.

[0071] FIG. 3 is a diagram illustrating the configuration of a server included in a system for generating a spatial map according to one embodiment of the present disclosure. Referring to FIG. 3, a server (100) according to one embodiment of the present disclosure may include a communication interface (110), a processor (120), and a memory (130). However, the components of the server (100) are not limited to the above-described examples, and the server (100) may include more or fewer components than the above-described components. Some or all of the communication interface (110), the processor (120), and the memory (130) may be implemented in the form of a single chip. According to one embodiment of the present disclosure, the server (100) may be a home IoT server that controls IoT devices in the home, such as a robot vacuum cleaner (200).

[0072] The communication interface (110) is a configuration for transmitting and receiving signals (control commands, data, etc.) with an external device via wire or wirelessly, and may be implemented to include a communication chipset that supports various communication protocols. The communication interface (110) may receive a signal from the outside and output it to the processor (120), or transmit a signal output from the processor (120) to the outside. The server (100) may communicate with the robot cleaner (200) or the mobile terminal (300) via the communication interface (110).

[0073] The processor (120) controls a series of processes to enable the server (100) to operate according to the embodiments described below, and may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU or a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. For example, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0074] The processor (120) can record data in the memory (130) or read data stored in the memory (130), and in particular, process data according to predefined operation rules or artificial intelligence models by executing a program stored in the memory (130). Accordingly, the processor (120) can perform the operations described in the following embodiments, and operations described as being performed by the server (100) in the following embodiments can be viewed as being performed by the processor (120) unless otherwise specified.

[0075] The memory (130) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (130) may not exist separately and may be configured to be included in the processor (120). The memory (130) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. A program for performing operations according to embodiments described below may be stored in the memory (130). The memory (130) may also provide stored data to the processor (120) at the request of the processor (120).

[0076] The server (100) may receive a space map (10) from the robot cleaner (200) through the communication interface (110), and store the received space map (10) in the memory (130) or transmit it to the mobile terminal (300) through the communication interface (110). Alternatively, the server (100) may generate a space map (10) using information (lidar scan data and captured images) received from the robot cleaner (200) through the communication interface (110), and store the generated space map (10) in the memory (130) or transmit it to the mobile terminal (300) through the communication interface (110).

[0077] FIG. 4 is a diagram illustrating a configuration of a robot cleaner included in a system for generating a spatial map according to an embodiment of the present disclosure. Referring to FIG. 4, a robot cleaner (200) according to an embodiment of the present disclosure may include a communication interface (210), an input / output interface (220), a memory (230), a processor (240), a lidar sensor (250), a camera (260), and a driving unit (270). However, the components of the robot cleaner (200) are not limited to the above-described examples, and the robot cleaner (200) may include more or fewer components than the above-described components. Some or all of the communication interface (210), the input / output interface (220), the memory (230), and the processor (240) may be implemented in the form of a single chip.

[0078] The communication interface (210) is a configuration for transmitting and receiving signals (control commands and data, etc.) with an external device via wire or wirelessly, and may be implemented to include a communication chipset that supports various communication protocols. The communication interface (210) may receive a signal from the outside and output it to the processor (240), or transmit a signal output from the processor (240) to the outside. The robot cleaner (200) may communicate with the server (100) through the communication interface (210). According to one embodiment of the present disclosure, the robot cleaner (200) may transmit the lidar scan data acquired through the lidar sensor (250) described below, or the photographed images acquired through the camera (260) described below, to the server (100) or the mobile terminal (300) through the communication interface (210). Alternatively, the robot cleaner (200) may transmit a spatial map (10) generated using lidar scan data and captured images to a server (100) or mobile terminal (300) via a communication interface (210).

[0079] The input / output interface (220) may include an input interface (e.g., touch screen, hard button, microphone, etc.) for receiving control commands or information from a user, and an output interface (e.g., display panel, speaker, etc.) for displaying the results of execution of an operation according to the user's control or the status of the robot cleaner (200).

[0080] The memory (230) is a configuration for storing various programs or data, and may be configured as a storage medium such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD, or a combination of storage media. The memory (230) may not exist separately and may be configured to be included in the processor (240). The memory (230) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. A program for performing operations according to the embodiments described below may be stored in the memory (230). The memory (230) may also provide stored data to the processor (240) upon request of the processor (240).

[0081] The processor (240) controls a series of processes to operate the robot cleaner (200) according to the embodiments described below, and may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. For example, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0082] The processor (240) can record data in the memory (230) or read data stored in the memory (230), and in particular, process data according to predefined operation rules or artificial intelligence models by executing a program stored in the memory (230). Accordingly, the processor (240) can perform the operations described in the following embodiments, and the operations described as being performed by the robot cleaner (200) in the following embodiments can be regarded as being performed by the processor (240) unless otherwise specifically described.

[0083] The lidar sensor (250) can be used when the robot cleaner (200) scans the distance (depth) to a wall or object in the surrounding space. The robot cleaner (200) can measure the depth of multiple areas included in the space using the lidar sensor (250) and create a space map based on the measured depth values.

[0084] The camera (260) can be used when the robot cleaner (200) captures images of the surrounding space. The robot cleaner (200) can be equipped with the camera (260) to recognize objects (e.g., obstacles) in front. According to one embodiment of the present disclosure, the robot cleaner (200) can increase the accuracy of the spatial map by using an image captured of the surrounding space through the camera (260) when creating a spatial map.

[0085] The driving unit (270) is a component that provides the power necessary for the robot cleaner (200) to perform cleaning operations. The robot cleaner (200) can move within a space and perform cleaning operations (e.g., suction, etc.) due to the driving force provided by the driving unit (270). According to one embodiment, the driving unit (270) may include a motor, a battery, etc.

[0086] Below, embodiments in which a robot cleaner (200) creates a spatial map using lidar scan data and captured images are described in detail.

[0087] FIG. 5 is a diagram illustrating the purpose and main features of a method for generating a spatial map according to one embodiment of the present disclosure. FIG. 5 illustrates a captured image (500) of a target area included in space.

[0088] In the photographed image (500) of FIG. 5, a scan line (hereinafter, “first line”) (L1) corresponding to a first height and a scan line (hereinafter, “second line”) (L2) corresponding to a second height are displayed.

[0089] According to one embodiment of the present disclosure, the robot cleaner (200) can measure the depth (distance from the lidar sensor (250) to the plurality of points) of a plurality of spots located on a first line (L1) close to the floor surface by scanning the target area using the lidar sensor (250). Therefore, according to one embodiment of the present disclosure, the lidar scan data acquired by the robot cleaner (200) can include depth values ​​for a plurality of points located on the first line (L1).

[0090] Referring to FIG. 5, since objects (e.g., an island table, etc.) exist on the first line (L1), if the robot cleaner (200) creates a space map using only the lidar scan data acquired for the first line (L1), the boundaries of the space map may be determined along the surfaces of the objects, not the walls of the space. In other words, if the robot cleaner (200) creates a space map using only the lidar scan data acquired at a height close to the floor, distortion may occur in the boundaries of the space map due to objects within the space.

[0091] A robot cleaner (200) according to one embodiment of the present disclosure can acquire (estimate) depth values ​​for a plurality of points located on a second line (L2) of a height where no object exists, and determine the boundaries of a spatial map based on the acquired depth values. Accordingly, a spatial map generated by the robot cleaner (200) according to one embodiment of the present disclosure can accurately reflect the structure of an actual space.

[0092] The robot cleaner (200) can use a photographed image (500) of a target area to obtain depth values ​​for a plurality of points located on a second line (L2). A method by which the robot cleaner (200) obtains depth values ​​for a plurality of points located on a second line (L2) using the photographed image (500) will be described in detail below with reference to the drawings.

[0093] Below, the process of creating a space map by a robot cleaner (200) is described with reference to the flowcharts of FIGS. 6 to 15, and additionally described with reference to necessary drawings for each step.

[0094] 1. Description of the overall process of generating a spatial map (Figs. 6 to 9)

[0095] FIG. 6 is a flowchart illustrating a method for generating a spatial map according to one embodiment of the present disclosure.

[0096] In step 601, the robot cleaner (200) can acquire a photographic image of a target area within a space. Subsequently, in step 602, the robot cleaner (200) can acquire lidar scan data that scans the depth of the target area with respect to a preset first height.

[0097] Referring to FIG. 16, steps 601 and 602 will be described in detail. FIG. 16 is a diagram for explaining a step of acquiring lidar scan data for a target area in a process of generating a spatial map according to one embodiment of the present disclosure.

[0098] The first image (1610) of FIG. 16 corresponds to a photographed image obtained by the robot cleaner (200) by photographing the target area through the camera (260). According to one embodiment of the present disclosure, the robot cleaner (200) can scan the depth of the target area for a first height (h1) set in advance through the lidar sensor (250), and as a result, depth values ​​(d) of a plurality of points located on the first line (L1) of the first height (h1) 11 , d 12 , d 13 , ..., d 1k ) can be obtained. The second image (1620) of Fig. 16 shows the depth values ​​(d) included in the lidar scan data (LH1). 11 , d 12 , d 13 , ..., d 1k ) is shown as a first line (L1) connecting points corresponding to each other.

[0099] Returning to FIG. 6, at step 603, the robot cleaner (200) may perform object detection on the captured image to determine whether an object exists in the captured image. FIG. 17 is a diagram illustrating a step of detecting an object in a captured image of a target area during the process of generating a spatial map according to one embodiment of the present disclosure.

[0100] Referring to FIG. 17, the robot cleaner (200) performs object detection (object recognition) on a captured image (1700) of a target area, and as a result, objects (an island table and a trash can) (1701) can be detected.

[0101] If no object is detected in the target area, the robot cleaner (200) can generate an accurate spatial map using only the lidar scan data acquired through the lidar sensor (250). Therefore, according to one embodiment of the present disclosure, the robot cleaner (200) can perform object detection on the captured image (1700) of the target area, thereby determining whether to use the captured image (1700) when generating the spatial map.

[0102] Meanwhile, the type of object identified by the robot cleaner (200) by performing object recognition on the captured image in step 603 can be reflected when displaying the type of object on the space map later.

[0103] Returning to FIG. 6, in step 604, the robot cleaner (200) can determine whether an object is detected from the captured image. If an object is detected from the captured image, the robot cleaner (200) can proceed to step 605, where the robot cleaner (200) can generate a spatial map for the target area based on both the lidar scan data and the captured image. Conversely, if an object is not detected from the captured image, the robot cleaner (200) can generate a spatial map for the target area based only on the lidar scan data, the robot cleaner (200) can proceed to step 606.

[0104] Below, the detailed steps included in step 605 of FIG. 6 are described in detail with reference to the flowcharts of FIGS. 7 to 15.

[0105] FIG. 7 is a flowchart for explaining detailed steps included in step 605 of FIG. 6. Referring to FIG. 7, in step 701, the robot cleaner (200) can obtain depth values ​​for a plurality of points located at a second height different from a first height in the target area by using a captured image. In other words, the robot cleaner (200) can obtain information about the depth of the target area corresponding to the second height different from the first height. Referring to FIG. 5, the robot cleaner (200) can obtain depth values ​​for a plurality of points located at a second line (L2) of a second height different from the first height (the height of the first line (L1)). The robot cleaner (200) can determine a height where no object exists in the target area (or a height where no object is expected to exist) as the second height (optimal height), and a specific method by which the robot cleaner (200) determines the second height will be described in detail below with reference to FIGS. 10 to 12.

[0106] Below, step 701 is described in detail with reference to FIG. 8.

[0107] FIG. 8 is a flowchart illustrating detailed steps included in step 701 of FIG. 7. Referring to FIG. 8, in step 801, the robot cleaner (200) can acquire a depth image from a photographed image of a target area. Subsequently, in step 802, the robot cleaner (200) can perform depth calibration to match the scales of the lidar scan data and the depth image.

[0108] Referring to FIGS. 18 and 19, steps 801 and 802 of FIG. 8 will be described in detail. FIG. 18 is a diagram for explaining a step of acquiring a depth image from a photographed image and acquiring depth values ​​from the depth image in a process of generating a spatial map according to one embodiment of the present disclosure. FIG. 19 is a diagram for explaining a step of performing depth calibration using depth values ​​acquired from lidar scan data and a depth image in a process of generating a spatial map according to one embodiment of the present disclosure.

[0109] The first image (1810) of FIG. 18 corresponds to a captured image of a target area acquired by the robot cleaner (200), and the second image (1820) corresponds to a depth image acquired by the robot cleaner (200) from the captured image. The robot cleaner (200) can convert the captured image (the first image (1810)) into a depth image (the second image (1820)) by performing depth estimation on the captured image (the first image (1810)).

[0110] Before explaining the detailed steps included in step 802 of FIG. 8 with reference to FIG. 9, the reason why the robot cleaner (200) performs depth calibration will first be explained.

[0111] The robot cleaner (200) can only know the relative depth (distance) between multiple points included in the depth image from the depth image. According to one embodiment of the present disclosure, the robot cleaner (200) must be able to confirm (estimate) the absolute depth (distance) of points included in the depth image in order to create or correct a spatial map. Therefore, the robot cleaner (200) can perform calibration using lidar scan data in order to confirm (estimate) the absolute depth (distance) of points included in the depth image, and this is called depth calibration. In other words, depth calibration can also be viewed as an operation for converting the scale of the depth image according to the lidar scan data.

[0112] The robot cleaner (200) can extract absolute depth information from a depth image by performing calibration on relative depth information for multiple points included in the depth image. In other words, the robot cleaner (200) can obtain a scale factor for converting relative depth values ​​included in the depth image into absolute depth values ​​that can be obtained through a lidar scan by performing depth calibration.

[0113] A specific method for performing depth calibration using a depth image obtained from a photographed image and lidar scan data by a robot cleaner (200) is described in detail below.

[0114] Figure 9 is a flowchart for explaining the detailed steps included in step 802 of Figure 8.

[0115] In step 901, the robot cleaner (200) can obtain absolute depth values ​​for a plurality of points located at a first height from the lidar scan data. Referring to FIG. 19, the robot cleaner (200) can obtain absolute depth values ​​(d) for a plurality of points located at a first line (L1) of a first height (h1) displayed in a first image (1910) from the lidar scan data (LH1). 11 , d 12 , d 13 , ..., d 1k ) can be obtained.

[0116] In step 902, the robot cleaner (200) can obtain relative depth values ​​for a plurality of points (points corresponding to depth values ​​included in the lidar scan data) located at a first height (h1) from the depth image. Referring to FIG. 18, a first line (L1) of a first height (h1) is displayed in a third image (1830), and points corresponding to the lidar scan data (points corresponding to depth values ​​included in the lidar scan data) are located on the first line (L1). The robot cleaner (200) can obtain relative depth values ​​(d') for a plurality of points located at the first line (L1) from the depth image (the third image (1830)). 11 , d' 12 , d' 13 , ..., d' 1k ) can be obtained by obtaining relative depth information (DH1).

[0117] A specific method for the robot cleaner (200) to identify a plurality of points (points corresponding to lidar scan data) located on the first line (L1) on the depth image is described in detail below with reference to FIGS. 13 to 15.

[0118] At step 903, the robot cleaner (200) can obtain a scale factor (θ) for converting the relative depth value included in the depth image into an absolute depth value based on the absolute depth values ​​and the relative depth values.

[0119] Referring to Fig. 19, the robot cleaner (200) obtains absolute depth values ​​(d) from the lidar scan data (LH1). 11 , d 12 , d 13 , ..., d 1k ) and relative depth values ​​(d') obtained from relative depth information (DH1). 11 , d' 12 , d' 13 , ..., d' 1k ) can be used to perform depth calibration. In detail, the robot cleaner (200) can find a scale factor θ that minimizes the value of the cost function expressed by the following mathematical expression 1.

[0120]

[0121] Returning to FIG. 8, in step 803, the robot cleaner (200) may determine a height at which no object detected in the target area exists as the second height (h2), i.e., the optimal height. Step 803 will be described in detail below with reference to FIGS. 10 to 12. According to one embodiment of the present disclosure, step 803 may be performed first, followed by step 802.

[0122] In step 804, the robot cleaner (200) can acquire depth values ​​(absolute depth values) for multiple points located at a second height (h2) from the depth image by reflecting the depth calibration results. Step 804 will be described in detail with reference to FIG. 20.

[0123] FIG. 20 is a diagram illustrating a step of acquiring depth values ​​of points located at optimal heights during a process of generating a spatial map according to one embodiment of the present disclosure. As described above, it is assumed that the second height (h2) determined in step 803 corresponds to the optimal height. The robot cleaner (200) can acquire absolute depth values ​​for a plurality of points located on the second line (L2) of the second height (h2) from the depth image (2010) by either of the following two methods.

[0124] According to the first method, the robot cleaner (200) can obtain relative depth information (DH2) corresponding to the second height (h2) from the depth image (2010) and convert the relative depth information (DH2) into absolute depth information (LH2) using a scale factor (θ). The relative depth information (DH2) corresponding to the second height (h2) can be converted into relative depth values ​​(d') for a plurality of points for the second line (L2). 21 , d' 22 , d' 23 , ..., d' 2k ) is included. The robot cleaner (200) includes relative depth values ​​(d') for a plurality of points for the second line (L2). 21 , d' 22 , d' 23 , ..., d' 2k ) by a scale factor (θ), the absolute depth information (LH2) corresponding to the second height (h2) can be obtained. In other words, the absolute depth information (LH2) corresponding to the second height (h2) is the relative depth values ​​(d') for a plurality of points located on the second line (L2). 21 , d' 22 , d' 23 , ..., d' 2k ) and a scale factor (θ). The absolute depth information (LH2) corresponding to the second height (h2) is the absolute depth values ​​(d) for a plurality of points located on the second line (L2).21 , d 22 , d 23 , ..., d 2k ) is included.

[0125] According to the second method, the robot cleaner (200) converts the scale of the depth image (2010) using a scale factor (θ) and stores it, and then, when the second height (h2) is determined, it converts the depth values ​​(absolute depth values) (d) for a plurality of points included in the second line (L2) from the scale-converted depth image (2010). 21 , d 22 , d 23 , ..., d 2k ) can be obtained. That is, the robot cleaner (200) multiplies the relative depth values ​​of all points included in the depth image (2010) by a scale factor (θ), and then, when the second height (h2) is determined, the robot cleaner (200) obtains depth values ​​(absolute depth values) (d) for a plurality of points included in the second line (L2) from the scale-converted depth image (2010). 21 , d 22 , d 23 , ..., d 2k ) can be obtained.

[0126] Returning to FIG. 7 again, in step 702, the robot cleaner (200) can generate a spatial map based on the acquired depth values. That is, the robot cleaner (200) can generate a spatial map based on absolute depth values ​​for a plurality of points located at the second height (h2).

[0127] 2. Specific description of the method for determining the optimal height (second height) (step 803 of Fig. 8 - Figs. 10 to 12, Figs. 24 and 25)

[0128] As previously described, the robot cleaner (200) can determine the optimal height as a height at which no object exists in the target area by performing a dynamic height scan. Hereinafter, with reference to FIGS. 10 to 12, FIGS. 24 and 25, a method for the robot cleaner (200) to determine the optimal height (second height) according to one embodiment of the present disclosure will be described in detail.

[0129] For reference, in order to clearly explain the conditions and method of dynamic height scanning, Figures 24 and 25 are shown for target areas different from the target areas used in the description of the embodiments above.

[0130] FIG. 24 is a drawing for explaining an operation of determining an optimal height in a process of generating a spatial map according to one embodiment of the present disclosure.

[0131] Referring to Fig. 24, the robot cleaner (200) can acquire a depth image for a target area in the first step (2410) and convert the depth image into a point cloud in the second step (2420). In Fig. 24, a first line (L1) corresponding to the lidar scan data is displayed in the depth image of the first step (2410) and the point cloud of the second step (2420). Since an object (bed) is located on the first line (L1), the robot cleaner (200) needs to determine a height different from the height of the first line (L1) as the optimal height.

[0132] The robot cleaner (200) can determine a height where no object exists as an optimal height in the third step (2430). In the point cloud of the third step (2430), a plurality of scan lines are displayed, and the scan lines included in the first range (2431) are located at a height where an object exists, and the scan lines included in the second range (2432) are located at a height where an object does not exist. The robot cleaner (200) can find a second range (2432) where no object exists by using various conditions described below, and determine the height of any one of the scan lines included in the second range (2432) as the optimal height.

[0133] FIG. 25 is a diagram illustrating three conditions used to select an optimal height in the process of generating a spatial map according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the robot cleaner (200) can determine a height where no object exists as the optimal height based on at least one of the three conditions illustrated in FIG. 25. The first to third conditions illustrated in FIG. 25 may correspond to the flowcharts of FIGS. 10 to 12 , respectively. Although the three conditions are described separately below, the robot cleaner (200) may also determine the optimal height by considering two or more of the three conditions together. In summary, the robot cleaner (200) may determine the optimal height by considering only one of the three conditions illustrated in FIG. 25 , or may determine the optimal height by sequentially considering two or more of the three conditions.

[0134] According to the first condition, the robot cleaner (200) can detect an object (2515) from a captured image (2510) and determine an optimal height from a second range (2512) excluding a first range (2511) in which the detected object (2515) is located. According to one embodiment of the present disclosure, the robot cleaner (200) can set a plurality of scan lines at regular intervals in the second range (2512) (e.g., set scan lines at regular height intervals from below) and determine the height of any one of the plurality of scan lines as the optimal height. For example, the robot cleaner (200) can select any one of the plurality of scan lines included in the second range (2512) according to a preset condition (e.g., select the highest scan line) and determine the height of the selected scan line as the optimal height. Or, for example, the robot cleaner (200) may select one scan line from among a plurality of scan lines included in the second range (2512) according to at least one of the second condition or the third condition described later, and determine the height of the selected scan line as the optimal height.

[0135] According to the second condition, the robot cleaner (200) can set a plurality of scan lines of different heights on the point cloud (2520) (e.g., set scan lines at a certain height interval from below) and divide the range based on the number of junctions of the plurality of scan lines. The junction of the scan lines may mean the point where the scan line breaks. According to one embodiment of the present disclosure, the robot cleaner (200) can alleviate noise by approximating the scan line by applying the Ramer-Douglas-Peucker algorithm to each scan line and then calculating the number of junctions.

[0136] In the embodiment illustrated in FIG. 25, the robot cleaner (200) can classify a height including scan lines having three connection points into a first range (2521), and a height including scan lines having one connection point into a second range (2522). In general, if an object exists on a scan line, the number of connection points included in the scan line is likely to increase. Therefore, according to one embodiment of the present disclosure, the robot cleaner (200) can determine the height of any one scan line among the scan lines included in the second range (2522) having a small number of connection points as the optimal height. For example, the robot cleaner (200) can select any one of a plurality of scan lines included in the second range (2522) according to a preset condition (e.g., selecting the highest scan line), and determine the height of the selected scan line as the optimal height. Or, for example, the robot cleaner (200) may select one scan line from among a plurality of scan lines included in the second range (2522) according to at least one of the first condition described above or the third condition described below, and determine the height of the selected scan line as the optimal height.

[0137] According to one embodiment of the present disclosure, the robot cleaner (200) may determine the boundaries of a spatial map to correspond to each of a plurality of scan lines set in a point cloud (2520), calculate the number of connection points of the determined boundaries, and then divide the height into a plurality of ranges based on the number of connection points according to the method described above and determine an optimal height.

[0138] According to one embodiment of the present disclosure, the robot cleaner (200) may determine the height of the scan line with the minimum number of connection points among the plurality of scan lines set in the point cloud (2520) as the optimal height. Alternatively, according to one embodiment of the present disclosure, the robot cleaner (200) may determine the border of the spatial map to correspond to each of the plurality of scan lines set in the point cloud (2520), and determine the height corresponding to the border with the minimum number of connection points among the determined borders as the optimal height.

[0139] According to the third condition, the robot cleaner (200) can set multiple scan lines of different heights on the point cloud (2530) (e.g., set scan lines at a certain height interval from below) and determine an optimal height based on the area of ​​the cross-section corresponding to the multiple scan lines. The area of ​​the cross-section corresponding to the scan line may mean the area of ​​the cross-section formed by the outline along the scan line, with the point where the target area is photographed as one vertex. An example of the cross-section corresponding to each scan line is illustrated in FIG. 25.

[0140] In the embodiment illustrated in FIG. 25, the robot cleaner (200) can classify the height including scan lines having relatively small corresponding cross-sectional areas into a first range (2531), and classify the height including scan lines having relatively large corresponding cross-sectional areas into a second range (2532). In general, if an object exists on a scan line, the area of ​​the cross-section corresponding to the scan line is likely to be small. Therefore, according to one embodiment of the present disclosure, the robot cleaner (200) can determine the height of any one of the scan lines included in the second range (2532) having a large corresponding cross-sectional area as the optimal height. For example, the robot cleaner (200) can select any one of a plurality of scan lines included in the second range (2532) according to a preset condition (e.g., selecting the highest scan line), and determine the height of the selected scan line as the optimal height. Alternatively, for example, the robot cleaner (200) may select one scan line from among a plurality of scan lines included in the second range (2532) according to at least one of the first condition or the second condition described above, and determine the height of the selected scan line as the optimal height.

[0141] According to one embodiment of the present disclosure, the robot cleaner (200) may calculate the area of ​​a cross-section corresponding to each of a plurality of scan lines set in a point cloud (2530), and determine the height of the scan line corresponding to the cross-section with the largest area as the optimal height. Alternatively, according to one embodiment of the present disclosure, the robot cleaner (200) may calculate the area of ​​a spatial map for a plurality of heights based on a depth image, and determine the height corresponding to the largest area among the calculated areas as the optimal height.

[0142] FIGS. 10 to 12 are flowcharts illustrating detailed steps included in step 803 of FIG. 8. FIG. 10 corresponds to the first condition of FIG. 25, FIG. 11 corresponds to the second condition of FIG. 25, and FIG. 12 corresponds to the third condition of FIG. 25. Accordingly, the contents of the three conditions described above with reference to FIG. 25 may be equally applied to the embodiments of FIGS. 10 to 12.

[0143] Referring to FIG. 10, in step 1001, the robot cleaner (200) can identify a range of heights in which no object exists in the target area by analyzing the captured image. In step 1002, the robot cleaner (200) can determine one of the identified ranges as the second height (optimal height).

[0144] Referring to FIG. 11, in step 1101, the robot cleaner (200) can determine the boundaries of a spatial map for a target area for a first height and a plurality of other heights based on a depth image. In step 1102, the robot cleaner (200) can determine the height corresponding to the boundary with the minimum number of connection points among the determined boundaries as the second height (optimal height).

[0145] Referring to FIG. 12, in step 1201, the robot cleaner (200) can calculate the areas of a spatial map for a target area for a first height and a plurality of other heights based on a depth image. In step 1202, the robot cleaner (200) can determine the height corresponding to the largest area among the calculated areas as the second height (optimal height).

[0146] 3. A specific description of a method for obtaining relative depth values ​​(relative depth values ​​corresponding to lidar scan data) for a plurality of points located at a first height on a depth image (step 902 of FIG. 9 - FIGS. 13 to 15 and FIGS. 21 to 23)

[0147] As explained above, to perform depth calibration, relative depth values ​​on the depth image are required that correspond to the absolute depth values ​​contained in the lidar scan data.

[0148] Hereinafter, with reference to FIGS. 13 to 15 and FIGS. 21 to 23, a method for a robot cleaner (200) to obtain relative depth values ​​(relative depth values ​​corresponding to the lidar scan data) for a plurality of points located at a first height (height corresponding to the lidar scan data) on a depth image according to an embodiment of the present disclosure will be described in detail. (Description of detailed steps included in step 902 of FIG. 9) Although three methods are introduced in the present disclosure, the robot cleaner (200) can obtain depth values ​​(relative depth values) corresponding to the lidar scan data from a depth image by various other methods.

[0149] FIG. 13 is a flowchart for explaining detailed steps included in step 902 of FIG. 9. Referring to FIG. 13, in step 1301, the robot cleaner (200) can obtain a transformation matrix for position matching between the lidar scan data and the captured image. The transformation matrix is ​​a matrix for converting points corresponding to depth values ​​included in the lidar scan data into positions (e.g., pixel positions) on the captured image. The transformation matrix may be obtained through factory calibration during the production stage in consideration of the positions of the lidar sensor (250) and the camera (260) of the robot cleaner (200), or may be obtained through a calibration process using a specific pattern after the production of the robot cleaner (200).

[0150] In step 1302, the robot cleaner (200) can use a transformation matrix to transform the positions of points corresponding to depth values ​​included in the lidar scan data into positions (pixel positions) on a depth image. In other words, the robot cleaner (200) can use the transformation matrix to determine which positions on the depth image a plurality of points located at the first height correspond to. Subsequently, in step 1303, the robot cleaner (200) can obtain relative depth values ​​corresponding to points at the transformed positions from the depth image.

[0151] Below, steps 1302 and 1303 are described in detail with reference to FIG. 21.

[0152] FIG. 21 is a diagram for explaining an operation of obtaining depth values ​​corresponding to lidar scan data from a depth image using a transformation matrix in a process of generating a spatial map according to one embodiment of the present disclosure.

[0153] Referring to Fig. 21, a first line (L1) of a first height (h1) corresponding to the lidar scan data (LH1) is displayed on the first image (2110), which is a photographed image of the target area. The points displayed on the first line (L1) are depth values ​​(d) included in the lidar scan data (LH1). 11 , d 12 , d 13 , ..., d 1k ) indicates the corresponding point.

[0154] According to one embodiment of the present disclosure, the robot cleaner (200) uses a transformation matrix (TM) for position matching between the lidar scan data (LH1) and the captured image (or a depth image converted therefrom) to obtain depth values ​​(d) included in the lidar scan data (LH1). 11 , d 12 , d 13 , ..., d 1k) can be converted into a position on the second image (2120), which is a depth image. The second image (2120) is a depth image converted from the first image (2110).

[0155] Note that in Fig. 21, for convenience of explanation, the depth values ​​(d) of the lidar scan data (LH1) are shown on the first image (2110), which is a photographed image. 11 , d 12 , d 13 , ..., d 1k ) are displayed, but according to one embodiment, the robot cleaner (200) actually uses the depth values ​​(d) of the lidar scan data (LH1) without using the transformation matrix. 11 , d 12 , d 13 , ..., d 1k ) cannot be identified on the first image (2110). That is, the robot cleaner (200) applies the lidar scan data (LH1) to the transformation matrix (TM), thereby obtaining the depth values ​​(d) of the lidar scan data (LH1) on the depth image (second image (2120)). 11 , d 12 , d 13 , ..., d 1k ) can be checked for the locations of points corresponding to the points (e.g. pixel locations).

[0156] In the second image (2120) of Fig. 21, a first line (L1) corresponding to the lidar scan data (LH1) is displayed. The points displayed on the first line (L1) of the second image (2120) are depth values ​​(d) included in the lidar scan data (LH1). 11 , d 12 , d 13 , ..., d 1k ) indicates a point corresponding to the first line (L1) of the second image (2120). The robot cleaner (200) can obtain the positions of the points indicated on the first line (L1) of the second image (2120) using the transformation matrix.

[0157] The robot cleaner (200) contains depth values ​​(d) included in the lidar scan data (LH1). 11 , d 12 , d 13 , ..., d 1k ) are converted to positions on the depth image (second image (2120)), then the relative depth values ​​(d') corresponding to the converted positions are 11 , d' 12 , d' 13 , ..., d' 1k ) can be extracted from the depth image (second image (2120)).

[0158] FIG. 14 is a flowchart for explaining detailed steps included in step 902 of FIG. 9, and FIG. 22 is a diagram for explaining an operation of obtaining depth values ​​corresponding to lidar scan data from a depth image through semantic segmentation in a process of generating a spatial map according to one embodiment of the present disclosure.

[0159] Referring to Fig. 14, in step 1401, the robot cleaner (200) can perform semantic segmentation on the captured image. Referring to Fig. 22, the result of performing semantic segmentation on the first image (2210), which is a captured image of the target area, by the robot cleaner (200) is shown in the second image (2220). The second image (2220) is divided into three areas: a ceiling, a wall, and a floor.

[0160] In step 1402, the robot cleaner (200) can identify the boundary between the wall and the floor in the captured image. Subsequently, in step 1403, the robot cleaner (200) can obtain relative depth values ​​corresponding to a plurality of points located on the boundary.

[0161] Referring to Fig. 22, a boundary line (2221) between a wall and a floor is indicated in the second image (2220). Since the robot cleaner (200) performs a lidar scan at a height close to the floor, the line on which the robot cleaner (200) performs a lidar scan is likely to be close to the boundary line (2221) between the wall and the floor. Therefore, according to one embodiment of the present disclosure, the robot cleaner (200) identifies the boundary line (2221) between the wall and the floor on the captured image (the second image (2220)) on which semantic segmentation has been performed, and the relative depth values ​​(d') corresponding to a plurality of points located on the identified boundary line (2221) 11 , d' 12 , d' 13 , ..., d' 1k ) can be obtained from the depth image (third image (2230)). In detail, the robot cleaner (200) finds a boundary line (2231) corresponding to the boundary line (2221) indicated in the second image (2220) in the third image (2230), and obtains relative depth values ​​(d') corresponding to a plurality of points located on the boundary line (2231) indicated in the third image (2230). 11 , d' 12 , d' 13 , ..., d' 1k ) can be obtained.

[0162] FIG. 15 is a flowchart for explaining detailed steps included in step 902 of FIG. 9, and FIG. 23 is a diagram for explaining an operation of obtaining depth values ​​corresponding to lidar scan data from a depth image through elbow point detection in a process of generating a spatial map according to one embodiment of the present disclosure.

[0163] Referring to FIG. 15, in step 1501, the robot cleaner (200) can extract multiple pixel columns from the depth image. Subsequently, in step 1502, the robot cleaner (200) can determine (detect) an elbow point for each of the pixel columns. At this time, the elbow point may refer to a point where the degree to which the depth value changes in the longitudinal direction of the pixel column changes below a preset standard.

[0164] Referring to FIG. 23, a first image (2310) displays a plurality of pixel columns (2311, 2312, 2313, 2314). According to one embodiment of the present disclosure, the robot cleaner (200) can extract pixel columns (2311, 2312, 2313, 2314) at regular intervals from a depth image (the first image (2310)). At this time, the distance between the extracted pixel columns (2311, 2312, 2313, 2314) may be the same as the distance between points corresponding to the depth values ​​of the lidar scan data.

[0165] The robot cleaner (200) can detect an elbow point for each of the pixel columns (2311, 2312, 2313, 2314) on the first image (2310). For example, the robot cleaner (200) can determine the degree to which the depth values ​​change by comparing the depth values ​​of the pixels in the longitudinal direction of the first pixel column (2311), and determine the point at which the degree to which the depth values ​​change is below a preset standard as the first elbow point (EP1). The robot cleaner (200) can also detect elbow points (EP2-EPk) from the second pixel column (2312) to the k-th pixel column (2314) in the same manner.

[0166] The reason why the robot cleaner (200) detects the elbow point is because there is a high possibility that the elbow point is located at the boundary line between the wall and the floor, and as described above, the line along which the lidar scan is performed up to the robot cleaner (200) is likely to be close to the boundary line between the wall and the floor. Therefore, in one embodiment of the present disclosure, it is assumed that the elbow points (EP1, EP2, EP3, EPk) detected from the pixel rows (2311, 2312, 2313, 2314) are located on the first line (L1) corresponding to the lidar scan data.

[0167] In step 1503, the robot cleaner (200) can obtain relative depth values ​​corresponding to points located at elbow points. Referring to FIG. 23, a first line (L1) corresponding to the lidar scan data is displayed on the second image (2320), and elbow points (EP1, EP2, EP3, EPk) are displayed on the first line (L1). The robot cleaner (200) obtains relative depth values ​​(d') corresponding to the elbow points (EP1, EP2, EP3, EPk) from the depth image (the second image (2320)). 11 , d' 12 , d' 13 , ..., d' 1k ) can be extracted.

[0168] The relative depth information (DH1) acquired by the robot cleaner (200) by the method described with reference to FIGS. 13 to 15 and FIGS. 21 to 23 corresponds to the relative depth information (DH1) in FIG. 18 described above. That is, the robot cleaner (200) can acquire depth values ​​(relative depth values) on the depth image for points corresponding to the lidar scan data by the three methods described above.

[0169] 4. How to display detected objects at their exact locations on a spatial map

[0170] According to one embodiment of the present disclosure, the robot cleaner (200) can display an object detected from a captured image on a spatial map. For example, the robot cleaner (200) can estimate the size, shape, type, etc. of an object by performing object recognition on the captured image, and display the estimated result on the spatial map.

[0171] According to one embodiment of the present disclosure, a specific method for a robot cleaner (200) to determine the location of an object and the area occupied by the object on a spatial map is as follows. The robot cleaner (200) may perform the following operations after steps 605 and 606 of FIG. 6.

[0172] The robot cleaner (200) can determine the area where the detected object is located by comparing the boundary of the spatial map determined solely based on the lidar scan data with the boundary of the spatial map determined based on both the lidar scan data and the captured image. Subsequently, the robot cleaner (200) can display the object in the determined area. At this time, the robot cleaner (200) can also display the shape or type of the object estimated from the captured image.

[0173] Referring to FIG. 2, the space map (10) on the left corresponds to a space map determined based on both the lidar scan data and the captured image, and the spherical space map (20) on the right corresponds to a space map determined based only on the lidar scan data. The robot cleaner (200) can determine that an object exists in each of the first area (21) to the sixth area (26) by comparing the border of the space map (10) on the left with the border of the spherical space map (20) on the right. Accordingly, the robot cleaner (200) can display the object in each of the first area (21) to the sixth area (26) on the space map (10). At this time, the robot cleaner (200) can also display the shape or type of the object estimated from the captured image.

[0174] By generating a spatial map according to the embodiments described above, it is expected that a spatial map that accurately reflects the structure of a space can be generated even when objects (e.g., furniture, home appliances, etc.) exist in the space.

[0175] A method for generating a spatial map according to one embodiment of the present disclosure may include the steps of: acquiring a captured image of a target area in space; acquiring LiDAR scan data that scans a depth of the target area with respect to a first height set in advance; performing object detection on the captured image; and, if an object is not detected from the captured image, generating a spatial map for the target area based on the LiDAR scan data; and if an object is detected from the captured image, generating a spatial map for the target area based on both the LiDAR scan data and the captured image.

[0176] A method for generating a spatial map according to one embodiment of the present disclosure may include a step of obtaining a captured image of a target area in space, a step of obtaining LiDAR scan data that scans a depth of the target area with respect to a first height, a step of performing object detection on the captured image, and a step of generating a spatial map of the target area based on the LiDAR scan data and the captured image according to the object detection.

[0177] According to one embodiment, when an object is detected from the captured image, the step of generating the spatial map may include the step of obtaining depth values ​​for a plurality of spots located at a second height different from the first height in the target area using the captured image, and the step of generating the spatial map based on the obtained depth values.

[0178] According to one embodiment, the step of obtaining the depth values ​​may include the steps of obtaining a depth image from the photographed image, performing depth calibration to match a scale of the lidar scan data and the depth image, determining a height at which the detected object does not exist in the target area as the second height, and reflecting the depth calibration result, obtaining depth values ​​for a plurality of points located at the second height from the depth image.

[0179] According to one embodiment, the step of performing the depth calibration may include the step of obtaining absolute depth values ​​for a plurality of points located at the first height from the lidar scan data, the step of obtaining relative depth values ​​for a plurality of points located at the first height from the depth image, and the step of obtaining a scale factor for converting a relative depth value included in the depth image into an absolute depth value based on the obtained absolute depth values ​​and relative depth values.

[0180] According to one embodiment, the step of obtaining the scale factor may determine the value of the scale factor by using a cost function such that a difference between absolute depth values ​​for a plurality of points located at the first height and values ​​obtained by multiplying relative depth values ​​for a plurality of points located at the first height by the scale factor is minimized.

[0181] According to one embodiment, the step of determining a height at which the detected object does not exist as the second height may include the step of analyzing the photographed image to determine a range of heights at which the object does not exist in the target area, and the step of determining one height among the determined ranges as the second height.

[0182] According to one embodiment, the step of determining a height at which the detected object does not exist as the second height may include the step of determining borders of a spatial map for the target area for a plurality of heights different from the first height based on the depth image, and the step of determining a height corresponding to a border with the minimum number of junctions among the determined borders as the second height.

[0183] According to one embodiment, the step of determining a height at which the detected object does not exist as the second height may include the step of calculating areas of a spatial map for the target area for a plurality of heights different from the first height based on the depth image, and the step of determining a height corresponding to a maximum area among the calculated areas as the second height.

[0184] According to one embodiment, the step of obtaining relative depth values ​​for a plurality of points located at the first height from the depth image may include the step of obtaining a transformation matrix for position matching between the lidar scan data and the photographed image, the step of converting positions of points corresponding to depth values ​​included in the lidar scan data into positions on the depth image using the transformation matrix, and the step of obtaining relative depth values ​​corresponding to points at the converted positions from the depth image.

[0185] According to one embodiment, the step of obtaining relative depth values ​​for a plurality of points located at the first height from the depth image may include the steps of performing semantic segmentation on the captured image, the step of confirming a boundary between a wall and a floor in the captured image, and the step of obtaining relative depth values ​​corresponding to a plurality of points located on the boundary.

[0186] According to one embodiment, the step of obtaining relative depth values ​​for a plurality of points located at the first height from the depth image includes the steps of extracting a plurality of pixel columns from the depth image, determining an elbow point for each of the pixel columns, and obtaining relative depth values ​​corresponding to points located at the determined elbow points, wherein the elbow point may be a point at which a degree of change in a depth value in a longitudinal direction of the pixel column is changed to be below a preset constant reference.

[0187] According to one embodiment, when an object is detected from the captured image, the method may further include a step of determining an area where the detected object is located by comparing a border of the spatial map determined based only on the lidar scan data with a border of the spatial map determined based on both the lidar scan data and the captured image, and a step of displaying the detected object in the determined area on the spatial map.

[0188] An electronic device for generating a spatial map according to one embodiment of the present disclosure comprises: a memory storing a program for generating a spatial map; and at least one processor, wherein the at least one processor executes the program to obtain a captured image of a target area in space, obtain LiDAR scan data obtained by scanning a depth of the target area with respect to a first height set in advance, perform object detection on the captured image, and then, if an object is not detected from the captured image, generate a spatial map for the target area based on the LiDAR scan data, and if an object is detected from the captured image, generate a spatial map for the target area based on both the LiDAR scan data and the captured image.

[0189] An electronic device for generating a spatial map according to one embodiment of the present disclosure comprises: a memory storing a program for generating a spatial map; and at least one processor, wherein the at least one processor executes the program stored in the memory to obtain a captured image of a target area in space, obtain LiDAR scan data obtained by scanning a depth of the target area with respect to a first height, perform object detection on the captured image, and then generate a spatial map of the target area based on the LiDAR scan data and the captured image based on the object detection.

[0190] According to one embodiment, when an object is detected from the captured image, the at least one processor may, in generating the spatial map, obtain depth values ​​for a plurality of spots located at a second height different from the first height in the target area using the captured image, and then generate the spatial map based on the obtained depth values.

[0191] According to one embodiment, the at least one processor may acquire a depth image from the captured image, perform depth calibration to match a scale of the lidar scan data and the depth image, determine a height at which the detected object does not exist in the target area as the second height, and then reflect the depth calibration result to acquire depth values ​​for a plurality of points located at the second height from the depth image.

[0192] According to one embodiment, the at least one processor, in performing the depth calibration, may obtain absolute depth values ​​for a plurality of points located at the first height from the lidar scan data, obtain relative depth values ​​for a plurality of points located at the first height from the depth image, and then obtain a scale factor for converting a relative depth value included in the depth image into an absolute depth value based on the obtained absolute depth values ​​and relative depth values.

[0193] According to one embodiment, the at least one processor may determine a value of the scale factor by using a cost function to obtain the scale factor, such that a difference between absolute depth values ​​for a plurality of points located at the first height and values ​​obtained by multiplying relative depth values ​​for a plurality of points located at the first height by the scale factor is minimized.

[0194] According to one embodiment, the at least one processor may determine a height at which the detected object does not exist as the second height by analyzing the captured image, confirming a range of heights at which the object does not exist in the target area, and then determining any one height among the confirmed ranges as the second height.

[0195] According to one embodiment, the at least one processor may determine a height at which the detected object does not exist as the second height by determining borders of a spatial map for the target area for a plurality of heights different from the first height based on the depth image, and then determining a height corresponding to a border with the minimum number of junctions among the determined borders as the second height.

[0196] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, "application" and "program" may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code, including source code, object code, and executable code. "Computer-readable medium" may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0197] Additionally, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, and removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0198] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0199] The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present disclosure. For example, suitable results can be achieved even if the described techniques are performed in a different order than the described method, and / or components of the systems, structures, devices, circuits, etc. described are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form.

[0200] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A step of obtaining a captured image of a target area in space; A step of acquiring LiDAR scan data that scans the depth of the target area for the first height; A step of performing object detection on the above captured image; and A method comprising the step of generating a spatial map for the target area based on the lidar scan data and the photographed image according to the object detection.

2. In paragraph 1, Based on the object being detected from the above captured image, The steps for generating the above spatial map are: A step of obtaining depth values ​​for a plurality of spots located at a second height different from the first height in the target area based on the above-described captured image; and A method characterized by comprising a step of generating the spatial map based on the acquired depth values.

3. In either of paragraphs 1 and 2, The steps for obtaining the above depth values ​​are: A step of obtaining a depth image from the above photographed image; A step of performing depth calibration to match the scale of the above lidar scan data and the above depth image; A step of determining a height at which the detected object is not detected in the target area as the second height; and A method characterized by comprising a step of obtaining depth values ​​for a plurality of points located at the second height from the depth image based on the depth calibration result.

4. In any one of paragraphs 1 to 3, The steps for performing the above depth calibration are: A step of obtaining absolute depth values ​​for a plurality of points located at the first height from the above lidar scan data; A step of obtaining relative depth values ​​for a plurality of points located at the first height from the depth image; and A method characterized by comprising a step of obtaining a scale factor for converting a relative depth value included in the depth image into an absolute depth value based on the obtained absolute depth values ​​and relative depth values.

5. In any one of paragraphs 1 to 4, The steps for obtaining the above scale factor are: A method characterized in that the value of the scale factor is determined such that the difference between the absolute depth values ​​for a plurality of points located at the first height and the values ​​obtained by multiplying the relative depth values ​​for a plurality of points located at the first height by the scale factor is minimized using a cost function.

6. In any one of paragraphs 1 to 5, The step of determining the second height is: A step of determining a range of heights in which the object is not detected in the target area based on the above-mentioned captured image; and A method characterized by comprising a step of determining one height among the above-mentioned confirmed ranges as the second height.

7. In any one of paragraphs 1 to 6, The step of determining the second height is: A step of determining borders of a spatial map for the target area for a plurality of heights different from the first height based on the depth image; and A method characterized by including a step of determining a height corresponding to a border having a minimum number of junctions among the determined borders as the second height.

8. In any one of paragraphs 1 to 7, The step of determining the second height is: A step of calculating areas of a spatial map for the target area for a plurality of heights different from the first height based on the depth image; and A method characterized by including a step of determining a height corresponding to a maximum area among the above-described calculated areas as the second height.

9. In any one of paragraphs 1 to 8, Based on the object detected from the above captured image, A step of determining an area where the detected object is located by comparing a border of the spatial map determined based only on the lidar scan data with a border of the spatial map determined based on both the lidar scan data and the captured image; and A method characterized by further comprising the step of displaying the detected object in the determined area on the space map.

10. A non-transitory computer-readable storage medium having recorded thereon a program for performing the method of any one of clauses 1 to 9 on a computer.

11. In an electronic device (100) for generating a spatial map, A memory (130) in which a program for generating a spatial map is stored; and comprising at least one processor (120), The above at least one processor (120) executes a program stored in the memory (130), Obtain a captured image of the target area in space, Obtain LiDAR scan data that scans the depth of the target area for the first height, After performing object detection on the above captured image, An electronic device that generates a spatial map for the target area based on both the lidar scan data and the captured image according to the object detection.

12. In paragraph 11, The at least one processor (120) generates the spatial map, Based on the object being detected from the above captured image, Based on the above-mentioned captured image, depth values ​​for a plurality of spots located at a second height different from the first height in the target area are obtained, An electronic device characterized by generating the spatial map based on the acquired depth values.

13. In either of paragraphs 11 and 12, The at least one processor (120) obtains the depth values, Obtain a depth image from the above captured image, Depth calibration is performed to match the scale of the above lidar scan data and the depth image, After determining the height at which the detected object is not detected in the target area as the second height, An electronic device characterized in that, based on the depth calibration result, depth values ​​for a plurality of points located at the second height are obtained from the depth image.

14. In any one of paragraphs 11 to 13, The at least one processor (120) performs the depth calibration, Obtain absolute depth values ​​for a plurality of points located at the first height from the above lidar scan data, After obtaining relative depth values ​​for a plurality of points located at the first height from the depth image, An electronic device characterized in that it obtains a scale factor for converting a relative depth value included in the depth image into an absolute depth value based on the acquired absolute depth values ​​and relative depth values.

15. In any one of paragraphs 11 to 14, The at least one processor (120) obtains the scale factor, An electronic device characterized in that the value of the scale factor is determined such that the difference between the absolute depth values ​​for a plurality of points located at the first height and the values ​​obtained by multiplying the relative depth values ​​for the plurality of points located at the first height by the scale factor is minimized using a cost function.

Images