Terminal device and method for generating map by terminal device

Abstract

A terminal device for generating a 3D map is disclosed. A processor of the present device: stores, in a memory, an image of a space where a robot cleaner is located, the image being captured by a camera; when data of a map for an indoor space generated by the robot cleaner is received through a communication unit, generates a 3D map for the indoor space on the basis of the captured image and the map generated by the robot cleaner; and controls a display to display the generated 3D map.

Description

Terminal devices and methods for creating maps of terminal devices

[0001] The present disclosure relates to a terminal device capable of generating a 3D map and a method for generating the map.

[0002] A robot vacuum cleaner is an electronic device that automatically cleans indoor spaces while moving around. Recently, robot vacuum cleaners that utilize cleaning maps for efficient and accurate cleaning have been developed and distributed.

[0003] Robot vacuum cleaners can create maps of indoor spaces using various sensors. However, because robot vacuums typically clean while attached to the floor, there is a significant difference between the maps generated by the robot vacuum's own sensors and the indoor space as seen from the user's perspective. Consequently, the usability and user satisfaction of maps generated by robot vacuums have been low.

[0004] According to the present disclosure, a terminal device according to at least one embodiment includes a communication unit, a memory, a camera, a display, and a processor for performing communication with a robot cleaner.

[0005] The processor stores an image captured by the camera for a space where the robot cleaner is located in the memory, and when data for a map generated by the robot cleaner for the space is received through the communication unit, the processor generates a 3D map for the space based on the captured image and the map generated by the robot cleaner, and controls the display to display the generated 3D map.

[0006] Meanwhile, the robot vacuum cleaner includes a driving unit for driving the robot vacuum cleaner, a communication unit for communicating with a terminal device, at least one sensor, a memory, and a processor.

[0007] The processor receives a sensing value sensed by the at least one sensor while controlling the driving unit to drive the robot cleaner, creates a map of a space where the robot cleaner is located based on the sensing value, stores the map in the memory, transmits data about the map to a terminal device through the communication unit, and receives data about a 3D map created by matching the map with a 3D point cloud map in the terminal device through the communication unit and stores the data in the memory.

[0008] Meanwhile, a method for generating a map of a terminal device including a 3D camera according to at least one embodiment of the present disclosure includes a step of generating a 3D point cloud map for a space in which a robot cleaner is located based on photographing data of the 3D camera for the space in which the robot cleaner is located, a step of receiving data for a map generated by the robot cleaner for the space, a step of generating a 3D map for the space by matching the 3D point cloud map with the map generated by the robot cleaner, and a step of displaying the generated 3D map.

[0009] Meanwhile, according to one embodiment of the present disclosure, a terminal device provides a non-transitory computer-readable recording medium storing a computer program for generating a 3D map.

[0010] The computer program may sequentially perform the steps of: generating a 3D point cloud map for a space where the robot cleaner is located based on the captured data of the 3D camera for the space where the robot cleaner is located; receiving data for a map generated by the robot cleaner for the space; generating a 3D map for the space by matching the 3D point cloud map with the map generated by the robot cleaner; and displaying the generated 3D map.

[0011] FIG. 1 is a drawing for explaining a method for generating a 3D map in a terminal device and a robot cleaner according to an embodiment of the present disclosure.

[0012] FIG. 2 is a block diagram showing the configuration of a terminal device according to an embodiment of the present disclosure.

[0013] FIG. 3 is a drawing showing an example of a 3D point cloud map generated by a terminal device according to an embodiment of the present disclosure.

[0014] FIG. 4 is a diagram showing a map generated by a robot vacuum cleaner according to an embodiment of the present disclosure.

[0015] FIG. 5 and FIG. 6 are diagrams for explaining a process of extracting floor information during 3D map generation of a terminal device according to an embodiment of the present disclosure.

[0016] FIG. 7 and FIG. 8 are drawings for explaining various examples of 3D maps generated by a terminal device according to an embodiment of the present disclosure.

[0017] FIG. 9 is a drawing for explaining a process of displaying a graphic object on a 3D map generated by a terminal device according to an embodiment of the present disclosure.

[0018] FIG. 10 is a drawing for explaining semantic information on a 3D map of a terminal device according to an embodiment of the present disclosure.

[0019] Figure 11 is a block diagram for explaining the configuration of a robot vacuum cleaner.

[0020] FIG. 12 is a flowchart of a map generation method of a terminal device according to an embodiment of the present disclosure.

[0021] FIG. 13 is a sequence diagram showing the order in which a terminal device and a robot cleaner operate according to an embodiment of the present disclosure.

[0022] FIG. 14 is a flowchart showing the overall process of a terminal device according to an embodiment of the present disclosure.

[0023] The terms used in the various embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of this disclosure, rather than simply their names.

[0024] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0025] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0026] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0027] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0028] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0029] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0030] In this disclosure, the term user may refer to a person using an electronic device or a device used by the person.

[0031] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0032] FIG. 1 is a drawing for explaining the operation of a terminal device and a robot vacuum cleaner according to one embodiment of the present disclosure.

[0033] Referring to FIG. 1, a user (10) can take a picture of the interior of an indoor space using his / her terminal device (100).

[0034] Separately, the robot vacuum cleaner (200) can create a map while moving within a space.

[0035] A robot cleaner (200) is a device that performs cleaning by driving itself using a drive motor and wheels. The robot cleaner (200) can sense the location within a space, the characteristics of the floor surface, the shape of the space, the position and shape of objects (such as home appliances or furniture) within the space, etc. by using various sensors such as a lidar sensor, an infrared sensor, an image sensor, and an ultrasonic sensor. In Fig. 1, a case where the robot cleaner (200) operates within a general household environment is illustrated, but the robot cleaner (200) can be used in various environments such as offices, inside buildings, factories, and public offices. Hereinafter, it is collectively referred to as a space.

[0036] A user (10) can take pictures of the inside of a space where a robot cleaner (200) is located by rotating the terminal device (100) while activating the camera of the terminal device (100).

[0037] The terminal device (100) can be implemented as various types of devices capable of taking pictures, such as a mobile phone, tablet PC, laptop PC, camera, PDA, or electronic notebook.

[0038] The terminal device (100) according to one embodiment of the present disclosure may be implemented in various forms and, depending on the form, may independently provide various services. Furthermore, the terminal device (100) may also provide services that link with other terminal devices.

[0039] The terminal device (100) can generate a three-dimensional image of the interior of an indoor space by connecting image frames continuously captured by the camera while rotating. Since these captured images represent the entire indoor space, they can be utilized as a map. The terminal device (100) can generate a 3D map of the corresponding space based on the captured images and the map generated by the robot cleaner (200).

[0040] For example, the terminal device (100) may create a 3D map by combining a map created by a robot cleaner (200) with the floor surface portion of a photographed image that three-dimensionally constructs the interior of an indoor space.

[0041] As another example, the terminal device (100) may generate a 3D point cloud map from a captured image and combine the 3D point cloud map with a map generated by a robot cleaner (200) to generate a 3D map.

[0042] The operation of creating a 3D map using the map created by the terminal device (100) and the map created by the robot cleaner (200) can be performed by the terminal device (100), but is not necessarily limited thereto, and the robot cleaner (200) can also receive data from the terminal device (100) and match it with the map created by the robot cleaner (200) to create a 3D map. Alternatively, an external device (e.g., a server device, etc.) other than the terminal device (100) and the robot cleaner (200) can receive the maps created by the terminal device (100) and the robot cleaner (200), respectively, and create a 3D map.

[0043] Since the robot cleaner (200) moves in close contact with the floor due to its nature, the map generated by the robot cleaner (200) is generated based on data sensed from a point of view below the robot cleaner's (200) height. In contrast, the user captures the same space from his or her own eye level. Therefore, by matching the maps generated by the terminal device (100) and the robot cleaner (200), the corresponding space can be expressed in three dimensions.

[0044] The generated 3D map can be utilized in a variety of ways, as detailed in the sections below.

[0045] FIG. 2 is a block diagram showing the configuration of a terminal device (100) according to one embodiment of the present disclosure.

[0046] The terminal device (100) includes a communication unit (110), a processor (120), a memory (130), a display (140), and a camera (150).

[0047] The communication unit (110) is a configuration for performing communication with an external device. The communication unit (110) may include a wired or wireless input / output interface (or input / output terminal) according to various standards. For example, the communication unit (110) may include various interfaces such as HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), AP-based Wi-Fi (Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc. The communication unit (110) may transmit and receive various signals and data while communicating with the robot cleaner (200).

[0048] The processor (120) is configured to control the operation of the terminal device (100). The processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a timing controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, or an artificial intelligence (AI) processor, or may be defined by the relevant term. In addition, the processor (120) may be implemented as a system on chip (SoC) or large scale integration (LSI) having a built-in processing algorithm, or may be implemented in the form of a field programmable gate array (FPGA). The processor (120) may perform various functions by executing computer executable instructions stored in the memory (130). In FIG. 2, only one processor (120) is illustrated, but the processor (120) The number can be implemented in multiples.

[0049] The memory (130) is a configuration for storing various data and programs required for the operation of the terminal device (100). The memory (130) may be implemented as a volatile memory such as S-RAM (Static Random Access Memory) and D-RAM (Dynamic Random Access Memory), a non-volatile memory such as Flash Memory, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory), a hard disk drive (HDD), or a solid state drive (SSD).

[0050] The memory (130) can be accessed by the processor (120). The processor (120) can perform operations such as reading / writing / modifying / deleting / updating data on the memory (130). In FIG. 2, only one memory (130) is included separately from the processor (120). However, depending on the embodiment, the memory (130) may be implemented in multiple units, and at least some of the memories may be mounted within the processor (120).

[0051] The display (140) is a configuration for displaying various screens. For example, the processor (120) controls the display (140) to display the generated 3D map. The display (140) may be implemented as various types of displays, such as an LCD (liquid crystal display), an OLED (organic light-emitting diode), an LCoS (Liquid Crystal on Silicon), a DLP (Digital Light Processing), a QD (quantum dot) display panel, a QLED (quantum dot light-emitting diodes), a μLED (Micro light-emitting diodes), a Mini LED, etc. Meanwhile, the display (140) may also be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc.

[0052] The camera (150) is a component for capturing images. In FIG. 2, one camera (150) is illustrated, but it may also include a stereo camera, a 3D camera, etc. A 3D camera is a camera used to capture and create three-dimensional images or videos. Unlike a typical 2D camera, it composes a captured image by including depth information, and can accurately determine the distance and spatial position of an object through the depth information, and is utilized in various application fields. For example, the camera (150) may be implemented as a stereo camera, a TOF (Time of Flight) camera, a depth camera, a multi-lens array camera, a stereo vision system, a fused lidar camera, etc. In addition, the terminal device (100) may include a 3D LIDAR (Light Detection and Ranging) sensor. A 3D LIDAR (Light Detection and Ranging) sensor is a sensor that can measure the physical properties of an object, such as distance, direction, speed, temperature, material distribution, and concentration characteristics, by irradiating light, such as a laser, on the object and analyzing the laser light reflected from the object. The processor (120) can create a map of an indoor space based on the captured image of the camera (150) and the sensing values ​​of various sensors. For example, the processor (120) can create a 3D point cloud map. A 3D point cloud map is a map that represents a data structure representing the position and shape of an object in three-dimensional space. This map is composed of a series of 3D points (or dots), each point representing a coordinate in space and generally having (x, y, z) coordinates. The 3D point cloud map is used to model the user's real environment and integrate virtual objects in virtual reality (VR) and augmented reality (AR). The processor (120) receives a map generated using various sensors in the robot vacuum cleaner (200) through the communication unit (110).At this time, the map generated by the robot cleaner (200) may be in 2D or 3D form. For convenience of explanation, the map generated by the robot cleaner (200) may be referred to as a cleaning map in the present disclosure. If the robot cleaner includes a vision sensor or camera, the cleaning map can also be generated as a 3D map.

[0053] The processor (120) generates a single 3D map based on the generated 3D point cloud map and the cleaning map received from the robot cleaner (200). The processor (120) compares the floor portion of the 3D point cloud map with the cleaning map, searches for a matching reference point, and can create a 3D map by matching the 3D point cloud, map, and cleaning map based on the reference point.

[0054] The specific details of matching the 3D point cloud map and the map generated by the robot cleaner (200) will be described later.

[0055] The 3D map generated by the processor (120) can be utilized in various ways. For example, the processor (120) may display the map through the display (140) of the terminal device (100) according to a user's command. The user (10) can intuitively check the generated 3D map through the display (140). In this case, the processor (120) may also control the display (140) to additionally display graphic objects related to electronic products within the space on the 3D map. This will be described in detail in the following section.

[0056] As another example, the processor (120) may transmit a 3D map to the robot cleaner (200) via the communication unit (110). The robot cleaner (200) can perform more precise cleaning with the received map. If the robot cleaner (200) does not have a vision sensor or an image sensor, the cleaning map generated by the robot cleaner (200) cannot accurately display the spatial structure, such as a room, a living room, or a bathroom, or the locations of various home appliances or furniture within the space. Therefore, the robot cleaner (200) cannot identify the internal features of the space with only the cleaning map, and thus cannot selectively clean only a specific area. However, if the processor (120) transmits a 3D map to the robot cleaner (200), the robot cleaner (200) can perform selective cleaning for each area. For example, if a user inputs a user command to clean under a table, the robot cleaner (200) can identify the table area based on the 3D map, move to that area, and perform cleaning. In addition, the processor (120) can transmit the generated 3D map to an external device.

[0057] Below, a detailed description will be given of the 3D point cloud map and the map generated by the robot cleaner (200).

[0058] FIG. 3 is a drawing showing an example of a 3D point cloud map (310) generated by a terminal device (100) according to one embodiment of the present disclosure.

[0059] When a terminal device (100) is equipped with a TOF or 3D LIDAR sensor, a 3D camera, etc., the processor (120) can obtain depth information of each object in space through the sensor and camera, and then generate a 3D point cloud map (310) based on the depth information. Hereinafter, a method for generating a 3D point cloud map (310) using a terminal device (100) including a 3D camera will be described.

[0060] A user (10) moves around an indoor space while holding a terminal device (100) including a 3D camera, thereby acquiring information within the indoor space. Based on the acquired information, a processor (120) analyzes images and data within the indoor space and extracts depth information for generating a 3D point cloud map (310). Here, the depth information may have data expressed as coordinates. Thereafter, based on the extracted information, the processor (120) generates a 3D point cloud map (310) that represents the location or information of an object and the structure of the indoor space as 3D points (or dots).

[0061] Referring to FIG. 3, the 3D cloud map (310) can express not only the structure and shape of a space such as a living room, a room, a kitchen, etc., but also various objects placed within the space, such as home appliances or furniture.

[0062] When the processor (120) displays a 3D point cloud map (310) on the display (110), if the user inputs an operation to move or rotate the 3D point cloud map (310), the processor (120) can control the display (110) to display the 3D point cloud map (310) at different angles, such as up, down, right, or left, or to display a changed appearance when the location is moved in space, depending on the operation.

[0063] In cases where a 3D camera or depth camera is not provided, the processor (120) may also generate a 3D point cloud map from captured images captured by the camera (150). For example, the processor (120) extracts feature points from captured images captured by the camera (150) and extracts pose information of the camera (150) or the terminal device (100) based on the feature points.

[0064] The processor (120) can extract feature points using the VSLAM (Visual Simultaneous Localization And Mapping) technique, but is not necessarily limited thereto. Each image frame can be divided into multiple blocks, and the pixel representative values ​​of each block can be compared to extract feature points that match each other.

[0065] The processor (120) can track pose information from a plurality of captured images. The pose information may include, for example, information on how the position of the camera moves based on a key frame among each captured image frame. Key frames may be frames that have matching feature points among the image frames, and thus the position of the camera (150) or the terminal device (100) can be estimated by comparing two image frames. The processor (120) can estimate depth information of each object included in the captured image based on the tracked pose information. The processor (120) can collect points corresponding to the estimated depth information to generate a 3D point cloud map.

[0066] FIG. 4 is a drawing showing a map generated by a robot cleaner (200) according to one embodiment of the present disclosure.

[0067] A robot cleaner (200) can create a cleaning map (410) for an indoor space by driving through a camera or various sensors. Fig. 4 illustrates a case where a robot cleaner (200) creates a 2D cleaning map.

[0068] When a robot cleaner (200) creates a 2D map, the robot cleaner (200) can create a 2D map (410) based on data acquired through various sensors while driving inside a space for cleaning. The robot cleaner (200) can create a 2D map using various sensors for detecting obstacles and SLAM technology. The robot cleaner (200) can identify obstacles and walls in a space based on the sensing values ​​of the sensors, and can identify the spatial structure, shape, location and shape of objects using a distance measurement method between identified objects. In addition, the robot cleaner (200) uses SLAM technology to measure the location and create a map of the surrounding environment. SLAM (Simultaneous Localization and Mapping) technology means simultaneous location estimation and map creation, and refers to a technology that estimates its own location and creates a map of the surrounding environment.

[0069] Meanwhile, when the robot cleaner (200) generates a 3D map, an additional process for generating the 3D map is required in addition to the process of generating the 2D map. In order to generate the 3D map, the robot cleaner (200) must be equipped with a vision sensor and Fusion SLAM must be operating. A vision sensor is a sensor that collects and processes initial information, obtains images or videos of the surrounding environment, and provides a realistic visual experience in VR and AR systems. Fusion SLAM is a SLAM that combines the function of a vision sensor and LIDAR technology, and is a technology that performs simultaneous location estimation and map creation by integrating various sensors and data sources. The robot cleaner (200) can obtain a 3D image or coordinates using the above sensors and technologies. The robot cleaner (200) can generate a partial 3D point cloud map using the obtained 3D information and 2D information. The robot cleaner (200) has limitations in generating a 3D point cloud map for the entire indoor space because the angle of view that the robot cleaner (200) itself can see during the driving or cleaning process for map generation is limited. Therefore, the robot cleaner (200) can generate a partial 3D point cloud map rather than an entire 3D point cloud map. Thereafter, the robot cleaner (200) generates a 3D map by matching 3D LandMark information of image information representing a specific point or object in a 3D space with LIDAR distance information.

[0070] Referring to FIG. 4, the 2D map generated by the robot cleaner (200), i.e., the 2D cleaning map (410), may additionally display the structure and shape of the entire space, the location and shape of objects within the space, as well as the cleaning path actually cleaned by the robot cleaner (200). However, the cleaning path may be deleted according to the user's selection.

[0071] FIGS. 5 and 6 are drawings for explaining a method for generating a 3D map of a terminal device according to an embodiment of the present disclosure.

[0072] The processor (120) of the terminal device (100) can create a 3D map by matching the generated 3D point cloud map (310) with the map (410) generated by the robot cleaner (200). In order to match the two maps, 2D information considering the height of the robot cleaner in the 3D point cloud map (310) must be extracted. Specifically, in order to match the 2D information extracted from the 3D point cloud map (310) with the map (410) generated by the robot cleaner (200), matching of reference points on the 2D map is required.

[0073] In Fig. 5, an image (510) captured from the viewpoint of the robot cleaner (200) and an image (520) captured by a user (10) considering the height information of the robot cleaner (200) are shown. The robot cleaner (200) creates a 2D map using a plurality of images captured while driving within a space, including the image (510) of Fig. 5. The terminal device (200) can also create a map using a plurality of images captured within a space, including the image (520) of Fig. 5.

[0074] In order to match a 2D map created through an image (510) taken from the viewpoint of a robot vacuum cleaner (200) and a 3D map created using an image (520) taken directly by a user (10) using a terminal device (100), the following specific process is required.

[0075] When the robot cleaner (200) generates a 2D map, the floor information considering the height information of the robot cleaner (200) is extracted from the 3D point cloud map (310) generated by the terminal device (100). Here, the floor information refers to map information on the 3D point cloud map (310) viewed from a height similar to the height information of the robot cleaner (200). The height of the robot cleaner (200) may vary for each robot cleaner device, and since a LIDAR sensor is attached, the height information may vary depending on the sensor size. In addition, the height information may vary depending on the inclination and angle of the image or video captured by the user (10). The processor (120) of the terminal device (100) extracts at least one piece of floor information from the 3D point cloud map (310) based on the height information of the robot cleaner (200).

[0076] Since the height information of the robot cleaner (200) may not be completely horizontal, and the height of the floor surface on which the robot cleaner (200) is placed may not be constant or may have steps, the processor (120) may extract not only floor information for one height but also multiple pieces of floor information according to various heights.

[0077] The processor (120) extracts at least one piece of 2D map information corresponding to the height information of the robot cleaner based on at least one piece of floor information extracted from the 3D point cloud map (310). When extracting multiple pieces of floor information, the processor (120) may also extract multiple pieces of 2D map information.

[0078] Meanwhile, noise such as the location of objects in an indoor space and the curvature of the floor may occur in a 2D map generated based on a plurality of pieces of floor information extracted from a 3D point cloud map (310). Noise refers to a curvature phenomenon that occurs when an object is located in a non-fixed location or during the shooting process, or a phenomenon in which images or videos overlap during the shooting of a 3D video. When a user (10) generates a 3D point cloud map (310) through a terminal device (100), noise may occur when extracting floor information due to objects located on the floor of an indoor space, for example, objects that are not located in a fixed location, such as toys, books, trash cans, etc. In addition, a curvature phenomenon may occur in the floor information depending on the shooting angle of the user (10), and noise may occur in which images or videos overlap each other when shooting a 3D video, such as a panorama.

[0079] Referring to FIG. 6, a 2D map (610) from which noise is not removed from the floor information, a 2D map (620) from which noise is removed to a certain extent, and a 2D map (630) from which all noise is removed can be seen. The processor (120) reconstructs height information (z) based on the 2D map (630) from which noise is removed from the floor information, and generates a plurality of 2D maps by considering the height information of the robot cleaner (200). Among the plurality of 2D map information, the processor (120) selects the 2D map information that has a similar form to the 2D map generated by the robot cleaner (200) and has the least error.

[0080] Thereafter, the processor (120) acquires information for matching the adopted 2D map information and the 3D map, such as the reference point, angle, and scale, from the 2D map generated by the robot cleaner (200). Based on the acquired information, the reference point, angle, and scale information are calculated, and a process of matching each map into one map is performed. For example, the processor (120) selects one reference point from the 2D map adopted from the 3D point cloud map (310) and the 2D map generated by the robot cleaner (200) according to the indoor structure or location. Thereafter, the processor (120) adjusts the angle so that each map can face the same direction based on the reference point. When the matching of the reference point and the angle is completed, the processor (120) adjusts the scale so that each map can be overlapped into one map. Specifically, the processor (120) calculates variables for a reference point (x, y), a reference angle (theta), and a scale (r) from a plurality of 2D maps extracted from a 3D point cloud map (310), and selects a 2D map on the 3D point cloud map (310) with the minimum RMS error among the 2D map generated by the robot cleaner (200) and the plurality of 2D maps based on the calculated values. The RMS error refers to an error value that measures the average difference between a value predicted by a prediction model and an actual value. The processor (120) calculates variables for a reference point (x, y), a reference angle (theta), and a scale (r) of the selected 2D map and the 2D map generated by the robot cleaner (200), and selects one reference point based on a feature point. Thereafter, the processor (120) matches each map into one map based on the reference point. In this way, in the process of matching with a single map, the order in which the reference point, angle, and scale information are calculated and applied is not limited to this and can be changed in various ways.

[0081] Meanwhile, if the robot cleaner (200) can also generate a 3D map, the processor (120) extracts height information of the 3D map generated by the robot cleaner (200) based on the floor information of the robot cleaner (200). Thereafter, the processor (120) performs matching to create a single map between the 3D point cloud map (310) corresponding to the height and the 3D map generated by the robot cleaner (200). Since the matching process is the same as described above, such as reference point, angle, and scale information, a duplicate description will be omitted.

[0082] FIGS. 7 and 8 are drawings for explaining various examples of 3D maps generated by a terminal device (100) according to one embodiment of the present disclosure.

[0083] Referring to FIG. 7, as an example, the terminal device (100) matches a 2D map (410) created by the robot cleaner (200) during the actual cleaning process through cleaning driving with a 3D point cloud map (310) created by directly photographing by the user (10) to create a single 3D map (710). In the created 3D map (710), not only the positions and shapes of various objects within a space, but also the driving path of the robot cleaner (200) on the floor of the space can all be expressed. Since the method for creating a 3D map has been described in detail in the above-described section, a redundant description will be omitted.

[0084] Fig. 8 is a map (810) converted into a 2D format from a 3D map (710) created by matching a 3D point cloud map (310) and a map (410) generated by a robot cleaner. The 2D map (810) has noise from the indoor floor removed for precision, and displays the area where the robot cleaner (200) actually cleaned. In addition, the 2D map (810) complements the limitations of the robot cleaner (200) in terms of the angle of view, and enables more detailed display than the generated 2D map (410). The 2D map (810) shown in Fig. 8 is merely an example, and the terminal device (100) can reconstruct and provide various types of maps based on the generated 3D map.

[0085] Meanwhile, the process of creating a 3D map (710) by matching maps is one embodiment of the present disclosure, and may be performed in the processor (120) of the terminal device (100) or directly in the processor of the robot cleaner (200). Furthermore, it may be performed without a matching task through a separate server. The entity performing the 3D map matching task is not limited thereto and may be implemented in various ways.

[0086] FIG. 9 is a drawing for explaining a process of displaying a graphic object on a 3D map generated by a terminal device (100) according to one embodiment of the present disclosure.

[0087] The processor (120) can add and display graphic objects (920, 930) related to the operation of various electronic products placed in an indoor space on a 3D map (910) created by matching each of the two maps.

[0088] In Fig. 9, a case is shown where a graphic object (920) corresponding to the shape of an air conditioner located in an indoor space and a graphic object (930) related to the operation of the air conditioner are displayed together.

[0089] When a user checks the 3D map in Fig. 9, the user can intuitively know that the air conditioner is currently operating.

[0090] The processor (120) can receive information about the operation of the air conditioner through the communication unit (110). When the operation status changes, the processor (120) can express the operation status by changing the graphic object (930). For example, the direction of air flow and the wind speed of the air conditioner in an indoor space can be expressed as a graphic object (930), and when the wind speed increases, the size of the graphic object (930) can be increased or the display (140) can be controlled to add a wind-shaped line. In addition, the processor (120) can display the cooling set temperature in text form. As another example, the processor (120) can display the operation status, air flow direction, and wind speed of the air purifier as a graphic object while communicating with the air purifier, and when a robot vacuum cleaner is cleaning, it can display the location of the robot vacuum cleaner. The processor (120) can also display the operation status of various other home appliances.

[0091] Although not illustrated in FIG. 9, graphic objects representing various menus may be additionally displayed on the map (910). Alternatively, the processor (120) may utilize various graphic objects displayed on the map (910) as menus. For example, when a user selects an air conditioner object (920) on the map (910) of FIG. 9, the processor (120) may execute a control application for controlling the operation of the air conditioner and display the execution screen on the display (140).

[0092] As described above, the 3D map generated by the terminal device (100) and the robot cleaner (200) in conjunction with each other can also be utilized as a user interface.

[0093] FIG. 10 is a drawing for explaining semantic information on a 3D map of a terminal device (100) according to one embodiment of the present disclosure.

[0094] Figure 10 shows an example of a map (1010) on which semantic information is displayed.

[0095] Semantic information encompasses information designed to enable a robot vacuum cleaner to identify the structure and shape of a space, as well as the types and locations of objects within it. For example, it refers to information necessary for cleaning tasks, such as detecting and recognizing obstacles like furniture, walls, and appliances, or gathering information necessary to identify and remove dust, hair, and other objects from the floor.

[0096] By utilizing a map (1010) containing semantic information, the robot vacuum cleaner can identify the locations of furniture, home appliances, and obstacles.

[0097] Meanwhile, if the processor (120) displays semantic information on the 3D map (710), the robot cleaner (200) can receive the 3D map (710) through the terminal device (100) and utilize the semantic information that the robot cleaner (200) could not search for. That is, if the robot cleaner (200) includes a vision sensor or an image sensor, it can directly generate a cleaning map including semantic information. However, in the case of a robot cleaner (200) that does not include such a sensor, it can only generate a map that simply represents a spatial structure or shape, and cannot identify the meaning of each point in the space. In this case, if the 3D map generated by the terminal device (100) is provided to the robot cleaner (200), the robot cleaner (200) can recognize the semantic information in the 3D map. The terminal device (100) or robot cleaner (200) can analyze the shape of each object included in the 3D map and identify what type of object it is, thereby utilizing it as semantic information.

[0098] As another example, a user (10) may directly add semantic information to a 3D map (710). The processor (120) may transmit data about the 3D map including the semantic information added by the user to the robot cleaner (200).

[0099] After matching the 3D map, the positions of objects such as furniture and home appliances may change, or new obstacle information may be updated. In such cases, it is necessary to update the object position information and obstacle update information in the 3D map. For example, after the 3D map is completed, if the robot cleaner (200) scans a new obstacle while cleaning and obtains new map information, or if the positions of furniture or home appliances change, the 3D map needs to be updated. The 3D map update task can be performed every time the robot cleaner (200) updates the cleaning map while performing cleaning. The process of updating the 3D map may include extracting 2D information considering the height information of the robot cleaner from the 3D point cloud map, and matching the map with the updated 2D map of the robot cleaner.

[0100] The process of matching the map has been specifically described in the above section, so it will be omitted. When an update is performed, the processor (120) can store the new 3D map in the memory (130) and transmit it to the robot cleaner (200) via the communication unit (110).

[0101] FIG. 11 is a block diagram for explaining the configuration of a robot vacuum cleaner (200) according to one embodiment of the present disclosure.

[0102] According to FIG. 11, the robot vacuum cleaner (200) includes a communication unit (210), a processor (220), a memory (230), a sensor (240), and a driving unit (250).

[0103] The communication unit (210) is a configuration for transmitting and receiving information with an external device and / or an external network. When connected to the terminal device (100) through the communication unit (210), the user can input various user operations through the terminal device (100). The communication unit (2110) receives a control signal corresponding to the user operation from the terminal device (100) and transmits it to the processor (220). Accordingly, the user can also remotely control the operation of the robot cleaner (200). The communication unit (210) may include at least one of a wireless Internet module, a short-range communication module, and a location information module. Here, the wireless Internet module refers to a module for wireless Internet access, and may be built into or external to the robot cleaner (200). At this time, wireless Internet technologies such as WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), and HSDPA (High Speed ​​Downlink Packet Access) may be used. Additionally, a short-range communication module refers to a module for short-range communication. Here, short-range communication technologies such as Bluetooth, RFID (Radio Frequency Identification), infrared communication (IrDA, infrared Data Association), UWB (Ultra Wideband), and ZigBee can be used.

[0104] The sensor (240) is a sensor that detects the surrounding environment. Specifically, it may include at least one of a LiDAR sensor, a vision sensor, an image sensor, an infrared sensor, an ultrasonic sensor, a gyro sensor, an acceleration sensor, and a proximity sensor. The processor (220) can detect the distance to a wall or obstacle in an indoor space to be cleaned based on the sensing value of the sensor (240).

[0105] The driving unit (250) is a component for moving the main body of the robot cleaner (200). The driving unit (250) may include wheels for movement, a motor for rotating the wheels, etc. The wheels may be left and right driving wheels installed on the lower sides of the robot cleaner main body, respectively. In addition, the driving unit (250) may be provided with at least one auxiliary wheel on the bottom of the cleaner main body to minimize friction between the robot cleaner (200) and the floor while guiding the movement of the robot cleaner (200).

[0106] The processor (220) can recognize a location within a space and determine a driving direction and path based on values ​​sensed by a sensor (240) while the robot cleaner is performing cleaning. The processor (220) generates a cleaning map for an indoor space based on the driving path and then stores the map in the memory (230). Once the generation of the cleaning map for the entire space is complete, the processor (220) can transmit data on the generated cleaning map to the terminal device. As described in the various embodiments described above, when the terminal device (100) generates a single 3D map based on the 3D point cloud map and the cleaning map and then transmits the data, the processor (220) can receive data on the 3D map through the communication unit (210) and store the data in the memory (230). The processor (220) performs cleaning by utilizing the stored map.

[0107] The processor (220) can identify each area of ​​an indoor space based on a 3D map. For example, when a user inputs a cleaning command for a specific area of ​​an indoor space, the processor (220) identifies the location of the specific area where the specific user inputs the cleaning command based on the 3D map received from the terminal device, and controls the driving unit (250) so that the robot cleaner (200) moves to the identified location.

[0108] FIG. 12 is a flowchart of a map generation method of a terminal device according to an embodiment of the present disclosure.

[0109] Referring to FIG. 12, when a user takes a picture of an indoor space where a robot cleaner is located using a camera of a terminal device, the terminal device generates a 3D point cloud map based on the captured data (S1210). Thereafter, the terminal device receives data regarding the map generated by the robot cleaner (S1220) and matches the 3D point cloud map with the map generated by the robot cleaner to generate a single 3D map (S1230). The terminal device may also display the generated 3D map (S1240).

[0110] Meanwhile, the step of generating a 3D map may include a step of extracting floor information from a 3D point cloud map, and extracting 2D map information corresponding to height information of the robot cleaner based on the floor information. At this time, there may be a plurality of 2D map information. Thereafter, the 2D map information generated by the terminal device and the map generated by the robot cleaner are compared, and the 2D map information with the lowest error value is selected, and one reference point, angle, and scale information are acquired from the selected 2D map information and the 2D map generated by the robot cleaner, and one 3D map is generated based on one reference point. The process of generating a 3D map has been described in detail in the above-mentioned part, so it will be omitted.

[0111] Meanwhile, if the robot cleaner can create a 3D map, the height information of the created 3D map is extracted, and the 3D map created by the robot cleaner and the 3D point cloud map are matched based on the height information to create a single 3D map.

[0112] Additionally, graphic objects related to the operation of electronic products placed within an indoor space can be additionally displayed on a 3D map.

[0113] FIG. 13 is a sequence diagram showing the order in which a terminal device and a robot cleaner operate according to an embodiment of the present disclosure.

[0114] Referring to FIG. 13, a terminal device collects movement data using a technology such as TOF (S1301). A robot cleaner collects movement data using a LIDAR sensor and SLAM technology (S1302). This process may occur simultaneously or sequentially for each of the terminal device and the robot cleaner. In addition, the terminal device acquires obstacle images and depth information (S1303) and generates a 3D point cloud map (S1304). The robot cleaner generates a 2D map based on values ​​sensed by a sensor (S1305). Floor information is extracted from the 3D point cloud map (S1306), and a plurality of 2D information considering height information of the robot cleaner based on the floor information is extracted (S1307). The robot cleaner transmits the generated 2D map information through a communication unit of the terminal device (S1308). Among the multiple 2D information of the terminal device, the 2D information with the lowest error value is selected from the 2D map of the robot cleaner, and a process of matching it with the 2D map of the robot cleaner is performed (S1309). At this time, in the matching process, reference point and angle scale information are extracted (S1310), and if semantic data is displayed on the map, the error is adjusted with the semantic data (S1311) to ultimately generate a single 3D map. The terminal device displays the generated 3D map on the display (S1312) or transmits it to the robot cleaner.

[0115] FIG. 14 is a flowchart showing the overall process of a terminal device according to an embodiment of the present disclosure.

[0116] Referring to FIG. 14, the terminal device generates a 3D point cloud map (S1410), and generates a cleaning map while the robot cleaner drives (S1420). Thereafter, the terminal device matches the 3D point cloud map with the map generated by the robot cleaner (S1430). If the robot cleaner obtains new map information while driving to clean (S1440), the terminal device may update the 3D map (S1450). The terminal device displays at least one of the generated 3D map and the updated 3D map on the display of the terminal device (S1460).

[0117] Meanwhile, the various embodiments described above may be applied to a product as an embodiment alone, but at least some of the contents may be implemented in combination with other embodiments of the present disclosure.

[0118] According to various embodiments of the present disclosure, a terminal device can generate a 3D map based on images captured by the terminal device itself and a 2D map generated by a robot cleaner. This map can be more accurate than maps generated independently by the terminal device or robot cleaner. By viewing the generated 3D map, a user can more easily understand the structure and shape of a space.

[0119] In particular, according to various embodiments of the present disclosure, problems caused by the limited field of view of a robot vacuum cleaner can be overcome, enabling efficient cleaning by the robot vacuum cleaner. Furthermore, even without a vision sensor or image sensor, the robot vacuum cleaner can identify and classify areas, enabling selective cleaning. Furthermore, when there are changes in object information and obstacle locations in an indoor space, the 3D map can be updated, enabling efficient and accurate cleaning. Furthermore, the 3D map can be used to control the robot vacuum cleaner and other home appliances through linkage with them.

[0120] Meanwhile, the methods according to the various embodiments described above may be implemented in the form of program code for performing each step, stored on a recording medium, and distributed. In this case, a device equipped with the recording medium can perform the map generation method according to the various embodiments described above.

[0121] For example, program code for sequentially performing the steps of creating a 3D point cloud map of a space based on photographed data of a space where a robot cleaner is located and the step of creating a 3D map of the space by matching the map created by the robot cleaner and the 3D point cloud map of the space may be stored in various recording media.

[0122] Such recording media may be various types of non-transitory computer-readable recording media, such as ROM, RAM, memory chips, memory cards, external hard drives, hard drives, CDs, DVDs, magnetic disks, or magnetic tapes.

[0123] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the terminal device, A communication unit for communicating with a robot vacuum cleaner; memory; camera; display; and Processor; including; The above processor, Store the image captured by the camera in the space where the robot cleaner is located in the memory, When data on a map generated by the robot cleaner for the space is received through the communication unit, a 3D map for the space is generated based on the captured image and the map generated by the robot cleaner, A terminal device that controls the display to display the generated 3D map.

2. In paragraph 1, The above processor, A terminal device that transmits the generated 3D map to the robot cleaner through the communication unit.

3. In paragraph 1, The above camera, Includes a 3D camera, The above processor, When a user command is input, the 3D camera is activated, and when the terminal device rotates within the space while the activated 3D camera is shooting the inside of the space, a 3D point cloud map is generated based on the captured image of the 3D camera regarding the inside of the space. A terminal device that creates the 3D map by matching the 3D point cloud map and the map created by the robot cleaner.

4. In paragraph 3, The above processor, Extract floor information from the above 3D point cloud map, and extract 2D map information corresponding to the height of the robot cleaner based on the above floor information, A terminal device that compares the 2D map information with the map generated by the robot cleaner to identify at least one reference point, and matches the 3D point cloud map with the map generated by the robot cleaner based on the at least one reference point.

5. In paragraph 4, The above processor, The above 2D map information is extracted into multiple pieces according to different heights based on the above floor information, A terminal device that compares a plurality of pieces of 2D map information with maps generated by the robot cleaner, identifies 2D map information having a minimum error, and matches the identified 2D map information with the map generated by the robot cleaner.

6. In paragraph 3, The map generated by the above robot vacuum cleaner is a 3D map. The above processor, Extract the height information of the 3D map generated by the above robot cleaner, A terminal device that matches a 3D map generated by the robot cleaner based on the above height information and the 3D point cloud map.

7. In paragraph 3, The above processor, A terminal device that controls the display to additionally display a graphic object related to the operation of at least one electronic product placed within the space on the 3D map.

8. In robot vacuum cleaners, A driving unit for driving the above robot cleaner; A communication unit for communicating with a terminal device; At least one sensor; memory; and Processor; including; The above processor, While controlling the driving unit to drive the robot cleaner, a sensing value sensed by at least one sensor is received, and a map of the space where the robot cleaner is located is created based on the sensing value and stored in the memory. Data on the above map is transmitted to the terminal device through the above communication unit, A robot vacuum cleaner that receives data on a 3D map created by matching a 3D point cloud map and the map from the terminal device through the communication unit and stores the data in the memory.

9. In paragraph 8, The above processor, A robot cleaner, which, when a cleaning command for a specific area within the space is input, identifies the location of the specific area within the space based on the 3D map and controls the driving unit to move to the identified location.

10. A method for generating a map of a terminal device including a 3D camera, A step of generating a 3D point cloud map based on the shooting data of the 3D camera for the space where the robot cleaner is located; A step of receiving data on a map generated by the robot cleaner for the above space; A step of creating a 3D map of the space by matching the 3D point cloud map and the map created by the robot cleaner; and A method for generating a map, comprising: a step of displaying the generated 3D map.

11. In paragraph 10, The step of creating a 3D map of the space by matching the above 3D point cloud map and the map created by the robot cleaner is as follows. A step of extracting floor information from the 3D point cloud map and extracting 2D map information corresponding to the height of the robot cleaner based on the floor information; and A step of identifying at least one reference point by comparing the above 2D map information with the map generated by the robot cleaner; and A method for generating a map, comprising: a step of matching the 3D point cloud map and the map generated by the robot cleaner based on the at least one reference point.

12. In paragraph 10, The step of creating a 3D map of the space by matching the above 3D point cloud map and the map created by the robot cleaner is as follows. A step of extracting floor information from the 3D point cloud map and extracting 2D map information corresponding to the height of the robot cleaner based on the floor information; A step of obtaining a plurality of 2D map information for a plurality of different heights based on the floor information from the extracted 2D map information; and A step of comparing the acquired plurality of 2D map information with the map generated by the robot cleaner to identify the 2D map information having the minimum error; A step of identifying at least one reference point by comparing the identified 2D map information with the map generated by the robot cleaner; and A method for generating a map, comprising: a step of matching the identified 2D map information with a map generated by the robot cleaner based on the at least one identified reference point.

13. In paragraph 10, The map generated by the above robot vacuum cleaner is a 3D map. The step of creating a 3D map of the space by matching the above 3D point cloud map and the map created by the robot cleaner is as follows. A step of extracting height information of a 3D map generated by the above robot cleaner; A method for generating a map, comprising: a step of matching a 3D map generated by the robot cleaner and the 3D point cloud map based on the height information.

14. In paragraph 10, A method for generating a map, comprising the step of additionally displaying a graphic object related to the operation of at least one electronic product placed within the space on the 3D map.

15. A non-transitory computer-readable recording medium storing a computer command that, when executed by a processor of a terminal device, causes the terminal device to perform an operation, wherein the operation is: A step of generating a 3D point cloud map of a space based on the photographed data of the space where the robot cleaner is located; and A non-transitory computer-readable recording medium, comprising: a step of creating a 3D map of the space by matching the map created by the robot cleaner for the space with the 3D point cloud map;

Images