Cleaning robot and method for performing tasks thereof

Abstract

A method for performing tasks of a cleaning robot is provided. The method according to an embodiment includes generating a navigation map for driving the cleaning robot by using at least one sensor to detect the result of a task area in which an object is arranged, obtaining recognition information of the object by applying an image of the object captured by at least one camera to a trained artificial intelligence model, generating a semantic map of an environment indicating the task area by mapping the area of ​​the object included in the navigation map with the recognition information of the object, and performing the task of the cleaning robot based on a control command of a user using the semantic map. An example of a trained artificial intelligence model may be a deep learning neural network model in which a plurality of network nodes having weighted values ​​are arranged in different layers and exchange data according to a convolution relationship, but the present disclosure is not limited thereto.

Description

Technical Field

[0001] Apparatus and methods consistent with what is disclosed herein relate to a cleaning robot and a task method thereof, and more particularly, to a cleaning robot and a control method thereof that use information about objects (eg, obstacles) near the cleaning robot to provide appropriate tasks. Background Art

[0002] With the development of robot technology, robots have been widely used in households and professional technical fields or industries that require a large amount of labor. Specifically, service robots, cleaning robots, pet robots, etc. that provide housekeeping services to users have been widely used.

[0003] In particular, in the case of a cleaning robot, it is very important to specifically identify information about objects (such as foreign objects, structures, obstacles, etc.) near the cleaning robot in depth and perform tasks suitable for each object. However, due to the limited combination of sensors, conventional cleaning robots are limited in obtaining detailed information about objects. In other words, conventional cleaning robots do not have information about what kind of object an object is, but rely solely on the detection capability (sensing capability) of the sensor to drive to avoid objects of the same pattern.

[0004] Therefore, there is a need to identify objects near a cleaning robot, determine tasks suitable for the objects that the cleaning robot can perform, and drive the cleaning robot more efficiently or avoid the objects. Summary of the invention

[0005] Technical solutions to the problem

[0006] Aspects of exemplary embodiments are directed to provide a cleaning robot which provides a service for performing a task suitable for a peripheral object using a plurality of sensors of the cleaning robot, and a control method thereof.

[0007] According to an exemplary embodiment, a method for performing a task of a cleaning robot is provided, the method including generating a navigation map for driving the cleaning robot based on receiving sensor data from at least one sensor that detects or senses a task area in which an object is arranged, obtaining recognition information of the object by applying an image of the object captured by at least one camera to a trained artificial intelligence model, generating a semantic map of an environment indicating the task area by mapping an area of ​​the object included in the navigation map with the recognition information of the object, and performing the task of the cleaning robot based on a user's control command using the semantic map.

[0008] According to an exemplary embodiment, a cleaning robot is provided, which includes at least one sensor, a camera, and at least one processor, the at least one processor being configured to generate a navigation map for driving the cleaning robot based on sensor data received from at least one sensor that detects (or senses) a task area in which an object is arranged, obtain recognition information of the object by applying an image of the object captured by the camera to a trained artificial intelligence model, provide a semantic map of an environment indicating the task area by mapping an area of ​​the object included in the navigation map with the recognition information of the object, and perform the task of the cleaning robot based on a user's control command using the semantic map.

[0009] According to the various exemplary embodiments described above, the cleaning robot may provide services for performing the most appropriate task, such as removing or avoiding one or more objects by considering identification information and / or additional information of the objects (eg, nearby objects).

[0010] According to an exemplary embodiment, a method is provided, the method comprising: receiving, by a cleaning robot, a captured image from a camera or sensor of the cleaning robot, sending, by the cleaning robot, the captured image to an external server, obtaining, by the external server, recognition result information by inputting the captured image into a trained artificial intelligence model, the recognition result information including information about an object, sending, by the server, the recognition result information to the cleaning robot, generating, by the cleaning robot, a semantic map based on mapping an area corresponding to an object included in a navigation map with the recognition information of the object, the semantic map including information indicating the location of an object in a task area in the navigation map, and performing, by the cleaning robot, a task based on a user's control command using the semantic map.

[0011] According to the various exemplary embodiments described above, the cleaning robot can provide a semantic map indicating the environment of the task area. Therefore, the user can control the task of the cleaning robot by using the name of the object or place, etc. and the provided semantic map, so that the usability can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a view explaining a method of recognizing and detecting (or sensing) an object of an obstacle of a cleaning robot according to an embodiment of the present disclosure;

[0013] Figure 2A and Figure 2B is a block diagram explaining a configuration of a cleaning robot according to an embodiment of the present disclosure;

[0014] Figure 3 is a detailed block diagram explaining the configuration of a cleaning robot according to an embodiment of the present disclosure;

[0015] Figure 4A , Figure 4B , Figure 4C and Figure 4D is a view explaining that the cleaning robot according to an embodiment of the present disclosure obtains additional information about an object based on a detection result through an infrared (IR) stereo sensor;

[0016] Figure 5A , Figure 5B and Figure 5C is a view explaining that the cleaning robot obtains additional information about an object based on a detection result through a LIDAR sensor;

[0017] Fig. 6A and Figure 6B is a view explaining that the cleaning robot obtains additional information about an object based on a detection result through an ultrasonic sensor;

[0018] Fig. 7A , Figure 7B and Figure 7C is a view showing a structure in which a cleaning robot recognizes a house;

[0019] Fig. 8A and Figure 8B is a view explaining that a cleaning robot according to an embodiment of the present disclosure generates a semantic map based on a structure of a house and additional information about an object;

[0020] Fig. 9A and Fig. 9B is a view explaining that the cleaning robot notifies the user that there is a hazardous object on the floor;

[0021] Fig.10 is a view explaining an area designated not to be cleaned by a cleaning robot according to an embodiment of the present disclosure;

[0022] Fig.11A and Fig. 11B is a block diagram illustrating a training module and a recognition module according to various embodiments of the present disclosure;

[0023] Fig.12 is a diagram showing an example in which a cleaning robot and a server may operate in association with each other to train and recognize data;

[0024] Fig.13 is a flowchart explaining a network system using a recognition model according to an embodiment of the present disclosure;

[0025] Fig.14 is a flowchart explaining an example in which a cleaning robot provides a search result of a first area using a recognition model according to an embodiment of the present disclosure;

[0026] Fig.15is a flow chart explaining a system using a recognition model according to an embodiment of the present disclosure;

[0027] Fig.16 is a diagram for explaining generation of a semantic map according to an embodiment of the present disclosure;

[0028] Fig.17 is a view showing a user interface for using a semantic map according to an embodiment of the present disclosure;

[0029] Fig.18 is a view showing a case where a cleaning robot is controlled by using a semantic map according to an embodiment of the present disclosure;

[0030] Fig.19 is a diagram for explaining generation of a semantic map according to an embodiment of the present disclosure;

[0031] Fig. 20 is a view showing a user interface using a semantic map according to an embodiment of the present disclosure;

[0032] Fig.21 is a view showing a case where a cleaning robot is controlled by using a semantic map according to an embodiment of the present disclosure;

[0033] Fig. 22 is a view showing a process of recognizing an object according to an embodiment of the present disclosure;

[0034] Fig.23 is a view explaining a process of generating a semantic map according to an embodiment of the present disclosure;

[0035] Fig.24 is a view showing a case where a cleaning robot is controlled by using a semantic map according to an embodiment of the present disclosure;

[0036] Fig.25 is a view showing a configuration of a cleaning robot according to an embodiment of the present disclosure;

[0037] Fig.26A and Fig.26B is a view showing a detection range (sensing range) of a cleaning robot according to an embodiment of the present disclosure; and

[0038] Fig. 27 , Fig.28 and Fig.29 is a flowchart explaining the cleaning robot according to the embodiment. DETAILED DESCRIPTION

[0039] It should be understood that the disclosure herein is not intended to limit the scope to the described embodiments, but includes various modifications, equivalents and / or alternatives of the embodiments.In the description of the drawings, the same reference numerals refer to the same elements throughout the description of the drawings.

[0040] Terms such as "first" and "second" may be used to modify various elements, regardless of their order and / or importance. These terms are only used for the purpose of distinguishing a component from other components. For example, a first user device and a second user device may represent different user devices, regardless of order or importance. For example, without departing from the scope of the claims described in this disclosure, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0041] When an element (e.g., a first component) is referred to as being “operably or communicatively coupled to” or “connected to” another element (e.g., a second component), it should be understood that each component is directly connected or indirectly connected via another component (e.g., a third component). However, when an element (e.g., a first component) is referred to as being “directly coupled to” or “directly connected to” another element (e.g., a second component), it should be understood that no other component (e.g., a third component) is interposed therebetween.

[0042] The terms used herein are for the purpose of describing specific embodiments, rather than for limiting the scope of other example embodiments. As used herein, the singular form is for ease of explanation, but is also intended to include the plural form, unless the context clearly indicates otherwise. In addition, the terms used in this specification may have the same meaning as those generally understood by those skilled in the art. The general predefined terms used herein may be interpreted as having the same or similar meaning as the contextual meaning of the relevant art, and unless clearly defined in this article, these terms should not be interpreted as ideal or too formal meanings. In some cases, the terms defined herein cannot be interpreted as excluding embodiments of the present disclosure.

[0043] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Figure 1 is a view explaining a method of recognizing and detecting (sensing) an object of an obstacle of a cleaning robot according to an embodiment of the present disclosure.

[0044] refer to Figure 1, the cleaning robot 100 can identify an object 200 near the cleaning robot. The cleaning robot 100 can be a device that moves by itself and provides cleaning services to a user, and it can be implemented as various types of electronic devices. For example, for various purposes, the cleaning robot 100 can be embodied in various forms, such as a cylindrical or rectangular parallelepiped shape, such as a household cleaning robot, a high-rise cleaning robot, an airport cleaning robot, etc. According to an embodiment, the cleaning robot 100 can not only perform the task of removing foreign objects on the floor, but also perform the task of moving objects according to the user's instructions.

[0045] The cleaning robot 100 can capture an image including the object 200 through a camera, and input the captured image into an artificial intelligence model trained for identifying the object so as to identify the object. The artificial intelligence model may be included in the cleaning robot 100, or included in an external server (not shown). For example, the artificial intelligence model may be a model trained according to a supervised learning method and an unsupervised learning method based on an artificial intelligence algorithm. As an example of an artificial intelligence model, a neural network model may include a plurality of network nodes having weighted values, and the plurality of network nodes may be located at different depths (or layers) to send or receive data according to a convolutional connection relationship. For example, a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), etc. may be used as a neural network model, but the present disclosure is not limited thereto.

[0046] When the object 200 is recognized as a specific type of object (e.g., a chair), the cleaning robot 100 may use sensors other than the red-green-blue (RGB) camera to obtain more specific information about the object 200. Specifically, when the object is a specific type of object (e.g., a chair), the cleaning robot 100 may be preset to preferentially use a light detection and ranging (LIDAR) sensor to obtain information about the position of a part (e.g., a chair leg) of the specific type of object or the distance between the parts (e.g., the legs), or to give a higher weighted value to the result of the LIDAR sensor among the results of multiple sensors.

[0047] The cleaning robot 100 may emit a laser pulse 220 for detecting the object 200 through the LIDAR sensor to obtain specific information about the recognized object 200. This will be described in detail below.

[0048] The cleaning robot 100 can store speed information, object image information, and speed information of the cleaning robot at different positions (e.g., a first position and a second position). The cleaning robot 100 can determine the distance (d) between the first position and the second position based on the stored information, and determine the distance from a specific position to the object 200.

[0049] The cleaning robot 100 may determine a task to be performed by the cleaning robot 100 with respect to the object 200 based on specific information about the position of a part of a specific object (e.g., a chair leg), the distance between the parts of a specific object (e.g., the chair legs), etc., obtained through an additional sensor (such as a LIDAR sensor), and distance information from the object 200. For example, the cleaning robot 100 may control the moving speed and direction based on the information about the distance between the parts of a specific object (e.g., the chair legs) to perform a task of cleaning the space between the parts of a specific object (e.g., the chair legs).

[0050] The cleaning robot 100 may determine a no-go area or a bounding box 210 for the recognized object 200. When there is no object recognition by the RGB camera and no additional detection information except information about the no-go area 210, the cleaning robot 100 may only approach the no-go area 210 at most.

[0051] In the above example, it is illustrated that the object is a chair and the sensor is a LIDAR sensor, but the present disclosure is not limited thereto. Various sensors may be used for various objects.

[0052] The cleaning robot 100 can determine the task differently according to the object recognition result. For example, the cleaning robot 100 can perform the following functions: when the object is recognized as grain, remove the grain; when the recognized object is a mat, push the mat away; and when the object is recognized as a fragile and therefore easily broken glass, reduce the moving speed to completely avoid the glass. In addition, when the object is recognized as a dangerous object, the cleaning robot 100 can capture an image of the object and send the image to the user terminal device. When the object is recognized as a dirty object (such as pet feces), the cleaning robot 100 can perform a specific recognition for completely avoiding the object, and send the image including the object to the user terminal device to notify the user of the pet's feces (or glass).

[0053] The cleaning robot 100 can obtain specific information about the object based on a combination of detection (sensing) results of at least one sensor, which can be used to determine whether to perform close avoidance driving or complete avoidance driving. This will be described in detail below.

[0054] Figure 2Ais a block diagram explaining a configuration of a cleaning robot according to an embodiment of the present disclosure.

[0055] refer to Figure 2A , the cleaning robot 100 may include a sensor 110 , a camera 120 , a memory 130 , and a processor 140 .

[0056] The sensor 110 may include various sensors. Specifically, the sensor 110 may include an IR stereo sensor, a LIDAR sensor, an ultrasonic sensor, etc. Each of the IR stereo sensor, the LIDAR sensor, and the ultrasonic sensor may be implemented as one sensor, or as a separate sensor.

[0057] IR stereo sensors can detect the three-dimensional shape and distance information of an object. IR stereo sensors can obtain three-dimensional (3D) depth information of an object, which may include length, height, and width information. However, IR stereo sensors have the disadvantage of not detecting black, transparent, or metallic colors.

[0058] The cleaning robot 100 can use the LIDAR sensor to obtain the two-dimensional (2D) line shape and distance information of the object. Therefore, the cleaning robot 100 can obtain information about the space of the object and distance information about nearby objects. However, the LIDAR stereo sensor has the disadvantage of not detecting black, transparent or metallic colors.

[0059] Ultrasonic sensors can obtain distance information about obstacles. Ultrasonic sensors have the disadvantage of a relatively limited sensing range, but have the advantage of detecting black, transparent or metallic colors.

[0060] In addition, the sensor 110 may include a sensor for detecting the surrounding environment, such as a dust sensor, an odor sensor, a laser sensor, an ultra-wideband (UWB) sensor, an image sensor, an obstacle sensor, and a sensor for detecting a moving state, such as a gyroscope sensor, a global positioning system (GPS) sensor, etc. The sensor for detecting the surrounding environment and the sensor for detecting the moving state of the cleaning robot may be implemented in different configurations or a single configuration. The sensor 110 may further include various sensors, and some of the sensors may not be included depending on the task performed by the cleaning robot 100.

[0061] The camera 120 may be configured to capture peripheral images of the cleaning robot 100 from various aspects. The camera 120 may capture a front image of the cleaning robot 100, or an image in a direction other than the driving direction, through an RGB camera. The camera 120 may be independently provided in the cleaning robot 100, or included in an object recognition sensor as part thereof.

[0062] The camera 120 may include a plurality of cameras. The camera 120 may be installed on at least one of the upper portion or the front portion of the cleaning robot 100.

[0063] The memory 130 may store images and movement status information captured by the camera 120, and may capture direction information of the cleaning robot 100 when capturing. The memory 130 may store navigation map information for placing the cleaning robot 100 to perform tasks by the cleaning robot 100. However, the present disclosure is not limited thereto, and the memory 130 may store various programs required to operate the cleaning robot 100.

[0064] The memory 130 can store a plurality of applications and / or applications driven by the cleaning robot 100, as well as data commands, etc., for operating the cleaning robot 100. Part of the application can be downloaded from an external server via wireless communication. When released, at least part of the application can be set in the cleaning robot 100 for basic functions. The application can be stored in the memory 130 and cause the cleaning robot 100 to operate (or function).

[0065] As various example embodiments, the memory 130 may generate a navigation map for driving the cleaning robot 100 using the result of at least one sensor detecting a task area in which an object is arranged, obtain recognition information of the object by applying an image of the object captured by at least one camera to a trained artificial intelligence model, map the area of ​​the object included in the navigation map with the recognition information of the object, and store at least one instruction set to generate a semantic map of the environment indicating the task area.

[0066] The memory 130 can store at least one instruction so that the processor 140 can capture an image of an object near the cleaning robot, obtain recognition information of the object included in the image by applying the captured image to a trained artificial intelligence model, detect the object by using at least one sensor selected based on the obtained recognition information of the object, and obtain additional information about the object using the result detected by the at least one sensor.

[0067] The memory 130 may be implemented with at least one of a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SDD). The memory 130 may be accessed by the processor 140, and any one of reading / writing / modifying / deleting / updating data (e.g., data stored in the memory) may be performed by the processor 140. According to the present disclosure, the term "memory" may include the memory 130, a read-only memory (ROM) (not shown), or a random access memory (RAM) (not shown) in the processor 140, or a memory card (not shown) (e.g., a micro SD card, a memory stick, etc.) installed in the cleaning robot 100.

[0068] The processor 140 can control the overall operation of the cleaning robot 100. For example, the processor 140 can control the camera 120 to capture images near the cleaning robot 100. The processor 140 may include a RAM and a ROM, or the system may include a ROM and a RAM and a processor 140. The ROM may store a command set for system booting. The CPU 141 may copy an operating system (O / S) stored in the cleaning robot 100 to the RAM according to the command stored in the ROM, and execute the O / S to perform system booting. When the system booting is completed, the CPU 141 may copy various programs stored in the memory 130 to the RAM, execute the programs copied to the RAM, and perform various operations.

[0069] According to the embodiment, reference Figure 2B , the cleaning robot 100 may include a plurality of processors 141 and 142. The plurality of processors 141 and 142 may include a central processing unit (CPU) 141 and a neural processing unit (NPU) 142. The NPU 142 may be a specific processor optimized for recognizing an object by using a trained artificial intelligence model. The NPU (and / or CPU) of the cleaning robot 100 may include at least one of a digital signal processor (DSP), a microprocessor, a time controller (TCON), a microcontroller unit (MCU), a microprocessing unit (MPU), an application processor (AP), a communication processor (CP), or an ARM processor, etc., for processing digital signals, or may be defined by different combinations of the corresponding terms. The processor 140 may be implemented as a system on chip (SoC), a large-scale integrated circuit (LSI) with a built-in processing algorithm, or in the form of a field programmable gate array (FPGA).

[0070] The processor 140 can recognize obstacles included in the image by an artificial intelligence model trained to recognize objects such as obstacles. The processor 140 can input an image including obstacles to the artificial intelligence model and obtain an output result including information about the type of obstacles. The processor 140 can determine the size of different restricted areas according to the type of obstacles. The restricted area can be an area including obstacles and can be an area that cannot be entered by the cleaning robot 100 performing a cleaning task.

[0071] The artificial intelligence model can be trained as an on-device type and stored in the memory 130 of the cleaning robot 100, or stored in an external server. It will be described in detail below. Hereinafter, an embodiment in which the artificial intelligence model is stored in the cleaning robot 100 will be illustrated.

[0072] The processor 140 may generate a second image obtained by overlapping the restricted area of ​​the identified obstacle with the first image. The processor 140 may identify the positions of the structures and obstacles near the cleaning robot 100 based on the information included in the second image, and determine the driving direction and speed of the cleaning robot 100. The processor 140 may control the driver 110 to move the cleaning robot 100 according to the determined moving direction and speed.

[0073] The processor 140 may generate a first image of the bottom, which is characteristically divided from the captured image. The processor 140 may divide the bottom image using an image division method.

[0074] The processor 140 can generate a navigation map for driving the cleaning robot 100 based on the result of the sensor 110 detecting (sensing) the task area in which the object is arranged. The processor 140 can obtain recognition information about the object by applying the image of the object captured by the camera 120 to the trained artificial intelligence model. The processor 140 can generate a semantic map including information indicating the environment of the task area by mapping the area of ​​the object included in the navigation map with the recognition information of the object. The processor 140 can use the semantic map to perform the task of the cleaning robot based on the control command of the user. Therefore, the user who is provided with the semantic map can use the recognition information of the object to control the task of the cleaning robot 100 in various ways, so that the usability can be significantly improved.

[0075] The processor 140 can use the identification information of the object to obtain the identification information of the place included in the task area. The processor 140 can use the identification information of the place and the identification information of the object included in the task area to generate a semantic map including information indicating the environment of the task area. Therefore, the user can control the task to be performed by the cleaning robot 100 based on the provided semantic map, with reference to one or both of the identification information of each place and the identification information of the object.

[0076] The processor 140 may map the region of the object included in the navigation map with the recognition information of the object based on at least one of the position or form of the object according to the detection result of the object to generate a semantic map of the environment indicating the task area. Therefore, the object may be mapped to an accurate position relative to the navigation map to provide a semantic map.

[0077] The processor 140 may apply the image of the object captured by the camera 120 to the trained artificial intelligence model provided in the external server to obtain recognition information of the object. By using the artificial intelligence model, the recognition rate of the object can be significantly improved. In particular, by using the artificial intelligence model provided in the external server, the limitation of the resources of the cleaning robot 100 can be overcome, so more resources can be used to improve the availability of the artificial intelligence model.

[0078] The processor 140 may identify a boundary of an object corresponding to an object in the navigation map. The processor 140 may map an area of ​​the object determined by the boundary of the object with the recognition information of the object to generate a semantic map of the environment indicating the mission area.

[0079] The processor 140 may apply the image of the object captured by the camera 120 to trained artificial intelligence provided in an external server and obtain recognition information of the object.

[0080] The processor 140 may control at least one sensor selected based on the recognition information of the object among the plurality of sensors included in the sensor 110 to detect (sense) the object. The processor 140 may obtain additional information about the object using the detection result of the at least one sensor.

[0081] The processor 140 may set priorities for the plurality of sensors according to the recognition information of the object. The processor 140 may obtain additional information about the object using a result detected by at least one sensor according to the priority among the plurality of sensors.

[0082] The processor 140 may control the camera 120 to capture an object near the cleaning robot 100. The processor 140 may apply the captured image to the trained artificial intelligence model to obtain recognition information of the object included in the image. The processor 140 may control at least one sensor selected based on the recognition information of the object among the multiple sensors included in the sensor 110 to detect the object. The processor 140 may use the result detected by at least one sensor to obtain additional information about the object, and determine the task to be performed by the cleaning robot 100 for the object based on the additional information about the object.

[0083] The processor 140 may set priorities for a plurality of sensors included in the sensor 110 according to the recognition information about the object. The processor 140 may obtain additional information about the object by using a result detected by at least one sensor according to the priority.

[0084] When a higher priority is given to the IR stereo sensor among the plurality of sensors according to the recognition information about the object, the processor 140 may give a weighted value to the result detected by the IR stereo sensor and obtain additional information about the object.

[0085] The processor 140 may identify a bounding box regarding the recognized object, and lower a threshold of the IR stereo sensor for an area where the recognition result of the bounding box does not overlap with the object detection result by the IR stereo sensor.

[0086] When the LIDAR sensor among the plurality of sensors included in the sensor 110 is given a higher priority according to the recognition information of the object, the processor 140 may give a weighted value to the result detected by the LIDAR sensor and obtain additional information about the object.

[0087] When an ultrasonic sensor among a plurality of sensors included in the sensor 110 is given a higher priority according to recognition information of an object, the processor 140 may give a weighted value to a result detected by the ultrasonic sensor to obtain additional information about the object.

[0088] When the priority of the ultrasonic sensor among the plurality of sensors included in the sensor 110 is set higher according to the recognition information of the object, the recognition object may be transparent or black.

[0089] The processor 140 may apply the captured image to a trained artificial intelligence model provided in an external server and obtain recognition information of the object.

[0090] Figure 3 is a detailed block diagram explaining the configuration of a cleaning robot according to an embodiment of the present disclosure.

[0091] refer to Figure 3 The cleaning robot 100 may include a sensor 110, a camera 120, a memory 130, a communicator 150, a dust collection unit 160, a driver 170, a power supply 180, and a processor 140 electrically connected to the above constituent elements.

[0092] The sensor 110 , the camera 120 , the memory 130 , and the processor 140 have been described, and thus repeated descriptions will be omitted.

[0093] The communicator 150 may send and / or receive data, control commands, etc. to and / or from an external device. For example, the communicator 150 may receive partial or entire map information about location information of a space including the cleaning robot 100 operating from an external device. The communicator 150 may send information for updating the entire map information to an external device. As another example, the communicator 150 may receive a signal for controlling the cleaning robot 100, which is sent by a user using a remote control device. The remote control device may be implemented in various forms (such as a remote controller, a mobile device, etc.).

[0094] The communicator 150 may send and / or receive data to and / or from an external server (not shown). For example, when the artificial intelligence model is stored in the external server, the communicator 150 may send an image captured by the camera 120 to the external server, and receive recognition information about an object (e.g., information about an obstacle) recognized by using the artificial intelligence model stored in the external server. However, the present disclosure is not limited thereto, but the communicator 150 may receive information about a movable area of ​​a space in which the cleaning robot 100 performs a task from an external server.

[0095] The communicator 150 may include a communication interface using various methods such as near field communication (NFC), wireless local area network (LAN), infrared (IR) communication, Zigbee communication, WiFi, Bluetooth, etc. as a wireless communication method.

[0096] The dust collection unit 160 may be configured to collect dust. Specifically, the dust collection unit 160 may inhale air and collect dust in the inhaled air. For example, the dust collection unit 160 may include a motor for passing air from an inlet through a duct to an outlet, a filter for filtering dust in the inhaled air, and a dust basket for collecting the filtered dust.

[0097] The driver 170 may be configured to drive the movement of the cleaning robot 100. For example, the driver 170 may move the cleaning robot 100 to a position to perform a task under the control of the processor 140. In this case, the driver 170 may include at least one wheel contacting the bottom, a motor for providing a driving force to the wheel, and a driver for controlling the motor. As another example, the driver 170 may be operated to perform a task. In the case of an object moving task, the driver 170 may include a motor for performing operations such as picking up an object.

[0098] The power supply 180 can provide the power required to drive the cleaning robot 100. For example, the power supply 180 can be embodied as a battery that can be charged or discharged. When the remaining power level is equal to or less than a predetermined level, or the task is completed, the processor 140 can control the driver 110 to move to the charging station. The power supply 180 can be charged using at least one of a contact method or a non-contact method.

[0099] Figure 4A , Figure 4B , Figure 4C and Figure 4D is a view explaining how the cleaning robot according to an embodiment of the present disclosure obtains additional information about an object based on a detection result through an IR stereo sensor.

[0100] The cleaning robot 100 can detect an object in front of itself through the IR stereo sensor. The cleaning robot 100 can identify an object at the bottom in front of it through the camera 120. The cleaning robot 100 can detect the object through the IR stereo sensor and obtain length, height and / or depth information about the object. Figure 4A , the cleaning robot 100 can recognize and detect multiple objects 410 and 420 in front of it through the camera 120 and the IR stereo sensor, and obtain length, height or depth information about the objects 410 and 420, and the objects 410 and 420 may be a flower pot 410 and a carpet 420.

[0101] When one of the length, height or depth of the object is greater than the threshold, the IR stereo sensor can detect the object, but when one of the length, height or width depth of the object is less than the threshold, the IR stereo sensor cannot detect the object. For example, when the height information of the carpet 420 is less than a predetermined threshold, the cleaning robot 100 may not detect the carpet 420. Figure 4B , the cleaning robot 100 can obtain depth information 411 about the flower pot 410 through the IR stereo sensor.

[0102] The camera 120 may detect and recognize the objects 410 and 420 in front of it regardless of height or depth information about the objects. Figure 4C, the cleaning robot 100 may capture and recognize images of the flower pot 410 and the carpet 420 , respectively, and determine bounding boxes 412 and 422 regarding the corresponding objects 410 and 420 .

[0103] refer to Figure 4B and Figure 4C , when the recognition result of the object by the camera 120 is different from the detection result of the object by the IR stereo sensor, the cleaning robot 100 may set the threshold of the IR stereo sensor to be smaller. The cleaning robot 100 may lower the threshold of the IR stereo sensor until the object recognition result by the camera 120 is consistent with the object detection result by the IR stereo sensor, for example, until the number of objects is the same. When the threshold of the IR stereo sensor is significantly lowered, the reference Figure 4D , the cleaning robot 100 can not only obtain depth information 411 about the flower pot 410, but also obtain size information (such as depth information) 421 about a specific object (such as a carpet 420) as additional information through the IR stereo sensor.

[0104] refer to Figure 4C , the cleaning robot 100 may lower the threshold of the IR stereo sensor for the areas corresponding to the obtained bounding boxes 412 and 422. The cleaning robot 100 may compare the object recognition result with the object detection result by the IR stereo sensor, and lower the threshold of the IR stereo sensor for the different areas 422. According to the above embodiment, the cleaning robot 100 may not only detect the carpet 420 that is not detected by the IR stereo sensor, but also maintain the existing threshold when detecting a new object.

[0105] Figure 5A , Figure 5B and Figure 5C It is a view explaining how the cleaning robot obtains additional information about an object based on the detection result through the LIDAR sensor.

[0106] refer to Figure 5A , the cleaning robot 100 can capture and recognize the object 500 near the cleaning robot 100 through the camera 120. As a result of recognizing the object, the cleaning robot 100 can determine that the recognized object 500 is a specific type of object, such as a table or a chair. When the object is a table or a chair, the cleaning robot 100 can detect the object by giving priority to the LIDAR sensor over other sensors as an additional sensor for obtaining additional information about the object.

[0107] Figure 5B5 is a plan view showing that the cleaning robot 100 recognizes the object 500 and determines the bounding box 510 relative to the object 500. As described above, the cleaning robot 100 can obtain detailed information of the object 500 as additional information through the LIDAR sensor, which is given a higher priority than another sensor. Specifically, when the cleaning robot 100 emits a laser pulse in the direction of the object recognized by the cleaning robot 100, the LIDAR sensor can obtain information about the bridge of the object 500.

[0108] refer to Figure 5C , the cleaning robot 100 can obtain information 520 about a part of the object 500 (e.g., the legs of the object 500) through the LIDAR sensor. In other words, the cleaning robot 100 can obtain information about the position of the legs of the object 500 and the space between the positions of the legs, and determine an appropriate task.

[0109] Fig. 6A and Figure 6B It is a view explaining that the cleaning robot obtains additional information about an object based on a detection result through an ultrasonic sensor.

[0110] refer to Fig. 6A , when the object is the black table 600 or the transparent glass 610, the IR stereo sensor or the LIDAR sensor may not detect the object. When the object is recognized as the black table 600 and the glass 610 in the image obtained by the camera 120, the cleaning robot 100 can detect the object by giving the ultrasonic sensor a higher priority than the IR stereo sensor or the LIDAR sensor.

[0111] The cleaning robot 100 may rotate right or left and attempt to detect an object because the ultrasonic sensor has a limited detection range. Figure 6B , the cleaning robot 100 can obtain additional information about the object, such as the position of the legs of the black table 600 and / or the position of the glass 610 through ultrasound by entering a distance that the ultrasonic sensor can detect.

[0112] The cleaning robot according to the embodiment may have an advantage of detecting an object that is difficult to detect through an IR stereo sensor and a LIDAR sensor.

[0113] Fig. 7A and Figure 7B 2 is a view showing a structure in which the cleaning robot recognizes a house.

[0114] refer to Fig. 7A, the cleaning robot 100 can use the result of detecting the object by using the camera 120 or at least one of the plurality of sensors to recognize the objects 700 and 710, which may be doors. The cleaning robot 100 can recognize the object as a door based on the result of inputting an image including the doors 700 and 710 to an artificial intelligence model.

[0115] When the detected object is a door, the cleaning robot 100 can determine the structure of the task area (e.g., the structure of the house) through the door. For example, the cleaning robot 100 can determine the two sides in the direction of the horizontal doors 700 and 710 as walls unless there are exceptions.

[0116] refer to Figure 7B , the cleaning robot 100 may determine two directions horizontal to the identified first door 700 as the first wall 701, and determine two directions horizontal to the identified second door 710 as the second wall 711. The cleaning robot may determine the portion 720 where the first wall 701 and the second wall 711 intersect as the edge or corner of one or more walls of the house.

[0117] refer to Figure 7C , the cleaning robot 100 may detect an empty space 730 of a portion where the first wall 701 and the second wall 711 are expected to cross each other.

[0118] For example, when an additional area 740 exists in a portion where two walls are expected to intersect each other, the cleaning robot 100 can use the LIDAR sensor to detect the additional area 740. Based on the detection results, the cleaning robot 100 can determine that there is a space between the first wall 701 and the second wall 711, and include the additional area 740 in the portion of the structure of the task area.

[0119] When the object is recognized as a door, the cleaning robot 100 according to the embodiment can recognize not only the object but also the structure of the mission area. In addition, the cleaning robot 100 according to the embodiment can generate a semantic map that reflects the structure of the mission area into the navigation map.

[0120] Fig. 8A and Figure 8B is a view explaining that a cleaning robot generates a semantic map based on a structure of a house and additional information about an object according to an embodiment of the present disclosure.

[0121] refer to Fig. 8A , the cleaning robot 100 may recognize objects in the house as doors 700 and 710 and a sofa 800 .

[0122] like Fig. 7A and Figure 7BAs explained in , the cleaning robot 100 can recognize the structure of the task area (e.g., the wall in the house) through the doors 700 and 710. The cleaning robot 100 can recognize the sofa 800 while driving toward the object. Fig. 8A , the cleaning robot 100 can obtain multiple images 801 to 803 of the object based on the detection result of the camera 120 or at least one of the multiple sensors when driving toward the object. For example, the cleaning robot 100 can capture the object at a predetermined distance interval (e.g., 20 cm to 40 cm) or a time interval period (e.g., 0.5 seconds to 2 seconds) and obtain the images 801 to 803 to store the images in the memory 130.

[0123] When the cleaning robot 100 returns to the charging station after completing a task or returns to the charging station for charging, the cleaning robot 100 can obtain recognition information of the object from the images 801 to 803 stored in the memory 130. The cleaning robot 100 can apply the stored images 801 to 803 to the artificial intelligence model to obtain recognition information about the object.

[0124] When the additionally recognized object belongs to a specific type of object (eg, the sofa 800 ) that is not an obstacle or a foreign object, the cleaning robot 100 may add this information to the navigation map relative to the task area and generate a semantic map.

[0125] For example, refer to Figure 8B , the cleaning robot 100 can generate a semantic map based on the sofa leg information 810 additionally obtained through the LIDAR sensor about the sofa 800 (which is a structure) and the structure of the task area inferred through the doors 700 and 710.

[0126] Fig. 9A and Fig. 9B is a view explaining that the cleaning robot notifies the user that there is a hazardous object on the floor.

[0127] refer to Fig. 9A , the cleaning robot 100 may recognize the object on the floor as a broken glass 900. The cleaning robot 100 may input an image including the object 900 into an artificial intelligence model and perform object recognition to recognize the object as a broken glass.

[0128] The cleaning robot 100 can notify the user that there is a dangerous object on the floor. The cleaning robot 100 can send alarm data to the user terminal device 90, so that an alarm statement 911 can be displayed on the user terminal device 910, for example, "something that shouldn't be here is on the floor" or "there is something that shouldn't be here". In addition, the cleaning robot 100 can send alarm data to the user terminal device 910, so that an alarm statement including identification information of the object is used as alarm data. For example, when the object is identified as a glass, the cleaning robot 100 can send alarm data to the user terminal device 910, so that an alarm statement (for example, "the glass is on the floor") can be displayed.

[0129] The cleaning robot 100 can 7A to 8B The navigation map or semantic map generated by the method shown is sent to the user terminal device 910. The user terminal device 910 can display the semantic map or navigation map on the user terminal device 910, and display the location information of the dangerous object received from the cleaning robot 100.

[0130] For example, refer to Fig. 9B , the user terminal device 910 can display a user interface (UI) 912 about the navigation map received from the cleaning robot 100, and display the location of the dangerous object 900 detected by the cleaning robot 100 on the UI.

[0131] Therefore, the user can easily recognize whether a dangerous object has fallen on the floor, or whether there is a dangerous object.

[0132] Fig.10 is a view explaining that the cleaning robot designates an area not to be cleaned according to an embodiment of the present disclosure.

[0133] The user terminal device 910 may receive a navigation map from the cleaning robot 100 and display a UI 912 thereof.

[0134] The user can specify an area 913 that is not to be cleaned by the cleaning robot 100. For example, when it is necessary to restrict access to a specific area 930 on the navigation map (for example, when a baby is sleeping), the user can instruct the cleaning robot 100 not to clean the specific area 930 by interacting (for example, touching, clicking, etc.) with the specific area 913 displayed on the user terminal device 910. For example, the touch can occur as a user input on a touch screen display device.

[0135] The cleaning robot 100 can perform a task of automatically avoiding entering a restricted specific area 913 without receiving (multiple) user commands. For example, before entering the specific area 913, the cleaning robot 100 can use an artificial intelligence model to identify an object (e.g., a sleeping baby) included in the specific area 913. As a result of identifying a specific object (e.g., a sleeping baby), if it is determined that the specific area 913 needs to be restricted, the cleaning robot 100 can perform the task while driving and avoiding the specific area 913.

[0136] Fig.11A and Fig. 11B is a block diagram illustrating a training module and a recognition module according to various embodiments of the present disclosure.

[0137] refer to Fig.11A , the processor 1100 may include at least one of a training module 1110 and a recognition module 1120 . Fig.11A The processor 1100 may correspond to the processor 140 of the cleaning robot 100 , or to a processor of an external server (not shown) capable of communicating with the cleaning robot 100 .

[0138] The training module 1110 may generate and train a recognition model having a predetermined standard for determining a situation. The training module 1110 may generate a recognition model having a determined standard using the collected training data.

[0139] The training module 1110 may generate, train, or renew an object recognition model having a criterion for determining which object is included in an image by using an image including an object as training data.

[0140] The training module 1110 may generate, train, or update a peripheral information recognition model having a standard for determining various additional information near an object included in an image by using peripheral information included in a screen including an object as training data.

[0141] The training module 1110 may generate, train, or update an obstacle recognition model having a criterion for determining an obstacle included in an image by using an image captured by a camera as training data.

[0142] The recognition module 1120 may use predetermined data as input data of a trained recognition model and assume an object to be recognized in the predetermined data.

[0143] For example, the recognition module 120 may obtain (or assume, infer, etc.) object information about an object included in an object region by using the object region (or image) including the object as input data of a trained recognition model.

[0144] For another example, the recognition module 1120 may apply the object information to the trained recognition model to assume (or determine, infer, etc.) the search category to provide search results. The search results may include a plurality of search results according to priority.

[0145] At least part of the training module 1110 or the recognition module 1120 may be implemented as a software module, or in the form of at least one hardware chip installed on an electronic device. For example, at least one of the training module 1110 and the recognition module 1120 may be manufactured in the form of a hardware chip used only for artificial intelligence (AI), or as part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics-specific processor (e.g., GPU) installed on the cleaning robot 100. The hardware chip used only for artificial intelligence (AI) may be a processor dedicated to probability calculation, which has higher parallel processing performance than a conventional general-purpose processor, thereby quickly performing arithmetic operations in the field of artificial intelligence (such as machine training). When the training module 1110 or the recognition module 1120 is implemented as a software module (or a program module including instructions), the software module may be a non-temporary computer-readable medium that is computer-readable. In this case, the software module may be provided by an operating system (OS), or provided by a predetermined application. Alternatively, some of the software modules may be provided by an operating system (OS), while others of the software modules may be provided by a predetermined application.

[0146] The training module 1110 and the recognition module 1120 may be installed on one electronic device, or may be installed on each of the electronic devices. For example, one of the training module 1210 and the recognition module 1320 may be included in the cleaning robot 100, and the other may be included in an external server. In addition, the training module 1110 and the recognition module 1120 may be connected in a wired / wireless manner to provide the model information established by the training module 1110 to the recognition module 1120, and the data input to the recognition module 1120 may be provided to the training module 1110 as additional training data.

[0147] Fig. 11B 11 is a block diagram explaining a training module 1110 and a recognition module 1120 according to various embodiments of the present disclosure.

[0148] refer to Fig. 11B According to part (a), the training module 1110 of the embodiment may include a data acquisition component 1110-1 and a model training module 1110-4. The training module 1110 may selectively include at least one of a training data preprocessor 1110-2, a training data selector 1110-3, or a model evaluation module 1110-5.

[0149] The training data acquisition unit 1110-1 may obtain the training data required for inferring the recognition model of the object to be recognized. The training data acquisition unit 1110-1 may obtain the entire image of the object, the image corresponding to the object area, and the object information as the training data. The training data may be data collected or tested by the training module 1110 or the manufacturer of the training module 1110.

[0150] The model training module 1110-4 can use the training data to train the recognition model to have a predetermined standard for determining how to determine the object to be recognized. For example, the model training module 1110-4 can use at least part of the training data as a determination standard to train the recognition model through supervised learning. For example, the model training module 1110-4 can train itself by using the training data without additional supervised learning, and train the recognition model through unsupervised learning to find a determination standard for determining a situation.

[0151] In addition, for example, the model training module 1110-4 can train the recognition model through reinforcement learning using feedback on whether the result of the training determination situation is appropriate. For example, the model training module 1110-4 can train the recognition model using a training algorithm including an error back propagation method or a gradient descent method.

[0152] The model training module 1110 - 4 may train determination criteria based on which the training data will be used to predict objects identified using the input data.

[0153] When there are multiple established recognition models, the model training module 1110-4 can determine the recognition model with greater correlation between the input training data and the basic training data as the recognition model. In this case, the basic training data can be classified according to the data type, and the recognition model can be pre-established according to the data type. For example, the basic training data can be pre-classified based on various criteria (such as at least one of the area where the training data is generated, the time when the training data is generated, the size of the training data, the type of the training data, the creator of the training data, or the type of the object in the training data, etc.).

[0154] When the recognition model is trained, the model training module 1110-4 can store the trained recognition model. The model training module 1110-4 can store the trained recognition model in the memory 130 of the cleaning robot 100. The model training module 1110-4 can store the trained recognition model in the memory of the server connected to the cleaning robot 100 in a wired / wireless manner.

[0155] The training module 1110 may further include a training data preprocessor 1110 - 2 and a training data selector 1110 - 3 for improving the results of analyzing the recognition model or saving resources or time required for generating the recognition model.

[0156] The training data preprocessor 1110-2 may preprocess the acquired data so that the acquired data may be used for training to determine the situation. The training data preprocessor 1110-2 may produce the acquired data in a predetermined format so that the model training module 1110-4 may use the acquired data for training to determine the situation.

[0157] The training data selector 1110-3 can select the data obtained from the training acquisition component 1110-1, or the data preprocessed by the training data preprocessor 1110-2 as the data required for training. The selected training data can be provided to the model training module 1110-4. The training data selector 1110-3 can select the training data required for training from the obtained or preprocessed data according to a predetermined standard. In addition, the training data selector 1110-3 can select the training data according to a predetermined standard through the training of the model training module 1110-4.

[0158] The training module 1110 may also include a model evaluation module 1110 - 5 to improve the analysis results of the data recognition model.

[0159] When the evaluation data is input to the recognition model, but the result output from the evaluation data analysis does not meet the predetermined standard, the model evaluation module 1110-5 may cause the model training module 1110-4 to train again. The evaluation data may be predefined data for evaluating the recognition model.

[0160] For example, when the number or ratio of evaluation data that is not accurately analyzed in the analysis result of the trained recognition model on the evaluation data exceeds a predetermined threshold, the model evaluation module 1110 - 5 may evaluate that the data does not meet the predetermined standard.

[0161] When the trained recognition model includes a plurality of trained recognition models, the model evaluation module 1110-5 may evaluate whether each trained recognition model satisfies a predetermined criterion, and determine the model satisfying the predetermined criterion as the final recognition model. In this case, when the recognition model satisfying the predetermined criterion includes a plurality of recognition models, the model evaluation module 1110-5 may determine any one or a predetermined number of recognition models preset in order of high evaluation scores as the final recognition model.

[0162] refer to Fig. 11B According to part (b), the recognition module 1120 according to some embodiments may include a data acquisition component 1120-1 and a recognition result provider 1120-4.

[0163] The recognition module 1120 may selectively include at least one of a recognition data preprocessor 1120 - 2 , a recognition data selector 1120 - 3 , and a model updating module 1120 - 5 .

[0164] The recognition data acquisition module 1120-1 can obtain data required for situation determination. The recognition result provider 1120-4 can apply the data obtained from the recognition data acquisition module 1120-1 to the trained recognition model to determine the situation. The recognition result provider 1120-4 can provide analysis results according to the analysis purpose of the data. The recognition result provider 1120-4 can apply the data selected by the recognition data preprocessor 1120-2 or the recognition data selector 1120-3 as an input value to the recognition model to obtain the analysis result. The analysis result can be determined by the recognition model.

[0165] For example, the recognition result provider 1120 - 4 may apply an object region including an object obtained from the recognition data acquisition module 1120 - 1 to the trained recognition model, and obtain (or assume) object information corresponding to the object region.

[0166] For another example, the recognition result provider 1120 - 4 may apply at least one of the object region, object information, or context information obtained from the recognition data acquisition module 1120 - 1 to the trained recognition model to obtain (or assume) a search category to provide search results.

[0167] The recognition module 120 may further include a recognition data preprocessor 1120 - 2 and a recognition data selector 1120 - 3 to improve the analysis result of the recognition model or save resources or time for providing the analysis result.

[0168] The recognition data preprocessor 1120-2 may preprocess the acquired data so that the acquired data for situation determination may be used. The recognition data preprocessor 1120-2 may make the acquired data in a predefined format so that the recognition result provider 1120-4 may use the acquired data for situation determination.

[0169] The recognition data selector 1120-3 may select data obtained from the recognition data acquisition module 1120-1 or data preprocessed by the recognition data preprocessor 1120-2 as data required for situation determination. The selected data may be provided to the recognition result provider 1120-4. The recognition data selector 1120-3 may select part or all of the obtained or preprocessed data according to a predetermined standard for situation determination. The recognition data selector 1120-3 may select data according to a standard preset by the training of the model training module 1110-4.

[0170] The model update module 1120-5 may control to update the recognition model based on the analysis result provided by the recognition result provider 1120-4. For example, the model update module 1120-5 may provide the analysis result provided by the recognition result provider 1120-4 to the model training module 1110-4 to request the model training module 1110-4 to additionally train or update the recognition model.

[0171] Fig.12 is a view showing an example in which the cleaning robot 100 and the server 200 may operate in association with each other to train and recognize data.

[0172] refer to Fig.12 , the server 200 may train a standard for situation determination, and the cleaning robot 100 may determine the situation based on the training result.

[0173] The model training module 1110-4 of the server 200 can execute Fig. 11B Functions of the model training module 1110-4 shown. The model training module 1110-4 of the server 200 can train which object image, object information or content information will be used to determine a predetermined situation, or a standard of how to determine a situation by using data.

[0174] The recognition result provider 1120-4 of the cleaning robot 100 may apply the data selected by the recognition data selector 1120-3 to the recognition model generated by the server 200 to determine the object information or the search category. The recognition result provider 1120-4 of the cleaning robot 100 may receive the recognition model generated by the server 200 from the server 200 and use the received recognition model to determine the situation. In this case, the recognition result provider 1120-4 of the cleaning robot 100 may apply the object image selected by the recognition data selector 1120-3 to the recognition model received from the server 200 to determine the object information corresponding to the object image. The recognition result provider 1120-4 may determine the search category for obtaining the search results by using at least one of the context information or the context recognition information.

[0175] Fig.13 is a flowchart explaining a network system using a recognition model according to an embodiment of the present disclosure.

[0176] The first component 1301 may be a cleaning robot 100, and the second component 1302 may be a server 200 storing a recognition model. The first component 1301 may be a general-purpose processor, and the second component 1302 may be an artificial intelligence dedicated processor. The first component 1301 may include at least one application, and the second component 1302 may include an operating system (OS). The second component 1302 may be more integrated, more dedicated, have shorter latency, better performance, or have more resources than the first component 1301 to process the calculations required to generate, update, or apply the data recognition model.

[0177] refer to Fig.13 In step S1311, the first component element 1301 may generate a captured image (e.g., a captured image) by capturing a surrounding environment including an object (e.g., an object arranged in a task area). For example, the first component element 1301 may include a camera that captures the captured image. In step S1312, the first component element 1301 may send the captured image to the second component element 1302. The first component element 1301 may send information about an object area corresponding to the selected object together with the captured image.

[0178] In step S1313, the second component element 1302 may divide the received captured image into the object area and the peripheral area. The second component element 1302 may divide the image into the object area and the peripheral area based on the received information about the object area.

[0179] In step S1314, the second component 1302 can obtain object information and additional information about the object by inputting the separated object area and the peripheral area into the recognition model. The second component 1302 can obtain object information by inputting the object area into the object recognition model, and obtain additional information about the object by inputting the peripheral area into the peripheral information recognition model. In addition, the second component 1302 can determine the search category and the priority of the search category based on the object information and the additional information about the object.

[0180] In step S1315, the second component 1302 can obtain a result related to the object by using the obtained object information and additional information. The second component 1302 can apply the object information and additional information as input data to the recognition model and obtain a result related to the object. The second component 1302 can obtain the result by using the search category. The second component 1302 can obtain the result by using additional data (for example, the risk level of the obstacle, and / or the importance of the obstacle relative to the user) in addition to the object information and additional information. The additional data can be sent from the first component 1301 or other components, or be pre-stored in the second component 1302.

[0181] When the second component element 1302 transmits the result associated with the object to the first component element 1301 in step S1316 , the first component element 1301 may detect the object through the sensor based on the received result associated with the object in step S1317 .

[0182] Fig.14 is a flowchart explaining an example in which the cleaning robot provides a search result of a first area using a recognition model according to an embodiment of the present disclosure.

[0183] refer to Fig.14 In step S1410, the cleaning robot 100 may generate an image by capturing or photographing the surrounding environment of the cleaning robot 100. The cleaning robot 100 may obtain first information about a first area by using a first model trained using the generated image as input data. The first model may be stored in the cleaning robot 100, but is not limited thereto. The first model may be stored in an external server.

[0184] In step S1430, the cleaning robot 100 may obtain second information about the second area through a trained second model using the first information and the generated image as input data. The first model may be stored in the cleaning robot 100, but is not limited thereto. The first model may be stored in an external server.

[0185] When the first model and the second model are stored in an external server, the cleaning robot 100 can send the generated image to the external server, and the external server can input the image to the first model to receive the first information, and input the image and the first information to the second model to receive the second information.

[0186] Therefore, by obtaining second information about the second area (which may be an area near the first area) and first information about the first area where the user input is detected, information about the first area may be obtained more accurately.

[0187] Fig.15 is a flowchart explaining a system using a recognition model according to an embodiment of the present disclosure.

[0188] refer to Fig.15 In step S1510, the cleaning robot 100 may generate an image by capturing or photographing the surrounding environment. In step S1520, the cleaning robot 100 may obtain first information about the first area through a trained first model using the generated image as input data.

[0189] In step S1530 , the cleaning robot 100 may transmit the generated image and the first information to the server 200 .

[0190] In step S1540, the server 200 may obtain second information about the second area through a trained second model using the first information and the generated image as input data.

[0191] In step S1550, the server 200 may retrieve information about the first area based on the first information and the second information.

[0192] In step S1560, the server 200 may send information about the first area (e.g., search results related to the first area) to the cleaning robot 100, and in step S1570, for example, the cleaning robot 100 may provide the received information (e.g., search results) by causing a display to display the received search results.

[0193] An operation of obtaining first information through a first model for recognizing an object may be performed by the cleaning robot 100, or an operation of obtaining second information through a second model for assuming context information may be performed by the server 200. In other words, an object recognition operation for processing information with a small amount of processing may be performed by the cleaning robot 100, and a context estimation operation with a large amount of processing may be performed by the server 200.

[0194] refer to Fig.15 , the server 200 may obtain the first information or the second information through the trained model and retrieve information related to the first area, but is not limited thereto. Multiple servers may each perform the above operations. That is, the first server may obtain the first information and the second information through the trained model, and the second server may retrieve information about the first area based on the first information and the second information obtained by the first server, but is not limited thereto. All processes performed by the server 200 may be performed by the cleaning robot 100.

[0195] Fig.16 It is a view explaining generation of a semantic map according to an embodiment of the present disclosure.

[0196] refer to Fig.16 , the cleaning robot 100 can be relative to Fig.16 The task area in part (a) is generated as follows Fig.16 For example, the cleaning robot may detect the task area using at least one of an IR stereo sensor, an ultrasonic sensor, a LIDAR sensor, a position sensitive diode (PSD) sensor, or an image sensor. The cleaning robot may generate a navigation map for driving the cleaning robot by using the detection result of the task area. Preferably, Fig.16 A 2-D (two-dimensional) navigation map is generated in. For example, the navigation map can be displayed as being divided by at least one 2-D line.

[0197] The cleaning robot 100 can detect objects in the task area relative to the task area using at least one of the camera 120, the object recognition sensor, the IR stereo sensor, the ultrasonic sensor, the LIDAR sensor, or the image sensor. The cleaning robot can apply the result of detecting the object to the trained artificial intelligence model and obtain recognition information of the object. For example, the result of detecting the object may include a captured image of the object, depth information of the object, material information of the object, and / or a reflection coefficient of the object, but is not limited thereto.

[0198] The cleaning robot 100 can obtain Fig.16 The name of each of the one or more objects in part (c) is used as the identification information of the object. In addition, the cleaning robot 100 can obtain at least one of the type of the object, the size of the object (e.g., the height of the object, the width of the object, the depth of the object, etc.), or the characteristics of the object (e.g., the color of the object, the material of the object, etc.), but is not limited thereto.

[0199] The cleaning robot 100 can generate a semantic map of environmental information indicating a task area where the cleaning robot 100 performs a task by mapping the area of ​​the object included in the navigation map with the identification information of the object. The cleaning robot 100 can identify the boundary of the object corresponding to the object in the navigation map. When the boundary of the object is identified, the cleaning robot 100 can determine the area of ​​the object by the boundary of the object.

[0200] When the area of ​​the object is determined, the cleaning robot 100 may map the area of ​​the object with the recognition information of the object.

[0201] The cleaning robot 100 can map the area of ​​the object included in the navigation map with the identification information of the object based on the detection result of the object and the position of the object. When the position of the object when the object is detected to generate the navigation map (for example, when the detection result of the object is obtained) and the position of the object when the object is detected for obtaining the identification information of the object (for example, when the detection result of the object is obtained, or when the image of the object is stored) are the same or similar to each other within a threshold range, the cleaning robot 100 can map the area of ​​the object included in the navigation map with the identification information of the object.

[0202] The cleaning robot 100 may map the area of ​​the object included in the navigation map with the identification information of the object based on the form of the object according to the result of detecting the object. The cleaning robot 100 may compare the form of the object according to the result of detecting the object for generating the navigation map with the form of the object included in the identification information of the object, and when the two forms are similar or identical to each other, the area of ​​the object included in the navigation map may be mapped with the identification information of the object.

[0203] refer to Fig.16 In part (d), as a result of mapping the area of ​​the object with the recognition information of the object, the cleaning robot 100 can generate a semantic map of the environment indicating the task area. The cleaning robot 100 can display the generated semantic map on a display. When the cleaning robot 100 sends the generated semantic map to an external user terminal device, the user terminal device can display the semantic map through the display.

[0204] The user terminal device or the external server may generate a semantic map. For example, when the cleaning robot 100 sends a navigation map and object recognition information to the user terminal device or the external server, the user terminal device or the external server may generate a semantic map. As another example, when the cleaning robot 100 sends a navigation map and the result of detecting an object to the user terminal device or the external server, the user terminal device or the external server may generate a semantic map.

[0205] The identification information of the object can be displayed in the object area of ​​the semantic map as text (e.g., the name of the object) or an icon. The identification information of the object can be displayed in the object area of ​​the semantic map in a reference form. Specifically, the identification information of the object can be displayed with respect to the instruction line indicating the object area, or the area of ​​the object can be distinguished by color, so that the identification information of the object can be displayed based on color.

[0206] Fig.17 is a view illustrating a user interface using a semantic map according to an embodiment of the present disclosure.

[0207] refer to Fig.17 In part (a), the user terminal device 1700 may display a semantic map. For example, the semantic map may be displayed by Fig.16 For example, the recognition information of the object may be displayed in the semantic map in the context of at least one of a 3-D map or a navigation map (eg, a LIDAR map) reflecting the structure of the mission area.

[0208] For example, the bed 1701 may be displayed as identification information of the object in at least a portion of the object region of the semantic map. In this case, the user may select the bed 1701 as identification information.

[0209] In response to the user's selection, the user terminal device 1700 may display a drop-down box 1710 associated with the selected identification information 1701, such as Fig.17 . The name of the object representing the selected recognition information may be displayed in the drop-down box 1710. For example, as a result of applying the object detection result to the trained artificial intelligence model, the object recognition result may be recognized as bed 50%, sofa 30%, table 20%, etc. In this case, the names applicable to the objects may be sequentially displayed in the drop-down box 1710 in the order of the high recognition results. For example, in the list of the drop-down box 1710, the bed 1701 may be displayed in the first field, the sofa 1702 may be displayed in the second field, and the table 1703 may be displayed in the third field.

[0210] When the user selects a name (e.g., table 1703), the reference Fig.17 In part (c), the user terminal device 1700 can display the identification information 1703 of the changed object on the navigation map.

[0211] Fig.18 is a view illustrating a case where a cleaning robot is controlled by using a semantic map according to an embodiment of the present disclosure.

[0212] refer to Fig.18 , the user can execute the control command by using the identification information of the object displayed on the semantic map provided by the user terminal device 1700. For example, the Fig.16 and Fig.17 The identification information of at least one object displayed on the semantic map of the TV. In this case, the user can issue "Please clean the front of the TV" to specify the cleaning area.

[0213] According to the user's speech command, the user terminal device 1700 can recognize the user's speech command and send the user's control command corresponding to the recognized speech command to the cleaning robot 100. The control command can be a command requesting the execution of a task regarding a specific object area. The cleaning robot 100 can execute the task based on the user's control command. For example, the cleaning robot 100 can move toward the TV and clean the front of the TV.

[0214] Fig.19 It is a view explaining generation of a semantic map according to an embodiment of the present disclosure.

[0215] refer to Fig.19 , the cleaning robot 100 can be relative to Fig.19 The task area in part (a) is generated Fig.19 The detailed description of generating the navigation map may correspond to the description of Fig.16 Therefore, the repeated description will be omitted.

[0216] The cleaning robot 100 can be used in Fig.19 The result of detecting the task area in part (a) is used to obtain the identification information of each location in the task area. For example, the cleaning robot 100 can obtain Fig.19 The name of each place in part (c) is used as identification information.

[0217] The cleaning robot 100 may apply a plurality of images capturing various locations in the task area to a trained artificial intelligence model to obtain recognition information of each location in the task area.

[0218] refer to Fig. 7A and Figure 7B , the cleaning robot 100 can detect the object to identify the doors 700 and 710, and use the identified doors 700 and 710 to determine the structure of the task area. The cleaning robot 100 can obtain identification information of each location of the structure that is distinguished as the task area. For example, the cleaning robot 100 can recognize that the doors 700 and 710 are the most, identify the largest area as the living room, and identify the second largest area as the bedroom.

[0219] The cleaning robot 100 can obtain the identification information of each location in the task area by using the identification information of the objects at each location in the task area. For example, the cleaning robot 100 can determine the area with a table as the kitchen, the area with a bed as the bedroom, and the area with a TV or a sofa as the living room.

[0220] When the identification information of each location is obtained, the cleaning robot 100 can use the navigation map and the obtained identification information of each location to generate a navigation map such as Fig.19 The cleaning robot 100 may display the generated semantic map through a display. In addition, when the cleaning robot 100 sends the generated semantic map to an external user terminal device, the user terminal device may display the semantic map through a display.

[0221] The user terminal device or the external server may generate a semantic map. For example, when the cleaning robot 100 sends a navigation map and identification information of each location to the user terminal device or the external server, the user terminal device or the external server may generate a semantic map. As another example, when the cleaning robot 100 sends a navigation map and a result of detecting an object at each location included in the task area to the user terminal device or the external server, the user terminal device or the external server may generate a semantic map.

[0222] Fig. 20 is a view illustrating a user interface using a semantic map according to an embodiment of the present disclosure.

[0223] refer to Fig. 20 In part (a), the user terminal device 2000 may display a semantic map. For example, the semantic map may be displayed by Fig.19 The recognition information of the object on the semantic map can be displayed, for example, in the background of at least one of a 3D map or a navigation map reflecting the structure of the mission area.

[0224] For example, a living room 2001 may be displayed on at least one part of the object as identification information about a place. In this case, the user may select the living room 2001 as the identification information.

[0225] In response to the user's selection, Fig. 20 In part (b), the user terminal device 2000 may display a drop-down box 2010 associated with the selected identification information 2001, and the name of the place to be replaced with the selected identification information may be displayed in the drop-down box 2010. For example, as a result of applying an image captured at a place to a trained artificial intelligence model, the recognition result for a place may be 50% living room, 30% bedroom, and 20% study. In this case, according to the order of the recognition result values, the names applicable to a place may be sequentially displayed in the drop-down box 2010. For example, in the list of the drop-down box 2010, the living room 2001 may be displayed in the first field, the bedroom 2002 may be displayed in the second field, and the study 2003 may be displayed in the third field.

[0226] When the user selects a name (e.g., Study 2003), the reference Fig. 20In part (c), the user terminal device 2000 can display identification information 2003 of the changed location on the navigation map.

[0227] Fig.21 is a view illustrating a case where a cleaning robot is controlled by using a semantic map according to an embodiment of the present disclosure.

[0228] refer to Fig.21 , the user can execute a control command by using the identification information of each location displayed on the semantic map provided from the user terminal device 2000. For example, Fig.19 and Fig. 20 The identification information of each location on the semantic map in can be displayed on the display of the user terminal device 2000. In this case, the user can say "please clean the living room" to specify the cleaning area.

[0229] According to the user's speech command, the user terminal device 2000 can recognize the user's speech command and send a user control command corresponding to the recognized user's speech command to the cleaning robot 100. The user's control command can be a command requesting the execution of a task about a specific location. The cleaning robot 100 can execute the task based on the user's control command. For example, the cleaning robot 100 can move to the living room and clean the living room.

[0230] The user terminal device can display a semantic map, which includes Fig.17 The identification information of the object in part (a) and Fig. 20 In this case, when one of the identification information of the object and the identification information of each place is selected, the user terminal device may provide a user interface for changing the selected identification information.

[0231] For example, the user terminal device may provide a candidate list capable of changing the selected identification information. A variable name may be included in the candidate list, and the names may be arranged in order of high probability values ​​in consideration of the recognition result of the artificial intelligence model. When a user input for selecting a name is input, the user terminal device may change the identification information of the existing object to the selected name and display the information.

[0232] Fig. 22 is a view illustrating a process of recognizing an object according to an embodiment of the present disclosure.

[0233] The cleaning robot 100 may apply the image of the object captured by the camera 120 to the trained artificial intelligence model to obtain recognition information of the object. The cleaning robot 100 may apply the image of the place in the task area captured by the camera 120 to obtain recognition information of the place.

[0234] As another example, when the cleaning robot 100 captures an object in a specific area, the cleaning robot 100 can apply the captured image to the trained artificial intelligence model to obtain recognition information of the object and recognition information of the place. Specifically, an electronic device including an artificial intelligence model 2200 (e.g., an external server) can apply a feature map 2220 in the vertical direction generated by an object recognition network 2210 (convolutional network model) to a classifier, and execute an object recognition module 2230 for recognizing an object and a place recognition module 2240 for recognizing a place of an object. However, the training step can be simplified because both the loss for recognizing an object and the loss for recognizing a place are trained.

[0235] Relative to the captured image, when the recognition of both the object and the place are performed, more accurate recognition of the object is possible. For example, the recognition results of the object included in the captured image can be table 50%, dining table 30%, and table 20%. When the place with the object is identified as a kitchen, the electronic device 2300 can identify the object as a dining table instead of a table. As another example, when the object and the place with the object are identified as a table and a study, respectively, the object can be identified as a desk. As another example, when the object and the place with the object are identified as a front door and a door, respectively, the object can be identified as a front door. As another example, when the object and the place with the object are identified as a room and a door, respectively, the object can be identified as a door.

[0236] As another example, when an object and a place having the object are respectively identified as a threshold and a room, the object can be identified as a threshold. As another example, when an object and a place having the object are respectively identified as a threshold and a balcony, the object can be identified as a balcony threshold.

[0237] With respect to the captured image, when the object and the location including the object are identified together through a single network, the electronic device 2200 or the cleaning robot 100 can effectively generate a semantic map indicating the environment of the task area. For example, not only the identification information of the object, but also the location information of the object can be displayed on the semantic map.

[0238] Fig.23 is a view explaining a process of generating a semantic map according to an embodiment of the present disclosure.

[0239] When the cleaning robot 100 applies the image captured by the camera 120 to the network included in the electronic device 2200 , at least one of recognition information of the object or recognition information of the place (eg, context of the place) may be displayed on the semantic map.

[0240] For example, the cleaning robot 100 may apply the captured image to Fig. 22 The object recognition module 2230 in the embodiment of the present invention generates Fig.23 The first semantic map in part (a) of Fig.23 The cleaning robot 100 can obtain the identification information of the object in the first semantic map of part (a), and the name of the object can be displayed at the position corresponding to the object. Fig. 22 The location recognition module 2240 generates Fig.23 The name of the place can be displayed as identification information of the place in the divided area of ​​the task area in the semantic map, in the area corresponding to the place.

[0241] The cleaning robot 100 can Fig.23 The first semantic map of part (a) is Fig.23 The names and locations of objects present in the task area can be displayed together as the environment information of the task area. Fig.23 In the final semantic map of part (c).

[0242] Fig.24 is a view illustrating a case where a cleaning robot is controlled by using a semantic map according to an embodiment of the present disclosure.

[0243] refer to Fig.24 , the user can execute a control command by using the identification information of each object and place displayed on the semantic map provided from the user terminal device 2400. For example, Fig.23 The identification information of each object and place on the semantic map can be displayed on the display of the user terminal device 2400. In this case, the user can say "Please clean the front of the table in the living room" to specify the cleaning area.

[0244] According to the user's speech command, the user terminal device 2400 can recognize the user's speech command. The user terminal device 2400 can send a control command of the user corresponding to the recognized speech command to the cleaning robot 100. The control command can be a command to perform a task relative to an area of ​​a specific object located at a specific location. The cleaning robot 100 can perform a task based on the user's control command. For example, the cleaning robot 100 can move toward a table in a living room among a location (living room or study) with a table, and clean the front of the table.

[0245] In various embodiments, Fig.18 , Fig.21 and Fig.24In the embodiment of the present invention, the cleaning robot 100 can directly recognize the user's verbal command according to the user's verbal command. In this case, the cleaning robot 100 may include at least one of an automatic speech recognition (ASR) module, a natural language understanding (NLU) module, a path planner module, etc. for recognizing the user's verbal command. In this case, when the user speaks to specify the cleaning area, the cleaning robot 100 can recognize the user's verbal command by using at least one of the modules. The cleaning robot 100 can perform tasks according to the recognized user's control command. For example, the cleaning robot 100 can perform tasks according to the user's control command based on the environmental information (e.g., the name of the object, the name of the object, etc.) included in the stored semantic map.

[0246] refer to Fig.18 , Fig.21 and Fig.24 According to the user's speech command, the third voice recognition device (e.g., a home voice recognition hub, an artificial intelligence speaker, etc.) can recognize the user's speech command. The third voice recognition device may include at least one of an automatic voice recognition module, an automatic voice recognition module, a natural language understanding module, and a pathway planning module. The third voice recognition device can recognize the user's speech command and send the user's control command corresponding to the recognized speech command to the cleaning robot 100. Therefore, the cleaning robot 100 can perform tasks according to the user's control command while minimizing restrictions on locations or equipment.

[0247] Fig.25 is a view illustrating a configuration of a cleaning robot according to an embodiment of the present disclosure.

[0248] Fig.25 Part (a) is a perspective view showing a cleaning robot 100 including a plurality of sensors, and Fig.25 Part (b) is a front view showing the cleaning robot 100 including a plurality of sensors.

[0249] Fig.25 The cleaning robot 100 shown may include a plurality of sensors, and include at least one of an IR stereo sensor, a LIDAR sensor, an ultrasonic sensor, a 3D sensor, a material recognition sensor, a fall detection sensor, a position sensitive diode (PSD) sensor, and the like.

[0250] The functions of the IR stereo sensor, the LIDAR sensor, and the ultrasonic sensor have been described in detail with reference to FIG. 2, and their detailed description will be omitted. For example, the ultrasonic sensor may be composed of two light emitting sensor modules and two light receiving sensor modules. The cleaning robot 100 may detect the object on the front side using a 3D sensor and extract the three-dimensional shape of the object. Therefore, the cleaning robot 100 may obtain information about the size and distance of the object. The cleaning robot 100 may detect the object on the front side by using an object recognition sensor to obtain the type of the object. For example, the cleaning robot 100 may capture the object using a camera included in the object recognition sensor, and apply the captured image to a trained artificial intelligence model to obtain the type of the object as an application result. The cleaning robot 100 may use a fall detection sensor when driving forward and backward to identify the steps at the bottom (e.g., the steps between the bottom of the living room and the front door entrance). For example, the fall detection sensor may be arranged in front of or behind the cleaning robot. The cleaning robot 100 may use a PSD sensor to detect the position of an object within a close distance (e.g., within 15 cm). For example, the cleaning robot 100 may use a PSD sensor to detect an object, or drive along a wall to perform cleaning. For example, the PSD sensor may be disposed about 45 degrees outward in the forward direction, or arranged in the right and left surfaces of the cleaning robot 100 .

[0251] refer to Fig.25 , a plurality of sensors may be provided on the front surface, the rear surface, and the side surface of the cleaning robot 100. For example, at least one ultrasonic sensor 2501 to 2504, at least one 3D sensor 2511, at least one camera 2521 (e.g., an RGB camera), and at least one of an IR stereo sensor 2531 (e.g., an IR stereo sensor for docking), PSD sensors 2541 and 2542, a LIDAR sensor 2551, or a bumper sensor 2561 may be provided. In addition, another PSD sensor (not shown) may be provided on the side of the cleaning robot 100.

[0252] Fig.26A and Fig.26B 2 is a view illustrating a detection (sensing) range of a cleaning robot according to an embodiment of the present disclosure.

[0253] Fig.26A Part (a) is a plan view showing the cleaning robot 100, Fig.26A Part (b) is a side view showing the cleaning robot 100, and Fig.26B Parts (c) and (d) are perspective views showing the cleaning robot 100 .

[0254] refer to Fig.26A and Fig.26B, the cleaning robot 100 can detect the structure of an object or a house by using a plurality of sensors while driving.

[0255] refer to Fig.26A and Fig.26B , the detection range of the 3D sensor 2511 may be a first range 2510a, the detection range of the ultrasonic sensors 2501 to 2504 may be a second range 2500a, the detection ranges 2541 and 2542 of the front PSD sensor may be third ranges 2540a and 2540b, the detection range of the side PSD sensor 2543 (not shown) may be fourth ranges 2540c and 2540d, and the detection range of the LIDAR sensor 2551 may be a fifth range 2550a. The viewing angle of the camera 2521 may be a sixth range 2520a, but is not limited thereto. The detection range or the capture range may be predicted differently based on the specifications of the plurality of sensors or the locations where the plurality of sensors are installed.

[0256] The cleaning robot 100 can select at least one sensor from a plurality of sensors based on the recognition information of the object. The cleaning robot 100 can detect the object by using at least one selected sensor and obtain additional information about the object by using the detection result. The cleaning robot 100 can determine the task to be performed by the cleaning robot 100 based on the additional information about the object.

[0257] Fig. 27 , Fig.28 and Fig.29 is a flowchart explaining the cleaning robot according to the embodiment.

[0258] refer to Fig. 27 In step S2701, the cleaning robot 100 may generate a navigation map for driving the cleaning robot 100 by detecting a result of a task area in which an object is arranged using at least one sensor.

[0259] In step S2702, the cleaning robot 100 may obtain recognition information of the object by applying the image of the object captured by the camera 120 to the trained artificial intelligence model. The operation of obtaining the recognition information of the object may be performed before the cleaning robot 100 generates the navigation map in step S2701, or the recognition information of the object may be obtained when the navigation map is generated.

[0260] The cleaning robot 100 may obtain recognition information of the object by applying the image of the object captured by the camera 120 to a trained artificial intelligence model located in an external server.

[0261] When the recognition information of the object is obtained, the cleaning robot 100 may map the area of ​​the object included in the navigation map with the recognition information of the object and generate a semantic map indicating the environment of the task area in step S2703.

[0262] In step S2704, the cleaning robot 100 may perform the task of the cleaning robot based on the user's control command by using the semantic map. The user's control command may be a command requesting the execution of a task with respect to an object area or a specific place.

[0263] The cleaning robot 100 may obtain recognition information of each location in the task area. The cleaning robot 100 may generate a semantic map indicating an environment of the task area by using the obtained recognition information of each location and the recognition information of the mapped object.

[0264] The cleaning robot 100 may generate a semantic map indicating an environment of a task area by mapping an area of ​​an object included in a navigation map with recognition information of the object based on at least one of a position or a form of the object according to a result of detecting the object.

[0265] The cleaning robot 100 may identify a boundary of an object corresponding to an object in the navigation map. The cleaning robot 100 may map an area of ​​the object determined by the boundary of the object with the recognition information of the object to generate a semantic map indicating an environment of the task area.

[0266] The cleaning robot 10 may detect the object by using at least one sensor selected based on the recognition information of the object among a plurality of sensors included in the sensor 110. The cleaning robot 100 may obtain additional information about the object by using a result detected by at least one sensor.

[0267] The cleaning robot 100 may set priorities for the plurality of sensors according to the recognition information of the object. The cleaning robot 100 may obtain additional information about the object by using a result detected by at least one sensor among the plurality of sensors according to the priority.

[0268] Fig.28 is a flowchart explaining a cleaning robot according to another embodiment of the present disclosure;

[0269] refer to Fig.28 In step S2801, the cleaning robot 100 may generate a navigation map for driving the cleaning robot 100 by detecting a result of a task area using at least one sensor.

[0270] In step S2802 , the cleaning robot 100 may obtain recognition information of a place included in the task area by applying an image of the place captured by the camera 120 included in the task area to a trained artificial intelligence model.

[0271] When the identification information of the place included in the task area is obtained, in step S2803, the cleaning robot 100 can generate a semantic map indicating the environment of the task area by mapping the area corresponding to the place included in the navigation map with the identification information of the place.

[0272] In step S2804 , the cleaning robot 100 may use the semantic map to perform the task of the cleaning robot 100 based on the user's control command.

[0273] Fig.29 is a flowchart explaining a cleaning robot according to an embodiment of the present disclosure.

[0274] refer to Fig.29 , in step S2901, the cleaning robot 100 may capture an object near the cleaning robot 100.

[0275] In step S2902, the cleaning robot 100 may obtain recognition information of an object included in the image by applying the captured image to a trained artificial intelligence model. For example, the cleaning robot 100 may obtain recognition information of an object by applying the captured image to a trained artificial intelligence model located in an external server.

[0276] In step S2903 , the cleaning robot 100 may obtain additional information about the object using a result detected by at least one sensor selected based on the obtained recognition information of the object among the plurality of sensors.

[0277] For example, the cleaning robot 100 can obtain additional information about the object by selectively using the result detected by at least one sensor selected based on the identification information of the object among the detection results detected by multiple sensors within a predetermined time (e.g., 10 milliseconds) based on the predetermined time period. The at least one sensor selected based on the identification information of the object may include one or more sensors. When multiple sensors are selected based on the identification information of the object, the selected multiple sensors may have a priority. The cleaning robot 100 may assign a weighted value to the detection result with the highest priority to obtain additional information about the object.

[0278] When the priority of the IR stereo sensor among the multiple sensors is set to be higher, the cleaning robot 100 can obtain additional information about the object by assigning a weighted value to the result detected by the IR stereo sensor to obtain additional information about the object. Specifically, the cleaning robot 100 can determine the bounding box of the object, and for an area where the determination result of the bounding box is inconsistent with the object detection result by the IR stereo sensor, lower the threshold of the IR stereo sensor to detect the object.

[0279] The cleaning robot 100 may set priorities for the plurality of sensors according to the recognition information of the object, and obtain additional information about the object through a result detected by at least one sensor among the plurality of sensors according to the priority.

[0280] When a higher priority is given to the LIDAR sensor among a plurality of sensors according to recognition information of the object, the cleaning robot 100 may assign a weighted value to a result detected by the LIDAR sensor to obtain additional information about the object.

[0281] When the ultrasonic sensor is given a higher priority among multiple sensors according to the identification information of the object, the cleaning robot 100 can give a weighted value to the result detected by the ultrasonic sensor and obtain additional information about the object. When the ultrasonic sensor is given a higher priority among multiple sensors according to the identification information of the object, the identified object can be transparent or black.

[0282] In step S2904 , based on the additional information about the object, the cleaning robot 100 may determine a task to be performed by the cleaning robot 100 .

[0283] Various embodiments of the present disclosure may also be implemented in mobile devices. For example, the mobile device may be implemented in various forms (such as service robots for public places, transport robots for production sites, operator-assisted robots, housekeeping robots, security robots, autonomous vehicles, etc.).

[0284] In this case, the task of the present disclosure may be a task according to the purpose of the mobile device. For example, if the cleaning robot is tasked with avoiding an object or sucking up dust in a house, the task of the operator-assisted robot may be to avoid or move an object. In addition, the task of the security robot may be to avoid an object, detect an intruder to provide an alarm, or photograph an intruder. In addition, the task of the autonomous vehicle may be to avoid another vehicle or an obstacle, or to control a steering device or an acceleration / deceleration device.

[0285] The term "module" used in the present disclosure may include units embodied in hardware, software, or firmware, and may be used compatibly with terms such as logic, logic block, component, circuit, etc. A module may be a component of an overall structure or a minimum unit of a component or part thereof that performs one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0286] Various embodiments of the present disclosure may be embodied as software including commands stored in a machine-readable storage medium. A machine may be a device that calls commands stored in a storage medium and may operate according to the called commands, including an electronic device (e.g., electronic device (A)) according to the disclosed example embodiments. When a command is executed by a processor, the processor may use other components to perform functions corresponding to the command directly or under the control of the processor.

[0287] Various embodiments of the present disclosure may be implemented as software (e.g., a program), the software including one or more instructions stored in a storage medium (e.g., memory 130, memory on a server (not shown)), the storage medium being readable by a machine (not shown) (e.g., the cleaning robot 100 and a server (not shown) communicating with the cleaning robot 100). For example, a processor of a device (e.g., processor 140, a processor of a server (not shown)) may call and execute at least one of the one or more instructions stored from the storage medium. This enables the device to be operated to perform at least one function according to at least one called command called. The command may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-temporary storage medium. "Non-temporary" means that the storage medium does not include a signal (e.g., an electromagnetic wave) and is tangible, but does not distinguish whether the data is stored semi-permanently or temporarily on the storage medium.

[0288] According to an embodiment, the method according to various embodiments disclosed herein may be provided in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or through an application store (e.g., PlayStore). TM ) is distributed online (e.g., downloaded or uploaded) between two user devices (e.g., smartphones). In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created on a storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0289] Each of the components (e.g., modules or programs) according to various embodiments may be composed of a single entity or multiple entities, and some of the subcomponents of the above-mentioned subcomponents may be omitted, or other components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by each component before integration. According to various embodiments, the operations performed by modules, programs or other components may be performed sequentially, in parallel, repeatedly or heuristically, or at least some operations may be performed or omitted in a different order, or another function may be further added.

[0290] Although exemplary embodiments have been shown and described, those skilled in the art will appreciate that changes may be made to these exemplary embodiments without departing from the principles and spirit of the present disclosure. Therefore, the scope of the present invention is not to be interpreted as being limited to the described exemplary embodiments, but is to be defined by the appended claims and their equivalents.

Claims

1. A cleaning robot, comprising: at least one sensor; At least one camera; Driver; and The processor is configured as: obtaining information about an object included in the captured image by inputting the image captured by at least one camera into a trained artificial intelligence model; obtaining information sensed by at least one sensor; obtaining recognition information about each of the plurality of regions using the obtained information about the object, wherein the recognition information about each of the plurality of regions includes information about a type of each of the plurality of regions; generating a map showing the plurality of areas using the obtained information about the object, the sensed information, and the recognition information about each of the plurality of areas; controlling a driver to move the cleaning robot to the area in the plurality of areas based on a user voice indicating a cleaning operation with respect to an area in the plurality of areas; and performing a cleaning operation with respect to an area among the plurality of areas based on movement of the cleaning robot to the area among the plurality of areas, wherein the user voice includes information about the cleaning operation and identification information about a region among the plurality of regions, Therein, the map includes information about a type of each of the plurality of areas.

2. The cleaning robot according to claim 1, wherein: The user voice includes a word referring to an area among the plurality of areas and a word instructing an execution operation of the cleaning robot, and Here, a word referring to a region among the plurality of regions is matched with identification information about the region among the plurality of regions.

3. The cleaning robot according to claim 1, wherein: The processor is further configured to control the driver to move the cleaning robot to the remaining area excluding the avoidance area based on a user command designating an area among the plurality of areas as an avoidance area received from the external terminal device through the communication interface.

4. The cleaning robot according to claim 1, wherein: The map is displayed by an external terminal device connected to the cleaning robot, and The map includes text or icons indicating identification information about the plurality of areas.

5. The cleaning robot according to claim 1, wherein: The at least one sensor includes at least one of an infrared (IR) stereo sensor, a light detection and ranging (LIDAR) sensor, or a position sensitive diode (PSD) sensor.

6. A method for controlling a cleaning robot comprising at least one sensor and at least one camera, the method comprising: obtaining information about an object included in the captured image by inputting the image captured by at least one camera into a trained artificial intelligence model; obtaining information sensed by at least one sensor; obtaining recognition information about each of the plurality of regions using the obtained information about the object, wherein the recognition information about each of the plurality of regions includes information about a type of each of the plurality of regions; generating a map showing the plurality of areas using the obtained information about the object, the sensed information, and the recognition information about each of the plurality of areas; moving the cleaning robot to an area in the plurality of areas based on a user voice instructing a cleaning operation with respect to an area in the plurality of areas; and performing a cleaning operation with respect to an area among the plurality of areas based on movement of the cleaning robot to the area among the plurality of areas, wherein the user voice includes information about the cleaning operation and identification information about a region among the plurality of regions, Therein, the map includes information about a type of each of the plurality of areas.

7. The method according to claim 6, wherein: The user voice includes a word referring to an area among the plurality of areas and a word instructing an execution operation of the cleaning robot, and The word referring to the region among the plurality of regions is matched with the identification information about the region among the plurality of regions.

8. The method according to claim 6, comprising: Based on a user command that designates an area among a plurality of areas as an avoidance area received from an external terminal device through a communication interface, the cleaning robot is moved to a remaining area that does not include the avoidance area.

9. The method according to claim 6, wherein: The map is displayed by an external terminal device connected to the cleaning robot, and The map includes text or icons indicating identification information about the plurality of areas.

10. The method according to claim 6, wherein: The at least one sensor includes at least one of an infrared (IR) stereo sensor, a light detection and ranging (LIDAR) sensor, or a position sensitive diode (PSD) sensor.

Images