Cleaning robot

Abstract

Cleaning robots, comprehensive: a communication interface; at least one sensor; at least one camera (120); a drive unit; and a processor (140) that is configured: to input an image taken by at least one camera into a trained artificial intelligence model in order to obtain information about an object contained in the captured image: to obtain information that is captured by at least one sensor; To obtain recognition information about each of several work surfaces in a home based on information about the object, wherein the recognition information about each of the several work surfaces includes type information about each of the several work surfaces; to generate a map that displays the multiple work areas, using the information obtained about the object, the captured information, and the recognition information for each of the multiple work areas; based on a user voice instructing a cleaning operation to control the drive unit to move to at least one of the multiple work surfaces; and based on moving to the at least one work surface, a cleaning operation for the at least one work surface is to be carried out, where the recognition information for each of the multiple work surfaces includes a name for each of the multiple work surfaces.

Description

[Technical field]

[0001] Devices and methods corresponding to those disclosed herein relate to a cleaning robot and a working method for it, and in particular to a cleaning robot for providing adequate work using information about objects (e.g. obstacles) in the vicinity of the cleaning robot and a control method for it. [Background]

[0002] With the development of robotics technology, robots are now commonly used in households, as well as in specialized technical fields and industries that require a significant amount of labor. In particular, service robots for performing household chores, cleaning robots, pet robots, and so on have become widely used.

[0003] In the case of a cleaning robot, it is particularly important to accurately identify information about objects such as foreign substances, structures, obstacles, etc., in the vicinity of the robot and to perform a cleaning process appropriate for each object. However, a conventional cleaning robot is limited in its ability to obtain detailed information about the object due to the restricted combination of sensors. In other words, the conventional cleaning robot has no information about the type of object but attempts to avoid objects in the same pattern solely based on the sensor's detection capability.

[0004] Therefore, it is necessary to identify an object near the cleaning robot, determine a task suitable for the object that can be performed by the cleaning robot, and drive the cleaning robot or avoid objects more effectively. [Disclosure of the invention][Problem solving]

[0005] One aspect of the exemplary embodiments concerns providing a cleaning robot to provide a service for performing work suitable for a peripheral object, using multiple sensors of the cleaning robot and a control method for it.

[0006] According to an exemplary embodiment, a method for performing a task by a cleaning robot is provided, wherein the method includes generating a navigation map for driving the cleaning robot based on receiving sensor data from at least one sensor that detects or captures a work surface in which an object is arranged, obtaining recognition information of the object by applying an image of the object, captured by at least one camera, to a trained artificial intelligence model, generating a semantic map that specifies an environment of the work surface by mapping an area of ​​the object contained in the navigation map with the object's recognition information, and performing a task by the cleaning robot based on a control command from a user using the semantic map.

[0007] According to an exemplary embodiment, a cleaning robot is provided which includes at least one sensor, one camera, and at least one processor configured to generate a navigation map for driving the cleaning robot based on receiving sensor data from the at least one sensor that detects (or captures) a work surface in which an object is arranged, obtaining object recognition information by applying an image of the object captured by the camera to a trained artificial intelligence model, thereby providing a semantic map specifying an environment of the work surface by mapping an area of ​​the object contained in the navigation map with the object's recognition information, and performing work of the cleaning robot based on a control command from a user using the semantic map.

[0008] According to the various exemplary embodiments described above, a cleaning robot can provide a service to perform the most suitable work, such as removing or avoiding one or more objects, taking into account recognition information and / or additional information, etc., about an object (e.g., a nearby object).

[0009] According to an exemplary embodiment, a method is provided comprising: Receiving, by a cleaning robot, an image captured by a camera or sensor of the cleaning robot; Transmitting, by the cleaning robot, the captured image to an external server; Obtaining, by the external server, recognition result information by inputting the captured image into a trained artificial intelligence model, wherein the recognition result information contains information about the object; Transmitting, by the server, the recognition result information to the cleaning robot, based on mapping an area corresponding to the object contained in a navigation map, with the object's recognition information; Generating, by the cleaning robot, a semantic map containing information specifying a position of the object in the work area in the navigation map.and carrying out, by the cleaning robot, a task based on a control command from a user using the semantic map.

[0010] According to the various exemplary embodiments described above, a cleaning robot can provide a semantic map that specifies the environment of a work surface. Therefore, a user can control the cleaning robot's work using the name, etc., of an object or location with the provided semantic map, thus significantly improving usability. [Brief description of the drawings] Fig. 1 is a view explaining a method for recognizing and detecting (or capturing) an object or obstacle of a cleaning robot according to an embodiment of the disclosure; Fig. 2A and Fig. 2B are block diagrams to explain a configuration of a cleaning robot according to an embodiment of the disclosure; Fig. Figure 3 is a detailed block diagram to explain a configuration of a cleaning robot according to an embodiment of the disclosure; Fig. 4A, Fig. 4B, Fig. 4C and Fig. 4D views are used to explain that a cleaning robot receives additional information about an object based on a detection result by an infrared (IR) stereo sensor according to an embodiment of the disclosure; Fig. 5A, Fig. 5B and Fig. 5C are views to explain that a cleaning robot receives additional information about an object based on a detection result by a LIDAR sensor; Fig. 6A and Fig. 6B are views to explain that a cleaning robot receives additional information about an object based on a detection result by an ultrasonic sensor; Fig. 7A, Fig. 7B and Fig. 7C are views that illustrate how a cleaning robot recognizes the structure of a house; Fig. 8A and Fig. 8B are views explaining that a cleaning robot generates a semantic map based on the structure of a house and additional information about an object according to one embodiment of the disclosure; Fig. 9A and Fig. 9B are views explaining that a cleaning robot informs a user about a hazardous material on the floor; Fig. 10 is a view explaining that a cleaning robot according to one embodiment of the disclosure indicates an area that is not to be cleaned; Fig. 11A and Fig. Figure 11B are block diagrams illustrating a training module and a recognition module according to different embodiments of the disclosure; Fig. Figure 12 is a view illustrating an example in which a cleaning robot and a server can be operated in conjunction with each other to train and recognize data; Fig. Figure 13 is a flowchart for explaining a network system using a recognition model according to an embodiment of the disclosure; Fig. Figure 14 is a flowchart to explain an example in which a cleaning robot provides a search result for a first area using a recognition model according to an embodiment of the disclosure; Fig. Figure 15 is a flowchart to explain a system that uses a recognition model according to one embodiment of the disclosure; Fig. 16 is a view explaining the generation of a semantic map according to one embodiment of the disclosure; Fig. Figure 17 is a view illustrating a user interface for the use of a semantic map according to one embodiment of the disclosure; Fig. 18 is a view illustrating a situation in which a cleaning robot is controlled using a semantic map according to an embodiment of the disclosure; Fig. 19 is a view explaining the generation of a semantic map according to one embodiment of the disclosure; Fig. Figure 20 is a view illustrating a user interface for the use of a semantic map according to one embodiment of the disclosure; Fig. Figure 21 is a view illustrating a situation in which a cleaning robot is controlled using a semantic map according to an embodiment of the disclosure; Fig. 22 is a view illustrating a process for recognizing an object according to one embodiment of the disclosure; Fig. 23 is a view explaining a process for generating a semantic map according to one embodiment of the disclosure; Fig. Figure 24 is a view illustrating a situation in which a cleaning robot is controlled using a semantic map according to an embodiment of the disclosure; Fig. 25 is a view illustrating a configuration of a cleaning robot according to an embodiment of the disclosure; Fig. 26A and Fig. Figure 26B are views illustrating a detection range (capture area) of a cleaning robot according to one embodiment of the disclosure; and Fig. 27, Fig. 28 and Fig. Figure 29 are flowcharts to explain a cleaning robot according to one embodiment. [Best form for carrying out the invention]

[0011] It is clear that the present disclosure is not intended to limit the scope of the described embodiments, but rather includes various modifications, equivalents, and / or alternatives of the embodiments. In the description of the drawings, the same reference numerals refer to the same elements throughout the entire description of the drawings.

[0012] Terms such as "first" and "second" can be used to modify different elements regardless of order and / or meaning. These terms are used only to distinguish one component from other components. For example, the first user device and the second user device can represent different user devices, regardless of order or meaning. For example, without deviating from the scope of the claims described in this disclosure, the first component can be referred to as a second component, and likewise, the second component can also be referred to as the first component.

[0013] When an element (e.g., a first component) is described as "operationally or communicatively coupled to" another element or "connected to" another element (e.g., a second component), it should be clear that each element is directly or indirectly connected via another element (e.g., a third component). However, when an element (e.g., a first component) is described as "directly coupled to" another element or "directly connected to" another element (e.g., a second component), it should be clear that there is no intervening element (e.g., a third component).

[0014] The terminology used herein serves the purpose of describing particular embodiments and is not intended to limit the scope of other exemplary embodiments. As used herein, singular forms are employed for the sake of simplicity, but plural forms should also be included unless the context clearly indicates otherwise. Additionally, terms used in this patent specification may have the same meaning as generally understood by those skilled in the art. General predefined terms used herein may be interpreted as having the same or similar meanings as the contextual meanings of related technology, and unless expressly defined herein, the terms are not to be interpreted in an ideal or overly formal sense. In some cases, the terms defined herein may not be interpreted as excluding embodiments of the disclosure.

[0015] In the following, various embodiments of the disclosure are described in detail with reference to the accompanying drawings. Fig. Figure 1 is a view explaining a method for recognizing and detecting (capturing) an object or obstacle of a cleaning robot according to an embodiment of the disclosure.

[0016] With reference to Fig. 1. A cleaning robot 100 can detect an object 200 in the vicinity of the cleaning robot. The cleaning robot 100 can be a self-moving device that provides a cleaning service to a user and could be designed as various types of electronic devices. For example, the cleaning robot 100 can be designed in different shapes for different purposes, such as a cylindrical shape or a rectangular parallelepiped shape, for example as a home cleaning robot, high-rise cleaning robot, airport cleaning robot, etc. According to one embodiment, the cleaning robot 100 can not only perform the task of removing foreign substances from the floor, but also the task of moving an object according to instructions from a user.

[0017] The cleaning robot 100 can capture an image containing object 200 with its camera and input the captured image into an artificial intelligence model trained to recognize the object. The artificial intelligence model can be located within the cleaning robot 100 or on an external server (not shown). For example, the artificial intelligence model could be a model trained using a supervised learning process based on an artificial intelligence algorithm and an unsupervised learning process. As an example, the artificial intelligence model could be a neural network model containing multiple network nodes with weighted values. These nodes could be positioned at different depths (or layers) to transmit or receive data according to a convolutional link relationship.For example, a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), and the like can be used as a neural network model, but the revelation is not limited to this.

[0018] If object 200 is recognized as a specific type of object (e.g., a chair), the cleaning robot 100 can use a sensor other than a red-green-blue (RGB) camera to obtain more specific information about object 200. More precisely, if the object is the specific type of object (e.g., a chair), the cleaning robot 100 can be preset to preferably use a LiDAR (Light Detection and Ranging) sensor to obtain information about the positions of parts of the specific type of object (e.g., chair legs) or the distances between the parts (e.g., legs), or to give the result from the LiDAR sensor a higher weighting among results from multiple sensors.

[0019] The cleaning robot 100 can emit a laser pulse 220 to detect the object 200 using the LIDAR sensor, providing specific information about the detected object 200. A detailed description follows.

[0020] The cleaning robot 100 can store speed information, object image information, and speed information at different positions (e.g., a first position and a second position) of the cleaning robot. Based on the stored information, the cleaning robot 100 can determine a distance (d) between the first and second positions and determine the distance from a specific position to the object 200.

[0021] The cleaning robot 100 can determine a task to be performed by the cleaning robot 100 with respect to object 200 based on specific information about the positions of parts of the specific object (e.g., chair legs), the distances between the parts of the specific object (e.g., chair legs), and distance information from object 200 obtained from an additional sensor such as the LiDAR sensor. For example, the cleaning robot 100 can control the speed and direction of movement to perform a task to clean a space between parts of the specific object (e.g., chair legs) based on information about the distances between the parts of the specific object (e.g., the chair legs).

[0022] The cleaning robot 100 can define a restricted area for the detected object 200 or a boundary frame 210. The cleaning robot 100 can only access a restricted area 210 if no object is detected by the RGB camera and no additional detection information is available other than the information about the restricted area 210.

[0023] The examples above illustrate that an object is a chair and a sensor is a LiDAR sensor, but the disclosure is not limited to these. Different sensors can be used for different objects.

[0024] The Cleaning Robot 100 can determine different tasks based on object recognition results. For example, the Cleaning Robot 100 can perform a function to remove grain if it recognizes the object as grain, move a cushion if it recognizes the object as a cushion, and reduce its speed to completely avoid a glass cup if it recognizes the object as fragile and easily broken. Additionally, if the object is detected as hazardous, the Cleaning Robot 100 can take an image of the object and transmit the image to the user device.If the object is detected as dirty, such as dog poop, the cleaning robot 100 can perform specific detection to completely avoid the object and transmit an image containing the object to the user terminal to notify the user of the dog poop (or glass cup).

[0025] The Cleaning Robot 100 can obtain specific information about the object based on a combination of detection (capture) results from at least one sensor, which is available to determine whether a near-avoidance or a full-avoidance cleaning route is performed. A detailed description of this is given below.

[0026] Fig. Figure 2A is a block diagram to explain a configuration of a cleaning robot according to an embodiment of the disclosure.

[0027] With reference to Fig. 2A the cleaning robot can contain a sensor 110, a camera 120, a memory 130 and a processor 140.

[0028] The Sensor 110 can contain various types of sensors. Specifically, the Sensor 110 can include an IR stereo sensor, a LiDAR sensor, an ultrasonic sensor, and the like. Each IR stereo sensor, LiDAR sensor, and ultrasonic sensor can be implemented as a single sensor or as a separate sensor.

[0029] The IR stereo sensor can detect the three-dimensional shape of an object and distance information. It can obtain three-dimensional (3D) depth information, including length, height, and width. However, the IR stereo sensor has the disadvantage of not detecting black, transparent, or metallic colors.

[0030] The Cleaning Robot 100 can obtain a two-dimensional (2D) line shape of an object and distance information using a LiDAR sensor. Therefore, the Cleaning Robot 100 can obtain information about the space available to the object and distance information about nearby objects. However, the LiDAR stereo sensor has the disadvantage that it cannot detect black, transparent, or metallic colors.

[0031] The ultrasonic sensor can obtain distance information about obstacles. While it has a disadvantage of a relatively limited detection range, it has the advantage of detecting black, transparent, or metallic colors.

[0032] Additionally, the Sensor 110 can include sensors for environmental detection, such as a dust sensor, an odor sensor, a laser sensor, an ultra-wideband (UWB) sensor, an image sensor, an obstacle sensor, and sensors for detecting motion, such as a gyroscope, a global navigation satellite system (GPS) sensor, and the like. The environmental sensors and the sensors for detecting the cleaning robot's motion can be implemented in different configurations or in a single configuration. The Sensor 110 can further include various types of sensors, and some sensors may be omitted depending on the specific tasks performed by the Cleaning Robot 100.

[0033] The camera 120 can be configured to capture a peripheral image of the cleaning robot 100 from various angles. The camera 120 can capture a front-facing image of the cleaning robot 100 using an RGB camera, or images taken from directions other than the robot's direction of travel. The camera 120 can be deployed independently within the cleaning robot 100 or integrated into an object detection sensor as part of that sensor.

[0034] The Camera 120 can contain multiple cameras. The Camera 120 can be installed on at least one of the top or front parts of the Cleaning Robot 100.

[0035] Memory 130 can store the image captured by camera 120 and motion status information, and can record directional information of the cleaning robot 100 at the time of recording. Memory 130 can store navigation map information for positioning the cleaning robot 100 to perform its tasks. However, the disclosure is not limited to this, and memory 130 can store various programs required for operating the cleaning robot 100.

[0036] Memory 130 can store multiple application programs and / or applications powered by the Cleaning Robot 100, as well as data commands, etc., for operating the Cleaning Robot 100. Some of the application programs can be downloaded from an external server via wireless communication. At least some of the application programs can be configured in the Cleaning Robot 100 for basic functionality, if enabled. The application programs can be stored in Memory 130 and cause the Cleaning Robot 100 to operate (or function).

[0037] As various exemplary embodiments, the memory 130 can generate a navigation map for driving the cleaning robot 100 using the result of at least one sensor that detects a work surface in which an object is arranged, obtain recognition information of the object by applying the image of the object, which is captured by at least one camera, to the trained model of artificial intelligence, map the area of ​​the object contained in the navigation map with the recognition information of the object, and store at least one instruction that is set up to generate a semantic map that specifies the environment of the work surface.

[0038] Memory 130 can store at least one instruction, enabling processor 140 to capture an image of an object near the cleaning robot, obtain recognition information of the object contained in the image by applying the captured image to the trained artificial intelligence model, detect the object using at least one sensor selected based on the obtained object recognition information, and obtain additional information about the object using the result detected by at least one sensor.

[0039] The memory 130 can be implemented with at least one of the following: non-volatile memory, volatile memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The processor 140 can access the memory 130 and perform one of the following operations: reading, writing, modifying, deleting, or updating data (e.g., data stored in the memory). According to the present disclosure, the term 'memory' can include the memory 130, a read-only memory (ROM) (not shown), a random access memory (RAM) (not shown) in the processor 140, or a memory card (not shown) (e.g., a micro SD card, a memory stick, etc.) that is installed in the cleaning robot 100.

[0040] The processor 140 can control all operations of the cleaning robot 100. For example, the processor 140 can control the camera 120 to take a picture near the cleaning robot 100. The processor 140 can contain RAM and ROM, or a system can contain ROM, RAM, and the processor 140. The ROM can store an instruction configured to start the system. The CPU 141 can copy the operating system (O / S) stored in the cleaning robot 100 into RAM according to an instruction stored in ROM and execute the O / S to start the system. Once the system has started, the CPU 141 can copy various programs stored in memory 130 into RAM, execute the programs copied into RAM, and perform various operations.

[0041] According to one embodiment, with reference to Fig. 2B, the cleaning robot 100 can contain multiple processors 141 and 142. The multiple processors 141 and 142 can include a central processing unit (CPU) 141 and a neural processing part (NPU) 142. The NPU 142 can be an optimized, specific processor for recognizing an object using the trained artificial intelligence model. The NPU (and / or the CPU) of the cleaning robot 100 can contain at least one digital signal processor (DSP) for processing digital signals, a microprocessor, a time controller (TCON), a microcontroller unit (MCU), a microprocessing unit (MPU), an application processor (AP), a communication processor (CP), or an ARM processor, or the like, or can be defined by various combinations of the corresponding terms.The 140 processor can be implemented as a system-on-a-chip (SoC), a large-scale integration (LSI) circuit with a built-in processing algorithm, or in the form of a field-programmable gate array (FPGA).

[0042] The Processor 140 can detect obstacles contained in an image using an artificial intelligence model trained to recognize objects like obstacles. The Processor 140 can input an image containing obstacles into the AI ​​model and receive an output that includes information about the types of obstacles. The Processor 140 can determine the size of the restricted area, which varies depending on the type of obstacle. The restricted area can be an area containing the obstacles or an area to which the Cleaning Robot 100, performing its cleaning task, has no access.

[0043] The artificial intelligence model can be trained and stored in memory 130 of the cleaning robot 100 as a device type, or it can be stored on an external server. A detailed description of this is given below. An embodiment in which an artificial intelligence model is stored in the cleaning robot 100 is presented as an example.

[0044] The processor 140 can generate a second image by overlapping the restricted area for the detected obstacle with the first image. Based on information contained in the second image, the processor 140 can identify the positions of structures and obstacles near the cleaning robot 100 and determine the robot's direction and speed. The processor 140 can then control the driver 110 to move the cleaning robot 100 according to the specified direction and speed.

[0045] The Processor 140 can generate the first image of the ground, which is typically separated from the captured image. The Processor 140 can use an image splitting method to divide the ground image.

[0046] The processor 140 can generate a navigation map to guide the cleaning robot 100 based on the data from sensor 110, which detects (captures) the work area where the object is located. The processor 140 can obtain object recognition information by applying the image of the object, captured by camera 120, to the trained artificial intelligence model. The processor 140 can generate a semantic map containing information about the work area by mapping the object's area, as shown in the navigation map, using the object's recognition information. Based on the user's control commands, the processor 140 can then execute a cleaning robot task using this semantic map.Therefore, the user to whom the semantic map is provided can control the work of the cleaning robot 100 using the object's recognition information in various methods, thus significantly improving usability.

[0047] The Processor 140 can obtain location information within the work area using the object's location information. The Processor 140 can generate a semantic map containing information specifying the work area's environment, using both the location and object location information. Therefore, the user can control the tasks to be performed by the Cleaning Robot 100 by referencing one or both of the location and object location information, based on the provided semantic map.

[0048] The Processor 140 can map the area of ​​the object contained in the navigation map using the object's recognition information, based on at least one of its location or shape characteristics according to the object's detection result, to generate a semantic map that specifies the workspace's environment. Therefore, the object can be mapped with an exact location relative to the navigation map to provide the semantic map.

[0049] Processor 140 can apply the image of the object captured by camera 120 to the trained artificial intelligence model hosted on the external server to obtain object recognition information. Using this artificial intelligence model significantly increases the object recognition rate. Specifically, by using the artificial intelligence model hosted on the external server, the limitations of the cleaning robot 100's resources can be overcome, thus improving the model's usability by utilizing more resources.

[0050] The Processor 140 can identify the object's boundary according to the object in the navigation map. The Processor 140 can map the object's area, defined by its boundary, using the object's recognition information to generate a semantic map that specifies the workspace's environment.

[0051] The processor 140 can apply the image of the object captured by the camera 120 to the trained artificial intelligence provided in the external server and obtain the object's recognition information.

[0052] The processor 140 can control at least one sensor, selected based on the object's detection information, from among several sensors contained in the sensor 110, to detect (capture) the object. The processor 140 can obtain additional information about the object using the detection result from at least one sensor.

[0053] The Processor 140 can prioritize multiple sensors based on object detection information. The Processor 140 can also obtain additional information about the object using the results detected by at least one sensor, according to its priority among the multiple sensors.

[0054] Processor 140 can control camera 120 to capture an object near cleaning robot 100. Processor 140 can apply the captured image to the trained artificial intelligence model to obtain the object's recognition information. Processor 140 can control at least one sensor, selected based on the object's recognition information, from among the multiple sensors contained in sensor 110 to detect the object. Processor 140 can obtain additional information about the object using the results detected by at least one sensor and determine the task to be performed by cleaning robot 100 based on this additional information.

[0055] Processor 140 can set priorities for the multiple sensors contained in Sensor 110 based on the object detection information. Processor 140 can also obtain additional object information based on the results detected by at least one sensor, according to the priority.

[0056] If the IR stereo sensor is given a higher priority among the multiple sensors according to the detection information about the object, the processor 140 can assign a weighted value to the result detected by the IR stereo sensor and obtain the additional information about the object.

[0057] The processor 140 can identify a bounding frame in relation to the detected object and reduce a threshold value of the IR stereo sensor in relation to an area where the identification result of the bounding frame does not match the object detection result by the IR stereo sensor.

[0058] If the LIDAR sensor is given a higher priority from the multiple sensors contained in the sensor 110, according to the object's detection information, the processor 140 can assign a weighted value to the result detected by the LIDAR sensor in order to obtain additional information about the object.

[0059] If the ultrasonic sensor is given a higher priority from the multiple sensors contained in the sensor 110, according to the object's detection information, the processor 140 can assign a weighted value to the result detected by the ultrasonic sensor in order to obtain additional information about the object.

[0060] If a priority is set higher with respect to the ultrasonic sensor from among the multiple sensors contained in the sensor 110, according to the object's detection information, the detection object can be transparent or black.

[0061] The Processor 140 can apply the captured image to the trained artificial intelligence model provided on the external server and obtain the object recognition information.

[0062] Fig. Figure 3 is a detailed block diagram to explain a configuration of a cleaning robot according to an embodiment of the disclosure.

[0063] With reference to Fig. 3. A cleaning robot 100 can contain a sensor 110, a camera 120, a memory 130, a communicator 150, a dust collection unit 160, a driver 170, a power source 180 and a processor 140, which is electrically connected to the components described above.

[0064] The sensor 110, the camera 120, the memory 130 and the processor 140 have been described, and therefore a repeated description is omitted.

[0065] The Communicator 150 can transmit data, control commands, etc., to and / or receive data from an external device. For example, the Communicator 150 can receive partial or complete map information, including location data for the room where the Cleaning Robot 100 is operating, from the external device. The Communicator 150 can then transmit information to the external device to update all map data. Alternatively, the Communicator 150 can receive a signal to control the Cleaning Robot 100, transmitted by a user via a remote control device. The remote control device can take various forms, such as a remote control unit, a mobile device, etc.

[0066] The Communicator 150 can transmit data to and / or receive data from an external server (not shown). For example, if an artificial intelligence model is stored on the external server, the Communicator 150 can transmit the image captured by the Camera 120 to the external server and receive the object recognition information (e.g., information about obstacles) detected using the artificial intelligence model stored on the external server. However, the disclosure is not limited to this; the Communicator 150 can also receive information from the external server about the moving area for the room in which the Cleaning Robot 100 is performing its work.

[0067] The Communicator 150 can include a communication interface that uses various methods such as Near Field Communication (NFC), Wireless Local Area Network (LAN), Infrared (IR) Communication, Zigbee Communication, WiFi, Bluetooth, etc. as a wireless communication method.

[0068] The 160 dust collection unit can be configured to collect dust. Specifically, the 160 dust collection unit can draw in air and collect dust from the drawn-in air. For example, the 160 dust collection unit can include a motor that forces air through a guide tube from an inlet to an outlet, a filter for filtering dust from the drawn-in air, and a dust basket for collecting the filtered dust.

[0069] The driver 170 can be configured to drive the movement of the cleaning robot 100. For example, the driver 170 can move the cleaning robot 100 to a position to perform a task under the control of the processor 140. In this case, the driver 170 can include at least one wheel that contacts the floor, a motor to provide driving force to the wheel, and a driver to control the motor. As another example, the driver 170 can operate to perform a task. In the case of an object movement task, the driver 170 can include a motor to perform an operation such as picking up an object.

[0070] The power source 180 can supply the current required to power the cleaning robot 100. For example, the power source 180 can be implemented as a battery that can be charged or discharged. The processor 140 can control the driver 110 to move to a charging station when the remaining current level is equal to or less than a predetermined level, or when the cleaning cycle is complete. The power source 180 can be charged using at least one contact method or a non-contact method.

[0071] Fig. 4A, Fig. 4B, Fig. 4C and Fig. 4D views are used to explain how a cleaning robot can obtain additional information about an object based on a detection result by an IR stereo sensor according to one embodiment of the disclosure.

[0072] The cleaning robot 100 can detect an object in front of it using the IR stereo sensor. The cleaning robot 100 can recognize an object in front of it using the camera 120. The cleaning robot 100 can detect the object using the IR stereo sensor and obtain length, height, and / or depth information about the object. For example, with reference to Fig. 4A, the cleaning robot 100 recognizes and detects several objects 410 and 420 in front of it through the camera 120 and the IR stereo sensor and obtains the length, height or depth information about the objects 410 and 420, which can be a flower pot 410 and a carpet 420.

[0073] The IR stereo sensor can detect an object if any of its length, height, or depth is greater than a threshold value, but it cannot detect an object if any of its length, height, or width / depth is less than a threshold value. For example, if the height information for carpet 420 is less than a predetermined threshold, the cleaning robot 100 will not be able to detect carpet 420. (With reference to...) Fig. 4B, the cleaning robot can obtain 100 depth information 411 about a flower pot 410 through the IR stereo sensor.

[0074] Camera 120 can detect and recognize objects 410 and 420 in front of it, regardless of height or depth information about the object. The cleaning robot 100 can, with reference to Fig. 4C, take and identify pictures of the flower pot 410 and the carpet 420, and determine the boundary frames 412 and 422 in relation to the respective objects 410 and 420.

[0075] With reference to Fig. 4B and Fig. 4C allows the cleaning robot 100 to set a low threshold for the IR stereo sensor if the object detection result by camera 120 differs from the object detection result by the IR stereo sensor. The cleaning robot 100 can further reduce this threshold until the object detection result by camera 120 matches the object detection result by the IR stereo sensor, for example, until the number of objects is the same. If the IR stereo sensor threshold is significantly lowered, the cleaning robot 100 can, with reference to Fig. 4D, not only the depth information 411 about the flower pot 410, but also size information (like depth information) 421 about the specific object (like carpet 420) as additional information obtained through the IR stereo sensor.

[0076] With reference to Fig. 4C The cleaning robot 100 can reduce the threshold value of the IR stereo sensor with respect to the areas corresponding to the obtained boundary frames 412 and 422. The cleaning robot 100 can compare the object recognition result with the object detection result by the IR stereo sensor and reduce the threshold value of the IR stereo sensor with respect to a different area 422. According to the embodiment described above, the cleaning robot 100 can not only detect the carpet 420, which is detected by the IR stereo sensor, but also maintain an existing threshold value when detecting a new object.

[0077] Fig. 5A, Fig. 5B and Fig. 5C are views to explain how a cleaning robot can obtain additional information about an object based on a detection result from a LIDAR sensor.

[0078] With reference to Fig. 5A The cleaning robot 100 can detect and recognize object 500 in its vicinity using camera 120. As a result of recognizing an object, the cleaning robot 100 can determine that the detected object 500 is a specific type of object, such as a desk or a chair. If the object is indeed a desk or a chair, the cleaning robot 100 can detect it by prioritizing the LiDAR sensor over other sensors as an additional sensor to obtain further information about the object.

[0079] Fig. Figure 5B is a top view illustrating that the cleaning robot 100 detects object 500 and defines the boundary frame 510 relative to object 500. As described above, the cleaning robot 100 can obtain detailed information about object 500 as additional information from the LiDAR sensor, which is given higher priority than any other sensor. More specifically, the LiDAR sensor can obtain information about the bridge of object 500, while the cleaning robot 100 emits the laser pulse in the direction of the object detected by the cleaning robot 100.

[0080] With reference to Fig. The cleaning robot 100 can obtain information 520 about parts of the object 500 (e.g., the legs of the object 500) via the LIDAR sensor. In other words, the cleaning robot 100 can obtain information about the positions of the legs and the spaces between the positions of the legs of the object 500 and determine an appropriate cleaning procedure.

[0081] Fig. 6A and Fig. 6B are views to explain that a cleaning robot can obtain additional information about an object based on a detection result object through an ultrasonic sensor.

[0082] With reference to Fig. 6A, if an object is a black table 600 or a transparent glass beaker 610, the IR stereo sensor or the LiDAR sensor cannot detect the object. If the object is recognized as the black table 600 and the glass beaker 610 in the image obtained with the camera 120, the cleaning robot 100 can detect the object by giving higher priority to the ultrasonic sensor rather than the IR stereo sensor or the LiDAR sensor.

[0083] The cleaning robot 100 can turn right or left and attempt to detect an object, as the ultrasonic sensor has a limited detection range. For example, with reference to Fig. 6B, the cleaning robot 100 obtains additional information about the object, such as the positions of the legs of a black table 600 and / or the position of a glass beaker 610, by means of an ultrasonic wave, by approaching it within a distance that can be detected by the ultrasonic sensor.

[0084] The cleaning robot according to one embodiment may have the advantage of detecting an object that is difficult to detect by the IR stereo sensor and the LIDAR sensor.

[0085] Fig. 7A and Fig. 7B are views that illustrate how a cleaning robot recognizes the structure of a house.

[0086] With reference to Fig. 7A The cleaning robot 100 can detect objects 700 and 710, which may be doors, using the result of object detection via camera 120 or at least one of several sensors. The cleaning robot 100 can recognize the object as a door based on the result of inputting the image containing doors 700 and 710 into the artificial intelligence model.

[0087] If the detected object is a door, the cleaning robot 100 can determine the structure of the work surface (e.g., the structure of a house) through the door. For example, the cleaning robot 100 can identify both sides horizontally to doors 700 and 710 as walls, assuming no exceptions.

[0088] With reference to Fig. 7B The cleaning robot 100 can determine both directions horizontally to a detected first door 700 as a first wall 701 and both directions horizontally to a detected second door 710 as a second wall 711. The cleaning robot can determine a section 720, where the first wall 701 intersects the second wall 711, as the edge or corner of one or more walls of the house.

[0089] With reference to Fig. 7C, the cleaning robot 100 can detect an empty space in section 730 where the first wall 701 and the second wall 711 are expected to intersect.

[0090] For example, if an additional area 740 exists in the section where the two walls are expected to intersect, the cleaning robot 100 can detect this additional area 740 using its LiDAR sensor. Based on the detection result, the cleaning robot 100 can determine that a space exists between the first wall 701 and the second wall 711 and can then include the additional area 740 in that section of the work surface structure.

[0091] According to one embodiment, when the cleaning robot 100 recognizes an object as a door, it can not only recognize the object but also the structure of the work surface. Additionally, according to one embodiment, the cleaning robot 100 can generate a semantic map that represents the structure of the work surface in the navigation map.

[0092] Fig. 8A and Fig. 8B are views explaining that a cleaning robot generates a semantic map based on the structure of a house and additional information about an object according to one embodiment of the disclosure.

[0093] With reference to Fig. 8A, the cleaning robot can recognize 100 objects in a house as doors 700 and 710 and a sofa 800.

[0094] As in Fig. 7A and Fig. As explained in section 7B, the cleaning robot 100 can recognize the structure (e.g., walls in the house) of the work surface through doors 700 and 710. The cleaning robot 100 can recognize the sofa 800 as it moves towards the object. For example, with reference to Fig. 8A, the cleaning robot 100 can detect multiple images 801 to 803 of the object based on the detection result by camera 120 or at least one of several sensors while it moves towards the object. For example, the cleaning robot 100 can capture the object at each predetermined distance interval (e.g., 20 cm to 40 cm) or interval period (e.g., 0.5 sec to 2 sec) and obtain images 801 to 803 to store the images in memory 130.

[0095] When the cleaning robot 100 returns to its charging station after completing its work or to recharge, it can obtain object recognition information from images 801 to 803, which are stored in memory 130. The cleaning robot 100 can then apply these stored images 801 to 803 to its artificial intelligence model to obtain the object recognition information.

[0096] If the additionally detected object is a specific type of object (e.g., the Sofa 800) that is not an obstacle or foreign substance, the Cleaning Robot 100 can add this information to a navigation map in relation to the work surface and generate a semantic map.

[0097] For example, with reference to Fig. 8B, the cleaning robot 100 can generate a semantic map based on sofa leg information 810, which is additionally obtained through the LIDAR sensor in relation to the sofa 800, which is the structure, and derive the structure of the work surface through the doors 700 and 710.

[0098] Fig. 9A and Fig. 9B are views explaining how a cleaning robot informs a user about a hazardous material on the floor.

[0099] With reference to Fig. 9A, the cleaning robot 100 can recognize the object on the floor as a broken glass cup 900. The cleaning robot 100 can input an image containing the object 900 into the artificial intelligence model and perform object recognition to identify the object as a broken glass cup.

[0100] The Cleaning Robot 100 can inform a user that a hazardous object is on the floor. The Cleaning Robot 100 can transmit alarm data to the user terminal 90, so that an alarm message 911, for example, "This object should not be here on the floor" or "There is something here that should not be," can be displayed on the user terminal 910. Additionally, the Cleaning Robot 100 can transmit alarm data to the user terminal 910 so that the alarm message includes the object's detection information. For example, if the object is detected as a glass cup, the Cleaning Robot 100 can transmit alarm data to the user terminal 910 so that the alarm message (e.g., "A glass cup is on the floor") can be displayed.

[0101] The cleaning robot 100 can use the navigation map or the semantic map, which is based on the information in Fig. 7A to Fig. The data generated in the procedure shown in 8B is transmitted to the user terminal 910. The user terminal 910 can display the semantic map or the navigation map and show location information about the hazardous object received from the cleaning robot 100.

[0102] For example, with reference to Fig. 9B, the user terminal device 910 can display a user interface (UI) 912 in relation to the navigation map received from the cleaning robot 100 and display the location of the hazardous object 900 detected by the cleaning robot 100 on the UI.

[0103] Therefore, a user can easily recognize whether a dangerous object is falling to the ground or whether a dangerous object is present.

[0104] Fig. 10 is a view to explain that a cleaning robot according to one embodiment of the disclosure indicates an area that is not to be cleaned.

[0105] The user terminal 910 can receive the navigation map from the cleaning robot 100 and display its UI 912.

[0106] The user can specify an area 913 that should not be cleaned by the cleaning robot 100. For example, if it is necessary to restrict access to a specific area 930 on the navigation map (e.g., if a baby is sleeping), the user can instruct the cleaning robot 100 not to clean the specific area 930 by interacting with the area 913 displayed on the user device 910 (e.g., by touching, clicking, etc.). For example, the touch input can be made on a touchscreen display device.

[0107] The cleaning robot 100 can perform a task to automatically avoid access to a restricted specific area 913 without receiving any user command(s). For example, before accessing the specific area 913, the cleaning robot 100 can use its artificial intelligence model to detect an object contained within the specific area 913 (e.g., a sleeping baby). If, as a result of detecting the specific object (e.g., a sleeping baby), it is determined that access to the specific area 913 is necessary, the cleaning robot 100 can perform a task while moving and avoid the specific area 913.

[0108] Fig. 11A and Fig. Figure 11B are block diagrams illustrating a training module and a recognition module according to different embodiments of the disclosure.

[0109] With reference to Fig. 11A can contain a processor 1100 with at least one training module 1110 and one recognition module 1120. The processor 1100 of Fig. 11A can correspond to the processor 140 of the cleaning robot 100 or to a processor of an external server (not shown) that can communicate with the cleaning robot 100.

[0110] Training module 1110 can generate and train a recognition model that has predefined criteria for determining a situation. Training module 1110 can generate a recognition model with determination criteria using the collected training data.

[0111] The training module 1110 can generate, train, or renew an object recognition model that has criteria for determining which object is contained in the image, using the image containing the object as training data.

[0112] The training module 1110 can generate, train, or renew a peripheral information recognition model that has criteria for determining various additional information near the object contained in the image, using peripheral information contained in the screen containing the object as training data.

[0113] The training module 1110 can generate, train, or renew an obstacle detection model that has criteria for determining obstacles contained in the image, using the image captured by the camera as training data.

[0114] The recognition module 1120 can use predetermined data as input data for the trained recognition model and assume an object to be recognized within the predetermined data.

[0115] For example, the recognition module can obtain (or assume, infer, etc.) object information about an object contained in an object surface using the object surface (or image) containing the object as input data for the trained recognition model.

[0116] As another example, the recognition module 1120 can apply the object information to the trained recognition model to assume (or determine, infer, etc.) a search category in order to provide a search result. The search result can contain multiple search results according to priority.

[0117] At least one part of the training module 1110 or the recognition module 1120 can be implemented as a software module or in the form of at least one hardware chip for mounting on an electronic device. For example, at least one of the training module 1110 and the recognition module 1120 can be manufactured as a hardware chip solely for artificial intelligence (AI), or as part of an existing general-purpose processor (e.g., a CPU or application processor), or as part of a processor for graphics purposes (e.g., a GPU) for mounting on the cleaning robot 100. The hardware chip solely for artificial intelligence (AI) can be a processor designed for probability calculations, possessing higher parallel processing power than the conventional general-purpose processor, thereby enabling rapid arithmetic operations in the field of artificial intelligence, such as machine training.If the training module 1110 or the recognition module 1120 is implemented as a software module (or a program module containing an instruction), the software module can be a non-transient, machine-readable medium. In this case, the software module can be provided by an operating system (OS) or by a predefined application. Alternatively, some of the software modules can be provided by an operating system (OS), and others can be provided by a predefined application.

[0118] The training module 1110 and the recognition module 1120 can be mounted on one electronic device or on each of the electronic devices. For example, one of the training module 1210 and one of the recognition module 1320 can be contained in the cleaning robot 100, and the other can be contained in the external server. Additionally, the training module 1110 and the recognition module 1120 can be wired / wirelessly connected to provide the model information generated by the training module 1110 to the recognition module 1120, and the data entered into the recognition module 1120 can be provided to the training module 1110 as additional training data.

[0119] Fig. Figure 11B is a block diagram to explain a training module 1110 and a recognition module 1120 according to various embodiments of the disclosure.

[0120] Referring to part (a) of Fig. In one embodiment, the training module 1110 can comprise a data acquisition part 1110-1 and a model training module 1110-4. The training module 1110 can selectively include at least one of the training data pre-processor 1110-2, the training data selector 1110-3, or the model evaluation module 1110-5.

[0121] The 1110-1 training data acquisition unit can receive training data necessary for the recognition model to derive a recognizable object. The 1110-1 training data acquisition unit can receive a complete image containing the object, an image corresponding to the object's surface, and object information as training data. This training data can be data collected or tested by the 1110 training module or by the manufacturer of the 1110 training module.

[0122] The Model Training Module 1110-4 can train a recognition model to have predetermined criteria for determining how to identify a target object using training data. For example, the Model Training Module 1110-4 can train a recognition model through supervised learning using at least a portion of the training data as identification criteria. The Model Training Module 1110-4 can, for example, train itself using training data without additional supervised learning and train a recognition model through unsupervised learning to determine identification criteria for a given situation.

[0123] Additionally, the 1110-4 model training module can, for example, train a recognition model using reinforcement learning and feedback to determine whether the result of assessing the situation is appropriate according to the training. The 1110-4 model training module can, for example, train a recognition model using a training algorithm that includes a backpropagation error procedure or a gradient descent procedure.

[0124] The model training module 1110-4 can train criteria for determining which training data to use to predict an object to be recognized, using input data.

[0125] The 1110-4 model training module can, when multiple established recognition models exist, determine a recognition model with higher relevance between the input training data and the underlying training data. In this case, the underlying training data can be classified by data type, and the recognition model can be created in advance based on this data type. For example, the underlying training data can be pre-classified based on various criteria, such as the area where the training data is generated, the time at which the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, or the type of object in the training data, etc.

[0126] When the recognition model is trained, the model training module 1110-4 can store the trained recognition model. The model training module 1110-4 can store the trained recognition model in memory 130 of the cleaning robot 100. The model training module 1110-4 can store the trained recognition model in the memory of the server that is wired / wirelessly connected to the cleaning robot 100.

[0127] The training module 1110 may further include a training data preprocessor 1110-2 and a training data selector 1110-3 to improve the analysis result of the recognition model or to save resources or time required to generate a recognition model.

[0128] The 1110-2 training data preprocessor can preprocess the received data so that it can be used for training to determine a situation. The 1110-2 training data preprocessor can also convert the received data into a predefined format so that the 1110-4 model training module can use it for training to determine a situation.

[0129] The training data selector 1110-3 can select data received from the training acquisition unit 1110-1 or data pre-processed by the training data preprocessor 1110-2 as the data required for training. The selected training data can be provided to the model training module 1110-4. The training data selector 1110-3 can select the necessary training data from the received or pre-processed data according to predefined criteria. Additionally, the training data selector 1110-3 can select training data according to predefined criteria through training by the model training module 1110-4.

[0130] The training module 1110 may also include a model evaluation module 1110-5 to improve the analysis result of the data recognition model.

[0131] The model evaluation module 1110-5 can, if evaluation data is input into a recognition model but the analysis result output from the evaluation data does not meet a predetermined criterion, trigger the model training module 1110-4 to perform retraining. The evaluation data can be predefined data for evaluating the recognition model.

[0132] For example, if the number or ratio of evaluation data that is not exactly analyzed from the analysis results of the trained recognition model exceeds a predetermined threshold in relation to the evaluation data, the model evaluation module 1110-5 can evaluate that the data does not meet the predetermined criterion.

[0133] If the trained recognition model contains multiple trained recognition models, the Model Evaluation Module 1110-5 can evaluate whether each trained recognition model meets predetermined criteria and designate one that meets the predetermined criteria as the final recognition model. In this case, if the recognition model that meets the predetermined criteria contains multiple recognition models, the Model Evaluation Module 1110-5 can designate one or a predetermined number of recognition models, pre-configured in order of high evaluation scores, as the final recognition model.

[0134] Referring to part (b) of Fig. In some embodiments, the recognition module 1120 can include a data acquisition part 1120-1 and a recognition result provider 1120-4.

[0135] The recognition module 1120 can selectively include at least one of a recognition data preprocessor 1120-2, a recognition data selector 1120-3 and a model renewal module 1120-5.

[0136] The recognition data acquisition module 1120-1 can receive data necessary for situation determination. The recognition result provider 1120-4 can apply the data received from the recognition data acquisition module 1120-1 to the trained recognition model to determine a situation. The recognition result provider 1120-4 can provide the analysis result according to the data analysis purpose. The recognition result provider 1120-4 can apply data selected by the recognition data preprocessor 1120-2 or the recognition data selector 1120-3 as an input value to the recognition model to obtain the analysis result. The analysis result can be determined by the recognition model.

[0137] For example, the recognition result provider 1120-4 can apply the object area containing the object, obtained from the recognition data acquisition module 1120-1, to the trained recognition model and obtain (or assume) the object information corresponding to the object area.

[0138] As another example, the recognition result provider 1120-4 can apply at least one of object area, object information, or context information obtained from the recognition data acquisition module 1120-1 to the trained recognition model to obtain (or assume) a search category in order to provide the search result.

[0139] The recognition module 120 can also include the recognition data preprocessor 1120-2 and the recognition data selector 1120-3 to improve the analysis result of the recognition model or to save resources or time in providing the analysis result.

[0140] The recognition data preprocessor 1120-2 can preprocess the received data so that the data obtained can be used for situation determination. The recognition data preprocessor 1120-2 can produce the received data in a predefined format so that the recognition result provider 1120-4 can use the data obtained for situation determination.

[0141] The recognition data selector 1120-3 can select data received from the recognition data acquisition module 1120-1 or data preprocessed by the recognition data preprocessor 1120-2 as data necessary for situation determination. The selected data can be provided to the recognition result provider 1120-4. The recognition data selector 1120-3 can select a subset of all received or preprocessed data according to predefined criteria for situation determination. The recognition data selector 1120-3 can select data according to criteria preset by training the model training module 1110-4.

[0142] The model renewal module 1120-5 can be controlled to renew the detection model based on the analysis result provided by the detection result provider 1120-4. For example, the model renewal module 1120-5 can provide the analysis result provided by the detection result provider 1120-4 to the model training module 1110-4 to instruct the model training module 1110-4 to further train or renew the detection model.

[0143] Fig. Figure 12 is a view illustrating an example in which a cleaning robot 100 and a server 200 can be operated in conjunction with each other to train and recognize data.

[0144] With reference to Fig. The server can train 200 criteria for situation determination, and the cleaning robot can determine the situation based on the training result.

[0145] The model training module 1110-4 of server 200 can perform the function of the model training module 1110-4, which is in Fig. 11B is shown. The model training module 1110-4 of server 200 can train criteria for which object image, object information, or content information should be used to determine a predetermined situation, or how a situation should be determined using the data.

[0146] The recognition result provider 1120-4 of the cleaning robot 100 can apply the data selected by the recognition data selector 1120-3 to the recognition model generated by the server 200 to determine object information or a search category. The recognition result provider 1120-4 of the cleaning robot 100 can receive the recognition model generated by the server 200 and determine the situation using the received recognition model. In this case, the recognition result provider 1120-4 of the cleaning robot 100 can apply the object image selected by the recognition data selector 1120-3 to the recognition model received from the server 200 to determine object information according to the object image. The recognition result provider 1120-4 can determine the search category to obtain the search result using at least one of the contextual or contextual recognition information.

[0147] Fig. Figure 13 is a flowchart for explaining a network system using a recognition model according to one embodiment of the disclosure.

[0148] The first component 1301 can be the cleaning robot 100, and the second component 1302 can be the server 200, which stores a recognition model. The first component 1301 can be a general-purpose processor, and the second component 1302 can be a specific artificial intelligence processor. The first component 1301 can contain at least one application, and the second component 1302 can contain an operating system (OS). The second component 1302 can be more integrated, more specialized, have lower latency, be more powerful, or have more resources than the first component 1301 to process calculations required to generate, renew, or apply the data recognition model more quickly or effectively.

[0149] With reference to Fig. In step S1311, the first component 1301 can generate a captured image (e.g., the captured image) by capturing the environment containing the object (e.g., an object positioned in the workspace). For example, the first component 1301 can contain a camera that captures the captured image. The first component 1301 can transfer the captured image to the second component 1302 in step S1312. The first component 1301 can transfer information about the object area corresponding to the selected object along with the captured image.

[0150] The second component, 1302, can separate the received image into an object area and a peripheral area in step S1313. The second component, 1302, can separate the image into the object area and the peripheral area based on the received information about the object area.

[0151] The second component, 1302, can obtain object information and additional information about the object in step S1314 by inputting the separated object area and the peripheral area into the recognition model. The second component, 1302, can obtain object information by inputting the object area into the object recognition model and additional information about the object by inputting the peripheral area into the peripheral information recognition model. Additionally, the second component, 1302, can determine the search category and the priority of the search category based on the object information and the additional information about the object.

[0152] The second component 1302 can obtain the result regarding the object using the object information obtained and the additional information in step S1315. The second component 1302 can apply the object information and the additional information to the recognition model as input data and obtain the result regarding the object. The second component 1302 can obtain the result using the search category. The second component 1302 can obtain a result using additional data (e.g., the risk level of obstacles and / or the significance of the obstacle in relation to the user) in addition to the object information and the additional information. The additional data can be transferred from the first component 1301 or the other component, or pre-stored in the second component 1302.

[0153] When the second component 1302 transmits the result regarding the object to the first component 1301 in step S1316, the first component 1301 can detect the object through the sensor based on the result regarding the received object in step S1317.

[0154] Fig. Figure 14 is a flowchart to explain an example in which a cleaning robot provides a search result for a first area using a recognition model according to an embodiment of the disclosure.

[0155] With reference to Fig. In step S1410, the cleaning robot 100 can generate an image by recording or photographing its surroundings. The cleaning robot 100 can then obtain initial information about the first area using the first model, which was trained using the generated image as input data. This first model can be stored on the cleaning robot 100, but is not limited to this. It can also be stored on an external server.

[0156] The cleaning robot 100 can obtain second information about the second area from the trained second model, which uses the first information and the generated image as input data, in step S1430. The first model can be stored in the cleaning robot 100, but is not limited to this. The first model can also be stored on the external server.

[0157] If the first model and the second model are stored on the external server, the cleaning robot 100 can transmit the generated image to the external server, which can input the image into the first model to receive the first information, and input the image and the first information into the second model to receive second information.

[0158] Therefore, information regarding the first surface can be obtained more accurately by obtaining second information about the second surface, which may be a surface located near the first surface, as well as first information about the first surface in which the user input is detected.

[0159] Fig. Figure 15 is a flowchart for explaining a system using a recognition model according to one embodiment of the disclosure.

[0160] With reference to Fig. In step S1510, the cleaning robot 100 can generate an image by capturing and photographing an environment. In step S1520, the cleaning robot 100 can obtain initial information about the first area through the trained first model, which uses the generated image as input data.

[0161] The cleaning robot 100 can transmit the generated image and the first information to the server 200 in step S 1530.

[0162] In step S1540, server 200 can obtain the second information about the second surface through the trained second model, which uses the first information and the generated image as input data.

[0163] Server 200 can retrieve information regarding the first area based on the first information and second information in step S1550.

[0164] The server 200 can transmit the information regarding the first area (e.g., a search result relating to the first area) to the cleaning robot 100 in step S1560, and the cleaning robot 100 can provide the received information (e.g., the search result) in step S1570, such as by causing a display field to show the received search result.

[0165] The operation to obtain the initial information by the first model for object recognition can be performed by the cleaning robot 100, or the operation to obtain the second information by the second model for assuming context information can be performed by the server 200. In other words, the object recognition operation, which requires minimal processing power, can be performed by the cleaning robot 100, and the context estimation operation, which requires significant processing power, can be performed by the server 200.

[0166] With reference to Fig. Server 200 can receive initial or secondary information from the trained model and retrieve information relating to the initial area, but is not limited to this. Each of the multiple servers can perform the operation described above. That is, the first server can receive initial and secondary information from the trained model, and the second server can retrieve information about the initial area based on the initial and secondary information received from the first server, but is not limited to this. All processes performed by Server 200 can also be performed by the cleaning robot 100.

[0167] Fig. 16 is a view explaining the generation of a semantic map according to one embodiment of the disclosure.

[0168] With reference to Fig. 16. The cleaning robot 100 can use a navigation map as described in part (b) of Fig. 16 in relation to the work surface in part (a) of Fig. 16. For example, the cleaning robot can detect a work surface using at least one of an IR stereo sensor, an ultrasonic sensor, a LiDAR sensor, a position-sensitive diode (PSD) sensor, or an image sensor. The cleaning robot can generate a navigation map for driving the cleaning robot using the detection result of a work surface. It is preferred to generate a navigation map in 2D (two dimensions) in Fig. 16 to generate. The navigation map can, for example, be displayed as being divided by at least one 2D line.

[0169] The Cleaning Robot 100 can detect an object on the work surface using at least one of its sensors: the Camera 120, the object detection sensor, the IR stereo sensor, the ultrasonic sensor, the LiDAR sensor, or the image sensor. The cleaning robot can then apply the object detection result to its trained artificial intelligence model and obtain the object's recognition information. This object detection result can include, but is not limited to, the captured image of the object, its depth information, its material information, and / or its reflectance.

[0170] The cleaning robot 100 can assign a name to each of the one or more objects as in part (c) of Fig. 16 as the object recognition information. In addition, the cleaning robot 100 can receive at least one piece of information about the type of object, the size of the object (e.g., the height of the object, the width of the object, the depth of the object, etc.) or the characteristic of the object (e.g., the color of the object, the material of the object, etc.), but is not limited to this.

[0171] The Cleaning Robot 100 can generate a semantic map, providing environmental information for a work area where it is performing its tasks, by mapping the area of ​​the object contained in the navigation map using the object's recognition information. The Cleaning Robot 100 can identify the object's boundary according to the object in the navigation map. Once the object's boundary is identified, the Cleaning Robot 100 can determine the object's area defined by that boundary.

[0172] Once the object's surface area is determined, the cleaning robot 100 can map the object's surface using the object's recognition information.

[0173] The Cleaning Robot 100 can map the area of ​​the object contained in the navigation map using the object's recognition information, based on the object's location according to the object's detection result. The Cleaning Robot 100 can map the area of ​​the object contained in the navigation map using the object's recognition information if the object's location in a situation where the object is detected to generate a navigation map (e.g., when the object's detection result is obtained) and the object's location in a situation where the object is detected to obtain the object's recognition information (e.g., when the object's detection result is obtained or the object's image is saved) are within a threshold range of the same or very similar.

[0174] The Cleaning Robot 100 can map the area of ​​an object contained in the navigation map using the object's shape recognition information, based on the object's detection result. The Cleaning Robot 100 can compare the object's shape, as determined by the object's detection result, with the object's shape contained in the object's recognition information to generate the navigation map. If the two shapes are similar or identical, the robot can map the object's area using the object's recognition information.

[0175] Referring to part (d) of Fig. 16. The cleaning robot 100 can generate a semantic map, based on a mapping of the object's surface using the object's recognition information. This map specifies the environment of the work area. The cleaning robot 100 can display the generated semantic map on its display panel. If the cleaning robot 100 transmits the generated semantic map to the external user device, the user device can then display the semantic map on its display panel.

[0176] The user device or external server can generate a semantic map. For example, if the cleaning robot 100 transmits the navigation map and object detection information to the user device or external server, the user device or external server can generate a semantic map. Alternatively, if the cleaning robot 100 transmits the navigation map and object detection result to the user device or external server, the user device or external server can generate the semantic map.

[0177] The object's recognition information can be displayed as text (e.g., the object's name) or a symbol within the object area of ​​the semantic map. The object's recognition information can also be displayed in reference form within the object area of ​​the semantic map. More specifically, the object's recognition information can be displayed as the instruction line defining the object's area, or the object's area can be distinguished by color, allowing the object's recognition information to be displayed based on that color.

[0178] Fig. Figure 17 is a view illustrating a user interface for the use of a semantic map according to one embodiment of the disclosure.

[0179] Referring to part (a) of Fig. 17. The user terminal 1700 can display a semantic map. The semantic map can be created, for example, through the process of Fig. 16 can be generated. For example, the object recognition information can be displayed in the background of at least one of the 3D (dimension) maps, which depicts the structure of the workspace, or in the navigation map (e.g., LiDAR map) in the semantic map.

[0180] For example, bed 1701 can be displayed as the object's recognition information in at least one part of the object area of ​​the semantic map. In this case, a user can select bed 1701, which contains the recognition information.

[0181] In response to a selection made by a user, the user terminal device 1700 can display a drop-down list 1710 relating to the selected recognition information 1701, as shown in part (b) of Fig. 17. The names of the objects representing the selected recognition information can be displayed in drop-down list 1710. For example, as a result of applying the object detection result to the trained artificial intelligence model, the object recognition result might be 50% for bed, 30% for sofa, 20% for table, and so on. In this case, the names applicable to the objects can be displayed sequentially in drop-down list 1710, in order of highest recognition results. For example, bed 1701 can be displayed in the first field, sofa 1702 in the second field, and table 1703 in the third field of drop-down list 1710.

[0182] If a user enters a name (e.g., table 1703), referring to part (c) of Fig. 17, selects, the user terminal device 1700 can display recognition information 1703 of the changed object on the navigation map.

[0183] Fig. Figure 18 is a view illustrating a situation in which a cleaning robot is controlled using a semantic map according to one embodiment of the disclosure.

[0184] With reference to Fig. 18. The user can execute a control command using recognition information of the object displayed on the semantic map provided by the user terminal device 1700. For example, recognition information of at least one object displayed on the semantic map in Fig. 16 and Fig. If 17 is displayed, 1700 will be shown on the user terminal's display panel. In this case, the user can say 'please clean the front of the TV' to specify the cleaning area.

[0185] Based on the user's spoken command, the user terminal 1700 can recognize the user's spoken command and transmit the user's control command to the cleaning robot 100 accordingly. The control command can be a command to perform work on the surface of a specific object. The cleaning robot 100 can then perform a task based on the user's control command. For example, the cleaning robot 100 can move to the TV and clean the front of the TV.

[0186] Fig. 19 is a view to explain how to generate a semantic map according to one embodiment of the disclosure.

[0187] With reference to Fig. 19. The cleaning robot 100 can be used in part (b) of Fig. 19 a navigation map in relation to the work area in part (a) of Fig. 19. Generate. The detailed description of generating the navigation map can be found in the descriptions of parts (a) and (b) of Fig. 16 correspond. Therefore, a repeated description is omitted.

[0188] The cleaning robot 100 can obtain recognition information for each location on the work surface using the work surface detection result in part (a) of Fig. 19 received. For example, the cleaning robot 100 can name each place in part (c) of Fig. 19 as identification information received.

[0189] The cleaning robot 100 can apply multiple images, capturing respective locations on the work surface, to the trained artificial intelligence model in order to obtain the recognition information for each location on the work surface.

[0190] With reference to Fig. 7A and Fig. 7B, the cleaning robot 100 can detect the object to recognize doors 700 and 710 and determine the structure of the work surface using the detected doors 700 and 710. The cleaning robot 100 can obtain recognition information for each area that is distinguished as part of the work surface structure. For example, the cleaning robot 100 can best recognize doors 700 and 710, identifying the larger area as the living room and the second largest area as the bedroom.

[0191] The Cleaning Robot 100 can obtain recognition information for each location on the work surface using the recognition information of the object located at each location. For example, the Cleaning Robot 100 can identify the area with the table as the kitchen, the area with the bed as the bedroom, and the area with the TV or sofa as the living room.

[0192] If recognition information for each location is obtained, the cleaning robot 100 can create a semantic map of the work surface environment as described in part (d) of Fig. As specified in section 19, the robot generates a semantic map using the navigation map and the received recognition information for each location. The cleaning robot 100 can display the generated semantic map on its display panel. Additionally, if the cleaning robot 100 transmits the generated semantic map to an external user device, the user device can display the semantic map on its display panel.

[0193] The user device or external server can generate a semantic map. For example, if the cleaning robot 100 transmits the navigation map and detection information for each location to the user device or external server, the user device or external server can generate the semantic map. Alternatively, if the cleaning robot 100 transmits the navigation map and the detection result of the object located at each location on the work surface to the user device or external server, the user device or external server can generate a semantic map.

[0194] Fig. Figure 20 is a view illustrating a user interface for the use of a semantic map according to one embodiment of the disclosure.

[0195] Referring to part (a) of Fig. 20. A user terminal device can display a semantic map. The semantic map can be generated, for example, by the process in Fig. 19 can be generated. For example, the object recognition information can be displayed in the background of at least one of the 3D maps that depict the structure of the workspace, or the navigation map on the semantic map.

[0196] For example, a living room 2001 can be displayed as recognition information for a space on at least part of the object. In this case, the user can select the living room 2001 as the recognition information.

[0197] In response to a user's selection, the user terminal device 2000 can display a drop-down list 2010 relating to the selected recognition information 2001 in part (b) of Fig. Display 20, where the names of places to be replaced by the selected recognition information can be displayed in the 2010 dropdown list. For example, the result of applying the image taken at a location to the trained artificial intelligence model might be a recognition result of 50% for a living room, 30% for a bedroom, and 20% for a study. In this case, the names applicable to a place can be displayed sequentially in the 2010 dropdown list, according to the recognition result value. For example, the living room (2001) could be displayed in the first field, the bedroom (2002) in the second field, and a study (2003) in the third field of the 2010 dropdown list.

[0198] If the user enters a name (e.g. the study 2003) with reference to part (c) of Fig. If 20 is selected, the user terminal device can display 2000 detection information 2003 of the changed location on the navigation map.

[0199] Fig. Figure 21 is a view illustrating a situation in which a cleaning robot is controlled using a semantic map according to an embodiment of the disclosure.

[0200] With reference to Fig. 21. The user can execute a control command using recognition information for each location displayed on the semantic map provided by the user terminal device 2000. For example, the recognition information for each location on the semantic map can be found in Fig. 19 and Fig. The number 20 will be displayed on the user terminal 2000's display panel. In this case, the user can say 'please clean the living room' to specify the cleaning area.

[0201] Based on a user-spoken command, the user terminal 2000 can recognize the user-spoken command and transmit the user's control command to the cleaning robot 100 accordingly. The user's control command can be a command requesting the robot to perform work in a specific location. The cleaning robot 100 can then perform a task based on the user's control command. For example, the cleaning robot 100 can move to the living room and clean it.

[0202] A user terminal device can display a semantic map that includes both the object recognition information in part (a) of Fig. 17 as well as recognition information for each place in part (a) of Fig. 20 contains. In this case, if either object recognition information or recognition information for each location is selected, the user terminal device can provide a user interface that modifies the selected recognition information.

[0203] For example, the user terminal can provide a candidate list that can modify the selected recognition information. The changeable names can be included in the candidate list and ordered according to their high probability values, taking into account the recognition outcome of the artificial intelligence model. When user input is provided to select a name, the user terminal can modify the existing object's recognition information to the selected name and display the information.

[0204] Fig. Figure 22 is a view illustrating a process for recognizing an object according to one embodiment of the disclosure.

[0205] The cleaning robot 100 can apply the image of the object captured by camera 120 to the trained artificial intelligence model to obtain object recognition information. The cleaning robot 100 can also apply the image of the space within the work area, captured by camera 120, to obtain space recognition information.

[0206] As another example, if the cleaning robot 100 detects an object in a specific area, it can apply the captured image to the trained artificial intelligence model to obtain the object detection information along with the location detection information. More precisely, an electronic device containing the artificial intelligence model 2200 (e.g., an external server) can apply the feature map in a vertical direction 2220, generated by an object detection network 2210 (a convolutional network model), to the classifier and execute an object detection module 2230 to detect the object and a location detection module 2240 to detect the object's location. However, the training steps can be simplified by training both the object detection and location detection modules.

[0207] Regarding the captured image, when object and location recognition is performed, a more precise object identification may be possible. For example, the object recognition result for the object contained in the captured image might be 50% for a table, 30% for a dining table, and 20% for a desk. If the location is identified as a kitchen, an electronic device might recognize the object as a dining table, but not as a table. As another example, if the object and location are identified as a table and a study, respectively, the object might be identified as a desk. Similarly, if the object and location are identified as a front door and a door, respectively, the object might be identified as the front door. Finally, if the object and location are identified as a room and a door, respectively, the object might be identified as the door.

[0208] As another example, if the object and the space are recognized as a threshold or room, the object can be recognized as a threshold. Similarly, if the object and the space are recognized as a threshold or balcony, the object can be recognized as a balcony threshold.

[0209] If, in relation to the captured image, an object and the location containing the object are jointly detected by a single network, the electronic device 2200 or the cleaning robot 100 can effectively generate a semantic map that specifies the environment of the work area. For example, not only the object's detection information but also its location information can be displayed on the semantic map.

[0210] Fig. 23 is a view explaining a process for generating a semantic map according to one embodiment of the disclosure.

[0211] When the cleaning robot 100 applies the image captured by the camera 120 to the network contained in the electronic device 2200, at least one of the object recognition information or the place recognition information (e.g., context of the place) can be displayed on the semantic map.

[0212] For example, the cleaning robot 100 can transfer the captured image to the object recognition module 2230 in Fig. 22 apply to create an initial semantic map as in part (a) of Fig. 23 to generate. As the recognition information of the object in the first semantic map of part (a) of Fig. 23. The object's name can be displayed in the position corresponding to the object. The cleaning robot 100 can display a second semantic map as in part (b) of Fig. 23 by applying the captured image to a place detection module 2240 of Fig. 22. The name of the place can be displayed on the area corresponding to the place, from subdivided areas of the workspace in the semantic map as the recognition information of the place.

[0213] The cleaning robot 100 can generate the first semantic map of part (a) of Fig. 23 with the second semantic map of part (b) of Fig. 23 combine to generate a final semantic map. The name and location of the object present in the workspace can be combined in the final semantic map of part (c) of Fig. 23 are displayed as environment information for the workspace.

[0214] Fig. Figure 24 is a view illustrating a situation in which a cleaning robot is controlled using a semantic map according to an embodiment of the disclosure.

[0215] With reference to Fig. 24. The user can execute a control command using recognition information for each object and location displayed on the semantic map provided by a user terminal device 2400. For example, the recognition information for each object and location displayed on the semantic map of Fig. 23 will be displayed, while 2400 will be displayed on the user terminal's display panel. In this case, a user can say 'please clean the front of the table in the living room' to specify the cleaning area.

[0216] Based on a user-spoken command, the user terminal 2400 can recognize the user's spoken command. The user terminal 2400 can then transmit the user's control command, corresponding to the recognized spoken command, to the cleaning robot 100. The control command can be a command to perform a task related to the surface of a specific object located in a specific place. The cleaning robot 100 can then perform a task based on the user's control command. For example, the cleaning robot 100 can move from one of the seating areas (the living room or the study) to the table in the living room and clean the front of the table.

[0217] In various embodiments, in Fig. 18, Fig. 21 and Fig. 24. The cleaning robot 100 can directly recognize the user-spoken command. In this case, the cleaning robot 100 may contain at least one automatic speech recognition module (ASR module), a natural language understanding module (NLU module), a path planning module, or the like. If a user speaks to specify the cleaning area, the cleaning robot 100 can recognize the user-spoken command using at least one of these modules. The cleaning robot 100 can then perform a task according to the recognized user control command.For example, the cleaning robot 100 can perform a job according to a control command from the user based on the environmental information contained in the stored semantic map (e.g. the name of the object, the name of the object, etc.).

[0218] With reference to Fig. 18, Fig. 21 and Fig. 24. Upon receiving a user-spoken command, a third voice recognition device (e.g., a voice recognition home hub, an artificial intelligence speaker, etc.) can recognize the user's spoken command. This third voice recognition device can include at least one automatic voice recognition module, a natural language understanding module, and a path planner module. The third voice recognition device can recognize the user's spoken command and transmit the corresponding user control command to the cleaning robot 100. Therefore, the cleaning robot 100 can perform a task according to the user's control command while minimizing the constraints of space or equipment.

[0219] Fig. Figure 25 is a view illustrating a configuration of a cleaning robot according to an embodiment of the disclosure.

[0220] part (a) of Fig. Figure 25 is a perspective view illustrating a cleaning robot 100 which contains several sensors, and part (b) of Fig. 25 is a 'front view' illustrating the cleaning robot 100, which contains several sensors.

[0221] The cleaning robot 100, which is in Fig. Figure 25 shows that it can contain multiple sensors and may include at least one IR stereo sensor, LIDAR sensor, ultrasonic sensor, 3D sensor, material detection sensor, fall detection sensor, position-sensitive diode sensor (PSD sensor) or the like.

[0222] The function of the IR stereo sensor, the LIDAR sensor, and the ultrasonic sensor was described in detail with reference to Fig. 2 described, and their detailed description is omitted. The ultrasonic sensor can, for example, consist of two light-emitting sensor modules and two light-receiving sensor modules. The Cleaning Robot 100 can detect the object on its front using a 3D sensor and extract a 3-dimensional shape of the object. Therefore, the Cleaning Robot 100 can obtain information about the size and distance of the object. The Cleaning Robot 100 can detect the object on its front using an object recognition sensor to obtain the type of object. For example, the Cleaning Robot 100 can capture an object using a camera included in the object recognition sensor and apply the captured image to the trained artificial intelligence model to obtain the type of object as the application result. The Cleaning Robot 100 can detect a step in the floor (e.g.,The robot can detect obstacles such as a step (between the living room floor and the front door) when moving back and forth using a fall detection sensor. The fall detection sensor can be located, for example, on the front or back of the cleaning robot. The cleaning robot 100 can detect the location of an object at close range (e.g., within 15 cm) using the PSD sensor. For example, the cleaning robot 100 can detect an object using the PSD sensor or move along a wall to perform cleaning. The PSD sensor can be located, for example, on the left or right side, angled outwards at 45 degrees when facing forward, or on the right and left surfaces of the cleaning robot 100.

[0223] With reference to Fig. 25. Several sensors can be arranged on the front, back, and sides of the cleaning robot 100. For example, at least one ultrasonic sensor 2501 to 2504, at least one 3D sensor 2511, at least one camera 2521 (e.g., an RGB camera), and at least one IR stereo sensor 2531 (e.g., a docking IR stereo sensor), PSD sensors 2541 and 2542, one LiDAR sensor 2551, or one obstacle sensor 2561 can be provided. Additionally, another PSD sensor (not shown) can be provided on the side of the cleaning robot 100.

[0224] Fig. 26A and Fig. Figure 26B are views illustrating a detection (capture) area of ​​a cleaning robot according to an embodiment of the disclosure.

[0225] part (a) of Fig. 26A is a top view illustrating the cleaning robot 100, part (b) of Fig. 26A is a side view illustrating the cleaning robot 100, and parts (c) and (d) of Fig. Figures 26B are perspective views illustrating the Cleaning Robot 100.

[0226] With reference to Fig. 26A and Fig. 26B, the cleaning robot 100 can detect the object or structure of a house while driving, using multiple sensors.

[0227] With reference to Fig. 26A and Fig. 26B, the detection range of the 3D sensor 2511 can be a first range 2510a, the detection range of the ultrasonic sensors 2501 to 2504 can be a second range 2500a, the detection ranges 2541 and 2542 of the front-mounted PSD sensor can be third ranges 2540a and 2540b, the detection ranges of the side-mounted PSD sensor 2543 (not shown) can be fourth ranges 2540c and 2540d, and the detection range of the LiDAR sensor 2551 can be a fifth range 2550a. The viewing angle of the camera 2521 can be a sixth range 2520a, but is not limited to it. The detection or recording range can be predicted differently based on the specifications of the multiple sensors or the locations where multiple sensors are mounted.

[0228] The Cleaning Robot 100 can select at least one sensor from several sensors based on the object's detection information. Using the selected sensor, the Cleaning Robot 100 can detect an object and obtain additional information about the object based on the detected result. The Cleaning Robot 100 can then determine the task to be performed based on this additional information.

[0229] Fig. 27, Fig. 28 and Fig. Figure 29 are flowcharts to explain a cleaning robot according to one embodiment.

[0230] With reference to Fig. 27. The cleaning robot 100 can generate a navigation map to drive the cleaning robot 100 using the result that at least one sensor detects the work surface in which the object is located, in step S2701.

[0231] The cleaning robot 100 can obtain object recognition information by applying the object image captured by camera 120 to the trained artificial intelligence model in step S2702. The operation to obtain object recognition information can be performed before the cleaning robot 100 generates a navigation map in step S2701, or the object recognition information can be obtained during the navigation map generation process.

[0232] The cleaning robot 100 can obtain the object recognition information by applying the image of the object, captured by the camera 120, to the trained artificial intelligence model located on the external server.

[0233] Once the object's recognition information is obtained, the cleaning robot 100 can map the area of ​​the object contained in the navigation map using the object's recognition information and generate a semantic map indicating the work area's environment in step S2703.

[0234] The cleaning robot 100 can perform a cleaning robot task based on the user's control command using a semantic map in step S2704. The user's control command can be a command to request the execution of the task with respect to the object surface or the specific location.

[0235] The Cleaning Robot 100 can obtain the recognition information for each location within the work area. Using this information and the object's recognition information, the Cleaning Robot 100 can generate a semantic map that defines the work area's environment.

[0236] The cleaning robot 100 can generate a semantic map that specifies the environment of the work surface by mapping the area of ​​the object contained in the navigation map with the object's recognition information based on at least one of the object's location or shape according to the object's detection result.

[0237] The Cleaning Robot 100 can identify the object's boundary according to the object in the navigation map. The Cleaning Robot 100 can map the area defined by the object's boundary using the object's recognition information to generate a semantic map that specifies the work area's surroundings.

[0238] The cleaning robot 10 can detect the object using at least one sensor, which is selected from several sensors contained in the sensor 110 based on the object's detection information. The cleaning robot 100 can obtain additional information about the object using the result detected by at least one sensor.

[0239] The Cleaning Robot 100 can prioritize multiple sensors based on object detection information. The Cleaning Robot 100 can also obtain additional object-related information based on the results detected by at least one sensor, according to a priority from the other sensors.

[0240] Fig. Figure 28 is a flowchart to explain a cleaning robot according to another embodiment of the disclosure.

[0241] With reference to Fig. 28 The cleaning robot 100 can generate a navigation map to drive the cleaning robot 100 using the result of at least one sensor that detects the work surface in step S2801.

[0242] The cleaning robot 100 can obtain recognition information of the space contained in the work surface by applying the image of the space captured by the camera 120 contained in the work surface to the trained artificial intelligence model in step S2802.

[0243] If recognition information of the place contained in the work area is obtained, the cleaning robot 100 can generate a semantic map indicating the environment of the work area by mapping the area according to the place contained in the navigation map, using the recognition information of the place in step S2803.

[0244] The cleaning robot 100 can perform the work of the cleaning robot 100 based on the user's control command using the semantic map in step S2804.

[0245] Fig. Figure 29 is a flowchart to explain a cleaning robot according to one embodiment of the disclosure.

[0246] With reference to Fig. 29. The cleaning robot 100 can pick up the object near the cleaning robot 100 in step S2901.

[0247] The cleaning robot 100 can obtain object recognition information from the image by applying the captured image to the trained artificial intelligence model in step S2902. For example, the cleaning robot 100 can obtain object recognition information by applying the captured image to the trained artificial intelligence model located on the external server.

[0248] The cleaning robot 100 can obtain the additional information about the object using the result detected by at least one sensor, which is selected from multiple sensors based on the object's detection information obtained, in step S2903.

[0249] For example, the cleaning robot 100 can obtain additional information about the object by selectively using the result detected by the at least one sensor, chosen based on the object's recognition information, from detection results detected by multiple sensors within a predetermined time (e.g., 10 ms), based on a predetermined time period. At least one sensor, selected based on the object's recognition information, can contain one or more sensors. When multiple sensors are selected based on the object's recognition information, the selected sensors can have priority. The cleaning robot 100 can assign a weighted value to the highest-priority detection result to obtain additional information about the object.

[0250] If the IR stereo sensor is given a higher priority among several sensors, the cleaning robot can obtain additional information about the object by assigning a weighted value to the result detected by the IR stereo sensor. Specifically, the cleaning robot can determine the boundary frame for the object and, with respect to the area where the boundary frame determination result does not match the object detection result from the IR stereo sensor, lower the IR stereo sensor's threshold to detect the object.

[0251] The cleaning robot 100 can set priorities for multiple sensors according to the object's detection information and obtain additional information about the object using the result detected by at least one sensor, according to the priority of the multiple sensors.

[0252] If the LIDAR sensor is given a higher priority among several sensors according to the object's detection information, the cleaning robot 100 can assign a weighted value to the result detected by the LIDAR sensor in order to obtain the additional information about the object.

[0253] If the ultrasonic sensor is given a higher priority among the multiple sensors based on the object's detection information, the cleaning robot 100 can assign a weighted value to the result detected by the ultrasonic sensor and obtain additional information about the object. If the ultrasonic sensor is given a higher priority among the multiple sensors based on the object's detection information, the detected object can be transparent or black.

[0254] Based on the additional information about the object, the cleaning robot 100 can determine the work to be performed by the cleaning robot 100 in step S2904.

[0255] Various embodiments of the disclosure can also be implemented in a mobile device. The mobile device can, for example, take various forms, such as a service robot for a public space, a transport robot at a production facility, an operator-assisted robot, a household robot, a security robot, an autonomous vehicle, or the like.

[0256] In this case, the work disclosed can be any work in accordance with the purpose of the mobile device. For example, if the work of a cleaning robot is to avoid an object or vacuum up dust in a house, the work of an operator-assisted robot can be to avoid or move an object. Additionally, the work of a security robot can be to avoid an object, detect an intruder to trigger an alarm, or photograph the intruder. Furthermore, the work of an autonomous vehicle can be to avoid another vehicle or obstacle, or to control a steering or acceleration / deceleration device.

[0257] The term “module,” as used in this disclosure, may include units implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, circuit, or the like. The module may be an integrally constructed component, a minimal unit of the component, or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0258] Various embodiments of the disclosure can be implemented as software containing instructions stored in machine-readable storage media. The machine can be a device that calls an instruction stored in a storage medium and is operational according to the called instruction, comprising an electronic device according to the disclosed exemplary embodiments (e.g., an electronic device (A)). When the instruction is executed by a processor, the processor can perform the function according to the instruction, either directly or under the control of the processor using other components.

[0259] Various embodiments of the disclosure can be implemented as software (e.g., a program) containing one or more instructions stored in a memory medium (e.g., memory 130, memory on the server (not shown)) that is readable by a machine (not shown) (e.g., the cleaning robot 100 and a server (not shown) communicating with the cleaning robot 100). For example, a processor of the device (e.g., processor 140, a processor of the server (not shown)) can call and execute at least one of the stored instructions from a memory medium. This enables the device to be operated in such a way that it performs at least one function according to the at least one called instruction. The instruction can contain code that is generated or executed by a compiler or an interpreter.The machine-readable storage medium can be provided in the form of a non-transient storage medium. 'Non-transient' means that the storage medium does not contain a signal (e.g., an electromagnetic wave) and is tangible, but it does not distinguish whether data is stored semi-permanently or temporarily on a storage medium.

[0260] According to one embodiment, the method disclosed herein according to various embodiments can be provided in a computer program product. A computer program product can be traded as a commodity between a seller and a buyer. A computer program product can be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only storage (CD-ROM)) or online (e.g., download or upload) between two user devices (e.g., smartphones) through an app store (e.g., Play Store). (R)) are distributed. In the case of online distribution, at least one section of the computer program product may be temporarily stored or temporarily created on a storage medium such as a manufacturer's server, an app store server, or a relay server.

[0261] Each of the components (e.g., modules or programs) according to different embodiments can consist of a single unit or multiple units, and some subcomponents of the aforementioned subcomponents can be omitted, or other components can be additionally included in different embodiments. Alternatively or additionally, some components (e.g., modules or programs) can be integrated into a single unit to perform the same or similar functions that each component performed before integration. Operations performed by modules, programs, or other components according to different embodiments can be executed sequentially, in parallel, repeatedly, or heuristically, or at least some operations can be performed in a different order or omitted, or an additional function can be added.

[0262] Although exemplary embodiments have been shown and described, it will be clear to those skilled in the art that modifications can be made to these exemplary embodiments without departing from the principles and essence of the present disclosure. Accordingly, the scope of the present invention is not to be construed as being limited to the exemplary embodiments described, but is defined by the appended claims and their equivalents. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Study 2003

[0198]

Claims

[1] Cleaning robots, comprehensive: a communication interface; at least one sensor; at least one camera (120); a drive unit; and a processor (140) that is configured: to input an image taken by at least one camera into a trained artificial intelligence model in order to obtain information about an object contained in the captured image: to obtain information that is captured by at least one sensor; To obtain recognition information about each of several work surfaces in a home based on information about the object, wherein the recognition information about each of the several work surfaces includes type information about each of the several work surfaces; to generate a map that displays the multiple work areas, using the information obtained about the object, the captured information, and the recognition information for each of the multiple work areas; based on a user voice instructing a cleaning operation to control the drive unit to move to at least one of the multiple work surfaces; and based on moving to the at least one work surface, a cleaning operation for the at least one work surface is to be carried out, where the recognition information for each of the multiple work surfaces includes a name for each of the multiple work surfaces. [2] Cleaning robot according to claim 1, wherein the user voice contains a word relating to at least one work surface and a word indicating an operation performed by the cleaning robot; and wherein the word relating to at least one work surface corresponds to recognition information about the at least one work surface. [3] Cleaning robot according to claim 1 or 2, wherein the processor (140) is configured: based on a user command that specifies one or more of the multiple work surfaces as an avoidance area, which is received by an external end device through the communication interface, to control the drive unit to drive through remaining surfaces excluding the avoidance area. [4] Cleaning robot according to one of the preceding claims, wherein the map is displayed by an external end device that cooperates with the cleaning robot; and wherein the map contains text or a symbol that provides recognition information about the multiple work surfaces. [5] Cleaning robot according to one of the preceding claims, wherein the at least one sensor includes at least one IR stereo sensor, an ultrasonic sensor, a LiDAR sensor or a PSD sensor.

Images