Robot and method for operating robot

Abstract

The present disclosure provides a method for operating a mobile robot. The mobile robot may include: an operation of detecting an object; an operation of identifying a type of the object; an operation of identifying whether user feedback for correction associated with the type of the object is necessary; an operation of acquiring information on a distance between the mobile robot and a user on the basis of identifying that the user feedback is necessary; an operation of identifying whether the distance between the mobile robot and the user is within a designated distance; and an operation of, on the basis of identifying that the distance between the mobile robot and the user is within the designated distance, providing call information for calling the user, and displaying, through a display, a user interface for acquiring user feedback.

Description

Robots and methods of robot operation

[0001] The present disclosure relates to a robot and a method of operating the robot.

[0002] With the recent advancement of electronic technology, various types of electronic devices are being developed and distributed.

[0003] For example, robots for various purposes are being deployed in factories and homes. These robots can perform specific functions / operations, such as grasping external objects and moving them from one location to another, or transporting them. For example, a robot deployed in a home may move objects that require organization to specific locations to organize them. To accurately perform these organizing operations, it is necessary to accurately identify or classify the objects detected by the robot (e.g., classify the object type).

[0004] To improve classification accuracy, robot manufacturers typically collect data directly and regularly or irregularly update the classification algorithms used for classification. However, this approach makes accurate classification of unrecognized or misrecognized objects impossible until the manufacturer updates the classification algorithm. Therefore, methods that enhance classification accuracy through direct interaction between the robot and the user are needed.

[0005] According to one embodiment of the present disclosure, a mobile robot may include a communication circuit; a display; a memory storing instructions; and at least one processor connected to the memory and executing the instructions stored in the memory. The at least one processor may: detect an object, identify a type of the object, wherein the type of the object is one of a cleaning object for which the mobile robot is configured to perform a cleaning operation, an avoidance object for which the mobile robot is configured to perform an avoidance operation, or an unknown object, identify whether user feedback for correction associated with the type of the object is required, and, based on the identification that the user feedback is required, obtain information about a distance between the mobile robot and a user, identify whether the distance between the mobile robot and the user is within a specified distance, provide call information for calling the user based on the identification that the distance between the mobile robot and the user is within the specified distance, and display a user interface for obtaining the user feedback through the display.

[0006] According to one embodiment, the at least one processor can: obtain user feedback through the user interface, and correct the type of the object based on the user feedback.

[0007] According to one embodiment, the at least one processor: identifies whether the corrected type matches the identified type, and if it is determined that the corrected type does not match the identified type, displays information for reconfirming the correction of the type of the object through the display, and if it is determined that the corrected type matches the identified type, transmits a feedback message including information about the detected object and information about the corrected type to a server through the communication circuit, wherein the feedback message can be used by the server to update a learned artificial intelligence (AI) model used to classify the type of the object.

[0008] According to one embodiment, the at least one processor: performs a designated action corresponding to the type of the mobile robot based on the identification that the user feedback is not required, wherein the designated action may be one of a cleaning action for cleaning the cleaning object, an avoidance action for avoiding the avoidance object, or an action for correcting the unknown object.

[0009] According to one embodiment, the at least one processor may: transmit a message to the electronic device through the communication circuit for displaying notification information on a display of the electronic device of the user connected to the mobile robot to inform the user of the current status of the mobile robot based on the distance between the mobile robot and the user being identified as being outside the specified distance.

[0010] In one embodiment, the at least one processor can identify the type of the object by using a learned AI model to classify the object as one of the cleaned object, the avoided object, or the unknown object.

[0011] According to one embodiment, the at least one processor: inputs input data based on image data of the object into the learned AI model, obtains information indicating a type of the object as output data of the learned AI model, and identifies the type of the object based on the information indicating the type, wherein the information indicating the type may be set to one of a first value indicating that the object is the cleaned object, a second value indicating that the object is the avoided object, or a third value indicating that the object is the unknown object.

[0012] According to one embodiment, the at least one processor may: identify that user feedback is required if the object is identified as the unknown object based on the type of the object.

[0013] According to one embodiment, the at least one processor may: obtain information about the accuracy of the type classification of the object, identify whether the accuracy of the type classification is below a reference accuracy, and, if the accuracy of the type classification is identified as being below a reference accuracy, identify that the user feedback is required.

[0014] According to one embodiment, the at least one processor may: stop the movement operation of the mobile robot for a preset period of time after providing the call information.

[0015] According to one embodiment, the at least one processor may: when a user confirmation response corresponding to the call information is received within a first preset period after providing the call information, stop the movement operation of the mobile robot for a second preset period after the time at which the user confirmation response is received or for a third preset period after providing the call information.

[0016] According to one embodiment, the mobile robot includes: at least one sensor; a driving unit for moving the mobile robot; and a manipulator for organizing the organizing object, wherein the manipulator may include a gripper for gripping the organizing object for organizing.

[0017] In one embodiment, the at least one processor: provides a second user interface for obtaining user feedback for correction of at least one unknown object detected during the operation of the mobile robot after the operation of the mobile robot has ended, wherein the second user interface may be a user interface for correcting a type of one of the at least one unknown object or a user interface for correcting a type of at least one of the clustered one or more unknown objects.

[0018] According to one embodiment of the present disclosure, a method for operating a mobile robot may include: detecting an object; identifying a type of the object, wherein the type of the object is one of a cleaning object, which is an object configured to perform a cleaning operation on the mobile robot, an avoidance object, which is an object configured to perform an avoidance operation on the mobile robot, or an unknown object; identifying whether user feedback for correction associated with the type of the object is required; acquiring information about a distance between the mobile robot and a user based on the identification that the user feedback is required; identifying whether the distance between the mobile robot and the user is within a specified distance; and providing call information for calling the user and displaying a user interface for obtaining the user feedback through a display based on the identification that the distance between the mobile robot and the user is within the specified distance.

[0019] According to one embodiment, the method may include: obtaining user feedback through the user interface; and correcting the type of the object based on the user feedback.

[0020] According to one embodiment, the method includes: an operation of identifying whether the corrected type matches the identified type; an operation of displaying information for reconfirming correction of the type of the object through the display when it is identified that the corrected type matches the identified type; and an operation of transmitting a feedback message including information about the detected object and information about the corrected type to a server through a communication circuit when it is identified that the corrected type matches the identified type, wherein the feedback message can be used by the server to update a learned AI model used to classify the type of the object.

[0021] According to one embodiment, the method comprises: performing a designated action corresponding to the type of the mobile robot based on the identification that the user feedback is not required, wherein the designated action may be one of a cleaning action for cleaning the cleaning object, an avoidance action for avoiding the avoidance object, or an action for correcting the unknown object.

[0022] According to one embodiment, the method may include: transmitting a message to an electronic device of the user connected to the mobile robot through a communication circuit for displaying notification information for informing the user of the current situation of the mobile robot on a display of the electronic device of the user based on the distance between the mobile robot and the user being identified as being outside the specified distance.

[0023] In one embodiment, the act of identifying the type of the object may include: identifying the type of the object by using a learned AI model to classify the object as one of the cleaned object, the avoided object, or the unknown object.

[0024] In one embodiment, the act of identifying that the user feedback is required may include: identifying that the user feedback is required if the object is identified as the unknown object based on the type of the object.

[0025] FIG. 1 illustrates a robot intelligence system according to one embodiment of the present disclosure.

[0026] FIG. 2A is a drawing for explaining a cleaning robot according to an embodiment of the present disclosure.

[0027] FIG. 2b is a block diagram illustrating a cleaning robot according to an embodiment of the present disclosure.

[0028] FIG. 3 is a flowchart illustrating a method of operating a mobile robot according to one embodiment of the present disclosure.

[0029] FIG. 4 is a drawing schematically illustrating a method for a mobile robot to perform correction on an object detected during operation, according to one embodiment of the present disclosure.

[0030] FIG. 5 is a flowchart illustrating a method for a mobile robot to perform a specified operation depending on the type of an object during operation, according to one embodiment of the present disclosure.

[0031] FIG. 6 is a flowchart illustrating a method for a mobile robot to perform a specified operation according to a type of object detected during operation, according to one embodiment of the present disclosure.

[0032] FIG. 7 illustrates an example of a user interface provided to perform correction on an object detected during the operation of a mobile robot according to one embodiment of the present disclosure.

[0033] FIG. 8 is a flowchart illustrating a method for a mobile robot to perform correction on an object detected during operation, according to one embodiment of the present disclosure.

[0034] FIG. 9 is a flowchart illustrating a method for a mobile robot to perform correction on an object detected during operation, according to one embodiment of the present disclosure.

[0035] FIGS. 10A and 10B are diagrams schematically illustrating a method for updating an AI model used in a mobile robot according to one embodiment of the present disclosure.

[0036] FIG. 11 is a drawing illustrating a screen provided to perform correction on an object detected after completion of a movement of a mobile robot according to one embodiment of the present disclosure.

[0037] FIG. 12 is a flowchart illustrating a method for performing correction on a detected object after completion of a motion of a mobile robot, according to one embodiment of the present disclosure.

[0038] FIG. 13 is a drawing illustrating a screen provided to perform correction on an object detected during an operation after the completion of an operation of a mobile robot according to one embodiment of the present disclosure.

[0039] FIG. 14 is a drawing illustrating a screen provided to perform correction on an object detected during operation after completion of an operation of a mobile robot according to one embodiment of the present disclosure.

[0040] FIG. 15A is a flowchart illustrating a method for performing correction on an object detected during an operation of a mobile robot after the completion of the operation, according to one embodiment of the present disclosure.

[0041] FIG. 15b is a flowchart illustrating a method for performing corrections on objects detected during operation through an individual list screen after the completion of an operation of a mobile robot, according to one embodiment of the present disclosure.

[0042] FIG. 15c is a flowchart illustrating a method for performing correction on an object detected during an operation through a clustering screen after the operation of a mobile robot is completed, according to one embodiment of the present disclosure.

[0043] FIG. 16 is a flowchart illustrating a method for performing clustering on unknown objects according to one embodiment of the present disclosure.

[0044] FIGS. 17A and 17B are diagrams illustrating a method for a mobile robot to interact with a user according to one embodiment of the present disclosure.

[0045] FIG. 18 is a diagram illustrating a method for a mobile robot to interact with a user to provide a reward to the user, according to one embodiment of the present disclosure.

[0046] FIG. 19 is a flowchart illustrating a method of operating a mobile robot according to one embodiment of the present disclosure.

[0047] FIG. 20 is a flowchart illustrating a method of operating a mobile robot according to one embodiment of the present disclosure.

[0048] FIG. 21 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0049] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0050] An embodiment of the present disclosure will be described below with reference to the attached drawings.

[0051] FIG. 1 illustrates a robot intelligence system according to one embodiment of the present disclosure.

[0052] Referring to FIG. 1, the robot intelligence system (1) may include a mobile robot (10), a user's electronic device (hereinafter, user device) (20), and / or a server (30).

[0053] According to one embodiment, the robot intelligence system (1) may be a system for making a mobile robot (10) intelligent. For example, the robot intelligence system (1) may be a system for making a mobile robot (10) intelligent by using user input (or user feedback) obtained through the mobile robot (10) or a user device (20).

[0054] According to one embodiment, the mobile robot (10) may be a robot of various forms capable of performing a movement function. For example, the mobile robot (10) may be a robot (e.g., the cleaning robot (100) of FIGS. 2A and 2B) capable of performing various computing functions, such as a movement function, an object organization function (or cleaning function), an object transportation function, a sensing function, a display function, a communication function, and / or an output function (e.g., a voice or audio output function). However, the present invention is not limited thereto, and various forms of robots may be implemented as the mobile robot (10).

[0055] According to one embodiment, mobile robots (10) may be classified into industrial, medical, household, military, and exploration robots, depending on the functions and operations they can perform. Industrial robots may be further subdivided into, for example, robots used in the product manufacturing process in factories, robots that perform customer service, order reception, and serving in stores or restaurants, etc.

[0056] According to one embodiment, the mobile robot (10) may be implemented as a serving robot or a cleaning robot capable of transporting objects to a user-desired location or target location in various locations, such as a home, restaurant, hotel, supermarket, hospital, or clothing store. However, these are merely examples, and robots may be classified in various ways depending on their field of application, purpose of use, and the functions and actions they can perform.

[0057] According to one embodiment, the mobile robot (10) can perform communication using any of various wired or wireless communication protocols, such as Ethernet, GSM (global system for mobile communications), EDGE (enhanced data GSM environment), CDMA (code division multiple access), TDMA (time division multiplexing access), LTE (long term evolution), LTE-A (LTE advance), NR (new radio), 6G, Wi-Fi, or Bluetooth. For example, the mobile robot (10) can perform communication with a user device (20) or a server (30) based on a wired or wireless communication protocol.

[0058] According to one embodiment, the user device (20) may be a device capable of performing various computing functions, such as a communication function, a display function, or an output function (e.g., a voice or audio output function). For example, the user device (20) may be a TV, a wearable device (e.g., earbuds, a hearing aid, or a head mounted display (HMD)), a mobile device (e.g., a smartphone or a mobile phone), a tablet, a personal computer (PC), a desktop computer, a notebook computer, a personal digital assistant (PDA), a laptop, a media player, an e-book reader, a digital broadcasting terminal, a navigation device, a kiosk, a digital camera, or a home appliance. The electronic device (102) is not limited to the above-described devices and may be another type of electronic device.

[0059] According to one embodiment, the user device (20) can perform various functions to support the intelligence of the mobile robot (10). For example, the user device (20) can provide a user interface for receiving user feedback for the intelligence of the mobile robot (10), and can transmit a feedback message including data related to the user feedback obtained through the user interface to a server (30) or the mobile robot (10). The data related to the user feedback thus transmitted can be used to intelligence the mobile robot (10).

[0060] According to one embodiment, the user device (20) can perform communication using any of various wired or wireless communication protocols, such as Ethernet, GSM, EDGE, CDMA, TDMA, LTE, LTE-A, NR, Wi-Fi, or Bluetooth. For example, the user device (20) can perform communication with the mobile robot (10) or the server (30) based on the wired or wireless communication protocol. For example, the user device (20) can perform direct communication with the mobile robot (10), or can perform communication with the mobile robot (10) through the server (30).

[0061] In one embodiment, the server (30) may be connected to the mobile robot (10) and / or the user device (20). For example, the server (108) may be a cloud server. In one example, the server (30) may receive a feedback message including data associated with user feedback from the mobile robot (10) and / or the user device (20). In one example, the server (30) may support the intelligence of the mobile robot (10) by updating an AI model used in the mobile robot (10) based on the data included in the feedback message.

[0062] FIG. 2A is a drawing for explaining a cleaning robot according to an embodiment of the present disclosure.

[0063] According to one embodiment, the cleaning robot (100) is an example of a mobile robot (10), and may refer to various types of devices capable of performing movement and cleaning functions. In the present disclosure, the mobile robot (10) and the cleaning robot (100) may be abbreviated as robots. The cleaning robot (100) of the present disclosure may also be referred to as a mobile robot.

[0064] Referring to FIG. 2A, the cleaning robot (100) may include a manipulator (110), at least one sensor (120), a main body (130), a driving unit (140), a display (150), and / or a lift (160). According to one embodiment, the manipulator (110), the sensor (120), the driving unit (140), the display (150), and / or the lift (160) may be included in the main body (130) of the cleaning robot (100).

[0065] According to one embodiment, the cleaning robot (100) can sense the surrounding environment of the robot (100) in real time based on sensing data of a sensor (120) (e.g., a LiDAR (light detection and ranging) sensor, a camera (e.g., a depth camera (120-1), an RGB camera (120-2), etc.), collect information, and autonomously perform operations. For example, the cleaning robot (100) can detect an object using the sensor (120) and obtain data (e.g., image data) of the detected object. For example, the cleaning robot (100) can move by avoiding the detected object using the sensor (120).

[0066] According to one embodiment, the organizing robot (100) may include a manipulator (110). For example, the manipulator (110) is one of the parts of the organizing robot (100), and may perform a function similar to a human upper limb (e.g., a human arm), and may grasp an object (or identify an object, adsorb an object, etc.) using a gripper provided at the distal end. The manipulator (110) may be configured with, for example, six axes, and may specifically grasp an object in a three-dimensional space.

[0067] According to one embodiment, the gripper (111) can grasp an object or move an object like a human hand. For example, the gripper (111) may be called a robot hand, an end effector, etc. provided at the end of a multi-joint robot, but for the convenience of explanation, it will be referred to as a gripper (111) hereinafter.

[0068] In one embodiment, the gripper (111) may include at least one jaw (112). For example, the jaw (112) may refer to a portion (e.g., a distal end) of the gripper (111) that comes into contact with an object when the gripper (111) grasps or moves the object.

[0069] In one embodiment, the cleaning robot (100) can perform an operation of opening and closing at least one jaw (112) to grasp an object or move the object to a target location. For example, the cleaning robot (100) can grasp an object using at least one jaw (112), move the object to a target location, and then release the object.

[0070] For convenience of explanation, the action of the cleaning robot (100) grasping an object is referred to as a pick action, and the action of the gripper (111) moving the grasped object and placing it at a target location is referred to as a place action.

[0071] In one embodiment, the drive unit (140) may include an actuator or a motor. For example, the drive unit (140) may include wheels, brakes, etc., and the cleaning robot (100) may move within a space by itself using the drive unit (140). The drive unit (140) may be, for example, an omni-directional drive unit composed of two wheels and two casters, but is not limited thereto.

[0072] According to one embodiment, the organizing robot (100) can control the gripper (111) included in the manipulator (110) at various locations to grasp an object and then transport it to a target location.

[0073] According to one embodiment, the cleaning robot (100) can execute a cleaning mode. The cleaning mode may be, for example, a mode in which the cleaning robot (100) performs a cleaning operation, an avoidance operation, and / or a correction operation depending on the type of object.

[0074] According to one embodiment, the organizing robot (100) may perform various operations depending on the type of object to perform an organizing operation to efficiently move the object to a target location, or may perform an avoidance operation to move by avoiding the object. For example, if the organizing robot (100) cannot grasp the object using the gripper (111) while performing the organizing operation, the object may be moved by performing a pushing operation that pushes the object instead of a pick-and-place operation, and considering the time required to grasp the object, the object may be moved by performing a pulling operation that sweeps the object instead of a pick-and-place operation.

[0075] FIG. 2b is a block diagram illustrating a cleaning robot according to an embodiment of the present disclosure.

[0076] Referring to FIG. 2b, the cleaning robot (100) may include a manipulator (110), at least one sensor (120), a main body (130), a driving unit (140), a display (150), a lift (160), one or more processors (170), a memory (180), and / or a communication unit (190).

[0077] According to one embodiment, the manipulator (110) includes a gripper (111), which may be implemented as an impactive gripper, an ingressive gripper, an astrictive gripper, a contiguitive gripper, or the like.

[0078] In one embodiment, an impactive gripper can physically grasp an object and then move the object. For example, the impactive gripper can grasp an object using at least one jaw and move the object grasped by the at least one jaw.

[0079] In one embodiment, an ingressive gripper can grip an object by physically penetrating the surface of the object, such as using pins or needles.

[0080] In one embodiment, an astrictive gripper can grasp an object using attractive forces such as vacuum suction, magnetic force, or electrical adhesion.

[0081] According to one embodiment, at least one sensor (120) may include at least one of a first sensor (120-1) or a second sensor (120-2).

[0082] For example, the first sensor (120-1) may include a depth camera including a ToF (time of flight) camera sensor, etc.

[0083] According to one embodiment, the first sensing data acquired through the first sensor (120-1) may include depth information. The depth information may include a depth value corresponding to each of a plurality of pixels.

[0084] For example, a ToF camera sensor can irradiate a signal (e.g., near-infrared, ultrasound, laser, etc.) and, when the irradiated signal is reflected by a subject (e.g., an object), receive the reflected signal. One or more processors (170) can measure the elapsed time from when the ToF camera sensor irradiates the signal until it receives the reflected signal, thereby measuring the distance (or depth) between the ToF camera sensor and the subject.

[0085] According to one embodiment, one or more processors (170) can identify (or detect) an object adjacent to the robot (100) based on first sensing data acquired through the first sensor (120-1) and identify the size of the object.

[0086] For example, one or more processors (170) can identify an object (e.g., an object located in front of the robot (100)) located within a field of view (FOV) of the first sensor (120-1) based on the first sensing data.

[0087] However, the present invention is not limited thereto, and the first sensor (120-1) may be implemented as various types of sensors capable of identifying objects. For example, the first sensor (120-1) may include a stereo vision camera including at least two cameras.

[0088] According to one embodiment, one or more processors (170) can identify the depth of an object and / or the size of an object based on first sensing data reflecting binocular parallax characteristics in which objects are captured differently through a stereo vision camera.

[0089] For example, a stereo vision camera can obtain a left-eye image and a right-eye image by capturing an object from different viewpoints using at least two cameras based on the principle that when a person's two eyes (left and right eyes) located about 6.5 cm apart look at an object, different images are formed in the left and right eyes. According to one embodiment, one or more processors (170) can obtain the depth and size of the object based on first sensing data including the left-eye image and the right-eye image.

[0090] However, it is not limited thereto, and the first sensor (120-1) may be implemented as a Lidar sensor, a radar sensor, an ultrasonic sensor, an infrared sensor, etc.

[0091] According to one embodiment, the second sensor (120-2) may include a camera including an RGB camera sensor.

[0092] For example, a camera can convert an image of an object into an electrical signal and generate image data based on the converted signal. For example, a camera can convert an image of a subject into an electrical image signal through a semiconductor optical element (charge-coupled device, CCD), amplify the converted image signal, convert it into a digital signal, and then process the signal.

[0093] For example, the second sensor (120-2) may include at least one of a general (or basic) camera, an RGB camera, and an ultra-wide-angle camera. However, the present invention is not limited thereto, and the second sensor (120-2) may also be implemented as a lidar sensor, a radar sensor, an ultrasonic sensor, an infrared sensor, or the like.

[0094] According to one embodiment, one or more processors (170) can identify (or detect) an object based on second sensing data acquired through a second sensor (120-2).

[0095] According to one embodiment, as illustrated in FIG. 2a, the first sensor (120-1) is provided in the main body (130) and may be positioned on the upper side of the main body (130) to properly identify an object located within the field of view (FOV) of the first sensor (120-1).

[0096] According to one embodiment, as illustrated in FIG. 2a, the second sensor (120-2) may be positioned on the gripper (111) to appropriately identify an object positioned adjacent to the gripper (111).

[0097] However, this is an example for convenience of explanation, and each of the first sensor (120-1) and the second sensor (120-2) may be positioned in various ways on the main body (130).

[0098] According to one embodiment, the driving unit (140) can control the movement of the cleaning robot (100). For example, the driving unit (140) can move the cleaning robot (100), stop the moving robot (100), and control the moving speed and / or moving direction of the cleaning robot (100).

[0099] According to one embodiment, the driving type of the cleaning robot (100) may be a wheel type or a walking type.

[0100] According to one embodiment, the wheel type refers to the way in which the cleaning robot (100) moves by rotating the wheels. If the cleaning robot (100) is a wheel type robot, the cleaning robot (100) may include one or more wheels. The driving unit (140) may include a device that generates power to rotate the wheels. For example, the driving unit (140) may be implemented as a gasoline engine, a diesel engine, an LPG (liquefied petroleum gas) engine, or an electric motor, depending on the fuel (or energy source) used.

[0101] According to one embodiment, the walking type refers to the way in which the cleaning robot (100) moves through the movement of its legs. If the cleaning robot (100) is a walking type (e.g., a bipedal walking robot, a tripedal walking robot, a quadrupedal walking robot, etc.), the cleaning robot (100) may include two or more legs that support the cleaning robot (100). The legs may include a plurality of links and joints connected to the links. The driving unit (140) may include a device that generates power to raise or lower the legs by rotating the links around the joints. For example, the driving unit (140) may be implemented with a motor and / or an actuator.

[0102] According to one embodiment, the driving unit (140) can control the movement of a part of the cleaning robot (100). The driving unit (140) can be coupled between a first part (e.g., a body) and a second part (e.g., a head, an arm, etc.) of the cleaning robot (100). The driving unit (140) can rotate the second part. For example, the driving unit (140) can be implemented with a motor and / or an actuator.

[0103] According to one embodiment, the display (150) may be implemented as various types of displays such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a quantum dot (QD) display panel, a quantum dot light-emitting diodes (QLED), a micro light-emitting diodes (μLED), a mini LED, etc. According to one embodiment, the display (150) may also be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc.

[0104] According to one embodiment, the display (150) can display status information of the cleaning robot (100) under the control of one or more processors (170).

[0105] For example, the display (150) can display symbols and icons representing actions performed by the cleaning robot (100), and can display the remaining battery capacity of the cleaning robot (100), error messages of the cleaning robot (100), the status of the cleaning robot (100) (e.g., smile, blank expression), animation effects (e.g., eye blinking, etc.), etc. However, the present invention is not limited thereto, and the display (150) can display various contents under the control of one or more processors (170). For example, when a user input (e.g., gesture, multi-touch, etc.) is received by touching the display (150) through an input device such as a stylus pen or one or more fingers, one or more processors (170) can control the cleaning robot (100) according to the user input.

[0106] According to one embodiment, the cleaning robot (100) includes a lift (160), and the lift (160) may include a plate on which an object can be positioned and a motor for lowering the plate so that it touches the ground or raising the plate so that it does not touch the ground under the control of one or more processors (170).

[0107] In one embodiment, one or more processors (170) control the overall operation of the cleaning robot (100). For example, one or more processors (170) may be connected to each component of the cleaning robot (100) to control the overall operation of the cleaning robot (100).

[0108] According to one embodiment, one or more processors (170) may perform operations of the cleaning robot (100) according to various embodiments by executing at least one instruction stored in the memory (180).

[0109] According to one embodiment, the one or more processors (170) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (170) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (170) may execute one or more programs or instructions stored in the memory (180). For example, the one or more processors (170) may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in the memory.

[0110] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0111] According to one embodiment, one or more processors (170) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (170) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0112] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0113] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) having at least one processor and other electronic components integrated therein, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0114] According to one embodiment, one or more processors (170) can control components of the cleaning robot (100). The one or more processors (170) can control each of the components of the cleaning robot (100) by providing signals to them (e.g., gripper (111), sensor (120), and driving unit (140), etc.).

[0115] According to one embodiment, one or more processors (170) can detect (or identify) an object based on sensing data received from a sensor (120) (e.g., a first sensor (120-1) and / or a second sensor (120-2)).

[0116] According to one embodiment, one or more processors (170) may perform a pushing operation to move the object to a target location. For example, if the size of the object is greater than or equal to the distance between the plurality of jaws (112) provided on the gripper (111), the one or more processors (170) may perform a pushing operation to move the object to a target location.

[0117] According to one embodiment, one or more processors (170) may perform a pick and place operation in which the gripper (111) picks up an object and then places it at a target location, thereby moving the object to the target location. For example, if the size of the object is smaller than the distance between the plurality of jaws (112) and larger than the length of each of the plurality of jaws (112), the one or more processors (170) may perform a pick and place operation in which the gripper (111) picks up an object and then places it at a target location, thereby moving the object to the target location.

[0118] In one embodiment, one or more processors (170) may perform a pulling operation to sweep the object to move the object to a target location. For example, if the size of the object is less than or equal to the length of each of the plurality of jaws, the one or more processors (170) may perform a pulling operation to sweep the object to move the object to the target location.

[0119] For example, one or more processors (170) can sweep an object using a gripper (111) and then push the object to the front of the main body (130) to move it. For example, one or more processors (170) can sweep an object onto a lift (160) touching the ground using a gripper (111) and then move the robot (100) to a target position using a driving unit (140). Subsequently, one or more processors (170) can make the lift (160) slope to lower the object from the lift (160), and can also use the gripper (111) to lower the object from the lift (160).

[0120] According to one embodiment, the memory (180) may store various information or data related to the operation of the cleaning robot (100). For example, the memory (180) may include one or more storage media storing at least one instruction. For example, the memory (180) may include instructions that, when individually or collectively executed by one or more processors (170), cause the cleaning robot (100) to perform at least one operation.

[0121] According to one embodiment, the communication unit (190) includes a communication circuit and can support wired or wireless communication between the cleaning robot (100) and an external electronic device (e.g., the user device (20) and / or the server (30) of FIG. 1). The wireless communication method may include, for example, the LTE method, the 5G NR method, the Wi-Fi method, Bluetooth, Bluetooth Low Energy, infrared communication (IrDA: Infrared Data Association), UWB (Ultra WideBand), and NFC (Near Field Communication).

[0122] According to one embodiment, the cleaning robot (100) may further include additional configurations (e.g., a lidar sensor used for driving the cleaning robot (100), a direction indicator lighting positioned on the rear of the main body (130) that indicates the moving direction of the cleaning robot (100)) in addition to the configuration shown, or some configurations may be omitted.

[0123] Hereinafter, various embodiments of the present disclosure will be described using the cleaning robot (100) of FIGS. 2A and 2B as an example of a mobile robot. However, the present disclosure is not limited thereto, and various types of mobile robots or robots may be implemented to perform the operations / methods of the various embodiments of the present disclosure.

[0124] FIG. 3 is a flowchart illustrating a method of operating a mobile robot according to one embodiment of the present disclosure.

[0125] Referring to FIG. 3, in operation 3010, a mobile robot (e.g., a mobile robot (10) of FIG. 1 or a cleaning robot (100) of FIG. 2a) can detect (or identify) at least one object.

[0126] According to one embodiment, a mobile robot can detect at least one object located within a surrounding area by sensing (or scanning) the surrounding area using at least one sensor (e.g., the first sensor (120-1) and / or the second sensor (120-2) of FIG. 2A). For example, while moving, the mobile robot can detect at least one object located within the area by scanning the area within the sensing range of at least one sensor.

[0127] In operation 3020, the mobile robot can classify (or identify) the type of at least one detected object. The type of the object may be, for example, one of a first type corresponding to an object to be cleaned up by the mobile robot (hereinafter, referred to as a cleanup object), a second type corresponding to an object to be avoided by the mobile robot (hereinafter, referred to as an avoidance object), or a third type corresponding to an unknown object. If the type of the object is a cleanup object (first type), the mobile robot may be configured to perform a cleanup operation on the object. If the type of the object is an avoidance object (second type), the mobile robot may be configured to perform an avoidance operation on the object. If the type of the object is an unknown object (third type), the mobile robot may be configured to perform a corrective operation on the object. The unknown object may be, but is not limited to, an untrained object, a trained but unrecognized object, a trained but incorrectly recognized object (e.g., a trained object that is a cleanup object but is classified as an avoidance object by the mobile robot, or a trained object that is an avoidance object but is classified as a cleanup object by the mobile robot), or a incorrectly trained object (e.g., an object that is to be cleaned but is learned as an avoidance object, or an object that is to be avoided but is learned as a cleanup object).

[0128] In one embodiment, the mobile robot can identify the type of the detected object by classifying it as a clean object, an avoidance object, or an unknown object using a learned AI model. For example, the mobile robot can input data of the detected object (e.g., image data or input data based on the image data) into the learned AI model and obtain information about the type of the object (hereinafter, “type information”) as output data of the AI ​​model. The mobile robot can identify the type of the object based on the obtained type information.

[0129] In one embodiment, the AI ​​model used to classify object types may be a neural network-based model. For example, models such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN) may be used as AI models, but are not limited thereto.

[0130] In one embodiment, the AI ​​model used to classify the type of object may be learned through supervised learning, unsupervised learning, or reinforcement learning.

[0131] In one embodiment, the AI ​​model used to classify the type of object may be trained using a learning algorithm including error back-propagation or gradient descent.

[0132] In one embodiment, the type information may indicate the type of the object. For example, the type information may be set to one of a first value (e.g., 0) indicating that the detected object is a cleaning object, a second value (e.g., 1) indicating that the detected object is an avoidance object, or a third value (e.g., 2) indicating that the detected object is an unknown object. The unknown object may be, but is not limited to, an untrained object, a trained but unrecognized object, a trained but incorrectly recognized object (e.g., trained as a cleaning object but classified as an avoidance object by the mobile robot, or trained as an avoidance object but classified as a cleaning object by the mobile robot), or an incorrectly trained object (e.g., an object to be cleaned but learned as an avoidance object, or an object to be avoided but learned as a cleaning object).

[0133] According to one embodiment, the mobile robot can obtain information on the accuracy of type classification of a detected object (hereinafter, classification accuracy information) together with type information.

[0134] In one embodiment, the classification accuracy information may be set to a value (e.g., a percentage (%) value) indicating the accuracy of the type classification of the detected object. For example, if the detected object is classified as a clean object and the accuracy of the type classification is 90%, the type information may be set to a first value (e.g., 0) indicating that the detected object is a clean object, and the classification accuracy information may be set to 90.

[0135] In operation 3030, the mobile robot may perform a designated operation on at least one detected object based on the type of the at least one detected object. The designated operation may include, for example, at least one of an operation for tidying up a detected object (e.g., a tidying object) (hereinafter, a tidying operation), an operation for avoiding a detected object (e.g., an avoidance object) (hereinafter, an avoidance operation), or an operation for correcting a detected object (e.g., an unknown object) (hereinafter, a correction operation). In the present disclosure, the tidying operation may be referred to as a first operation, the avoidance operation may be referred to as a second operation, and the correction operation may be referred to as a third operation.

[0136] In one embodiment, if the detected object is a cleaning object, the mobile robot may perform a cleaning operation on the detected object. For example, the mobile robot may perform at least one of the pick-and-place operation, the pushing operation, or the pulling operation described above to clean the cleaning object (e.g., to move it to a target location).

[0137] In one embodiment, if the detected object is an evasion object, the mobile robot may perform an evasion action with respect to the detected object. For example, the mobile robot may perform an action to move while avoiding the evasion object.

[0138] According to one embodiment, if the detected object is an unknown object, the mobile robot can perform a correction operation on the detected object. Alternatively, if the detected object is classified as a cleanup object or an avoidance object, but the type classification accuracy is below a reference accuracy (e.g., 70, which indicates 70% accuracy), the mobile robot can perform a correction operation. In the present disclosure, an object that is the target of a correction operation (e.g., an object classified as an unknown object, or an object classified as a cleanup object or an avoidance object, but the type classification accuracy is below a reference accuracy) may be referred to as a correction target object. If the detected object is identified as a correction target object, the mobile robot can identify that user feedback (or user input) for correction associated with the type of the object is required.

[0139] According to one embodiment, the correction operation may include at least one operation for correcting a correction target object into a cleaning object or an avoidance object. For example, the correction operation may include an operation for providing (e.g., displaying) a user interface (hereinafter, a correction user interface) for obtaining user feedback (or user input) for correcting a correction target object to a mobile robot or a user device (e.g., the user device (20) of FIG. 1) connected to the mobile robot, an operation for obtaining user feedback (or user input) through the provided user interface, and / or an operation for correcting a correction target object into a cleaning object or an avoidance object based on the user feedback (or user input).

[0140] According to one embodiment, the correction user interface may include an image of a target object to be corrected, a first selectable item (or first object) for correcting (or selecting or classifying) the target object to be corrected as a clean object, and / or a second selectable item (or second object) for correcting (or selecting or classifying) the target object to be corrected as an avoidance object. An example of a correction user interface is described below with reference to FIG. 4, FIG. 7, FIG. 11, FIG. 13, or FIG. 14.

[0141] FIG. 4 is a drawing schematically illustrating a method for a mobile robot to perform correction on an object detected during operation, according to one embodiment of the present disclosure.

[0142] Referring to FIG. 4, a mobile robot (e.g., a mobile robot (10) of FIG. 1 or a cleaning robot (100) of FIGS. 2A and 2B) can perform an operation (e.g., operation 3010 of FIG. 3) of detecting (or identifying) an object. The mobile robot can perform an operation (e.g., operation 3020 of FIG. 3) of classifying the type of the detected object as an avoidance object, a cleaning object, or an unknown object. Through the operation of classifying the type of the detected object, the mobile robot can obtain classification result data including type information and / or classification accuracy information of the detected object. The classification result data can be transmitted from the mobile robot to a user device (e.g., the user device (20) of FIG. 1) connected to the mobile robot.

[0143] According to one embodiment, the mobile robot may display a first screen (4100) on the display of the mobile robot, which provides information on the progress of the operation (e.g., the progress of the operation according to the organizing mode of the organizing robot (100) of FIGS. 2A and 2B) based on the classification result data. The organizing mode may be, for example, a mode in which the organizing robot (100) performs an organizing operation, an avoidance operation, and / or a corrective operation depending on the type of object.

[0144] According to one embodiment, a first screen (4100) displayed on a display of a mobile robot may include a first user interface (4110) for providing information about at least one detected object, and / or a second user interface (4120) for obtaining user feedback for correcting at least one detected object to be corrected (e.g., an object classified as an unknown object, or an object classified as a cleanup object or an avoidance object but whose type classification accuracy is lower than or equal to a reference accuracy value). The second user interface (4120) may be, for example, the correction user interface described above with reference to FIG. 3. Examples of the first user interface (4110) and the second user interface (4120) of the first screen (4100) are described below with reference to FIG. 7.

[0145] According to one embodiment, the mobile robot may transmit a message (or information, or command) to the user device to display a second screen (4200) on a display of the user device connected to the mobile robot, the second screen (4200) providing information about the progress of the operation (e.g., the progress of the operation according to the organizing mode of the organizing robot (100) of FIGS. 2A and 2B). The message may include, for example, classification result data. The user device may display the second screen (4200) on the display of the user device based on the received classification result data.

[0146] According to one embodiment, the second screen (4200) displayed on the display of the user device may include a third user interface (4230) for providing information about at least one detected object. An example of the third user interface (4230) of the second screen (4200) is described below with reference to FIG. 7.

[0147] In one embodiment, the mobile robot can obtain user feedback (or user input) for performing correction on a target object to be corrected through a user interface (e.g., the second user interface (4120) or the third user interface (4230)). The mobile robot can perform correction on the target object to be corrected based on the obtained user feedback. By correcting the target object to be corrected using such user feedback, the rate at which detected objects are classified as unknown objects is gradually reduced. In this way, the mobile robot can be made intelligent or advanced through user feedback (or interaction with the user).

[0148] FIG. 5 is a flowchart illustrating a method for a mobile robot to perform a specified operation depending on the type of an object during operation, according to one embodiment of the present disclosure.

[0149] In the embodiment of FIG. 5, a mobile robot (e.g., a mobile robot (10) of FIG. 1 or a cleaning robot (100) of FIGS. 2a and 2b) can classify a detected object through a single judgment process.

[0150] Referring to FIG. 5, at operation 5010, the mobile robot may initiate an operation (e.g., a cleanup mode of the cleanup robot (100) of FIGS. 2A and 2B). In one embodiment, the mobile robot may initiate the operation in response to identifying an input (e.g., a user input) for initiating the operation.

[0151] In operation 5020, the mobile robot may scan (or sense) an area (e.g., an area to be cleaned by the cleaning robot (100) of FIGS. 2A and 2B) using at least one sensor (e.g., the first sensor (120-1) and / or the second sensor (120-2) of FIGS. 2A and 2B). In one embodiment, the mobile robot may scan an area within a sensing range of at least one sensor while moving.

[0152] In operation 5030, the mobile robot can identify whether an object exists within the scanned area. For example, the mobile robot can identify whether an object exists within the scanned area based on sensing data.

[0153] In operation 5040, if an object is identified as existing within the scanned area, the mobile robot may detect (or identify) the object. The operation of detecting the object in operation 5040 may include, for example, operation 3010 of FIG. 3. Duplicate descriptions thereof will be omitted.

[0154] In operation 5050, the mobile robot can classify the detected object. For example, the mobile robot can use the learned AI model to classify the type of the detected object as an avoidance object, a cleanup object, or an unknown object (or an object to be corrected). Through this, the mobile robot can identify the type of the detected object. The operation of classifying the detected object in operation 5050 may include, for example, operation 3020 of FIG. 3. Duplicate descriptions thereof will be omitted.

[0155] In operation 5060, if the detected object is identified as a cleanup object, the mobile robot may perform a cleanup operation on the detected object. For a description of the cleanup operation, see, for example, the description given above in operation 3030 of FIG. 3 . Therefore, any duplicate description will be omitted.

[0156] In operation 5070, if the detected object is identified as an evasion object, the mobile robot may perform an evasion action against the detected object. For a description of the evasion action, see, for example, the description given above in operation 3030 of FIG. 3 . Therefore, any duplicate description will be omitted.

[0157] In operation 5080, if the detected object is identified as an unknown object (or an object to be corrected), the mobile robot may perform a correction operation on the detected object. For a description of the correction operation, see, for example, the description given above in operation 3030 of FIG. 3. Therefore, any duplicate description will be omitted.

[0158] After performing operation 5060, operation 5070, or operation 5080, the mobile robot may perform operation 5020 to operation 5080 again for an area different from the previously scanned area. Through this repeated process, the mobile robot may perform the cleaning operation for the entire cleaning target area.

[0159] In operation 5090, if it is determined that no object exists within the scanned area, the mobile robot may terminate the operation (e.g., the cleanup mode of the cleanup robot (100) of FIGS. 2A and 2B).

[0160] FIG. 6 is a flowchart illustrating a method for a mobile robot to perform a specified operation according to a type of object detected during operation, according to one embodiment of the present disclosure.

[0161] Unlike the embodiment of FIG. 5, in the embodiment of FIG. 6, a mobile robot (e.g., the mobile robot (10) of FIG. 1 or the organizing robot (100) of FIGS. 2A and 2B) can classify a detected object through a plurality of step-by-step judgment processes.

[0162] Referring to FIG. 6, at operation 6010, a mobile robot (e.g., the mobile robot (10) of FIG. 1 or the cleaning robot (100) of FIGS. 2A and 2B) may initiate an operation (e.g., a cleaning mode of the cleaning robot (100) of FIGS. 2A and 2B). In one embodiment, the mobile robot may initiate the operation in response to identifying an input (e.g., a user input) for initiating the operation.

[0163] In operation 6020, the mobile robot may scan (or sense) an area (e.g., an area to be cleaned by the cleaning robot (100) of FIGS. 2A and 2B) using at least one sensor (e.g., the first sensor (120-1) and / or the second sensor (120-2) of FIGS. 2A and 2B). In one embodiment, the mobile robot may scan an area within a sensing range of at least one sensor while moving.

[0164] In operation 6030, the mobile robot can identify whether an object exists within the scanned area. For example, the mobile robot can identify whether an object exists within the scanned area based on sensing data.

[0165] In operation 6040, if an object is identified as existing within the scanned area, the mobile robot may detect (or identify) the object. The operation of detecting the object in operation 6040 may include, for example, operation 3010 of FIG. 3. Duplicate descriptions thereof will be omitted.

[0166] In operation 6050, the mobile robot can identify whether the detected object corresponds to a cleaning object. According to one embodiment, the mobile robot can identify whether the detected object corresponds to a cleaning object based on data (e.g., image data) of the detected object using a first classifier (or a first AI model). For example, if the distribution of the image data of the object detected by the first classifier is identified as similar to the distribution of the image data of the cleaning object, the mobile robot can identify the detected object as a cleaning object. For example, if the distribution of the image data of the object detected by the first classifier is identified as not similar to the distribution of the image data of the cleaning object, the mobile robot can identify the detected object as not corresponding to a cleaning object.

[0167] In operation 6060, if the detected object is identified as a cleaning object, the mobile robot may perform a cleaning operation on the detected object. For a description of the cleaning operation, see, for example, the description given above in operation 3030 of FIG. 3 . Therefore, any duplicate description will be omitted.

[0168] In operation 6070, if the detected object is identified as not being a clearing object, the mobile robot can identify whether the detected object is an avoidance object. According to one embodiment, the mobile robot can identify whether the detected object is an avoidance object based on data (e.g., image data) of the detected object using a second classifier (or a second AI model). For example, if the distribution of the image data of the object detected by the second classifier is identified as being similar to the distribution of the image data of the avoidance object, the mobile robot can identify the detected object as being an avoidance object. For example, if the distribution of the image data of the object detected by the second classifier is identified as not being similar to the distribution of the image data of the avoidance object, the mobile robot can identify the detected object as not being an avoidance object. Meanwhile, operation 6070 may be performed before operation 6050. In other words, the determination as to whether the detected object is an avoidance object may be performed before the determination as to whether the detected object is a clearing object. In this case, action 6060 associated with action 6050 may also be performed after action 6080 associated with action 6070.

[0169] In operation 6080, if the detected object is identified as an evasion object, the mobile robot may perform an evasion action against the detected object. For a description of the evasion action, see, for example, the description given above in operation 3030 of FIG. 3. Therefore, any duplicate description will be omitted.

[0170] In operation 6090, if the detected object is determined not to be an avoidance object, the mobile robot may perform a correction operation on the detected object. In this case, since the detected object is neither a cleanup object nor an avoidance object, it may be classified as an unknown object (or an object to be corrected), and thus, the mobile robot may perform a correction operation on the object. For a description of the correction operation, reference may be made to the description given above in operation 3030 of FIG. 3 . Therefore, a duplicate description will be omitted.

[0171] After performing operation 6060, operation 6080, or operation 6090, the mobile robot may perform operation 6020 to operation 6090 again for an area different from the previously scanned area. Through this repeated process, the mobile robot may perform the cleaning operation for the entire cleaning target area.

[0172] In operation 6100, if it is determined that no object exists within the scanned area, the mobile robot may terminate the operation (e.g., the cleanup mode of the cleanup robot (100) of FIGS. 2A and 2B).

[0173] Meanwhile, there may be errors in the judgment process for classifying the object type in the embodiments of FIGS. 5 and 6. For example, if the AI ​​model used to classify the object type is not trained based on sufficient training data, the accuracy of the type classification using the AI ​​model may be low, resulting in classification errors. Such classification errors may cause motion errors in the mobile robot. For example, if an object actually requires correction but is classified as an avoidance object instead of a correction object, motion errors such as performing an avoidance action against the object may occur. Therefore, a method for reducing such motion errors is needed.

[0174] For example, a method could be considered to reduce the occurrence of type classification errors by making mobile robots more intelligent or advanced through updating AI models using user feedback on object type classification results. Since updating AI models using user feedback is based on data collected by users in real-world environments, it can reduce the training costs manufacturers would otherwise incur to improve AI model performance, for example, in environments with high entropy. This can provide optimal learning effects tailored to real-world environments. Furthermore, mobile robots customized by users can be provided.

[0175] FIG. 7 illustrates an example of a user interface provided to perform correction on an object detected during the operation of a mobile robot according to one embodiment of the present disclosure.

[0176] Referring to FIG. 7, a user interface for correcting the type of an object detected during the operation of a mobile robot (e.g., operation according to the cleaning mode of the cleaning robot (100) of FIGS. 2A and 2B) may be displayed through a display of the mobile robot (e.g., the mobile robot (10) of FIG. 1 or the cleaning robot (100) of FIGS. 2A and 2B) or a display of a user device (e.g., the user device (20) of FIG. 1).

[0177] According to one embodiment, a first screen (4100) displayed on a display of a mobile robot may include a first user interface (4110) for providing information about at least one detected object, and / or a second user interface (4120) for obtaining user feedback for correcting at least one detected object to be corrected (e.g., an object classified as an unknown object, or an object classified as a cleanup object or an avoidance object but whose type classification accuracy is below a reference accuracy). The second user interface (4120) may be, for example, the correction user interface described above with reference to FIG. 3.

[0178] According to one embodiment, the first user interface (4110) may include image and type display information for at least one detected object sorted according to the order in which they were detected or the order in which a specified action was performed. For example, as illustrated in FIG. 7, in the first user interface (4110), the image (4111a) and type display information (4111b) of the first object (e.g., an object classified as a cleanup object) that is detected first may be displayed on the far left, the image (4112a) and type display information (4112b) of the second object (e.g., an object classified as an avoidance object) that is detected next to the first object may be displayed on the right side of the image (4111a) and type display information (4111b) of the first object, and the image (4113a) and type display information (4113b) of the third object (e.g., an object classified as an unknown object) that is detected next to the second object may be displayed on the right side of the image (4112a) and type display information (4112b) of the second object. However, the arrangement of image and type display information for each detected object shown in the first user interface (4110) of FIG. 7 is merely an example and is not limited thereto.

[0179] According to one embodiment, the second user interface (4120) may include an image of a target object to be corrected, a selectable item (hereinafter, referred to as a cleanup item) for correcting (or selecting, or classifying) the target object to be corrected as a cleanup object, and / or a second selectable item (hereinafter, referred to as an avoidance item) for correcting (or selecting, or classifying) the object as an avoidance object. For example, as illustrated in FIG. 7, the second user interface (4120) may include an image (4123) of a third object (e.g., an object classified as an unknown object) corresponding to a target object to be corrected among at least one detected object displayed in the first user interface (4110) (e.g., an image enlarged from the image (4113a) of the third object displayed in the first user interface (4110), an avoidance item (4121) for correcting the third object as an avoidance object, and / or a cleanup item (4122) for correcting the third object as a cleanup object.

[0180] In one embodiment, the mobile robot can receive user input (or user feedback) for selecting an item to be organized or an item to be avoided through the second user interface (4120). Based on the received user input, the mobile robot can correct an object to be corrected. For example, as illustrated in FIG. 7, the mobile robot can receive user input (7010) (e.g., touch input) for selecting an item to be avoided (4121) within the second user interface (4120), and based on the received user input (7010), correct a third object, which is an unknown object, to an avoidance object.

[0181] According to one embodiment, the second screen (4200) displayed on the display of the user device may include a third user interface (4230) for providing information about at least one detected object.

[0182] According to one embodiment, the third user interface (4230) may include image and type display information for at least one detected object sorted according to the order in which they were detected or the order in which a specified action was performed. For example, as illustrated in FIG. 7, in the third user interface (4230), the image (4231a) and type display information (4231b) of the first object detected first (e.g., an object classified as a cleanup object) may be displayed at the top, the image (4232a) and type display information (4232b) of the second object detected next to the first object (e.g., an object classified as an avoidance object) may be displayed below the image (4231a) and type display information (4231b) of the first object, and the image (4233a) and type display information (4233b) of the third object detected next to the second object (e.g., an object classified as an unknown object) may be displayed below the image (4232a) and type display information (4232b) of the second object. However, the arrangement of image and type display information for each detected object shown in the third user interface (4230) of FIG. 7 is only an example and is not limited thereto.

[0183] According to one embodiment, the mobile robot can receive a user input for selecting type indication information of a detected object through the third user interface (4230). Based on the received user input, the mobile robot can display on the screen a user interface for obtaining user feedback (or user input) for correcting a target object to be corrected. For example, as illustrated in FIG. 7, the mobile robot can receive a user input (7020) (e.g., a touch input) for selecting type information (4233b) of a third object (e.g., an object classified as an unknown object) within the third user interface (4230), and based on the received user input (7020), display on the screen a user interface for obtaining user feedback for correcting the third object. The user interface for obtaining user feedback for correcting the target object to be corrected may include, for example, the second user interface (4120).

[0184] In one embodiment, the mobile robot can obtain user feedback (or user input) for correcting a target object for correction through a user interface (e.g., the second user interface (4120) or the third user interface (4230)). The mobile robot can correct the target object for correction based on the obtained user feedback. By correcting the target object for correction using such user feedback, the rate at which detected objects are classified as unknown objects gradually decreases. In this way, the mobile robot can be made more intelligent or advanced through user feedback (or interaction with the user).

[0185] FIG. 8 is a flowchart illustrating a method for a mobile robot to perform correction on an object detected during operation, according to one embodiment of the present disclosure.

[0186] Referring to FIG. 8, at operation 8010, a mobile robot (e.g., a mobile robot (10) of FIG. 1 or a cleaning robot (100) of FIGS. 2A and 2B) may initiate an operation (e.g., a cleaning mode of the cleaning robot (100) of FIGS. 2A and 2B). According to one embodiment, the mobile robot may initiate the operation in response to identifying an input (e.g., a user input) for initiating the operation. Operation 8010 may include, for example, operation 5010 of FIG. 5. Duplicate descriptions thereof will be omitted.

[0187] In operation 8020, the mobile robot may scan (or sense) an area (e.g., an area to be cleaned by the cleaning robot (100) of FIGS. 2A and 2B) using at least one sensor. According to one embodiment, the mobile robot may scan an area within the sensing range of at least one sensor while moving. Operation 8020 may include, for example, operation 5020 of FIG. 5. Duplicate descriptions thereof will be omitted.

[0188] In operation 8030, the mobile robot can identify whether an object exists within the scanned area. Operation 8030 may include, for example, operation 5030 of FIG. 5 . Duplicate descriptions thereof will be omitted.

[0189] In operation 8040, if an object is identified as existing within the scanned area, the mobile robot may detect (or identify) the object. Operation 8040 may include, for example, operation 5040 of FIG. 5 . Duplicate descriptions thereof will be omitted.

[0190] In operation 8050, the mobile robot can classify the detected object. For example, the mobile robot can use a learned AI model to classify the type of the detected object as an avoidance object, a cleanup object, or an unknown object (or an object to be corrected). Operation 8050 may include, for example, operation 5050 of FIG. 5 . Duplicate descriptions thereof will be omitted.

[0191] In operation 8060, if the detected object is identified as being classified as a cleaning object, the mobile robot may perform a cleaning operation on the detected object. Operation 8060 may include, for example, operation 5060 of FIG. 5 . Duplicate descriptions thereof will be omitted.

[0192] In operation 8070, if the detected object is identified as an evasion object, the mobile robot may perform an evasion action against the detected object. Operation 8070 may include, for example, operation 5070 of FIG. 5 . A duplicate description thereof will be omitted.

[0193] In operation 8080, if the detected object is identified as an unknown object (or an object to be corrected), the mobile robot may perform a correction operation on the detected object. Operation 8080 may include, for example, operation 5080 of FIG. 5 . A duplicate description thereof will be omitted.

[0194] In operation 8090, if it is determined that no object exists within the scanned area, the mobile robot may terminate the operation. Operation 8090 may include, for example, operation 5090 of FIG. 5 . Duplicate descriptions thereof will be omitted.

[0195] After performing operation 8060, operation 8070, or operation 8080, at operation 8100, the mobile robot can identify whether user feedback for correction of the detected object is required.

[0196] In one embodiment, the mobile robot can identify that user feedback is needed for the detected object if the detected object is identified as an unknown object based on type information (or type information within the classification result data).

[0197] According to one embodiment, the mobile robot can identify that user feedback is required for the detected object when the accuracy of the type classification of the detected object is identified as being below a reference accuracy based on the classification accuracy information (or the classification accuracy information in the classification result data).

[0198] According to one embodiment, the mobile robot can identify that user feedback for a detected object is needed when a user input for selecting one of the at least one detected object displayed on a user interface (e.g., the first user interface (4110) or the third user interface (4230) of FIG. 4 or 7) for providing information about the at least one detected object is identified. The user input for selecting one of the at least one detected object may be, for example, a user input for selecting an image (e.g., the image (4113a) of FIG. 7) or type indication information (e.g., the type information (4113b) of FIG. 7) of the corresponding object displayed on the user interface (e.g., the first user interface (4110) or the third user interface (4230) of FIG. 4 or 7).

[0199] In one embodiment, if user feedback is identified as being required, the mobile robot may display a user interface (e.g., the second user interface (4120) of FIG. 4 or FIG. 7) on the display for obtaining user feedback for correction of the detected object. If user feedback is identified as not being required, the mobile robot may perform operations 8020 through 8080 again.

[0200] In operation 8110, if it is determined that user feedback for correction of the detected object is required, the mobile robot may obtain user feedback for correction of the detected object. In one embodiment, the mobile robot may obtain user feedback for correction of the detected object through a user interface displayed on a display (e.g., the second user interface (4120) of FIG. 4 or FIG. 7).

[0201] In operation 8120, the mobile robot can determine whether the corrected type matches the classified type. In one embodiment, the mobile robot can determine whether the type of the corrected object (corrected type) matches the type of the object classified through operation 8050 (classified type) based on user feedback.

[0202] In one embodiment, if the type of the corrected object (corrected type) and the type of the classified object (classified type) are identified as matching, the mobile robot may transmit a feedback message to the server including information about the detected object and information about the corrected type. The feedback message may be used by the server to update the learned AI model used to classify the type of the object. Thereafter, operations 8020 to 8080 may be performed again.

[0203] According to one embodiment, if the type of the corrected object (corrected type) and the type of the classified object (classified type) are identified as not matching, operation 8130 may be performed. In operation 8130, the mobile robot may perform an operation to reconfirm whether to correct the type of the detected object. For example, for correction reconfirmation, the mobile robot may display information for reconfirming (or inquiring about) the correction of the type of the detected object or a message including the information (e.g., a pop-up message) on the display.

[0204] In one embodiment, the mobile robot can correct the type of an object to the type of an object corrected based on user feedback, based on receiving user input reconfirming the correction. If no user input reconfirming the correction is received, or if user input indicating that correction is unnecessary is received, the mobile robot can maintain the type of the object as the type of the object classified through operation 8050. This correction reconfirmation operation can reduce the probability of the object type being incorrectly corrected due to user input errors, etc.

[0205] FIG. 9 is a flowchart illustrating a method for a mobile robot to perform correction on an object detected during operation, according to one embodiment of the present disclosure.

[0206] Referring to FIG. 9, at operation 9010, a mobile robot (e.g., the mobile robot (10) of FIG. 1 or the cleaning robot (100) of FIGS. 2A and 2B) may initiate an operation (e.g., a cleaning mode of the cleaning robot (100) of FIGS. 2A and 2B). In one embodiment, the mobile robot may initiate the operation in response to identifying an input (e.g., a user input) for initiating the operation. Operation 9010 may include, for example, operation 6010 of FIG. 6. Duplicate descriptions thereof will be omitted.

[0207] In operation 9020, the mobile robot may scan (or sense) an area (e.g., an area to be cleaned by the cleaning robot (100) of FIGS. 2A and 2B) using at least one sensor. According to one embodiment, the mobile robot may scan an area within the sensing range of at least one sensor while moving. Operation 9020 may include, for example, operation 6020 of FIG. 6. Duplicate descriptions thereof will be omitted.

[0208] In operation 9030, the mobile robot can identify whether an object exists within the scanned area. Operation 9030 may include, for example, operation 6030 of FIG. 6 . Duplicate descriptions thereof will be omitted.

[0209] In operation 9040, if an object is identified as existing within the scanned area, the mobile robot may detect (or identify) the object. Operation 9040 may include, for example, operation 6040 of FIG. 6. Duplicate descriptions thereof will be omitted.

[0210] In operation 9050, the mobile robot can identify whether the detected object corresponds to a cleaning object. Operation 9050 may include, for example, operation 6050 of FIG. 6 . Duplicate descriptions thereof will be omitted.

[0211] In operation 9060, if the detected object is identified as a cleaning object, the mobile robot may perform a cleaning operation on the detected object. Operation 9060 may include, for example, operation 6060 of FIG. 6 . Duplicate descriptions thereof will be omitted.

[0212] In operation 9070, if the detected object is identified as not being a cleanup object, the mobile robot can identify whether the detected object is an avoidance object. Operation 9070 may include, for example, operation 6070 of FIG. 6. Duplicate descriptions thereof will be omitted. Meanwhile, operation 9070 may be performed before operation 9050. In other words, the determination as to whether the detected object is an avoidance object may be performed before the determination as to whether the detected object is a cleanup object. In this case, operation 6060 associated with operation 6050 may also be performed after operation 6080 associated with operation 6070.

[0213] In operation 9080, if the detected object is identified as an evasion object, the mobile robot may perform an evasion action against the detected object. Operation 9080 may include, for example, operation 6080 of FIG. 6 . Duplicate descriptions thereof will be omitted.

[0214] In operation 9090, if the detected object is identified as not being an avoidance object, the mobile robot may perform a correction operation on the detected object. In this case, since the detected object is neither a cleanup object nor an avoidance object, it may be classified as an unknown object (or an object to be corrected), and thus, the mobile robot may perform a correction operation on the object. Operation 9090 may include, for example, operation 6090 of FIG. 6 . Duplicate descriptions thereof will be omitted.

[0215] In operation 9100, if it is determined that no object exists within the scanned area, the mobile robot may terminate the operation. Operation 9100 may include, for example, operation 6100 of FIG. 6. Duplicate descriptions thereof will be omitted.

[0216] After performing action 9060, action 9080, or action 9090, at action 9110, the mobile robot can identify whether user feedback is required for correction of the detected object.

[0217] In one embodiment, the mobile robot can identify that user feedback is needed for the detected object if the detected object is identified as an unknown object based on type information (or type information within the classification result data).

[0218] According to one embodiment, the mobile robot can identify that user feedback is required for the detected object when the accuracy of the type classification of the detected object is identified as being below a reference accuracy based on the classification accuracy information (or the classification accuracy information in the classification result data).

[0219] According to one embodiment, the mobile robot can identify that user feedback for a detected object is needed when a user input for selecting one of the at least one detected object displayed on a user interface (e.g., the first user interface (4110) or the third user interface (4230) of FIG. 4 or 7) for providing information about the at least one detected object is identified. The user input for selecting one of the at least one detected object may be, for example, a user input for selecting an image (e.g., the image (4113a) of FIG. 7) or type indication information (e.g., the type information (4113b) of FIG. 7) of the corresponding object displayed on the user interface (e.g., the first user interface (4110) or the third user interface (4230) of FIG. 4 or 7).

[0220] In one embodiment, if user feedback is identified as being required, the mobile robot may display a user interface (e.g., the second user interface (4120) of FIG. 4 or FIG. 7) on the display for obtaining user feedback for correction of the detected object. If user feedback is identified as not being required, the mobile robot may perform operations 9020 to 9100 again.

[0221] In operation 9120, if it is determined that user feedback for correction of the detected object is required, the mobile robot may obtain user feedback for correction of the detected object. In one embodiment, the mobile robot may obtain user feedback for correction of the detected object through a user interface displayed on a display (e.g., the second user interface (4120) of FIG. 4 or FIG. 7).

[0222] In operation 9130, the mobile robot can determine whether the corrected type matches the classified type. In one embodiment, the mobile robot can determine whether the type of the corrected object (corrected type) matches the type of the object classified through operation 9050 and / or operation 9070 based on user feedback (classified type).

[0223] In one embodiment, if the type of the corrected object (corrected type) and the type of the classified object (classified type) are identified as matching, the mobile robot may transmit a feedback message to the server including information about the detected object and information about the corrected type. The feedback message may be used by the server to update the learned AI model used to classify the type of the object. Thereafter, operations 9020 to 9100 may be performed again.

[0224] According to one embodiment, if the type of the corrected object (corrected type) and the type of the classified object (classified type) are identified as not matching, operation 9140 may be performed. In operation 9140, the mobile robot may perform an operation to reconfirm whether to correct the type of the detected object. For example, for correction reconfirmation, the mobile robot may display information for reconfirming (or inquiring about) the correction of the type of the detected object or a message including the information (e.g., a pop-up message) on the display.

[0225] In one embodiment, the mobile robot can correct the type of the object to the type of the object corrected based on user feedback, based on receiving user input reconfirming the correction. If no user input reconfirming the correction is received, or if user input indicating that correction is unnecessary is received, the mobile robot can maintain the type of the object as the type of the object classified through operation 9050 and / or operation 9070. Through such correction reconfirmation operations, the probability of the type of the object being incorrectly corrected due to user input errors, etc., can be reduced.

[0226] FIGS. 10A and 10B are diagrams schematically illustrating a method for updating an AI model used in a mobile robot according to one embodiment of the present disclosure.

[0227] Referring to FIG. 10A, a mobile robot (e.g., a mobile robot (10) of FIG. 1 or a cleaning robot (100) of FIGS. 2A and 2B) can perform an operation (e.g., operation 3010 of FIG. 3) of detecting (or identifying) an object. The mobile robot can perform an operation (e.g., operation 3020 of FIG. 3) of classifying the type of the detected object as an avoidance object, a cleaning object, or an unknown object using a learned AI model. Through the operation of classifying the type of the detected object, the mobile robot can obtain classification result data including type information and / or classification accuracy information of the detected object. The classification result data can be transmitted from the mobile robot to a user device (e.g., the user device (20) of FIG. 1) connected to the mobile robot.

[0228] According to one embodiment, the mobile robot may display a first screen (4100) on the display of the mobile robot, which provides information on the progress of the operation (e.g., the progress of the operation according to the organizing mode of the organizing robot (100) of FIGS. 2A and 2B) based on the classification result data.

[0229] According to one embodiment, the mobile robot may transmit a message (or information, or command) to the user device to display a second screen (4200) on a display of the user device connected to the mobile robot, which provides information about the progress of the operation (e.g., the progress of the operation according to the organizing mode of the organizing robot (100) of FIGS. 2A and 2B). The message may include, for example, classification result data. The user device may display the second screen (4200) on the display of the user device based on the received classification result data.

[0230] According to one embodiment, the mobile robot and the user device can obtain user feedback (or user input) for correcting a target object to be corrected through a user interface included in the screen (e.g., the second user interface (4120) or the third user interface (4230) of FIG. 4 or FIG. 7). The mobile robot can correct the target object to be corrected based on the obtained user feedback. By correcting the target object to be corrected using such user feedback, the rate at which detected objects are classified as unknown objects gradually decreases. In this way, the mobile robot can be made intelligent or advanced through user feedback (or interaction with the user).

[0231] According to one embodiment, the mobile robot and the user device may transmit a user feedback message (1001a, 1001b) including data associated with the acquired user feedback (user feedback data) to a server (e.g., server (30) of FIG. 1). The data associated with the user feedback transmitted to the server may be used to update a learned AI model used to classify the type of an object in the mobile robot. According to one embodiment, the user feedback message (1001a, 1001b) may include information about the detected object (e.g., image data of the object) and / or information about the type of the object corrected by the user feedback (corrected type). In the present disclosure, the user feedback message may be referred to as a feedback message.

[0232] In one embodiment, the server can use feedback data to update the trained AI model. For example, the server can update the AI ​​model by retraining the AI ​​model using the feedback data as training data. The server can then transmit the updated AI model (1002) to the mobile robot.

[0233] In the embodiment of Fig. 10a, the update of the learned AI model is exemplified as being performed by the server, but this is not limited to this. The update of the learned AI model may also be performed by the mobile robot, or through collaboration between the mobile robot and the server. Alternatively, the update of the learned AI model may be performed by a separate electronic device.

[0234] Referring to FIG. 10b, a server (e.g., server (30) of FIG. 1) can collect user feedback data (e.g., user feedback messages (1001a, 1001b) of FIG. 10a) and operational data from at least one mobile robot (e.g., mobile robot (10) of FIG. 1 or cleaning robot (100) of FIGS. 2a and 2b) and at least one user device (e.g., user device (20) of FIG. 1).

[0235] According to one embodiment, the data acquisition unit (1010) may acquire (or store) a data set (hereinafter, referred to as the collected data set) (1010) including collected user feedback data and operational data. The clustering unit (1020) may perform similarity-based clustering on the data of the collected data set so that the data of the collected data set can be used for learning an AI model. The learning unit (1030) may train (e.g., continuous training) using at least a portion of the data on which similarity-based clustering has been performed, to train an AI model for object detection and / or object type classification. The evaluation unit (1040) may perform an automated evaluation on the learned AI model using the evaluation data. If the evaluation results indicate that the learned AI model is not suitable, it may be retrained. If the evaluation results of the evaluation unit (1040) indicate that the learned AI model is suitable, the model update unit (1050) may update the AI ​​model to a final AI model. The server may transmit the updated AI model to the mobile robot.

[0236] According to one embodiment, at least one of the data acquisition unit (1010), the clustering execution unit (1020), the learning unit (1030), the evaluation unit (1040), or the model update unit (1050) may be implemented in the form of a hardware chip. For example, at least one of the data acquisition unit (1010), the clustering execution unit (1020), the learning unit (1030), the evaluation unit (1040), or the model update unit (1050) may be implemented in the form of a dedicated hardware chip for AI or as part of a general-purpose processor (e.g., CPU) or a graphics-only processor (GPU).

[0237] According to one embodiment, at least one of the data acquisition unit (1010), the clustering execution unit (1020), the learning unit (1030), the evaluation unit (1040), or the model update unit (1050) may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer readable recording medium that can be read by a computer. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by the OS, and the remaining some may be provided by a predetermined application.

[0238] According to one embodiment, at least one of the data acquisition unit (1010), the clustering execution unit (1020), the learning unit (1030), the evaluation unit (1040), or the model update unit (1050) may be included in a mobile robot (e.g., the mobile robot (10) of FIG. 1, or the mobile robot of FIGS. 2A and 2B), a user device (e.g., the user device (20) of FIG. 1), a server (e.g., the server (30) of FIG. 1), or other devices. In this case, the update of the learned AI model may be performed solely by the server, but may also be performed solely by the mobile robot or other devices. For example, some of the data acquisition unit (1010), the clustering execution unit (1020), the learning unit (1030), the evaluation unit (1040), or the model update unit (1050) may be included in the server, and the remaining some may be included in the mobile robot. In this case, the mobile robot and the server may cooperate with each other to update the learned AI model. there is.

[0239] FIG. 11 is a drawing illustrating a screen provided to perform correction on an object detected after completion of a movement of a mobile robot according to one embodiment of the present disclosure.

[0240] In the embodiment of FIG. 11, for convenience of explanation, the screens (or user interfaces) are described as being displayed through a display of a user device (e.g., the user device (20) of FIG. 1), but the present invention is not limited thereto. For example, the screens may also be displayed through a display of a mobile robot (e.g., the mobile robot (10) of FIG. 1, or the cleaning robot (100) of FIGS. 2A and 2B).

[0241] Referring to FIG. 11, an electronic device (e.g., a user device or a mobile robot) may display a first screen (1110) on a display that provides information about the operation record (or history) of the mobile robot after the mobile robot completes its operation (e.g., after the cleaning mode of the cleaning robot (100) of FIGS. 2A and 2B) ends. In the present disclosure, the first screen (1110) may be referred to as an operation record screen.

[0242] According to one embodiment, the first screen (1110) may include a selectable item (hereinafter, “cleaning record item”) (1111) for providing information about a record (or history) (hereinafter, “cleaning record”) for an object on which a cleanup operation or an avoidance operation has been performed and / or a selectable item (hereinafter, “unknown record item”) (1112) for providing information about a record (or history) (hereinafter, “unknown record”) for an unknown object.

[0243] According to one embodiment, in response to receiving a user input (1101) selecting a cleanup record item (1111), the electronic device may display a second screen (1120) on the display that provides information about the cleanup record. In the present disclosure, the second screen (1120) may be referred to as a cleanup record screen.

[0244] According to one embodiment, the second screen (1120) may include, for each of at least one object for which a cleaning operation or an avoidance operation has been performed, an image (1121), type display information (1122) indicating the classified type of the object, and / or a selectable item (hereinafter, a correction item) (1123) for correcting the type of the object.

[0245] According to one embodiment, in response to receiving a user input (1102) (e.g., a touch input) for selecting a modification item (1123), the electronic device may display a third screen (1130) on the display for modifying the type of an object associated with the modification item (1123). In the present disclosure, the third screen (1130) may be referred to as a modification screen.

[0246] According to one embodiment, the third screen (1130) may include an image (1131) of an object associated with the selected modification item, a selectable item (hereinafter, “avoid item”) (1132) for correcting the object to an avoidance object, a selectable item (hereinafter, “cleanup item”) (1133) for correcting the object to a cleanup object, and / or a selectable item (hereinafter, “update item”) (1134) for updating the corrected type according to the selected item to the type of the object.

[0247] In one embodiment, the electronic device may correct (or update) the type of a detected object to an avoidance object in response to receiving user input (e.g., a touch input) sequentially selecting an avoidance item (1132) and an update item (1134).

[0248] According to one embodiment, the electronic device may correct (or update) the type of a detected object to a cleanup object in response to receiving user input (1103) (e.g., touch input) sequentially selecting a cleanup item (1133) and an update item (1134).

[0249] FIG. 12 is a flowchart illustrating a method for performing correction on a detected object after completion of a motion of a mobile robot, according to one embodiment of the present disclosure.

[0250] The method of the embodiment of FIG. 12 can be performed by an electronic device (e.g., a mobile robot (10) of FIG. 1, a cleaning robot (100) of FIGS. 2a and 2b, or a user device (20)).

[0251] Referring to FIG. 12, in operation 12010, an electronic device (e.g., a mobile robot or a user device) may display an operation recording screen (e.g., a first screen (1110) of FIG. 11) on a display after the operation of the mobile robot is completed (e.g., after the cleaning mode of the cleaning robot (100) of FIGS. 2a and 2b) is terminated.

[0252] In operation 12020, the electronic device may identify that a cleanup record item (e.g., a cleanup record item (1111) of FIG. 11) within the operation record screen is selected. In one embodiment, the electronic device may identify that a cleanup record item is selected based on a user input (e.g., a touch input).

[0253] In operation 12030, in response to identifying that a cleanup record item is selected, the electronic device may display a cleanup record screen (e.g., second screen (1120) of FIG. 11) on the display.

[0254] In operation 12040, the electronic device may identify that a modification item (e.g., modification item (1123) of FIG. 11) within the cleanup record screen is selected. In one embodiment, the electronic device may identify that a modification item is selected based on user input.

[0255] In operation 12050, in response to identifying that a modification item is selected, the electronic device may identify a modification screen (e.g., the third screen (1130) of FIG. 11) of an object associated with the modification item.

[0256] In operation 12060, the electronic device may obtain user feedback (or user input) for correcting the object through the correction screen. For example, the electronic device may receive user input sequentially selecting an avoidance item (1132) and an update item (1134). In this case, the electronic device may correct (or update) the type of the object to an avoidance object. For example, the electronic device may receive user input (1103) (e.g., a touch input) sequentially selecting a cleanup item (1133) and an update item (1134). In this case, the electronic device may correct (or update) the type of the object to a cleanup object.

[0257] In operation 12070, the electronic device can determine whether the corrected type matches the classified type. Operation 12070 may include, for example, operation 8120 of FIG. 8 and / or operation 9130 of FIG. 9 . Duplicate descriptions thereof will be omitted.

[0258] In operation 12080, in response to identifying that the type of the corrected object (corrected type) matches the type of the classified object (classified type), the electronic device may transmit (or upload) a feedback message to the server, the feedback message including data associated with the user feedback. The feedback message may be used by the server to update a learned AI model used to classify the type of the object. The data associated with the user feedback may include, for example, information about the detected object and information about the corrected type.

[0259] In operation 12090, in response to identifying that the type of the corrected object (corrected type) does not match the type of the classified object (classified type), the electronic device may perform an operation to re-verify whether to correct the type of the detected object. Operation 12090 may include, for example, operation 8130 of FIG. 8 and / or operation 9140 of FIG. 9 . Duplicate descriptions thereof will be omitted.

[0260] FIG. 13 is a drawing illustrating a screen provided to perform correction on an object detected during an operation after the completion of an operation of a mobile robot according to one embodiment of the present disclosure.

[0261] In the embodiment of FIG. 13, for convenience of explanation, the screens (or user interfaces) are described as being displayed through a display of a user device (e.g., the user device (20) of FIG. 1), but the present invention is not limited thereto. For example, the screens may also be displayed through a display of a mobile robot (e.g., the mobile robot (10) of FIG. 1, or the cleaning robot (100) of FIGS. 2A and 2B).

[0262] Referring to FIG. 13, an electronic device (e.g., a user device or a mobile robot) may display a first screen (1310) on a display that provides information about the operation record of the mobile robot after the mobile robot completes its operation (e.g., after the cleaning mode of the cleaning robot (100) of FIGS. 2A and 2B) ends. In the present disclosure, the first screen (1310) may be referred to as an operation record screen.

[0263] According to one embodiment, the first screen (1310) may include a selectable item (hereinafter, “cleaning record item”) (1311) for providing information about a record of an object on which a cleanup operation or an avoidance operation has been performed (hereinafter, “cleaning record”) and / or a selectable item (hereinafter, “unknown record item”) (1312) for providing information about a record of an unknown object (hereinafter, “unknown record”).

[0264] According to one embodiment, the electronic device may display a second screen (1320) on the display, which provides information about the unknown records in an individual list format, in response to receiving a user input (1301) (e.g., a touch input) for selecting an unknown record item (1312). In the present disclosure, the second screen (1320) may be referred to as an unknown record screen in an individual list format, a first unknown record screen, or an individual list screen.

[0265] According to one embodiment, the second screen (1320) may include an image of at least one unknown object (1321) and / or a selectable item (hereinafter, a correction item) (1322) for correcting the type of the object.

[0266] According to one embodiment, in response to receiving a user input (1302) (e.g., a touch input) for selecting a modification item (1322), the electronic device may display a third screen (1330) on the display for modifying the type of object associated with the modification item (1322). In the present disclosure, the third screen (1330) may be referred to as a modification screen.

[0267] According to one embodiment, the third screen (1330) may include an image (13331) of an object associated with a selected modification item, a selectable item (hereinafter, "avoid item") (1332) for correcting the object to an avoidance object, a selectable item (hereinafter, "cleanup item") (1333) for correcting the object to a cleanup object, and / or a selectable item (hereinafter, "update item") (1334) for updating the corrected type according to the selected item to the type of the object.

[0268] According to one embodiment, the user device may correct (or update) the type of a detected object from an unknown object to an avoidance object in response to receiving user input (1303) (e.g., a touch input) sequentially selecting an avoidance item (1332) and an update item (1333).

[0269] According to one embodiment, the user device can correct (or update) the type of a detected object from an unknown object to a clean object in response to receiving user input sequentially selecting a clean item (1333) and an update item (1334).

[0270] According to the embodiment of Fig. 13, the type of individual objects can be individually corrected.

[0271] FIG. 14 is a drawing illustrating a screen provided to perform correction on an object detected during operation after completion of an operation of a mobile robot according to one embodiment of the present disclosure.

[0272] In the embodiment of FIG. 14, for convenience of explanation, the screens (or user interfaces) are described as being displayed through a display of a user device (e.g., the user device (20) of FIG. 1), but the present invention is not limited thereto. For example, the screens may also be displayed through a display of a mobile robot (e.g., the mobile robot (10) of FIG. 1, or the cleaning robot (100) of FIGS. 2A and 2B).

[0273] Referring to FIG. 14, an electronic device (e.g., a user device or a mobile robot) may display a first screen (1410) on a display that provides information about the operation record of the mobile robot after the mobile robot completes its operation (e.g., after the cleaning mode of the cleaning robot (100) of FIGS. 2A and 2B) ends. In the present disclosure, the first screen (1410) may be referred to as an operation record screen.

[0274] According to one embodiment, the first screen (1410) may include a selectable item (hereinafter, “cleaning record item”) (1411) for providing information about a record of an object on which a cleanup operation or an avoidance operation has been performed (hereinafter, “cleaning record”) and / or a selectable item (hereinafter, “unknown record item”) (1412) for providing information about a record of an unknown object (hereinafter, “unknown record”).

[0275] According to one embodiment, the electronic device may display a second screen (1420) on the display, which provides information about an unknown record in a clustering manner, in response to receiving a user input (1401) (e.g., a touch input) selecting an unknown record item (1412). In the present disclosure, the second screen (1420) may be referred to as a clustering-style unknown record screen, a second unknown record screen, or a clustering screen.

[0276] According to one embodiment, the second screen (1420) may include, for each of at least one cluster, selectable items (hereinafter, “modification items”) (1422) for correcting the type of the image (1421) and / or at least one object belonging to the cluster. The at least one cluster may include a first cluster including one or more objects clustered as avoidance objects, a second cluster including one or more objects clustered as cleaning objects, and / or a third cluster including one or more objects clustered as other objects. In the present disclosure, the first cluster may be referred to as an avoidance cluster, the second cluster may be referred to as a cleaning cluster, and the third cluster may be referred to as an other cluster. A description of the clustering method is described below with reference to FIG. 16.

[0277] According to one embodiment, in response to receiving a user input (1402) (e.g., a touch input) for selecting a modification item (1422), the electronic device may display a third screen (1430) on the display for modifying the type of object associated with the modification item (1422). In the present disclosure, the third screen (1430) may be referred to as a modification screen.

[0278] According to one embodiment, the third screen (1430) may include an image (1431) for each of the objects included in a cluster (e.g., an avoidance cluster) associated with the selected modification item (1422), at least one selection item (hereinafter, object selection item) (1432) for selecting at least one object among the object(s) included in the cluster, a selectable item (hereinafter, avoidance item) (1433) for correcting the at least one selected object to an avoidance object, a selectable item (hereinafter, cleanup item) (1434) for correcting the at least one selected object to a cleanup object, and / or a selection item (hereinafter, update item) (1435) for updating the corrected type according to the selected item to the type of the corresponding object.

[0279] According to one embodiment, in response to receiving a user input (1403) (e.g., a touch input) sequentially selecting at least one selection item (1432), an avoidance item (1433), and an update item (1435), the electronic device can correct (or update) the type of one or more objects selected through at least one selection item (1432) to an avoidance object.

[0280] According to one embodiment, in response to receiving user input sequentially selecting at least one selection item (1432), a cleanup item (1434), and an update item (1435), the electronic device can correct (or update) the type of one or more objects selected through at least one selection item (1432) to a cleanup object.

[0281] Unlike the embodiment of FIG. 13, according to the embodiment of FIG. 14, the type of at least one object belonging to a cluster can be corrected in bulk.

[0282] FIG. 15A is a flowchart illustrating a method for performing correction on an object detected during an operation of a mobile robot after the completion of the operation, according to one embodiment of the present disclosure.

[0283] The method of the embodiment of FIG. 15a can be performed by an electronic device (e.g., a mobile robot (10) of FIG. 1, a cleaning robot (100) of FIGS. 2a and 2b, or a user device (20)).

[0284] Referring to FIG. 15a, in operation 15010a, an electronic device (e.g., a mobile robot or a user device) may display an operation recording screen (e.g., a first screen (1310) of FIG. 13 or a first screen (1410) of FIG. 14) on a display after the operation of the mobile robot is completed (e.g., after the cleaning mode of the cleaning robot (100) of FIGS. 2a and 2b) is terminated.

[0285] In operation 15020a, the electronic device may identify whether an unknown record item (e.g., unknown record item (1312) of FIG. 13 or unknown record item (1412) of FIG. 14) within the operation record screen is selected. In one embodiment, the electronic device may identify whether an unknown record item is selected based on a user input.

[0286] In response to identifying that an unknown record item is selected in operation 15030a, the electronic device can identify whether the number of unknown objects is less than or equal to a specified number (N).

[0287] In operation 15040a, in response to identifying that the number of unknown objects is N or less, the electronic device may display an individual list screen (e.g., the second screen (1320) of FIG. 13) on the display.

[0288] In operation 15050a, in response to identifying that the number of unknown objects exceeds N, the electronic device may display a clustering screen (e.g., the second screen (1420) of FIG. 14) on the display.

[0289] In this way, by performing clustering only when the number of unknown objects is greater than a certain number, the power consumption resulting from performing clustering can be reduced.

[0290] FIG. 15b is a flowchart illustrating a method for performing corrections on objects detected during operation through an individual list screen after the completion of an operation of a mobile robot, according to one embodiment of the present disclosure.

[0291] The method of the embodiment of FIG. 15b can be performed by an electronic device (e.g., a mobile robot (10) of FIG. 1, a cleaning robot (100) of FIGS. 2a and 2b, or a user device (20)). The operations of FIG. 15b may be operations performed after operation 1540a of FIG. 15a.

[0292] Referring to FIG. 15B, at operation 1510b, an electronic device (e.g., a mobile robot or a user device) may identify that a modification item (e.g., a modification item (1322) of FIG. 13) within an individual list screen (e.g., a second screen (1320) of FIG. 13) is selected. In one embodiment, the electronic device may identify that a modification item is selected based on a user input.

[0293] In operation 1520b, in response to identifying that a modification item is selected, the electronic device may display on the display a modification screen (e.g., the third screen (1330) of FIG. 13) of an unknown object associated with the selected modification item.

[0294] In operation 1530b, the electronic device may obtain user feedback (or user input) for correcting the object through the correction screen. For example, the electronic device may correct (or update) the type of the detected object from an unknown object to an avoidance object in response to receiving user input (or user feedback) sequentially selecting an avoidance item (e.g., an avoidance item (1332) of FIG. 13) and an update item (e.g., an update item (1334) of FIG. 13). For example, the electronic device may correct (or update) the type of the detected object from an unknown object to a cleanup object in response to receiving user input sequentially selecting a cleanup item (e.g., a cleanup item (1333) of FIG. 13) and an update item (e.g., an update item (1334) of FIG. 13).

[0295] In operation 1540b, the electronic device may transmit (or upload) data associated with the user feedback to a server. The data associated with the user feedback may include, for example, information about the detected object and / or information about the corrected type.

[0296] FIG. 15c is a flowchart illustrating a method for performing correction on an object detected during an operation through a clustering screen after the operation of a mobile robot is completed, according to one embodiment of the present disclosure.

[0297] The method of the embodiment of FIG. 15c can be performed by an electronic device (e.g., a mobile robot (10) of FIG. 1, a cleaning robot (100) of FIGS. 2a and 2b, or a user device (20)).

[0298] According to one embodiment, the operations of FIG. 15c may be operations performed after operation 1550a of FIG. 15a.

[0299] Referring to FIG. 15c, at operation 1510c, an electronic device (e.g., a mobile robot or a user device) may identify that a modification item (e.g., a modification item (1422) of FIG. 14) within a clustering screen (e.g., a second screen (1420) of FIG. 14) is selected. In one embodiment, the electronic device may identify that a modification item is selected based on a user input.

[0300] In operation 1520c, in response to identifying that a modification item has been selected, the electronic device may display a modification screen (e.g., the third screen (1430) of FIG. 14) of a cluster (e.g., an avoidance cluster) associated with the selected modification item on the display. A description of the clustering method is described below with reference to FIG. 16.

[0301] In operation 1530c, the electronic device may obtain user feedback (or user input) for correcting at least one object within the cluster through a correction screen. For example, in response to receiving user input sequentially selecting at least one object selection item (e.g., selection item (1432) of FIG. 14), an avoidance item (e.g., avoidance item (1433) of FIG. 14), and an update item (e.g., update item (1435) of FIG. 14), the electronic device may correct (or update) the type of one or more objects selected through at least one selection item (1432) to an avoidance object.

[0302] According to one embodiment, in response to receiving user input sequentially selecting at least one object selection item (e.g., selection item (1432) of FIG. 14), a cleanup item (e.g., cleanup item (1434) of FIG. 14), and an update item (e.g., update item (1435) of FIG. 14), the user device may correct (or update) the type of one or more objects selected through at least one selection item to a cleanup object.

[0303] At step 1540c, the electronic device may transmit (or upload) data associated with the user feedback to a server. The data associated with the user feedback may include, for example, information about the detected object and / or information about the corrected type.

[0304] FIG. 16 is a flowchart illustrating a method for performing clustering on unknown objects according to one embodiment of the present disclosure.

[0305] The method of the embodiment of FIG. 16 can be performed by an electronic device (e.g., a mobile robot (10) of FIG. 1, a cleaning robot (100) of FIGS. 2a and 2b, or a user device (20)).

[0306] According to one embodiment, the operations of FIG. 16 may be performed when the number of unknown objects is identified as exceeding N through operation 1520a of FIG. 15a.

[0307] According to one embodiment, the operations of FIG. 16 may be performed prior to operation 1550a of FIG. 15a.

[0308] Referring to FIG. 16, at operation 16010, the electronic device may initiate clustering.

[0309] In operation 16020, the electronic device can collect a list of unknown objects.

[0310] In operation 16030, the electronic device may perform a similarity comparison between an unknown object in the list and existing objects (e.g., objects whose types are already known). According to one embodiment, the electronic device may perform the similarity comparison using a preset similarity comparison algorithm (e.g., a Euclidean distance algorithm or a Hamming distance algorithm).

[0311] At operation 16040, the electronic device can identify whether the unknown object is similar to the cleaned object.

[0312] In operation 16050, if the unknown object is identified as similar to a cleanup object, the electronic device may classify the unknown object as an object belonging to a cleanup cluster.

[0313] In operation 16060, if the unknown object is identified as not similar to the clearing object, the electronic device may identify whether the unknown object is similar to the avoiding object.

[0314] In operation 16070, if the unknown object is identified as similar to an avoidance object, the electronic device may classify the unknown object as an object belonging to an avoidance cluster.

[0315] In operation 16080, if the unknown object is identified as not similar to the evaded object, the electronic device may classify the unknown object as an object belonging to another cluster.

[0316] In operation 16090, the electronic device may transmit a notification containing information about objects belonging to other clusters to an external electronic device (e.g., a server or a developer's electronic device). Using the data contained in the notification, classification of objects belonging to other clusters may be performed. This may reduce the number of objects belonging to other clusters, thereby improving clustering accuracy.

[0317] At operation 16100, the electronic device can determine whether clustering has been performed on all unknown objects in the list. If it is determined that clustering has not been performed on all unknown objects in the list, the electronic device can perform operations 16030 to 16080 on the next unknown object in the list. In this manner, clustering can be performed on all unknown objects in the list.

[0318] In operation 16110, if it is identified that clustering has been performed on all unknown objects in the list, the electronic device may terminate clustering.

[0319] FIGS. 17A and 17B are diagrams illustrating a method for a mobile robot to interact with a user according to one embodiment of the present disclosure.

[0320] According to one embodiment, a mobile robot (e.g., the mobile robot (10) of FIG. 1 or the cleaning robot (100) of FIGS. 2A and 2B) may provide a user with an encouraging or appreciative message to inform the user that the user is contributing to the intelligence of the mobile robot. For example, as illustrated in FIG. 17A, the mobile robot (100) may display a message encouraging user feedback, such as "I'm done cleaning. Please check if I did it well," on the display of the mobile robot (100) or provide a voice response. For example, as illustrated in FIG. 17B, the mobile robot (100) may display a message of appreciation for user feedback, such as "Thank you for letting me know. I'll do better next time I clean." Through such interaction between the mobile robot and the user, the user can be provided with an element of fun, allowing the user to provide more active feedback. Through such active and continuous feedback, a mobile robot that is more optimized and intelligent for the user's real environment can be provided.

[0321] FIG. 18 is a diagram illustrating a method for a mobile robot to interact with a user to provide a reward to the user, according to one embodiment of the present disclosure.

[0322] According to one embodiment, a mobile robot (e.g., a mobile robot (10) of FIG. 1 or a cleaning robot (100) of FIGS. 2A and 2B) may encourage user feedback by providing a reward to the user according to the degree of the user's contribution to the intelligence of the mobile robot (100). For example, as illustrated in FIG. 18, the mobile robot (100) may display a phrase such as "Thanks to you, oo, I've become better at cleaning than before. I'll give you a thank you gift." on the display of the mobile robot (100) or provide it by voice. For example, as illustrated in FIG. 18, the mobile robot (100) or the user device (20) may display a screen (or user interface) (1830) including selectable items (hereinafter, "gift selection items") (1831) for selecting a gift, along with a phrase such as "If you press the gift, my appearance will be upgraded." Based on the user input (1801) identifying a gift selection item (1831) within the screen (1830), the mobile robot (100) can apply a gift (e.g., an icon, a skin) associated with the gift selection item (1831) to the mobile robot (100). Through this, the appearance of the mobile robot (100) can be upgraded. The screen (1830) can provide, along with the gift selection item (1831), an interaction count indicating the number of times the user has provided feedback and a reflection rate information indicating the reflection rate of the user feedback. Through this rewarding interaction between the mobile robot and the user, the user can provide more active feedback. Through this active and continuous feedback, a mobile robot that is more optimized and intelligent for the user's real environment can be provided.

[0323] FIG. 19 is a flowchart illustrating a method of operating a mobile robot according to one embodiment of the present disclosure.

[0324] Referring to FIG. 19, at operation 19010, a mobile robot (e.g., the mobile robot (10) of FIG. 1, the cleaning robot (100) of FIGS. 2A and 2B) can identify whether user feedback is required for a detected object.

[0325] In one embodiment, user feedback for a detected object may include, for example, user feedback on whether the motion of the mobile robot performed on the detected object is normal, or user feedback for correcting an unknown object (or a correction target object) to an object of a specified type.

[0326] In one embodiment, the mobile robot can identify whether user feedback is required based on type information of the detected object. The object type may be one of a cleaning object, which is an object configured to perform a cleaning operation on the mobile robot, an avoidance object, which is an object configured to perform an avoidance operation on the mobile robot, or an unknown object. For example, if the type of the detected object is identified as an unknown object based on the type information, the mobile robot can identify that user feedback is required for the detected object.

[0327] In one embodiment, the mobile robot can identify whether user feedback is required for a detected object based on classification accuracy information. For example, if the classification accuracy of the detected object is identified as being below a reference accuracy (e.g., 70, i.e., a value indicating 70% accuracy) based on the classification accuracy information, the mobile robot can identify that user feedback is required for the detected object. For example, if the detected object is identified as a cleaning object or an avoidance object based on the type information, but the classification accuracy is identified as being below a reference accuracy based on the classification accuracy information, the mobile robot can identify that user feedback is required for the detected object.

[0328] According to one embodiment, the mobile robot may identify that user feedback is needed for a detected object when a user input is identified selecting one of the at least one detected object displayed on a user interface (e.g., the first user interface (4110) or the third user interface (4230) of FIG. 4 or FIG. 7) for providing information about the at least one detected object.

[0329] In operation 19020, the mobile robot can obtain information about the distance between the mobile robot and the user based on identifying that user feedback is required for the object.

[0330] According to one embodiment, the mobile robot can obtain information about the distance between the mobile robot and the user by measuring the distance between the mobile robot and the user device (e.g., the user device (20) of FIG. 1). For example, the mobile robot can measure the distance between the mobile robot and the user device using a preset distance measurement method, and identify the measured distance or the distance obtained by correcting the measured distance as the distance between the mobile robot and the user. The preset distance measurement method may include, but is not limited to, a first method of measuring the distance between the mobile robot and the user using position information of the mobile robot (e.g., two-dimensional or three-dimensional coordinates of the mobile robot) obtained through position sensing of the mobile robot and position information of the user device (e.g., two-dimensional or three-dimensional coordinates of the user device) obtained through position sensing of the user device, or a second method of measuring the distance between the mobile robot and the user device using ranging technology (e.g., two-way ranging or one-way ranging based on ultra-wide band (UWB) communication).

[0331] In one embodiment, a mobile robot can obtain information about the distance between the mobile robot and the user by measuring the distance between the mobile robot and the user. For example, if the mobile robot recognizes the user through voice or image recognition, the mobile robot can measure the distance between the recognized user and the mobile robot based on the voice or image data, and identify the measured distance as the distance between the mobile robot and the user.

[0332] In operation 19030, the mobile robot can identify whether the distance between the mobile robot and the user is less than or equal to a specified distance (or, reference distance).

[0333] In operation 19040, based on identifying that the distance between the mobile robot and the user is less than or equal to a specified distance, the mobile robot can perform the user call operation.

[0334] In one embodiment, the user call action may include at least one action for calling the user and obtaining user feedback about the detected object via the mobile robot (e.g., a display of the mobile robot). For example, the user call action may include, but is not limited to, an action for providing call information for calling the user and / or an action for displaying a user interface (e.g., the second user interface (4120) of FIG. 4 or FIG. 7) on the display of the mobile robot for obtaining user feedback about the detected object.

[0335] In one embodiment, the call information may include information (e.g., auditory or visual) that indicates to the user that the mobile robot needs to approach the user immediately to provide user feedback. For example, the call information may include a voice message such as, "Please come to the mobile robot and provide user feedback on the currently detected object."

[0336] In one embodiment, the mobile robot may provide call information audibly via a voice user interface (VUI) or visually via a graphical user interface (GUI). This call information allows a user in close proximity to the mobile robot to approach the mobile robot to provide user feedback through direct interaction with the mobile robot.

[0337] In one embodiment, the mobile robot may suspend its motions (e.g., positioning, stopping, and / or avoiding motions) for a preset period of time after providing call information. This allows the mobile robot to wait for a user to approach the mobile robot to provide user feedback.

[0338] In one embodiment, the mobile robot may suspend a motion (e.g., a positioning motion, a stopping motion, and / or an avoidance motion) of the mobile robot for a second preset period of time after the time at which the user confirmation is received or for a third preset period of time after the call information is provided, if a user confirmation for the call information is received within a first preset period of time after the call information is provided. The third period of time may be longer than the first period of time. This allows the mobile robot to wait for a user to approach the mobile robot to provide user feedback only if a user confirmation for the call information is received.

[0339] In operation 19050, based on identifying that the distance between the mobile robot and the user exceeds a specified distance, the mobile robot may perform a user notification action.

[0340] In one embodiment, the user notification action may include at least one action to inform the user of the current status of the mobile robot. For example, the user notification action may include an action to transmit information (or a command) to the user device, which causes a status message to be displayed on the user device to inform a remote user of information about the current status of the mobile robot. In this case, a user at a remote location from the mobile robot can check the current status of the mobile robot through the status message, and notify another user at a nearby location from the mobile robot so that the other user can provide user feedback, or the user can provide user feedback himself when he or she moves within a nearby location from the mobile robot at a later time. The status message may include information to notify, for example, that a situation has occurred that requires user feedback regarding a detected object.

[0341] In one embodiment, the mobile robot may perform a designated action on a detected object after performing a user notification action or in conjunction with the user notification action. For example, the mobile robot may perform a designated action on a detected object after performing a user notification action or in conjunction with the user notification action without stopping the mobile robot's operation. Even in a situation where user feedback is required, since the user is at a distance from the mobile robot and cannot immediately obtain user feedback, the mobile robot may continue to perform the designated action without waiting for user feedback.

[0342] In action 19060, the mobile robot can autonomously perform a function based on identifying that no user feedback is required.

[0343] In one embodiment, in response to identifying that user feedback is not required, the mobile robot may perform a designated action corresponding to the type of detected object. For example, if the detected object is identified as a cleaning object, the mobile robot may perform a designated cleaning action on the detected object.

[0344] FIG. 20 is a flowchart illustrating a method of operating a mobile robot according to one embodiment of the present disclosure.

[0345] Referring to FIG. 20, in operation 20010, a mobile robot (e.g., the mobile robot (10) of FIG. 1, the cleaning robot (100) of FIGS. 2A and 2B) can detect (or identify) an object. Operation 20010 may include, for example, operation 3010 of FIG. 3. Duplicate descriptions thereof will be omitted.

[0346] In operation 20020, the mobile robot can identify the type of the object. In one embodiment, the type of the object may be one of a cleaning object, which is an object configured to perform a cleaning operation on the mobile robot, an avoidance object, which is an object configured to perform an avoidance operation on the mobile robot, or an unknown object. Operation 20030 may include, for example, operation 5050 of FIG. 5 or operations 6050 and / or 6070 of FIG. 6. Duplicate descriptions thereof will be omitted.

[0347] In operation 20030, the mobile robot can identify whether user feedback is required for corrections associated with the object type. The corrections associated with the object type may be, for example, corrections to the object type or corrections to a specified action based on the object type, but are not limited thereto. Operation 20030 may include, for example, operation 8100 of FIG. 8 or operation 9110 of FIG. 9 . Duplicate descriptions thereof will be omitted.

[0348] In operation 20040, the mobile robot may obtain information about the distance between the mobile robot and the user based on the identification that user feedback is required. Operation 20040 may include, for example, operation 19020 of FIG. 19 . Duplicate descriptions thereof will be omitted.

[0349] In operation 20050, the mobile robot can determine whether the distance between the mobile robot and the user is within a specified distance. Operation 20050 may include, for example, operation 19030 of FIG. 19. Duplicate descriptions thereof will be omitted.

[0350] In operation 20060, the mobile robot may provide call information for calling the user based on the determination that the distance between the mobile robot and the user is within the specified distance, and display a user interface for obtaining user feedback on the display. Operation 20060 may include, for example, operation 19040 of FIG. 19 . Duplicate descriptions thereof will be omitted.

[0351] In operation 20070, the mobile robot may transmit a message to an electronic device of the user connected to the mobile robot to display notification information on a display of the electronic device of the user, based on the determination that the distance between the mobile robot and the user is outside the specified distance, to inform the user of the current status of the mobile robot. Operation 20070 may include, for example, operation 19050 of FIG. 19. Duplicate descriptions thereof will be omitted.

[0352] In operation 20080, the mobile robot may perform a designated operation corresponding to the type of the mobile robot based on the identification that the user feedback is not required. The designated operation may be one of a cleaning operation for cleaning the cleaning object, an avoidance operation for avoiding the avoidance object, or an operation for correcting the unknown object. Operation 20080 may include, for example, operation 19060 of FIG. 19 . Duplicate descriptions thereof will be omitted.

[0353] According to one embodiment, the mobile robot can obtain the user feedback through the user interface and correct the type of the object based on the user feedback.

[0354] According to one embodiment, the mobile robot identifies whether the corrected type matches the identified type, and if it is identified that the corrected type does not match the identified type, it displays information for reconfirming the correction of the type of the object through the display, and if it is identified that the corrected type matches the identified type, it transmits a feedback message including information about the detected object and information about the corrected type to a server, and the feedback message can be used by the server to update a learned AI model used to classify the type of the object.

[0355] In one embodiment, the mobile robot can identify the type of the object by using the learned AI model to classify the object as one of the cleaning object, the avoidance object, or the unknown object.

[0356] According to one embodiment, the mobile robot inputs image data of the object or input data based on the image data into the learned AI model, obtains information indicating the type of the object as output data of the learned AI model, and identifies the type of the object based on the information indicating the type. The information indicating the type may be set to one of a first value indicating that the object is the cleaning object, a second value indicating that the object is the avoidance object, or a third value indicating that the object is the unknown object.

[0357] In one embodiment, the mobile robot can identify that user feedback is required when the object is identified as the unknown object based on the type of the object.

[0358] According to one embodiment, the mobile robot can obtain information about the accuracy of the type classification of the object, identify whether the accuracy of the type classification is below a reference accuracy, and if the accuracy of the type classification is identified as being below the reference accuracy, identify that the user feedback is required.

[0359] According to one embodiment, the mobile robot may stop its movement for a preset period of time after providing the call information.

[0360] According to one embodiment, if a user confirmation response corresponding to the call information is received within a preset first period after providing the call information, the mobile robot may stop the movement operation of the mobile robot for a preset second period after the time at which the user confirmation response is received or for a preset third period after providing the call information.

[0361] According to one embodiment, a mobile robot includes at least one sensor; a driving unit for moving the mobile robot; and a manipulator for organizing the organizing object, wherein the manipulator may include a gripper for gripping the organizing object for organizing.

[0362] In one embodiment, the mobile robot may provide a second user interface for obtaining user feedback for correction of at least one unknown object detected during the operation of the mobile robot after the operation of the mobile robot has ended. The second user interface may be a user interface for correcting the type of one of the at least one unknown object or a user interface for correcting the type of at least one of the clustered one or more unknown objects.

[0363] FIG. 21 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0364] The electronic device (21000) of the embodiment of FIG. 21 may be, for example, a mobile robot (e.g., a mobile robot (10) of FIG. 1, a cleaning robot (100) of FIGS. 2A and 2B), a user device (e.g., a user device (20) of FIG. 1), or a server (e.g., a server (30) of FIG. 1).

[0365] Referring to FIG. 21, an electronic device (21000) may include a display (21010), a communication unit (transceiver) (21020), a memory (21030), and / or at least one processor (21040). According to an example, the electronic device (21000) may include additional components (e.g., a display, an audio output unit (or speaker)) other than the illustrated components, or may omit at least one of the illustrated components.

[0366] According to one embodiment, depending on the type of electronic device (21000), at least some of the configurations disclosed in FIG. 21 may be omitted, or additional configurations may be included. For example, if the electronic device (21000) is a server, the display (21010) may be omitted from the configuration of FIG. 21. For example, if the electronic device (21000) is a mobile robot, in addition to the configuration of FIG. 21, the configurations of FIGS. 2A and 2B (e.g., a manipulator, a sensor, a driving unit, a lift, etc.) may be added.

[0367] According to one embodiment, the display (21010) may be implemented as a display of various types, such as an LCD, an OLED, an LCoS, a DLP, a QD display panel, a QLED, a μLED, a Mini LED, etc. According to one embodiment, the display (21010) may also be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc.

[0368] According to one embodiment, the display (21010) can display status information of the electronic device under the control of at least one processor (21040).

[0369] According to one embodiment, the communication unit (21020) may include a communication circuit that supports wired or wireless communication between the electronic device and an external electronic device. The wireless communication method may include, for example, the LTE method, the 5G NR method, the Wi-Fi method, Bluetooth, Bluetooth Low Energy, infrared communication (IrDA: Infrared Data Association), UWB (Ultra Wide Band), and NFC (Near Field Communication).

[0370] According to one embodiment, the memory (21030) may store various information or data related to the operation of the electronic device. For example, the memory (21030) may include one or more storage media that store at least one instruction. For example, the memory (21030) may include instructions that, when individually or collectively executed by at least one processor (21040), cause the electronic device to perform at least one operation. According to one example, the memory (21030) may include instructions that, when individually or collectively executed by at least one processor (21040), cause the electronic device to perform at least one of the operations described in FIGS. 1 to 20 .

[0371] According to one embodiment, at least one processor (21040) may be electrically or operatively connected to the display (21010), the communication unit (21020), and the memory (21030). The at least one processor (21040) may include a processing circuit that executes at least one instruction stored in the memory (21030).

[0372] According to one embodiment, at least one processor (21040) may include various processing circuits and / or multiple processors. One or more of the at least one processor (21040) may be individually and / or collectively configured to perform various functions described herein. In this disclosure, when "a processor," "at least one processor," and "one or more processors" are described as being configured to perform numerous functions, these terms encompass, for example, but are not limited to, a situation where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also encompass a situation where a single processor can perform all of the recited functions. Additionally, the at least one processor (1920) may include a combination of processors that perform the various recited / disclosed functions, for example, in a distributed manner. At least one processor (21040) can execute program instructions to accomplish or perform various functions.

[0373] According to one embodiment, at least one processor (21040) may include at least one of a central processing unit (CPU), a neural processing unit (NPU), a graphics processing unit (GPU), a micro processing unit (MPU), a micro controller unit (MCU), an application processor (AP), a communication processor (CP), a system on chip (SoC), or an integrated circuit (IC), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0374] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0375] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0376] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In mobile robots, communication circuit; display; memory for storing commands; and At least one processor connected to the memory and executing instructions stored in the memory, wherein the at least one processor: Detect objects, Identify the type of the object, and the type of the object is one of a cleaning object, which is an object set to perform a cleaning operation by the mobile robot, an avoidance object, which is an object set to perform an avoidance operation by the mobile robot, or an unknown object. Identify whether user feedback for corrections associated with the type of the above object is required, Based on the identification that the above user feedback is required, information about the distance between the mobile robot and the user is obtained, Identify whether the distance between the mobile robot and the user is within a specified distance, A mobile robot that provides call information for calling the user based on the distance between the mobile robot and the user being identified as being within the specified distance, and displays a user interface for obtaining user feedback through the display.

2. In the first paragraph, the at least one processor: Through the above user interface, the user feedback is obtained, A mobile robot that corrects the type of the object based on the user feedback.

3. In the second paragraph, the at least one processor: Identify whether the above corrected type matches the above identified type, If it is identified that the above corrected type does not match the above identified type, information for reconfirming the correction to the type of the object is displayed through the display, A mobile robot, wherein when it is identified that the corrected type matches the identified type, a feedback message including information about the detected object and information about the corrected type is transmitted to a server through the communication circuit, and the feedback message is used by the server to update a learned artificial intelligence (AI) model used to classify the type of the object.

4. In the first paragraph, the at least one processor: Based on the identification that the above user feedback is not required, perform a designated action corresponding to the type of the mobile robot, A mobile robot, wherein the above-mentioned specified action is one of a cleaning action for cleaning the cleaning object, an avoidance action for avoiding the avoidance object, or an action for correcting the unknown object.

5. In the first paragraph, the at least one processor: A mobile robot that transmits a message to an electronic device of the user connected to the mobile robot through the communication circuit to display notification information for informing the user of the current status of the mobile robot on the display of the electronic device of the user based on the distance between the mobile robot and the user being identified as being outside the specified distance.

6. In any one of paragraphs 1 to 5, the at least one processor: A mobile robot that identifies the type of the object by classifying the object as one of the cleaning object, the avoidance object, or the unknown object using a learned AI model.

7. In the 6th paragraph, the at least one processor: Input data based on the image data of the object is input to the learned AI model, and information indicating the type of the object is obtained as output data of the learned AI model. Identifying the type of the object based on information indicating the type, A mobile robot, wherein the information indicating the type is set to one of a first value indicating that the object is the cleaning object, a second value indicating that the object is the avoidance object, or a third value indicating that the object is the unknown object.

8. In any one of paragraphs 1 to 7, the at least one processor: A mobile robot that identifies that user feedback is required when the object is identified as an unknown object based on the type of the object.

9. In any one of paragraphs 1 to 8, the at least one processor: Obtain information about the accuracy of the type classification of the above object, Identify whether the accuracy of the above type classification is below the standard accuracy, A mobile robot that identifies that user feedback is required when the accuracy of the above type classification is identified as being below the standard accuracy.

10. In the first paragraph, the at least one processor: A mobile robot that stops the movement of the mobile robot for a preset period of time after providing the above call information.

11. In the first paragraph, the at least one processor: A mobile robot that stops the movement of the mobile robot during a preset second period after the time at which the user confirmation response is received or a preset third period after the call information is provided, when a user confirmation response corresponding to the call information is received within a preset first period after the call information is provided.

12. In paragraph 1, the mobile robot: At least one sensor; A driving unit for movement of the above mobile robot; and A mobile robot comprising a manipulator for organizing the above-mentioned organizing object, wherein the manipulator comprises a gripper for gripping the above-mentioned organizing object for organizing.

13. In the first paragraph, the at least one processor: After the operation of the mobile robot is completed, a second user interface is provided for obtaining user feedback for correction of at least one unknown object detected during the operation of the mobile robot. A mobile robot, wherein the second user interface is a user interface for correcting the type of at least one of the at least one unknown object or a user interface for correcting the type of at least one of the clustered one or more unknown objects.

14. In the method of operating a mobile robot, The action of detecting an object; An action for identifying the type of the object, wherein the type of the object is one of a cleaning object, which is an object set to perform a cleaning operation on the mobile robot, an avoidance object, which is an object set to perform an avoidance operation on the mobile robot, or an unknown object; An action to identify whether user feedback for correction is required associated with the type of said object; An operation of obtaining information about the distance between the mobile robot and the user based on the identification that the user feedback is required; An action for identifying whether the distance between the mobile robot and the user is within a specified distance; and A method comprising: providing call information for calling the user based on identification that the distance between the mobile robot and the user is within the specified distance; and displaying a user interface for obtaining user feedback through a display.

15. In paragraph 14, the method: An operation of obtaining user feedback through the user interface; and A method comprising an action of correcting the type of the object based on the user feedback.

Images