Robot and operation method thereof

Abstract

A robot is disclosed. The robot comprises: a first camera sensor disposed at a first capturing angle to capture a driving space of the robot; a mirror disposed at a first placement angle at a first position of the robot; at least one processor including a processing circuit; and a memory for storing instructions and including one or more storage media, wherein the instructions cause, when individually or collectively executed by the at least one processor, the robot to: acquire a first image corresponding to a first area within the driving space by using the first camera sensor; on the basis of the first image, identify a second image which corresponds to a second capturing angle and in which a second area within the driving space is reflected through the mirror; on the basis of the acquired first image, identify a third image obtained by capturing the second area at the first capturing angle; and on the basis of the second image corresponding to the second capturing angle and the third image corresponding to the first capturing angle, identify a liquid-type object within the second area.

Description

Robot and method of operation thereof

[0001] The present disclosure relates to a robot and a method of operation thereof, and more specifically, to a robot that travels in space and a method of operation thereof.

[0002] Driven by advancements in electronic technology, various types of electronic devices are being developed. Technological development for robots that provide services to users is also becoming active. In the case of robots navigating specific spaces to provide services, situations may arise where they must pass through or remove objects present within their path. Robots can navigate these spaces efficiently by accurately identifying the location and type of objects.

[0003] A drivable robot according to one embodiment of the present disclosure may include a first camera sensor positioned at a first shooting angle to photograph the driving space of the robot, a mirror positioned at a first positioning angle at a first position of the robot, at least one processor including a processing circuit, and a memory including one or more storage media for storing instructions.

[0004] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the robot may be able to acquire a first image corresponding to a first area within the driving space using the first camera sensor.

[0005] According to one embodiment, the instructions may enable the robot to identify a second image corresponding to a second shooting angle, in which a second area within the driving space is reflected through the mirror, based on the first image.

[0006] According to one embodiment, the instructions may enable the robot to identify a third image in which the second region is captured at the first shooting angle, based on the acquired first image.

[0007] According to one embodiment, the instructions may enable the robot to identify a liquid-type object within the second area based on a second image corresponding to the second shooting angle and a third image corresponding to the first shooting angle.

[0008] A method of operation of a robot according to one embodiment of the present disclosure may include the operation of acquiring a first image corresponding to a first area within the driving space of the robot using a first camera sensor positioned at a first shooting angle.

[0009] According to one embodiment, the operation method may include, based on the first image, an operation of identifying a second image corresponding to a second shooting angle, which is reflected through a mirror positioned at a first positioning angle at a first positioning angle of a second region within the driving space.

[0010] According to one embodiment, the operation method may include an operation of identifying a third image in which the second region is captured at the first shooting angle, based on the acquired first image.

[0011] According to one embodiment, the operation method may include an operation of identifying a liquid-type object within the second area based on a second image corresponding to the second shooting angle and a third image corresponding to the first shooting angle.

[0012] In a storage medium for storing computer-readable instructions according to one embodiment of the present disclosure, the instructions may cause the robot to acquire a first image corresponding to a first area within the robot's travel space by using a first camera sensor positioned at a first shooting angle when executed by at least one processor of the robot.

[0013] According to one embodiment, the instructions may cause the robot to identify a second image corresponding to a second shooting angle, which is reflected through a mirror positioned at a first position angle at a first position angle of the second area within the driving space based on the first image.

[0014] According to one embodiment, the instructions may cause the robot to identify a third image in which the second region is captured at the first shooting angle, based on the acquired first image.

[0015] According to one embodiment, the instructions may cause the robot to identify a liquid-type object within the second area based on a second image corresponding to the second shooting angle and a third image corresponding to the first shooting angle.

[0016] The following and other aspects, features and / or advantages of the embodiments of this disclosure will become more apparent from the following description, which is referenced together with the accompanying drawings.

[0017] FIG. 1a is a block diagram showing the configuration of a robot according to one embodiment.

[0018] FIG. 1b is a drawing for explaining the operation method of a robot according to one embodiment.

[0019] FIG. 1c is a drawing for explaining the operation method of a robot according to one embodiment.

[0020] FIG. 1d is a drawing for explaining the operation method of a robot according to one embodiment.

[0021] FIG. 1e is a drawing for explaining a method of operation of a robot according to one embodiment.

[0022] FIG. 2 is a flowchart illustrating a method of operation of a robot according to one embodiment.

[0023] FIG. 3 is a flowchart illustrating a method for identifying the existence of an object according to one embodiment.

[0024] FIG. 4 is a flowchart for explaining the operation of a robot related to an object according to one embodiment.

[0025] FIG. 5 is a flowchart illustrating a method for performing avoidance driving on an object according to one embodiment.

[0026] FIG. 6a is a flowchart illustrating a method for identifying whether an object exists according to one embodiment.

[0027] FIG. 6b is a drawing for explaining a method for identifying whether an object exists according to one embodiment.

[0028] FIG. 7a is a flowchart illustrating a method for identifying whether an object exists according to one embodiment.

[0029] FIG. 7b is a drawing for explaining a method for identifying whether an object exists according to one embodiment.

[0030] FIG. 7c is a drawing for explaining a method for identifying whether an object exists according to one embodiment.

[0031] FIG. 8 is a flowchart for explaining the operation of a robot related to an object according to one embodiment.

[0032] FIG. 9a is a flowchart illustrating a method for identifying whether an object exists according to one embodiment.

[0033] FIG. 9b is a drawing for explaining a method for identifying whether an object exists according to one embodiment.

[0034] FIG. 10a is a drawing for explaining the role of a mirror according to one embodiment.

[0035] FIG. 10b is a drawing for explaining the role of a mirror according to one embodiment.

[0036] FIG. 11 is a block diagram showing the detailed configuration of a robot according to one embodiment.

[0037] The present disclosure will be described in detail below with reference to the attached drawings.

[0038] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0039] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0040] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0041] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0042] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0043] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0044] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0045] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0046] FIGS. 1a to 1e are drawings for explaining the operation of a robot (100) according to one embodiment of the present disclosure.

[0047] Referring to FIGS. 1a through 1e, according to one embodiment, a robot (100) may be implemented as a robot (100) capable of driving in a driving space. According to one example, the robot (100) may include a first camera sensor (110), a mirror (Mirror, 120), at least one processor (130), and a memory (140).

[0048] According to one embodiment, the robot (100) can travel through a travel space to reach a destination. The robot (100) may be a robot that moves to a specific location and provides a service to a user. According to one example, the robot may be a different type of travel robot including a drone or a wheel robot, but is not limited thereto.

[0049] According to one embodiment, the first camera sensor (110) may be positioned within the robot (100) at a first shooting angle (114). According to one example, the first camera sensor (110) may be implemented as an RGB (Red, Green and Blue) camera. Alternatively, according to one example, the first camera sensor (110) may be implemented as a stereo camera. According to one example, the stereo camera may each include a camera corresponding to a person's left eye and a camera corresponding to a person's right eye, and according to one example, when the first camera sensor (110) is implemented as a stereo camera, the first camera sensor (110) may obtain three-dimensional image information including disparity information through images obtained from a plurality of cameras (e.g., a camera corresponding to the left eye and a camera corresponding to the right eye).

[0050] Alternatively, according to one example, the first camera sensor (110) may be a stereo camera implemented as an IR camera (Infrared Camera). However, it is not limited thereto, and the first camera sensor (110) may be implemented as a camera of a different type. According to one example, the first camera sensor (110) may be positioned at a location corresponding to a predetermined height from the ground.

[0051] According to one example, the first shooting angle (114) may be the angle (114) at which the first camera sensor (110) is tilted relative to a line (Line, 11) perpendicular to the ground (1). According to one example, the angle (114) at which the first camera sensor (110) is tilted may be the angle between a line perpendicular to the ground (1) and a line perpendicular to the shooting direction of the first camera sensor (110).

[0052] According to one embodiment, the first camera sensor (110) can photograph the driving space of the robot (100). According to one example, the first camera sensor (110) can photograph the driving space of the robot (100) within a shooting range within a preset field of view (FOV, 113). According to one example, the driving space may include a first area (112). According to one example, an image (101) corresponding to the driving space obtained (or photographed) through the first camera sensor (110) may include an image (103) corresponding to the first area (112) within the driving space of the robot (100).

[0053] According to one example, the first area (112) may be a floor area located within a preset distance from the robot (100). For example, the first area (112) may be an area on the driving space corresponding to a shooting range (111) between a shooting direction and a field of view (113) corresponding to the first camera sensor (110). According to one example, as illustrated in FIG. 1c, the first area (112) may be an area formed based on a field of view (113) corresponding to the first camera sensor (110), and according to one example, the first area (112) may be an area within a preset range of distance from the robot (100). According to one example, the first area (112) may be at least a portion of the area that can be captured through the first camera sensor (110).

[0054] According to one embodiment, the mirror (120) may be positioned at a first position of the robot (100) at a first position angle (115). According to one example, the first position may be a position relatively higher than the first camera sensor (110) and may be included in an area corresponding to the field of view (113) of the first camera sensor (110). According to one example, the area corresponding to the field of view (113) may be an area located within the field of view (113) of the first camera sensor (110) and included in the image (101) obtained through the first camera sensor (110). According to one example, the mirror (120) may be included in an area corresponding to the field of view (113), but is not limited thereto, and as shown in FIG. 1d, a part of the mirror (120) may be included in an area corresponding to the field of view (113). According to one example, the mirror (120) may be positioned in the robot (100) at a relatively higher position than the first camera sensor (110).

[0055] According to one example, the first positioning angle (115) may be the angle between a line (11) perpendicular to the ground and a line parallel to the reflective surface of the mirror (120). According to one example, the image (101) obtained through the first camera sensor (110) may include an image (102) of a specific area (or a second area (122)) on the driving space reflected through the mirror (120) positioned at the first positioning angle (115). According to one example, different types of images may be obtained depending on the positioning angle of the mirror (120).

[0056] For example, as the angle of placement of the mirror (120) increases, the range of the area reflected through the mirror (120) can be relatively wider. According to one example, if the angle of placement of the mirror (120) is greater than a preset angle (or if the range of the area reflected through the mirror (120) is relatively wide), the robot (100) can divide the area reflected through the mirror (120) into multiple regions and detect (or identify) a liquid-type object (10) based on an image corresponding to each of the divided multiple regions. This will be explained in detail through FIGS. 7a and 7b.

[0057] Alternatively, for example, as the angle of placement of the mirror (120) decreases, the range of the area reflected through the mirror (120) may be relatively narrower. In one example, if the angle of placement of the mirror (120) is less than a preset angle, the area reflected through the mirror (120) may include a blind area that cannot be captured through the first camera sensor (110), as an area other than the first area (112). In one example, the robot (100) may detect an object (10) present in the blind area based on an image (102) of the area reflected through the mirror (120). This will be explained in detail through FIGS. 9a and 9b.

[0058] At least one processor (130) (hereinafter, processor) is electrically connected to the first camera sensor (110), mirror (120), and memory (140) to control the overall operation of the robot (100). The processor (130) may be composed of one or more processors. Specifically, the processor (130) may perform the operation of the robot (100) according to various embodiments of the present disclosure by executing at least one instruction stored in the memory (140).

[0059] According to one embodiment, the processor (130) may be implemented as a digital signal processor (DSP) that processes digital video signals, a microprocessor, a Graphics Processing Unit (GPU), an Artificial Intelligence (AI) processor, a Neural Processing Unit (NPU), or a Time Controller (TCON). However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), a Micro Controller Unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor. Additionally, the processor (130) may be implemented as a System on Chip (SoC) or Large Scale Integration (LSI) with a built-in processing algorithm, or may be implemented in the form of an Application Specific Integrated Circuit (ASIC) or Field Programmable Gate Array (FPGA).

[0060] The memory (140) can store data for various embodiments. Depending on the purpose of data storage, the memory (140) may be implemented in the form of a memory embedded in the robot (100) or in the form of a memory that can be attached to and detached from the robot (100). For example, data for driving the robot (100) may be stored in a memory embedded in the robot (100), and data for the expansion function of the robot (100) may be stored in a memory that can be attached to and detached from the robot (100).

[0061] In the case of memory embedded in the robot (100), it may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), etc.), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD). Additionally, in the case of memory that can be attached to the robot (100), it may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), or external memory that can be connected to a USB port (e.g., USB memory).

[0062] According to one embodiment, the processor (130) can acquire an image through the first camera sensor (110). According to one example, the image acquired through the first camera sensor (110) may be an image (101) corresponding to the driving space of the robot (100). According to one example, the processor (130) can acquire a first image (103) corresponding to a first area (112) through the first camera sensor (110). According to one example, the processor (130) can acquire a first image (103) corresponding to the first area (112) among the images (101) corresponding to the driving space. For example, the processor (130) can acquire a first image (103) corresponding to the first area (112) within the images (101) corresponding to the driving space by using coordinate information corresponding to an area within a preset distance from the robot (100). For example, the memory (140) may store coordinate information of an image (103) corresponding to a first region (112) among images (101) corresponding to a driving space acquired through the first camera sensor (110). The processor (130) can identify the first image (103) from the images (101) corresponding to the driving space based on the information stored in the memory (140).

[0063] According to one example, if the first camera sensor (110) is implemented as an RGB camera, the image (101) corresponding to the driving space may be an image having RGB color values ​​per pixel included in the image. Alternatively, according to one example, if the first camera sensor (110) is implemented as an IR camera, the image (101) corresponding to the driving space may be a thermal image.

[0064] According to one embodiment, the processor (130) can identify an image corresponding to an area in the driving space that is reflected through the mirror (120). According to one example, the processor (130) can identify a second image (102) corresponding to a second shooting angle in which a second area (122) in the driving space is reflected through the mirror (120), based on an image (101) obtained through the first camera sensor (110).

[0065] According to one example, the second area (122) may be an area reflected through the mirror (120) with respect to the first camera sensor (110). For example, the second area (122) may be an area in the driving space corresponding to the reflection range (121) of the light reflected through the mirror among the light incident on the first camera sensor (110). According to one example, the second image (102) corresponding to the second area (122) may be included in the image (101) obtained through the first camera sensor (110).

[0066] According to one example, the processor (130) can identify a second image (102) in which a second region (122) is reflected through the mirror (120) by using coordinate information corresponding to the mirror (120) within an image (101) corresponding to the driving space. For example, the memory (140) may store coordinate information of an image corresponding to the mirror among the images (101) corresponding to the driving space obtained through the first camera sensor (110). The processor (130) can identify the second image (102) from the image (101) corresponding to the driving space based on the information stored in the memory (140).

[0067] According to one example, the second image (102) reflected through the mirror (120) positioned at the first positioning angle (115) may be an image corresponding to a shooting angle (or a second shooting angle) different from the first shooting angle (114) corresponding to the first camera sensor (110). According to one example, the second image (102) may correspond to an image in which the second region (122) is captured at the second shooting angle.

[0068] According to one embodiment, the processor (130) can identify a third image (104) in which a second area (122) is captured at a first shooting angle (114). According to one example, the processor (130) can identify a third image (104) corresponding to the second area (122) among a first image (103) corresponding to the first area (112) captured at a first shooting angle (114). According to one example, the third image (104) corresponding to the second area (122) may be an image captured at the first shooting angle (114). According to one example, the processor (130) can identify the third image (104) by using coordinate information corresponding to the second area (122) within the first image (103) obtained through the first camera sensor (110). For example, the memory (140) may store coordinate information of an image corresponding to a second region among the first image (103) acquired through the first camera sensor (110). The processor (130) can identify a third image (104) from the first image (103) based on the information stored in the memory (140).

[0069] According to one embodiment, the processor (130) can detect an object (10) present within the second area (122). According to one example, the processor (130) can detect a liquid-type object (10) within the second area (122) based on a second image (102) corresponding to a second shooting angle and a third image (104) corresponding to a first shooting angle (114).

[0070] According to one example, the object (10) may be a liquid type object, but is not limited thereto, and may be an object of a different type (e.g., an object type, etc.). According to one example, the operation of detecting the object (10) may include at least one of the operation of identifying whether the object (10) is present and the operation of identifying the exact location of the object (10) when the object (10) is identified.

[0071] According to one example, the processor (130) can detect a liquid-type object (10) present in the second area (122) by using images of different shooting angles corresponding to the second area (122). According to one example, the processor (130) can classify whether a liquid-type object (10) is present in the second area (122) by inputting (or providing) the second image (102) and the third image (104) to a trained neural network model. According to one example, the trained neural network model may be a model trained to classify images based on whether a liquid-type object (10) is present in the input image.

[0072] According to one example, the processor (130) may input each of the second image (102) and the third image (104) corresponding to the second region (122) into a learned neural network model. For example, the processor (130) may input the second image (102) into the learned neural network model to identify whether the second image (102) contains a liquid-type object (10). Alternatively, for example, the processor (130) may input the third image (104) into the learned neural network model to identify whether the third image (104) contains a liquid-type object (10).

[0073] According to one example, if the processor (130) identifies that an object (10) exists in a second area (122) based on at least one of a second image (102) and a third image (104), it can identify location information of a liquid-type object (10) based on at least one of the second image (102) and the third image (104). According to one example, the location information may be information about the location of a liquid-type object (10) within a driving space. According to one example, the processor (130) can identify information about the location of a liquid-type object (10) using a learned neural network model. For example, if it is identified that an object (10) exists in a second area (122), at least one of the second image (102) and the third image (104) can be input into a learned neural network model to identify location information of a liquid-type object (10) existing in the second area (122).

[0074] According to one example, a neural network model trained to identify whether a liquid-type object (10) exists within a second region (122) and a neural network model trained to identify the location of a liquid-type object (10) existing within the second region may be implemented as different models, but are not limited thereto, and the above-described neural network model may, of course, be implemented as a single neural network model. According to one example, the trained neural network model of the present disclosure may be stored in memory (140), but is not limited thereto, and the trained neural network model may be stored in an external device (e.g., a server).

[0075] According to one embodiment, the processor (130) may perform evasive driving on the liquid-type object (10) or perform a removal operation on the liquid-type object (10) based on location information of the identified liquid-type object (10). According to one example, the processor (130) may identify a driving path to avoid the object (10) based on location information of the liquid-type object (10) and drive through the driving space based on the identified driving path. Alternatively, according to one example, the processor (130) may perform a driving operation to move to the location of the object (10) based on location information of the liquid-type object (10), and then perform a removal operation on the object (10).

[0076] In the case of a liquid-type object (10), the reflectivity of the liquid surface may vary depending on the shooting angle, and when the liquid-type object (10) is photographed at a specific shooting angle, the liquid-type object (10) may not be identifiable. According to the example described above, the robot (100) of the present invention can detect the liquid through images corresponding to multiple shooting angles for a specific area (e.g., a second area (122)) in the driving space, and thus can detect the liquid with a high detection rate.

[0077] FIG. 2 is a flowchart illustrating a method of operation of a robot according to one embodiment.

[0078] Referring to FIG. 2, according to one embodiment, the operation method may include an operation (S210) of acquiring a first image (e.g., the first image (103) of FIG. 1e) corresponding to a first area (e.g., the first area (112) of FIG. 1b) within a driving space based on an image (e.g., the first camera sensor (110) of FIG. 1a) acquired through a first camera sensor (e.g., the first camera sensor (110) of FIG. 1a)). According to one example, when an image is acquired through the first camera sensor, a robot (e.g., the robot (100) of FIG. 1a) may acquire a first image corresponding to a first area within a driving space based on the acquired image.

[0079] According to one embodiment, the operation method may include an operation (S220) of identifying a second image (e.g., a second image (102) in FIG. 1e) corresponding to a second shooting angle (e.g., a second shooting angle in FIG. 1d) in which a second area (e.g., a second area (122) in FIG. 1b) in a driving space is reflected through a mirror (e.g., a mirror (102) in FIG. 1a) based on an image acquired through a first camera sensor. According to one example, a robot may identify a second image corresponding to a second shooting angle in which a second area in a driving space is reflected through a mirror based on an image acquired through a first camera sensor.

[0080] According to one embodiment, the operation method may include an operation (S230) of identifying a third image (e.g., a third image (104) in FIG. 1e) in which a second region is captured at a first shooting angle (e.g., a first shooting angle (114) in FIG. 1d) based on a first image obtained. According to one example, when a first image is obtained, the robot may identify a third image in which a second region is captured at a first shooting angle.

[0081] According to one embodiment, the operation method may include an operation (S240) of detecting a liquid-type object (e.g., object (10) of FIG. 1b) within a second area based on a second image corresponding to a second shooting angle and a third image corresponding to a first shooting angle. According to one example, a robot may detect a liquid-type object within a second area based on a second image corresponding to a second shooting angle and a third image corresponding to a first shooting angle.

[0082] FIG. 3 is a flowchart illustrating a method for identifying the existence of an object according to one embodiment.

[0083] Referring to FIG. 3, according to one embodiment, the operation method may include an operation (S310) of performing calibration on a second image (e.g., the second image (102) of FIG. 1a and FIG. 1e).

[0084] According to one example, when a second image is acquired, a robot (e.g., the robot (100) of FIG. 1a) may perform a correction on the second image using a preset algorithm. For example, the robot may perform a correction on the robot's state (e.g., an error due to the attachment state of the mirror or the robot's tolerance). Or, the robot may perform a correction on the distance error due to the reflection of the image through the mirror. Or, the robot may perform a correction on the image of the object (or mirror image) due to the reflection of the image through the mirror.

[0085] According to one embodiment, the operation method may include an operation (S320) of identifying a third image (e.g., a third image (104) in FIG. 1e) that corresponds to an area that matches a second image among a first image corresponding to a first area (e.g., a first area (112) in FIG. 1b). According to one example, a second area that matches a second image (e.g., a second area (122) in FIG. 1b) may be an area included in the first area.

[0086] According to one example, the robot can identify a third image among the first images (e.g., the first image (103) of FIG. 1e) as a captured image of a second area corresponding to a second image within the driving space. According to one example, information regarding the second area within the driving space according to the placement angle of a mirror (e.g., the mirror (102) of FIG. 1a) may be stored in a memory (e.g., the memory (140) of FIG. 1a). Based on the information stored in the memory, the robot can identify a third image corresponding to the second area among the first images.

[0087] According to one embodiment, the operation method may include an operation (S330) of inputting a corrected second image and an identified third image into a learned neural network model (e.g., the learned neural network model of FIGS. 1a to 1e) to identify whether a liquid-type object (e.g., object (10) of FIG. 1b) exists within a second region.

[0088] According to one example, the trained neural network model may be a model trained to output whether a liquid-type object exists within the received images when a second image and a third image, on which correction has been performed, are received. Alternatively, according to one example, the trained neural network model may be a model trained to identify the location of a liquid-type object when it is identified that a liquid-type object exists.

[0089] According to one example, the robot may input a corrected second image and an identified third image into a trained neural network model to identify whether a liquid-type object exists within a second region corresponding to the second image and the third image. Alternatively, the robot may input the corrected second image and the third image into a trained neural network model to identify the exact location of the liquid-type object within the second region.

[0090] According to the example described above, the robot can identify whether a liquid-type object exists within a second area or determine (or identify) the exact location of a liquid-type object by using a second image corresponding to a second area where correction has been performed and a third image corresponding to an area that matches the second image.

[0091] According to the example described above, the robot can identify a liquid-type object existing within a second area by using a third image corresponding to a first shooting angle and a second image corresponding to a second shooting angle. Accordingly, the robot can identify the presence of a liquid-type object by using images of different shooting angles corresponding to the same area, and the detection rate of the liquid-type object can be improved.

[0092] FIG. 4 is a flowchart for explaining the operation of a robot related to an object according to one embodiment.

[0093] According to FIG. 4, according to one embodiment, the operation method may include an operation (S410) of inputting a corrected second image (e.g., the corrected second image of FIG. 3) and an identified third image (e.g., the third image (104) of FIG. 1e) into a learned first neural network model to identify whether a liquid-type object (e.g., the object (10) of FIG. 1b) exists within a second region (e.g., the second region (122) of FIG. 1b).

[0094] According to one example, the trained first neural network model may be a model trained to classify whether a liquid-type object exists within an input image when an image is input. According to one example, a robot (e.g., the robot (100) of FIG. 1a) may input a second image and a third image into the trained first neural network model to identify whether a liquid-type object exists within a second region corresponding to the second image and the third image.

[0095] For example, the trained first neural network model can classify whether a liquid-type object exists within the image for each of the second image and the third image. The trained first neural network model can classify the second image as an image in which a liquid-type object exists and the third image as an image in which a liquid-type object does not exist. Based on the output result, the robot can identify that a liquid-type object exists within the second region.

[0096] According to one embodiment, the operation method may include an operation (S420) of identifying location information of a liquid-type object by inputting a corrected second image and an identified third image into a trained second neural network model when it is identified that a liquid-type object exists within a second region.

[0097] According to one example, the trained second neural network model may be a model trained to output location information of a liquid-type object present within an input image when an image is input. According to one example, the image input to the trained second neural network model may be an image containing a liquid-type object, but is not limited thereto, and according to one example, the trained second neural network model may classify whether a liquid-type object is present in the image and output location information of the object based on this.

[0098] According to one example, if the robot identifies that a liquid-type object exists within a second area, it may input a second image and a third image into a trained second neural network model. According to one example, the robot may identify location information of the liquid-type object existing within the second area based on the output result. According to one example, the location information of the liquid-type object may be information identified based on the coordinate information of the liquid-type object within the image corresponding to the second area, and may be information regarding the location of the liquid-type object in the second area on the driving space.

[0099] According to one embodiment, the operation method may include an operation (S430) of performing evasive driving on a liquid-type object or performing a removal operation on a liquid-type object based on identified location information.

[0100] According to one example, when the location information of a liquid-type object is identified, the robot can perform avoidance driving on the object based on the identified location information. For example, the robot can identify a driving path based on the location information and perform driving operations along the identified driving path. Alternatively, for example, the robot may perform a removal operation of the liquid-type object.

[0101] According to one example, the trained first neural network model and the trained second neural network model may be implemented as separate neural network models, but are not limited thereto, and according to one example, the trained first neural network model and the trained second neural network model may be implemented as a single neural network model. According to one example, the robot may identify location information of a liquid-type object within a second area using only the trained second neural network model, without performing an operation to classify whether a separate object exists.

[0102] According to the example described above, the robot can identify the location of an object existing within a second area by using images of multiple shooting angles of a second area in the driving space, and based on this, perform at least one of an avoidance action or a removal action. Accordingly, the driving performance of the robot can be improved.

[0103] FIG. 5 is a flowchart illustrating a method for performing avoidance driving on an object according to one embodiment.

[0104] According to FIG. 5, according to one embodiment, the operation method may include an operation (S510) of inputting a corrected second image (e.g., the corrected image of FIG. 4) and an identified third image (e.g., the third image (104) of FIG. 1e) into a learned third neural network model to identify location information of a preset type of object existing within a second region (e.g., the second region (122) of FIG. 1b).

[0105] For example, a pre-set type of object may be an object of a different type from a liquid type object. For example, a pre-set type of object may be an object of a different type existing in the house (e.g., furniture, objects, people, animals, etc.), but is not limited thereto.

[0106] According to one example, a robot (e.g., the robot (100) of FIG. 1a) can identify location information of an object existing within a second area by inputting a second image and a third image into a learned third neural network model. For example, the robot can identify location information of a pre-set type of object existing in the second area based on the output result obtained from the learned third neural network model.

[0107] According to one embodiment, the operation method may include an operation (S520) of performing avoidance driving for a preset type of object based on identified location information.

[0108] According to one example, when location information for an object of a preset type is identified, the robot can identify a driving path based thereon and perform driving operations along the identified driving path. For example, when location information for an object of a preset type is identified within a second area, the robot can perform avoidance driving.

[0109] FIG. 6a is a flowchart illustrating a method for identifying whether an object exists according to one embodiment. FIG. 6b is a diagram illustrating a method for identifying whether an object exists according to one embodiment.

[0110] According to FIGS. 6a and 6b, according to one embodiment, the operation method may include an operation (S610) of acquiring a fourth image corresponding to a first area (612, e.g., the first area (112) of FIG. 1a) within a driving space based on an image acquired through a second camera sensor (610).

[0111] According to one example, the robot (600, e.g., the robot (100) of FIG. 1a) may further include a second camera sensor (610) positioned at a third shooting angle to photograph a driving space (e.g., the driving space of FIG. 1a). According to one example, the third shooting angle may be a shooting angle different from the first shooting angle (e.g., the first shooting angle (114) of FIG. 1d) and the second shooting angle (e.g., the second shooting angle of FIG. 1d), but is not limited thereto, and the third shooting angle may be the same as at least one of the first shooting angle or the second shooting angle.

[0112] According to one example, the second camera sensor (610) can capture the driving space of the robot (100) within a pre-set field of view (FOV, 613) within a shooting range. According to one example, the driving space may include a first area (612). According to one example, an image corresponding to the driving space obtained (or captured) through the second camera sensor (610) may include an image corresponding to the first area (612) within the driving space of the robot (600).

[0113] According to one example, the first area (612) may be an area on the driving space corresponding to the shooting range (611) between the shooting direction and the field of view (613) corresponding to the second camera sensor (610). According to one example, the first area (612) may be an area within a preset range of distance from the robot (600). According to one example, the first area (612) may be at least a portion of the area that can be captured through the second camera sensor (610).

[0114] According to one example, the robot (600) can acquire a fourth image as an image of an area corresponding to a first area among the images (or images corresponding to the driving space) acquired through the second camera sensor (610).

[0115] According to one embodiment, the operation method may include an operation (S620) of identifying a fifth image reflected through a mirror (620) of a second area (622, e.g., the second area (122) of FIG. 1b) based on an image (or an image corresponding to a driving space) obtained through a second camera sensor (610).

[0116] According to one embodiment, the mirror (620) may be positioned at a second position of the robot (600) at a second position angle. According to one example, the mirror (620) may be a mirror different from the mirror (e.g., mirror (120) of FIG. 1b) corresponding to the first camera sensor (e.g., first camera sensor (110) of FIG. 1b). According to one example, the second position angle may be different from the first position angle shown in FIG. 1d, but is not limited thereto.

[0117] According to one example, the second position may be a position relatively higher than the second camera sensor (610) and included in the area corresponding to the viewing angle (613). According to one example, the area corresponding to the viewing angle (613) may be an area located within the viewing angle (613) and included in the image obtained through the second camera sensor (610). According to one example, the mirror (620) may be included in the area corresponding to the viewing angle (613), but is not limited thereto, and a part of the mirror (620) may be included in the area corresponding to the viewing angle (613).

[0118] According to one example, the second camera sensor (610) is implemented as a stereo camera including camera sensors corresponding to the left eye and the right eye, respectively, and the first camera sensor may be implemented as an RGB (Red, Green and Blue) camera sensor, but is not limited thereto, and if the first camera sensor is implemented as a stereo camera, the second camera sensor (610) may be implemented as an RGB camera sensor.

[0119] According to one example, if the robot (600) includes a first camera sensor and a second camera sensor (610), respectively, the first camera sensor and the second camera sensor (610) may be positioned at different locations within the robot (600). For example, if the first camera sensor is implemented as an RGB camera, the first camera sensor may be positioned at a position corresponding to a preset height (e.g., 63 mm (millimeter)) from the ground (1). For example, if the second camera sensor (610) is implemented as a stereo camera, the second camera sensor (610) may be positioned within the robot (600) at a preset angle (tilted angle) at a preset height (e.g., 46 mm) from the ground (1).

[0120] According to one example, if the robot (600) includes a second camera sensor (610), the robot (600) may include a light-emitting element (630, e.g., an LED (Light Emitting Diode)) that emits light in a preset wavelength band (e.g., an infrared wavelength band). However, it is not limited thereto, and the wavelength band (or frequency band corresponding thereto) corresponding to the light-emitting element may be an effective wavelength band for detecting a liquid-type object (10). According to one example, the light-emitting element may be positioned within the robot (600) at a preset height (e.g., 34.5 mm) from the ground (1), but is not limited thereto.

[0121] According to one example, if the second camera sensor (610) includes a camera sensor corresponding to the left eye and the right eye, respectively, a mirror (620) corresponding to each camera sensor may be placed within the robot (600). For example, a mirror (620) corresponding to the left eye and a mirror (620) corresponding to the right eye may each be placed at a pre-set position within the robot (600). According to one example, if the robot (600) includes a first camera sensor and a second camera sensor (610), respectively, at least three mirrors (620) may be placed within the robot (600), and the robot (600) may identify an image reflected through each mirror (620) and use this to identify a liquid-type object (10) present in the driving space.

[0122] According to one example, when an image corresponding to the driving space is acquired through the second camera sensor (610), the robot (600) can identify a fifth image in which the second region (622) among the acquired images is reflected through the mirror (620). According to one example, in a memory (e.g., memory (140) of FIG. 1a), coordinate information for an image corresponding to the mirror (620) among the images corresponding to the driving space acquired through the second camera sensor (610) may be stored. Based on the information stored in the memory, the robot (600) can identify the fifth image from the image corresponding to the driving space.

[0123] According to one example, the fourth image may be an image taken at a third shooting angle corresponding to the second camera sensor (610). According to one example, the fifth image may be an image corresponding to a fourth shooting angle in which the second area (622) is reflected through the mirror (620). According to one example, the fourth shooting angle may be a shooting angle corresponding to the arrangement angle of the mirror (620). According to one example, the fifth image corresponding to the second area (622) may be included in an image corresponding to the driving space obtained through the second camera sensor (610).

[0124] According to one example, the second area (622) may be an area reflected through the mirror (620) with respect to the second camera sensor (610). For example, the second area (622) may be an area in the driving space corresponding to the reflection range (621) of the light reflected through the mirror (620) among the light incident on the second camera sensor (610).

[0125] According to one embodiment, the operation method may include an operation (S630) of identifying a sixth image corresponding to a second region (622) based on a fourth image.

[0126] According to one example, the robot (600) can identify a sixth image corresponding to a second area (622) among a fourth image corresponding to a first area (612). According to one example, the sixth image may be an image in which the second area (622) is captured at a third shooting angle. According to one example, coordinate information for an image corresponding to the second area (622) among the fourth images may be stored in memory. The robot (600) can identify the sixth image from the fourth image based on the information stored in memory.

[0127] According to one embodiment, the operation method may include an operation (S640) of detecting a liquid-type object (10, e.g., the liquid-type object (10) of FIG. 1b) within a second region (622) based on an identified fifth image and an identified sixth image.

[0128] According to one example, the robot (600) can input the identified fifth image and the identified sixth image into a learned neural network model (e.g., the learned neural network model of FIG. 1a) to detect a liquid-type object (10) within the second area (622). For example, the robot (600) can input the fifth image and the sixth image into a learned neural network model to identify whether an object exists in at least one of the fifth image or the sixth image. Alternatively, for example, the robot (600) can input the fifth image and the sixth image into a learned neural network model to identify location information (e.g., location information of FIG. 1a) of a liquid-type object (10) existing within the second area (622).

[0129] According to one example, the robot (600) may detect a liquid-type object (10) in the driving space based on each of the identified second image, identified third image, identified fifth image, and identified sixth image. According to one example, the robot (600) may classify at least one image containing a liquid-type object (10) by inputting each of the identified second image, identified third image, identified fifth image, and identified sixth image into a learned neural network model (e.g., the learned first neural network model of FIG. 4).

[0130] According to one example, when at least one image containing a liquid-type object (10) is classified, the robot (600) inputs the classified at least one image into a learned neural network model (e.g., the learned second neural network model of FIG. 4) to identify location information of the liquid-type object (10). According to one example, the robot (600) can perform avoidance driving or removal of the object based on the identified location information.

[0131] According to the example described above, the robot (600) can identify a liquid-type object (10) existing within a second area (622) using a second camera sensor (610) and, based on this, perform avoidance driving or removal operations on the object. The robot (600) can detect objects existing on the driving path of the robot (600) more accurately through a plurality of camera sensors.

[0132] FIG. 7a is a flowchart illustrating a method for identifying whether an object exists according to one embodiment. FIG. 7b and FIG. 7c are drawings illustrating a method for identifying whether an object exists according to one embodiment.

[0133] According to FIGS. 7a to 7c, according to one embodiment, the operation method may include an operation (S710) of identifying a seventh image corresponding to a third region among a second image (e.g., the second image (102) of FIG. 1e).

[0134] According to one example, a first arrangement angle (e.g., the first arrangement angle (115) of FIG. 1d) corresponding to a mirror (720, e.g., the mirror (120) of FIG. 1b) may satisfy the following mathematical formula 1.

[0135]

[0136] In the above-described mathematical formula 1, 'A' may be a first shooting angle (e.g., the first shooting angle (114) of FIG. 1d) or a third shooting angle (e.g., the third shooting angle of FIG. 6a). In one example, 'B' may be a first placement angle. In one example, when the mirror (720) is placed within the robot (700, e.g., the robot (100) of FIG. 1a) so that the first placement angle satisfies the above-described mathematical formula 1, the second area (722, e.g., the second area (122) of FIG. 1b) may be included within the first area (712), e.g., the first area (112) of FIG. 1b).

[0137] According to one example, the first region (712) may be a region on the driving space corresponding to a shooting range (711) between a shooting direction and a field of view (713) corresponding to a camera sensor (710, e.g., at least one of the first camera sensor (110) of FIG. 1a or the second camera sensor (610) of FIG. 6b). According to one example, if the camera sensor (710) is implemented as a stereo camera (e.g., the second camera sensor (610) of FIG. 6b), the robot (700) may include a light-emitting element (e.g., the light-emitting element (630) of FIG. 6b).

[0138] According to one example, the second area (722) may be an area reflected through the mirror (720) with respect to the second camera sensor (710). For example, the second area (722) may be an area in the driving space corresponding to the reflection range (721) of the light reflected through the mirror (720) among the light incident on the camera sensor (710).

[0139] According to one example, the second area (722) may include a third area greater than a preset distance from the robot (700). According to one example, the robot (700) may identify a seventh image corresponding to the third area among the second images, among the second areas (722) reflected through the mirror (720).

[0140] According to one example, as the first placement angle corresponding to the mirror (720) increases, the range of the area reflected through the mirror (720) can be relatively wider. According to one example, if the placement angle of the mirror (720) is greater than a preset angle while satisfying Equation 1 (or if the range of the area reflected through the mirror (720) is relatively wide), the robot (700) can divide the area reflected through the mirror (720) into multiple regions and detect a liquid-type object (10) based on an image corresponding to each of the divided multiple regions.

[0141] For example, unlike as illustrated in FIG. 1b, if the first placement angle is greater than a preset angle while satisfying Equation 1, the size of the second area (722) may be relatively larger, as illustrated in FIG. 7b. In one example, if the first placement angle is greater than or equal to a preset angle, the second area (722) may not be included in the first area (712), as illustrated in FIG. 9b below. This will be described later.

[0142] According to one embodiment, the operation method may include an operation (S720) of detecting a liquid-type object (10) within a third region based on an identified seventh image and an identified third image (e.g., the third image (104) of FIG. 1e).

[0143] According to one example, the robot (700) can identify whether a liquid-type object (10) exists within the third area by inputting the seventh image corresponding to the third area and the third image into a learned neural network model (e.g., the learned neural network model of FIG. 1a). According to one example, the robot (700) can perform a correction on the seventh image (e.g., the correction of FIG. 3) and input the corrected seventh image into the learned neural network model. According to one example, the robot (700) can identify whether a liquid-type object (10) exists within the third area by extracting an image corresponding to the third area within the third image and inputting the extracted image together with the seventh image into the learned neural network model.

[0144] According to one example, the robot (700) may perform only the operation of classifying images (e.g., classification in FIG. 4) based on whether there is a liquid-type object (10) in the third area (or, distant area), and may perform the classification operation and the position information identification operation (e.g., position information in FIG. 4) in the remaining area (or, near area) of the second area (722) excluding the third area. Afterwards, if the robot (700) performs a driving operation and it is identified that the distance is less than a preset distance from the third area, the robot (700) may identify the position information of the object existing in the third area and perform a driving operation based on the identified position information.

[0145] According to the example described above, the robot (700) can perform the operation of identifying whether an object exists and the operation of identifying the location of an object for an area within a preset distance from the robot (700), and can perform only the operation of identifying whether an object exists for an area greater than a preset distance from the robot (700). Accordingly, the amount of data processing for object detection by the robot (700) can be reduced, and efficient control of the robot (700) becomes possible.

[0146] FIG. 8 is a flowchart for explaining the operation of a robot related to an object according to one embodiment.

[0147] According to FIG. 8, according to one embodiment, the operation method may include an operation (S810) of identifying a driving path based on the location of the third area when it is identified that an object (e.g., the liquid type object (10) of FIG. 1b) exists within the third area (e.g., the third area of ​​FIG. 7a).

[0148] According to one example, a robot (e.g., the robot (100) of FIG. 1a) can identify whether an object exists within a third area based on a seventh image (e.g., the seventh image of FIG. 7a) and a third image (e.g., the third image (104) of FIG. 1e) corresponding to the third area. According to one example, if the robot identifies that an object exists within the third area, it can identify a driving path based on the location of the third area. For example, the robot can identify a driving path to approach the third area. Or, for example, the robot can identify a driving path to pass through the third area.

[0149] According to one embodiment, the operation method may include an operation (S820) of identifying location information of a liquid-type object (e.g., location information of FIG. 4) when the robot is identified to be less than a preset distance from a third area while driving along an identified driving path.

[0150] According to one example, when a robot is identified to be less than a preset distance from a third area while performing a driving operation, the robot can input a seventh image corresponding to the third area and a third image into a trained neural network model (e.g., the trained neural network model of FIG. 4) to identify the position information of an object.

[0151] According to one embodiment, the operation method may include an operation (S830) of performing evasive driving on a liquid-type object or performing a removal operation on a liquid-type object based on position information.

[0152] According to one example, when the robot identifies the location information of an object existing in a third area, it may perform evasive driving on the object or perform a removal operation on the object based on the location information of the object.

[0153] However, this is not limited thereto, and for example, even if an object exists in a third area, the robot may perform an operation regardless of the object's location if the third area is not included in the robot's existing driving path.

[0154] According to the example described above, the robot can perform the operation of identifying whether an object exists and the operation of identifying the location of an object for an area within a preset distance from the robot, and can perform only the operation of identifying whether an object exists for an area greater than a preset distance from the robot. Accordingly, the amount of data throughput for object detection by the robot can be reduced, and efficient robot control becomes possible.

[0155] FIG. 9a is a flowchart illustrating a method for identifying whether an object exists according to one embodiment. FIG. 9b is a diagram illustrating a method for identifying whether an object exists according to one embodiment.

[0156] According to FIGS. 9a and 9b, according to one embodiment, the operation method may include an operation (S910) of identifying a second image (e.g., a second image (102) of FIG. 1e) reflected through a mirror (920, e.g., a mirror (120) of FIG. 1b) arranged at a second angle, based on an image (e.g., an image (101) corresponding to the driving space of FIG. 1e) obtained through a first camera sensor (910, e.g., a first camera sensor (110) of FIG. 1a)) obtained through a first camera sensor (910, e.g., a first camera sensor (110) of FIG. 1a).

[0157] According to one example, a robot (900, e.g., robot (100) of FIG. 1a) can acquire an image corresponding to a driving space through a first camera sensor (910). According to one example, the robot (900) can identify a second image in which a second region (931) among the images corresponding to the driving space is reflected through a mirror (920).

[0158] However, this is not limited thereto, and according to one example, the robot (900) may acquire an image corresponding to the driving space through a second camera sensor (e.g., the second camera sensor (610) of FIG. 6b) and identify the second image based thereon. However, for the convenience of explanation, the following description will be limited to the case where an image corresponding to the driving space is acquired through a first camera sensor.

[0159] According to one embodiment, the operation method may include an operation (S920) of identifying an eighth image corresponding to a fourth region (932) that is not included in the first region (912, e.g., the first region (112) of FIG. 1b) among the second regions based on the second image.

[0160] According to one example, the first area (912) may be an area on the driving space corresponding to the shooting range (911) between the shooting direction and the field of view (913) corresponding to the first camera sensor (910).

[0161] According to one example, the second area (931) may be an area reflected through the mirror (920) with respect to the first camera sensor (910). For example, the second area (931) may be an area on the driving space corresponding to the reflection range (930) of the light reflected through the mirror (920) among the light incident on the first camera sensor (910).

[0162] According to one example, the second arrangement angle corresponding to the mirror (920, e.g., mirror (120) in FIG. 1b) can satisfy the following mathematical formula 2.

[0163]

[0164] In the above-described mathematical formula 2, 'A' may be a first shooting angle (e.g., the first shooting angle (114) of FIG. 1d) or a third shooting angle (e.g., the third shooting angle of FIG. 6a). In one example, 'C' may be a second placement angle. In one example, when the mirror (920) is placed within the robot (900) such that the second placement angle satisfies the above-described mathematical formula 2, some area (933) of the second area (931) may be included in the first area (912), and the remaining area (932, or a fourth area) may not be included in the first area (912). In one example, the first placement angle (e.g., the first placement angle (115) of FIG. 1d) and the second placement angle may be different.

[0165] According to one example, as the second placement angle corresponding to the mirror (920) becomes smaller, the range of the area reflected through the mirror (920) may become relatively narrower. According to one example, when the second placement angle becomes smaller, a blind area may exist as an area (932) that is outside the field of view (913) of the first camera sensor (910). Alternatively, according to one example, when the second placement angle becomes smaller, only a portion of the area (933) may be included in the first area (912) as described above.

[0166] According to one example, the second area (931) may include a fourth area (932) that is less than a preset distance from the robot (900). According to one example, the robot (900) may identify an eighth image corresponding to the fourth area (932) among the second images among the second areas (931) reflected through the mirror (920). For example, the robot (900) may identify an eighth image corresponding to the fourth area (932) among the second images by using coordinate information corresponding to the fourth area (932). According to one example, the coordinate information corresponding to the fourth area (932) may be stored in a memory (e.g., memory (140) of FIG. 1a).

[0167] According to one embodiment, the operation method may include an operation (S930) of identifying whether an object exists within the fourth region (932) based on the eighth image.

[0168] According to one example, when the robot (900) identifies an 8th image corresponding to a 4th area, it inputs the identified 8th image into a learned neural network model (e.g., at least one of the learned 1st neural network model of FIG. 4 or the learned 3rd neural network model of FIG. 5) to identify whether a liquid-type object (10) exists within the 4th area.

[0169] According to one example, the robot (900) may identify whether a liquid-type object (10) exists within the fourth area, but is not limited thereto, and the robot may also identify location information of a preset type of object (e.g., a preset type of object of FIG. 5) within the fourth area.

[0170] According to one example, the robot (900) may simultaneously perform the operation of identifying whether a liquid-type object (10) exists within the fourth area and the operation of identifying location information of a preset type of object within the fourth area. According to one example, if the robot (900) identifies that a liquid-type object (10) exists within the fourth area, it may identify location information of the liquid-type object (10).

[0171] According to one example, the robot (900) may perform a rotation operation based on whether an object is detected within the fourth area. According to one example, the robot (900) may perform a rotation operation if it is identified that no object exists within the fourth area. For example, one can assume a case where the robot (900) plans to perform a rotation operation in the fourth area to provide a service. The robot (900) detects whether an object exists within the fourth area, and if it is identified that no object exists, it may perform a rotation operation in the fourth area. According to one example, if an object exists within the fourth area, the robot (900) may not perform a rotation operation, but instead perform an avoidance operation against the object or a removal operation against the object.

[0172] Returning to FIG. 2, according to one embodiment, the processor (130) can compare the location of the first object in the second image and the location of the second object in the third image when a first object of liquid type is detected in the second image and a second object of liquid type is detected in the third image.

[0173] According to one example, the processor (130) may input each of the identified second image and the identified third image into a learned neural network model. According to one example, the processor (130) may detect a first object of liquid type in the second image and detect a second object of liquid type in the third image based on the output result.

[0174] According to one example, the processor (130) can identify the location of the first object and the location of the second object, respectively. According to one example, the processor (130) can identify the location of each object within the second region based on the output result of a learned neural network model. According to one example, the processor (130) can compare each location.

[0175] According to one embodiment, the processor (130) can determine whether the first object and the second object are the same object based on the comparison result. According to one example, if the first object and the second object are identified as existing at the same location (or at a location within a preset error range), the processor (130) can determine that the first object and the second object are the same object.

[0176] According to one embodiment, the processor (130) may determine that an object exists within a second area if the first object and the second object are determined to be the same object. Based on the location of the object existing within the second area, the processor (130) may perform evasive driving for the object or perform an operation to remove the object.

[0177] According to the example described above, the robot (100) can detect the presence of an object by using multiple images of the same area. Accordingly, the presence of an object and the location of the object can be accurately determined, thereby improving the performance of the robot (100).

[0178] FIGS. 10a and FIGS. 10b are drawings for explaining the role of a mirror according to one embodiment.

[0179] Referring to FIGS. 10a and FIGS. 10b, according to one embodiment, a robot (e.g., robot (100) of FIG. 1a) may include a camera sensor (1010, e.g., first camera sensor (110) of FIG. 1a or second camera sensor (610) of FIG. 6b).

[0180] According to one example, the camera sensor (1010) can photograph a liquid-type object (10) present within a first shooting range (1011) at a first angle (1012).

[0181] According to one example, the mirror (1020) may be positioned within the robot at a preset positioning angle (e.g., the first positioning angle of FIG. 1b). According to one example, among the light reflected through the mirror (1020), light within the first reflection range (1021) may be incident on the camera sensor (1010) through the mirror (1020).

[0182] According to one example, when a liquid-type object (10) exists in an area corresponding to a first reflection range (1021), the liquid-type object (10) can be photographed at a second angle (1022) through a mirror (1020). According to one example, when light within the first reflection range (1021) is incident on a camera sensor (1010) through the mirror (1020), the image output through the camera sensor (1010) may include an image of the liquid-type object (10) photographed at the second angle (1022).

[0183] According to the example described above, the robot of the present invention acquires images taken at different angles for the same area and can detect an object (10) within the area using the acquired images at different angles. In the case of a liquid-type object (10), the detection of the object (10) may vary depending on the shooting angle. According to the example described above, since the liquid-type object (10) can be captured at different shooting angles, the detection rate of the object (10) can be increased.

[0184] Returning to FIG. 2, according to one embodiment, the first position angle (115) of the mirror (120) may be changed based on context information of the robot (100). According to one example, the context information of the robot (100) may include at least one of the driving speed of the robot (100), the operating mode of the robot (100), information about the region of interest of the robot (100), and information about the driving space. According to one example, the region of interest of the robot (100) may be a second region (122), but is not limited thereto. According to one example, the information about the driving space may include information about the terrain slope corresponding to the driving space.

[0185] According to one example, the processor (130) can identify a first positioning angle (115) of the mirror (120) based on the driving speed of the robot (100). For example, if the driving speed of the robot (100) increases based on a preset event (e.g., an event of driving around an object or an event of driving around), the processor (130) can identify a first positioning angle (115) such that the position of the second area (122) is located at a position relatively farther from the robot (100) compared to the existing position. Or, for example, if the driving speed of the robot (100) decreases, the processor (130) can identify a first positioning angle (115) such that the position of the second area (122) is located at a position relatively closer to the robot (100) compared to the existing position.

[0186] According to one example, the processor (130) may identify a first placement angle (115) of the mirror (120) based on information regarding the operating mode of the robot (100). For example, the processor (130) may identify a first placement angle (115) such that when the robot (100) is operating in a 'high-speed cleaning mode', the position of the second area (122) is located at a position relatively far from the robot (100) compared to the existing position. Or, for example, the processor (130) may identify a first placement angle (115) such that when the robot (100) is operating in a 'high-precision cleaning mode', the position of the second area (122) is located at a position relatively close to the robot (100) compared to the existing position.

[0187] According to one example, the processor (130) may identify a first positioning angle (115) of the mirror (120) based on information about the driving space. For example, the processor (130) may identify a first positioning angle (115) such that when the robot (100) is driving on steep terrain, the position of the second area (122) is located relatively closer to the robot (100) than the existing position. Alternatively, the processor (130) may identify a first positioning angle (115) such that when the robot (100) is driving on a downhill slope, the position of the second area (122) is located relatively further away from the robot (100) than the existing position.

[0188] According to one example, the processor (130) may identify a first placement angle (115) of the mirror (120) based on the output result from the learned neural network model. For example, a case may be assumed where an image corresponding to a specific area within the driving space is input to the learned neural network model. If the processor (130) identifies, based on the output result from the learned neural network model, that it is uncertain whether an object exists in the aforementioned specific area, it may identify a first placement angle (115) for obtaining an additional image of the aforementioned specific area.

[0189] According to one example, the robot (100) may further include a drive unit (e.g., a motor) for changing the positioning angle of the mirror (120). According to one example, when a first positioning angle (115) of the mirror (120) is identified, the robot (100) may control the drive unit so that the mirror (120) is positioned at the identified first positioning angle (115). According to one example, the robot (100) may change the first positioning angle (115) of the mirror (120) in real time based on the context information of the robot (100), even when the robot (100) is driving in a driving space.

[0190] FIG. 11 is a block diagram showing the detailed configuration of a robot according to one embodiment.

[0191] According to FIG. 11, the robot (100') may include at least one sensor (145) comprising a first camera sensor (110), a second camera sensor (150, e.g., the second camera sensor (610) of FIG. 6b), and a third sensor (160), a mirror (120), at least one processor (130), a memory (140), a display (170), a user interface (180), a communication interface (185), a speaker (190), and a microphone (195). A detailed description of configurations shown in FIG. 11 that overlap with configurations shown in FIG. 2 will be omitted.

[0192] At least one sensor (145) may include different types of sensors, including a first camera sensor (110), a second camera sensor (150), and a third sensor (160). At least one sensor (145) may include a plurality of sensors of various types. The third sensor (160) may measure physical quantities or detect the operating state of the robot (100') and convert the measured or detected information into an electrical signal. The third sensor (160) may include a camera, and the camera may include a lens that focuses visible light or other optical signals received by being reflected by an object onto an image sensor, and an image sensor capable of detecting visible light or other optical signals. Here, the image sensor may include a 2D pixel array divided into a plurality of pixels. Alternatively, the third sensor (160) may include a temperature sensor or an infrared sensor.

[0193] The display (170) may be implemented as a display including a self-emissive element or as a display including a non-emissive element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, or a QLED (Quantum dot light-emitting diodes). The display (170) may also include a driving circuit, a backlight unit, etc., which can be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (170) may be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc. The processor (130) can control the display (170) to output an output image obtained according to the various embodiments described above. The output image may be a high-resolution image of 4K or 8K or higher. According to one embodiment, the output image may be a game image.

[0194] According to one embodiment, the display (170) may include a plurality of haptic elements. The haptic elements may be implemented as motors to provide haptic feedback (e.g., vibration feedback) to a user, but are not limited thereto. According to one example, the display (170) may include a predetermined number of haptic elements. For example, the display (170) may include a predetermined number of haptic elements corresponding to a predetermined number of sub-regions of the display, but is not limited thereto, and it is obvious that the display may include a number of haptic elements different from the number of sub-regions corresponding to the display.

[0195] The user interface (180) is configured for the robot (100') to perform interaction with the user. For example, the user interface (180) may include at least one of a touch sensor, a motion sensor, a button, a jog dial, a switch, a microphone, or a speaker, but is not limited thereto.

[0196] The communication interface (185) can input and output various types of data. For example, the communication interface (185) can transmit and receive various types of data to and from an external device (e.g., source device), an external storage medium (e.g., USB memory), an external server (e.g., web hard drive) through communication methods such as AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc.

[0197] According to one example, the communication interface (185) may include a Bluetooth Low Energy (BLE) module. BLE refers to Bluetooth technology capable of transmitting and receiving low-power, low-capacity data in a 2.4 GHz frequency band with a range of about 10 m. However, it is not limited thereto, and the communication interface (185) may include a Wi-Fi communication module. That is, the communication interface (185) may include at least one of a Bluetooth Low Energy (BLE) module or a Wi-Fi communication module.

[0198] According to one embodiment, the speaker (190) may be composed of a tweeter for reproducing high-frequency sound, a midrange for reproducing mid-frequency sound, a woofer for reproducing low-frequency sound, a subwoofer for reproducing ultra-low-frequency sound, an enclosure for controlling resonance, and a crossover network for dividing the frequency of an electrical signal input to the speaker into bands.

[0199] According to one embodiment, the speaker (190) can output an acoustic signal to the outside of the robot (100'). The speaker (190) can output multimedia playback, recording playback, various notification sounds, voice messages, etc. The robot (100') may include an audio output device such as the speaker (190), but may also include an output device such as an audio output terminal. In particular, the speaker (190) can provide acquired information, information processed or produced based on the acquired information, response results to user voice, or operation results, etc., in the form of voice.

[0200] The microphone (195) may refer to a module that acquires sound and converts it into an electrical signal, and may be a condenser microphone, ribbon microphone, moving coil microphone, piezoelectric element microphone, carbon microphone, or MEMS (Micro Electro Mechanical System) microphone. Additionally, it may be implemented in omnidirectional, bidirectional, unidirectional, subcardioid, supercardioid, or hypercardioid modes. According to one embodiment, the robot (100') may include the microphone (195) and an inner microphone, and the microphone (195) may be a microphone located relatively outside the body. According to one example, the robot (100') may acquire an audio signal including external noise through the microphone (195). According to one embodiment, the microphone (195) may be positioned in a direction opposite to the direction in which the speaker (190) emits sound.

[0201] According to the example described above, the robot (100') of the present invention can detect liquid through images taken at multiple shooting angles for a specific area in the driving space, and thus can detect liquid with a high detection rate.

[0202] The methods according to the various embodiments of the present disclosure described above may be implemented in the form of an application that can be installed on an existing robot. Alternatively, the methods according to the various embodiments of the present disclosure described above may be performed using a deep learning-based learned neural network (or deep learned neural network), that is, a learning network model. Furthermore, the methods according to the various embodiments of the present disclosure described above may be implemented solely through a software upgrade or a hardware upgrade of the existing robot. Additionally, the various embodiments of the present disclosure described above may also be performed through an embedded server equipped in the robot or an external server of the robot.

[0203] According to the exemplary embodiments of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include a display device (e.g., a display device (A)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code provided or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0204] Additionally, according to one embodiment, the method according to the various embodiments described above may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed online in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or provided on a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0205] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0206] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In robots, A first camera sensor positioned at a first shooting angle to photograph the driving space of the robot; A mirror positioned at a first position of the above-mentioned robot at a first positioning angle; At least one processor including a processing circuit; and Memory that stores instructions and includes one or more storage media; and When the above instructions are executed individually or collectively by the at least one processor, the robot, Using the first camera sensor, a first image corresponding to a first area within the driving space is obtained, and Based on the first image above, a second image corresponding to a second shooting angle, in which a second area within the driving space is reflected through the mirror, is identified, and Based on the first image obtained above, the second region identifies a third image captured at the first shooting angle, and A robot that identifies a liquid-type object within the second area based on a second image corresponding to the second shooting angle and a third image corresponding to the first shooting angle.

2. In Paragraph 1, The above first shooting angle is, The angle at which the first camera sensor is tilted relative to a line perpendicular to the ground, and The above second shooting angle is, A robot, which is a shooting angle corresponding to an image reflected through a mirror positioned at the first positioning angle.

3. In Paragraph 1, The above-mentioned first region is, Includes the above-mentioned second region, When the above instructions are executed individually or collectively by the at least one processor, the robot, Calibrate the above second image, and Identifying a third image corresponding to an area that matches the second image among the first images corresponding to the first area, and A robot that provides the second image in which the correction is performed and the third image identified to the first learned neural network model to identify whether the liquid-type object exists within the second region.

4. In Paragraph 3, When the above instructions are executed individually or collectively by the at least one processor, the robot, The second image with the correction performed and the third image identified are provided to a trained second neural network model to identify whether the liquid-type object exists within the second region, and If it is identified that the liquid-type object exists within the second region, the corrected second image and the identified third image are provided to a trained third neural network model to identify the location information of the liquid-type object. A robot that performs at least one of avoidance driving or removal operation on the liquid-type object based on the above-mentioned identified location information.

5. In Paragraph 3, When the above instructions are executed individually or collectively by the at least one processor, the robot, The second image with the correction performed and the third image identified are provided to a trained fourth neural network model to identify location information of other objects of a pre-set type existing within the second region, and A robot that performs avoidance driving against other objects of the aforementioned preset type based on the above-mentioned identified location information.

6. In Paragraph 1, The above first position is, A robot that is included in an area corresponding to the field of view of the first camera sensor and is located at a position relatively higher than the first camera sensor.

7. In Paragraph 1, It further includes a second camera sensor positioned at a third shooting angle to photograph the first area; and When the above instructions are executed individually or collectively by the at least one processor, the robot, Using the second camera sensor above, a fourth image corresponding to a first area within the driving space is obtained, and Based on the above fourth image, the second region identifies the fifth image reflected through the mirror, and Based on the fourth image obtained above, identify the sixth image corresponding to the second region, and A robot that identifies the liquid-type object within the second region based on the identified fifth image and the identified sixth image.

8. In Paragraph 7, The first camera sensor above is, It is implemented with an RGB (Red, Green, and Blue) camera, and The second camera sensor above is, It is implemented as a stereo camera including camera sensors corresponding to the left eye and the right eye, respectively, and When the above instructions are executed individually or collectively by the at least one processor, the robot, A robot that identifies the liquid-type object within the driving space based on each of the identified second image, the identified third image, the identified fifth image, and the identified sixth image.

9. In Paragraph 1, The above second region is, It further includes a third area greater than a pre-set distance from the above robot, and When the above instructions are executed individually or collectively by the at least one processor, the robot, Identifying the seventh image corresponding to the third region among the second images above, and A robot that identifies a liquid-type object within the third region based on the identified seventh image and the identified third image.

10. In Paragraph 9, When the above instructions are executed individually or collectively by the at least one processor, the robot, If it is identified that the object exists within the third area, a driving path is identified based on the location of the third area, and While driving along the above-identified driving path, if it is identified that the robot is less than the distance between the above-identified third area and the above-preset distance, the location information of the liquid-type object is identified, and A robot that performs at least one of avoidance driving or removal operation on the liquid-type object based on the above-mentioned identified location information.

11. In Paragraph 1, The above mirror is, It is arranged at a second arrangement angle different from the first arrangement angle above, and When the above instructions are executed individually or collectively by the at least one processor, the robot, Using the first camera sensor, the second image reflected through a mirror positioned at the second positioning angle in the second area within the driving space is identified, and Based on the second image above, identify an eighth image corresponding to a fourth region among the second regions that is not included in the first region, and A robot that identifies whether other objects exist within the fourth area based on the eighth image above.

12. In Paragraph 11, The above-mentioned fourth region is, It is a blind area that extends beyond the field of view of the first camera sensor, and When the above instructions are executed individually or collectively by the at least one processor, the robot, A robot that performs a rotation operation based on whether the object within the fourth area is identified.

13. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the robot, When a first object of liquid type is identified in the second image and a second object of liquid type is identified in the third image, the position of the first object in the second image and the position of the second object in the third image are compared, and A robot that identifies whether the first object and the second object are the same object based on the above comparison.

14. In the method of operation of the robot, An operation of acquiring a first image corresponding to a first area within the driving space of the robot using a first camera sensor positioned at a first shooting angle; Based on the first image above, an operation of identifying a second image corresponding to a second shooting angle, which is reflected through a mirror positioned at a first positioning angle at the first positioning angle of the robot in a second region within the driving space; Based on the first image obtained above, the operation of identifying the third image captured at the first shooting angle in the second region; and A method of operation comprising: identifying a liquid-type object within the second area based on a second image corresponding to the second shooting angle and a third image corresponding to the first shooting angle.

15. A storage medium storing computer-readable instructions, wherein the instructions, when executed by at least one processor of a robot, cause the robot, Based on an image obtained through a first camera sensor positioned at a first shooting angle, a first image corresponding to a first area within the driving space of the robot is obtained, and Based on the first image above, a second image corresponding to a second shooting angle is identified, in which a second area within the driving space is reflected through a mirror positioned at a first positioning angle at the first positioning angle of the robot, and Based on the first image obtained above, the second region identifies a third image captured at the first shooting angle, and A storage medium that causes to identify a liquid-type object within the second area based on a second image corresponding to the second shooting angle and a third image corresponding to the first shooting angle.

Images