Robot and control method therefor

Abstract

A robot and a control method therefor are provided. The robot comprises: a camera; a driving unit; at least one memory for storing instructions; and at least one processor operatively connected to the at least one memory. The at least one processor may, by executing the instructions, recognize at least one user located around the robot on the basis of an image obtained by the camera, identify a first rotation angle and a first rotation direction of the robot on the basis of a position of at least one target area in which the at least one user is located among a plurality of areas within an angle of view of the camera and a first gaze order for the at least one target area, and control the driving unit so that the robot rotates on the basis of the first rotation angle and the first rotation direction.

Description

Robot and its control method

[0001] The present disclosure relates to a robot and a control method thereof. More specifically, the present disclosure relates to a robot and a control method thereof that determines a gaze order for multiple users and sequentially gazes at the multiple users based on the determined gaze order.

[0002] Recent advancements in electronic technology have led to the increasing use of robots in various industries. Object recognition technology has enabled robots to accurately distinguish various objects, while advancements in autonomous driving technology have enabled robots to navigate safely without interfering with human traffic. Even examples like serving robots that deliver food to customers in restaurants or guide robots that guide customers in airports and large supermarkets demonstrate the diversification of the fields and methods in which robots are being utilized.

[0003] As robots are utilized in various fields, there are instances where multiple users utilize them simultaneously. For example, multiple visitors at an exhibition may be guided by robots, or multiple travelers may be guided by robots at an airport.

[0004] According to one aspect of the present disclosure, a robot includes a camera, a driving unit, at least one memory storing instructions, and at least one processor operatively connected to the at least one memory. The at least one processor executes the instructions,

[0005] At least one user located around the robot is recognized based on an image acquired through the camera, and a first rotation angle and a first rotation direction of the robot are identified based on a position of at least one target area in which the at least one user is located among a plurality of areas within the field of view of the camera and a first gaze order for the at least one target area, and the driving unit is controlled to rotate the robot based on the first rotation angle and the first rotation direction, and while the robot is rotating, if a new user is recognized in the image, a second gaze order including a new target area in which the new user is located among a plurality of areas within the field of view of the camera is generated, and a second rotation angle and a second rotation direction of the robot are identified based on the position of the new target area and the second gaze order, and the driving unit is controlled to rotate the robot based on the second rotation angle and the second rotation direction.

[0006] The at least one processor may, by executing the instruction, divide the angle of view of the camera into preset angles to identify the plurality of regions, and if there are multiple target regions, identify a first target region among the plurality of target regions based on the first gaze order, identify a first rotation angle and a first rotation direction of the robot corresponding to the first target region based on the position of the first target region, identify a first rotation angle and a first rotation direction of the robot corresponding to the remaining target regions among the plurality of target regions based on the position of the first target region and the position of the remaining target regions, and control the driving unit so that the robot rotates according to the first gaze order based on the first rotation angle and the first rotation direction.

[0007] The at least one processor may, by executing the instruction, divide the angle of view of the camera of the robot into preset angles to identify a plurality of areas, and when the new user is recognized based on an image acquired through the camera of the rotated robot, identify a rotation angle and a rotation direction of the robot corresponding to the new target area based on a location of the new user among the plurality of areas, and control the driving unit to rotate the robot based on the identified rotation angle and rotation direction.

[0008] The at least one processor, by executing the instruction, can identify a next target area in the next order of the target area that the rotated robot is gazing at based on the second gaze order, if the at least one target area is a plurality of target areas including a non-gaze target area and a new target area, identify a next rotation angle and a next rotation direction of the robot corresponding to the next target area based on a position of the next target area, and identify a remaining rotation angle and a remaining rotation direction of the robot corresponding to the remaining target areas after the next target area based on a position of the next target area and a position of the remaining target areas.

[0009] The at least one processor may, by executing the instruction, identify the second rotation angle and the second rotation direction based on the position of the new target area if the next target area is the new target area, identify a third rotation angle and a third rotation direction of the robot corresponding to the non-gaze target area based on the position of the new target area, and control the driving unit so that the robot rotates based on the third rotation angle and the third rotation direction after the robot gazes at the new target area.

[0010] The at least one processor can, by executing the instruction, identify the second rotation angle and the second rotation direction based on the position of the new target area and the position of the non-gaze target area before the new target area if the next target area is the non-gaze target area, and control the driving unit so that the robot rotates based on the second rotation angle and the second rotation direction after the robot gazes at the non-gaze target area before the new target area.

[0011] The at least one processor may, by executing the instruction, maintain the first gaze order without including the new target area in the first gaze order if the rotation angle of the robot for gazing at the new target area is greater than or equal to a preset threshold angle.

[0012] The at least one processor can identify a plurality of first users located around the robot based on an image acquired through the camera by executing the instruction, and identify a plurality of users of the same user group among the plurality of first users based on at least one of a distance between the plurality of first users, a time when the plurality of first users were located within the field of view of the camera, and whether a conversation is taking place between the plurality of first users.

[0013] The robot may further include a microphone for receiving a user's voice. The at least one processor may execute the instructions so that, if the at least one user is a plurality of users and the at least one target area is a plurality of target areas, the processor may identify gaze times of the plurality of users toward the robot based on images acquired through the camera, identify the number of voice inputs of the plurality of users toward the robot based on user voices acquired through the microphone, calculate interaction scores of the plurality of users based on the identified gaze times and number of voice inputs, identify interaction scores for each area corresponding to each target area based on the interaction scores of each of the at least one user located in each target area, and determine a first gaze order for the plurality of target areas based on the interaction scores for each area corresponding to each target area.

[0014] The at least one processor can determine a gaze time for each target area based on an interaction score for each area corresponding to each target area by executing the instruction, and control the driving unit so that the robot gazes at each target area during the gaze time for each area.

[0015] According to one aspect of the present disclosure, a method for controlling a robot may include: recognizing at least one user located around the robot based on an image acquired through a camera; identifying a first rotation angle and a first rotation direction of the robot based on a position of at least one target area in which the at least one user is located among a plurality of areas within a field of view of the camera and a first gaze order for the at least one target area; controlling the driving unit to rotate the robot based on the first rotation angle and the first rotation direction; generating a second gaze order including a new target area in which the new user is located among a plurality of areas within a field of view of the camera when a new user is recognized in the image while the robot is rotating; identifying a second rotation angle and a second rotation direction of the robot based on the position of the new target area and the second gaze order; and controlling the driving unit to rotate the robot based on the second rotation angle and the second rotation direction.

[0016] The step of identifying the first rotation angle and the first rotation direction may include the step of dividing the angle of view of the camera into preset angles to identify the plurality of areas, the step of identifying a first target area among the plurality of target areas based on the first gaze order when the at least one target area is plural, the step of identifying a first rotation angle and a first rotation direction of the robot corresponding to the first target area based on the position of the first target area, and the step of identifying a first rotation angle and a first rotation direction of the robot corresponding to the remaining target areas among the plurality of target areas based on the position of the first target area and the position of the remaining target areas.

[0017] The step of identifying the second rotation angle and the second rotation direction may include a step of dividing the angle of view of the camera into preset angles to identify a plurality of regions. The at least one target region and the new target region may be included in the plurality of regions.

[0018] The step of identifying the first rotation angle and the first rotation direction may further include, if the at least one target area is a plurality of target areas including a non-gaze target area and a new target area, a step of identifying a next target area in the next order of target areas that the robot is gazing at based on the second gaze order, a step of identifying a next rotation angle and a next rotation direction of the robot corresponding to the next target area based on a position of the next target area, and a step of identifying the remaining rotation angles and the remaining rotation directions of the robot corresponding to the remaining target areas after the next target area based on the position of the next target area and the position of the remaining target areas.

[0019] The step of identifying the next rotation angle and the next rotation direction may include a step of identifying the second rotation angle and the second rotation direction based on a position of the new target area if the next target area is the new target area. The step of identifying the remaining rotation angle and the remaining rotation direction may include a step of identifying a third rotation angle and a third rotation direction of the robot corresponding to the non-attended target area based on a position of the new target area. The step of controlling the driving unit to rotate the robot based on the third rotation angle and the third rotation direction may include a step of controlling the driving unit to rotate the robot based on the third rotation angle and the third rotation direction after the robot gazes at the new target area.

[0020] The control method may further include a step of identifying the second rotation angle and the second rotation direction based on the position of the new target area and the position of the non-gaze target area before the new target area, if the next target area is the non-gaze target area, and a step of controlling the driving unit so that the robot rotates based on the second rotation angle and the second rotation direction after the robot gazes at the non-gaze target area before the new target area.

[0021] The above control method may include a step of maintaining the first gaze order without including the new target area in the first gaze order if the rotation angle of the robot for gazing at the new target area is greater than or equal to a preset threshold angle.

[0022] The control method may include a step of identifying a plurality of first users located around the robot based on an image acquired through the camera, and a step of identifying a plurality of users of the same user group among the plurality of first users based on at least one of a distance between the plurality of first users, a time when the plurality of first users were located within the field of view of the camera, and whether or not there is a conversation between the plurality of first users.

[0023] The control method may include, if the at least one user is a plurality of users and the at least one target area is a plurality of target areas, a step of identifying a gaze time of the plurality of users toward the robot based on an image acquired through the camera, a step of identifying a number of voice inputs of the plurality of users toward the robot based on user voices acquired through a microphone, a step of calculating an interaction score of the plurality of users based on the identified gaze time and number of voice inputs, a step of identifying an interaction score for each area corresponding to each target area based on the interaction score of each of the at least one users located in each target area, and a step of determining a first gaze order for the plurality of target areas based on the interaction score for each area corresponding to each target area.

[0024] The control method may include a step of determining a gaze time for each target area based on an interaction score for each area corresponding to each target area, and a step of controlling the driving unit so that the robot gazes at each target area during the gaze time for each area.

[0025] Other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the accompanying drawings and the description below.

[0026] Figure 1 is an example of a robot that responds to multiple users.

[0027] FIG. 2 is a block diagram illustrating a configuration of a robot according to one or more embodiments of the present disclosure.

[0028] FIG. 3 is a flowchart of a method for controlling a robot according to one or more embodiments of the present disclosure.

[0029] FIG. 4 is an exemplary diagram illustrating a method for identifying multiple users included in the same user group according to one or more embodiments of the present disclosure.

[0030] FIG. 5 is an exemplary diagram illustrating identification of a target area among multiple areas within a field of view of a camera according to one or more embodiments of the present disclosure.

[0031] FIG. 6 is a flowchart illustrating a method for determining a gaze order based on an interaction score according to one or more embodiments of the present disclosure.

[0032] FIG. 7A is an exemplary diagram illustrating a method for determining a gaze order based on an interaction score according to one or more embodiments of the present disclosure.

[0033] FIG. 7b is an exemplary diagram illustrating a method for determining a gaze order based on an interaction score according to one or more embodiments of the present disclosure.

[0034] FIG. 8 is a flowchart illustrating a control method for identifying rotational directions and rotational angles for multiple target areas according to a gaze order according to one or more embodiments of the present disclosure.

[0035] FIG. 9 is an exemplary diagram illustrating identification of rotational directions and rotational angles for multiple target areas according to a gaze order according to one or more embodiments of the present disclosure.

[0036] FIG. 10 is an exemplary diagram illustrating identification of rotational directions and rotational angles for multiple target areas according to a gaze order according to one or more embodiments of the present disclosure.

[0037] FIG. 11 is an exemplary diagram illustrating recognizing a new area in which a new user is located when a new user is recognized according to one or more embodiments of the present disclosure.

[0038] FIG. 12 is a flowchart illustrating a control method for re-identifying rotation angles and rotation directions for multiple target areas, according to one or more embodiments, when a new user is recognized.

[0039] FIG. 13 is an exemplary diagram showing that a new target area is not included in the gaze order when the rotation angle of the robot for gaze at a new target area according to one or more embodiments of the present disclosure is greater than a preset threshold angle.

[0040] FIG. 14 is an exemplary diagram illustrating re-identifying the rotational direction and rotational angle of a robot based on the location of a new target area according to one or more embodiments of the present disclosure.

[0041] FIG. 15 is an exemplary diagram illustrating re-identifying the rotational direction and rotational angle of a robot based on the location of a new target area according to one or more embodiments of the present disclosure.

[0042] FIG. 16 is an exemplary diagram illustrating re-identifying the rotational direction and rotational angle of a robot based on the location of a new target area according to one or more embodiments of the present disclosure.

[0043] FIG. 17 is a detailed schematic diagram of a robot according to one or more embodiments of the present disclosure.

[0044] The present embodiments may be modified and have various embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0045] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0046] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0047] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0048] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0049] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0050] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0051] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0052] When it is said that a component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it is understood that no other component (e.g., a third component) exists between said component and said other component.

[0053] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0054] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0055] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0056] The various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacings drawn in the attached drawings.

[0057] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0058] Figure 1 is an example of a robot that responds to multiple users.

[0059] Referring to FIG. 1, as robots (100) are being utilized in various fields recently, there are cases where the robot (100) must respond to multiple users (200-1, 200-2, and 200-3, hereinafter referred to as 200). Here, the robot (100) responding to multiple users (200) may mean simultaneously interacting with multiple users (200). For example, this may include a robot (100) that guides multiple visitors at an exhibition or fair, or a robot (100) that provides directions or information to multiple visitors at an airport or large mart.

[0060] At this time, the existing robot (100) interacted with a specific user among multiple users (200) or interacted with multiple users (200) randomly. Accordingly, in most cases, multiple users (200) did not feel that they were interacting appropriately with the robot (100). In particular, when multiple users (200) requested that the robot (100) provide specific information, if the robot (100) interacts only with a specific user among the multiple users (200), the information provided by the robot (100) is not properly conveyed to the remaining users except for the specific user, which causes a problem.

[0061] In addition, the existing robot (100) did not include a specific user among multiple users (200) who was not recognized by the robot (100) as a target of interaction even if the specific user was recognized later. In this case, even though the specific user recognized later is included in the same user group (e.g., the same group of visitors or spectators, etc.) as the multiple users (200), the specific user is disadvantaged in that he / she is excluded from the target of interaction of the robot (100) simply because he / she was recognized later than other users. In other words, the specific user recognized later cannot interact with the robot (100) due to the problem.

[0062] The present disclosure is intended to address these challenges. When multiple users (200) utilize a robot (100), the robot (100) determines an appropriate interaction order so that it can interact evenly with the multiple users (200), and sequentially interacts with the multiple users (200) according to the gaze order. Furthermore, even when a new user is recognized, the robot (100) does not omit the new user, but interacts with the new user along with the existing multiple users (200). Hereinafter, embodiments of the present disclosure related to this will be described in detail.

[0063] FIG. 2 is a block diagram illustrating a configuration of a robot according to one or more embodiments of the present disclosure.

[0064] Referring to FIG. 2, the robot (100) includes a camera (110), a driving unit (120), and one or more processors (130).

[0065] A robot (100) according to one or more embodiments of the present disclosure may be implemented as a variety of electronic devices that provide various services to a user while performing interactions with the user. For example, the robot (100) may be a guide robot (100) located in an airport or a supermarket, etc., that, upon receiving input (e.g., voice input, touch input, etc.) from a user, provides the user with specific information or provides various services, such as providing directions, in response to the received input.

[0066] According to one or more embodiments of the present disclosure, the robot (100) may be divided into a head and a body. As one or more examples, the head of the robot (100) may include a camera (110) or a sensor, and may perform a function of acquiring images of objects around the robot (100) or detecting objects. In addition, the head of the robot (100) may include a display, and the robot (100) may display various image information through the display. In particular, the robot (100) may provide a vivid interaction experience to a user using the robot (100) by displaying graphic objects representing eyes, a nose, and a mouth through the display included in the head while performing an interaction with a user.

[0067] The body of the robot (100) can be connected to a driving unit (120) to move the robot (100) or perform the function of a frame that supports the robot (100). In addition, the body of the robot (100) can include a display, and the robot (100) can display various image information through the display. In particular, the robot (100) can display information requested by a user or required by the user (e.g., UI, etc.) through the display, and can also receive user commands from the user through a touch panel included in the display.

[0068] The camera (110) captures images of objects around the robot (100) to obtain multiple images of the objects. Specifically, the camera (110) may also obtain images of objects (e.g., people, animals, other objects, etc.) existing around the robot (100).

[0069] For this purpose, the camera (110) may be implemented with an imaging element such as an imaging element having a CMOS structure (CIS, CMOS Image Sensor) or an imaging element having a CCD structure (Charge Coupled Device). However, the present invention is not limited thereto, and the camera (110) may be implemented with a camera (110) module having various resolutions capable of photographing a subject.

[0070] Additionally, the camera (110) may be implemented as a depth camera (110), a stereo camera (110), or an RGB camera (110). Through this, the camera (110) can obtain depth information of the object together with an image of the object.

[0071] The driving unit (120) drives at least a part of the robot (100). As one or more examples, the driving unit (120) may be implemented as a rotatable motor connecting the head and the body of the robot (100). Accordingly, the driving unit (120) rotates the head connected to the driving unit (120) by 360° so that the robot (100) can obtain information about objects around the robot (100) through the camera (110). However, the present invention is not limited thereto, and the robot (100) may be implemented as an integrated body in which the head and the body are not distinguished, and in this case, the body and the head may be distinguished on the robot (100) according to the functions each performs.

[0072] In addition, as one or more examples, the driving unit (120) may be provided at the lower part of the body of the robot (100) to move the robot (100). That is, the driving unit (120) may be a device that can drive the robot (100). At this time, the driving unit (120) may adjust the driving direction and driving speed according to the control of the processor. To this end, the driving unit (120) may include a power generation device that generates power for driving the robot (100) (e.g., a gasoline engine, a diesel engine, an LPG (liquefied petroleum gas) engine, an electric motor, etc. depending on the fuel (or energy source) used), and a steering device for controlling the driving direction (e.g., manual steering, hydraulics steering, electronic control power steering (EPS), etc.).

[0073] Additionally, the driving unit (120) may include a plurality of wheels. The plurality of wheels may be rotated based on the control of the processor, thereby moving the robot (100). Alternatively, the plurality of wheels may be rotated at different rotational speeds based on the control of the processor (or only some of the plurality of wheels may be rotated), thereby rotating the robot (100).

[0074] One or more processors are electrically connected to the camera (110) and the driving unit (120) to control the overall operation and function of the robot (100).

[0075] The one or more processors (130) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (130) may control one or any combination of other components of the robot (100) and may perform operations related to communication or data processing. The one or more processors (130) may execute one or more programs or instructions stored in a memory. For example, the one or more processors (130) may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0076] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-dedicated processor).

[0077] One or more processors (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (130) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal memory of the processor (130), such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0078] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0079] In one or more embodiments of the present disclosure, the processor (130) may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0080] Hereinafter, for convenience of explanation, one or more processors (130) will be referred to as processors (130).

[0081] FIG. 3 is a flowchart of a method for controlling a robot according to one or more embodiments of the present disclosure.

[0082] Referring to FIG. 3, the processor (130) recognizes at least one user (200) located around the robot (100) based on an image acquired through a camera (S310).

[0083] Specifically, the processor (130) can acquire multiple images of objects, people, animals, etc. around the robot (100) through the camera. Then, the processor (130) can extract feature information about at least one user (200) included in the images acquired through the camera, and recognize at least one user (200) by matching the extracted feature information with at least one user (200) in the images. Then, the robot (100) can track and recognize at least one user (200) within the multiple images based on the feature information matched to at least one user (200).

[0084] At this time, the processor (130) can select and recognize as users (200) only those people who interact with the robot (100) or who wish to interact with the robot (100) among the plurality of people around the robot (100). In other words, the processor (130) can select and recognize as users (200) those people who are located around the robot (100) for the purpose of using the robot (100) among the plurality of people. For example, the processor (130) may not recognize as users (200) those people who are simply located around the robot (100) or who pass by the robot (100), and may recognize as users (200) those people who are located around the robot (100) for the purpose of using the robot (100).

[0085] To this end, the processor (130) may identify the distances between a plurality of people and the robot (100) based on the acquired images, and select at least one user (200) from among the plurality of people based on the identified distances. For example, the processor (130) may identify the distances between a plurality of people and the robot (100) based on depth information included in the acquired images, and may distinguish and recognize as users (200) only those people whose identified distances are within a preset distance among the plurality of people. At this time, the processor (130) may identify as users (200) those people who stay within a preset distance for a preset time and are identified as gazing at the robot (100). The processor (130) may extract feature information about the users (200) from the acquired images, and may continuously track and recognize the users (200) included in the plurality of images based on the extracted feature information.

[0086] In addition, the processor (130) can select and identify at least one user (200) among a plurality of people based on whether or not each person interacts with the robot (100). Here, the interaction may include whether or not the robot (100) gazes (or gaze time), whether or not a voice command is input (or input count) to the robot (100), whether or not a touch is input (and input count), a change in the distance between the robot (100) and the person, etc. For example, the processor (130) can extract feature information about a plurality of people included in an image, and recognize the eyes of the plurality of people based on the extracted feature information, thereby identifying whether or not each person gazes at the robot (100). In addition, the processor (130) can distinguish and identify at least one user (200) who wants to use the robot (100) among the plurality of people based on whether or not each person gazes at the robot (100) and the gaze time.

[0087] According to one or more embodiments of the present disclosure, the processor (130) may select and recognize only the plurality of users (200) included in the same user group among the plurality of users (200). Here, the user group may be a group including the plurality of users (200) located around the robot (100) and having the same purpose of use for the robot (100). The processor (130) may control the robot (100) so that the robot (100) looks only at the plurality of users (200) included in the same user group.

[0088] To this end, the processor (130) can identify whether a plurality of users (200) around the robot (100) are included in the same user group. As one or more examples, the processor (130) can identify a plurality of users (200) included in the same user group based on whether there is interaction between the plurality of users (200), the distance between the plurality of users (200), the time difference at which the plurality of users (200) are recognized, etc. based on images acquired through the camera.

[0089] FIG. 4 is an exemplary diagram illustrating a method for identifying multiple users included in the same user group according to one or more embodiments of the present disclosure.

[0090] According to one or more embodiments of the present disclosure, the processor (130) may identify a plurality of users (200) located around the robot (100) based on images acquired through the camera (110). In addition, the processor (130) may identify a plurality of users (200) included in the same user group among the plurality of users (200) based on at least one of the distance between the plurality of users (200), the time at which the plurality of users (200) are located within the field of view (10) of the camera (110), and whether the plurality of users (200) are in conversation.

[0091] Specifically, the processor (130) can identify a plurality of users (200) located around the robot (100) based on an image acquired through the camera (110). In particular, the processor (130) can identify a plurality of users (200) among a plurality of people located around the robot (100) based on an image acquired through the camera (110). As one or more examples, the processor (130) can identify a distance between the robot (100) and each person based on depth information for each person included in the image. In addition, the processor (130) can identify only a plurality of people located within a preset distance among the plurality of people as the plurality of users (200).

[0092] Referring to FIG. 4, the processor (130) can identify a plurality of people (200-1 to 200-8) located within 2 m from the robot (100) within the field of view (10) (150°) of the camera (110) as a plurality of users (200). Accordingly, the processor (130) can identify only 6 people (200-1 to 200-6) located within 2 m from the robot (100) within the field of view (10) (150°) of the camera (110) among 8 people (200-1 to 200-8) around the robot (100) as a plurality of users (200). In particular, the processor (130) may not identify a person (200-7) located more than 2 m away from the robot (100) as a user. Additionally, the processor (130) may not identify a person (200-8) located outside the field of view (10) (150°) of the camera (110) as a user. In this case, the person (200-9) located outside the field of view (10) (150°) of the camera (110) is not included in the image acquired through the camera (110), so the processor (130) may not recognize the person located outside the field of view (10) (150°) of the camera (110).

[0093] The processor (130) can identify multiple users (200) and then select multiple users (200) that are included in the same user group from among the multiple users (200). In particular, the processor (130) can identify multiple users (200) that are included in the same user group from among the multiple users (200) based on at least one of the distance between the multiple users (200), the time at which the multiple users (200) are located within the field of view (10) of the camera (110), and whether or not there is a conversation between the multiple users (200).

[0094] Specifically, the processor (130) can identify the distance between the identified plurality of users (200). Specifically, the processor (130) can identify the distance between each user in the image acquired through the camera (110). Through this, the processor (130) can identify the distance between the plurality of users (200). In addition, the processor (130) can identify only the plurality of users (200) identified as having a distance (or interval) between the plurality of users (200) within a preset distance (or interval) as being included in the same user group. For example, referring to FIG. 4, the processor (130) can identify only the plurality of users (200) identified as having a distance between the plurality of users (200) (or adjacent users) within 30 cm among the six identified users (200-1 to 200-6) as being included in the same user group.

[0095] At this time, the processor (130) may identify whether the distance between the remaining users is within a preset distance (or interval) based on a specific user among the plurality of users (200), and may include the specific user and at least one user within the preset distance from the specific user in the user group. Here, the specific user may be the user who is first located within the field of view (10) of the camera (110) and the preset distance (the preset distance from the robot (100)) among the plurality of users (200).

[0096] In addition, the processor (130) can identify the time at which the plurality of users (200) are located within the field of view (10) of the camera (110). Specifically, the processor (130) can identify the time at which each user starts to be included in the plurality of images acquired through the camera (110). Through this, the processor (130) can identify the time at which each user is located within the field of view (10) of the camera (110). In particular, the processor (130) can identify the time at which each user is located not only within the field of view (10) of the camera (110) but also within a preset distance. This can be identified based on the depth information included in the image as described above. As such, the processor (130) can identify the time difference between the plurality of users (200) located within the field of view (10) of the camera (110) (and the preset distance). And, the processor (130) can identify only the plurality of users (200) for which the time difference between the plurality of users (200) is within a preset time as being included in the same user group. For example, referring to FIG. 4, the processor (130) can identify the times at which the six identified users (200-1 to 200-6) are located within the field of view (10) (and the preset distance) of the camera (110). And, the processor (130) can identify only the plurality of users (200) for which the time difference between the plurality of users (200) (or adjacent users) within the field of view (10) (and the preset distance) of the camera (110) is 1 second or less as being included in the same user group.

[0097] At this time, the processor (130) can identify the time when the remaining users are located within the field of view (10) (and the preset distance) of the camera (110) based on the time when a specific user among the plurality of users (200) is located within the field of view (10) (and the preset distance) of the camera (110). In addition, the processor (130) can identify at least one user located within the field of view (10) (and the preset distance) of the camera (110) within a preset time (for example, 3 seconds) based on the time when the specific user is located within the field of view (10) (and the preset distance) of the camera (110). In addition, the processor (130) can include the at least one identified user and the specific user in a user group. Here, the specific user may be the user who is located within the field of view (10) of the camera (110) and the preset distance (the preset distance from the robot (100)) for the first time among the plurality of users (200).

[0098] In addition, the processor (130) can select multiple users (200) belonging to the same user group from among the multiple users (200) based on whether or not a conversation (or gaze) takes place between the multiple users (200). Specifically, the processor (130) can recognize the face of each user or directly detect the mouth and eyes of the user based on multiple images acquired through the camera (110). For example, the processor (130) can detect the eyes and mouth of a person in an image using the AdaBoost algorithm, the HOG (Histogram of Oriented Gradients) algorithm, the Haar Cascade algorithm, etc. Meanwhile, the processor (130) can track the mouths and eyes of the detected users to identify the object that each user is gazing at or conversing with. Through this, the processor (130) can identify whether or not a conversation (or gaze) takes place between the multiple users (200). In addition, the processor (130) can recognize the voice of each user input through the microphone of the robot (100), match the voice of each user with the user, and then identify whether a conversation is taking place between multiple users (200). To this end, the processor (130) can utilize an HMM (Hidden Markov Models) algorithm, an LSTM (Long Short-Term Memory) model, a CNN model, a Transformer model, etc.

[0099] The processor (130) can identify multiple users (200) having a conversation as the same user group. For example, referring to FIG. 4, the processor (130) can identify only the multiple users (200) having a conversation among the six identified users (200-1 to 200-6) as the same user group.

[0100] The processor (130) can select multiple users (200) that are included in the same user group among the multiple users (200) based on the distance between the multiple users (200), the time at which the multiple users (200) are located within the field of view (10) of the camera (110), and whether or not there is a conversation between the multiple users (200).

[0101] Specifically, the processor (130) can calculate a probability value that each user is included in the same user group based on the distance between the plurality of users (200), the time that the plurality of users (200) are located within the field of view (10) of the camera (110), and whether or not there is a conversation between the plurality of users (200).

[0102] As one or more examples, the processor (130) may calculate, for each user, a first probability value of being included in the same group based on the distance between the plurality of users (200). At this time, the processor (130) may calculate a higher first probability value of being included in the same user group as the distance between the plurality of users (200) decreases. In particular, the processor (130) may calculate a higher first probability value of being included in the same user group as the specific user as the distance between the users decreases.

[0103] In addition, the processor (130) can calculate a second probability value for each user that the plurality of users (200) are included in the same group based on the time at which the plurality of users (200) are located within the field of view (10) of the camera (110). At this time, the processor (130) can calculate a second probability value for the plurality of users (200) to be included in the user group as a higher value as the time difference between the time at which the plurality of users (200) are located within the field of view (10) of the camera (110) is smaller. In particular, the processor (130) can calculate a second probability value as a high value for a user that has a short time difference from the time at which the specific user described above is located within the field of view (10) of the camera (110).

[0104] In addition, the processor (130) may calculate a third probability value for each user included in the same group based on whether or not a conversation is taking place between multiple users (200). At this time, the processor (130) may calculate a third probability value for whether or not a conversation is taking place between multiple users (200). That is, the processor (130) may identify whether or not multiple users (200) are conversing based on the acquired image, and calculate a third probability value for each user to conduct a conversation.

[0105] And, the processor (130) can obtain the probability value of inclusion in the same user group of each user based on the calculated first, second, and third probability values. As one or more examples, the processor (130) can obtain the probability value of inclusion in the same user group of each user by applying weights to the first, second, and third probability values, respectively. In particular, the processor (130) can apply a higher weight to the third probability value than to the first and second probability values. That is, the processor (130) can identify that multiple users (200) conducting a conversation are likely to be included in the same user group. And, the processor (130) can obtain the probability value of inclusion in the same user group of each user by adding up the first, second, and third probability values ​​to which different weights are applied. And, the processor (130) can identify only multiple users (200) whose obtained probability values ​​are greater than or equal to a preset value as the same user group.

[0106] For example, referring to FIG. 4, the processor (130) can obtain a probability value for inclusion in the same user group for each of the six identified users (200-1 to 200-6), based on the distance between the six users (200-1 to 200-6), the time at which the six users (200-1 to 200-6) were located within the field of view (10) of the camera (110), and whether or not the six users (200-1 to 200-6) conversed with each other. At this time, when the preset value is 0.5, the processor (130) can identify only four users (200-1 to 200-4) among the six users (200-1 to 200-6) having a probability value of 0.5 or higher as being included in the same user group.

[0107] In the following, it is assumed that the robot (100) interacts with four users identified as being included in the same user group.

[0108] Referring again to FIG. 3, the processor (130) can identify the rotation angle and rotation direction of the robot (100) based on the position of at least one target area (30) in which at least one user (200) is located among a plurality of areas (20) within the camera's field of view (10) and the gaze order for the at least one target area (30) (S320).

[0109] Specifically, the processor (130) can identify a plurality of areas (20) within the camera's field of view (10). For example, the processor (130) can set a preset range within the camera's field of view (10) and identify the preset range by dividing it into a plurality of areas (20) of preset sizes.

[0110] As one or more examples, the processor (130) can identify a plurality of areas (20) by dividing the camera's field of view (10) into preset angles. Here, the preset angles can be preset according to the performance of the camera's field of view (10). Alternatively, the preset angles can be set by the processor (130) according to the number of users (200) identified within the camera's field of view (10). For example, the processor (130) can reduce the size of the preset angle as the number of users (200) identified within the field of view (10) decreases.

[0111] In addition, the processor (130) can identify a plurality of areas (20) within a preset distance. Accordingly, the processor (130) can identify a plurality of areas (20) having preset angles and sizes. That is, the plurality of areas (20) have the same size and can be identified within the camera's field of view (10) in a non-overlapping form. The preset distance can be identified based on depth information included in an image acquired through the camera or distance information acquired through a sensor of the robot (100).

[0112] In addition, the processor (130) can identify a plurality of areas (20) within the camera's field of view (10) and then identify at least one target area (30) where at least one user (200) is located among the plurality of areas (20). Here, the target area (30) is an area where at least one user (200) is located among the plurality of areas (20), and can then become an area at which the robot (100) will gaze by rotating the robot (100).

[0113] The robot (100) looking at the target area (30) may mean that the robot (100) or a part of the robot (100) is directed toward the target area (30). For example, when the head of the robot (100) rotates toward the target area (30) as the processor (130) controls a driving unit (e.g., a motor, etc.) provided between the head and body of the robot (100), this may correspond to the robot (100) looking at the target area (30). Alternatively, when the body of the robot (100) rotates toward the target area (30) as the processor (130) controls a driving unit (e.g., a wheel, etc.) provided on the lower side of the body of the robot (100), this may correspond to the robot (100) looking at the target area (30). In this way, by having the robot (100) or a part of the robot (100) look at the target area (30) under the control of the processor (130), a plurality of users (200) located in the target area (30) can feel that the robot (100) is actively interacting with the users (200).

[0114] FIG. 5 is an exemplary diagram illustrating identification of a target area among multiple areas within a field of view of a camera according to one or more embodiments of the present disclosure.

[0115] Hereinafter, for the convenience of explanation of the present disclosure, it is assumed that there are multiple users (200) and multiple target areas (30).

[0116] Referring to FIG. 5, the processor (130) identifies five areas by dividing a 150° field of view (10) into 30° angles within a distance of 2 m. Then, the processor (130) can identify a first area where one user (200) is located, a second area where two users (200) are located, and a fifth area where one user (200) is located among the five areas as target areas (30). Accordingly, the processor (130) can rotate the robot (100) to identify the first, second, and fifth areas as areas to be gazed at by the robot (100).

[0117] After identifying the target area (30), the processor (130) can identify the rotation angle and rotation direction of the robot (100) based on the positions of the plurality of target areas (30) where the plurality of users (200) are located and the gaze order for the plurality of target areas (30).

[0118] Specifically, the processor (130) can identify the rotation angle and rotation direction of the robot (100) based on the position of the target area (30) within the field of view (10). Since the plurality of areas (20) are identified within the field of view (10) in a form that does not overlap with each other as described above, the processor (130) can identify the position of the target area (30) identified among the plurality of areas (20) within the field of view (10) of the camera. In addition, the processor (130) can identify the rotation direction of the robot (100) based on the position of the identified target area (30). Since the plurality of areas (20) are divided into the same preset angle, the processor (130) can also identify the rotation angle at which the robot (100) should be rotated in order for the robot (100) to gaze at (or face) the target area (30).

[0119] Referring back to FIG. 5, the processor (130) can identify the rotation direction of the robot (100) with respect to the first and second regions as being leftward since the first and second regions among the plurality of target regions (30) (i.e., the first, second, and fifth target regions) are identified as being located on the left side of the robot (100). In addition, the processor (130) can identify the rotation angle of the robot (100) with respect to the first region as 60°, and the rotation angle of the robot (100) with respect to the second region as 30°.

[0120] The processor (130) can identify the rotation direction of the robot (100) with respect to the fifth region as being in the right direction since the fifth region among the plurality of target regions (30) (i.e., the first, second, and fifth target regions) is identified as being located on the right side of the robot (100). In addition, the processor (130) can identify the rotation angle of the robot (100) with respect to the first region as 60°.

[0121] However, when multiple target areas (30) are identified, the processor (130) can identify the rotation direction and rotation angle of the robot (100) for each target area (30) by considering the positions and gaze order of the multiple target areas (30). Here, the gaze order may be the order in which the robot (100) gazes at the multiple target areas (30).

[0122] As one or more examples, the gaze order may be sequentially set for a plurality of target areas (30) based on a preset rotation direction. For example, referring to FIG. 5, if the preset rotation direction is clockwise, the gaze order for the plurality of target areas (i.e., the first, second, and fifth areas) may be in the order of the first area, the second area, and the fifth area.

[0123] As one or more examples, the gaze order may be set based on the distance between the user (200) located in the target area (30) and the robot (100). Specifically, the processor (130) may identify the distance between at least one user (200) located in each target area (30) and the robot (100) based on the location of the user (200). Here, the location of the user (200) may be identified based on depth information included in the acquired image or distance information acquired through a sensor.

[0124] And, the processor (130) can identify distance information corresponding to the target area (30) based on the distance between at least one identified user (200) and the robot (100). And, the processor (130) can determine the gaze order for the target area (30) in order of the distance between the user (200) and the robot (100) based on the identified distance information. At this time, when multiple users (200) are located in the target area (30), the processor (130) can identify the distance information of the target area (30) as the distance of the user (200) that is closer to the robot (100) among the multiple users (200).

[0125] Additionally, one or more examples of the gaze order may be set based on the interaction score of a user (200) located in the target area (30). Here, the interaction score may be a score calculated based on the frequency, time, etc. of the interaction performed by the user (200) with respect to the robot (100). Hereinafter, one or more embodiments of the present disclosure that determine the gaze order based on the interaction of each user (200) will be described.

[0126] FIG. 6 is a flowchart illustrating a method for determining a gaze order based on an interaction score according to one or more embodiments of the present disclosure. FIGS. 7A and 7B are exemplary diagrams illustrating a method for determining a gaze order based on an interaction score according to one embodiment of the present disclosure. Steps S605 and S635 to S655 illustrated in FIG. 6 may correspond to steps S310 to S360 illustrated in FIG. 3 , respectively.

[0127] The processor (130) can first obtain interaction information of multiple users (200). In this regard, according to one embodiment of the present disclosure, the processor (130) can identify the gaze time of multiple users (200) toward the robot (100) based on images obtained through the camera (110) (S610). Specifically, the processor (130) can recognize the face of each user or directly detect the eyes of the users based on the multiple images obtained through the camera (110). For example, the processor (130) can detect human eyes within the images using the AdaBoost algorithm, the HOG (Histogram of Oriented Gradients) algorithm, the Haar Cascade algorithm, etc. In addition, the processor (130) can track the eyes of the users detected within the multiple images to identify whether each user gazes at the robot (100) and the gaze time toward the robot (100).

[0128] In addition, the processor (130) can identify the number of times a plurality of users (200) input voices to the robot (100) based on the user voices acquired through the microphone of the robot (100) (S615). Specifically, the processor (130) can recognize the voices of the plurality of users (200) (200) input through the microphone of the robot (100), and identify the number of times a plurality of users (200) (200) input voices by matching each user with the recognized user voice. For this purpose, the processor (130) can use an HMM (Hidden Markov Models) algorithm, an LSTM (Long Short-Term Memory) model, a CNN model, a Transformer model, etc.

[0129] At this time, the processor (130) may calculate interaction scores of multiple users (200) based on the identified gaze time and number of voice inputs (S620). Specifically, the processor (130) may calculate an interaction score corresponding to the gaze time and number of voice inputs of each user. As one or more examples, the processor (130) may calculate a first interaction score corresponding to the gaze time of each user on the robot (100). In particular, the processor (130) may calculate a first interaction score as a higher value as the gaze time of the user on the robot (100) becomes longer. In addition, the processor (130) may calculate a second interaction score corresponding to the number of voice input strokes of each user on the robot (100). In particular, the processor (130) may calculate a second interaction score as a higher value as the number of voice inputs becomes greater.

[0130] In addition, the processor (130) can identify the number of touch inputs of a user (200) input through the display of the robot (100) and calculate a third interaction score corresponding to the identified number of touch inputs for each user (300).

[0131] The processor (130) can calculate an interaction score for each user by adding up the first and second interaction scores. Referring to FIGS. 7A and 7B , the processor (130) calculated interaction scores of 0.66, 0.55, 0.59, and 0.78, respectively, based on the gaze time and number of voice inputs for the robot (100) of four users (200-1 to 200-4).

[0132] And the processor (130) can identify an interaction score corresponding to each target area based on the interaction score of at least one user located in each target area (S625).

[0133] As one or more examples, the processor (130) may identify a value obtained by adding up the interaction scores of multiple users (200) located in each target area as an interaction score for each area corresponding to each target area. Specifically, referring to FIG. 7A, the processor (130) may identify an interaction score for each area corresponding to each target area based on the interaction scores (0.66, 0.55, 0.59, and 0.78) of four users (200-1 to 200-4). In this case, the processor (130) may identify 1.14, which is the sum of the interaction scores (0.55 and 0.59) of the two users, as an interaction score corresponding to a second target area in which two users are located.

[0134] As another example, the processor (130) may identify the interaction score corresponding to the highest value among the interaction scores of multiple users (200) located in each target area as the interaction score corresponding to each target area. Specifically, referring to FIG. 7A, the processor (130) may identify the interaction score corresponding to a second target area where two users are located as the interaction score (0.59) of the user with the highest value.

[0135] In addition, the processor (130) can determine the gaze order for the plurality of target areas (30) based on the interaction scores corresponding to each identified target area (S630). Specifically, the processor (130) can determine the gaze order for the plurality of target areas (30) in descending order of the interaction scores.

[0136] For example, as illustrated in FIG. 7a, when identifying an interaction score corresponding to each target area based on a value obtained by adding up the interaction scores of a plurality of users (200) located in the target area, the processor (130) may determine the gaze order for the plurality of target areas (30) (i.e., the first, second, and fifth areas) in the order of the second area, the fifth area, and the first area.

[0137] For another example, as illustrated in FIG. 7b, when identifying an interaction score corresponding to each target area based on the highest interaction score among the interaction scores of multiple users (200) located in each target area, the processor (130) may determine the gaze order for the multiple target areas (30) (i.e., the first, second, and fifth areas) in the order of the fifth area, the first area, and the second area.

[0138] The processor (130) may determine the gaze order by considering both the distance between the user located in each target area and the robot (100) and the interaction score of each user. Specifically, the processor (130) may identify the main user with the highest interaction score among the plurality of users (200) based on the interaction scores of the plurality of users (200). At this time, the processor (130) may determine the gaze order by making the target area where the main user is located the first gaze target area. In addition, the processor (130) may determine the gaze order for the remaining target areas in the order of the distance between the user located in each target area (30) and the robot (100). However, the present invention is not limited thereto, and the processor (130) may determine the gaze order in various ways based on the distance between the user located in each target area and the robot (100) and the interaction score of each user.

[0139] Hereinafter, for the convenience of explanation of the present disclosure, it will be explained assuming that the processor (130) determines the gaze order for multiple target areas in the order of the fifth area, the first area, and the second area.

[0140] The processor (130) can rotate the robot (100) to sequentially gaze at multiple target areas based on the gaze order for multiple targets. For example, if the gaze order for multiple target areas in FIG. 5 is the fifth area, the first area, and the second area, the processor (130) can control the driving unit so that the robot (100) gazes at the fifth area first, then gazes at the first area, and finally gazes at the second area.

[0141] To this end, the processor (130) can identify the rotation direction and rotation angle for each target area (30) by taking into account the gaze order. In particular, in order to control the driving unit so that the robot (100) gazes at another target area (30) after gazing at one of the plurality of target areas (30), the processor (130) must identify the rotation direction and rotation angle of the robot (100) for the next target area (30) by taking into account the position of the target area (30) of the previous order within the gaze order.

[0142] FIG. 8 is a flowchart illustrating a control method for identifying rotation directions and rotation angles for a plurality of target areas according to a gaze order according to one or more embodiments of the present disclosure. S510, S550 to S580 illustrated in FIG. 8 may correspond to S310, S330 to S360 illustrated in FIG. 3. FIG. 9 and FIG. 10 are exemplary diagrams illustrating identifying rotation directions and rotation angles for a plurality of target areas according to a gaze order according to one or more embodiments of the present disclosure.

[0143] Referring to FIG. 8, according to one embodiment of the present disclosure, the processor (130) can identify a first target area (30) among a plurality of target areas (30) based on the gaze order, and can identify a rotation angle and rotation direction of the robot (100) corresponding to the first target area (30) based on the position of the first target area (30) (S830).

[0144] Here, the initial target area (30) may be the target area (30) corresponding to the first order in the gaze sequence. In addition, the rotation angle and rotation direction corresponding to the initial target area (30) may be the rotation angle and rotation direction required for the robot (100) to face the initial target area (30).

[0145] Specifically, the processor (130) can identify a rotation angle and a rotation direction corresponding to the initial target area (30) based on the posture of the robot (100) at the time of identifying the plurality of target areas (30). At this time, the processor (130) can identify the position of the initial target area (30) from the posture of the robot (100) at the time of identifying the plurality of target areas (30), and can identify a rotation angle and a rotation direction corresponding to the initial target area (30) based on the identified position of the initial target area (30).

[0146] In particular, the processor (130) can identify the rotation angle and rotation direction corresponding to the initial target area (30) based on the preset angle used to divide the camera's field of view (10).

[0147] Referring to FIG. 9, the processor (130) can identify the first target area (30) as the fifth area in the gaze sequence and identify the location of the fifth area. After identifying the location of the fifth area, which is located fifth among the plurality of areas (20), the processor (130) can identify the rotation angle and rotation direction required to face the fifth area from the current posture of the robot (100). In addition, the processor (130) can identify the rotation angle for the fifth area as 60° and the rotation direction as clockwise. At this time, the rotation angle for the fifth area (60° = 2 X 30°) can be identified based on a preset angle (30°) used to divide the angle of view of the camera.

[0148] After identifying the rotation direction and rotation angle corresponding to the initial target area (30), the processor (130) can identify the rotation direction and rotation angle for the remaining target areas (30) (S840). At this time, the remaining target areas (30) may be target areas (30) corresponding to the order after the initial target area (30) in the gaze order.

[0149] In this regard, according to one or more embodiments of the present disclosure, the processor (130) can identify the remaining rotation angle and the remaining rotation direction of the robot (100) corresponding to the remaining target areas (30) among the plurality of target areas (30) based on the position of the target area (30) preceding the remaining target areas (30) and the position of the remaining target areas (30) (S840).

[0150] For the remaining target areas (30) after the first target area (30), the processor (130) must control the robot (100) to face a specific target area (30) and then successively control the robot (100) to face another target area (30), and therefore, the rotation direction and rotation identification for each target area (30) must be considered in consideration of the position of the target area (30) of the previous order.

[0151] Specifically, the processor (130) can identify the position of the target area (30) corresponding to the previous order in the order corresponding to each target area (30) in the gaze order for the remaining target areas (30). Then, the processor (130) can identify the rotation angle and rotation direction required for the robot (100) to gaze at the target area (30) of the next order after gazing at the target area (30) of the previous order based on the position of the target area (30) of the previous order and the position of the target area (30) of the next order. At this time, the processor (130) can identify the rotation angle and rotation direction corresponding to the remaining target areas (30) based on the preset angle used to divide the field of view (10) of the camera.

[0152] Referring to FIGS. 9 and 10 , the processor (130) can identify the rotation direction and rotation angle for the first target area (30) corresponding to the first order in the gaze order, and then identify the rotation direction and rotation angle for the remaining target areas (30) (i.e., the first and second areas) after the first order. At this time, the processor (130) can identify the second target area (30) as the first area and the third target area (30) as the second area based on the gaze order.

[0153] Referring to FIG. 9, the processor (130) may consider the position of the first region and the position of the fifth region (i.e., the first target region (30)) that is the previous order (i.e., the first) of the first region in order to identify the rotation direction and rotation angle of the robot (100) with respect to the first region (30), which is the second target region (30) in the gaze order.

[0154] Specifically, the processor (130) can identify a rotation direction and rotation angle for rotating the robot (100) from the fifth region to the first region by considering the location of the fifth region and the location of the first region. At this time, the processor (130) can identify the rotation direction for the first region as counterclockwise and the rotation angle as 120°.

[0155] Thereafter, referring to FIG. 10, the processor (130) may consider the position of the second region and the position of the first region (i.e., the second target region (30)) that is the previous order (i.e., the second) of the second region to identify the rotation direction and rotation angle of the robot (100) for the second region, which is the third target region (30).

[0156] Specifically, the processor (130) can identify a rotation direction and a rotation angle for rotating the robot (100) from the first region to the second region by considering the location of the first region and the location of the second region. In addition, the processor (130) can identify the rotation direction for the second region as clockwise and the rotation angle as 30°.

[0157] The processor (130) can control the driving unit to rotate the robot (100) based on the identified rotation angle and rotation direction (S330, S550).

[0158] At this time, the processor (130) can control the driving unit so that the robot (100) faces each target area (30) based on the rotation angle and rotation direction. As described above, the processor (130) can control the driving unit so that the head of the robot (100) or the body of the robot (100) can rotate to sequentially look at a plurality of target areas (30).

[0159] Accordingly, multiple users (200) located in each target area (30) can recognize that the robot (100) is actively interacting with the user (200). In particular, in the case of the existing robot (100), when multiple users (200) are located around the robot (100), it was difficult to feel that the multiple users (200) were interacting appropriately with the robot (100) because the robot only stared at a specific user (200) or stared at the users (200) indiscriminately. However, the present disclosure does not stare at only a specific user (200) among the multiple users (200), but sequentially and repeatedly stares at the multiple users (200), thereby allowing the multiple users (200) using the robot (100) to feel that they are interacting with the robot (100) more actively and smoothly.

[0160] The processor (130) can control the robot (100) to gaze at each target area (30) for a preset period of time. Hereinafter, the time set to gaze at the target area (30) will be referred to as a gaze time. The processor (130) can control the driving unit so that when the robot (100) rotates toward the target area (30), the driving unit can be controlled so that the robot (100) stops toward the target area (30) for the preset gaze time. Accordingly, the robot (100) can stand toward the target area (30) in a stopped state for the preset gaze time for a plurality of target areas (30).

[0161] The gaze time for each target area (30) may be set differently. As one or more examples, the processor (130) may identify an interaction score corresponding to each target area based on the interaction score of at least one user (200) located in each target area (30), and determine an area-specific gaze time for each target area based on the interaction score corresponding to each target area. In particular, the processor (130) may determine an area-specific gaze time for each target area in proportion to the interaction score corresponding to each target area. That is, the processor (130) may set a longer gaze time for the target area as the interaction score corresponding to the target area increases.

[0162] Additionally, as one or more examples, the processor (130) may set a gaze time for each target area (30) based on the number of users (200) located in each target area. In particular, the processor (130) may determine a gaze time for each target area in proportion to the number of users (200) located in each target area (30). That is, the processor (130) may set a gaze time for each target area to be longer as the number of users (200) located in the target area (30) increases.

[0163] The processor (130) can control the driving unit so that the robot (100) gazes at each target area for a determined gaze time.

[0164] The processor (130) can control the driving unit so that the robot (100) repeatedly gazes at the plurality of target areas (30) according to the gaze order based on the identified (first) rotation angle and (first) rotation direction. To explain again with the above example, if the gaze order is the fifth area, the first area, and the second area, the processor (130) can control the driving unit so that the robot (100) gazes at the second area, which is the last area in the gaze order, and then control the driving unit so that the robot (100) gazes at the fifth area again. More specifically, if the gaze order of the fifth area, the first area, and the second area is referred to as one cycle (or 1 cycle), the processor (130) can control the driving unit so that the robot (100) gazes at the fifth area, the first area, and the second area periodically (or repeatedly). The gaze order may include the gaze order for the target areas (30) during a plurality of cycles. That is, although FIGS. 9 and 10 illustrate that only one cycle of the gaze sequence is included within the gaze sequence, depending on the embodiment, the gaze sequence of the second and third cycles after the first cycle may also be included.

[0165] The processor (130) can identify a new user (210) while the robot (100) rotates based on the identified rotation angle and rotation direction. Here, the new user (210) may be a user (200) that the processor (130) did not recognize previously but has newly recognized. For example, as the robot (100) rotates, the camera's field of view (10) may change, and at this time, a user (200) newly recognized within the changed field of view (10) may correspond to the new user (210). In particular, the new user (210) may also be a user (200) identified as being included in the same user group as multiple users (200).

[0166] Alternatively, the processor (130) may recognize a user (200) located around the robot (100) previously, but may identify the user (200) as not being included in the same user group and thus excluded. In this case, if the processor (130) identifies the excluded user (200) as having later joined the same user group through interaction with multiple users (200) included in the user group, the processor (130) may recognize the excluded user (200) as a new user (210).

[0167] According to one or more embodiments of the present disclosure, when a new user (210) is recognized based on an image acquired through a camera while the robot (100) is rotating, the processor (130) may adjust the gaze order (e.g., generate a second gaze order) so that a new target area (31) in which the new user (210) is located among a plurality of areas (20) within the field of view (10) of the camera of the rotated robot (100) is included in the gaze order (S360).

[0168] Specifically, when a new user (210) other than the previously recognized plurality of users (200) is recognized as the robot (100) rotates, the processor (130) can control the robot (100) to gaze at the new user (210). In particular, when the new user (210) is identified as being included in the same user group as the existing plurality of users (200), the processor (130) can control the robot (100) to include the new user (210) in the user group and allow the robot (100) to gaze at the new user (210) along with the plurality of users (200).

[0169] To this end, the processor (130) may adjust the existing gaze order. In particular, if a new user (210) is located in a new target area (31) rather than in one of the existing identified multiple target areas (30), the processor (130) may adjust the existing gaze order. In addition, if the new target area (31) is not identified among the existing identified multiple areas (20) but is identified in a different location (or in another multiple areas (20)), the processor (130) may adjust the gaze order so that the new target area (31) is included in the gaze order.

[0170] Hereinafter, embodiments of the present disclosure related thereto will be described in detail with reference to FIGS. 11 to 16.

[0171] FIG. 11 is an exemplary diagram illustrating a method for recognizing a new area in which a new user is located when a new user is recognized according to one or more embodiments of the present disclosure. FIG. 12 is a flowchart illustrating a control method for re-identifying a (second) rotation angle and a (second) rotation direction for a plurality of target zeros when a new user is recognized. S1210, S1220, S1230, S1270, and S1280 illustrated in FIG. 12 may correspond to S310, S320, S330, S350, and S360 illustrated in FIG. 3, respectively.

[0172] The processor (130) can recognize a new user (210) located around the robot (100) based on an image acquired through a camera while the robot (100) is rotating (S1240). Specifically, the processor (130) can store feature information about a plurality of users (200) in a memory, and recognize a new user (210) in an image acquired by the robot (100) through the camera based on the stored feature information. After acquiring feature information about the user (200) in the image, the processor (130) can compare the acquired feature information with the feature information about the plurality of users (200) stored in the memory, and identify whether the user (200) is one of the plurality of previously recognized users (200) or a new user (210). In particular, if the acquired feature information is identified as being different from the feature information stored in the memory, the processor (130) can recognize that the user (200) corresponding to the acquired feature information is a new user (210).

[0173] Referring to FIG. 12, when a new user (210) is recognized (S1240), the processor (130) can identify multiple areas (20) by dividing the field of view (10) of the camera of the robot (100) into preset angles (S1250). Then, the robot (100) can identify a new target area (31) in which the new user (210) is located among the multiple areas (20). Since the description of the present disclosure described above can be equally applied to this, a detailed description thereof will be omitted.

[0174] Specifically, referring to FIG. 11, the processor (130) can acquire multiple images through the camera while the robot (100) rotates according to the gaze order and gazes at the fifth area. Then, the processor (130) can recognize a new user (210) around the robot (100) based on the acquired images. The new user (210) may be a user (200-8) located outside the camera's field of view before the robot rotates.

[0175] At this time, when a new user (210) is recognized, the processor (130) can identify multiple areas (21) by dividing the camera's field of view (10) into preset angles while the robot (100) is looking at the fifth area. Then, the processor (130) can identify the sixth area as the target area (30) where the new user (210) is located among the multiple areas (21).

[0176] Referring to FIG. 11, the processor (130) may be different from the existing plurality of regions (20) in that the plurality of regions (20) identified while the robot (100) is rotated are identified while the robot (100) is looking at the target region (30). However, for the convenience of explanation of the present disclosure, the reference numbers added to the existing plurality of regions (20) are duplicated for the new plurality of regions (21) so that one or more embodiments of the present disclosure will be described.

[0177] In addition, the processor (130) can adjust the gaze order so that the new target area (31) is included in the gaze order (S1260). At this time, the processor (130) can adjust the gaze order based on the distance between the new user (210) and the robot (100) or the interaction score of the new user (210).

[0178] As one or more examples, the processor (130) may adjust the gaze order between a target area (30) that the robot (100) has not yet gazed at (or is scheduled to gaze at) (hereinafter, “un-gaze target areas”) within the gaze order and a new target area (31).

[0179] As one or more examples, the processor (130) may adjust the gaze order between the non-gaze target area and the new target area (31) based on the interaction scores of the user (200) located in the non-gaze target area and the new user (210) located in the new target area (31). The processor (130) may calculate the interaction score of the new user (210) located in the new target area (31). Specifically, based on the image acquired through the camera (110) while the robot (100) is rotated, the processor (130) may identify the gaze time of the new user (210) toward the robot (100). In addition, the processor (130) may identify the number of voice inputs of the new user (210) toward the robot (100) based on the voice of the new user (210) acquired through the microphone. In addition, the processor (130) may also identify the number of touch inputs of the new user (210) input through the display. Additionally, the processor (130) can calculate a new user's interaction score based on the gaze time and the number of voice inputs (and touch inputs). Since the description of the above-described embodiment of the present disclosure is equally applicable to this, a detailed description will be omitted.

[0180] The processor (130) can compare the interaction scores corresponding to the non-gaze target area and the new target area (31), respectively, and adjust the gaze order between the non-gaze target area and the new target area (31). Since the method for adjusting the gaze order between the new target area (31) and the non-gaze target area based on the interaction score can be equally applied to the description of the embodiment of the present disclosure described above, a detailed description thereof will be omitted.

[0181] In addition, the processor (130) may adjust the gaze order between the non-gaze target area and the new target area (31) by comparing the distance between the user (200) and the robot (100) located in the non-gaze target area and the distance between the user (200) and the robot (100) located in the new target area (31). In this regard, since the description of the embodiment of the present disclosure described above can be equally applied, a detailed description will be omitted.

[0182] FIG. 13 is an exemplary diagram showing that a new target area is not included in the gaze order when the rotation angle of the robot for gaze at a new target area according to one or more embodiments of the present disclosure is greater than a preset threshold angle.

[0183] According to one or more embodiments of the present disclosure, the processor (130) may adjust the gaze order so that the new target area (31) is included in the gaze order when the rotation angle of the robot (100) for gazing at the new target area (31) is less than a preset threshold angle. Accordingly, the processor (130) may maintain the gaze order without including the new target area (31) in the gaze order when the rotation angle of the robot (100) for gazing at the new target area (31) is equal to or greater than a preset threshold angle.

[0184] Specifically, if the processor (130) adjusts the gaze order to include a new target area (31) whenever a new user and a new target area (31) in which the new user is located are identified while the robot (100) is rotating, a problem may arise in which the opportunity for existing multiple users to interact with the robot (100) is reduced (or the waiting time for interaction is increased).

[0185] Therefore, the processor (130) can set a threshold angle for the rotation angle of the robot (100) at the time of first determining the gaze order. In addition, the processor (130) can identify whether the rotation angle of the robot (100) for gazing at a new target area (31) where the new user is located is within the set threshold angle whenever a new user is recognized. For example, the processor (130) can set a threshold range (40) at the time of first determining the gaze order. Here, the threshold range (40) can be set based on the posture of the robot (100), the field of view range (10), the angles of the plurality of areas (20), the threshold angle of the rotation angle of the robot, etc. at the time of first determining the gaze order.

[0186] And, the processor (130) can identify whether the rotation angle of the robot (100) for gazing at the new target area (31) is within the set threshold angle by identifying whether the position of the new target area (31) is within the preset threshold range (40). If the processor (130) identifies that the position of the new target area (31) is within the preset threshold range (40), the processor (130) can adjust the gazing order so that the new target area (31) is included in the gazing order. If the processor (130) identifies that the position of the new target area (31) is outside the preset threshold range (40), the processor (130) can maintain the gazing order as is without including the new target area (31) in the gazing order.

[0187] Referring to FIG. 13, the processor (130) may set the threshold range (40) related to the rotation angle of the robot (100) to 210° at the time when the robot (100) determines the initial gaze order. At this time, the processor (130) may identify a target area where a newly recognized new user (220) is located among the new plurality of areas (21) while the robot (100) gazes at the user in the fifth area. At this time, if the new target area (31) is the seventh area, the processor (130) may identify that the seventh area is outside the set threshold range (40) (210°). Therefore, the processor (130) may determine that the robot (100) will not gaze at the new target area (31) (the seventh area). That is, the processor (130) does not include the new target area (31) (the seventh area) in the gaze order, and can maintain the gaze order of the existing target areas (the first, second, and fifth areas) as is.

[0188] After adjusting the gaze order, the processor (130) can re-identify the rotation angle and rotation direction of the robot (100) based on the position of the new target area (31) and the adjusted gaze order (S1270). Then, the processor (130) can control the driving unit to rotate the robot (100) based on the re-identified rotation angle and rotation direction (S1280).

[0189] For example, when a new user (210) is recognized based on an image acquired through a camera of a rotated robot (100), the processor (130) can identify a rotation angle and a rotation direction of the robot (100) corresponding to the new target area (31) based on the location of the new target area (31) where the new user (210) is located among the plurality of areas (20). Then, the processor (130) can control a driving unit so that the robot (100) rotates based on the rotation angle and the rotation direction of the robot (100) corresponding to the identified target area (30).

[0190] Specifically, referring to FIG. 11, if a new user (210) identified while the robot (100) is gazing at the fifth area is identified as being located in the sixth area, the processor (130) can identify a location for the newly identified target area (31) (i.e., the sixth area). At this time, the processor (130) can identify the sixth area as being located fourth among the newly identified plurality of areas (21), and can identify the rotation angle and rotation direction for the sixth area as 30° clockwise. In addition, the processor (130) can control the driving unit so that the robot (100) gazes not only at the existing plurality of target areas (30) but also at the newly identified new target area (31) based on the rotation angle and rotation direction for the identified sixth area.

[0191] However, when the gaze order is adjusted as a new target area (31) is identified, the rotation direction and rotation angle for the existing target area (30) may change. In addition, the processor (130) must identify the rotation direction and rotation angle of the robot (100) for the new target area (31) by considering not only the position of the new target area (31) but also the positions of other target areas (30) according to the order of the new target area (31) within the adjusted gaze order.

[0192] To this end, as one or more examples, the processor (130) can identify the next target area in the next order of the target area (30) that the rotated robot (100) is gazing at among the non-gaze target areas and the new target areas (31) based on the adjusted gaze order.

[0193] Here, the next target area may be the next target area (30) of the area that the robot (100) gazes at when the processor (130) recognizes a new user (210) while the robot (100) is rotated and identifies a new target area where the new user (210) is located. At this time, the next target area may be identified based on the adjusted gaze order.

[0194] Specifically, referring to FIG. 11, while the rotated robot (100) is gazing at the fifth area among the plurality of target areas (30), the processor (130) recognizes a new user (210) around the robot (100). At this time, the processor (130) can adjust the gaze order so that the sixth area, which is a new target area (31), is included in the gaze order. In addition, the processor (130) can identify the next target area (30) after the fifth area that the robot (100) is gazing at within the adjusted gaze order.

[0195] FIGS. 14, 15 and 16 are exemplary diagrams illustrating re-identifying the rotational direction and rotational angle of a robot based on the location of a new target area according to one or more embodiments of the present disclosure.

[0196] For example, referring to FIG. 14, if the robot (100) adjusts the existing gaze order (5th area -> 1st area -> 2nd area) to include the 6th area, which is a new target area (31), the next target area within the adjusted gaze order (5th area -> 1st area -> 2nd area -> 6th area) may be the 1st area, which is the order after the 5th area. That is, even if the adjusted gaze order is followed in the same way as the existing gaze order, the processor (130) can control the driving unit to rotate the robot (100) toward the 1st area so that the robot (100) gazes at the 1st area after gazing at the 5th area.

[0197] Referring to FIG. 15, if the robot (100) adjusts the existing gaze order (5th area -> 1st area -> 2nd area) to include the 6th area, which is a new target area (31), the next target area within the adjusted gaze order (5th area -> 6th area -> 1st area -> 2nd area) may be the 6th area, which is the order after the 5th area. In this case, the processor (130) may control the driving unit so that the robot (100) gazes at the 6th area, not the 1st area, after gazing at the 5th area according to the adjusted gaze order, thereby rotating the robot (100) toward the 6th area.

[0198] The processor (130) can re-identify the rotation angle and rotation direction of the robot (100) corresponding to the next target area based on the location of the next target area. In addition, the processor (130) can re-identify the rotation angle and rotation direction of the robot (100) corresponding to the remaining target areas (30) after the next target area among the unattended target areas and the new target areas (31) based on the location of the target area (30) preceding the remaining target areas (30) and the location of the remaining target areas (30).

[0199] Specifically, the processor (130) can identify the rotation angle and rotation direction corresponding to the next target area based on the posture of the robot (100) at the time of identifying the new target area (31). At this time, the processor (130) can identify the location of the next target area based on the location of the target area (30) that the robot (100) is looking at at the time of identifying the new target area (31). In addition, the processor (130) can identify the rotation angle and rotation direction corresponding to the next target area based on the location of the identified next target area. In particular, the processor (130) can identify the rotation angle and rotation direction corresponding to the first target area (30) based on the preset angle used to divide the field of view (10) of the camera.

[0200] Referring to FIG. 14, the processor (130) can identify the next target area as the first area in the adjusted gaze order and identify the position of the first area. In particular, the processor (130) can identify the position of the first area that is not identified within the field of view (10) based on the rotation direction and rotation angle for the first area before the gaze order is adjusted. In addition, the processor (130) can identify the rotation angle and rotation direction required for the robot (100) gazing at the fifth area to face the first area, which is the next target area. In addition, the processor (130) can identify the rotation angle for the first area as 120° and the rotation direction as clockwise. At this time, the rotation angle (120° = 4 X 30°) for the first area can be identified based on the preset angle (30°) used to divide the field of view (10) of the camera. That is, even within the adjusted gaze sequence, the rotation direction and rotation angle of the robot (100) with respect to the first region may not change in that the next target region after the fifth region remains the first region.

[0201] Referring to FIG. 15, the processor (130) can identify the next target area as the sixth area in the adjusted gaze sequence and identify the location of the sixth area. The processor (130) can identify a plurality of areas (20) within the camera's field of view (10) while the robot (100) is gazing at the fifth area, and can identify the location of the sixth target area (30) within the identified plurality of areas (20). In addition, the processor (130) can identify a rotation angle and a rotation direction required for the robot (100) gazing at the fifth area to face the sixth area, which is the next target area. In addition, the processor (130) can identify the rotation angle for the sixth area as 30° and the rotation direction as clockwise.

[0202] At this time, the rotation angle (30° = 1 X 30°) for the 6th area can be identified based on the preset angle (30°) used to divide the camera's field of view (10). That is, even within the adjusted gaze sequence, the next target area after the 5th area has changed from the 1st area to the 6th area, so the angle and direction in which the processor (130) must control the driving unit to rotate the robot (100) after gaze at the 5th area have changed.

[0203] After the processor (130) identifies the next rotation direction and the next rotation angle corresponding to the next target area, it can identify the rotation direction and rotation angle for the remaining target areas (30). At this time, the remaining target areas (30) may be target areas (30) corresponding to the order after the next target area in the gaze order. The description of the present disclosure for identifying the rotation direction and rotation angle for the remaining target areas (30) after the first target area (30) described above can be equally applied.

[0204] In particular, if the next target area is a non-gaze target area, the processor (130) can identify the rotation angle and rotation direction of the robot (100) corresponding to the new target area (31) based on the position of the new target area (31) and the position of the non-gaze target area before the new target area (31). Specifically, if the next target area is a non-gaze target area, the processor (130) can identify the rotation direction and rotation angle for the new target area (31) by considering the position of the target area (30) in the previous order of the new target area (31) added to the adjusted gaze order and the position of the new target area (31).

[0205] Referring back to FIG. 14, the processor (130) can identify the gaze order of the sixth area within the adjusted gaze order. Then, the processor (130) can identify the second area as the target area (30) having the third gaze order, which is the order before the sixth area having the fourth gaze order. After identifying the location of the second area, the processor (130) can identify the rotation direction and rotation angle for rotating the robot (100) from the second area to the sixth area based on the location of the second area and the location of the sixth area.

[0206] Specifically, the processor (130) can identify the location of the second region based on the rotation angle relative to the rotation direction for the second region identified before the gaze order is adjusted. Then, the processor (130) can identify the rotation direction and rotation angle that the robot (100) must rotate to gaze at the sixth region after gazing at the second region, considering the location of the sixth region in the plurality of regions (21) based on the location in the second region. At this time, the processor (130) can identify the rotation direction and rotation angle of the robot (100) relative to the sixth region as being 120° clockwise.

[0207] And, the processor (130) can control the driving unit to rotate the robot (100) so that the robot (100) gazes at the new target area (31) (i.e., the sixth area) based on the rotation angle and rotation direction (i.e., 120° clockwise) of the robot (100) corresponding to the new target area (31) after the robot (100) gazes at the ungaze target area (i.e., the second area) before the new target area (31) according to the adjusted gaze order.

[0208] The processor (130) can identify the rotation angle and rotation direction of the robot (100) corresponding to the new target area (31) based on the location of the new target area (31), if the next target area is a new target area (31), and can re-identify the rotation angle and rotation direction of the robot (100) corresponding to the unattended target area based on the location of the new target area (31).

[0209] Specifically, the next target area is a new target area (31), and the processor (130) can re-identify the rotation direction and rotation angle for the target area (30) of the subsequent order of the new target area (31) added to the adjusted gaze order by taking into account the position of the new target area (31).

[0210] Referring to FIGS. 15 and 16 , the processor (130) can identify a target area (30) having a next order of the sixth area within the adjusted gaze order. In addition, the processor (130) can identify the first area as a target area (30) having a third gaze order, which is a third order after the sixth area having a second gaze order. Accordingly, the processor (130) can identify the position of the sixth area and then identify a rotation direction and a rotation angle for rotating the robot (100) from the sixth area to the first area based on the position of the first area and the position of the sixth area.

[0211] Specifically, the processor (130) can identify the location of the first region based on the rotation angle relative to the rotation direction for the first region identified before the gaze order is adjusted. Then, the processor (130) can identify the rotation direction and rotation angle that the robot (100) must rotate to gaze at the first region after gazing at the sixth region by considering the location in the first region based on the location of the sixth region in the plurality of regions (21). At this time, the processor (130) can identify the rotation direction and rotation angle of the robot (100) relative to the first region as being 150° counterclockwise.

[0212] And, the processor (130) can control the driving unit to rotate the robot (100) so that the robot (100) gazes at the first area based on the rotation angle and rotation direction (i.e., 150° counterclockwise) of the robot (100) corresponding to the next target area (30) (i.e., the first area) after the robot (100) gazes at the new target area (31) (i.e., the sixth area) according to the adjusted gaze order.

[0213] FIG. 17 is a detailed schematic diagram of a robot according to one or more embodiments of the present disclosure.

[0214] Referring to Fig. 17, the robot (100) includes a camera (110), a driving unit (120), a display (140), a memory (150), a sensor (160), a speaker (170), a microphone (180), and a communication interface (190). Among the configurations illustrated in Fig. 12, a detailed description of configurations that overlap with those illustrated in Fig. 2 will be omitted.

[0215] The display (140) can display various visual information. As one or more examples, the display (140) can display various information requested by multiple users. To this end, the display (140) can be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, the display (140) can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc.

[0216] The display (140) may also include a driving circuit, a backlight unit, etc. that can be implemented in the form of a-si TFT, LTPS (low temperature poly silicon) TFT, OTFT (organic TFT), etc.

[0217] The display (140) may be implemented as a touch screen combined with a touch panel and a touch sensor, a flexible display, a rollable display, a 3D display, a display in which multiple display modules are physically connected, etc. When the display (140) is implemented as a touch screen together with a touch panel, the display (140) may function as an output unit that outputs information between the robot (100) and the user, and at the same time, may function as an input unit that provides an input interface between the robot (100) and the user. In particular, the processor may calculate an interaction score for multiple users based on touch inputs input through the display (140).

[0218] The memory (150) can store data required for various embodiments of the present disclosure. As one or more examples, the memory (150) can store map data regarding the driving space in which the robot (100) is located. The memory (150) may be implemented in the form of memory embedded in the robot (100) or in the form of memory that can be attached or detached from the robot (100), depending on the purpose of data storage. For example, data for driving the robot (100) may be stored in a memory embedded in the robot (100), and data for the expansion functions of the robot (100) may be stored in a memory that can be attached or detached from the robot (100).

[0219] In the case of memory embedded in the robot (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0220] In addition, in the case of a memory that can be attached or detached to the robot (100), it can be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), an external memory that can be connected to a USB port (e.g., USB memory), etc.

[0221] According to one or more examples, the memory (150) may store information regarding a plurality of neural network (or artificial intelligence) models. Here, storing information regarding the neural network models may mean storing various information related to the operation of the neural network models, for example, information regarding at least one layer included in the neural network model, information regarding parameters, biases, etc. used in each of at least one layer, etc. However, it goes without saying that, depending on the implementation form of the processor (130), the information regarding the neural network models may be stored in the internal memory of the processor (130). For example, when the processor (130) is implemented with dedicated hardware, the information regarding the neural network models may also be stored in the internal memory of the processor (130).

[0222] The robot (100) includes at least one sensor (160). In addition to the sensors for detecting objects around the robot (100) described above (e.g., lidar sensor, ToF sensor, etc.) and the sensors for detecting the posture of the robot (100) (e.g., IMU sensor, etc.), the at least one sensor (160) may further include at least one of a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor (e.g., an RGB (red, green, blue) sensor), a biometric sensor, a temperature / humidity sensor, an illuminance sensor, or a UV (ultra violet) sensor. The processor may detect a plurality of users (200) around the robot (100) based on sensing information acquired through the at least one sensor (160), and may also identify the distance between the robot (100) (200) and the users.

[0223] The speaker (170) can output an audio signal to the outside of the robot (100). The speaker (170) can output multimedia playback, recording playback, various notification sounds, voice messages, etc. The robot (100) may include an audio output device such as the speaker (170), but may also include an output device such as an audio output terminal. In particular, the speaker (170) can provide acquired information, information processed and produced based on acquired information, response results to user voice, operation results, etc. in voice form.

[0224] The microphone (180) can receive a user's voice. In particular, the microphone (180) can receive the voice of a user located around the robot (100). Furthermore, the microphone (180) can convert the input user's voice into an electrical signal and transmit it to the processor. Accordingly, the processor (130) can identify the number of voice command inputs from multiple users (200). Alternatively, the processor (130) can also identify whether a conversation is taking place between multiple users (200).

[0225] The communication interface (190) can transmit or receive various types of content. As one or more examples, the communication interface (190) can receive or transmit signals in a streaming or download manner from an external device (e.g., a user terminal), an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as AP-based Wi-Fi (Wireless Local Area Network), Bluetooth, Zigbee, wired / wireless Local Area Network (LAN), Wide Area Network (WAN), Ethernet, IEEE 1394, High-Definition Multimedia Interface (HDMI), Universal Serial Bus (USB), Mobile High-Definition Link (MHL), Audio Engineering Society / European Broadcasting Union (AES / EBU), Optical, Coaxial, etc.

[0226] In addition, the robot (100) may further include a user interface. The user interface is a component used by the robot (100) to perform interaction with a user. The processor (130) may receive various information, such as control information of the robot (100), through the user interface, and may calculate an interaction score for each user (200) based on the received information. The user interface may include, but is not limited to, at least one of a touch sensor, a motion sensor, a button, a jog dial, and a switch.

[0227] The methods according to various embodiments of the present disclosure described above may be implemented in the form of applications that can be installed on existing robots. Alternatively, the methods according to various embodiments of the present disclosure described above may be performed using a deep learning-based trained neural network (or a deep learned neural network), i.e., a learning network model. In addition, the methods according to various embodiments of the present disclosure described above may be implemented only with a software upgrade or a hardware upgrade for an existing robot. In addition, the various embodiments of the present disclosure described above may also be performed through an embedded server provided in the robot or an external server of the robot.

[0228] According to one or more exemplary embodiments of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include a display device (e.g., display device (A)) according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When an instruction is executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instruction using other components. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0229] Furthermore, according to one embodiment, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0230] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0231] While specific embodiments of the present disclosure have been specifically illustrated and described, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the following claims.

Claims

1. In robots, camera; drive unit; At least one memory storing instructions; and At least one processor operatively connected to the camera, the driving unit and the at least one memory; The at least one processor executes the instructions to recognize at least one user located around the robot based on an image acquired through the camera, Identifying a first rotation angle and a first rotation direction of the robot based on a position of at least one target area where the at least one user is located among a plurality of areas within the field of view of the camera and a first gaze order for the at least one target area, Controlling the driving unit so that the robot rotates based on the first rotation angle and the first rotation direction; When a new user is recognized in the image while the robot is rotating, a second gaze sequence is generated that includes a new target area where the new user is located among a plurality of areas within the field of view of the camera, Identifying the second rotation angle and the second rotation direction of the robot based on the position of the new target area and the second gaze order; A robot that controls the driving unit to rotate the robot based on the second rotation angle and the second rotation direction.

2. In paragraph 1, At least one processor executes the instruction, Divide the angle of view of the above camera into preset angles to identify the plurality of areas, If the at least one target area is plural, the first target area among the plurality of target areas is identified based on the first gaze order, and the first rotation angle and the first rotation direction of the robot corresponding to the first target area are identified based on the position of the first target area. Identifying the first rotation angle and the first rotation direction of the robot corresponding to the remaining target areas among the plurality of target areas based on the position of the initial target area and the position of the remaining target areas, A robot that controls the driving unit so that the robot rotates according to the first gaze order based on the first rotation angle and the first rotation direction.

3. In paragraph 1, At least one processor executes the instruction, The camera angle of the above rotated robot is divided into preset angles to identify multiple areas, When the new user is recognized based on the image acquired through the camera of the rotated robot, the rotation angle and rotation direction of the robot corresponding to the new target area are identified based on the location of the new target area where the new user is located among the plurality of areas, A robot that controls the driving unit to rotate the robot based on the identified rotation angle and rotation direction.

4. In paragraph 3, At least one processor of the above, If the at least one target area is a plurality of target areas including an unattended target area and a new target area, the next target area in the next order of the target area that the robot is gazing at is identified based on the second gaze order, Based on the location of the next target area, identify the next rotation angle and the next rotation direction of the robot corresponding to the next target area, A robot that identifies the remaining rotation angle and the remaining rotation direction of the robot corresponding to the remaining target areas after the next target area based on the position of the next target area and the position of the remaining target areas.

5. In paragraph 4, At least one processor executes the instruction, If the above next target area is the new target area, the second rotation angle and the second rotation direction are identified based on the position of the new target area, Based on the location of the new target area, the third rotation angle and the third rotation direction of the robot corresponding to the unattended target area are identified, A robot that controls the driving unit so that the robot rotates based on the third rotation angle and the third rotation direction after the robot gazes at the new target area.

6. In paragraph 4, At least one processor executes the instruction, If the next target area is the unattended target area, the second rotation angle and the second rotation direction are identified based on the location of the new target area and the location of the unattended target area before the new target area, A robot that controls the driving unit so that the robot rotates based on the second rotation angle and the second rotation direction after the robot gazes at an unattended target area before the new target area.

7. In paragraph 1, At least one processor executes the instruction, A robot that maintains the first gaze order without including the new target area in the first gaze order if the rotation angle of the robot for gazing at the new target area is greater than or equal to a preset threshold angle.

8. In paragraph 1, At least one processor executes the instruction, Identifying multiple first users located around the robot based on images acquired through the camera; A robot that identifies multiple users of the same user group among the plurality of first users based on at least one of the distance between the plurality of first users, the time at which the plurality of first users were located within the field of view of the camera, and whether or not there is a conversation between the plurality of first users.

9. In paragraph 1, further comprising a microphone for receiving a user's voice; At least one processor executes the instruction, If the at least one user is a plurality of users and the at least one target area is a plurality of target areas, the gaze time of the plurality of users toward the robot is identified based on the image acquired through the camera, Identifying the number of voice inputs to the robot from the plurality of users based on the user voices acquired through the microphone; Based on the identified gaze time and number of voice inputs, the interaction scores of the plurality of users are calculated, Based on the interaction score of each of at least one user located in each target area, an area-specific interaction score corresponding to each target area is identified, A robot that determines a first gaze order for the plurality of target areas based on the interaction scores for each area corresponding to each target area.

10. In paragraph 9, At least one processor executes the instruction, Based on the interaction score for each region corresponding to each target region, the gaze time for each target region is determined, A robot that controls the driving unit so that the robot gazes at each target area during the gaze time for each area.

11. In a method of controlling a robot, A step of recognizing at least one user located around the robot based on an image acquired through a camera; A step of identifying a first rotation angle and a first rotation direction of the robot based on a position of at least one target area where the at least one user is located among a plurality of areas within the field of view of the camera and a first gaze order for the at least one target area; A step of controlling the driving unit to cause the robot to rotate based on the first rotation angle and the first rotation direction; When a new user is recognized in the image while the robot is rotating, a step of generating a second gaze sequence including a new target area where the new user is located among a plurality of areas within the field of view of the camera; A step of identifying a second rotation angle and a second rotation direction of the robot based on the position of the new target area and the second gaze order; and A control method comprising a step of controlling the driving unit to rotate the robot based on the second rotation angle and the second rotation direction.

12. In paragraph 11, The step of identifying the first rotation angle and the first rotation direction comprises: A step of identifying the plurality of areas by dividing the angle of view of the camera into preset angles; If the at least one target area is plural, a step of identifying a first target area among the plurality of target areas based on the first gaze order; A step of identifying a first rotation angle and a first rotation direction of the robot corresponding to the first target area based on the position of the first target area; and A control method comprising a step of identifying a first rotation angle and a first rotation direction of the robot corresponding to a remaining target area among the plurality of target areas based on a position of the initial target area and a position of the remaining target areas.

13. In paragraph 11, The step of identifying the second rotation angle and the second rotation direction comprises: A step of identifying a plurality of areas by dividing the angle of view of the camera into preset angles; A control method, wherein at least one target area and the new target area are included in the plurality of areas.

14. In paragraph 13, The step of identifying the first rotation angle and the first rotation direction comprises: If the at least one target area is a plurality of target areas including an unattended target area and a new target area, a step of identifying a next target area in the next order of target areas that the robot is gazing at based on the second gaze order; A step of identifying the next rotation angle and the next rotation direction of the robot corresponding to the next target area based on the location of the next target area; and A control method further comprising the step of identifying a remaining rotation angle and a remaining rotation direction of the robot corresponding to the remaining target areas after the next target area based on the position of the next target area and the position of the remaining target areas.

15. In paragraph 14, The steps of identifying the next rotation angle and the next rotation direction are: If the above next target area is the new target area, the second rotation angle and the second rotation direction are identified based on the position of the new target area, The step of identifying the remaining rotation angle and the remaining rotation direction is: Based on the location of the new target area, the third rotation angle and the third rotation direction of the robot corresponding to the unattended target area are identified, The step of controlling the driving unit to rotate the robot based on the third rotation angle and the third rotation direction is: A control method for controlling the driving unit so that the robot rotates based on the third rotation angle and the third rotation direction after the robot gazes at the new target area.

Images