Robot and control method thereof

Abstract

A robot and a control method thereof are provided. The robot includes: a camera; a driving unit; at least one memory to store instructions; and at least one processor operably connected to the at least one memory. The at least one processor may, by executing the instructions, identify at least one user located around the robot based on an image obtained by the camera; identifying a first rotation angle and a first rotation direction of the robot based on a position of at least one target area in which at least one user is located among a plurality of areas within a field angle of view of the camera and a first gaze order for the at least one target area; and controlling the driving unit such that the robot rotates based on the first rotation angle and the first rotation direction.

Description

Technical Field

[0001] This disclosure relates to a robot and a control method thereof, and more particularly, to a robot and a control method thereof for determining a gaze order for multiple users and sequentially gazing at multiple users based on the determined gaze order. Background Technology

[0002] In recent years, with the development of electronic technology, robots have been increasingly used in various industrial fields. Advances in object recognition technology have enabled robots to accurately distinguish between different objects, and thanks to the development of autonomous driving technology, robots can drive stably within a driving space without interfering with pedestrian traffic. Considering service robots taking orders in restaurants or guiding robots providing directions in airports and large supermarkets, it is evident that the fields and methods of robot use have become increasingly diversified.

[0003] As robots are used in various fields, there are situations where multiple users are using them simultaneously. This may happen, for example, when robots guide a large number of visitors at an exhibition or when robots guide a large number of tourists at an airport. Summary of the Invention

[0004] Technical solution According to one aspect of this disclosure, a robot includes: a camera; a actuator; at least one memory storing instructions; and at least one processor operatively connected to the at least one memory. The at least one processor can be configured to execute instructions to perform the following operations: The robot is identified based on images acquired by a camera; a first rotation angle and a first rotation direction of the robot are identified based on the position of at least one target area where at least one user is located in one of multiple areas within the camera's field of view and a first gaze sequence for the at least one target area; a control actuator rotates the robot based on the first rotation angle and the first rotation direction; a second gaze sequence is generated based on the identification of a new user from the images during the robot's rotation, wherein the second gaze sequence may include a new target area where the new user is located in one of multiple areas within the camera's field of view; a second rotation angle and a second rotation direction of the robot are identified based on the position of the new target area and the second gaze sequence; and the control actuator rotates the robot based on the second rotation angle and the second rotation direction.

[0005] At least one processor may also be configured to execute instructions to perform the following operations: identify multiple regions by dividing the field of view of a camera at a predetermined angle; identify a first target region among the multiple target regions based on a first gaze order, based on the fact that at least one target region is multiple target regions; identify a first rotation angle and a first rotation direction of the robot corresponding to the first target region based on the position of the first target region; identify a first rotation angle and a first rotation direction of the robot corresponding to the remaining target regions based on the position of the first target region and the positions of the remaining target regions among the multiple target regions; and control a driver to rotate the robot according to the first gaze order based on the first rotation angle and the first rotation direction.

[0006] At least one processor may also be configured to execute instructions to identify multiple regions by dividing the camera's field of view at predetermined angles. At least one target region and new target regions may exist within these multiple regions.

[0007] At least one processor may be configured to: identify the next target region in the next order of the current target region being gazed upon by the robot based on a second gaze order, since at least one target region is a plurality of target regions; identify the next rotation angle and the next rotation direction of the robot corresponding to the next target region based on the position of the next target region; and identify the remaining rotation angle and the remaining rotation direction of the robot corresponding to the remaining target regions based on the position of the next target region and the position of the remaining target regions after the next target region.

[0008] At least one processor may also be configured to execute instructions to perform the following operations: based on the position of the new target region, identify a second rotation angle and a second rotation direction based on the position of the new target region; based on the position of the new target region, identify a third rotation angle and a third rotation direction of the robot corresponding to the unprocessed target region; and after the robot gazes at the new target region, control the actuator to rotate the robot based on the third rotation angle and the third rotation direction.

[0009] At least one processor may also be configured to execute instructions to: identify a second rotation angle and a second rotation direction based on the fact that the next target region is an unprocessed target region, based on the position of the new target region and the position of the unprocessed target region preceding the new target region; and after viewing the unprocessed target region preceding the new target region, control the actuator to rotate the robot based on the second rotation angle and the second rotation direction.

[0010] At least one processor may also be configured to execute instructions to maintain a first gaze order without including the new target region in the first gaze order, based on the fact that the rotation angle of the robot used for gazing at the new target region is equal to or greater than a predetermined critical angle.

[0011] At least one processor may also be configured to execute instructions to perform the following operations: identify multiple first users located around the robot based on images acquired by the camera; and identify multiple users in the same user group among the multiple first users based on at least one of the distance between the multiple first users, the time during which the multiple first users are within the field of view of the camera, and whether the multiple first users are engaged in dialogue.

[0012] The robot may also include a microphone configured to receive user speech. At least one processor may also be configured to execute instructions to: identify the gaze duration of multiple users looking at the robot based on images acquired via a camera, given that at least one user is multiple users and at least one target region is multiple target regions; identify the number of voice inputs from multiple users to the robot based on user speech acquired via the microphone; calculate interaction scores for multiple users based on the gaze duration and the number of voice inputs; identify a region interaction score corresponding to each target region based on the interaction scores of at least one user located in each target region; and determine a first gaze order for the multiple target regions based on the region interaction scores corresponding to each target region.

[0013] At least one processor may also be configured to run instructions to: determine the region gaze duration for each target region based on the region interaction score corresponding to each target region; and control the actuator to cause the robot to gaze at each target region during the region gaze duration.

[0014] According to one aspect of this disclosure, a robot control method includes: identifying at least one user located around the robot based on images acquired by a camera; identifying a first rotation angle and a first rotation direction of the robot based on the position of at least one target area where at least one user is located in a plurality of areas within the field of view of the camera and a first gaze sequence for the at least one target area; controlling a driver to rotate the robot based on the first rotation angle and the first rotation direction; generating a second gaze sequence based on identifying a new user from the images during the rotation of the robot, wherein the second gaze sequence may include a new target area where the new user is located in a plurality of areas within the field of view of the camera; identifying a second rotation angle and a second rotation direction of the robot based on the position of the new target area and the second gaze sequence; and controlling a driver to rotate the robot based on the second rotation angle and the second rotation direction.

[0015] The operation of identifying the first rotation angle and the first rotation direction may include: identifying multiple regions by dividing the field of view of the camera by a predetermined angle; identifying a first target region among the multiple target regions based on a first gaze order, since at least one target region is multiple target regions; identifying the first rotation angle and the first rotation direction of the robot corresponding to the first target region based on the position of the first target region; and identifying the first rotation angle and the first rotation direction of the robot corresponding to the remaining target regions based on the position of the first target region and the position of the remaining target regions among the multiple target regions.

[0016] The operation of identifying the second rotation angle and the second rotation direction may include: identifying multiple regions by dividing the camera's field of view at a predetermined angle. At least one target region and a new target region may be located in the multiple regions.

[0017] The operation of identifying the first rotation angle and the first rotation direction may include: identifying the next target region in the next order of the current target region being gazed at by the robot based on a second gaze order, since at least one target region is multiple target regions, wherein the multiple target regions may include unprocessed target regions and new target regions; identifying the next rotation angle and the next rotation direction of the robot corresponding to the next target region based on the position of the next target region; and identifying the remaining rotation angle and the remaining rotation direction of the robot corresponding to the remaining target regions based on the position of the next target region and the position of the remaining target regions after the next target region.

[0018] The operation of identifying the next rotation angle and the next rotation direction may include: identifying a second rotation angle and a second rotation direction based on the position of the new target area, since the next target area is a new target area. The operation of identifying the remaining rotation angle and the remaining rotation direction may include: identifying a third rotation angle and a third rotation direction of the robot corresponding to the unprocessed target area, based on the position of the new target area. The operation of controlling the actuator to rotate the robot based on the third rotation angle and the third rotation direction may include: after the robot gazes at the new target area, controlling the actuator to rotate the robot based on the third rotation angle and the third rotation direction.

[0019] The control method may further include: identifying a second rotation angle and a second rotation direction based on the fact that the next target area is an unprocessed target area, based on the position of the new target area and the position of the unprocessed target area before the new target area; and after viewing the unprocessed target area before the new target area, controlling the actuator to rotate the robot based on the second rotation angle and the second rotation direction.

[0020] The control method may further include: maintaining a first gaze sequence without including the new target region in the first gaze sequence, based on the fact that the rotation angle of the robot used for gazing at the new target region is equal to or greater than a predetermined critical angle.

[0021] The control method may further include: identifying multiple first users located around the robot based on images acquired by the camera; and identifying multiple users in the same user group among the multiple first users based on at least one of the distance between the multiple first users, the time during which the multiple first users are within the field of view of the camera, and whether the multiple first users are engaged in dialogue.

[0022] The control method may further include: identifying the gaze duration of multiple users gazing at the robot based on images acquired by a camera, given that at least one user is multiple users and at least one target region is multiple target regions; identifying the number of voice inputs from multiple users to the robot based on user voices acquired by a microphone; calculating interaction scores for multiple users based on the gaze duration and the number of voice inputs; identifying a region interaction score corresponding to each target region based on the interaction scores of at least one user located in each target region; and determining a first gaze order for the multiple target regions based on the region interaction scores corresponding to each target region.

[0023] The control method may also include: determining the region gaze duration for each target region based on the region interaction score corresponding to each target region; and controlling the actuator to cause the robot to gaze at each target region during the region gaze duration. Attached Figure Description

[0024] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which: Figure 1 This is a diagram of a robot responding to multiple users; Figure 2 This is a block diagram illustrating the configuration of a robot according to one or more embodiments of the present disclosure; Figure 3 This is a flowchart of a method for controlling a robot according to one or more embodiments of the present disclosure; Figure 4 This is a diagram illustrating a method for identifying multiple users included in the same user group according to one or more embodiments of the present disclosure; Figure 5 This is a diagram illustrating the identification of a target region within a plurality of regions in the field of view of a camera according to one or more embodiments of the present disclosure; Figure 6 This is a flowchart illustrating a method for determining gaze order based on interaction score according to one or more embodiments of the present disclosure; Figure 7a This is a diagram illustrating a method for determining gaze order based on interaction scores according to one or more embodiments of the present disclosure; Figure 7b This is a diagram illustrating a method for determining gaze order based on interaction scores according to one or more embodiments of the present disclosure; Figure 8 This is a flowchart illustrating a control method for identifying the rotation direction and rotation angle of multiple target regions according to the gaze sequence according to one or more embodiments of the present disclosure; Figure 9 This is a diagram illustrating the identification of rotation direction and rotation angle of multiple target regions according to gaze order according to one or more embodiments of the present disclosure; Figure 10 This is a diagram illustrating the identification of rotation direction and rotation angle of multiple target regions according to gaze order according to one or more embodiments of the present disclosure; Figure 11 This is a diagram illustrating the identification of a new area where a new user is located based on the identification of a new user, according to one or more embodiments of this disclosure; Figure 12 This is a flowchart illustrating a control method for re-identifying the rotation angle and rotation direction of multiple target regions based on the identification of a new user, according to one or more embodiments; Figure 13 This is a diagram illustrating how, according to one or more embodiments of the present disclosure, the rotation angle of a robot used for gazing at a new target region is equal to or greater than a predetermined critical angle without including the new target region in the gazing sequence; Figure 14 This is a diagram illustrating the re-identification of the robot's rotation direction and rotation angle based on the position of a new target region according to one or more embodiments of this disclosure; Figure 15 This is a diagram illustrating the re-identification of the robot's rotation direction and rotation angle based on the position of a new target region according to one or more embodiments of this disclosure; Figure 16 This is a diagram illustrating the re-identification of the robot's rotation direction and rotation angle based on the position of a new target region according to one or more embodiments of this disclosure; and Figure 17 This is a detailed configuration diagram of a robot according to one or more embodiments of the present disclosure. Detailed Implementation

[0025] This disclosure can be modified in various ways and has multiple embodiments, and specific embodiments of this disclosure are shown in the accompanying drawings and described in detail in the detailed description. However, it should be understood that the techniques mentioned in this disclosure are not limited to the specific embodiments and include various modifications, equivalents, and / or alternatives to the embodiments of this disclosure. Throughout the drawings, similar components are indicated by similar reference numerals.

[0026] In describing this disclosure, detailed descriptions of known functions or configurations that are determined to be relevant to this disclosure may unnecessarily obscure the main points of this disclosure, such detailed descriptions are omitted.

[0027] Furthermore, the following embodiments can be modified in several different forms, and the scope and spirit of this disclosure are not limited to the following embodiments. Rather, these embodiments are provided to make this disclosure comprehensive and complete, and to fully convey the spirit of this disclosure to those skilled in the art.

[0028] The terminology used in this disclosure is for describing particular embodiments only and is not intended to limit the scope of this disclosure. Unless otherwise expressly stated, singular terms may also include plural forms.

[0029] In this disclosure, the expressions “have,” “may have,” “include,” “may include,” etc., indicate the presence of a corresponding feature (e.g., a value, function, operation, or component (such as a part)) and do not exclude the presence of additional features.

[0030] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A and / or B” can include all possible combinations of the items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can indicate all of the following: 1) the case that includes at least one A, 2) the case that includes at least one B, or 3) the case that includes both at least one A and at least one B.

[0031] The terms “first,” “second,” etc., used in this disclosure may refer to various components, regardless of the order and / or importance of the components. These terms are used only to distinguish one component from another and do not limit the corresponding components.

[0032] When referring to any component (e.g., the first component) being "operably or communicatively combined" with "another component (e.g., the second component)" or "operably or communicatively combined" to "another component (e.g., the second component)" or "connected" to "another component (e.g., the second component), it should be understood that any component is directly combined to the other component, or can be combined to the other component through yet another component (e.g., the third component).

[0033] When it is mentioned that any component (e.g., the first component) is “directly coupled to” or “directly connected to” another component (e.g., the second component), it should be understood that there is no other component (e.g., the third component) between any two components.

[0034] The expression “configured (or set) as” as used in this disclosure may be replaced by the expressions “suitable for,” “capable of,” “designed to,” “adapted to,” “manufactured as,” or “capable”, depending on the context. The term “configured (or set) as” may not necessarily indicate “specifically designed for” in the hardware.

[0035] Conversely, the phrase "a device configured as..." can indicate that the device can "perform" together with other devices or components. For example, "a processor configured (or set) to perform A, B, and C" can indicate a dedicated processor (e.g., an embedded processor) for performing the respective operations or a general-purpose processor (e.g., a central processing unit (CPU) or application processor) that can perform the respective operations by running one or more software programs stored in a memory device.

[0036] In the embodiments, a "module" or "device" may perform at least one function or operation and may be implemented by hardware or software, or by a combination of hardware and software. Furthermore, aside from "modules" or "devices" that require implementation by specific hardware, multiple "modules" or multiple "devices" may be integrated into at least one module and implemented by at least one processor.

[0037] Various elements and areas in the accompanying drawings are shown schematically. Therefore, the spirit of this disclosure is not limited to the relative dimensions or spacing shown in the drawings.

[0038] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, so that those skilled in the art to which this disclosure pertains can readily practice the present disclosure.

[0039] Figure 1 This is a diagram of a robot responding to multiple users.

[0040] Reference Figure 1 As robot 100 has recently been used in various fields, there are situations where robot 100 must respond to multiple users 200-1, 200-2, and 200-3 (hereinafter referred to as 200). Here, robot 100 responding to multiple users 200 can mean that robot 100 interacts with multiple users 200 simultaneously. For example, robot 100 may include a robot that guides large numbers of visitors at exhibitions or markets, or a robot that guides or provides information to large numbers of visitors at airports or large supermarkets.

[0041] Currently, the existing robot 100 interacts with a specific user among multiple users 200, or randomly with multiple users 200. Therefore, in most cases, the multiple users 200 do not perceive that they have effectively interacted with the robot 100. Specifically, when multiple users request specific information from the robot 100, if the robot 100 only interacts with the specific user among the multiple users 200, the information provided by the robot 100 may not be effectively conveyed to other users besides the specific user.

[0042] Furthermore, even if the existing robot 100 subsequently identifies a specific user among the multiple users 200 that it did not initially identify, the robot 100 does not include that specific user as an interaction target. In this case, even if the subsequently identified specific user is included in the same user group (e.g., the same audience group or visitor group) as the multiple users 200, that specific user is disadvantaged simply because they were identified later than other users, and is excluded from the robot 100's interaction targets. In other words, there is a problem that the subsequently identified specific user cannot interact with the robot 100.

[0043] This disclosure aims to solve this problem. When multiple users 200 use robot 100, robot 100 determines an appropriate interaction order to interact with the multiple users 200 equally, and performs interactions with the multiple users 200 sequentially according to the gaze order. Furthermore, even when a new user is identified, robot 100 does not ignore the new user, and interacts with the new user as well as the existing multiple users 200. Embodiments of this disclosure related to this will be described in detail below.

[0044] Figure 2 This is a block diagram illustrating the configuration of a robot according to one or more embodiments of the present disclosure.

[0045] Reference Figure 2 The robot 100 includes a camera 110, a driver 120, and one or more processors 130.

[0046] The robot 100 according to one or more embodiments of this disclosure can be implemented as various electronic devices that provide various services to a user while interacting with the user. For example, the robot 100 may be a guide robot 100 located in an airport or shopping mall, used to provide various services (such as providing specific information, guiding directions, etc.) to the user in response to input received from the user (e.g., voice input, touch input, etc.).

[0047] According to one or more embodiments of this disclosure, robot 100 may be divided into a head and a body. As one or more examples, the head of robot 100 may include a camera 110 or a sensor to acquire images of objects around robot 100 or perform object detection functions. Furthermore, the head of robot 100 may include a display, and robot 100 may display various image information on the display. In particular, when robot 100 is interacting with a user, the robot may display graphic objects representing eyes, nose, and mouth on the display included in its head, providing a vivid interactive experience to the user using robot 100.

[0048] The body of robot 100 can be connected to drive 120 and can perform the functions of moving robot 100 or supporting the frame of robot 100. Furthermore, the body of robot 100 may include a display, and various image information can be displayed on the display. Specifically, robot 100 can display information requested or needed by the user (e.g., user interface (UI) etc.) on the display, and can receive user commands from the user via a touch panel included in the display.

[0049] Camera 110 images objects around robot 100 and acquires multiple images of the objects. Specifically, camera 110 can acquire images of objects (e.g., people, animals, other objects, etc.) existing around robot 100.

[0050] Therefore, camera 110 can be implemented as an imaging device (such as a CMOS image sensor (CIS) with a CMOS structure or a charge-coupled device (CCD) with a CCD structure). However, this disclosure is not limited thereto, and camera 110 can be implemented as a camera 110 module with various resolutions capable of imaging objects.

[0051] Furthermore, the camera 110 can be implemented as a depth camera 110, a stereo camera 110, or an RGB camera 110. Therefore, the camera 110 can acquire depth information about the object and an image of the object.

[0052] A actuator 120 drives at least a portion of the robot 100. As one or more examples, the actuator 120 may be implemented as a rotatable motor connecting the head and body of the robot 100. Thus, the actuator 120 rotates the head connected to the actuator 120 360° to allow the robot 100 to obtain information about objects around the robot 100 via the camera 110. However, this disclosure is not limited thereto, and the robot 100 may be implemented as an integrated body with a non-separable head and body. In this case, the body and head may be distinguishable from each other on the robot 100 according to the functions they each perform.

[0053] Furthermore, as one or more examples, the actuator 120 may be located at the lower part of the robot 100's body and move the robot 100. That is, the actuator 120 may be a device capable of driving the robot 100. In this case, the actuator 120 may adjust the direction of travel and the speed of travel under the control of a processor. For this purpose, the actuator 120 may include a power generation device (e.g., a gasoline engine, diesel engine, liquefied petroleum gas (LPG) engine, electric motor, etc.) that generates power for the robot 100's movement based on the fuel (or energy source) used, and a steering device for controlling the direction of travel (e.g., mechanical steering, hydraulic steering, electronic power steering (EPS), etc.).

[0054] Furthermore, the drive 120 may include multiple wheels. Multiple wheels can be rotated based on processor control, and the robot 100 can be moved accordingly. Optionally, multiple wheels can be rotated at different speeds (or only some of the multiple wheels can be rotated) based on processor control, and the robot 100 can be rotated accordingly.

[0055] One or more processors are electrically connected to the camera 110 and the driver 120 and control the overall operation and function of the robot 100.

[0056] One or more processors 130 may include one or more of a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), integrated many-core (MIC), digital signal processor (DSP), neural processing unit (NPU), hardware accelerator, or machine learning accelerator. One or more processors 130 may control one or any combination of other components of robot 100 and may perform operations related to communication or data processing. One or more processors 130 may run one or more programs or instructions stored in memory. For example, one or more processors 130 may perform methods according to one or more embodiments of this disclosure by running one or more instructions stored in memory.

[0057] When a method according to one or more embodiments of this disclosure includes multiple operations, the multiple operations may be performed by one or more processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor), while the third operation may be performed by a second processor (e.g., an artificial intelligence-specific processor).

[0058] One or more processors 130 may be implemented as a single-core processor including one core, or as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). When one or more processors 130 are implemented as multi-core processors, each of the multiple cores included in the multi-core processor may include the processor 130's internal memory (such as cache memory and on-chip memory), and a common cache shared by the multiple cores may be included in the multi-core processor. Furthermore, each of the multiple cores (or some of the multiple cores) included in the multi-core processor may independently read and execute program instructions for implementing the methods according to one or more embodiments of the present disclosure, and all (or some) of the multiple cores may be linked to read and execute program instructions for implementing the methods according to one or more embodiments of the present disclosure.

[0059] When a method according to one or more embodiments of this disclosure includes multiple operations, the multiple operations may be executed by one core of a plurality of cores included in a multi-core processor, or they may be executed by multiple cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, all of the first operation, the second operation, and the third operation may be executed by the first core of the multi-core processor, the first operation and the second operation may be executed by the first core of the multi-core processor, and the third operation may be executed by the second core of the multi-core processor.

[0060] In one or more embodiments of this disclosure, processor 130 may be a system-on-a-chip (SoC) integrating one or more processors and other electronic components, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, and here, the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of this disclosure are not limited thereto.

[0061] In the following text, for ease of description, one or more processors 130 will be referred to as processor 130.

[0062] Figure 3 This is a flowchart of a method for controlling a robot according to one or more embodiments of the present disclosure.

[0063] Reference Figure 3 The processor 130 identifies at least one user 200 located around the robot 100 based on images acquired by the camera (S310).

[0064] Specifically, the processor 130 can acquire multiple images of objects, people, animals, etc., around the robot 100 via a camera. The processor 130 can then extract feature information about at least one user 200 included in the images acquired by the camera, and match the extracted feature information with the at least one user 200 in the images to identify the at least one user 200. Furthermore, the robot 100 can track and identify at least one user 200 within multiple images based on the feature information matched with the at least one user 200.

[0065] At this time, the processor 130 may select and identify only those people who interact with or want to interact with the robot 100 from among the multiple people surrounding the robot 100 as users 200. In other words, the processor 130 may identify those who are located around the robot 100 to use the robot 100 as users 200. For example, the processor 130 may not only identify those who are located around or pass by the robot 100 as users 200, but may also identify those who are located around the robot 100 to use the robot 100 as users 200.

[0066] To this end, processor 130 can identify the distance between each of the multiple people and robot 100 based on the acquired images, and select at least one user 200 from the multiple people based on the identified distance. For example, processor 130 can identify the distance between each of the multiple people and robot 100 based on depth information included in the acquired images, and only distinguishably identify people within a predetermined distance as users 200. At this time, processor 130 can identify people who are identified as watching robot 100 among those who stay within the predetermined distance for a predetermined time as users 200. Processor 130 can extract feature information about users 200 from the acquired images, and continuously track and identify users 200 included in multiple images based on the extracted feature information.

[0067] Furthermore, the processor 130 can select and identify at least one user 200 among multiple people based on whether each person interacts with the robot 100. Here, interaction refers to whether the robot 100 gazes (or the duration of gaze), whether a voice command is input to the robot 100 (or the number of times it is input), whether a touch is input (and the number of times it is input), and changes in the distance between the robot 100 and the person. For example, the processor 130 can extract feature information about multiple people included in an image, identify the eyes of multiple people based on the extracted feature information, and identify whether each person is gazing at the robot 100. Furthermore, the processor 130 can distinguishably identify at least one user 200 among multiple people who wants to use the robot 100 based on whether each person is gazing at the robot 100 and the duration of gaze.

[0068] According to one or more embodiments of this disclosure, processor 130 may selectively identify only a plurality of users 200 that are included in the same user group. Here, a user group may be a group of multiple users located around robot 100 who have the same purpose of using robot 100. Processor 130 may control robot 100 such that robot 100 only looks at the plurality of users 200 included in the same user group.

[0069] To this end, processor 130 can identify whether multiple users 200 around robot 100 are included in the same user group. As one or more examples, processor 130 can identify multiple users 200 included in the same user group based on images acquired by a camera, based on whether there is interaction between multiple users 200, the distance between multiple users 200, the time difference in the identification of multiple users 200, etc.

[0070] Figure 4 This is a diagram illustrating a method for identifying multiple users included in the same user group according to one or more embodiments of the present disclosure.

[0071] According to one or more embodiments of this disclosure, processor 130 may identify multiple users located around robot 100 based on images acquired by camera 110. Furthermore, processor 130 may identify multiple users 200 belonging to the same user group based on at least one of the following: distance between multiple users 200, time during which multiple users 200 are within the field of view 10 of camera 110, and whether multiple users 200 are engaged in conversation.

[0072] Specifically, processor 130 can identify multiple users 200 located around robot 100 based on images acquired by camera 110. More specifically, processor 130 can identify multiple users 200 among multiple people located around robot 100 based on images acquired by camera 110. As one or more examples, processor 130 can identify the distance between robot 100 and each person based on depth information about each person included in the image. Furthermore, processor 130 can identify only multiple people within a predetermined distance as multiple users 200.

[0073] Reference Figure 4The processor 130 can identify multiple people 200-1 to 200-8 located within 2m of the robot 100 within the field of view 10 (150°) of the camera 110 as multiple users 200. Therefore, the processor 130 can identify only six of the eight people 200-1 to 200-8 surrounding the robot 100, located within 2m of the robot 100 within the field of view 10 (150°) of the camera 110, as multiple users 200. Specifically, the processor 130 may not identify person 200-7 located more than 2m away from the robot 100 as a user. Furthermore, the processor 130 may not identify person 200-8 located outside the field of view 10 (150°) of the camera 110 as a user. At this time, since the person 200-8 located outside the field of view 10 (150°) of the camera 110 is not included in the image acquired by the camera 110, the processor 130 may not be able to recognize the person outside the field of view 10 of the camera 110.

[0074] After identifying multiple users 200, the processor 130 can select multiple users 200 that are included in the same user group. In particular, the processor 130 can identify multiple users 200 that are included in the same user group based on at least one of the following: the distance between the multiple users 200, the time during which the multiple users 200 are within the field of view 10 of the camera 110, and whether the multiple users 200 are having a conversation.

[0075] Specifically, processor 130 can identify the distances between multiple identified users 200. Specifically, processor 130 can identify the distance between each user in the image acquired by camera 110. Therefore, processor 130 can identify the distances between multiple users 200. Furthermore, processor 130 can only identify multiple users 200 whose distances are within a predetermined distance (or interval) as belonging to the same user group. For example, refer to... Figure 4 The processor 130 can identify multiple users 200 among the six identified users 200-1 to 200-6 whose distance between each other (or between adjacent users) is within 30cm as the same user group.

[0076] At this point, the processor 130 can identify whether the interval between the remaining users is within a predetermined distance (or interval) based on a specific user among the multiple users 200, and include at least one user within the predetermined distance from the specific user and the specific user in the user group. Here, the specific user can be the first user among the multiple users 200 located within the field of view 10 of the camera 110 and at a predetermined distance (a predetermined distance from the robot 100).

[0077] Furthermore, the processor 130 can identify the time when multiple users 200 are located within the field of view 10 of the camera 110. Specifically, the processor 130 can identify the time when each user begins to be included in multiple images acquired by the camera 110. In this way, the processor 130 can identify the time when each user is located within the field of view 10 of the camera 110. In particular, the processor 130 can even identify the time when each user is not only located within the field of view 10 of the camera 110 but also within a predetermined distance. As described above, this can be identified based on depth information included in the images. Therefore, the processor 130 can identify the time difference between multiple users 200 located within the field of view 10 (and the predetermined distance) of the camera 110. Furthermore, the processor 130 can identify only multiple users 200 whose time differences are within a predetermined time as belonging to the same user group. For example, refer to... Figure 4 The processor 130 can identify the time during which each of the six identified users 200-1 to 200-6 is located within the field of view 10 (and a predetermined distance) of the camera 110. Furthermore, the processor 130 can identify only multiple users 200 located within the field of view 10 (and a predetermined distance) of the camera 110 that have a time difference of 1 second or less between them as the same user group.

[0078] At this time, the processor 130 can identify the time during which each of the remaining users is within the field of view 10 (and predetermined distance) of the camera 110 based on the time a specific user among the plurality of users 200 is within the field of view 10 (and predetermined distance) of the camera 110. Furthermore, the processor 130 can identify at least one user within the field of view 10 (and predetermined distance) of the camera 110 within a predetermined time (e.g., 3 seconds) based on the time the specific user is within the field of view 10 (and predetermined distance) of the camera 110. Additionally, the processor 130 can include at least one identified user and the specific user in a user group. Here, the specific user can be the first user among the plurality of users 200 to be within the field of view 10 of the camera 110 and at a predetermined distance (a predetermined distance from the robot 100).

[0079] Furthermore, processor 130 can select multiple users from the multiple users 200 to be included in the same user group based on whether the multiple users 200 are conversing with each other (or whether they are looking at each other). Specifically, processor 130 can identify each user's face or directly detect each user's mouth and eyes based on multiple images acquired by camera 110. For example, processor 130 can use algorithms such as AdaBoost, HOG (Histogram of Oriented Gradients), and Haar cascade algorithms to detect a person's eyes and mouth in the images. Processor 130 can track the detected mouth and eyes of users to identify the object each user is looking at or the object each user is conversing with. Therefore, processor 130 can identify whether the multiple users 200 are conversing with each other (or whether they are looking at each other). In addition, processor 130 can recognize the speech of each user input through the microphone of robot 100, match each user with the user's speech, and then identify whether there is a conversation between the multiple users 200. For this purpose, processor 130 can use Hidden Markov Model (HMM) algorithms, Long Short-Term Memory (LSTM) models, CNN models, or Transformer models.

[0080] Processor 130 can identify multiple users 200 engaged in a conversation as the same user group. For example, refer to Figure 4 The processor 130 can identify multiple users 200 who are having conversations with only six identified users 200-1 to 200-6 as the same user group.

[0081] The processor 130 can select multiple users 200 to be included in the same user group from among the multiple users 200 based on the distance between the multiple users 200, the time during which the multiple users 200 are within the field of view 10 of the camera 110, and whether the multiple users 200 are having a conversation.

[0082] Specifically, the processor 130 can calculate the probability value of each user being included in the same user group based on the distance between the multiple users 200, the time during which the multiple users 200 are within the field of view 10 of the camera 110, and whether the multiple users 200 are having a conversation.

[0083] As one or more examples, processor 130 may calculate a first probability value for each user to be included in the same group based on the distance between the multiple users 200. In this case, as the distance between the multiple users 200 decreases, processor 130 may calculate a higher first probability value for a user to be included in the same user group. Specifically, processor 130 may calculate a higher first probability value for a user who is closer to the aforementioned specific user to be included in the same user group as that specific user.

[0084] Furthermore, the processor 130 can calculate a second probability value for each user based on the time that the multiple users 200 are within the field of view 10 of the camera 110. At this time, as the time difference between the multiple users 200 being within the field of view 10 of the camera 110 decreases, the processor 130 can calculate a higher second probability value for the multiple users 200 being included in the user group. Specifically, the processor 130 can calculate a higher second probability value for users whose time difference with the aforementioned specific user's time difference within the field of view 10 of the camera 110 is shorter.

[0085] Furthermore, the processor 130 can calculate a third probability value for each user in the same group based on whether the multiple users 200 are conversing with each other. In this case, the processor 130 can calculate a third probability value regarding whether the multiple users 200 are conversing with each other. That is, the processor 130 can identify whether the multiple users 200 are conversing with each other based on the acquired image and calculate a third probability value for each user that they will converse.

[0086] Furthermore, the processor 130 can obtain the probability value of each user being included in the same user group based on the calculated first probability value, second probability value, and third probability value. As one or more examples, the processor 130 can obtain the probability value of each user being included in the same user group by applying weights to the first probability value, the second probability value, and the third probability value, respectively. In particular, the processor 130 can apply a higher weight to the third probability value than the weights applied to the first and second probability values. That is, the processor 130 can identify that multiple users 200 engaging in a conversation are likely to be included in the same user group. Furthermore, the processor 130 can add the first probability value, the second probability value, and the third probability value with different weights applied to obtain the probability value of each user being included in the same user group. In addition, the processor 130 can identify only multiple users 200 whose obtained probability values ​​are greater than or equal to a predetermined value as belonging to the same user group.

[0087] For example, refer to Figure 4 Regarding the six identified users 200-1 to 200-6, processor 130 can obtain a probability value for each user being included in the same user group based on the distance between the six users 200-1 to 200-6, the time during which the six users 200-1 to 200-6 are within the field of view 10 of camera 110, and whether the six users 200-1 to 200-6 are interacting with each other. In this case, if the predetermined value is 0.5, processor 130 can identify four users 200-1 to 200-4 among the six users 200-1 to 200-6 who have a probability value of 0.5 or greater as being included in the same user group.

[0088] In the following description, we assume that robot 100 interacts with four users identified as being included in the same user group.

[0089] Return to reference Figure 3 The processor 130 can identify the rotation angle and rotation direction of the robot 100 based on the position of at least one target region 30 where at least one user 200 is located in one of a plurality of regions 20 within the field of view 10 of the camera and the order of gaze toward at least one target region 30 (S320).

[0090] Specifically, the processor 130 can identify multiple regions 20 within the camera's field of view 10. For example, the processor 130 can set a predetermined range within the camera's field of view 10 and divide the predetermined range into multiple regions 20 with predetermined sizes to identify them.

[0091] As one or more examples, processor 130 can identify multiple regions 20 by dividing the camera's field of view 10 at a predetermined angle. Here, the predetermined angle can be preset based on the performance of the camera's field of view 10. Alternatively, the predetermined angle can be set by processor 130 based on the number of users 200 identified within the camera's field of view 10. For example, processor 130 can decrease the size of the predetermined angle as the number of users 200 identified within the field of view 10 decreases.

[0092] Furthermore, the processor 130 can identify multiple regions 20 within a predetermined distance. Therefore, the processor 130 can identify multiple regions 20 with predetermined angles and sizes. That is, the multiple regions 20 can have the same size and can be identified within the camera's field of view 10 in a non-overlapping manner. The predetermined distance can be identified based on depth information included in the image acquired by the camera or distance information acquired by the robot 100's sensors.

[0093] Furthermore, the processor 130 can identify multiple regions 20 within the camera's field of view 10, and then identify at least one target region 30 in which at least one user 200 is located. Here, the target region 30 is the region in which at least one user 200 is located among the multiple regions 20, and can be an area in which the robot 100 can be rotated for viewing.

[0094] The robot 100's gaze at the target area 30 can be caused by the robot 100 or a portion of the robot 100 facing the target area 30. For example, the rotation of the robot 100's head to face the target area 30, controlled by the processor 130 with actuators (e.g., motors, etc.) located between the head and body of the robot 100, corresponds to the robot 100 gazing at the target area 30. Alternatively, the rotation of the robot 100's body to face the target area 30, controlled by the processor 130 with actuators (e.g., wheels, etc.) located under the body of the robot 100, corresponds to the robot 100 gazing at the target area 30. In this way, when the robot 100 or a portion of the robot 100 gazes at the target area 30 under the control of the processor 130, multiple users 200 located in the target area 30 may feel as if the robot 100 is actively interacting with the users 200.

[0095] Figure 5 This is a diagram illustrating the identification of a target region within a plurality of regions in the field of view of a camera according to one or more embodiments of the present disclosure.

[0096] In the following text, for the purpose of describing this disclosure, it is assumed that there are multiple users 200 and at least one target area 30.

[0097] Reference Figure 5 The processor 130 divides a 150° field of view 10 at a 30° angle within a distance of 2m and identifies five regions. Furthermore, the processor 130 can identify the target region 30 as the first region where one user 200 is located, the second region where two users 200 are located, and the fifth region where one user 200 is located. Therefore, the processor 130 can rotate the robot 100 to identify the first, second, and fifth regions as areas that the robot 100 can gaze upon.

[0098] After identifying the target region 30, the processor 130 can identify the rotation angle and rotation direction of the robot 100 based on the multiple target regions 30 where the multiple users 200 are located and the order of gazes toward the multiple target regions 30.

[0099] Specifically, the processor 130 can identify the rotation angle and rotation direction of the robot 100 based on the position of the target region 30 within the field of view 10. Since multiple regions 20 are identified in a non-overlapping manner within the field of view 10 as described above, the processor 130 can identify the position of the identified target region 30 among the multiple regions 20 within the camera's field of view 10. Furthermore, the processor 130 can identify the rotation direction of the robot 100 based on the position of the identified target region 30. Since the multiple regions 20 are divided at the same predetermined angle, the processor 130 can identify the rotation angle at which the robot 100 should be rotated so that the robot 100 looks at (or faces) the target region 30.

[0100] Return to reference Figure 5 Since the first and second regions among the multiple target regions 30 (i.e., the first target region, the second target region, and the fifth target region) are identified as being located to the left of the robot 100, the processor 130 can identify the rotation direction of the robot 100 relative to the first and second regions as the left direction. Furthermore, the processor 130 can identify the rotation angle of the robot 100 relative to the first region as 60° and the rotation angle of the robot 100 relative to the second region as 30°.

[0101] Since the fifth target region among the multiple target regions 30 (i.e., the first target region, the second target region, and the fifth target region) is identified as being located to the right of the robot 100, the processor 130 can identify the rotation direction of the robot relative to the fifth target region as the rightward direction. Furthermore, the processor 130 can identify the rotation angle of the robot 100 relative to the first target region as 60°.

[0102] However, when multiple target regions 30 are identified, the processor 130 can consider the positions and gaze order of the multiple target regions 30 to identify the rotation direction and rotation angle of the robot 100 relative to each target region 30. Here, the gaze order can be the order in which the robot 100 gazes at the multiple target regions 30.

[0103] As one or more examples, the gaze order can be sequentially set for multiple target regions 30 based on a predetermined rotation direction. For example, refer to Figure 5 When the predetermined rotation direction is clockwise, the order of gaze for multiple target areas (i.e., the first area, the second area, and the fifth area) can be the order of the first area, the second area, and the fifth area.

[0104] As one or more examples, the gaze order can be set based on the distance between the user 200 and the robot 100 located in the target area 30. Specifically, the processor 130 can identify the distance between at least one user 200 and the robot 100 based on the position of at least one user 200 located in each target area 30. Here, the position of the user 200 can be identified based on depth information included in the acquired image or distance information acquired by a sensor.

[0105] Furthermore, the processor 130 can identify distance information corresponding to the target region 30 based on the distance between at least one user 200 and the robot 100. Moreover, based on the identified distance information, the processor 130 can determine the gaze order for the target region 30 according to the order of the closest distances between the users 200 and the robot 100. At this time, when multiple users 200 are located in the target region 30, the processor 130 can identify the distance information of the target region 30 as the distance of the user 200 closest to the robot 100 among the multiple users 200.

[0106] Furthermore, as one or more examples, the gaze order can be set based on the interaction scores of users 200 located in the target area 30. Here, the interaction score can be a score calculated based on the frequency, duration, etc., of user 200 interacting with robot 100. Below, one or more embodiments of this disclosure for determining the gaze order based on the interactions of each user 200 will be described.

[0107] Figure 6 This is a flowchart illustrating a method for determining gaze order based on interaction scores according to one or more embodiments of the present disclosure. Figure 7a and Figure 7b This is a diagram illustrating a method for determining gaze order based on interaction scores according to one or more embodiments of the present disclosure. Figure 6 Operations S605 and S635 to S655 shown herein can respectively correspond to Figure 3 Operations S310 to S360 are shown in the figure.

[0108] Processor 130 may first acquire interaction information from multiple users 200. In this regard, according to one or more embodiments of this disclosure, processor 130 may identify the duration of gaze of multiple users 200 toward robot 100 based on images acquired by camera 110 (S610). Specifically, processor 130 may identify the face of each user or directly detect the user's eyes based on multiple images acquired by camera 110. For example, processor 130 may use algorithms such as AdaBoost, HOG (Histogram of Oriented Gradients), and Haar cascade algorithms to detect human eyes in images. Furthermore, processor 130 may track the eyes of users detected in multiple images to identify whether each user is gazing at robot 100 and the time spent gazing at robot 100.

[0109] Furthermore, the processor 130 can identify the number of voice inputs from multiple users 200 to the robot 100 based on user voices acquired through the microphone of the robot 100 (S615). Specifically, the processor 130 can identify the voices of multiple users 200 input through the microphone of the robot 100, match each user with the identified user voice, and then identify the number of voice inputs from multiple users 200. For this purpose, the processor 130 can use a Hidden Markov Model (HMM) algorithm, a Long Short-Term Memory (LSTM) model, a CNN model, or a Transformer model.

[0110] At this point, processor 130 can calculate the interaction score for multiple users 200 based on the recognized gaze duration and the number of voice inputs (S620). Specifically, processor 130 can calculate an interaction score corresponding to the gaze duration and the number of voice inputs for each user. As one or more examples, processor 130 can calculate a first interaction score corresponding to the gaze duration of each user towards robot 100. In particular, as the gaze duration of a user towards robot 100 increases, processor 130 can calculate a higher value for the first interaction score. Furthermore, processor 130 can calculate a second interaction score corresponding to the number of voice inputs from each user towards robot 100. In particular, as the number of voice inputs increases, processor 130 can calculate a higher value for the second interaction score.

[0111] In addition, the processor 130 can identify the number of touch inputs from the user 200 via the display of the robot 100, and calculate a third interaction score corresponding to the number of identified touch inputs from each user 200.

[0112] Processor 130 can calculate an interaction score for each user by adding a first interaction score and a second interaction score. (See reference...) Figure 7a and Figure 7bThe processor 130 calculated interaction scores of 0.66, 0.55, 0.59 and 0.78 based on the duration of gaze at the robot 100 by four users 200-1 to 200-4 and the amount of voice input, respectively.

[0113] In addition, the processor 130 can identify the interaction score corresponding to each target area based on the interaction score of at least one user located in each target area (S625).

[0114] As one or more examples, processor 130 can identify the sum of interaction scores of multiple users 200 located in each target region as a region interaction score corresponding to each target region. Specifically, see... Figure 7a The processor 130 can identify the region interaction score corresponding to each target region based on the interaction scores of 0.66, 0.55, 0.59, and 0.78 for the four users 200-1 to 200-4. At this point, the processor 130 can identify 1.14, obtained by adding the interaction scores of two users (0.55 and 0.59), as the interaction score corresponding to the second target region where the two users are located.

[0115] As another example, processor 130 can identify the interaction score corresponding to the highest value among the interaction scores of multiple users 200 located in each target region as the interaction score corresponding to each target region. Specifically, see... Figure 7a The processor 130 can identify the interaction scores corresponding to the second target areas where the two users are located as user interaction scores with high values ​​(0.59).

[0116] Furthermore, the processor 130 may determine the gaze order of the plurality of target regions 30 based on the interaction score corresponding to each identified target region (S630). Specifically, the processor 130 may determine the gaze order of the plurality of target regions 30 sequentially, starting from the highest interaction score.

[0117] For example, such as Figure 7a As shown, when identifying the interaction score corresponding to each target region based on the sum of the interaction scores of multiple users 200 located in each target region, the processor 130 can determine the gaze order of the multiple target regions 30 (i.e., the first region, the second region, and the fifth region) as the order of the second region, the fifth region, and the first region.

[0118] As another example, such as Figure 7bAs shown, when identifying the interaction score corresponding to each target region based on the interaction score corresponding to the highest value among the interaction scores of multiple users 200 located in each target region, the processor 130 can determine the gaze order of the multiple target regions 30 (i.e., the first region, the second region, and the fifth region) as the order of the fifth region, the first region, and the second region.

[0119] Processor 130 can determine the gaze order by considering the distance between the user and robot 100 located in each target area and the interaction score of each user. Specifically, processor 130 can identify the master user with the highest interaction score among the multiple users 200 based on their interaction scores. At this time, processor 130 can determine the gaze order such that the target area where the master user is located is the first gaze target area. In addition, processor 130 can determine the gaze order for the remaining target areas according to the order of the closest distance between the user and robot 100 located in each target area 30. However, this disclosure is not limited to this, and processor 130 can determine the gaze order in various ways based on the distance between the user and robot 100 located in each target area and the interaction score of each user.

[0120] In the following text, for the purpose of describing this disclosure, it is assumed that the processor 130 determines the gaze order of the multiple target regions in the order of the fifth region, the first region, and the second region.

[0121] The processor 130 can rotate the robot 100 to sequentially gaze at multiple target regions based on the gaze order of multiple targets. For example, if for Figure 5 If the gaze order of multiple target regions in the process is the fifth region, the first region, and the second region, then the processor 130 can control the driver to make the robot 100 gaze at the fifth region first, then the first region, and finally the second region.

[0122] To this end, processor 130 may consider the gaze sequence to identify the rotation direction and rotation angle for each target region 30. Specifically, in order to control the actuator so that robot 100 gazes at one target region 30 after gazing at another target region 30 among a plurality of target regions 30, processor 130 should consider the position of the target region 30 in the previous sequence of gaze to identify the rotation direction and rotation angle of robot 100 for the target region 30 in the next sequence.

[0123] Figure 8 This is a flowchart illustrating a control method for identifying the rotation direction and rotation angle of multiple target regions according to the gaze sequence, based on one or more embodiments of the present disclosure. Figure 8 S510, S550 to S580 shown in the figure can correspond to Figure 3S310, S330 to S360 are shown in the figure. Figure 9 and Figure 10 This is a diagram illustrating the identification of the rotation direction and rotation angle of multiple target regions according to the order of gaze, based on one or more embodiments of the present disclosure.

[0124] Reference Figure 8 According to one or more embodiments of the present disclosure, the processor 130 may identify a first target region 30 among a plurality of target regions 30 based on the gaze sequence, and identify the rotation angle and rotation direction of the robot 100 corresponding to the first target region 30 based on the position of the first target region 30 (S830).

[0125] Here, the first target region 30 can be the target region 30 corresponding to the first order in the gaze sequence. In addition, the rotation angle and rotation direction corresponding to the first target region 30 can be the rotation angle and rotation direction required for the robot 100 to face the first target region 30.

[0126] Specifically, the processor 130 can identify the rotation angle and rotation direction corresponding to the first target region 30 based on the posture of the robot 100 when identifying multiple target regions 30. At this time, the processor 130 can identify the position of the first target region 30 based on the posture of the robot 100 when identifying multiple target regions 30, and identify the rotation angle and rotation direction corresponding to the first target region 30 based on the identified position of the first target region 30.

[0127] Specifically, the processor 130 can identify the rotation angle and rotation direction corresponding to the first target region 30 based on a predetermined angle used to divide the field of view 10 of the camera.

[0128] Reference Figure 9 The processor 130 can identify the first target region 30 as the fifth region according to the gaze sequence, and identify the position of the fifth region. After identifying the position of the fifth region among the plurality of regions 20, the processor 130 can identify the rotation angle and rotation direction required for the robot 100 to face the fifth region from its current pose. In addition, the processor 130 can identify the rotation angle for the fifth region as 60°, and the rotation direction as clockwise. At this time, the rotation angle for the fifth region (60° = 2 × 30°) can be identified based on a predetermined angle (30°) used to divide the field of view of the camera.

[0129] After identifying the rotation direction and rotation angle corresponding to the first target region 30, the processor 130 can identify the rotation direction and rotation angle for the remaining target region 30 (S840). At this time, the remaining target region 30 can be the target region 30 corresponding to the sequence following the first target region 30 in the gaze sequence.

[0130] In this regard, according to one or more embodiments of the present disclosure, the processor 130 may identify the remaining rotation angle and remaining rotation direction of the robot 100 corresponding to the remaining target region 30 based on the position of the previous sequence of target regions 30 and the position of the remaining target region 30 among a plurality of target regions 30 (S840).

[0131] In the case of the remaining target areas 30 after the first target area 30, the processor 130 should control the robot 100 to face a specific target area 30, and then sequentially control the robot 100 to face another target area 30. Therefore, the processor 130 should consider the position of the target areas 30 in the previous sequence to identify the rotation direction and rotation angle for each of the target areas 30.

[0132] Specifically, for the remaining target region 30, the processor 130 can identify the position of the target region 30 corresponding to the previous sequence in the gaze sequence. Furthermore, based on the positions of the target regions 30 in the previous and next sequences, the processor 130 can identify the rotation angle and direction required for the robot 100 to gaze at the target region 30 in the previous sequence and then at the target region 30 in the next sequence. At this point, the processor 130 can identify the rotation angle and direction corresponding to the remaining target region 30 based on a predetermined angle for dividing the field of view 10 of the camera.

[0133] Reference Figure 9 and Figure 10 The processor 130 can identify the rotation direction and rotation angle of the first target region 30 corresponding to the first order in the gaze sequence, and then identify the rotation direction and rotation angle for the remaining target regions 30 after the first order (i.e., the first region and the second region). At this time, the processor 130 can identify the first region as the second target region 30 and the second region as the third target region 30 based on the gaze sequence.

[0134] Reference Figure 9 The processor 130 can consider the position of the first region and the position of the fifth region (i.e., the first target region 30) in the preceding sequence (i.e., the first sequence) to identify the rotation direction and rotation angle of the robot 100 for the first region, which is the second target region 30 in the gaze sequence.

[0135] Specifically, the processor 130 can consider the positions of the fifth region and the first region to identify the rotation direction and rotation angle for rotating the robot 100 from the fifth region to the first region. At this time, the processor 130 can identify the rotation direction for the first region as counterclockwise and the rotation angle as 120°.

[0136] Then, refer to Figure 10 The processor 130 can identify the rotation direction and rotation angle of the robot 100 for the second region, which is the third target region, by taking into account the position of the second region and the position of the first region (i.e., the second sequence) preceding the second region.

[0137] Specifically, the processor 130 may consider the positions of the first region and the second region to identify the rotation direction and rotation angle for rotating the robot 100 from the first region to the second region. Furthermore, the processor 130 may identify the rotation direction for the second region as clockwise and the rotation angle as 30°.

[0138] The processor 130 can control the actuator to make the robot 100 rotate based on the recognized rotation angle and rotation direction (S330, S550).

[0139] At this time, the processor 130 can control the actuators to make the robot 100 face each target area 30 based on the rotation angle and rotation direction. As described above, the processor 130 can control the actuators to make the head or body of the robot 100 sequentially gaze at multiple target areas 30.

[0140] Therefore, multiple users 200 located in each target area 30 can recognize that the robot 100 is actively interacting with them. Specifically, in the case of existing robots 100, when multiple users 200 are around the robot 100, the robot 100 may only look at a specific user 200 or look at users 200 indiscriminately, thus making it difficult for the multiple users 200 to perceive that they are interacting appropriately with the robot 100. However, in this disclosure, the robot 100 does not only look at a specific user 200 among the multiple users 200, but rather looks at the multiple users 200 sequentially and repeatedly, so that the multiple users 200 using the robot 100 can perceive that they are actively and smoothly interacting with the robot 100.

[0141] The processor 130 can control the robot 100 to gaze at each target region 30 for a predetermined time. Hereinafter, the time set for the robot 100 to gaze at the target region 30 will be referred to as the gaze duration. When the robot 100 rotates toward the target region 30 via a control actuator, the processor 130 can control the actuator to stop the robot 100 from gazing at the target region 30 for the predetermined gaze duration. Therefore, the robot 100 can stand stationary facing the target regions 30 relative to multiple target regions 30 for a predetermined gaze duration.

[0142] The gaze duration for each target region 30 can be set differently. As one or more examples, the processor 130 can identify an interaction score corresponding to each target region based on the interaction score of at least one user 200 located in each target region 30, and determine the region gaze duration for each target region based on the interaction score corresponding to each target region. In particular, the processor 130 can determine the region gaze duration for each target region proportionally to the interaction score corresponding to each target region. That is, as the interaction score corresponding to the target region increases, the processor 130 can set the gaze duration for the target region to be longer.

[0143] Furthermore, as one or more examples, the processor 130 may set the gaze duration for each target region 30 based on the number of users 200 located in each target region. Specifically, the processor 130 may determine the gaze duration for each target region proportionally to the number of users 200 located in each target region 30. That is, as the number of users 200 located in the target region 30 increases, the processor 130 may set the gaze duration for the target region to be longer.

[0144] The processor 130 controls the actuators to enable the robot 100 to gaze at each target area for a determined duration.

[0145] Processor 130 can control the actuator to cause robot 100 to repeatedly gaze at multiple target regions 30 according to a gaze sequence based on a recognized (first) rotation angle and (first) rotation direction. Describing the example above again, if the gaze sequence is the order of fifth region, first region, and second region, then processor 130 can control the actuator to cause robot 100 to gaze at the second region as the last in the gaze sequence, and then control the actuator to cause robot 100 to face the fifth region again. More specifically, if the gaze sequence of fifth region, first region, and second region is referred to as one cycle (or one loop), then processor 130 can control the actuator to cause robot 100 to periodically (or repeatedly) gaze at the fifth region, first region, and second region. The gaze sequence may include a gaze sequence for target regions 30 for multiple cycles. That is, Figure 9 and Figure 10 The illustration shows a gaze sequence that includes only one cycle, but according to an embodiment, it may also include a gaze sequence that includes a second and third cycle after the first cycle.

[0146] The processor 130 can identify a new user 210 while the robot 100 rotates based on the identified rotation angle and direction. Here, the new user 210 can be a user 200 that was not previously identified by the processor 130 but is now being identified. For example, as the robot 100 rotates, the camera's field of view 10 can change, and at this time, the newly identified user 200 within the changed field of view 10 can correspond to the new user 210. In particular, the new user 210 can also be a user 200 identified as being included in the same user group as multiple users 200.

[0147] Optionally, a new user 210 may have already been identified by the processor 130 as a user 200 who has been located around the robot 100 multiple times, but has been identified as not included in the same user group and excluded. In this case, if the excluded user 200 is identified as being delayed in joining the same user group through interactions with multiple users 200 included in the user group, the processor 130 may identify the excluded user 200 as the new user 210.

[0148] According to one or more embodiments of this disclosure, when a new user 210 is identified based on an image acquired by a camera while the robot 100 is rotating, the processor 130 may adjust the gaze order such that a new target region 31 in which the new user 210 is located (e.g., generating a second gaze order) in a plurality of regions 20 within the field of view 10 of the camera of the rotated robot 100 is included in the gaze order (S360).

[0149] Specifically, when a new user 210 is identified based on the rotation of the robot 100, in addition to the multiple previously identified users 200, the processor 130 can control the robot 100 to look at the new user 210. In particular, if the new user 210 is identified as being included in the same user group as the multiple existing users 200, the processor 130 can include the new user 210 in the user group and can control the robot 100 to look at both the new user 210 and the multiple users 200.

[0150] To this end, processor 130 can adjust the existing gaze order. Specifically, processor 130 can adjust the existing gaze order when the new user 210 is located in a new target region 31 instead of multiple existing identified target regions 30. Furthermore, when the new target region 31 is identified not in multiple existing identified regions 20 but in different locations (or multiple different regions 20), processor 130 can adjust the gaze order so that the new target region 31 is included in the gaze order.

[0151] In the following text, reference will be made to Figures 11 to 16 A detailed description of embodiments of this disclosure relating to it is provided.

[0152] Figure 11This is a diagram illustrating the identification of a new area where a new user is located when a new user is identified, according to one or more embodiments of the present disclosure. Figure 12 This is a flowchart illustrating a control method for re-identifying multiple target areas with a (second) rotation angle and (second) rotation direction when a new user is identified. Figure 12 S1210, S1220, S1230, S1270 and S1280 shown can respectively correspond to Figure 3 S310, S320, S330, S350 and S360 are shown in the figure.

[0153] The processor 130 can identify a new user 210 located around the robot 100 based on images acquired by a camera while the robot 100 is rotating (S1240). Specifically, the processor 130 can store feature information about multiple users 200 in a memory and identify the new user 210 in the images acquired by the robot 100 through the camera based on the stored feature information. After acquiring feature information about user 200 in the image, the processor 130 can compare the feature information about user 200 with the feature information about multiple users 200 stored in the memory to identify whether user 200 is one of multiple existing identified users 200 or a new user 210. In particular, if the acquired feature information is identified as different from the feature information stored in the memory, the processor 130 can identify the user 200 corresponding to the acquired feature information as a new user 210.

[0154] Reference Figure 12 When a new user 210 is identified (S1240), the processor 130 can divide the field of view 10 of the robot 100's camera by a predetermined angle to identify multiple regions 20 (S1250). Furthermore, the robot 100 can identify a new target region 31 within the multiple regions 20 where the new user 210 is located. In this respect, the description of this disclosure given above can be applied in the same manner, and therefore a detailed description will be omitted.

[0155] Specifically, refer to Figure 11 When the robot 100, rotating according to the gaze sequence, gazes at the fifth region, the processor 130 can acquire multiple images via the camera. Furthermore, the processor 130 can identify a new user 210 around the robot 100 based on the acquired images. The new user 210 could be a user 200-8 who was outside the camera's field of view before the robot rotated.

[0156] At this time, when a new user 210 is identified, the processor 130 can divide the camera's field of view 10 at a predetermined angle to identify multiple regions 21 while the robot 100 is looking at the fifth region. Furthermore, the processor 130 can identify the sixth region among the multiple regions 21 as the target region 30 where the new user 210 is located.

[0157] Reference Figure 11 The multiple regions 20 identified while the robot 100 is rotating may differ from the multiple existing regions 20 because the multiple regions 20 are identified when the robot 100 is gazing at the target region 30. However, for ease of description, one or more embodiments of this disclosure are described with respect to the multiple new regions 21 using repeating reference numerals added to the multiple existing regions 20.

[0158] Furthermore, the processor 130 can adjust the gaze order so that the new target region 31 is included in the gaze order (S1260). At this time, the processor 130 can adjust the gaze order based on the distance between the new user 210 and the robot 100 or the interaction score of the new user 210.

[0159] As one or more examples, processor 130 may adjust the gaze order between target region 30 (or region planned to be gazed upon) (hereinafter, unprocessed target region) that robot 100 has not yet gazed upon in the gaze order and a new target region 31.

[0160] As one or more examples, processor 130 may adjust the gaze order between the unprocessed target area and the new target area 31 based on the interaction scores of user 200 located in the unprocessed target area and new user 210 located in the new target area 31. Processor 130 may calculate the interaction score of new user 210 located in the new target area 31. Specifically, based on images acquired by camera 110 while the robot 100 is rotating, processor 130 may identify the gaze duration of new user 210 towards robot 100. Furthermore, processor 130 may identify the amount of voice input from new user 210 towards robot 100 based on voice acquired by microphone. Additionally, processor 130 may also identify the amount of touch input from new user 210 via display. Furthermore, processor 130 may calculate the interaction score of the new user based on the gaze duration and the amount of voice input (and touch input). In this regard, the description of embodiments of the present disclosure given above is equally applicable, and therefore a detailed description will be omitted.

[0161] The processor 130 can compare the interaction scores corresponding to the unprocessed target region and the new target region 31, respectively, and adjust the gaze order between the unprocessed target region and the new target region 31. Since the description of the embodiments of this disclosure above is equally applicable to methods for adjusting the gaze order between the new target region 31 and the unprocessed target region based on interaction scores, a detailed description will be omitted.

[0162] Furthermore, the processor 130 can adjust the gaze order between the unprocessed target area and the new target area 31 by comparing the distance between the user 200 and the robot 100 located in the unprocessed target area and the distance between the user 200 and the robot 100 located in the new target area 31. In this regard, the description of the embodiments of this disclosure given above can be applied in the same way, and therefore a detailed description will be omitted.

[0163] Figure 13 This is a diagram illustrating that, according to one or more embodiments of the present disclosure, a new target region is not included in the gaze sequence when the rotation angle of the robot used for gazing at a new target region is greater than or equal to a predetermined critical angle.

[0164] According to one or more embodiments of this disclosure, when the rotation angle of the robot 100 used for gazing at a new target region 31 is less than a predetermined critical angle, the processor 130 may adjust the gaze order so that the new target region 31 is included in the gaze order. Therefore, if the rotation angle of the robot 100 used for gazing at the new target region 31 is equal to or greater than the predetermined critical angle, the processor 130 may not include the new target region 31 in the gaze order and maintain the gaze order.

[0165] Specifically, if the processor 130 adjusts the gaze order to include the new target area 31 whenever a new user and the new target area 31 where the new user is located are identified while the robot 100 is rotating, there may be a problem that reduces the opportunity for multiple existing users to interact with the robot 100 (or increases the waiting time for interaction).

[0166] Therefore, the processor 130 can set a critical angle for the rotation angle of the robot 100 when the gaze order is determined for the first time. Furthermore, whenever a new user is identified, the processor 130 can determine whether the rotation angle of the robot 100 used to gaze at the new target area 31 where the new user is located is within the set critical angle. For example, the processor 130 can set a critical range 40 when the gaze order is determined for the first time. Here, the critical range 40 can be set based on the robot 100's posture, the field of view 10, the angles of the multiple areas 20, the critical angle of the robot's rotation angle, etc., when the gaze order is determined for the first time.

[0167] Furthermore, the processor 130 can identify whether the position of the new target region 31 is within a predetermined critical range 40, thereby identifying whether the rotation angle of the robot 100 used to gaze at the new target region 31 is within a set critical angle. If the position of the new target region 31 is identified as being within the predetermined critical range 40, the processor 130 can adjust the gaze order so that the new target region 31 is included in the gaze order. If the position of the new target region 31 is identified as being outside the predetermined critical range 40, the processor 130 can exclude the new target region 31 from the gaze order and maintain the gaze order.

[0168] Reference Figure 13 The processor 130 can set a critical range 40 related to the rotation angle of the robot 100 to 210° when the robot 100 determines the first gaze sequence. At this time, when the robot 100 gazes at the user in the fifth region, the processor 130 can identify the target region where the newly identified user 220 is located among multiple new regions 21. At this time, when the new target region 31 is the seventh region, the processor 130 can identify the seventh region as being outside the set critical range 40 (210°). Therefore, the processor 130 can determine that the robot 100 is not gazing at the new target region 31 (the seventh region). That is, the processor 130 can exclude the new target region 31 (the seventh region) from the gaze sequence and can maintain the existing gaze sequence of the target regions (the first region, the second region, and the fifth region) as is.

[0169] After adjusting the gaze sequence, the processor 130 can re-identify the rotation angle and rotation direction of the robot 100 based on the position of the new target region 31 and the adjusted gaze sequence (S1270). In addition, the processor 130 can control the actuator to make the robot 100 rotate based on the re-identified rotation angle and rotation direction (S1280).

[0170] For example, when a new user 210 is identified based on an image acquired by a camera on a rotated robot 100, the processor 130 can identify the rotation angle and direction of the robot 100 corresponding to the new target region 31, based on the position of the new user 210 in the new target region 31 among the multiple regions 20. Furthermore, the processor 130 can control a actuator to cause the robot 100 to rotate based on the rotation angle and direction of the robot 100 corresponding to the identified target region 30.

[0171] Specifically, refer to Figure 11If a new user 210 identified while the robot 100 is looking at the fifth region is identified as being located in the sixth region, the processor 130 can identify the location of the newly identified target region 31 (i.e., the sixth region). At this point, the processor 130 can identify the sixth region as being located in the fourth of a plurality of newly identified regions 21, and identify the rotation angle and direction of the sixth region as clockwise 30°. Furthermore, the processor 130 can control the actuators to cause the robot 100 to also look at the newly identified target region 31 and a plurality of existing target regions 30 based on the rotation angle and direction of rotation relative to the identified sixth region.

[0172] However, when the gaze sequence is adjusted as a new target region 31 is identified, the rotation direction and angle relative to the existing target regions 30 may change. Furthermore, the processor 130 should, based on the order of the new target regions 31 in the adjusted gaze sequence, consider the positions of the other target regions 30 and the position of the new target region 31, and identify the rotation direction and angle of the robot 100 relative to the new target region 31.

[0173] To this end, as one or more examples, processor 130 may identify the next target region in the next order of the target region 30 being gazed at by the rotated robot 100, based on the adjusted gaze order, in the unprocessed target region and the new target region 31.

[0174] Here, the next target region can be the next sequential target region 30 of the region that the robot is currently looking at when the processor 130 identifies the new user 210 and the new target region where the new user 210 is located while the robot 100 is rotating. At this time, the next target region can be identified based on the adjusted gaze order.

[0175] Specifically, refer to Figure 11 When the rotated robot 100 gazes at the fifth of multiple target areas 30, the processor 130 identifies a new user 210 around the robot 100. At this time, the processor 130 can adjust the gaze order to include the sixth area, which is a new target area 31, in the gaze order. In addition, the processor 130 can identify the next target area 30 after the fifth area that the robot 100 is gazing at in the adjusted gaze order.

[0176] Figure 14 , Figure 15 and Figure 16 This is a diagram illustrating the re-identification of the robot's rotation direction and rotation angle based on the position of a new target region according to one or more embodiments of this disclosure.

[0177] For example, refer to Figure 14When robot 100 adjusts its existing gaze sequence (fifth region → first region → second region) to include a sixth region as the new target region 31, the next target region within the adjusted gaze sequence (fifth region → first region → second region → sixth region) can be the first region in the sequence following the fifth region. That is, even when following the gaze sequence adjusted to be the same as the existing gaze sequence, processor 130 can control the actuator to make robot 100 gaze at the first region after gazing at the fifth region, and to rotate robot 100 toward the first region.

[0178] Reference Figure 15 When robot 100 adjusts its existing gaze sequence (fifth region → first region → second region) to include a sixth region as a new target region 31, the next target region within the adjusted gaze sequence (fifth region → sixth region → first region → second region) can be the sixth region in the sequence following the fifth region. In this case, processor 130 can control the driver to rotate robot 100 toward the sixth region after gazing at the fifth region, so as to gaze at the sixth region instead of the first region, according to the adjusted gaze sequence.

[0179] The processor 130 can re-identify the rotation angle and rotation direction of the robot 100 corresponding to the next target region based on the position of the next target region. In addition, the processor 130 can re-identify the rotation angle and rotation direction of the robot 100 corresponding to the remaining target region 30 based on the position of the target region 30 in the previous sequence of the remaining target region 30 after the next target region in the unprocessed target region and the position of the remaining target region 30 in the new target region 31.

[0180] Specifically, the processor 130 can identify the rotation angle and rotation direction corresponding to the next target region based on the posture of the robot 100 when identifying the new target region 31. At this time, the processor 130 can identify the position of the next target region based on the position of the target region 30 that the robot 100 is looking at when identifying the new target region 31. Furthermore, the processor 130 can identify the rotation angle and rotation direction corresponding to the next target region based on the identified position of the next target region. In particular, the processor 130 can identify the rotation angle and rotation direction corresponding to the first target region 30 based on a predetermined angle used to divide the field of view 10 of the camera.

[0181] Reference Figure 14The processor 130 can identify the next target region in the adjusted gaze sequence as the first region and identify the position of the first region. Specifically, the processor 130 can identify the position of an unidentified first region within the field of view 10 based on the rotation direction and rotation angle relative to the first region before adjusting the gaze sequence. Furthermore, the processor 130 can identify the rotation angle and rotation direction required for the robot 100, which is gazing at the fifth region, to face the first region as the next target region. Additionally, the processor 130 can identify the rotation angle relative to the first region as 120° and the rotation direction as clockwise. At this time, the rotation angle relative to the first region (120° = 4 × 30°) can be identified based on a predetermined angle (30°) used to divide the field of view 10 of the camera. That is, the rotation direction and rotation angle of the robot 100 relative to the first region can remain unchanged because even within the adjusted gaze sequence, the next target region after the fifth region is maintained as the first region.

[0182] Reference Figure 15 The processor 130 can identify the next target region in the adjusted gaze sequence as the sixth region and identify the position of the sixth region. The processor 130 can identify multiple regions 20 within the camera's field of view 10 while the robot 100 is gazing at the fifth region, and identify the position of the sixth target region 30 within the multiple identified regions 20. Furthermore, the processor 130 can identify the rotation angle and rotation direction required for the robot 100, which is gazing at the fifth region, to face the sixth region as the next target region. Additionally, the processor 130 can identify the rotation angle for the sixth region as 30° and the rotation direction as clockwise.

[0183] At this point, the rotation angle (30° = 1 × 30°) for the sixth region can be identified based on a predetermined angle (30°) used to divide the field of view 10 of the camera. That is, since the next target region after the fifth region actually changes from the first region to the sixth region within the adjusted gaze sequence, the angle and direction of the rotation of the robot 100 have changed after the gaze of the fifth region.

[0184] After identifying the next rotation direction and the next rotation angle corresponding to the next target region, the processor 130 can identify the rotation direction and rotation angle for the remaining target region 30. At this time, the remaining target region 30 can be the target region 30 corresponding to the next target region in the gaze sequence. The description of identifying the rotation direction and rotation angle for the remaining target region 30 after the first target region 30 described above can be applied in the same way.

[0185] Specifically, if the next target region is an unprocessed target region, the processor 130 can identify the rotation angle and rotation direction of the robot 100 corresponding to the new target region 31 based on the position of the new target region 31 and the positions of the unprocessed target regions preceding the new target region 31. Specifically, if the next target region is an unprocessed target region, the processor 130 can consider the position of the previous target region 30 added to the adjusted gaze sequence and the position of the new target region 31 to identify the rotation direction and rotation angle for the new target region 31.

[0186] Return to reference Figure 14 The processor 130 can identify the gaze order of the sixth region within the adjusted gaze order. Furthermore, the processor 130 can identify the second region as the target region 30 with a third gaze order, which is the order preceding the sixth region with a fourth gaze order. After identifying the position of the second region, the processor 130 can determine the rotation direction and angle for rotating the robot 100 from the second region to the sixth region based on the positions of the second and sixth regions.

[0187] Specifically, processor 130 can identify the position of the second region based on the rotation angle and direction relative to the identified second region before adjusting the gaze sequence. Furthermore, processor 130 can, based on the position of the second region and taking into account the position of the sixth region among the plurality of regions 21, identify the rotation direction and rotation angle of the robot 100 after gazing at the second region to gaze at the sixth region. In this case, processor 130 can identify the rotation direction and rotation angle of the robot 100 relative to the sixth region as 120° clockwise.

[0188] Furthermore, the processor 130 can control the driver to rotate the robot 100 to gaze at the new target region 31 after gazing at the unprocessed target region (i.e., the second region) before the new target region 31 (i.e., the sixth region) according to the adjusted gaze order.

[0189] If the next target area is a new target area 31, the processor 130 can identify the rotation angle and rotation direction of the robot 100 corresponding to the new target area 31 based on the position of the new target area 31, and re-identify the rotation angle and rotation direction of the robot 100 corresponding to the unprocessed target area based on the position of the new target area 31.

[0190] Specifically, the next target region can be a new target region 31, and the processor 130 can take into account the position of the new target region 31 and re-identify the rotation direction and rotation angle of the target regions 30 in the order after the new target region 31 added to the adjusted gaze order.

[0191] Reference Figure 15 and Figure 16 The processor 130 can identify the target region 30 in the next order after the sixth region within the adjusted gaze sequence. Furthermore, the processor 130 can identify the first region as the target region 30 with a third gaze sequence, which is the sequence following the sixth region with a second gaze sequence. Therefore, after identifying the position of the sixth region, the processor 130 can identify the rotation direction and rotation angle for rotating the robot 100 from the sixth region to the first region based on the positions of the first and sixth regions.

[0192] Specifically, processor 130 can identify the position of the first region based on the rotation angle and direction relative to the identified first region before adjusting the gaze sequence. Furthermore, processor 130 can identify the rotation direction and angle of the robot 100 after gazing at the sixth region to gaze at the first region, based on the position of the sixth region among the plurality of regions 21, taking into account the position of the first region. In this case, processor 130 can identify the rotation direction and angle of the robot 100 relative to the first region as 150° counterclockwise.

[0193] Furthermore, the processor 130 can control the drive to rotate the robot 100 to gaze at the new target area 31 (i.e., the sixth area), and then gaze at the first area based on the rotation angle and rotation direction (i.e., 150° counterclockwise) of the robot 100 corresponding to the next target area 30 (i.e., the first area) according to the adjusted gaze order.

[0194] Figure 17 This is a detailed configuration diagram of a robot according to one or more embodiments of the present disclosure.

[0195] Reference Figure 17 The robot 100 includes a camera 110, a actuator 120, a display 140, a memory 150, a sensor 160, a speaker 170, a microphone 180, and a communication interface 190. (The remaining text is omitted.) Figure 12 The one shown in the middle is the same as Figure 2 The component shown is a repeated component. Detailed description of the component.

[0196] Display 140 can display various visual information. As one or more examples, display 140 can display various information requested by multiple users. For this purpose, display 140 can be implemented as a display including a self-emissive device or a display including a non-self-emissive device and a backlight. For example, display 140 can be implemented as various types of displays, such as liquid crystal display (LCD), organic light-emitting diode (OLED) display, light-emitting diode (LED), micro LED, mini LED, plasma display panel (PDP), quantum dot (QD) display, quantum dot light-emitting diode (QLED), etc.

[0197] The display 140 may also include driving circuitry and a backlight unit that can be implemented in the form of a-si TFT, low-temperature polysilicon (LTPS) TFT or organic TFT (OTFT).

[0198] Display 140 can be implemented as a touchscreen combined with a touch panel and touch sensors, a flexible display, a rollable display, a 3D display, or a display in which multiple display modules are physically connected. When display 140 is implemented as a touchscreen together with a touch panel, display 140 can be used as an output unit for outputting information between robot 100 and user, and as an input unit for providing an input interface between robot 100 and user. In particular, the processor can calculate an interaction score for multiple users based on touch input received through display 140.

[0199] The memory 150 can store data required for various embodiments of this disclosure. As one or more examples, map data about the travel space of the robot 100 can be stored in the memory 150. Depending on the data storage purpose, the memory 150 can be implemented as a memory embedded in the robot 100 or as a memory removable from the robot 100. For example, data for driving the robot 100 can be stored in a memory embedded in the robot 100, and data for expanding the functionality of the robot 100 can be stored in a memory removable from the robot 100.

[0200] The memory embedded in the robot 100 may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM) or synchronous dynamic RAM (SDRAM)) and non-volatile memory (e.g., one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard disk drive or solid-state drive (SSD)).

[0201] Furthermore, the memory removable from the robot 100 can be implemented as a memory card (e.g., Compact Flash (CF), Secure Digital (SD), Micro-SD, Mini-SD, xD, Multimedia Card (MMC), etc.), or an external memory (e.g., a USB memory) that can be connected to a USB port.

[0202] According to one or more examples, memory 150 may store information about multiple neural network (or artificial intelligence) models. Here, storing information about the neural network models may refer to storing various information related to the operation of the neural network models (such as information about at least one layer included in the neural network model, information about the parameters used in each of the at least one layer, biases, etc.). However, of course, depending on the implementation type of processor 130, information about the neural network models may be stored in the internal memory of processor 130. For example, if processor 130 is implemented as dedicated hardware, information about the neural network models may be stored in the internal memory of processor 130.

[0203] Robot 100 includes at least one sensor 160. In addition to at least one of a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor (e.g., red, green, and blue (RGB) sensor), biometric sensor, temperature / humidity sensor, illuminance sensor, or ultraviolet (UV) sensor, at least one sensor 160 may also include sensors for detecting objects around robot 100 (e.g., radar sensor, ToF sensor, etc.) and sensors for detecting the posture of robot 100 (e.g., IMU sensor, etc.). The processor can detect multiple users around robot 100 based on sensing information acquired through at least one sensor 160 and identify the distance between robot 100 and user 200.

[0204] The speaker 170 can output acoustic signals to the outside of the robot 100. The speaker 170 can output multimedia playback, recording playback, various notification sounds, voice messages, etc. The robot 100 may include audio output devices (such as the speaker 170), or may include output devices (such as audio output terminals). Specifically, the speaker 170 can provide, in voice form, acquired information, information processed and generated based on the acquired information, response results or operation results regarding user voice, etc.

[0205] Microphone 180 can receive user voice. Specifically, microphone 180 can receive voice from users located around robot 100. Furthermore, microphone 180 can convert the input user voice into an electrical signal and send the electrical signal to the processor. Therefore, processor 130 can recognize the number of voice command inputs from multiple users 200. Optionally, processor 130 can recognize whether the multiple users 200 are conversing with each other.

[0206] Communication interface 190 can send or receive various types of content. As one or more examples, communication interface 190 can receive signals from or send signals to external devices (e.g., user terminals), external storage media (e.g., USB memory), or external servers (e.g., network drives) via communication methods such as AP-based wireless LAN networks (Wi-Fi), Bluetooth, Zigbee, wired / wireless LAN, wide area network (WAN), Ethernet, IEEE 1394, high-definition multimedia interface (HDMI), universal serial bus (USB), mobile high-definition link (MHL), Audio Engineering Society / European Broadcasting Union (AES / EBU), fiber optic, coaxial cable, etc.) in a streamlined or download manner.

[0207] In addition, robot 100 may also include a user interface. The user interface is a component of robot 100 used to interact with users, and processor 130 can receive various information (such as control information for robot 100) through the user interface and can calculate an interaction score for each user 200 based on the input information. The user interface may include, but is not limited to, at least one of touch sensors, motion sensors, buttons, micro-dials, and switches.

[0208] The methods according to the various embodiments of this disclosure described above can be implemented as applications that can be installed on existing robots. Optionally, the methods according to the various embodiments of this disclosure described above can be performed using a trained neural network (or a deeply trained neural network) based on deep learning (i.e., a learned network model). Furthermore, the methods according to the various embodiments of this disclosure described above can be implemented simply by upgrading the software or hardware of an existing robot. Additionally, the various embodiments of this disclosure described above can also be performed via an embedded server located in the robot or an external server of the robot.

[0209] According to one or more embodiments of this disclosure, the foregoing various embodiments can be implemented as software including instructions stored in a machine-readable storage medium, which can be read by a machine (e.g., a computer). A machine refers to a means that invokes the instructions stored in the storage medium and can operate according to the invoked instructions, and the means may include a display device according to the foregoing embodiments. When the instructions are executed by a processor, the processor may perform the function corresponding to the instructions itself, or by using other components under its control. Instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" simply means that the storage medium does not include signals and is tangible, but does not indicate whether data is semi-permanently stored or temporarily stored in the storage medium.

[0210] According to another embodiment, methods according to the various examples disclosed herein may be included and incorporated into a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or through an app store (e.g., the Play Store). TM Online distribution. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily disposed in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0211] Furthermore, each component (e.g., a module or program) in the various embodiments described above may consist of a single object or multiple objects. Additionally, some sub-components may be omitted in the corresponding sub-components described above, or other sub-components may be included in the various embodiments. Optionally or additionally, some components (e.g., modules or programs) may be integrated into an object and perform the functions performed by each component prior to integration in the same or similar manner. Operations performed by modules, programs, or other components according to the various embodiments may be performed sequentially, in parallel, repeatedly, or heuristically. Alternatively, at least some operations may be performed in a different order or omitted, or other operations may be added.

[0212] While specific embodiments of this disclosure have been specifically shown and described, it will be understood that various changes in form and detail may be made without departing from the spirit and scope of the appended claims.

Claims

1. A robot, comprising: camera; drive; At least one memory to store instructions; as well as At least one processor is operatively connected to the camera, the driver, and the at least one memory, and is configured to execute the instructions to perform the following operations: Identify at least one user located around the robot based on images acquired by the camera. The robot's first rotation angle and first rotation direction are identified based on the position of at least one target region where the at least one user is located in one of multiple regions within the camera's field of view, and a first gaze sequence for the at least one target region. The actuator is controlled to rotate the robot based on the first rotation angle and the first rotation direction. Based on the identification of a new user from the image during the robot's rotation, a second gaze sequence is generated, wherein the second gaze sequence includes a new target region where the new user is located in one of the plurality of regions within the camera's field of view. The robot's second rotation angle and second rotation direction are identified based on the location of the new target region and the second gaze sequence. The actuator is controlled to rotate the robot based on the second rotation angle and the second rotation direction.

2. The robot as claimed in claim 1, wherein, The at least one processor is also configured to execute the instructions to perform the following operations: The multiple regions are identified by dividing the camera's field of view at a predetermined angle. Based on the premise that the at least one target region is multiple target regions, the first target region among the multiple target regions is identified based on the first gaze order. Based on the location of the first target region, the first rotation angle and the first rotation direction of the robot corresponding to the first target region are identified. Based on the position of the first target region and the positions of the remaining target regions among the plurality of target regions, the first rotation angle and first rotation direction of the robot corresponding to the remaining target regions are identified, and Based on a first rotation angle and a first rotation direction, the actuator is controlled to rotate the robot according to the first gaze sequence.

3. The robot as described in claim 1, wherein, The at least one processor is also configured to execute the instructions to perform the following operations: The multiple regions are identified by dividing the camera's field of view at a predetermined angle, and The at least one target area and the new target area are located in the plurality of areas.

4. The robot as described in claim 3, wherein, The at least one processor is configured to: Based on the premise that the at least one target region is multiple target regions, the next target region in the next order of the current target region being gazed upon by the robot is identified based on the second gaze order, wherein the multiple target regions include unprocessed target regions and the new target region. Based on the location of the next target area, identify the robot's next rotation angle and next rotation direction corresponding to the next target area; Based on the position of the next target area and the position of the remaining target area after the next target area, the remaining rotation angle and remaining rotation direction of the robot corresponding to the remaining target area are identified.

5. The robot as described in claim 4, wherein, The at least one processor is also configured to execute the instructions to perform the following operations: Based on the fact that the next target region is the new target region, the second rotation angle and the second rotation direction are identified based on the position of the new target region. Based on the location of the new target region, identify the robot's third rotation angle and third rotation direction corresponding to the unprocessed target region, and After the robot gazes at the new target area, the actuator is controlled to rotate the robot based on the third rotation angle and the third rotation direction.

6. The robot as claimed in claim 4, wherein, The at least one processor is also configured to execute the instructions to perform the following operations: Based on the fact that the next target region is an unprocessed target region, the second rotation angle and the second rotation direction are identified based on the position of the new target region and the positions of the unprocessed target regions preceding the new target region. After observing the unprocessed target area prior to the new target area, the actuator is controlled to rotate the robot based on the second rotation angle and the second rotation direction.

7. The robot as claimed in claim 1, wherein, The at least one processor is also configured to execute the instructions to perform the following operations: Based on the fact that the rotation angle of the robot used to gaze at the new target region is equal to or greater than a predetermined critical angle, the first gaze order is maintained, and the new target region is not included in the first gaze order.

8. The robot as claimed in claim 1, wherein, The at least one processor is also configured to execute the instructions to perform the following operations: Based on the images acquired by the camera, multiple first users located around the robot are identified; as well as Based on at least one of the following: the distance between the plurality of first users, the time during which the plurality of first users are within the field of view of the camera, and whether the plurality of first users are engaged in a conversation, multiple users in the same user group among the plurality of first users are identified.

9. The robot of claim 1, further comprising: The microphone is configured to receive user voice. The at least one processor is further configured to execute the instructions to perform the following operations: Based on the premise that the at least one user is multiple users and the at least one target region is multiple target regions, the duration of gaze of the multiple users at the robot is identified based on the images acquired by the camera. The number of voice inputs from the multiple users to the robot is identified based on the user's voice acquired through the microphone. The interaction scores of the multiple users are calculated based on the gaze duration and the amount of voice input. Based on the interaction scores of at least one user located in each target region, identify the regional interaction scores corresponding to each target region; The first gaze order for the plurality of target regions is determined based on the region interaction score corresponding to each target region.

10. The robot of claim 9, wherein, The at least one processor is also configured to execute the instructions to perform the following operations: The duration of regional gaze for each target region is determined based on the regional interaction score corresponding to each target region. as well as The actuator is controlled to cause the robot to gaze at each target area during the gaze duration of the area.

11. A method for controlling a robot, the method comprising: Identify at least one user located around the robot based on images acquired by the camera; The robot's first rotation angle and first rotation direction are identified based on the position of at least one target area where at least one user is located in one of multiple areas within the field of view of the camera and a first gaze sequence for the at least one target area; The control actuator rotates the robot based on the first rotation angle and the first rotation direction; Based on the identification of a new user from the image during the robot's rotation, a second gaze sequence is generated, wherein the second gaze sequence includes a new target region where the new user is located in one of the plurality of regions within the camera's field of view; The robot's second rotation angle and second rotation direction are identified based on the location of the new target region and the second gaze sequence; and The actuator is controlled to rotate the robot based on the second rotation angle and the second rotation direction.

12. The control method as described in claim 11, wherein, The operation of identifying the first rotation angle and the first rotation direction includes: The multiple regions are identified by dividing the field of view of the camera at a predetermined angle; Based on the premise that the at least one target region is multiple target regions, the first target region among the multiple target regions is identified based on the first gaze order; Based on the location of the first target region, identify the first rotation angle and the first rotation direction of the robot corresponding to the first target region; and Based on the position of the first target region and the position of the remaining target regions among the plurality of target regions, the first rotation angle and the first rotation direction of the robot corresponding to the remaining target regions are identified.

13. The control method as described in claim 11, wherein, The operation of identifying the second rotation angle and the second rotation direction includes: The multiple regions are identified by dividing the camera's field of view at a predetermined angle, and The at least one target area and the new target area are located in the plurality of areas.

14. The control method as described in claim 13, wherein, The operation of identifying the first rotation angle and the first rotation direction includes: Based on the premise that the at least one target region is multiple target regions, the next target region in the next order of the current target region being gazed at by the robot is identified based on the second gaze order, wherein the multiple target regions include unprocessed target regions and the new target region; Based on the location of the next target region, identify the robot's next rotation angle and next rotation direction corresponding to the next target region; and Based on the position of the next target area and the position of the remaining target area after the next target area, the remaining rotation angle and remaining rotation direction of the robot corresponding to the remaining target area are identified.

15. The control method as described in claim 14, wherein, The operation of identifying the next rotation angle and the next rotation direction includes: identifying the second rotation angle and the second rotation direction based on the position of the new target region, since the next target region is the new target region. The operation of identifying the remaining rotation angle and the remaining rotation direction includes: based on the position of the new target region, identifying the robot's third rotation angle and third rotation direction corresponding to the unprocessed target region, and The operation of controlling the actuator to rotate the robot based on the third rotation angle and the third rotation direction includes: after the robot gazes at the new target area, controlling the actuator to rotate the robot based on the third rotation angle and the third rotation direction.

Images