Robot and method for controlling same

Abstract

A robot and a method for controlling same are provided. This robot comprises at least one sensor, a speaker, a microphone, a driving unit, at least one memory for storing one or more instructions, and at least one processor configured to execute the one or more instructions. The one or more instructions, when executed by the at least one processor, generate a map including information regarding multiple objects on the basis of sensing information acquired through the at least one sensor; generate ultrasonic waves with regard to the multiple objects, respectively, through the speaker; acquires reflectivity information regarding the multiple objects on the basis of reflected sounds which have been reflected from the multiple objects, respectively, and received through the microphone; and store the reflectivity information. The reflected sound reflected from each object is at least a part of an ultrasonic wave reflected from each object. If a user voice is received through the microphone, information regarding the intensity of the user voice is acquired with regard to each of multiple directions. Information regarding multiple candidate directions in which the user voice is received, among the multiple directions, is acquired on the basis of the information regarding the intensity of the user voice with regard to each of multiple directions. Priority information regarding the multiple candidate directions is acquired on the basis of the robot's position and the stored reflectivity information. Information regarding the direction in which the user voice is uttered, among the multiple directions, is acquired on the basis of the priority information.

Description

Robot and method for controlling the same

[0001] The present disclosure relates to a robot and a control method thereof, and more particularly, to a robot and a control method for estimating a location where a user's voice is uttered.

[0002] Recently, technologies for estimating sound source direction have been developed to estimate the location of the sound source of a user's voice or other audio signals. Source direction estimation refers to a technique for estimating the user's direction relative to a microphone by measuring the intensity of a voice signal in different directions using sound containing user voice information.

[0003] However, this method is less usable in indoor environments where sound is easily reflected from walls and other objects. To compensate for this, it is often used in conjunction with other sensors, such as cameras. However, when used in conjunction with other sensors, such as lidar or cameras, it is possible to detect objects in a given location. However, this requires the robot to be repeatedly moved or rotated, requiring additional actions to determine the location using the lidar or camera.

[0004] According to one embodiment of the present disclosure, a robot comprises: at least one sensor; a speaker; a microphone; a driving unit; at least one memory storing one or more instructions; And at least one processor configured to execute the one or more instructions; wherein the one or more instructions, when executed by the at least one processor, generate a map including information on a plurality of objects based on sensing information acquired through the at least one sensor, generate an ultrasonic wave for each of the plurality of objects through the speaker, obtain reflectivity information on the plurality of objects based on reflected sounds reflected from each of the objects and received through the microphone, and store the reflectivity information, wherein the reflected sounds reflected from each object are at least a portion of the ultrasonic waves reflected from each object, and when a user voice is received through the microphone, obtain information on the intensity of the user voice for each of a plurality of directions, obtain information on a plurality of candidate directions in which the user voice is received among the plurality of directions based on the information on the intensity of the user voice for each of the plurality of directions, obtain priority information on the plurality of candidate directions based on the location of the robot and the stored reflectivity information, and obtain information on the direction in which the user voice is spoken among the plurality of candidate directions based on the priority information.

[0005] The one or more commands, when executed by the at least one processor, generate ultrasonic waves at preset distance intervals for a wall object among the plurality of objects, receive reflected sounds reflected from the wall object through the microphone, and obtain reflectivity information for the wall object, wherein the reflected sounds reflected from the wall object may be at least a portion of ultrasonic waves reflected from the wall object and generated at preset intervals.

[0006] The one or more commands, when executed by the at least one processor, generate ultrasonic waves in two or more directions for an object among the plurality of objects excluding a wall object, and obtain an average value of reflected sounds reflected from the object and received through the microphone to obtain reflectivity information for the object, and the reflected sounds reflected from the object may be at least a part of the ultrasonic waves reflected in two or more directions.

[0007] The one or more commands, when executed by the at least one processor, can obtain information about the intensity of user voices received from the plurality of directions based on the position of the robot, and identify a preset number of directions in which the intensity of the user voice is greater than a preset threshold value among the plurality of directions as the plurality of candidate directions.

[0008] The one or more commands, when executed by the at least one processor, can identify an object located in the plurality of candidate directions among the plurality of objects based on the position of the robot, and identify a priority for the plurality of candidate directions based on information about the intensity of the user's voice measured from each of the plurality of candidate directions and reflectivity information corresponding to the identified object.

[0009] The one or more commands, when executed by the at least one processor, multiply the intensity of the user voice corresponding to each of the plurality of candidate directions by a weight to obtain a corrected intensity of the user voice obtained from the plurality of candidate directions, and identify a priority based on the corrected intensity for each of the plurality of candidate directions, wherein the weight may correspond to reflectivity information corresponding to the identified object.

[0010] The weight and reflectance information corresponding to the above-identified object may be inversely proportional to each other.

[0011] The priority of each candidate direction among the plurality of candidate directions is directly proportional to the corrected century, and when the one or more commands are executed by the at least one processor, the candidate direction having the highest priority among the plurality of candidate directions can be identified as the direction in which the user's voice was spoken.

[0012] The one or more commands, when executed by the at least one processor, can perform voice recognition for the user's voice by performing beamforming in the direction in which the user's voice was spoken.

[0013] According to one embodiment of the present disclosure, a method for controlling a robot includes the steps of: generating a map including information on a plurality of objects based on sensing information acquired through at least one sensor of the robot; generating an ultrasonic wave for each of the plurality of objects through a speaker of the robot; obtaining reflectivity information on the plurality of objects based on reflected sounds reflected from each of the objects and received through a microphone of the robot, and storing the reflectivity information; obtaining information on the intensity of the user's voice for each of a plurality of directions when a user's voice is received through the microphone, wherein the reflected sounds reflected from each object are at least a portion of the ultrasonic waves reflected from each object; obtaining information on a plurality of candidate directions in which the user's voice is spoken among the plurality of directions based on the information on the intensity of the user's voice in the plurality of directions; obtaining priority information on the plurality of candidate directions based on a location of the robot and the stored reflectivity information; and obtaining information on a direction in which the user's voice is spoken among the plurality of candidate directions based on the priority information.

[0014] The step of generating ultrasonic waves for each of the plurality of objects may include generating ultrasonic waves at preset distance intervals for a wall object among the plurality of objects, and the step of obtaining and storing the reflectivity information may include receiving a reflected sound reflected from the wall object through the microphone to obtain the reflectivity information for the wall object, and the reflected sound reflected from the wall object may be at least a portion of the ultrasonic waves reflected from the wall object and generated at preset intervals.

[0015] The step of generating ultrasonic waves for each of the plurality of objects may include generating ultrasonic waves in two or more directions for objects remaining among the plurality of objects except for a wall object, and the step of obtaining and storing the reflectivity information may include obtaining an average value of reflected sounds reflected from the objects and received through the microphone to obtain reflectivity information for the objects, and the reflected sounds reflected from the objects may be at least a part of the ultrasonic waves reflected in two or more directions.

[0016] The step of obtaining information about the plurality of candidate directions may identify a preset number of directions in which the intensity of the user's voice is greater than a preset threshold value among the plurality of directions as the plurality of candidate directions.

[0017] The step of obtaining the priority information may include identifying an object located in the plurality of candidate directions among the plurality of objects based on the position of the robot, and identifying a priority for the plurality of candidate directions based on information about the intensity of a user's voice corresponding to each of the plurality of candidate directions and reflectivity information corresponding to the identified object.

[0018] The step of obtaining the above priority information may include obtaining a corrected intensity of the user voice obtained from the plurality of candidate directions by multiplying the intensity of the user voice corresponding to each of the plurality of candidate directions by a weight, and identifying a priority based on the corrected intensity for each of the plurality of candidate directions, wherein the weight may correspond to reflectivity information corresponding to the identified object.

[0019] According to one embodiment of the present disclosure, a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to execute a *?* method for controlling a robot, the method comprising: generating a map including information on a plurality of objects based on sensing information acquired through at least one sensor of the robot; generating an ultrasonic wave for each of the plurality of objects through a speaker of the robot; obtaining reflectivity information on the plurality of objects based on reflected sounds reflected from each of the objects and received through a microphone of the robot, and storing the reflectivity information; obtaining information on the intensity of a user's voice for each of a plurality of directions when a user's voice is received through the microphone, the reflected sounds reflected from each object being at least a portion of the ultrasonic waves reflected from each object; obtaining information on a plurality of candidate directions in which the user's voice is spoken among the plurality of directions based on the information on the intensity of the user's voice in the plurality of directions; obtaining priority information on the plurality of candidate directions based on a location of the robot and the stored reflectivity information; And a step of obtaining information about the direction in which the user's voice was spoken among the plurality of candidate directions based on the priority information;

[0020] The step of generating ultrasonic waves for each of the plurality of objects may include generating ultrasonic waves at preset distance intervals for a wall object among the plurality of objects, and the step of obtaining and storing the reflectivity information may include receiving a reflected sound reflected from the wall object through the microphone to obtain the reflectivity information for the wall object, and the reflected sound reflected from the wall object may be at least a portion of the ultrasonic waves reflected from the wall object and generated at preset intervals.

[0021] The step of generating ultrasonic waves for each of the plurality of objects may include generating ultrasonic waves in two or more directions for objects remaining among the plurality of objects except for a wall object, and the step of obtaining and storing the reflectivity information may include obtaining an average value of reflected sounds reflected from the objects and received through the microphone to obtain reflectivity information for the objects, and the reflected sounds reflected from the objects may be at least a part of the ultrasonic waves reflected in two or more directions.

[0022] The step of obtaining information about the plurality of candidate directions may identify a preset number of directions in which the intensity of the user's voice is greater than a preset threshold value among the plurality of directions as the plurality of candidate directions.

[0023] The step of obtaining the priority information may include identifying an object located in the plurality of candidate directions among the plurality of objects based on the position of the robot, and identifying a priority for the plurality of candidate directions based on information about the intensity of a user's voice corresponding to each of the plurality of candidate directions and reflectivity information corresponding to the identified object.

[0024] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0025] FIG. 1 is a block diagram showing the configuration of a robot according to one embodiment of the present disclosure;

[0026] FIG. 2 is a block diagram showing a plurality of configurations for generating a map including reflectance information and identifying a direction from which a user's voice was spoken using the generated map, according to one embodiment of the present disclosure;

[0027] FIG. 3A, FIG. 3B and FIG. 3C are diagrams illustrating a method for generating a map including information about an object according to one embodiment of the present disclosure;

[0028] FIGS. 4A, 4B, and 4C are diagrams illustrating a method for obtaining reflectance information for an object and storing it together with a map, according to one embodiment of the present disclosure.

[0029] FIG. 5A, FIG. 5B, FIG. 5C and FIG. 5D are diagrams illustrating a method for identifying a direction in which a user's voice is spoken using reflectance information according to one embodiment of the present disclosure, and

[0030] FIG. 6 is a flowchart for explaining a method for controlling a robot according to one embodiment of the present disclosure.

[0031] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0032] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0033] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0034] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0035] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0036] In this disclosure, expressions such as "A or B," "at least one of A and / or B," or "one or more of A or / and B" can include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0037] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0038] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0039] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0040] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0041] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0042] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0043] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0044] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily carry out the present disclosure.

[0045] FIG. 1 is a block diagram showing the configuration of a robot according to one embodiment of the present disclosure. As illustrated in FIG. 1, the robot (100) may include at least one sensor (110), a speaker (120), a microphone (130), a driving unit (140), at least one memory (150), and at least one processor (160). Meanwhile, the configuration of the robot (100) is not limited to the configuration illustrated in FIG. 1, and it goes without saying that configurations obvious to those skilled in the art may be added or deleted. In addition, the robot (100) according to the present disclosure may be a robot vacuum cleaner, but this is merely one embodiment, and the robot (100) may be a robot that provides various services in a home, or a service robot (for example, a guide robot, etc.) that provides services in various places such as an airport, a hotel, a mart, a clothing store, a logistics center, a hospital, etc.

[0046] At least one sensor (110) can obtain various information about the status of the robot (100) or the surrounding environment of the robot (100). In particular, at least one sensor (110) may include a lidar sensor and an IMU (Inertial Measurement Unit) sensor. The lidar sensor can obtain sensing information for obtaining information about the distance to the object by projecting light (e.g., laser, near-infrared light, visible light, ultraviolet light, etc.) to an object and detecting light reflected by the object. The IMU sensor is a sensor for detecting the movement of the robot (100) and may include at least one of a geomagnetic sensor, an acceleration sensor, and a gyro sensor. However, as described above, using the lidar sensor to obtain information about the distance to the object is only one embodiment, and information about the distance to the object may be obtained using various sensors such as a depth sensor.

[0047] In particular, at least one processor (160) can obtain information on distances to multiple objects and movement information of the robot (100) based on sensing information acquired through the lidar sensor and the IMU sensor. However, this is only one embodiment, and at least one sensor (110) can obtain information on distances to multiple objects and movement information of the robot (100) based on the acquired sensing information.

[0048] Additionally, at least one sensor (110) may include a camera for capturing images. In particular, the camera may capture images of the surroundings of the robot (100) to obtain images of objects. The robot (100) may input the images of the objects into a trained neural network model (e.g., an object recognition model) to obtain information about the objects (e.g., the type of the objects, the size of the objects, the shape of the objects, etc.).

[0049] The speaker (120) is configured to output an audio signal. In particular, the speaker (120) can output ultrasonic waves toward an object under the control of at least one processor (160). At this time, the speaker (120) can generate ultrasonic waves at preset distance intervals for a wall object among a plurality of objects. In addition, the speaker (120) can generate ultrasonic waves in two or more directions with respect to the remaining objects, excluding the wall object among the plurality of objects.

[0050] The microphone (130) is configured to receive various audio signals. In particular, the microphone (130) can receive reflected sound from an object that is an ultrasonic wave output through the speaker (120). In addition, the microphone (130) can receive a user's voice. In this case, a plurality of microphones (130) may be provided, and the user's voice received from a plurality of directions can be received through the plurality of microphones.

[0051] The driving unit (140) can cause the robot (100) to drive or move under the control of at least one processor (160). In this case, driving may include a motion in which the robot (100) moves with power. According to embodiments of the present disclosure, driving may include a motion in which the robot (100) moves with power in any direction. Alternatively, driving may include a motion in which the robot (100) moves with power along a preset track or path.

[0052] In particular, the driving unit (140) may include wheels that drive the robot (100) and a wheel drive motor that rotates the wheels. In particular, the driving unit (140) may move the robot (100) within a space within a home in order to photograph the space within the home. In addition, when the direction in which the user's voice was spoken is identified, the driving unit (140) may move the robot (100) in the direction in which the user's voice was spoken.

[0053] At least one memory (150) may store an operating system (OS) for controlling the overall operation of components of the robot (100) and instructions or data related to components of the robot (100). In particular, at least one memory (150) may include a plurality of modules for generating a map including reflectivity information and identifying a direction in which a user's voice is spoken using the generated map. For example, at least one memory (150) may include a map generation module (210) including a terrain information acquisition module (211), an object information acquisition module (212), and a reflectivity information acquisition module (213), and a recognition module (220) including a user voice acquisition module (221), a preprocessing module (222), an intensity measurement module (223), a candidate direction acquisition module (224), a priority information acquisition module (225), a direction recognition module (226), and a user voice recognition module (227), as illustrated in FIG. 2.

[0054] In particular, when a function for generating a map including reflectivity information and identifying the direction in which a user's voice was spoken using the generated map is executed, the robot (100) can load data for various modules stored in non-volatile memory to perform various operations into volatile memory. Here, loading means an operation of calling and storing data stored in non-volatile memory into volatile memory so that at least one processor (160) can access it.

[0055] Additionally, at least one memory (150) can store information about a neural network model trained to obtain information about an object by inputting an image acquired by a camera.

[0056] At least one memory (150) may be implemented as a non-volatile memory (e.g., a hard disk, a solid state drive (SSD), a flash memory), a volatile memory (which may also include memory within at least one processor (160)), etc.).

[0057] At least one processor (160) can control the robot (100) according to at least one instruction stored in the memory (150).

[0058] In particular, at least one processor (160) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the robot (100) and perform operations or data processing related to communication. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0059] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor). For example, the robot (100) may perform an operation such as generating a map using the general-purpose processor, and may perform an operation such as recognizing an object or recognizing a user's voice using the artificial intelligence-specific processor.

[0060] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0061] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among a plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0062] In one or more embodiments of the present disclosure, at least one processor (160) generates a map including information on a plurality of objects based on sensing information acquired through at least one sensor (110). The plurality of objects may include items, walls, windows, etc. contained within a home. In addition, the map may include a drawing showing the state of a space within a home on a plane by reducing it by a certain ratio. According to embodiments of the present disclosure, the map may include a drawing showing the planar structure of the interior of a home by reducing it by a certain ratio and representing it with a predetermined symbol. In addition, the map may include a drawing showing the planar structure of the interior of the home with lines. However, the present disclosure is not limited thereto, and the map may also include the locations of major objects within the home. The map may be implemented as at least one of a two-dimensional grid map, a two-dimensional linear vector map, a three-dimensional grid map, and a three-dimensional map.

[0063] At least one processor (160) generates ultrasonic waves for each of a plurality of objects through a speaker (120), receives reflected sounds reflected from each of the objects through a microphone (130), obtains reflectivity information for each of the plurality of objects, and stores the information in at least one memory (150). The reflectivity information for each of the plurality of objects may include information about the ratio of the intensity of the ultrasonic waves output through the speaker (120) to the intensity of the reflected sounds received through the microphone. Information about the reflectivity information may be obtained for each object, but this is only one embodiment, and the same object may be obtained depending on the direction.

[0064] In particular, at least one processor (160) can obtain information about reflectivity in different ways depending on the type of object.

[0065] In one embodiment, at least one processor (160) may generate ultrasonic waves at a preset distance interval for a wall object among a plurality of objects, and receive reflected sounds reflected from the wall object through a microphone (130) to obtain reflectivity information for the wall object. In one embodiment, at least one processor (160) may generate ultrasonic waves in two or more directions for the remaining objects excluding the wall object among a plurality of objects, and receive reflected sounds reflected from two or more directions with respect to the remaining objects through a microphone (130), and obtain an average value of the received sounds reflected from the sounds generated in the two or more directions to obtain reflectivity information for the remaining objects.

[0066] In a manner as described above, at least one processor (160) can store reflectance information for a plurality of objects together with the generated map.

[0067] When a user's voice is received through a microphone (130), at least one processor (160) obtains information about the intensity of the user's voice in a plurality of directions. The intensity of the user's voice refers to sound wave energy transmitted per unit time and per unit area, and can be replaced with other terms such as the amplitude of the user's voice, the size of the user's voice, the energy of the user's voice, etc. In addition, at least one processor (160) can obtain information about the intensity of the user's voice in a plurality of directions at preset intervals (e.g., 1 degree) with respect to the robot (100).

[0068] At least one processor (160) obtains information about a plurality of candidate directions from which the user voice is received among the plurality of directions based on information about the intensity of the user voice. The plurality of candidate directions are directions from which the user voice is likely to be uttered among the plurality of directions. Specifically, the at least one processor (160) may obtain information about the intensity of the user voice received from the plurality of directions based on the position of the robot (100). In addition, the at least one processor (160) may identify a predetermined number of directions (e.g., five directions) from among the plurality of directions in which the intensity of the user voice is high as the plurality of candidate directions. As used herein, an intensity may be considered "high" if it exceeds a predetermined threshold. In general, an intensity may be considered "high" if it belongs to a subset of the highest intensity values ​​among the entire set of acquired intensity values ​​(e.g., the five highest intensity values ​​among a larger set of intensity values).

[0069] At least one processor (160) may obtain priority information for a plurality of candidate directions based on the position and stored reflectivity information of the robot (100). The priority information may include information about the direction in which the user's voice is likely to be spoken among the plurality of directions.

[0070] Specifically, at least one processor (160) can identify objects located in a plurality of candidate directions based on the position of the robot (100). In addition, at least one processor (160) can identify priorities for the plurality of candidate directions based on the intensity of the user voice measured from each of the plurality of candidate directions and the reflectivity information for the identified objects. The at least one processor (160) can correct the intensity of the user voice obtained from the plurality of candidate directions by multiplying the intensity of the user voice obtained from the plurality of candidate directions by a weight corresponding to each reflectivity of the objects located in the plurality of candidate directions. At this time, the weight may be set to be lower as the reflectivity of the object located in the candidate direction is higher, and may be set to be higher as the reflectivity of the object located in the candidate direction is lower. In addition, at least one processor (160) can identify priorities based on the intensity of the corrected user voice for each of the plurality of candidate directions.

[0071] At least one processor (160) obtains information about the direction in which the user's voice was spoken among a plurality of candidate directions based on priority information. Specifically, at least one processor (160) can identify a higher priority for each of the plurality of candidate directions as the greater the intensity of the corrected user's voice, and can obtain information about the direction in which the user's voice was spoken by identifying the candidate direction with the highest priority among the plurality of candidate directions as the direction in which the user's voice was spoken.

[0072] At least one processor (160) can perform various functions based on information about the direction in which the user's voice was spoken. In one embodiment, the at least one processor (160) can perform voice recognition for the user's voice by performing beamforming in the direction in which the user's voice was spoken. The voice recognition for the user's voice can include at least one of an operation of obtaining text corresponding to the user's voice, an operation of performing natural language understanding using the text corresponding to the user's voice, and an operation of providing a response or controlling the robot (100) based on the natural language understanding result. In one embodiment, the at least one processor (160) can rotate or move the robot (100) so that the robot (100) faces the direction in which the user's voice was spoken.

[0073] As described above, by recognizing the direction in which the user's voice is spoken using reflectivity information, the robot (100) can more accurately recognize the direction in which the user is located and provide various functions.

[0074] Hereinafter, a method for generating a map including reflectance information and identifying the direction in which a user's voice was spoken using the generated map will be described in detail with reference to FIGS. 2 to 5d.

[0075] FIG. 2 is a block diagram illustrating a plurality of components for generating a map including reflectance information and identifying the direction from which a user's voice was spoken using the generated map, according to one embodiment of the present disclosure. As illustrated in FIG. 2 , the robot (100) may include a map generation module (210) and a recognition module (220).

[0076] The map generation module (210) can generate a map including various information about an object based on sensing information received from at least one sensor (110) and reflected sounds acquired through a speaker (120) and a microphone (130). In particular, the map generation module (210) can include a terrain information acquisition module (211), an object information acquisition module (212), and a reflectivity information acquisition module (213), as illustrated in FIG. 2.

[0077] The terrain information acquisition module (211) can acquire terrain information of a space within a home based on sensing information acquired through a lidar sensor and an IMU sensor while the robot (100) is driving. For example, as illustrated in FIG. 3A, when the robot (100) of a space within a home (310) generates a map according to a user command, the terrain information acquisition module (211) can control the robot (100) to drive in a preset pattern. The terrain information acquisition module (211) can acquire sensing information through a lidar sensor and an IMU sensor while the robot (100) is driving. Based on the acquired sensing information, the terrain information acquisition module (211) can acquire a map (320) for terrain information including the outermost line (321) of the space within a home (310), as illustrated in FIG. 3B.

[0078] The object information acquisition module (212) can acquire information about an object through an image acquired through a camera. Specifically, the object information acquisition module (212) can acquire information about a plurality of objects included in a space within a home by inputting an image acquired through a camera into a trained neural network model (e.g., an object recognition model). For example, as illustrated in FIG. 3A, when a robot (100) of a space within a home (310) generates a map according to a user command, the object information acquisition module (212) can acquire information about a first object (331), information about a second object (332), and information about a third object (333) by inputting an image acquired through a camera into a trained neural network model. At this time, the information about the object can include at least one of shape information, type information, size information, and location information of the object. In addition, the object information acquisition module (212) can store the acquired information about the object in the form of metadata in the generated map (320).

[0079] The reflectivity information acquisition module (213) can acquire and store reflectivity information for a plurality of objects by generating ultrasonic waves toward each of the plurality of objects through the speaker (120) after generating the map (320) or while generating the map (320) and receiving reflected sounds reflected from each of the objects through the microphone (130). The reflectivity information acquisition module (213) can acquire information on reflectivity in different ways depending on the type of object.

[0080] In one embodiment, the reflectivity information acquisition module (213) can acquire information on reflectivity by generating ultrasonic waves against the wall at a specific distance (e.g., 2 m) interval for the outermost object (e.g., a wall object, a door object, etc.) and measuring the size of the reflected sound. For example, as illustrated in FIG. 4A, the reflectivity information acquisition module (213) can generate ultrasonic waves against the wall at a first location (410-1) and measure the size of the reflected sound to measure the first reflectivity, and then, after moving a specific distance (e.g., 2 m) from the first location (410-1), generate ultrasonic waves against the wall at a second location (410-2) and measure the size of the reflected sound to measure the second reflectivity. In this manner, the reflectivity information acquisition module (213) can acquire reflectivity information on the outermost object while moving through the indoor space (310). And, the reflectivity information acquisition module (213) can acquire the representative value (e.g., average value or mode value, etc.) of the reflectivities acquired in this manner and store it in the generated map (320). However, the above description is only an example, and the reflectivity information acquisition module (213) can store reflectivity information for the outermost object acquired from multiple locations in the map (320) generated for each location.

[0081] In one embodiment, the reflectivity information acquisition module (213) may generate ultrasonic waves in two or more directions for objects (e.g., home appliances, furniture, etc.) other than the outermost object among a plurality of objects, receive reflected sounds reflected from two or more directions toward the remaining objects, and obtain an average value of the received sounds reflected from two or more directions toward each of the remaining objects, thereby acquiring reflectivity information for the remaining objects. For example, as illustrated in FIG. 4B, the reflectivity information acquisition module (213) may generate ultrasonic waves to the second object (332) in a first direction (420-1) based on the second object (332), measure the magnitude of the reflected reflected sound to measure the first reflectivity, generate ultrasonic waves to the second object (332) in a second direction (420-2), measure the magnitude of the reflected reflected sound to measure the second reflectivity, and generate ultrasonic waves to the second object (332) in a third direction (420-3). The reflectivity information acquisition module (213) can acquire reflectivity information about the second object (332) based on the reflected sound reflected from multiple directions in this manner. In addition, the reflectivity information acquisition module (213) can store the representative values ​​for the first to third reflectivities acquired in this manner in the generated map (320). However, this is only one embodiment, and the reflectivity information acquisition module (213) can store reflectivity information about the remaining objects acquired from multiple directions in the map (320) generated for each direction.

[0082] The reflectivity information acquisition module (213) can store the reflectivity information acquired in the manner described above in the form of metadata of the map (320). For example, as illustrated in FIG. 4C, the reflectivity information acquisition module (213) can store the reflectivity information for the first object (331) as 0.2, the reflectivity information for the second object (332) as 0.7, the reflectivity information for the third object (333) as 0.4, and the reflectivity information for the wall object as 0.8.

[0083] The recognition module (220) can identify the speech location of the user's voice using the map acquired by the map generation module (210) and recognize the user's voice using the identified speech location. As illustrated in FIG. 2, the recognition module (220) can include a user voice acquisition module (221), a preprocessing module (222), an intensity measurement module (223), a candidate direction acquisition module (224), a priority information acquisition module (225), a direction recognition module (226), and a user voice recognition module (227).

[0084] The user voice acquisition module (221) can acquire the user voice through the microphone (130). The user voice is a voice spoken by the user and can be distinguished from noise, etc. In particular, the user voice acquisition module (221) can input the acquired audio signal into a trained neural network model to identify whether the audio signal includes the user voice. For example, as illustrated in FIG. 5A, the user voice acquisition module (221) can acquire the user voice "Bot Nano, would you like to look at me?" spoken by the user (10).

[0085] The preprocessing module (222) can perform a preprocessing operation on the acquired user voice. Specifically, the preprocessing module (222) can obtain an audio signal for a portion of the acquired user voice band through a band pass filter. In addition, the preprocessing module (222) can perform a preprocessing operation to remove noise or echo, etc.

[0086] The intensity measurement module (223) can measure the intensity of the user's voice acquired from multiple directions. In addition, the intensity measurement module (223) can perform sampling on the user's voice acquired from multiple directions. In particular, the intensity measurement module (223) can measure the intensity of the user's voice acquired from multiple directions using the SRP-PHAT (steered response power phase transform) algorithm. At this time, the intensity measurement module (223) can measure the intensity of the user's voice in multiple directions (e.g., 360 directions) at preset intervals (e.g., 1 degree) based on the robot (100).

[0087] The candidate direction acquisition module (224) can identify a preset number of directions with a large user voice intensity among a plurality of directions as the plurality of candidate directions. For example, the candidate direction acquisition module (224) can acquire five directions with a large user voice intensity among a plurality of directions (e.g., 360 directions) as candidate directions, as illustrated in FIG. 5b. That is, the intensity of the user voice acquired in the first candidate direction (510-1) may be 1.2, the intensity of the user voice acquired in the second candidate direction (510-2) may be 0.8, the intensity of the user voice acquired in the third candidate direction (510-3) may be 0.8, the intensity of the user voice acquired in the fourth candidate direction (510-4) may be 0.7, and the intensity of the user voice acquired in the fifth candidate direction (510-5) may be 1.1. At this time, the intensities illustrated in FIG. 5b may be relative values. The candidate direction acquisition module (224) identifies the largest information among the intensity information of the user voice received in adjacent directions (for example, directions within a 10 degree range) among multiple directions as the candidate direction, and excludes the remaining adjacent directions from the candidate directions even if the intensity of the user voice is greater than that of other directions.

[0088] The priority information acquisition module (225) can acquire priority information for multiple candidate directions based on the location of the robot (100) and stored reflectivity information.

[0089] Specifically, the priority information acquisition module (225) can identify objects located in multiple candidate directions based on the position of the robot. For example, the priority information acquisition module (225) can identify first to fifth objects located in the first to fifth candidate directions based on the position of the robot (100). Objects may not exist in some of the multiple candidate directions.

[0090] Additionally, the priority information acquisition module (225) can identify weights corresponding to the reflectivity of objects located in multiple candidate directions. The weights may be set lower as the reflectivity of objects located in the candidate directions increases, and higher as the reflectivity of objects located in the candidate directions decreases. For example, the weights and reflectivity may have a relationship as expressed in the following equation.

[0091] <Mathematical Formula 1>

[0092] Weight = 1 - Reflectance

[0093] However, the mathematical expression 1 described above is only one example, and the relationship between the weight and the reflectivity can be expressed in another expression in which the weight and the reflectivity have a different inverse relationship.

[0094] The priority information acquisition module (225) can correct the intensity of the user voice acquired from the plurality of candidate directions by multiplying the intensity of the user voice acquired from the plurality of candidate directions by a weight corresponding to the reflectivity of an object located in the plurality of candidate directions. For example, the priority information acquisition module (225) can correct the intensity of the user voice acquired from the first candidate direction to 0.36 by multiplying the intensity information of 0.3, which is a weight corresponding to the reflectivity of a first object located in a first candidate direction among the plurality of candidate directions, and can correct the intensity of the user voice acquired from the third candidate direction to 0.56 by multiplying the intensity information of 0.7, which is a weight corresponding to the reflectivity of a third object located in a third candidate direction among the plurality of candidate directions.

[0095] In addition, the priority information acquisition module (225) can identify priority information for a plurality of candidate directions based on the intensity of the user voice corrected for each of the plurality of candidate directions. That is, the priority information acquisition module (225) can identify a candidate direction in which the intensity of the user voice corrected for each of the plurality of candidate directions is high in priority, and can identify a candidate direction in which the intensity of the user voice is low in priority. For example, the priority information acquisition module (225) can identify the intensity of the user voice for the first candidate direction (510-1) as 0.36, the intensity of the user voice for the second candidate direction (510-2) as 0.8, the intensity of the user voice for the third candidate direction (510-3) as 0.56, the intensity of the user voice for the fourth candidate direction (510-4) as 0.7, and the intensity of the user voice for the fifth candidate direction (510-5) as 1.1, as illustrated in FIG. 5c. And, the priority information acquisition module (225) can identify the fifth candidate direction (510-5) as the first priority, the second candidate direction (510-2) as the second priority, the fourth candidate direction (510-4) as the third priority, the third candidate direction (510-3) as the fourth priority, and the first candidate direction (510-1) as the fifth priority.

[0096] The direction recognition module (226) can recognize the direction in which the user's voice was spoken based on priority information. Specifically, the direction recognition module (226) can recognize the candidate direction of the first priority as the direction in which the user's voice was spoken. For example, as illustrated in FIG. 5D , the direction recognition module (226) can recognize the fifth candidate direction (510-5) identified as the first priority as the direction in which the user's voice was spoken. As another example, the direction recognition module (226) can capture candidate directions in order of priority with a camera to identify a candidate area in which the user is located, and identify the identified candidate area as the direction in which the user's voice was spoken.

[0097] The speech recognition module (227) can perform speech recognition on a user's voice. The speech recognition module (227) can perform speech recognition on an (additional) user's voice by performing beamforming in the direction in which the user's voice was spoken. The speech recognition module (227) converts the user's additional speech into text, performs natural language understanding on the converted text, and provides a response to the user (10) based on the natural language understanding result or controls the robot (100) according to the user's voice. For example, the speech recognition module (227) can recognize the user's voice, "Bot, look at me," and rotate the robot (100) in the direction in which the user's voice was spoken.

[0098] FIG. 6 is a flowchart for explaining a method for controlling a robot according to one embodiment of the present disclosure.

[0099] First, the robot (100) can obtain sensing information through at least one sensor (S605). Specifically, when an event for creating a map (e.g., an event in which a robot vacuum cleaner is first installed and a user command for creating a map is received, etc.) occurs, the robot (100) can drive within a space within a home using the power of the driving unit (140). In addition, the robot (100) can obtain sensing information for creating a map using at least one sensor (110) while driving within a space within a home. For example, the robot (100) can obtain sensing values ​​for obtaining distance information between the robot (100) and an object using a lidar sensor. For example, the robot (100) can obtain sensing values ​​for obtaining movement information of the robot (100) using an IMU sensor. For example, the robot (100) can obtain images for obtaining information about objects around the robot (100) using a camera.

[0100] The robot (100) can generate a map including information on a plurality of objects based on sensing information (S610). Specifically, the robot (100) can obtain topographic information of a space within a home using an IMU sensor and a lidar sensor, and can input images obtained through a camera into a trained neural network model to obtain information on a plurality of objects included in the space within a home. The information on the plurality of objects can include various information such as type information on each of the plurality of objects, size information on each of the plurality of objects, shape information on each of the plurality of objects, location information on each of the plurality of objects, etc. In addition, the robot (100) can store information on the objects in the form of metadata together with the map including information on the space within a home.

[0101] The robot (100) can generate ultrasonic waves for each of a plurality of objects (S615). At this time, the robot (100) can generate ultrasonic waves using different methods based on the types of the plurality of objects.

[0102] For example, the robot (100) can generate ultrasonic waves at a specific distance interval (e.g., 3 m) for a wall object among a plurality of objects. In addition to the wall object, the robot (100) can generate ultrasonic waves at a specific distance interval for the outermost objects such as windows or doors as described above. For example, the robot (100) can determine two or more sample numbers (e.g., 5) within the angle range that the robot (100) can view for the remaining objects (e.g., furniture, household items, etc.) excluding the outermost object among a plurality of objects, and can evenly divide the angles according to the number of samples to generate ultrasonic waves in two or more directions with respect to the object.

[0103] The robot (100) can generate ultrasonic waves at each of a plurality of objects while driving around the space in the house to create a map, but this is only one embodiment, and after creating the map, the robot (100) can of course generate ultrasonic waves at each of a plurality of objects.

[0104] The robot (100) can receive reflected sounds for each of a plurality of objects, obtain reflectivity information for the objects, and store the obtained reflectivity information (S620). Specifically, the robot (100) can receive reflected sounds for each of a plurality of objects, and obtain information on the intensity of the reflected sounds. In addition, the robot (100) can obtain information on the reflectivity of each of the plurality of objects based on the ratio of the intensity of the output ultrasonic waves and the intensity of the received reflected sounds.

[0105] The robot (100) can receive reflected sounds for ultrasonic waves generated at specific distance intervals for the outermost object, and obtain information about the reflectivity at each specific distance interval. At this time, the robot (100) can store information about the reflectivity at each specific distance interval, and can store representative values ​​(e.g., average value, mode, etc.) of the reflectivity obtained at each specific distance interval as information about the reflectivity of the object. In addition, the robot (100) can receive reflected sounds obtained from at least two directions for the remaining objects excluding the outermost object, and obtain information about the reflectivity according to at least two directions. At this time, the robot (100) can store information about the reflectivity at each of at least two directions, and can store representative values ​​(e.g., average value, mode, etc.) of the reflectivity obtained at each of at least two directions as information about the reflectivity of the object.

[0106] The robot (100) can store information about the reflectivity of an object along with the map. As described above, the robot (100) can store information about the reflectivity of an object on the map in the form of metadata.

[0107] The robot (100) can generate a map containing reflectivity information about an object through steps S605 to S620 and store the map in the robot (100). Meanwhile, the robot (100) can not only store the map within the robot (100), but can also transmit the map to an external server to store the map in the external server. Furthermore, it goes without saying that steps S605 to S620 can also be performed by an electronic device other than the robot (100).

[0108] After step S620 (i.e., after the map is generated), the robot (100) can identify whether it has received a user's voice via the microphone (130) (S625). Specifically, the robot (100) can identify whether it has received a user's voice using a neural network model trained to identify whether it is a human voice. However, this is merely an example, and it is possible to identify whether it has received a user's voice using other methods.

[0109] When a user voice is identified as being received (S625-Y), the robot (100) can preprocess the user voice (S630). Specifically, the robot (100) can acquire the user voice in a specific frequency band using a preset filter (e.g., a band pass filter, etc.) and perform preprocessing on the user voice, such as noise removal.

[0110] The robot (100) can obtain information on the intensity of the user's voice in multiple directions (S635). Specifically, the robot (100) can obtain information on the intensity of the user's voice in multiple directions at preset intervals (e.g., 1 degree) relative to the robot (100). For example, the robot (100) can obtain information on the intensity of the user's voice in the first direction to the 360th direction.

[0111] The robot (100) can obtain information on a plurality of candidate directions in which the user voice is received (S640). Specifically, the robot (100) can identify a preset number of directions (e.g., 5) in which the user voice is louder as the plurality of candidate directions based on information on the intensity of the user voice obtained for each of the plurality of directions. The robot (100) can identify the information with the greatest intensity of the user voice received in an adjacent direction (e.g., a direction within a 10-degree range) among the plurality of directions as the candidate direction, and can exclude the remaining adjacent directions from the candidate directions even if the user voice is louder than the other directions.

[0112] The robot (100) can obtain priority information for multiple candidate directions (S645). Specifically, the robot (100) can correct the intensity of the user voice received from the candidate directions based on the robot's (100) position and reflectivity information, and then obtain priority information for the multiple candidate directions based on the corrected intensity of the user voice.

[0113] Specifically, the robot (100) can identify objects located in multiple candidate directions based on the robot's position. For example, the robot (100) can identify a first object located in a first candidate direction based on the robot's (100) position, and a second object located in a second candidate direction.

[0114] In addition, the robot (100) can identify weights corresponding to the reflectivity of objects located in multiple candidate directions. At this time, the weight may be set lower as the reflectivity of the object located in the candidate direction is higher, and may be set higher as the reflectivity of the object located in the candidate direction is lower. In other words, the higher the reflectivity of the object located in the candidate direction, the higher the possibility that the sound is a reflection of the user's voice reflected off the object rather than the actual user's voice, and therefore the weight may be set lower. The value of the weight may be a value between 0 and 1, but is not limited thereto.

[0115] The robot (100) can correct the intensity of the user voice acquired from the plurality of candidate directions by multiplying the intensity of the user voice acquired from the plurality of candidate directions by a weight corresponding to the reflectivity of the object located in the plurality of candidate directions. That is, the robot (100) can multiply the intensity of the user voice received from the direction in which the object with high reflectivity is located by a small weight, and can multiply the intensity of the user voice received from the direction in which the object with low reflectivity is located by a high weight.

[0116] In addition, the robot (100) can identify priority information for each of the plurality of candidate directions based on the intensity of the user voice corrected for each of the plurality of candidate directions. That is, the robot (100) can identify a higher priority as the intensity of the user voice corrected for each of the plurality of candidate directions increases, and can identify a lower priority as the intensity of the user voice decreases.

[0117] The robot (100) can obtain information about the direction in which the user's voice was spoken based on priority information (S650). Specifically, the robot (100) can identify the candidate direction with the highest priority as the direction in which the user's voice was spoken. Alternatively, the robot (100) can use a camera to capture candidate directions in descending order of priority to identify the candidate area in which the user is located, and then identify the identified candidate area as the direction in which the user's voice was spoken.

[0118] In addition, the robot (100) can recognize the user's voice (S655). Specifically, the robot (100) can drive the robot (100) to face the direction in which the user is located based on the direction in which the user's voice was spoken. In addition, the robot (100) can perform voice recognition for (additional) user voice by performing beamforming in the direction in which the user's voice was spoken.

[0119] The artificial intelligence-related functions according to the present disclosure are operated through the processor and memory of the robot (100).

[0120] The processor may be comprised of one or more processors. The one or more processors may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Processing Unit), but are not limited to the examples of the processors described above.

[0121] CPUs are general-purpose processors capable of performing not only general calculations but also artificial intelligence calculations. Their multi-layered cache structure allows for the efficient execution of complex programs. CPUs are advantageous for serial processing, enabling organic linking of previous and subsequent calculation results through sequential calculations. General-purpose processors are not limited to the examples described above, except where specifically identified as CPUs.

[0122] A GPU is a processor designed for large-scale computations, such as floating-point operations used in graphics processing. It integrates a large number of cores to perform large-scale computations in parallel. In particular, GPUs may be advantageous over CPUs in parallel processing methods, such as convolution operations. Furthermore, GPUs can be used as coprocessors to supplement the functions of CPUs. Processors for large-scale computations are not limited to the examples described above, except in cases where they are specifically referred to as GPUs.

[0123] An NPU is a processor specialized in artificial intelligence computation using artificial neural networks. Each layer of an artificial neural network can be implemented in hardware (e.g., silicon). Since NPUs are designed specifically according to a company's specifications, they have less freedom than CPUs or GPUs, but they can efficiently process the AI ​​computations requested by companies. As a processor specialized in AI computation, an NPU can be implemented in various forms, such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), or a Vision Processing Unit (VPU). Except as specifically designated as an NPU, an AI processor is not limited to the examples described above.

[0124] Additionally, one or more processors may be implemented as a System on Chip (SoC). In addition to one or more processors, the SoC may further include memory and a network interface, such as a bus, for data communication between the processor and the memory.

[0125] If the SoC (System on Chip) included in the robot includes multiple processors, the robot can perform artificial intelligence-related operations (e.g., operations related to learning or inference of an artificial intelligence model) using some of the multiple processors. For example, the robot can perform artificial intelligence-related operations using at least one of a GPU, NPU, VPU, TPU, or hardware accelerator specialized in artificial intelligence operations such as convolution operations or matrix multiplication operations among the multiple processors. However, this is only one embodiment, and it is obvious that artificial intelligence-related operations can be processed using a CPU or other general-purpose processor.

[0126] Additionally, the robot can perform computations related to AI-related functions by utilizing multiple cores (e.g., dual cores, quad cores, etc.) contained within a single processor. In particular, the robot can perform AI operations such as convolution and matrix multiplication in parallel by utilizing the multiple cores contained within the processor.

[0127] One or more processors are controlled to process input data according to predefined operating rules or artificial intelligence models stored in memory. The predefined operating rules or artificial intelligence models are characterized by being created through learning.

[0128] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0129] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0130] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0131] Methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0132] The methods according to various embodiments of the present disclosure may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call commands stored in the storage medium and operate according to the called commands, and may include a robot according to the disclosed embodiments.

[0133] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0134] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0135] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In robots, At least one sensor speaker; mike; driving part; At least one memory storing one or more instructions; and At least one processor configured to execute one or more of the above instructions; When said one or more instructions are executed by said at least one processor, Generating a map including information on a plurality of objects based on sensing information acquired through at least one sensor, Generate ultrasonic waves for each of the plurality of objects through the above speakers, Based on the reflected sound reflected from each of the objects and received through the microphone, reflectivity information for the plurality of objects is obtained, and the reflectivity information is stored, and the reflected sound reflected from each of the objects is at least a part of the ultrasonic wave reflected from each object, and when a user's voice is received through the microphone, information on the intensity of the user's voice is obtained for each of the plurality of directions, Based on information about the intensity of the user voice for each of the plurality of directions, information about a plurality of candidate directions from which the user voice was received among the plurality of directions is obtained, Obtain priority information for the plurality of candidate directions based on the position of the robot and the stored reflectivity information, A robot that obtains information about the direction in which the user's voice was spoken among the plurality of candidate directions based on the above priority information.

2. In paragraph 1, When said one or more instructions are executed by said at least one processor, For the wall object among the above multiple objects, ultrasonic waves are generated at preset distance intervals, and the reflected sound reflected from the wall object is received through the microphone to obtain reflectivity information for the wall object. A robot in which the reflected sound reflected from the wall object is at least a portion of the ultrasonic waves generated at preset intervals by reflecting from the wall object.

3. In paragraph 1, When said one or more instructions are executed by said at least one processor, Generate ultrasonic waves from two or more directions for the remaining objects, excluding the wall object, among the above multiple objects, A robot that obtains reflectivity information for the object by obtaining an average value of reflected sounds reflected from the object and received through the microphone, wherein the reflected sounds reflected from the object are at least part of ultrasonic waves reflected in two or more directions.

4. In paragraph 1, When said one or more instructions are executed by said at least one processor, Obtain information about the intensity of user voice received from the plurality of directions based on the position of the robot, A robot that identifies a preset number of directions in which the intensity of the user's voice is greater than a preset threshold value among the plurality of directions as the plurality of candidate directions.

5. In paragraph 1, When said one or more instructions are executed by said at least one processor, Based on the position of the robot, identify an object located in the candidate direction among the plurality of objects, A robot that identifies priorities for the plurality of candidate directions based on information about the intensity of a user's voice measured from each of the plurality of candidate directions and reflectivity information corresponding to the identified object.

6. In paragraph 5, When said one or more instructions are executed by said at least one processor, Obtain a corrected intensity of the user voice obtained from the plurality of candidate directions by multiplying the intensity of the user voice corresponding to each of the plurality of candidate directions by a weight, Identify priorities based on the corrected century for each of the above multiple candidate directions, The above weights correspond to the robot reflectivity information corresponding to the identified object.

7. In paragraph 6, A robot in which the weight and reflectivity information corresponding to the identified object are inversely proportional to each other.

8. In paragraph 6, The priority for each candidate direction among the above multiple candidate directions is directly proportional to the corrected century, When said one or more instructions are executed by said at least one processor, A robot that identifies the candidate direction with the highest priority among the above multiple candidate directions as the direction in which the user's voice was spoken.

9. In paragraph 1, When said one or more instructions are executed by said at least one processor, A robot that performs voice recognition for a user's voice by performing beamforming in the direction in which the user's voice is spoken.

10. In the method of controlling a robot, A step of generating a map including information on a plurality of objects based on sensing information acquired through at least one sensor of the robot; A step of generating ultrasonic waves for each of the plurality of objects through the speaker of the robot; A step of obtaining reflectivity information for the plurality of objects based on reflected sounds reflected from each of the objects and received through a microphone of the robot, and storing the reflectivity information; The reflected sounds reflected from each of the objects are at least a portion of the ultrasonic waves reflected from each object, A step of obtaining information on the intensity of the user's voice in multiple directions when a user's voice is received through the above microphone; A step of obtaining information about a plurality of candidate directions in which the user voice was spoken among the plurality of directions based on information about the intensity of the user voice in the plurality of directions; A step of obtaining priority information for the plurality of candidate directions based on the position of the robot and the stored reflectivity information; and A control method comprising: a step of obtaining information about the direction in which the user's voice was spoken among the plurality of candidate directions based on the priority information.

11. In paragraph 10, The step of generating ultrasonic waves for each of the above multiple objects is: For the wall object among the above multiple objects, ultrasonic waves are generated at preset distance intervals, The step of obtaining and storing the above reflectivity information is: A control method for obtaining reflectivity information about the wall object by receiving reflected sound reflected from the wall object through the microphone, wherein the reflected sound reflected from the wall object is at least a part of ultrasonic waves reflected from the wall object and generated at preset intervals.

12. In paragraph 10, The step of generating ultrasonic waves for each of the above multiple objects is: Generate ultrasonic waves from two or more directions for the remaining objects, excluding the wall object, among the above multiple objects, The step of obtaining and storing the above reflectivity information is: Obtain the average value of the reflected sound reflected from the object and received through the microphone to obtain reflectivity information for the object. A control method wherein the reflected sound reflected from the object is at least a part of the ultrasonic waves reflected in two or more directions.

13. In paragraph 10, The step of obtaining information about the above multiple candidate directions is: A control method for identifying a preset number of directions in which the intensity of the user's voice is greater than a preset threshold value among the plurality of directions as the plurality of candidate directions.

14. In paragraph 10, The step of obtaining the above priority information is: Based on the position of the robot, identify an object located in the candidate direction among the plurality of objects, A control method for identifying priorities for the plurality of candidate directions based on information about the intensity of a user's voice corresponding to each of the plurality of candidate directions and reflectivity information corresponding to the identified object.

15. In paragraph 14, The step of obtaining the above priority information is: Obtain a corrected intensity of the user voice obtained from the plurality of candidate directions by multiplying the intensity of the user voice corresponding to each of the plurality of candidate directions by a weight, Identify priorities based on the corrected century for each of the above multiple candidate directions, The above weight is a control method corresponding to the reflectance information corresponding to the identified object.

Images