Electronic device for identifying location of user and control method thereof

Abstract

Disclosed is an electronic device comprising: a microphone; a communication interface; a memory; and one or more processors for controlling the electronic device. The processor may: control the communication interface such that if one or more commands stored in the memory are executed and thus a preset user voice of a user is received through the microphone, one or more pieces of first sound information are requested from the one or more home appliances; allowing one or more pieces of corresponding first sound information corresponding to preset user voice to be received from each of the one or more household appliances; receiving one or more pieces of first sound information from one or more household appliances through the communication interface; and identifying the position of the user based on the preset user voice and the one or more pieces of first voice information.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for identifying a user's location and a method for controlling the electronic device. Background Art

[0002] Due to the development of electronic technology, various types of electronic devices have been developed, such as electronic devices equipped with a driver and capable of positional movement (eg, robot devices), in which the electronic device can move to the user's location according to the user's call.

[0003] However, there is a problem that the electronic device cannot accurately recognize the user's location due to the surrounding environment.

[0004] Furthermore, when the robotic device moves in a plurality of spaces in a house to perform functions, it is necessary to restrict the movement of the robotic device in a privacy area or space having a higher risk of privacy invasion under certain conditions. Summary of the invention

[0005] According to one aspect of the present disclosure, an electronic device includes: a microphone; a communication interface; a memory storing at least one instruction; and at least one processor connected to the microphone and the communication interface. The at least one processor is configured to run at least one instruction to: based on a preset user voice of a user received through the microphone, control the communication interface to request at least one first sound information from at least one home appliance, wherein each of the at least one first sound information corresponds to a preset user voice received at each of the at least one home appliance, receive at least one first sound information from at least one home appliance through the communication interface, and identify the user's location based on the preset user voice and the at least one first sound information.

[0006] The processor can also be configured to run at least one instruction to: based on receiving a preset user voice through the microphone, control the communication interface to send time information indicating when the preset user voice is received to at least one household appliance, and receive at least one first sound information corresponding to the time information from at least one household appliance through the communication interface.

[0007] The memory may be further configured to store location information of at least one home appliance, and the processor may be further configured to execute at least one instruction to identify the user's location based on the location information, a preset user voice, and at least one first sound information.

[0008] The electronic device may also include: a speaker, wherein the processor may also be configured to run at least one instruction to: output an inaudible sound through the speaker based on the occurrence of a preset event, receive at least one second sound information corresponding to the inaudible sound from at least one household appliance through the communication interface, and identify the location of at least one household appliance based on the location of the electronic device that outputs the inaudible sound and the at least one second sound information.

[0009] The electronic device may also include: a driver, wherein the processor may also be configured to run at least one instruction to: control the driver to move the electronic device to each of a plurality of positions, output an inaudible sound through a speaker at each of the plurality of positions, and identify the position of at least one household appliance based on the plurality of positions and at least one second sound information received at each of the plurality of positions.

[0010] The preset event includes at least one of an event of initializing the electronic device and an event of not responding to a request for at least one first sound information.

[0011] The memory may be configured to store delay information of at least one household appliance in each of multiple areas, and the processor may be further configured to run at least one instruction to: identify delay information of each of at least one first sound information, and identify the user's location based on the identified delay information and the delay information stored in the memory.

[0012] The electronic device may further include: a driver, wherein the processor may be further configured to execute at least one instruction to control the driver to move the electronic device to a location of a user.

[0013] The electronic device may further include a camera, wherein the processor may be further configured to execute at least one instruction to capture an image through the camera based on the electronic device moving to the position of the user, and recognize the user based on the image.

[0014] The memory may also be configured to store user information, wherein facial information of each of a plurality of users and voice information of each of a plurality of users are mapped in the user information, and the processor may also be configured to run at least one instruction to: obtain facial information corresponding to a preset user voice based on the user information, and identify a user corresponding to the facial information from an image.

[0015] The electronic device may further include: a speaker, wherein the processor may further be configured to run at least one instruction to output a preset sound through the speaker based on not obtaining facial information corresponding to a preset user voice based on the user information or based on not recognizing a user corresponding to the facial information from the image.

[0016] According to one aspect of the present disclosure, a method for controlling an electronic device includes: receiving a preset user voice of a user; requesting at least one first sound information from at least one household appliance, wherein each of the at least one first sound information corresponds to the preset user voice received at each of the at least one household appliance; receiving at least one first sound information from at least one household appliance; and identifying the user's location based on the preset user voice and the at least one first sound information.

[0017] Requesting at least one first sound information may include: based on receiving a preset user voice, sending time information about when the preset user voice is received to at least one home appliance, wherein the at least one first sound information received from the at least one home appliance corresponds to the time information.

[0018] Identifying the location of the user may include: identifying the location information of at least one home appliance by calculating delay information of each of at least one first sound information received from the at least one home appliance, and identifying the location of the user based on the location information of the at least one home appliance.

[0019] Identifying the location of the user may include identifying the location of the user based on location information of at least one home appliance, a preset user voice, and at least one first sound information.

[0020] Identifying the location information of at least one household appliance may include: outputting an inaudible sound based on the occurrence of a preset event; receiving at least one second sound information corresponding to the inaudible sound from at least one household appliance; and identifying the location of at least one household appliance based on the location of the electronic device that outputs the inaudible sound and the at least one second sound information.

[0021] The method may also include: controlling the electronic device to move to each of a plurality of positions; outputting an inaudible sound at each of the plurality of positions; and identifying the position of at least one household appliance based on the plurality of positions and at least one second sound information received at each of the plurality of positions.

[0022] The method may further include controlling the electronic device to move to the location of the user.

[0023] The method may further include: capturing an image at the user's location based on the electronic device moving to the user's location; and recognizing the user based on the image and facial information corresponding to a preset user voice.

[0024] According to one aspect of the present disclosure, a non-transitory computer-readable medium stores a computer-readable program code or instruction executable by a processor to perform a method of controlling an electronic device. The method includes: receiving a preset user voice of a user; requesting at least one first sound information from at least one home appliance, wherein each of the at least one first sound information corresponds to a preset user voice received at each of the at least one home appliance; receiving at least one first sound information from at least one home appliance; and identifying the user's location based on the preset user voice and the at least one first sound information. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which: Figure 1 is a diagram illustrating a mobile operation of a device according to a user call according to one or more embodiments; Figure 2 is a block diagram showing a configuration of an electronic system according to one or more embodiments; Figure 3 is a block diagram showing a configuration of an electronic device according to one or more embodiments; Figure 4 is a block diagram showing a detailed configuration of an electronic device according to one or more embodiments; Figure 5 is a diagram illustrating a multi-channel microphone environment according to one or more embodiments; Figures 6 to 9 is a diagram illustrating a method of recognizing a location of a user without using a location of at least one home appliance according to one or more embodiments; Figures 10 to 12 is a diagram illustrating a method of identifying a user's location using a location of at least one home appliance according to one or more embodiments; and Fig.13 is a flowchart illustrating a control method of an electronic device according to one or more embodiments. DETAILED DESCRIPTION

[0026] The example embodiments of the present disclosure may be modified in various ways. Therefore, specific example embodiments are shown in the drawings and described in detail in the detailed description. However, it will be understood that the present disclosure is not limited to specific example embodiments, but includes all modifications, equivalents and alternatives without departing from the scope and spirit of the present disclosure. In addition, well-known functions or configurations are not described in detail because they would obscure the present disclosure with unnecessary details.

[0027] The present disclosure provides an electronic device for identifying a user's location according to a call of the user and a control method thereof.

[0028] Although the general terms used in the embodiments of the present disclosure are selected to describe the embodiments in consideration of their functions, these general terms may change according to the intentions of those of ordinary skill in the art, legal or technical interpretations, the emergence of new technologies, etc. Some terms are arbitrarily selected by the applicant of the embodiments. In this case, their meanings will be described in detail in the description of the present disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the overall content of the present disclosure, rather than based on the name definitions of simple terms.

[0029] In this specification, the expressions “have”, “may have”, “include” and “comprise” or “may include” and “may include” used herein indicate the existence of corresponding features (for example, elements such as values, functions, operations or components), but do not exclude the existence of additional features.

[0030] The expression at least one of A or B should be understood to mean "A" or "B" or "A and B".

[0031] Terms such as “first”, “second”, etc. used herein may refer to various elements regardless of the order and / or priority of the elements and may be used to distinguish one element from another rather than to limit the elements.

[0032] Unless the context clearly indicates otherwise, the singular includes the plural. In this application, the terms "comprising" or "configuration" etc. specify the presence of features, numbers, steps, operations, elements, components or combinations thereof, but do not exclude the presence or addition of one or more features, numbers, steps, operations, components, parts or combinations thereof.

[0033] In this specification, the term "user" may refer to a person using an electronic device or a device (eg, an artificial intelligence electronic device) using an electronic device.

[0034] Hereinafter, various example embodiments of the present disclosure will be described with reference to the accompanying drawings, wherein like reference characters denote corresponding features throughout.

[0035] Figure 1 is a diagram illustrating a mobile operation of a device according to a user's call according to one or more embodiments.

[0036] The device may move to the user's location according to the user's call in the room. However, in some cases, the device may not be able to accurately identify the user's location due to noise, obstacles, reflection, absorption, etc. In particular, when the device cannot identify the user's location, the device may need to search the space in sequence, such as Figure 1 shown.

[0037] To improve this, the device may include a plurality of microphones to more accurately identify the user's position, but in this case, the manufacturing cost of the device may increase.

[0038] Figure 2 1 is a block diagram showing a configuration of an electronic system 1000 according to one or more embodiments. Figure 2 As shown, the electronic system 1000 includes an electronic device 100 and a plurality of home appliances 200-1 to 200-N. However, the electronic system 1000 is not limited thereto and may include only the electronic device 100 and one home appliance. Hereinafter, for ease of description, the plurality of home appliances 200-1 to 200-N are described as at least one home appliance 200.

[0039] The electronic device 100 is a device that identifies the location of the user and may be a mobile device such as a robot vacuum cleaner. However, the electronic device 100 is not limited thereto and may be a device implemented as a computer host, a set-top box (STB), an artificial intelligence speaker, a television, a desktop computer, a laptop computer, a smart phone, a tablet computer, smart glasses, a smart watch, etc., to identify the location of the user and provide information about the location of the user to a mobile device such as a robot vacuum cleaner. Any device that can identify the location of the user may be used.

[0040] When receiving a preset user voice of a user, the electronic device 100 may request sound information corresponding to the user's voice from at least one home appliance 200, and recognize the user's location based on the preset user voice and the sound information received from at least one home appliance 200. According to an embodiment, the electronic device 100 may pre-store location information of at least one home appliance 200, and recognize the user's location based on the location information, the preset user voice, and the sound information.

[0041] The at least one home appliance 200 is a device that receives a user voice and sends sound information corresponding to the user voice to the electronic device 100, and may be a computer host, STB, artificial intelligence speaker, television, desktop computer, laptop computer, smart phone, tablet computer, smart glasses, smart watch, refrigerator, washing machine, air conditioner, etc. However, the at least one home appliance 200 is not limited thereto, and any device that can receive a user voice through a microphone and send sound information corresponding to the user voice to the electronic device 100 through a communication interface may be used.

[0042] When receiving a preset user voice of a user and receiving a request for sound information corresponding to the user voice from the electronic device 100 , at least one home appliance 200 may transmit the sound information to the electronic device 100 .

[0043] Figure 3 is a block diagram illustrating a configuration of an electronic device 100 according to one or more embodiments.

[0044] like Figure 3 As shown, the electronic device 100 includes a microphone 110 , a communication interface 120 , and a processor 130 .

[0045] The microphone 110 is a component for receiving sound and converting the sound into an audio signal. The microphone 110 may be electrically connected to the processor 130 and receive the sound by the control of the processor 130.

[0046] For example, the microphone 110 may be integrally formed with the electronic device 100 on the upper side, front side, or side side of the electronic device 100. According to an embodiment, the microphone 110 may be provided in a remote controller separated from the electronic device 100. In this case, the remote controller may receive sound through the microphone 110 and provide the received sound to the electronic device 100.

[0047] The microphone 110 may include various configurations, such as a microphone for collecting analog sounds, an amplifier circuit for amplifying the collected sounds, an A / D conversion circuit for sampling the amplified sounds and converting the sounds into digital signals, a filter circuit for removing noise components from the converted digital signals, and the like.

[0048] According to an embodiment, the microphone 110 may be implemented in the form of a sound sensor, and any component capable of collecting sound may be used.

[0049] The communication interface 120 is a component that performs communication with various types of external devices according to various types of communication methods. For example, the electronic device 100 can communicate with at least one home appliance 200 through the communication interface 120.

[0050] The communication interface 120 may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. According to an embodiment, each communication module may be implemented in the form of at least one hardware chip.

[0051] The Wi-Fi module and the Bluetooth module perform communication using the Wi-Fi method and the Bluetooth method, respectively. When using the Wi-Fi module or the Bluetooth module, the electronic device 100 may first send and receive various types of connection information (such as SSID, session key, etc.), connect the communication using the connection information, and then send and receive various types of information. The infrared communication module performs communication according to the Infrared Data Association (IrDA) technology that wirelessly transmits data within a short distance using infrared rays between visible light and millimeter waves.

[0052] In addition to the above-mentioned communication methods, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards (such as Zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G), fifth generation (5G), etc.).

[0053] According to an embodiment, the communication interface 120 may include a wired communication interface (such as HDMI, DP, Thunderbolt interface, USB, RGB, D-SUB, DVI, etc.).

[0054] According to an embodiment, the communication interface 120 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module that performs communication using a twisted pair cable, a coaxial cable, or an optical fiber cable.

[0055] The processor 130 generally controls the operation of the electronic device 100. Specifically, the processor 130 may be connected to each component of the electronic device 100 to control the overall operation of the electronic device 100. For example, the processor 130 may be connected to components such as the microphone 110, the communication interface 120, a memory (e.g., the memory 140), and a display (e.g., the display 170) to control the operation of the electronic device 100.

[0056] According to one or more embodiments, the processor 130 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON). However, the processor 130 is not limited thereto, and may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, or may be defined as corresponding terms. According to an embodiment, the processor 130 may be implemented as a system on chip (SoC) or a large-scale integrated circuit (LSI) in which a processing algorithm is embedded, or may be implemented as a field programmable gate array (FPGA).

[0057] The processor 130 may be implemented as one processor or a plurality of processors. However, hereinafter, for convenience of explanation, the operation of the electronic device 100 will be described using the expression of the processor 130 .

[0058] When a preset user voice of a user is received through the microphone 110, the processor 130 may control the communication interface 120 to request at least one first sound information corresponding to the received preset user voice of the user from at least one home appliance 200. For example, when a user voice "Robot, come here!" is received, the processor 130 may control the communication interface 120 to request at least one first sound information corresponding to the received "Robot, come here!" from at least one home appliance 200.

[0059] The processor 130 may receive at least one first sound information from at least one home appliance 200 through the communication interface 120, and identify the user's location based on a preset user voice and the at least one first sound information. The method for identifying the user's location may be a rule-based method or a method using a neural network model. A detailed description of this aspect will be described below with the accompanying drawings.

[0060] When a preset user voice is received through the microphone 110, the processor 130 may control the communication interface 120 to send time information about when the preset user voice is received to at least one home appliance 200, and may receive at least one first sound information corresponding to the time information from at least one home appliance 200 through the communication interface 120. For example, when a user voice "Robot, come here!" is received, the processor 130 may control the communication interface 120 to send the time 15:00 when the user voice "Robot, come here!" is received to at least one home appliance 200, and may receive at least one first sound information from at least one home appliance 200 after 15:00 through the communication interface 120. Through this operation, unnecessary time periods other than the preset user voice in the first sound information may be reduced.

[0061] According to an embodiment, when a preset user voice is received through the microphone 110, the processor 130 may control the communication interface 120 to send time information about when the preset user voice is received and the duration of the preset user voice to at least one household appliance 200, and receive at least one first sound information corresponding to the time information and the duration from at least one household appliance 200 through the communication interface 120.

[0062] The electronic device 100 may further include a memory (e.g., memory 140) storing location information of at least one home appliance, and the processor 130 may identify the location of the user based on the location information, a preset user voice, and at least one first sound information. Therefore, the processor 130 may identify the location of the user in the same manner as arranging microphones at multiple locations.

[0063] The electronic device 100 may also include a speaker (e.g., speaker 150), and when a preset event occurs, the processor 130 may output an inaudible sound through the speaker, receive at least one second sound information corresponding to the inaudible sound from at least one household appliance 200 through the communication interface 120, and identify the location of at least one household appliance 200 based on the location of the electronic device 100 that outputs the inaudible sound and the at least one second sound information.

[0064] According to an embodiment, the preset event may include at least one of an event of initializing the electronic device 100 and an event of no response to a request for at least one first sound information. That is, when the electronic device 100 is arranged in a new place, the processor 130 may identify the location of at least one home appliance 200 in the same manner as described above. According to an embodiment, when there is no response to a request for at least one first sound information, the processor 130 may recognize that the location of the existing home appliance has changed, and identify the location of at least one home appliance 200 in the same manner as described above. According to an embodiment, the processor 130 may obtain information about the space in which the electronic device 100 is located while identifying the location of at least one home appliance 200 in the same manner as described above. That is, the processor 130 may obtain information about the space in which the electronic device 100 is located and the location of at least one home appliance 200 in the space as a map. According to an embodiment, the space in which the electronic device 100 is located may include all spaces in which the electronic device 100 can move.

[0065] In the above, a state in which the electronic device 100 pre-stores the location information of at least one home appliance 200 has been described. However, the present disclosure is not limited thereto, and the processor 130 may identify the location of the user without the location information of at least one home appliance 200. For example, the electronic device 100 may further include a memory storing delay information of at least one home appliance in each of a plurality of areas, and the processor 130 may identify delay information of each of at least one first sound information, and identify the location of the user based on the identified delay information and the delay information stored in the memory.

[0066] The electronic device 100 may also include a driver (e.g., driver 160), and the processor 130 may control the driver to move the electronic device 100 to each of a plurality of positions, output an inaudible sound through a speaker at each of the plurality of positions, and identify the position of at least one household appliance 200 based on the plurality of positions and at least one second sound information received from each of the plurality of positions.

[0067] According to an embodiment, the processor 130 may control the driver to move the electronic device 100 to the user's location.

[0068] The electronic device 100 may further include a camera (eg, camera 190 ), and when the electronic device 100 moves to a location of a user, the processor 130 may capture an image through the camera and recognize the user based on the image.

[0069] For example, the electronic device 100 may further include a memory (e.g., memory 140) storing user information, in which facial information of each of the plurality of users and voice information of each of the plurality of users are mapped, and the processor 130 may obtain facial information corresponding to a preset user voice based on the user information, and recognize the user corresponding to the facial information from the image. According to an embodiment, the processor 130 may recognize the user from the image through a neural network model.

[0070] According to an embodiment, when facial information corresponding to a preset user voice is not obtained based on user information or a user corresponding to the facial information is not recognized from an image, the processor 130 may output a preset sound through a speaker. For example, when facial information corresponding to a preset user voice is not obtained based on user information or a user corresponding to the facial information is not recognized from an image, the processor 130 may output a sound such as “Who is calling?” through a speaker.

[0071] According to an embodiment, functions related to artificial intelligence (AI) according to the present disclosure may be performed by the processor 130 and the memory.

[0072] The processor 130 may include one processor or multiple processors, such as a general-purpose processor (such as a CPU, an application processor (AP), a digital signal processor (DSP), etc.), a dedicated processor (such as a GPU, a visual processing unit (VPU), etc.), or an AI dedicated processor (such as a neural processing unit (NPU)).

[0073] The processor 130 may process input data according to a predefined operating rule or AI model stored in the memory. According to an embodiment, when the processor 130 includes an AI dedicated processor, the AI ​​dedicated processor may be designed to have a hardware structure dedicated to processing a specific AI model. The predefined operating rule or AI model may be formulated through training.

[0074] In this article, when an AI model is formulated by training, this may mean that a basic AI model is trained based on a learning algorithm using multiple training data sets, thereby formulating a predefined operating rule or AI model that performs a desired characteristic (or purpose). Such learning may be performed by a device implementing an AI according to the present disclosure or by a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0075] The AI ​​model may include multiple neural network layers. Each of the multiple neural network layers may have multiple weight values, and perform neural network operations by operating between the operation results of the previous layer and the multiple weight values. The multiple weight values ​​in the multiple neural network layers may be optimized by the training results of the AI ​​model. For example, the multiple weight values ​​may be updated to reduce or minimize the loss value or cost value obtained in the AI ​​model during the training process.

[0076] Examples of AI neural networks may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), or a deep Q-network.

[0077] Figure 4 is a block diagram illustrating a detailed configuration of the electronic device 100 according to one or more embodiments.

[0078] The electronic device 100 may include a microphone 110, a communication interface 120, and a processor 130. Figure 4 , the electronic device 100 may further include a memory 140, a speaker 150, a driver 160, a display 170, a user interface 180, and a camera 190. Figure 4 In the components shown, the Figure 3 Detailed description of the redundant components shown.

[0079] The memory 140 may refer to hardware that stores information such as data in an electrical or magnetic form so that the information can be accessed by the processor 130, etc. To this end, the memory 140 may be implemented as at least one hardware of a nonvolatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a RAM, a ROM, etc.

[0080] At least one instruction required for the operation of the electronic device 100 or the processor 130 may be stored in the memory 140. According to an embodiment, the instruction is a code unit that instructs the operation of the electronic device 100 or the processor 130, and may be written in a machine language understandable by a computer. According to an embodiment, a plurality of instructions for performing a specific task of the electronic device 100 or the processor 130 may be stored in the memory 140 as an instruction set.

[0081] The memory 140 may store data, which is information in units of bits or bytes that can represent characters, numbers, images, etc. For example, mapping information, user information, and a neural network model may be stored in the memory 140 .

[0082] The memory 140 may be accessed by the processor 130 , and instructions, instruction sets, or data may be read / recorded / modified / deleted / updated by the processor 130 .

[0083] The speaker 150 is a component that outputs not only various audio data processed by the processor 130 but also various notification sounds or voice messages.

[0084] The driver 160 is a component of the mobile electronic device 100, and may include, for example, at least one wheel and a motor driving the wheel. However, the driver 160 is not limited thereto, and may include legs for four-legged walking and motors driving the legs, and any component capable of moving the electronic device 100 may be used.

[0085] The display 170 is a component that displays an image, and may be implemented as various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display (PDP), etc. The display 170 may further include a driving circuit that may be implemented in the form of an a-si TFT, a low temperature polycrystalline silicon (LTPS) TFT, or an organic TFT (OTFT), a backlight unit, etc. According to an embodiment, the display 170 may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0086] The user interface 180 may be implemented as a button, a touch pad, a mouse, a keyboard, etc., or may be implemented as a touch screen capable of performing a display function and an operation input function. According to an embodiment, the button may be various types of buttons (such as a mechanical button, a touch pad, a scroll wheel, etc.) formed in any area (such as the front, side, or rear of the main body appearance of the electronic device 100).

[0087] The camera 190 is a component that captures a still image or a moving image. The camera 190 may capture a still image at a specific point in time, but may also capture still images continuously.

[0088] The camera 190 may capture a real environment in front of the electronic device 100 by capturing the front of the electronic device 100. The processor 130 may recognize a user from an image captured by the camera 190.

[0089] The camera 190 includes a lens, a shutter, an aperture, a solid-state imaging device, an analog front end (AFE), and a timing generator (TG). The shutter adjusts the time taken for light reflected from the subject to enter the camera 190, and the aperture adjusts the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which the light passes. When the light reflected from the subject is accumulated as photocharges, the solid-state imaging device outputs an image generated by the photocharges as an electrical signal. The TG outputs a timing signal for reading out pixel data of the solid-state imaging device, and the AFE samples and digitizes the electrical signal output from the solid-state imaging device.

[0090] As described above, the electronic device 100 can more accurately identify the user's position by using at least one home appliance as a microphone, and can identify a user who makes a preset user voice based on the user's facial information and voice information even if there are multiple users.

[0091] In the following, reference will be made to Figures 5 to 12 The operation of the electronic device 100 is described in more detail. For the convenience of description, Figures 5 to 12 Individual embodiments are described. However, they can be implemented in any combination. Figures 5 to 12 A separate embodiment of the invention.

[0092] Figure 5 is a diagram illustrating a multi-channel microphone environment according to one or more embodiments. Figure 5 In the figure, for the convenience of description, the electronic device 100 is shown as a robot.

[0093] like Figure 5 As shown, the electronic device 100 can communicate with the AI ​​speaker, refrigerator, or air conditioner. In addition, the electronic device 100 may be in a state where information about the space where the electronic device 100 is located and location information of each of the AI ​​speaker, refrigerator, and air conditioner are pre-stored.

[0094] When receiving the user voice "Robot, come here!", the electronic device 100 may request the first sound information corresponding to the preset user voice from each of the AI ​​speaker, refrigerator, and air conditioner, and receive the first sound information from each of the AI ​​speaker, refrigerator, and air conditioner. According to an embodiment, the first sound information may be information corresponding to the preset user voice received by each of the AI ​​speaker, refrigerator, and air conditioner. According to an embodiment, since the AI ​​speaker, refrigerator, and air conditioner are arranged in different positions, the first sound information of the AI ​​speaker, the first sound information of the refrigerator, and the first sound information of the air conditioner may all be different.

[0095] The electronic device 100 can identify the user's location based on the location information of each of the AI ​​speaker, refrigerator and air conditioner, the preset user voice received by the electronic device 100, and the first sound information received from each of the AI ​​speaker, refrigerator and air conditioner.

[0096] As described above, even if the electronic device 100 includes only one microphone 110, a multi-channel microphone environment can be constructed by using at least one home appliance 200 as a microphone, and the position of the user can be recognized more accurately.

[0097] Figures 6 to 9is a diagram illustrating a method of recognizing a location of a user without using a location of at least one home appliance 200 according to one or more embodiments.

[0098] Reference Figure 6 In operation S610, the processor 130 may generate a search area. Figure 7 As shown, the electronic device 100 may be a robot vacuum cleaner, and the processor 130 may move the electronic device 100 to a next position and generate an area where a search will be performed. The processor 130 may determine the next position based on the current position of the electronic device 100 and the diameter of the electronic device 100. For example, the processor 130 may move the electronic device 100 according to the diameter of the electronic device 100. In some cases, the next position may be the same as the current position, and the processor 130 may determine that the electronic device 100 is moved to the next position without activating the driver 160. The processor 130 may move the electronic device 100 to a plurality of positions, and at each position, the processor 130 may perform the following steps. Figure 6 One or more operations. Figure 7 As shown, the processor 130 may sequentially move the electronic device 100 to each of the positions 710, 720, 730, 740, etc. The processor 130 may execute at the position 710 Figure 6 One or more operations are then performed at position 720, and the same operations are sequentially performed at position 730, position 740, and so on.

[0099] In operation S620, the processor 130 may output an inaudible sound, and in operation S630, the processor 130 may receive sound information corresponding to the inaudible sound from at least one home appliance 200. In operation S640, the processor 130 may calculate delay information by obtaining a timestamp when the inaudible sound is first received from each sound information, and in operation S650, the processor 130 may store the delay information in a database. According to an embodiment, the delay information may be obtained based on a plurality of timestamps obtained from a plurality of sound information received from at least one home appliance 200.

[0100] For example, the processor 130 may receive sound information from an air conditioner and sound information from a refrigerator. The processor 130 may identify a timestamp of 0.2s when an inaudible sound is first received based on the sound information from the air conditioner, and may identify a timestamp of 0.4s when an inaudible sound is first received based on the sound information from the refrigerator. The processor 130 may identify a timestamp of each sound information received from a household appliance in at least one household appliance 200, and obtain an intermediate value. For example, assuming that there are only the above-mentioned air conditioner and refrigerator, the intermediate value is 0.3s, and the processor 130 may calculate (each timestamp intermediate value) and store the intermediate value in a database.

[0101] In operation S660, the processor 130 may identify whether there is a next area to be searched. If it is identified that there is a next area to be searched, in operation S670, the processor 130 may move the electronic device 100 to a position corresponding to the next area to be searched, and the processor 130 may return to operation S620.

[0102] like Figure 8 As shown, the database may include timestamped median values ​​for each of the plurality of household appliances at each location.

[0103] However, the present disclosure is not limited thereto, and the processor 130 may use an average value instead of a median value. According to an embodiment, the processor 130 may determine that the next position is at a distance other than the diameter of the electronic device 100, and the processor 130 may move the electronic device 100 to the next position at such a distance.

[0104] When the database is established as described above, even if there is no location information of at least one home appliance 200, the processor 130 can recognize the location of the user and refer to the location of the home appliance 200. Fig. 9 Describe this.

[0105] Reference Fig. 9 In operation S910, the processor 130 may detect a voice (e.g., receive a preset user voice), and in operation S920, the processor 130 may send a voice detection period (e.g., a timestamp) of the preset user voice to another home appliance (e.g., at least one home appliance 200). According to an embodiment, the voice detection period may include a reception time of the preset user voice. According to an embodiment, the voice detection period may include a reception time of the preset user voice and a duration of the preset user voice.

[0106] In operation S930, the processor 130 may receive voice data (sound information) corresponding to the voice detection period from at least one home appliance 200. In operation S940, the processor 130 may calculate the delay information by obtaining a timestamp of the first reception of the sound related to the voice detection from each sound information. Figure 6 The delay information is obtained in the same manner as described in .

[0107] In operation S950, the processor 130 may identify whether the delay information has a value having a similarity of 95% or more with the information included in the database (DB). If the delay information has a value having a similarity of 95% or more, the processor 130 may complete the position estimation in operation S960. According to an embodiment, the similarity threshold of 95% is only an example, and various values ​​(e.g., 99%, 85%, 50%, etc.) may be used as the similarity threshold.

[0108] If the delay information does not include a value having a similarity of 95% or more, the processor 130 may recognize that the location estimation has failed, and provide a user with a notification of the location estimation failure, in operation S970.

[0109] The processor 130 may identify whether the user agrees to perform the update, and if the user agrees, data generation may begin. According to an embodiment, data generation means Figure 7 operation.

[0110] The processor 130 may identify the location information of the user based on the sound information obtained in each search area through the above method.

[0111] Figures 10 to 12 is a diagram illustrating a method of recognizing a user's location using a location of at least one home appliance 200 according to one or more embodiments.

[0112] The processor 130 may obtain a power value according to each position based on the position of at least one home appliance 200, a preset user voice, and at least one first sound information received from at least one home appliance 200 to identify the position of the user. Fig.10 As shown, the power value can be obtained as a value between the maximum value and the minimum value, and as Fig.12 As shown, each of the plurality of positions may be associated with a power value between a maximum value and a minimum value.

[0113] Reference Fig.12 , the processor 130 may identify the space where the electronic device 100 is located as a plurality of circular areas. In addition, the processor 130 may obtain a power value based on the position of at least one home appliance 200 for each area, a preset user voice, and at least one first sound information.

[0114] Reference Fig.11 The processor 130 may shift each of at least one first sound information based on the delay information of the first sound information, obtain a waveform by matching the shifted first sound information with a preset user voice, and obtain a power value based on the waveform.

[0115] For example, the processor 130 may use a controllable response power phase transform (SRP-PHAT) to obtain a power value for each region. The SRP-PHAT method is robust in estimating distant sound sources, and may divide the space into a plurality of grids and calculate each controllable response power (SRP) value by the following equation.

[0116]

[0117] The processor 130 may identify a maximum power value from among a plurality of SRP values ​​calculated for each of a plurality of regions through the following equation, and identify a location of the user based on the maximum power value among the plurality of SRP values.

[0118]

[0119] For example, Fig.12 As shown, a power value for each region may be represented, and a region (eg, location 1210 ) having a maximum power value may be identified as the location of the user.

[0120] Fig.13 is a flowchart illustrating a control method of an electronic device according to one or more embodiments.

[0121] Reference Fig.13 In operation S1310, a preset user voice of a user is received. In operation S1320, at least one first sound information corresponding to the received preset user voice is requested from at least one home appliance. In operation S1330, at least one first sound information is received from at least one home appliance. In operation S1340, a location of the user is identified based on the preset user voice and the at least one first sound information.

[0122] According to an embodiment, in operation S1320, when a preset user voice is received, time information about when the preset user voice is received may be sent to at least one home appliance, and in operation S1330, at least one first sound information received from at least one home appliance corresponds to the time information.

[0123] According to an embodiment, in operation S1340, the location of the user may be recognized based on location information of at least one home appliance, a preset user voice, and at least one first sound information.

[0124] According to an embodiment, the method may further include: an operation of outputting an inaudible sound when a preset event occurs, an operation of receiving at least one second sound information corresponding to the inaudible sound from at least one household appliance, and an operation of identifying the location of at least one household appliance based on the location of the electronic device that outputs the inaudible sound and the at least one second sound information corresponding to the inaudible sound.

[0125] According to an embodiment, the method may further include: an operation of moving the electronic device to each of a plurality of positions. In the output operation, an inaudible sound may be output at each of the plurality of positions. In the operation of identifying the position of at least one household appliance, the position of at least one household appliance may be identified based on the plurality of positions and at least one second sound information received at each of the plurality of positions.

[0126] According to an embodiment, the preset event may include at least one of an event of initializing the electronic device and an event of no response to a request for at least one first sound information.

[0127] According to an embodiment, the method may further include an operation of identifying delay information of each of the at least one first sound information. In the identifying operation, in S1340, the position of the user may be identified based on the identified delay information and the pre-stored delay information.

[0128] According to an embodiment, the method may further include an operation of moving the electronic device to the user's location.

[0129] According to an embodiment, the method may further include: when the electronic device moves to the user's location, capturing an image and recognizing the user's operation based on the image.

[0130] According to an embodiment, the method may further include an operation of obtaining facial information corresponding to a preset user voice based on user information mapping facial information of each of a plurality of users and voice information of each of a plurality of users. In the operation of identifying a user, a user corresponding to the facial information may be identified from an image.

[0131] According to an embodiment, the method may further include an operation of outputting a preset sound when facial information corresponding to a preset user voice is not obtained based on the user information or a user corresponding to the facial information is not recognized from the image.

[0132] According to various embodiments of the present disclosure as described above, the electronic device can more accurately recognize the location of the user by using at least one home appliance as a microphone.

[0133] According to an embodiment, even if there are a plurality of users, the electronic device can identify a user who utters a preset user voice based on facial information and voice information of the users.

[0134] The various embodiments described above may be implemented as software including instructions stored in a storage medium readable by a machine (e.g., a computer). A machine is a device that can call a storage command from a storage medium and operate according to the called command, and may include an electronic device (e.g., electronic device A) according to an embodiment of the present disclosure. When a command is executed by a processor, the processor may directly execute a function corresponding to the command, or execute a function corresponding to the instruction by using other components under the control of the processor. The command may include code generated or run by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether the data is semi-permanently stored in the storage medium or temporarily stored in the storage medium.

[0135] The methods according to the various embodiments described above may be provided in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may include a product in the form of a machine-readable storage medium (e.g., a CD-ROM) or may be available through an application store (e.g., Google Play Store). TM or AppStore TM ) Online distribution. In the case of electronic distribution, at least parts of the computer program product may be at least temporarily stored or created on a storage medium, such as a memory of a manufacturer's server, a server of an application store or a relay server.

[0136] Various embodiments as described above can be implemented in a recording medium readable by a computer or similar device using software, hardware, or a combination of software and hardware. In some cases, the embodiments described herein can be implemented in the processor itself. According to software implementation, embodiments such as the programs and functions described herein can be implemented as separate software. Each software can perform one or more functions and operations described herein.

[0137] Computer instructions for executing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. According to the various embodiments described above, the computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause a specific device to perform processing operations in the device. A non-transitory computer-readable medium is not a medium that stores data for a short period of time (such as a register, cache, or memory), but a medium that stores data semi-permanently and is readable by a device. Specific examples of non-transitory computer-readable media may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, and the like.

[0138] Each component (e.g., module or program) according to the various embodiments described above may be composed of a single entity or multiple entities, and in various embodiments, some of the above subcomponents may be omitted, or other subcomponents may also be included. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform the functions performed by each corresponding component before integration in the same or similar manner. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, iteratively, or heuristically, or may be performed in a different order, at least some operations may be omitted, or another operation may be added.

[0139] Although specific example embodiments of the present disclosure have been shown and described above, the embodiments of the present disclosure are not limited to the above-described specific embodiments, and various modifications may be made by those skilled in the relevant art without departing from the gist of the present disclosure as claimed in the claims and their equivalents, and such modifications should not be understood solely from the technical ideas or prospects of the present disclosure.

Claims

1. An electronic device, comprising: microphone; Communication interface; A memory storing at least one instruction; as well as At least one processor is connected to the microphone and the communication interface and is configured to execute the at least one instruction to perform the following operations: Based on receiving a preset user voice of a user through the microphone, controlling the communication interface to request at least one first sound information from at least one household appliance, wherein each of the at least one first sound information corresponds to the preset user voice received at each of the at least one household appliance, receiving the at least one first sound information from the at least one household appliance through the communication interface, and The location of the user is identified based on the preset user voice and the at least one first sound information.

2. The electronic device according to claim 1, wherein: The processor is further configured to execute the at least one instruction to perform the following operations: based on receiving the preset user voice through the microphone, controlling the communication interface to send time information indicating when the preset user voice is received to the at least one home appliance, and The at least one first sound information corresponding to the time information is received from the at least one home appliance through the communication interface.

3. The electronic device according to claim 1, wherein: The memory is further configured to store location information of the at least one household appliance, and The processor is further configured to execute the at least one instruction to identify the location of the user based on the location information, the preset user voice and the at least one first sound information.

4. The electronic device according to claim 3, further comprising: speaker, The processor is further configured to execute the at least one instruction to perform the following operations: Based on the occurrence of a preset event, an inaudible sound is output through the speaker, receiving at least one second sound information corresponding to the inaudible sound from the at least one home appliance through the communication interface, and The location of the at least one home appliance is identified based on the location of the electronic device that outputs the inaudible sound and the at least one second sound information.

5. The electronic device according to claim 4, further comprising: Drive, The processor is further configured to execute the at least one instruction to perform the following operations: controlling the actuator to move the electronic device to each of a plurality of positions, outputting the inaudible sound at each of the plurality of locations through the speaker, and The location of the at least one home appliance is identified based on the plurality of locations and the at least one second sound information received at each of the plurality of locations.

6. The electronic device according to claim 4, wherein: The preset event includes at least one of an event of initializing the electronic device and an event of no response to a request for the at least one first sound information.

7. The electronic device according to claim 1, wherein: The memory is configured to store delay information of the at least one home appliance in each of a plurality of areas, and The processor is further configured to execute the at least one instruction to perform the following operations: identifying delay information of each of the at least one first sound information, and The location of the user is identified based on the identified delay information and the delay information stored in the memory.

8. The electronic device according to claim 1, further comprising: Drive, The processor is further configured to execute the at least one instruction to control the driver to move the electronic device to the user's location.

9. The electronic device according to claim 8, further comprising: camera, The processor is further configured to execute the at least one instruction to perform the following operations: capturing an image with the camera based on the electronic device moving to the position of the user, and The user is identified based on the image.

10. The electronic device according to claim 9, wherein: The memory is further configured to store user information, wherein facial information of each of a plurality of users and voice information of each of the plurality of users are mapped in the user information, and The processor is further configured to execute the at least one instruction to perform the following operations: obtaining facial information corresponding to the preset user voice based on the user information, and The user corresponding to the facial information is identified from the image.

11. The electronic device according to claim 10, further comprising: speaker, The processor is further configured to execute the at least one instruction to output a preset sound through the speaker based on not obtaining the facial information corresponding to the preset user voice according to the user information or based on not identifying the user corresponding to the facial information from the image.

12. A method for controlling an electronic device, comprising: Receive a preset user voice of the user; Requesting at least one first sound information from at least one home appliance, wherein each of the at least one first sound information corresponds to the preset user voice received at each of the at least one home appliance; receiving the at least one first sound information from the at least one home appliance; and The location of the user is identified based on the preset user voice and the at least one first sound information.

13. The method of claim 12, wherein: Requesting the at least one first sound information comprises: Based on receiving the preset user voice, sending time information about when the preset user voice is received to the at least one home appliance, and The at least one first sound information received from the at least one household appliance corresponds to the time information.

14. The method of claim 13, wherein: Identifying the location of the user includes: identifying location information of the at least one home appliance by determining delay information of each of the at least one first sound information received from each of the at least one home appliance, and The location of the user is identified based on the location information of the at least one home appliance.

15. The method of claim 12, wherein: Identifying the location of the user includes: The location of the user is identified based on the location information of the at least one home appliance, the preset user voice and the at least one first sound information.

Images