Electronic apparatus and control method therefor

Abstract

An electronic apparatus and a control method therefor are provided. The electronic device comprises: a driving unit for driving the electronic apparatus; a microphone for receiving audio; a camera; a projection unit for projecting an image; a memory for storing one or more computer programs; and one or more processors communicatively connected to the driving unit, the microphone, the camera, the projection unit and the memory. When executed individually or collectively by the one or more processors, one or more computer-executable instructions included in the one or more computer programs acquire, through the microphone, information about audio generated by an object located in an indoor space during driving thereof using the driving unit, acquire an image of the object through the camera, acquire information about expected audio related to the object on the basis of the image of the object, store, in a map corresponding to the indoor space, information about the expected audio and the audio generated by the object, identify an object related to a detected sound on the basis of the information about the audio stored in the map when the sound is detected in the indoor space, and controls the projection unit such that a message including information about the sound is projected on the basis of the sound and the information about the related object.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device capable of projecting a message including information about a sound generated by an object during driving and a method for controlling the same.

[0002] Nowadays, various video content is provided through electronic devices (e.g., projectors) that project images onto walls or separate screens. In particular, recent projectors are designed to be portable, allowing them to be displayed in various spaces within the home, in addition to providing diverse video content in a fixed location.

[0003] In particular, users with hearing loss or those in situations where they cannot perceive sound (e.g., wearing earphones) may not be able to recognize sounds occurring in their home. This can lead to users being unable to recognize events occurring in their home and respond appropriately.

[0004] Therefore, there is a need to find a way to provide information related to sounds occurring in the home to users who have hearing loss or who are unable to perceive sounds.

[0005] The above information is provided solely as background information to aid in understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0006] One aspect of the present disclosure addresses at least the problems and / or drawbacks mentioned above, and provides at least the advantages described below. Accordingly, one aspect of the present disclosure provides an electronic device capable of projecting a message containing information about sounds generated by a moving object.

[0007] Additional aspects will be partly explained in the following description, partly will be obvious from the description, or may be learned by practice of the presented embodiments.

[0008] According to one aspect of the present disclosure, an electronic device is provided. An electronic device includes a driving unit for driving the electronic device, a microphone for receiving audio, a camera, a projection unit for projecting an image, a memory for storing one or more computer programs, and one or more processors communicatively coupled to the driving unit, the microphone, the camera, the projection unit, and the memory; wherein one or more computer-executable instructions included in the one or more computer programs, when individually or collectively executed by the one or more processors, acquire information about audio generated by an object located in an indoor space through the microphone while driving through the driving unit, acquire an image of the object through the camera, acquire information about expected audio related to the object based on the image of the object, store the audio generated by the object and information about the expected audio on a map corresponding to the indoor space, and when a sound is detected within the indoor space, identify an object related to the detected sound based on information about audios stored on the map, and control the projection unit to project a message including information about the sound based on information about the related object and the sound.

[0009] The electronic device further includes a communication interface; wherein one or more computer-executable instructions included in the one or more computer programs, when individually or collectively executed by the one or more processors, obtain identification information of the object based on an image of the object, transmit the obtained identification information of the object to an external server through the communication interface, and obtain information about expected audio that the object may generate from the external server.

[0010] One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, obtain information about emergency audio requiring an urgent response in relation to a location where said object is located, and store said emergency audio on a map corresponding to said indoor space.

[0011] One or more computer-executable instructions included in the one or more computer programs, when individually or collectively executed by the one or more processors, identify whether an object related to the detected sound exists based on information about audios stored on the map, and if it is identified that an object related to the detected sound exists, identify whether the sound is generated from a registered person object or a registered thing object, and if it is identified that the detected sound is generated from the registered person object, control the driving unit to move to the location of the registered person object by identifying the location of the registered person object, and if the person object exists at the moved location, control the projection unit to project a message including a text that recognizes the sound around the person object.

[0012] One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, control said projection unit to project a message including information about the estimated location of said person object and text that recognizes said sound, if said person object is an unregistered person or said person object does not exist at the moved location.

[0013] One or more computer-executable instructions included in the one or more computer programs, when individually or collectively executed by the one or more processors, determine the importance of the detected sound based on a pre-stored criterion when the detected sound is identified as a sound generated from the registered object, and control the projection unit to project a message including information about the sound at a current location or a location where the sound generated based on the importance of the detected sound.

[0014] One or more computer-executable instructions included in the one or more computer programs, when individually or collectively executed by the one or more processors, determine the first priority if the detected sound is identified as an emergency sound, determine the second priority if the waveform of the detected sound is repeated a preset number of times or more or if the detected sound matches an audio including an irregular noise among audios matching the registered object or if the detected sound is identified as a call sound, and determine the third priority if the sound is identified as not having the first priority or the second priority.

[0015] The one or more computer-executable instructions included in the one or more computer programs, when individually or collectively executed by the one or more processors, output a notification message to a user and control the driving unit to move to a location where the detected sound occurred if the importance of the detected sound is a first importance, and control the projection unit to project a message including information about the sound at the location where the sound occurred; output a message asking the user whether to move to the location where the detected sound occurred if the user moves to the location where the detected sound occurred, and control the projection unit to project a message including information about the sound at the location where the sound occurred if the user does not move to the location where the detected sound occurred, and control the projection unit to project information about the location where the sound occurred and a message including information about the sound at the current location if the importance of the detected sound is a third importance, and control the projection unit to project a message including information about the sound to the user at the current location.

[0016] One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, identify the significance of the detected sound based on at least one of a loudness of the detected sound and a duration of the detected sound, if said detected sound is identified as a sound originating from the unregistered human object.

[0017] One or more computer-executable instructions included in the one or more computer programs, when individually or collectively executed by the one or more processors, output a notification message to a user and control the driving unit to move to a location where the detected sound occurred, if the importance of the detected sound is a first importance, and control the projection unit to project a message including information about the sound from the location where the sound occurred; if the importance of the detected sound is a second importance, control the projection unit to project a message including information about the sound in a direction corresponding to a location where the sound occurred from a current location; and if the importance of the detected sound is a third importance, do not output a message.

[0018] According to another aspect of the present disclosure, a method for controlling an electronic device is provided. The method includes: obtaining information about audio generated by an object located in an indoor space through a microphone while the electronic device is driving, and obtaining an image of the object through a camera; obtaining information about expected audio related to the object based on the image of the object, and storing the audio generated by the object and information about the expected audio on a map corresponding to the indoor space; identifying an object related to the detected sound based on information about audios stored on the map when a sound is detected within the indoor space; and projecting a message including information about the sound based on the information about the related object and the sound.

[0019] The step of storing includes: a step of obtaining identification information of the object based on an image of the object; a step of transmitting the obtained identification information of the object to an external server; and a step of obtaining information about expected audio that the object may generate from the external server.

[0020] The step of storing includes: a step of obtaining information about emergency audio requiring an urgent response in relation to a location where the object is located; and a step of storing the emergency audio on a map corresponding to the indoor space.

[0021] The step of identifying the object includes: a step of identifying whether an object related to the detected sound exists based on information about audios stored on the map; and a step of identifying whether the sound generated by the registered person object or the sound generated by the registered thing object is present if the object related to the detected sound is identified as existing; and the step of projecting the message includes: if the detected sound is identified as being a sound generated by the registered person object, identifying the location of the registered person object and moving to the location of the person object, and if the person object exists at the moved location, projecting a message including a text recognizing the sound around the person object.

[0022] The step of projecting the message projects a message including information on the estimated location of the person object and text recognizing the sound, if the person object is not registered or the person object does not exist at the moved location.

[0023] The above identifying step includes: a step of determining the importance of the detected sound based on a pre-stored criterion if the detected sound is identified as a sound generated from the registered object; and a step of controlling the projection unit to project a message including information about the sound at a current location or a location where the sound generated based on the importance of the detected sound.

[0024] The identifying step includes: a step of determining the first importance if the detected sound is identified as an emergency sound; a step of determining the second importance if the waveform of the detected sound is repeated a preset number of times or more or the detected sound matches an audio including irregular noise among audios matching the registered object or the detected sound is identified as a call sound; and a step of determining the third importance if the sound is identified as not having the first or second importance.

[0025] The step of projecting the message includes: if the importance of the detected sound is a first importance, outputting a notification message to the user and controlling the driving unit to move to a location where the detected sound was generated, and projecting a message including information about the sound at the location where the sound was generated; if the importance of the detected sound is a second importance, outputting a message asking the user whether to move to the location where the detected sound was generated, and if the user moves to the location where the detected sound was generated, controlling the projection unit to project a message including information about the sound at the location where the sound was generated, and if the user does not move to the location where the detected sound was generated, controlling the projection unit to project information about the location where the sound was generated and a message including information about the sound at the current location; if the importance of the detected sound is a third importance, controlling the projection unit to project a message including information about the sound to the user at the current location.

[0026] The identifying step identifies the importance of the detected sound based on at least one of the magnitude of the detected sound and the duration of the detected sound, if the detected sound is identified as a sound generated from the unregistered object.

[0027] The above projecting step outputs a notification message to the user if the importance of the detected sound is the first importance, moves to a location where the detected sound occurred, and projects a message including information about the sound at the location where the sound occurred; if the importance of the detected sound is the second importance, projects a message including information about the sound in a direction corresponding to a location where the sound occurred from the current location; and if the importance of the detected sound is the third importance, does not output a message.

[0028] According to another embodiment of the present disclosure, one or more non-transitory computer-readable storage media are provided that store computer-executable instructions that, when individually or collectively executed by one or more processors of an electronic device, cause the electronic device to perform tasks including: obtaining information about audio generated by an object located in an indoor space through a microphone while the electronic device is driving and obtaining an image of the object through a camera; obtaining information about expected audio related to the object based on the image of the object, and storing the audio generated by the object and information about the expected audio on a map corresponding to the indoor space; identifying an object related to the detected sound based on information about audios stored on the map when a sound is detected within the indoor space; and projecting a message including information about the sound based on the information about the related object and the sound.

[0029] Other aspects, advantages and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the present disclosure taken in conjunction with the accompanying drawings.

[0030] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0031] FIG. 1 is a diagram illustrating a system for projecting a message containing information about a sound detected in a home, according to one embodiment of the present disclosure;

[0032] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure;

[0033] FIG. 3 is a flowchart illustrating a method of controlling an electronic device that projects a message including information about a sound detected in a home, according to one embodiment of the present disclosure;

[0034] FIG. 4A is a flowchart illustrating a method of storing information about audio generated by an object acquired through a microphone on a map according to one embodiment of the present disclosure;

[0035] FIG. 4b is a diagram illustrating a method for storing information about audio generated by an object acquired through a microphone on a map according to one embodiment of the present disclosure;

[0036] FIG. 5A is a flowchart illustrating a method of storing information about expected audio that may occur in an object acquired based on an image acquired through a camera in a map according to one embodiment of the present disclosure;

[0037] FIG. 5b is a diagram illustrating a method for storing information about expected audio that may occur from an object acquired based on an image acquired through a camera in a map according to one embodiment of the present disclosure;

[0038] FIG. 6A is a flowchart illustrating a method of storing information about emergency audio associated with a location where an object is located on a map, according to one embodiment of the present disclosure;

[0039] FIG. 6b is a diagram illustrating a method for storing information about emergency audio related to a location where an object is located on a map, according to one embodiment of the present disclosure;

[0040] FIG. 7 is a flowchart illustrating a method for learning and storing a neural network model based on information about object-generated audio and expected audio, according to one embodiment of the present disclosure;

[0041] FIG. 8 is a drawing for explaining a method for identifying a location where a sound is generated, according to one embodiment of the present disclosure;

[0042] FIG. 9 is a flowchart illustrating a method of controlling an electronic device that projects a message containing information about a sound in various ways depending on whether a sound detected in a home is registered audio, according to one embodiment of the present disclosure;

[0043] FIG. 10A and FIG. 10B are diagrams illustrating a message including information about a sound associated with a human object, according to various embodiments of the present disclosure;

[0044] FIG. 11 is a flowchart illustrating a method for determining the importance of a sound associated with a registered thing object according to one embodiment of the present disclosure;

[0045] FIG. 12 is a flowchart illustrating a method for providing a message based on the importance of a sound associated with a registered thing object, according to one embodiment of the present disclosure;

[0046] FIG. 13 is a flowchart illustrating a method for determining the importance of a sound associated with an unregistered thing object according to one embodiment of the present disclosure;

[0047] FIG. 14 is a flowchart illustrating a method for providing a message based on the importance of a sound associated with an unregistered thing object, according to one embodiment of the present disclosure;

[0048] FIG. 15 is a diagram illustrating a method for providing a message including information related to sound generated in a separate space within a home, according to one embodiment of the present disclosure;

[0049] FIG. 16 is a diagram illustrating a method for providing a message including information about a sound related to a failure of an object, according to one embodiment of the present disclosure;

[0050] FIG. 17 is a diagram illustrating a method for providing a message including information about sound related to emergency audio, according to one embodiment of the present disclosure;

[0051] FIG. 18A and FIG. 18B are diagrams illustrating a method for providing a message when sounds related to a thing object and a person object are sequentially generated according to various embodiments of the present disclosure;

[0052] FIG. 19 is a diagram illustrating a method for providing a message including information about a sound generated in a display device according to one embodiment of the present disclosure, and

[0053] FIGS. 20, 21a, and 21b are diagrams illustrating a method for providing a message including information related to noise occurring in a home, according to various embodiments of the present disclosure.

[0054] It should be noted that throughout the drawings, similar reference numbers are used to describe identical or similar elements, features and structures.

[0055] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and their equivalents. While it includes numerous specific details to aid understanding, these are to be considered merely exemplary. Accordingly, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of known functions and configurations may be omitted for clarity and brevity.

[0056] The terms and words used in the following description and claims are not to be construed as limited by their bibliographic meanings, but rather have been used by the inventors solely to facilitate a clear and consistent understanding of the present invention. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present invention is provided for illustrative purposes only and is not intended to limit the present invention as defined by the appended claims and their equivalents.

[0057] The singular forms "a," "an," and "the" should be understood to include the plural unless the context clearly dictates otherwise. Thus, for example, reference to "a surface of a component" includes reference to one or more such surfaces.

[0058] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0059] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0060] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0061] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0062] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0063] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0064] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0065] In various embodiments of the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented in specific hardware.

[0066] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0067] Hereinafter, various embodiments of the present invention will be described using the attached drawings.

[0068] It should be recognized that each block of the flowchart and the combination of flowcharts can be performed by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory device, or the one or more computer programs may be divided into different parts stored in multiple different memory devices.

[0069] Any function or operation described herein may be processed by a single processor or a combination of processors. A single processor or a combination of processors may include circuits that perform processing, such as an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (e.g., a CPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a wireless fidelity (Wi-Fi) chip, a Bluetooth™ chip, a global positioning system (GPS) chip, a near-field communication (NFC) chip, a connection chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on a chip (SoC), an IC, and the like.

[0070] FIG. 1 is a diagram illustrating a system for projecting a message containing information about a sound detected in a home, according to one embodiment of the present disclosure.

[0071] As illustrated in FIG. 1, the system may include a server (10), an electronic device (100), and a plurality of external devices (20-1, 20-2, 20-3). At this time, the server (10) may be implemented as at least one server. In addition, the electronic device (100) may be implemented as a movable projector, but is not limited thereto, and may be implemented as a movable device (e.g., a robot vacuum cleaner, a serving robot, etc.). In addition, as illustrated in FIG. 1, the external devices may be implemented as a television (20-1), a washing machine (20-2), a refrigerator (20-3), etc., but this is only one example, and may be implemented as various home appliances or IoT (Internet of Things) devices located in a home. At this time, the server (10), the electronic device (100), and a plurality of external devices (20-1, 20-2, 20-3) may be communicatively connected to each other.

[0072] The electronic device (100) can register at least one user. Specifically, the electronic device (100) can register a primary user (e.g., a user with hearing impairment, etc.) based on user input, and can also register family members (or surrounding members) of the primary user. In addition, the electronic device (100) can obtain and store information about the primary user and his / her family members (e.g., relationship information with the primary user, voice information, etc.).

[0073] In addition, the electronic device (100) can generate a map of an indoor space based on sensing information acquired through at least one sensor (e.g., a ToF sensor, a lidar sensor, or an IMU (inertial measurement unit) sensor, etc.) and an image acquired through a camera. In this case, the indoor space may be a space inside a house, but this is only an example, and may be implemented as various indoor spaces (e.g., a space inside a restaurant, etc.). In addition, the map may include a picture that represents the state of the indoor space on a plane by reducing it by a certain ratio. According to embodiments of the present disclosure, the map may include a picture that represents the planar structure inside a house by reducing it by a certain ratio and representing it with a predetermined symbol. For example, the map may include a picture that represents the planar structure inside a house with lines. However, the present invention is not limited thereto, and the map may also include the locations of major objects inside the house.

[0074] In addition, the electronic device (100) can obtain and store information about audio related to an object located in an indoor space on the generated map. In an embodiment of the present disclosure, the electronic device (100) can obtain information about audio generated by an object acquired through a microphone while driving and store the information on the map. At this time, the electronic device (100) can obtain information about audio generated by an object not only through a microphone equipped in the electronic device (100) but also through microphones equipped in a plurality of external devices (20-1, 20-2, 20-3). In an embodiment of the present disclosure, the electronic device (100) can obtain information about expected audio that may be generated by an object based on an image of the object captured while driving and store the information on the map. Specifically, the electronic device (100) can transmit an image of the object captured while driving (or identification information of the object acquired through the image) to the server (10). The server (10) can transmit information about expected audio that may occur when an object is captured based on an image of the captured object (or identification information of the object obtained through the image) to the electronic device (100). The electronic device (100) can store the information about the expected audio transmitted by the server (10) on a map. In an embodiment of the present disclosure, the electronic device (100) can obtain information about urgent audio that requires an urgent response related to a location where an object is located and store the information on the map. Specifically, the electronic device (100) can receive information about urgent audio that requires an urgent response related to a location where an object is located from the server (10) and store the information on the map.

[0075] After storing information about audio related to an object on a map, the electronic device (100) can detect sounds generated in an indoor space. In this case, the electronic device (100) can detect (or monitor) sounds generated in an indoor space while driving around a user (e.g., within a threshold distance from a primary user) in the indoor space.

[0076] When the occurrence of a sound is detected, the electronic device (100) can identify an object associated with the detected sound using a map that stores information about audio associated with the object. Furthermore, the electronic device (100) can identify information about the location where the detected sound originated based on the information about the identified object. Furthermore, the electronic device (100) can project a message including information about the object associated with the detected sound and information about the sound based on the detected sound. Various embodiments of projecting a message including information about the detected sound will be described in more detail later with reference to the drawings.

[0077] As described above, by providing sound-related messages using a map that stores information about audio associated with objects, users with hearing loss or difficulty perceiving sound can be provided with information about sounds occurring in their homes. Consequently, users with hearing loss or difficulty perceiving sound can respond quickly and accurately to various events occurring in their homes.

[0078] FIG. 2 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0079] As illustrated in FIG. 2, the electronic device (100) may include at least one sensor (110), a camera (120), a microphone (130), a communication interface (140), a driving unit (150), a projection unit (160), a memory (170), and at least one processor (180). Meanwhile, the configuration of the electronic device (100) is not limited to the configuration illustrated in FIG. 2, and it goes without saying that configurations obvious to those skilled in the art may be added or deleted. For example, the electronic device (100) may have additional configurations such as a speaker and a display, and some of the configurations illustrated in FIG. 2 may be deleted.

[0080] At least one sensor (110) can obtain various information about the status of the electronic device (100) or the surrounding environment of the electronic device (100). In particular, at least one sensor (110) may include a ToF (Time of Flight) sensor and an IMU (Inertial Measurement Unit) sensor. The ToF sensor can obtain sensing information for obtaining information about the distance to the object by projecting light (e.g., laser, near-infrared light, visible light, ultraviolet light, etc.) to an object and detecting light reflected by the object. The IMU sensor is a sensor for detecting the movement of the electronic device (100) and may include at least one of a geomagnetic sensor, an acceleration sensor, and a gyro sensor. However, as described above, using a lidar sensor to obtain information about the distance to the object is only one embodiment, and information about the distance to the object may be obtained using various sensors such as a depth sensor. In particular, at least one processor (180) can generate a map of an indoor space based on sensing information acquired through a ToF sensor and an IMU sensor.

[0081] The camera (120) can capture images of objects located in an indoor space by photographing the surroundings of the electronic device (100). The electronic device (100) can input the images of the objects into a trained neural network model (e.g., an object recognition model) to obtain identification information of the objects (e.g., the type of the objects, the size of the objects, the shape of the objects, etc.). At this time, the trained neural network model can be stored in the memory (170), but this is only one embodiment, and can be stored in an external server (10).

[0082] The microphone (130) is a configuration for receiving audio generated from an object (e.g., human voice, audio generated from an object, etc.) and converting it into audio data. The microphone (130) can receive audio when activated. In particular, a plurality of microphones (130) may be provided on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (130) may include various configurations such as a microphone for collecting audio in analog form, an amplifier circuit for amplifying the collected audio, an A / D (analog-to-digital) conversion circuit for sampling the amplified audio and converting it into a digital signal, and a filter circuit for removing noise components from the converted digital signal.

[0083] In particular, the microphone (130) may include a plurality of microphones, and may identify the direction from which the sound is received based on the size and reception time of the sound received through the plurality of microphones.

[0084] The communication interface (140) includes at least one circuit and can communicate with various types of external devices. The communication interface (140) can include at least one of a BLE (Bluetooth Low Energy) module, a Wi-Fi communication module, a cellular communication module, a 3G (third generation) mobile communication module, a 4G (fourth generation) mobile communication module, a 4th generation LTE (Long Term Evolution) communication module, and a 5G (fifth generation) mobile communication module.

[0085] In particular, the communication interface (140) can transmit an image (or identification information of an object) captured by the camera (120) to the server (10), and can receive information about expected audio or emergency audio that may occur when an object is generated from the server (10). In addition, the communication interface (140) can receive information about audio (or sound) that an object is generated from an external device (20). At this time, the electronic device (100) can communicate with the server (10) through a first type of communication interface (e.g., a long-distance communication interface), and can communicate with the external device (20) through a second type of communication interface (e.g., a short-distance communication interface).

[0086] The driving unit (150) is a component for driving the electronic device (100). In particular, the driving unit (150) may include wheels and a driving unit to move the electronic device (100). The driving unit may generate physical force and transmit it to the wheels included in the electronic device (100). In particular, the driving unit (150) may move to a location where a sound is generated under the control of at least one processor (180).

[0087] The projection unit (160) is a component that projects an image to the outside. According to various embodiments of the present disclosure, the projection unit (212) may be implemented in various projection methods (e.g., CRT (cathode-ray tube) method, LCD (Liquid Crystal Display) method, DLP (Digital Light Processing) method, laser method, etc.). In addition, the projection unit (160) may perform various functions for adjusting the output image by the control of at least one processor (180). For example, the projection unit (160) may perform functions such as zoom, keystone, quick corner (4 corner) keystone, lens shift, etc. In particular, the projection unit (160) may provide a message including information related to sound. At this time, the message is for guiding the user with information related to sound, and may be called a UI, a guide window, a guide screen, etc.

[0088] Meanwhile, in the embodiment of the present disclosure described above, the electronic device (100) projects a message through the projection unit (160), but this is only one embodiment, and the electronic device (100) can of course output a message through an output unit such as a display.

[0089] The memory (170) may store an operating system (OS) for controlling the overall operation of the components of the electronic device (100) and instructions or data related to the components of the electronic device (100). In particular, the memory (170) may include a plurality of modules for generating a map including information on audio related to an object and providing a message including information related to sound using the generated map. In particular, when a function for generating a map including information on audio related to an object and providing a message including information related to sound using the generated map is executed, the electronic device (100) may load data for various modules stored in a non-volatile memory to perform various operations into a volatile memory. Here, loading means an operation of loading and storing data stored in a non-volatile memory into a volatile memory so that at least one processor (180) can access it.

[0090] In particular, the memory (170) may store information about a map generated by at least one processor (180). At this time, the map may include information about an object (e.g., object identification information, object location information, etc.) as well as information about audio related to the object (e.g., audio generated by the object, expected audio that may be generated by the object, emergency audio that may be generated at a location related to the object, etc.).

[0091] Additionally, the memory (170) can store data for various neural network models. For example, the memory (170) can store a neural network model trained to input an image and obtain identification information about an object included in the image, or a neural network model trained to input a detected sound and obtain information about an object related to the sound, etc. However, this is merely an example, and it is to be understood that the neural network model can be stored in an external server (10).

[0092] Additionally, the memory (170) may include a STT (Speech-To-Text) module for converting the user's voice into text. However, this is only one embodiment, and the STT module may of course be stored on an external server.

[0093] Meanwhile, the memory (170) may be implemented as a non-volatile memory (e.g., hard disk, SSD (Solid state drive), flash memory), volatile memory (which may also include memory within at least one processor (180)), etc.

[0094] At least one processor (180) can control the electronic device (100) according to at least one instruction stored in the memory (170).

[0095] In particular, at least one processor (180) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the electronic device (100) and perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0096] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment of the present disclosure, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an AI-specific processor). For example, the electronic device (100) may perform an operation of generating a map or an operation of providing a message using a general-purpose processor, and may perform an operation of acquiring recognition information of an object or an operation of identifying an object related to a sound using an AI-specific processor.

[0097] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0098] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among a plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment of the present disclosure, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0099] In one or more embodiments of the present disclosure, at least one processor (180) obtains information about audio generated by an object located in an indoor space through a microphone (130) while driving through a driving unit (150) and obtains an image of the object through a camera (120). The at least one processor (180) obtains information about expected audio related to the object based on the image of the object, and stores information about the audio generated by the object and the expected audio on a map corresponding to the indoor space. When a sound is detected within the indoor space, the at least one processor (180) identifies an object related to the detected sound based on information about audios stored on the map. The at least one processor (180) controls the projection unit (160) to project a message including information about the sound based on the information about the related object and the sound.

[0100] Below, with reference to the drawings, we will describe a method of creating a map that stores information about audio associated with an object, and using the created map to project a message containing information about sounds detected in a home.

[0101] FIG. 3 is a flowchart illustrating a method for controlling an electronic device that projects a message including information about a sound detected in a home, according to one embodiment of the present disclosure.

[0102] Referring to FIG. 3, the electronic device (100) registers a user (step S310). At this time, the user can register a primary user and the primary user's family members (also referred to as "sub-users"). At this time, the electronic device (100) can register the primary user and family members through a separate registration procedure. At this time, the electronic device (100) can store information about the primary user and family members (e.g., the user's identification information, the user's voice information, etc.). Therefore, the registered primary user and family members at this time can be referred to as registered person objects.

[0103] The electronic device (100) generates a map containing information about audio (step S320). Specifically, the electronic device (100) can store information about objects (e.g., object identification information, etc.) on the map, as well as information about audio associated with the objects. A method for generating a map containing information about audio will be described in detail later with reference to FIGS. 4A, 4B, 5A, 5B, 6A, 6B, and 7.

[0104] Through steps S310 and S320, the electronic device (100) can generate a map for providing a message including information related to sound.

[0105] After the map is generated, the electronic device (100) determines whether a sound has been detected (step S330). Specifically, the electronic device (100) can detect the occurrence of sounds within the home while driving around the primary user. The sound may be a sound louder than a preset volume, but is not limited thereto.

[0106] When a sound is identified as being detected (step S330-Y), the electronic device (100) identifies an object associated with the detected sound (step S340). That is, the electronic device (100) can identify an object associated with the detected sound based on information about the detected sound and audio associated with the object stored on the map. At this time, the sound may include audio generated from multiple objects, but the electronic device (100) can identify an object that generates audio with the largest volume (or amplitude) among the audio generated from multiple objects. In addition, the electronic device (100) can identify a location where the detected sound is generated based on the location of the identified object.

[0107] The electronic device (100) projects a message containing information about a sound based on information about a related object and the detected sound (step S350). Specifically, the electronic device (100) can project a message containing information about a sound in various locations or in various ways based on information about a related object and the sound. This will be described in detail later with reference to FIGS. 9, 10a, 10b, and 14.

[0108] FIG. 4A is a flowchart illustrating a method of storing information about audio generated by an object acquired through a microphone in a map, according to one embodiment of the present disclosure.

[0109] FIG. 4b is a diagram illustrating a method for storing information about audio generated by an object acquired through a microphone on a map according to one embodiment of the present disclosure.

[0110] Referring to FIGS. 4A and 4B , first, the electronic device (100) can receive audio generated by an object while the electronic device (100) is driving (step S410). Specifically, when a user command for map generation is input, the electronic device (100) can receive audio generated by an object while the electronic device (100) is driving for map generation. At this time, the electronic device (100) can receive audio generated while driving for map generation, but this is only one embodiment, and the electronic device (100) can receive audio generated while driving in general.

[0111] The electronic device (100) can identify the location of the electronic device (100) (step S420). That is, when audio generated from an object is received, the electronic device (100) can identify the location of the electronic device (100) receiving the audio generated by the object based on sensing information acquired through at least one sensor (110) (e.g., an IMU sensor). At this time, the location of the electronic device (100) receiving the audio can be referred to as the collection location of the electronic device (100). The location of the electronic device (100) can be expressed as (x coordinate, y coordinate, rotation direction) based on a map of an indoor space. For example, the electronic device (100) can express the location of the electronic device (100) receiving the audio generated by the object as (112, 241, 52), as illustrated in FIG. 4B.

[0112] The electronic device (100) can identify the location of an object (step S430). In particular, the electronic device (100) can identify the location of an object that generates audio based on sensing information acquired through at least one sensor (110) (e.g., an IMU sensor and a ToF sensor, etc.). At this time, the electronic device (100) can acquire information about the relative location between the electronic device (100) and the object that generates audio (i.e., the distance between the electronic device (100) and the object, etc.) based on the sensing value acquired through the ToF sensor. In addition, the electronic device (100) can identify the location of the object based on the information about the location of the electronic device (100) acquired in step S420 and the relative location between the electronic device (100) and the object that generates audio. At this time, the location of the object can be expressed as (x coordinate, y coordinate) on a map of an indoor space. For example, the electronic device (100) can be expressed as the position of the object as (120, 270), as shown in FIG. 4b.

[0113] The electronic device (100) can analyze the collected audio to obtain information about the audio (step S440). Specifically, the electronic device (100) can collect audio generated from an object and analyze the collected audio. In addition, the electronic device (100) can obtain and store information about the size (or amplitude), pitch (or frequency), wavelength, and period of the audio. For example, as illustrated in FIG. 4b, the electronic device (100) can store information about the audio, such as a sound size of 38 dB, a pitch of 40 Hz, a wavelength of 12, and a period of 1 / 10 s.

[0114] The electronic device (100) can store information about the acquired audio on the map (step S450). Specifically, the electronic device (100) can acquire identification information of an object (e.g., an air purifier) ​​based on an image captured by the camera (120), and store information about audio generated by the object on the map based on the identification information of the object. At this time, the electronic device (100) can match information about the location of the electronic device (100), the location of the object, and the audio and store the information in the form of log data (or metadata). For example, as illustrated in FIG. 4B, the electronic device (100) can match information about the location of the electronic device (100), the location of the object, and the audio and store the information as log data (460), such as (112.241.52 / 10.270 / 38.40.12.1 / 10). At this time, the operation of storing information about the acquired audio on the map by the electronic device (100) may have the same meaning as the operation of registering the acquired audio in the electronic device (100).

[0115] FIG. 5A is a flowchart illustrating a method for storing information about expected audio that may occur from an object acquired based on an image acquired through a camera in a map, according to one embodiment of the present disclosure.

[0116] FIG. 5b is a diagram illustrating a method for storing information about expected audio that may occur from an object acquired based on an image acquired through a camera in a map, according to one embodiment of the present disclosure.

[0117] Referring to FIGS. 5A and 5B , first, the electronic device (100) can capture an image by photographing an object while the electronic device (100) is driving (step S510). Specifically, when a user command for map generation is input, the electronic device (100) can capture an image by photographing an object through the camera (120) while the electronic device (100) is driving for map generation. In this case, the electronic device (100) can capture an image by photographing an object while driving for map generation, but this is only one embodiment, and the image can be captured during general driving.

[0118] The electronic device (100) can identify the location of the electronic device (100) (step S520). That is, when photographing an object, the electronic device (100) can identify the location of the electronic device (100) photographing the object based on sensing information acquired through at least one sensor (110) (e.g., an IMU sensor). At this time, the location of the electronic device (100) photographing the object can be referred to as the photographing location of the electronic device (100). The location of the electronic device (100) can be expressed as (x coordinate, y coordinate, rotation direction) based on a map of an indoor space. For example, the electronic device (100) can express the location of the electronic device (100) receiving audio generated by the object as (140, 281, 92), as illustrated in FIG. 5B.

[0119] The electronic device (100) can identify the location of an object (step S530). In particular, the electronic device (100) can identify the location of the photographed object based on sensing information acquired through at least one sensor (110) (e.g., an IMU sensor and a ToF sensor, etc.). At this time, the electronic device (100) can acquire information on the relative location between the electronic device (100) and the photographed object from the sensing value acquired through the ToF sensor. In addition, the electronic device (100) can identify the location of the object based on the information on the location of the electronic device (100) and the relative location between the electronic device (100) and the photographed object acquired in step S520. At this time, the location of the object can be expressed as (x coordinate, y coordinate) on a map of an indoor space. For example, the electronic device (100) can express the location of the object as (147, 282), as illustrated in FIG. 5B.

[0120] The electronic device (100) can obtain identification information of an object based on an image (step S540). Specifically, the electronic device (100) can input an image of an object into a trained neural network model (e.g., an object recognition model) to obtain identification information of an object included in the image. Alternatively, the electronic device (100) can transmit an image of an object to an external server and receive identification information of the object obtained by the neural network model stored in the external server from the external server. At this time, the identification information of the object can include various information such as the type of the object (e.g., air purifier, etc.), as well as the product name of the object, the manufacturing date of the object, etc.

[0121] The electronic device (100) transmits identification information of the object to an external server (step S550). At this time, the external server (10) may be a cloud server that stores information about expected audio that may occur when the object is generated.

[0122] The electronic device (100) can obtain information about expected audio that may be generated by an object from an external server (step S560). For example, if the captured object is a "front door object," the electronic device (100) can obtain expected audio, such as the sound of a front door opening, closing, and locking, as shown in FIG. 5B. At this time, the information about the expected audio may include information about the size (or amplitude), pitch (or frequency), wavelength, and period of the expected audio, as previously described in FIG. 4A.

[0123] The electronic device (100) may store information about expected audio on the map (step S570). Specifically, the electronic device (100) may store information about expected audio that may occur when an object is detected on the map based on the object's identification information. At this time, the electronic device (100) may match information about the location of the electronic device (100), the location of the object, and the expected audio, and store the information in the form of log data, as described in FIG. 4A.

[0124] FIG. 6A is a flowchart illustrating a method for storing information about emergency audio related to a location where an object is located on a map, according to one embodiment of the present disclosure. Meanwhile, steps S610 to S630 illustrated in FIG. 6A are identical to steps S510 to S530 described in FIG. 5A , and thus, a redundant description thereof will be omitted.

[0125] FIG. 6B is a diagram illustrating a method for storing information about emergency audio related to a location where an object is located on a map, according to one embodiment of the present disclosure.

[0126] The electronic device (100) can obtain information about emergency audio that requires an urgent response in relation to the location of an object (step S640). Specifically, the electronic device (100) can input an image of the object into a neural network model to obtain identification information of the object, and identify the location of the object based on the identification information of the object. In addition, the electronic device (100) can obtain information about emergency audio related to the location of the object among information about emergency audio previously stored in the electronic device (100). Alternatively, the electronic device (100) can obtain identification information about the object by inputting an image of the object into a neural network model. In addition, the electronic device (100) can transmit the identification information of the object to an external server, and can receive information about emergency audio related to the location of the object from the external server. For example, if the object included in the captured image is a "front door object", the electronic device (100) can obtain emergency audio such as a fire alarm sound, an intruder alarm sound, a siren alarm sound, etc., related to the location where the front door is located (e.g., the front door), as illustrated in FIG. 6B. At this time, information about the emergency audio may include information about the size (or amplitude), pitch (or frequency), wavelength, and period of the emergency audio, as described above in FIG. 4A. As another example, if the object included in the captured image is a "window object", the electronic device (100) can obtain emergency audio such as a gas leak alarm sound, a carbon monoxide alarm sound, etc., related to the location where the window is located (e.g., the veranda). As another example, if the object included in the captured image is a “gas range object,” the electronic device (100) can obtain emergency audio, such as a gas range alarm sound, related to the location where the gas range is located (e.g., a kitchen).

[0127] The electronic device (100) may store information about emergency audio on the map (step S650). Specifically, the electronic device (100) may store information about emergency audio that may occur on the map based on the object's identification information. At this time, the electronic device (100) may match the location of the electronic device (100), the location of the object, and information about emergency audio and store them in the form of log data, as described in FIG. 4A.

[0128] As described in FIGS. 5A, 5B, 6A, and 6B, the electronic device (100) can acquire and store more diverse audio related to an object by storing not only audio directly generated by the object but also anticipated audio that the object may generate or emergency audio related to the object, thereby enabling the primary user to respond more quickly and accurately to various events that may occur urgently.

[0129] Meanwhile, information on various audio acquired through FIGS. 4a, 5b, 6a to 6b can be stored in the form of log data on the map, but this is only one embodiment, and various audio acquired through FIGS. 4a to 6b can be used as learning data to train a neural network model.

[0130] FIG. 7 is a flowchart illustrating a method for learning and storing a neural network model based on information about object-generated audio and expected audio, according to one embodiment of the present disclosure.

[0131] Referring to FIG. 7, the electronic device (100) can obtain information on audio, expected audio, and emergency audio generated by an object (step S710). At this time, information on audio, expected audio, and emergency audio generated by an object has been described in detail in FIGS. 4A, 5B, 6A, and 6B, so a redundant description will be omitted.

[0132] The electronic device (100) can extract features of audio, expected audio, and emergency audio generated by an object (step S720). At this time, the features may be named feature data or feature values, and may be values ​​(or data) that visualize each of the audio, expected audio, and emergency audio generated by an object based on information about the audio, expected audio, and emergency audio generated by the object.

[0133] The electronic device (100) trains a neural network model using the extracted features (step S730). At this time, the neural network model may be a neural network model trained to take features corresponding to audio in which an object occurs, expected audio, and emergency audio as input values ​​and output information about an object corresponding to each of the audio in which an object occurs, expected audio, and emergency audio.

[0134] The electronic device (100) can store the learned neural network model (step S740). Later, the electronic device (100) can use the stored neural network model to obtain information about objects related to the sound detected within the indoor space (object identification information, object location information, etc.).

[0135] FIG. 8 is a diagram illustrating a method for identifying a location where a sound is generated, according to one embodiment of the present disclosure.

[0136] Referring to FIG. 8, the electronic device (100) may detect the occurrence of a sound (step S810). At this time, the sound may be acquired while the electronic device (100) is driving after the electronic device (100) creates a map storing audio associated with objects. At this time, the sound may include audio generated from multiple objects. In particular, according to one embodiment of the present disclosure, the electronic device (100) may detect the occurrence of a sound greater than a preset volume while driving around the primary user.

[0137] The electronic device (100) can analyze a sound and separate sound elements contained in the sound (step S820). Specifically, the detected sound may include sound elements that are audio generated from multiple objects. For example, the detected sound may include a first sound element generated from an air conditioner, a second sound element generated from an air purifier, and a third sound element generated from an object that is not a pre-registered object. The electronic device (100) can analyze the detected sound and separate the three sound elements contained in the sound. At this time, the electronic device (100) can identify a sound element having the largest volume among the multiple sound elements. In addition, the electronic device (100) can compare the identified sound element with information about audio stored on a map to identify an object related to the detected sound. For example, if the detected sound includes a first sound element generated from an air conditioner, a second sound element generated from an air purifier, and a third sound element generated from an object that is not a pre-registered object, the electronic device (100) can identify the second sound element having the largest volume among the first to third sound elements. Additionally, the electronic device (100) can identify an object related to the detected sound as an air purifier by comparing the second sound element with information about the audio stored on the map.

[0138] The electronic device (100) can extract a map coordinate location (step S830). Specifically, when an object associated with a detected sound is identified, the electronic device (100) can extract location information of the object associated with the sound based on log data of the object associated with the detected sound stored on the map.

[0139] In the manner described above, the electronic device (100) can obtain information about objects and the positions of the objects related to sounds detected while the electronic device (100) is driving.

[0140] Meanwhile, although the above-described embodiment has described a method for identifying the location of an object registered in the electronic device (100), this is merely an example, and the location of an object not registered in the electronic device (100) can also be identified (or estimated). Specifically, the electronic device (100) can identify the direction from which the sound is received based on the size and reception time of the sound received through a plurality of microphones. In addition, the electronic device (100) can identify information about the location where the object related to the sound is estimated to be located based on the direction from which the sound is received.

[0141] FIG. 9 is a flowchart illustrating a method of controlling an electronic device that projects a message including information about sound in various ways depending on whether a sound detected in a home is registered audio, according to one embodiment of the present disclosure.

[0142] Referring to FIG. 9, the electronic device (100) can detect sound generation (step S905). That is, the electronic device (100) can detect sound generation exceeding a preset size while driving around the main user.

[0143] The electronic device (100) can identify whether the detected sound is registered audio (step S910). Specifically, the electronic device (100) can identify whether the detected sound is registered audio based on the detected sound and log data stored on the map or information about the user's voice that has been previously stored. That is, the electronic device (100) can identify whether the detected sound is registered audio by searching for log data that includes information about audio identical to the detected sound among the log data stored on the map. Alternatively, the electronic device (100) can identify whether the detected sound is registered audio by identifying whether the detected sound is identical to the user's voice that has been previously stored.

[0144] If it is determined that the detected sound is registered audio (step S910-Y), the electronic device (100) can identify whether the object related to the detected sound is a human object (step S915). Specifically, the electronic device (100) can identify whether the object related to the detected sound is a human object based on the voice information of the user registered in step S310 of FIG. 3. For example, if it is determined that the voice identical to the detected sound among the voice information of the previously stored user is the brother of the main user, the electronic device (100) can identify the object related to the detected sound as a "brother object."

[0145] If the object associated with the detected sound is identified as a human object (step S915-Y), the electronic device (100) can identify the location of the human object and move there (step S920). Specifically, the electronic device (100) can identify the location of the human object based on the direction of the detected sound and move there to the identified location.

[0146] The electronic device (100) can identify whether a human object exists at the moved location (step S930). Specifically, the electronic device (100) can identify whether a human object exists by continuously detecting surrounding sounds at the moved location or by photographing the surroundings of the moved location. For example, the electronic device (100) can identify whether a "type object" exists at the moved location by continuously detecting the voice of the "type object" detected at the moved location or by identifying whether the "type object" is included in an image photographed at the moved location.

[0147] If it is identified that a human object exists at the moved location (step S925-Y), the electronic device (100) can project a message including information about sound at the moved location (step S940). At this time, the electronic device (100) can recognize the detected sound through the STT module and obtain text corresponding to the detected sound. Then, the electronic device (100) can generate a message including information about the obtained text and project the generated message at the moved location. At this time, the generated message can be projected in the direction in which the main user is located. Specifically, the electronic device (100) can rotate from the moved location toward the main user to project the generated message toward the main user. At this time, the text included in the generated message can be arranged so that the main user can read it.

[0148] FIGS. 10A and 10B are diagrams illustrating a message including information about a sound related to a human object according to one embodiment of the present disclosure.

[0149] Referring to FIG. 10a, the electronic device (100) can project a message (1010) containing text corresponding to the sound, “Cheolsu, did you go to school today?”, toward the main user around a person object (1000-1) associated with the sound.

[0150] If it is determined that a human object does not exist at the moved location (step S930-N), the electronic device (100) can project a message including information on the estimated location of the human object and text that recognizes the sound (step S935). Specifically, the electronic device (100) can identify the direction from which the sound is received based on the magnitude and reception time of the sound received through the plurality of microphones. Then, the electronic device (100) can obtain information on the estimated location where the human object related to the sound is to be located based on the direction from which the sound is to be received. The electronic device (100) can generate a message including text corresponding to the sound acquired through the STT module together with information on the estimated location where the human object related to the sound is to be located, and project the generated message. For example, as illustrated in FIG. 10b, the electronic device (100) can project a message (1020) including information about a human object, information about an estimated location of the human object, and recognized text, in the direction of the primary user, such as “A voice saying ‘Cheolsu, did you go to school today?’ was detected from the direction of the living room fan” from the moved location.

[0151] If the object associated with the detected sound is identified as not being a human object (step S915-N) (i.e., if the object associated with the detected sound is identified as being an object), the electronic device (100) may determine the importance of the sound (step S940). At this time, the importance of the sound may refer to the degree to which a user needs to quickly confirm the sound generated from the object. The method by which the electronic device (100) determines the importance of the sound will be described with reference to FIG. 11.

[0152] FIG. 11 is a flowchart illustrating a method for determining the importance of a sound associated with a registered thing object according to one embodiment of the present disclosure.

[0153] Referring to FIG. 11, the electronic device (100) can identify whether the detected sound is an emergency audio (step S1110). That is, the electronic device (100) can identify whether the detected sound among the audio stored on the map is an emergency audio requiring a quick response at a location related to the detected sound.

[0154] If the detected sound is identified as an emergency audio (step S1110-Y), the electronic device (100) may determine the importance of the detected sound as the first importance (step S1120). At this time, a sound of the first importance may be a sound that requires a quick response from the user. For example, if the detected sound is a gas leak alarm sound coming from the kitchen or a fire alarm sound coming from the front door, the electronic device (100) may determine the importance of the detected sound as the first importance.

[0155] If the detected sound is determined to be non-emergency audio (step S1110-N), the electronic device (100) can determine whether the waveform of the detected sound is repeated n or more times (step S1130). For example, audio in which the waveform of the detected sound is repeated n or more times may be audio such as a refrigerator door open alarm sound, a door lock not closing sound, or a microwave oven food completion notification sound.

[0156] If the waveform of the detected sound is repeated n or more times (step S1130-Y), the electronic device (100) may determine the importance of the detected sound as a second importance (step S1160). At this time, a sound of second importance may not be a sound that requires a quick response from the user, but may be a sound that requires a response.

[0157] If the waveform of the detected sound is not repeated n times or more (step S1130-N), the electronic device (100) can identify whether the detected sound matches an audio including an irregular noise among the audios matching the registered object (step S1140). For example, an audio including an irregular noise among the audios matching the registered object may be an audio such as a sound of metal (e.g., keys, etc.) hitting each other in a washing machine, or a sound of a motor malfunctioning in a refrigerator or a robot vacuum cleaner.

[0158] If the detected sound matches an audio that includes irregular noise among the audios that match the registered object (step S1140-Y), the electronic device (100) can determine the importance of the detected sound as the second importance (step S1160).

[0159] If the detected sound does not match an audio that includes irregular noise among the audios matching the registered object (step S1140-N), the electronic device (100) can identify whether the detected sound is a call sound (step S1150). For example, the call sound may be the same audio as the front door, the intercom call sound of the entrance door, or a call sound set by the user.

[0160] If the detected sound is a call sound (step S1150-Y), the electronic device (100) can determine the importance of the detected sound as the second importance (step S1160).

[0161] If the detected sound is not a call sound (step S1150-N), the electronic device (100) may determine the importance of the detected sound as the third importance (step S1170). That is, if the detected sound is not an emergency audio, an audio whose waveform is repeated n or more times, an audio including irregular noise among the audios matching the registered object, or a call sound, the electronic device (100) may determine the importance of the detected sound as the third importance. At this time, the sound of the third importance may be a sound that does not require a response from the user. For example, the sound of the third importance may be a sound of an object moving that may occur in daily life.

[0162] Meanwhile, the criteria for dividing importance described above are merely exemplary; importance levels can be further simplified or subdivided. For example, first and second importance levels can be simplified into a single importance level, and various audios included in second importance level can be further subdivided into multiple importance levels.

[0163] Referring back to FIG. 9, the electronic device (100) may project a message containing information about the sound based on the perceived importance of the sound (step S945). At this time, the electronic device (100) may project the message containing information about the sound at the current location or the location where the sound originated, based on the perceived importance of the sound. This will be described in more detail with reference to FIG. 12.

[0164] FIG. 12 is a flowchart illustrating a method for providing a message based on the importance of a sound associated with a registered thing object, according to one embodiment of the present disclosure.

[0165] Referring to FIG. 12, if the detected sound is determined to be of the first priority (step S1205-Y), the electronic device (100) may project a notification message to the user (step S1210). At this time, the electronic device (100) may output a notification message informing the user of an emergency situation to enable a quick response. For example, the electronic device (100) may output a notification message informing that emergency audio has been detected on a screen of a preset color (e.g., red) while rotating 360 degrees from the current location. In addition, the electronic device (100) may include information about the emergency audio (e.g., the location where the emergency audio was generated, the volume of the emergency audio, etc.) and information guiding movement to the location where the emergency audio was generated in the notification message, and may output an LED (light emitting diode) warning light and an alarm sound together with the notification message.

[0166] The electronic device (100) may move to the location where the detected sound occurred (step S1215). That is, the electronic device (100) may move to the location where the detected sound occurred to guide the user's quick response. For example, if the detected sound is a gas leak alarm, the electronic device (100) may move to the kitchen.

[0167] The electronic device (100) can project a message containing information about the sound at the location where the sound was generated (step S1220). Specifically, the electronic device (100) can output a message containing information about the emergency audio (e.g., the type of emergency audio, the volume of the emergency audio, etc.) and information about how to respond at the location where the sound was generated. For example, after moving to the kitchen, the electronic device (100) can output a message saying, "A 100 dB gas leak alarm has occurred. After checking the gas, please close the gas valve and open the window."

[0168] If the detected sound is determined to be not the first priority (step S1205-N) but the second priority (step S1225-Y), the electronic device (100) may project a message asking the user whether to move to the location where the detected sound originated (step S1230). Specifically, the electronic device (100) may project a message asking the user whether to move to the location where the sound originated, along with information about the detected sound. For example, the electronic device (100) may project a message saying, "A strange sound has occurred from the refrigerator in the kitchen. Should I move to the location where the sound originated? Yes / No."

[0169] The electronic device (100) can identify whether the user is moving (step S1235). At this time, the electronic device (100) can identify whether the user is moving based on the voice spoken by the user. However, this is only an example, and the electronic device (100) can identify whether the user is moving based on the user's movements.

[0170] If the user is identified as not moving (step S1235-N), the electronic device (100) may project a message containing information about the location where the sound originated from the current location and information about the sound (step S1240). For example, the electronic device (100) may project a message from the current location toward the user, saying, "A motor malfunction has occurred in the refrigerator located in the kitchen. Please check this."

[0171] If the user is identified as moving (step S1235-Y), the electronic device (100) may project a message containing information about the sound at the location where the sound originated (step S1245). Specifically, the electronic device (100) may move with the user to the location where the sound originated and project a message containing information about the sound at the location where the sound originated. For example, the electronic device (100) may move with the user to the kitchen and project a message in the kitchen that reads, "A motor malfunction has occurred in the refrigerator. Please check this."

[0172] If the detected sound is determined to be not of the second priority (step S1225-N), the electronic device (100) may determine that the detected sound is of the third priority (step S1250). The electronic device (100) may project a message containing information about the location where the sound originated and information about the sound to the user at the current location (step S1255). For example, the electronic device (100) may project a message containing information about the sound, such as "The air conditioner in the living room is operating in wind-free mode," at the current location.

[0173] Meanwhile, the electronic device (100) may display the text included in the projected message in different sizes, thicknesses, colors, etc., depending on the importance, or may provide separate indicators, etc. For example, the electronic device (100) may display the text in a larger and bolder size and provide text in a red color series as the importance increases, and may display the text in a smaller and thinner size and provide text in a black color series as the importance decreases. Alternatively, the electronic device (100) may display indicators such as "emergency situation" and "confirmation required" together depending on the importance.

[0174] Referring back to FIG. 9, if it is determined that the detected sound is not a registered audio (step S910-N), the electronic device (100) can identify whether the detected sound is a human object (step S950). That is, the electronic device (100) can identify whether a sound generated from an unregistered object is generated from a human object.

[0175] If the detected sound is identified as having originated from a human object (step S950-Y), the electronic device (100) may project a message including information on the estimated location of the human object and a text that recognizes the sound (step S955). Specifically, the electronic device (100) may generate a message including text corresponding to the sound acquired through the STT module together with information on where the human object associated with the sound will be located, and project the generated message. For example, the electronic device (100) may project a message including information on the estimated location of the human object and a text that recognizes the detected sound, such as "A voice saying "Is your mother here?" was detected from the front door direction" at the current location, toward the primary user.

[0176] If it is determined that the detected sound is not generated from a human object (step S950-N) (i.e., if it is determined that the detected sound is generated from a physical object), the electronic device (100) may determine the importance of the sound (step S960). At this time, the electronic device (100) may determine the importance of the detected sound based on at least one of the loudness of the detected sound and the duration of the detected sound. This will be described with reference to FIG. 13.

[0177] FIG. 13 is a flowchart illustrating a method for determining the importance of a sound associated with an unregistered thing object according to one embodiment of the present disclosure.

[0178] Referring to FIG. 13, the electronic device (100) can identify whether the detected sound level is greater than or equal to a first threshold (step S1310). At this time, the first threshold is a value related to the level of sound generated in connection with an emergency situation, and may be, for example, 120 dB.

[0179] If the size of the detected sound is greater than or equal to the first threshold value (step S1310-Y), the electronic device (100) can identify whether the duration of the detected sound is greater than or equal to the first threshold time (step S1320). For example, the first threshold time may be 10 seconds.

[0180] If the size of the detected sound is greater than or equal to the first threshold value (step S1310-Y) and the duration of the detected sound is greater than or equal to the first threshold time (step S1320-Y), the electronic device (100) may determine the detected sound as having a first priority (step S1330). For example, among sounds generated from unregistered objects, sounds of first priority may be sounds requiring a response, such as a siren, a baby crying, or a dog barking.

[0181] If the duration of the detected sound is less than the first threshold time (step S1320-N), the electronic device (100) can determine the detected sound as having a second priority (step S1360).

[0182] If the size of the detected sound is less than a first threshold value (step S1310-N), the electronic device (100) can identify whether the size of the detected sound is greater than or equal to a second threshold value (step S1340). At this time, the second threshold value is a value related to the size of the sound related to the event requiring confirmation, and may be, for example, 70 db.

[0183] If the size of the detected sound is greater than or equal to the second threshold value (step S1340-Y), the electronic device (100) can identify whether the duration of the detected sound is greater than or equal to the second threshold time (step S1350). For example, the second threshold time may be 8 seconds. Meanwhile, the first threshold time and the second threshold time may be set differently, but this is merely an example, and of course, they may be set to the same time.)

[0184] If the duration of the detected sound is longer than the second threshold time (step S1350-Y), the electronic device (100) may determine the detected sound as having a second priority (step S1360). For example, sounds of second priority among sounds generated from unregistered objects may be a single scream, a sound of an object colliding, a sound of an object falling, a knocking sound, etc.

[0185] If the size of the detected sound is less than the second threshold value (step S1340-N) or the duration of the detected sound is less than the second threshold time (step S1350-N), the electronic device (100) may determine the detected sound as having a third importance (step S1370). For example, sounds of third importance among sounds generated from unregistered objects may be ambient noise, white noise, etc.

[0186] Meanwhile, in the above-described embodiment of the present disclosure, the importance of a sound related to an unregistered object is determined using the size and duration of the detected sound, but this is only one example, and it is of course possible to determine the importance of a sound related to an unregistered object using only one of the size and duration of the detected sound.

[0187] In addition, although the above-described embodiment of the present disclosure has described three levels of importance of sounds related to unregistered objects, this is only one example, and it is obvious that the level can be simplified or subdivided beyond three levels.

[0188] Referring back to FIG. 9, the electronic device (100) may project a message containing information about the sound based on its significance (step S965). At this time, the electronic device (100) may project a message containing information about the sound at the current location or the location where the sound originated, based on the significance of the detected sound. This will be described with reference to FIG. 14.

[0189] FIG. 14 is a flowchart illustrating a method for providing a message based on the importance of a sound associated with an unregistered thing object, according to one embodiment of the present disclosure.

[0190] Referring to FIG. 14, if the detected sound is determined to be of the first priority (step S1410-Y), the electronic device (100) may project a notification message to the user (step S1420). At this time, the electronic device (100) may output a notification message informing the user of an emergency situation for a quick response. For example, the electronic device (100) may output a notification message informing that emergency audio has been detected on a screen of a preset color (e.g., red) while rotating 360 degrees from the current location. In addition, the electronic device (100) may include information about the emergency audio (e.g., the location where the emergency audio was generated, the volume of the emergency audio, etc.) and information guiding movement to the location where the emergency audio was generated in the notification message, and may output an LED warning light and an alarm sound together with the notification message.

[0191] The electronic device (100) may move to the location where the detected sound occurred (step S1430). That is, the electronic device (100) may move to the location where the detected sound occurred to guide the user's quick response. For example, if the detected sound is a siren, the electronic device (100) may move to a window.

[0192] The electronic device (100) can project a message containing information about the sound at the location where the sound was generated (step S1440). Specifically, the electronic device (100) can output a message containing information about the emergency audio (e.g., the type of emergency audio, the volume of the emergency audio, etc.) and information about how to respond at the location where the sound was generated. For example, after moving to a window, the electronic device (100) can output a message saying, "A 128 dB siren sound has occurred. Please open the window to confirm and then evacuate."

[0193] If the detected sound is determined to be not the first priority (step S1410-N) but the second priority (step S1450-Y), the electronic device (100) may project a message containing information about the sound in a direction corresponding to the location where the sound originated from the current location (step S1460). For example, the electronic device (100) may project a message from the current location toward the window saying, "A scream has occurred outside the window. Please check this."

[0194] If the detected sound is determined to be not of the second priority (step S1450-N), the electronic device (100) may determine that the detected sound is of the third priority (step S1470). The electronic device (100) may not output a message (step S1480). That is, the electronic device (100) may determine that the currently detected sound is noise and may not output a separate message.

[0195] Hereinafter, a method for an electronic device (100) to provide a message including information about a detected sound will be described according to various embodiments of the present disclosure.

[0196] FIG. 15 is a diagram illustrating a method for providing a message including information related to sound generated in a separate space within a home, according to one embodiment of the present disclosure.

[0197] Referring to FIG. 15, specifically, the electronic device (100) can navigate within a space within a home, but can also detect sounds occurring within all spaces within the home. For example, if a sound occurs in a space with a closed door or is too far away to be detected by the microphone (120), the electronic device (100) cannot detect the sound even if the sound occurs within the home. In this case, the electronic device (100) can detect sounds occurring within the home through an external device (20) within the home. At this time, the external device (20) may be a device registered to detect sounds within the space where the external device (20) is located by linking with the electronic device (100). At this time, the external device (20) may store information regarding a map that stores audio related to objects stored in the electronic device (100).

[0198] For example, as illustrated in FIG. 15, when an external device (20) (e.g., an AI speaker) located in a kitchen detects the sound of a glass bottle (1510) breaking, the electronic device (100) can obtain information about the sound detected from the external device (20). At this time, the electronic device (100) may receive the sound detected by the external device (200), but this is only one embodiment, and the electronic device (100) can obtain information about the sound analyzed by the external device (20) (e.g., information about the size of the sound, the pitch of the sound, the wavelength, and the period).

[0199] In addition, the electronic device (100) may provide a message including information about the sound based on information about the sound received from the external device (20). For example, the electronic device (100) may project a message (1520) that reads, "A sound of breaking glass was detected around the kitchen at 123 dB," based on an object and sound associated with the sound, as illustrated in FIG. 15.

[0200] That is, the electronic device (100) can project a message containing information about sound by linking with an external device (20) located in the home.

[0201] FIG. 16 is a diagram illustrating a method for providing a message including information about a sound related to a failure of an object, according to one embodiment of the present disclosure.

[0202] Referring to FIG. 16, specifically, the electronic device (100) can recognize information about a failure or problem that may occur in existing home appliances that are not connected to the electronic device (100) or an AP in the home, and project a message notifying the user of the same.

[0203] Specifically, when excessive noise is detected during a spin-drying operation in a washing machine (1610), which is an object within a home, the electronic device (100) can obtain information about the detected sound based on information about audio related to the washing machine stored in the electronic device (100). That is, the electronic device (100) can identify information about the detected sound by searching for audio related to noise that may occur during a spin-drying operation among information about audio related to the washing machine.

[0204] In addition, the electronic device (100) may output a message asking the user whether to move to the space where the washing machine (1610) is located. When the user moves to the space where the washing machine (1610) is located together with the electronic device (100), the electronic device (100) may project a message (1620) in the direction of the user, which is information related to sound, such as “The washing machine is making an abnormal noise and needs to check the motor.”

[0205] At this time, the electronic device (100) can output a button or message for making an after-sales service (A / S) reservation. In addition, the electronic device (100) can provide data (e.g., a recording) regarding noise generated by the repair technician's terminal device during A / S, and can play back the noise generated to the repair technician.

[0206] FIG. 17 is a diagram illustrating a method for providing a message including information about sound related to emergency audio, according to one embodiment of the present disclosure.

[0207] Referring to FIG. 17, specifically, the electronic device (100) may store information about audio at a specific location on the map, rather than storing emergency audio on the map in relation to an object. Specifically, when storing information about audio on the map, the electronic device (100) may analyze objects included in an image acquired through the camera (120) to obtain information about the area where the electronic device (100) is currently located. For example, if objects included in the image include a front door, a shoe rack, shoes, etc., the electronic device (100) may identify the area where the electronic device (100) is currently located as the front door. In addition, the electronic device (100) may store information about emergency audio related to the front door at the front door location on the map.

[0208] And, when a sound corresponding to an emergency audio is detected, the electronic device (100) can move to a specific area on the map where the emergency audio is stored based on information about the emergency audio stored on the map and project a message including information related to the sound. For example, when a fire alarm sound is detected, the electronic device (100) can move to the front door (1710) where information about the fire alarm sound is stored on the map within the house, as illustrated in FIG. 17, and project a message (1720) saying "A 132 dB fire alarm sound has been detected. Please evacuate quickly" at the front door (1710).

[0209] FIGS. 18A and 18B are diagrams illustrating a method for providing a message when sounds related to a thing object and a person object are sequentially generated according to various embodiments of the present disclosure.

[0210] When a first sound related to a thing object and a second sound related to a person object are sequentially generated, the electronic device (100) can sequentially provide a message including information about the first sound and the second sound.

[0211] Referring to FIG. 18a, when a first sound corresponding to a front door bell is detected, the electronic device (100) can move to the front door (1810) where the first sound is detected based on information about the first sound, and project a first message (1820) that is information related to the first sound, “Front doorbell ringing occurred.”, at the moved front door (1810).

[0212] Referring to FIG. 18b, when a second sound corresponding to a human voice is detected outside the front door, the electronic device (100) recognizes the second sound through the STT module, acquires the text corresponding to the second sound, “Are you there? I am a new resident who moved in next door. I would like to say hello,” and projects a second message (1830) including the acquired text at the current front door (1810).

[0213] FIG. 19 is a diagram illustrating a method for providing a message including information about sound generated in a display device according to one embodiment of the present disclosure.

[0214] Referring to FIG. 19, the electronic device (100) can also recognize sound generated from the display device and project subtitles corresponding to the sound generated from the display device.

[0215] Specifically, when a sound is detected from the display device (1910), the electronic device (100) can move to the area where the display device (1910) is located and project a first message (1920) that is information about the sound, which is “television sound generation detected.”

[0216] And, when a sound corresponding to a human voice is detected from the display (1910), the electronic device (100) can recognize the sound output by the display device (1910) through the STT module and obtain a text corresponding to the sound. And, the electronic device (100) can project a second message (1930) including a text corresponding to the sound obtained in the area where the display (1910) is located, i.e. a subtitle, toward the user.

[0217] FIGS. 20, 21a, and 21b are diagrams illustrating a method for providing a message including information related to noise occurring in a home, according to various embodiments of the present disclosure.

[0218] Referring to FIGS. 20, 21a, and 21b, specifically, the electronic device (100) may store information regarding standards for noise that may occur within a home where the electronic device (100) is located. For example, the electronic device (100) may store information regarding standards for inter-floor noise permitted in an apartment where the electronic device (100) is located.

[0219] In addition, if the sound generated within the home exceeds the noise standard that may be generated within the home, the electronic device (100) may project a message including information about the detected sound and a warning about the noise. For example, if the sound of a chair being dragged exceeding the noise standard is detected, the electronic device (100) may project a message (2010) toward the user including information about the chair being dragged and a warning about the noise, such as "The chair being dragged is continuously above 57 db. Please be careful!"

[0220] At this time, if a sound that deviates from the noise standard is detected more than a preset number of times or the size of the sound that deviates from the noise standard is greater than a threshold value, the electronic device (100) may provide a message (2010) as illustrated in FIG. 20. In addition, the electronic device (100) may provide a message (2010) as illustrated in FIG. 20 based on the time at which the noise was generated (e.g., 10 p.m. to 7 a.m.).

[0221] In addition, the electronic device (100) can detect low-frequency noise that cannot be heard by a person in the house where the electronic device (100) is located, but can cause stress, and provide information about the detected noise to the user. For example, if the electronic device (100) detects low-frequency noise below a preset frequency (e.g., 100 Hz) for a preset period of time or at a set cycle, the electronic device (100) can obtain information about the low-frequency noise (e.g., the location where the low-frequency noise was generated, the duration, the repetition time, etc.) and project a message including the information about the low-frequency noise. For example, as illustrated in FIG. 21A, the electronic device (100) can project a first message (2110) toward the user that says, "Low-frequency noise has been continuously detected around the living room (TV) for the past week."

[0222] Additionally, the electronic device (100) may project a message containing information about the effects of low-frequency noise on the user's body. For example, as illustrated in FIG. 21b, the electronic device (100) may project a second message (2120) toward the user stating, "High low-frequency noise intensity may cause physical stress, such as headaches."

[0223] As described above, by projecting information about various sounds occurring within a home as messages, it is possible to provide information about the situation within the home more quickly and accurately to users who have hearing loss or are unable to detect sounds.

[0224] Meanwhile, the order shown in the flowchart according to the various embodiments described above is only one example, and each step shown in the flowchart may be performed in parallel, and the order of each step included in the flowchart may be changed.

[0225] Meanwhile, the methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0226] Various embodiments of the present disclosure, as described in the claims and specification, may be implemented in the form of hardware, software, or a combination of hardware and software.

[0227] Such software may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), and the one or more computer programs include computer-executable instructions that, when executed by one or more processors of the electronic device, cause the electronic device to perform the tasks described above.

[0228] Any such software may be stored in the form of a volatile or non-volatile storage medium. For example, a storage device such as a read-only memory (ROM), whether erasable or rewritable, or may be stored in the form of memory such as a random access memory (RAM), a memory chip, a device or an integrated circuit, or an optically or magnetically readable medium such as a compact disk (CD), a digital versatile disk (DVD), a magnetic disk or a magnetic tape. It will be appreciated that the storage device and the storage medium are various embodiments of a non-transitory machine-readable storage device suitable for storing a computer program or a computer program that includes instructions for implementing various embodiments of the present disclosure when executed. Accordingly, various embodiments provide a program comprising code for implementing an apparatus or method as recited in any of the claims of this specification, and a non-transitory machine-readable storage medium storing such a program.

[0229] The methods according to various embodiments of the present disclosure may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call commands stored in the storage medium and operate according to the called commands, and may include an electronic device (e.g., a TV) according to the disclosed embodiments.

[0230] Meanwhile, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0231] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0232] While the present disclosure has been illustrated and described with reference to various embodiments, it will be apparent to those skilled in the art that various changes in form and detail may be made therein without departing from the knowledge and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In electronic devices, A driving unit for driving the above electronic device; A microphone for receiving audio; camera; A projection unit for projecting images; A memory storing one or more computer programs; and It comprises one or more processors communicatively coupled with the driving unit, the microphone, the camera, the projection unit and the memory; One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, While driving through the above driving section, information about audio generated by an object located in an indoor space is acquired through the above microphone, and an image of the object is acquired through the above camera. Obtain information about expected audio related to the object based on an image of the object, and store the information about the audio generated by the object and the expected audio on a map corresponding to the indoor space. When a sound is detected within the above indoor space, an object related to the detected sound is identified based on information about audios stored on the map, An electronic device that controls the projection unit to project a message including information about the sound based on information about the related object and information about the sound.

2. In paragraph 1, further comprising a communication interface; One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, Obtain identification information of the object based on the image of the object, The identification information of the acquired object is transmitted to an external server through the above communication interface, An electronic device that obtains information about expected audio that may occur from said object from said external server.

3. In paragraph 1, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, Obtain information about emergency audio that requires urgent response related to the location where the above object is located, An electronic device that stores the emergency audio on a map corresponding to the indoor space.

4. In paragraph 1, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, Identifying whether an object related to the detected sound exists based on information about the audios stored on the map, When it is identified that there is an object related to the detected sound, it identifies whether the sound originated from a registered person object or a registered thing object. If the detected sound is identified as a sound generated from the registered human object, the driving unit is controlled to identify the location of the registered human object and move to the location of the human object. An electronic device that controls the projection unit to project a message including text that recognizes the sound around the human object when the human object exists at the moved location.

5. In paragraph 4, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, An electronic device that controls the projection unit to project a message including information on the estimated location of the human object and a text that recognizes the sound, if the human object is an unregistered person or the human object does not exist at the moved location.

6. In paragraph 4, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, If the detected sound is identified as a sound generated from the registered object, the importance of the detected sound is determined based on the stored criteria, An electronic device that controls the projection unit to project a message containing information about the sound at the current location or at the location where the sound occurred, depending on the importance of the detected sound.

7. In paragraph 6, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, If the above detected sound is identified as an emergency sound, it is judged as having the first priority, If the waveform of the detected sound is repeated more than a preset number of times, or if the detected sound matches audio that includes irregular noise among audios matching the registered object, or if the detected sound is identified as a call sound, it is determined to be of second importance. An electronic device that determines that the sound is of a third priority if the sound is identified as not of the first priority or the second priority.

8. In paragraph 7, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, If the importance of the detected sound is the first importance, the driving unit is controlled to output a notification message to the user and move to the location where the detected sound occurred, and the projection unit is controlled to project a message including information about the sound at the location where the sound occurred. If the importance of the detected sound is the second importance, a message is output asking the user whether to move to the location where the detected sound occurred, and if the user moves to the location where the detected sound occurred, the projection unit is controlled to project a message including information about the sound at the location where the sound occurred, and if the user does not move to the location where the detected sound occurred, the projection unit is controlled to project a message including information about the location where the sound occurred and information about the sound at the current location. An electronic device that controls the projection unit to project a message containing information about the sound to the user at the current location if the importance of the detected sound is the third importance.

9. In paragraph 5, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, An electronic device that identifies the significance of the detected sound based on at least one of the loudness of the detected sound and the duration of the detected sound, if the detected sound is identified as a sound generated from the unregistered human object.

10. In paragraph 8, One or more computer-executable instructions included in said one or more computer programs, when individually or collectively executed by said one or more processors, If the importance of the detected sound is the first importance, the driving unit is controlled to output a notification message to the user and move to the location where the detected sound occurred, and the projection unit is controlled to project a message including information about the sound at the location where the sound occurred. If the importance of the detected sound is the second importance, the projection unit is controlled to project a message including information about the sound in a direction corresponding to the location where the sound occurred from the current location. An electronic device that does not output a message if the priority of the detected sound is the third priority.

11. In a method for controlling an electronic device, A step of obtaining information about audio generated by an object located in an indoor space through a microphone while the electronic device is driving and obtaining an image of the object through a camera; A step of obtaining information about expected audio related to the object based on an image of the object, and storing the information about the audio generated by the object and the expected audio on a map corresponding to the indoor space; When a sound is detected within the indoor space, a step of identifying an object related to the detected sound based on information about audios stored on the map; and A control method comprising: a step of projecting a message including information about the sound based on information about the related object and information about the sound.

12. In paragraph 11, The above saving steps are: A step of obtaining identification information of the object based on an image of the object; A step of transmitting the identification information of the acquired object to an external server; and A control method comprising: a step of obtaining information about expected audio that may occur from the object from the external server; 13. In paragraph 11, The above saving steps are: A step of obtaining information about emergency audio that requires urgent response in relation to the location where the above object is located; and A control method comprising: a step of storing the emergency audio on a map corresponding to the indoor space; 14. In paragraph 11, The step of identifying the above object is: A step of identifying whether an object related to the detected sound exists based on information about audios stored on the map; and If it is identified that there is an object related to the detected sound, a step of identifying whether the sound originated from a registered person object or a registered thing object is included; The steps to project the above message are: If the detected sound is identified as a sound generated from the registered human object, the location of the registered human object is identified and movement is made to the location of the human object. A control method for projecting a message including text that recognizes the sound around the person object when the person object exists at the moved location.

15. In paragraph 14, The steps to project the above message are: A control method for projecting a message including information on the estimated location of the person object and a text recognizing the sound, if the person object is not registered or the person object does not exist at the moved location.

Images