Electronic device and operation method thereof

Abstract

An electronic device according to the present disclosure comprises: a memory for storing at least one instruction; and at least one processor for executing the at least one instruction stored in the memory, wherein the at least one processor can: identify a final state of a user on the basis of first state information acquired according to a sound signal pattern acquired from a sound signal and second state information acquired according to a Wi-Fi signal pattern acquired from a Wi-Fi signal; and control that operation occurs in a state corresponding to the final state of the user.

Description

Electronic device and method of operation thereof

[0001] The disclosed various embodiments relate to electronic devices and methods of operating the same, and more particularly to electronic devices that operate in response to a user's state, and methods of operating the same.

[0002] Many users tend to watch content on electronic devices like TV before bed. If the device continues to output content after the user falls asleep, the content output can actually interfere with a good night's sleep.

[0003] An electronic device according to an embodiment of the present invention may include a memory storing at least one instruction and at least one processor executing the at least one instruction stored in the memory.

[0004] In an embodiment, the at least one processor can identify the user final state based on first state information obtained according to an acoustic signal pattern obtained from an acoustic signal and second state information obtained according to a Wi-Fi signal pattern obtained from a Wi-Fi signal.

[0005] In an embodiment, the at least one processor can be controlled to operate in a state corresponding to a user end state.

[0006] FIG. 1 is a drawing showing an electronic device operating in a mode corresponding to a user state according to an embodiment.

[0007] Figure 2 is a block diagram of an electronic device according to an embodiment.

[0008] Figure 3 is a block diagram of an electronic device according to an embodiment.

[0009] FIG. 4 illustrates a Wi-Fi signal pattern acquired by the processor of FIG. 3 based on a CSI channel value input through a Wi-Fi signal acquisition unit, according to an embodiment.

[0010] FIG. 5 is a graph showing the strength of an ACF signal according to the number of users obtained based on CSI channel values, according to an embodiment.

[0011] Figure 6 is a graph showing changes in Wi-Fi signals according to the user's movement.

[0012] Figure 7 is an internal block diagram of a processor according to an embodiment.

[0013] Figure 8 is a drawing explaining how the distance recognition unit of Figure 7 obtains the sound source distance using a neural network that has learned distance-specific sound data.

[0014] FIG. 9 is a table explaining that the signal processing unit of FIG. 7 determines the state weight based on the first state information and the second state information.

[0015] Fig. 10 is a block diagram according to an example of the signal processing unit of Fig. 7.

[0016] FIG. 11 is a diagram illustrating, according to an embodiment, separation of a voice signal into user-specific voice signals.

[0017] FIG. 12 is a diagram illustrating, according to an embodiment, how the signal separation unit of FIG. 10 identifies a WiFi channel corresponding to an acoustic signal using a neural network.

[0018] Fig. 13 is an internal block diagram of an electronic device according to an embodiment.

[0019] Fig. 14 is a flowchart illustrating an operation method of an electronic device according to an embodiment.

[0020] Fig. 15 is a flowchart illustrating an operation method of an electronic device according to an embodiment.

[0021] According to an embodiment, a method of operating an electronic device may include a step of identifying a user final state based on first state information obtained according to an acoustic signal pattern obtained from an acoustic signal and second state information obtained according to a Wi-Fi signal pattern obtained from a Wi-Fi signal.

[0022] In an embodiment, a method of operating an electronic device may include a step of operating in a state corresponding to the user final state.

[0023] A recording medium according to an embodiment may be a computer-readable recording medium having recorded thereon a program that can perform a method of operating an electronic device by a computer, the method including a step of identifying a user final state based on first state information acquired according to an acoustic signal pattern acquired from an acoustic signal and second state information acquired according to a Wi-Fi signal pattern acquired from a Wi-Fi signal.

[0024] In an embodiment, the recording medium may be a computer-readable recording medium having recorded thereon a program that can perform a method of operating an electronic device, the method including a step of operating in a state corresponding to the user's final state, by a computer.

[0025] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0026] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0027] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0028] Additionally, the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure.

[0029] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where it is "directly connected" but also the cases where it is "electrically connected" with another element in between.

[0030] As used herein, and particularly in the claims, the terms "said" and "said" and similar referents may refer to both the singular and the plural. Furthermore, unless the order of steps in a method according to the present disclosure is explicitly specified, the steps described may be performed in any appropriate order. The present disclosure is not limited by the order in which the steps are described.

[0031] The appearances of phrases such as “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

[0032] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0033] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0034] Additionally, terms such as “part”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0035] Additionally, the term "user" in the specification means a person who uses an electronic device, and may include a consumer, evaluator, viewer, administrator, or installer.

[0036] The present disclosure will be described in detail with reference to the attached drawings below.

[0037] The present disclosure may relate to artificial intelligence (AI) systems and / or applications of AI systems utilizing machine learning algorithms.

[0038] FIG. 1 is a drawing showing an electronic device (100) operating in a mode corresponding to a user state according to an embodiment.

[0039] In an embodiment, the electronic device (100) may be an electronic device capable of outputting content. In an embodiment, the content may be in various forms, such as video, audio, subtitles, other additional information, etc., such as still images or moving images.

[0040] In an embodiment, the electronic device (100) may be a video display device capable of outputting images or videos through a display, and / or an audio device capable of outputting audio. The electronic device (100) may be stationary or mobile.

[0041] In an embodiment, the electronic device (100) may include at least one of a television, a desktop, a smartphone, a tablet personal computer, a game console, an audio device, a mobile phone, a video phone, an e-book reader, a laptop personal computer, a netbook computer, a digital camera, a personal digital assistant (PDA), a portable multimedia player (PMP), a camcorder, a navigation device, a wearable device, a smart watch, a home network system, a security system, and a medical device.

[0042] When the electronic device (100) is a video display device, the electronic device (100) may be implemented as a flat display device, a curved display device having a screen with a curvature, or a flexible display device whose curvature can be adjusted. The output resolution of the electronic device (100) may have various resolutions, such as, for example, HD (High Definition), Full HD, Ultra HD, or a resolution clearer than Ultra HD.

[0043] In an embodiment, if the electronic device (100) is a television (TV), the electronic device (100) may include a digital TV equipped with an operating system (OS) and an Internet connection function.

[0044] The electronic device (100) can output various types of content provided by content providers.

[0045] A content provider may refer to a terrestrial broadcaster, cable broadcaster, satellite broadcaster, IPTV (Internet Protocol Television) service provider, OTT (Over the Top) service provider, or server operator that provides various types of content to consumers.

[0046] Content can take many forms, including video, including still images or moving images, audio, subtitles, and other additional information.

[0047] Content may include content for viewing, such as movies or dramas, content for listening, such as music, game content, and art content that introduces or guides works of art, such as famous paintings or sculptures.

[0048] In an embodiment, the electronic device (100) can receive and output various contents generated by a content provider through an external device. The external device can be implemented as a source device of various forms, such as a PC, a set-top box, a Blu-ray disc player, a mobile phone, a game console, a home theater, an audio player, a USB, etc.

[0049] An external device can be connected to the electronic device (100) via a wired or wireless communication network, such as HDMI, and provide various contents to the electronic device (100). The electronic device (100) can receive VOD (Video On Demand) contents provided by an IPTV service provider or an OTT service provider via a set-top box and output the same. The VOD service is a service that provides a user with desired video at a desired time through a communication network connection, and can include various types of contents provided by an OTT service provider or an IPTV service provider. The IPTV service provider or an OTT service provider can provide not only VOD contents but also real-time broadcast programs.

[0050] In an embodiment, when an operating system is installed in the electronic device (100), the electronic device (100) can stream and output various types of VOD content created by an OTT service provider using the operating system installed therein in addition to real-time broadcasting programs.

[0051] In an embodiment, the electronic device (100) can connect to the Internet and provide web surfing services, social network services, etc. In addition, the electronic device (100) can function as a communication center that can check news, weather, email, etc. in real time.

[0052] In an embodiment, the electronic device (100) can execute various types of applications. The electronic device (100) may have various types of apps (applications) installed by default. Alternatively, the electronic device (100) may, under user control, access the Internet, search for apps requested by the user, and install them. The electronic device (100) can execute apps to provide various services.

[0053] In an embodiment, the electronic device (100) may be wirelessly connected to an external device, such as a set-top box, through a wireless network that follows a communication standard such as Bluetooth, WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), CDMA, or WCDMA, and may receive a video signal from the external device.

[0054] In an embodiment, the electronic device (100) may be connected to an external device via a wired cable to receive a video signal from the external device or transmit a video signal to the external device. The wired cable may include a port capable of simultaneously transmitting a video signal and an audio signal, such as a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB), a Display Port (DP), or Thunderbolt. Alternatively, the wired cable may include a port for separately transmitting a video signal and an audio signal.

[0055] In an embodiment, the electronic device (100) may be implemented as an electronic device that does not include a video display device such as a display. For example, if the electronic device (100) is an external device itself, such as a set-top box, a satellite broadcast receiving device, an Internet receiving device that receives content from an OTT (Over The Top) service provider, the electronic device (100) may be of a form that does not include a video display device.

[0056] In this case, the electronic device (100) can be connected to the video display device by a wire, and transmit a signal input from an external source to the video display device. For example, the electronic device (100) can be connected to the video display device, such as a monitor, using a port that can simultaneously transmit video signals and audio signals, such as a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI), a Display Port (DP), or Thunderbolt. Alternatively, the electronic device (100) can be connected to the video display device using a port that separately transmits video signals and audio signals.

[0057] In an embodiment, the electronic device (100) may be controlled by a control device. In an embodiment, the control device may be a device used to control the electronic device (100), such as a remote controller. A user may control various functions of the electronic device (100) using the control device.

[0058] In an embodiment, the control device may be a dedicated control device for the electronic device (100). Alternatively, in an embodiment, the control device may be an electronic device, such as a smartphone or an AI speaker, whose main function is to perform operations other than controlling the electronic device (100). A user may control the electronic device (100) using the control device by installing a remote control app or the like on the control device. In this case, the control device may include at least one module capable of performing Wi-Fi, Bluetooth, or infrared communication. The control device may transmit and receive data with the electronic device (100) using a Wi-Fi, Bluetooth, or infrared communication module, etc.

[0059] In an embodiment, the control device may have an input unit. The input unit may receive user input for controlling the electronic device (100). The input unit included in the control device may include a plurality of keys. The keys may be in various forms, such as physical buttons that receive a user's push operation, jog shuttles, touch buttons displayed on a touchpad that detects touch, etc. The user may control various functions of the electronic device (100) using the plurality of keys included in the control device.

[0060] A plurality of keys included in the control device can be used to control various functions of the electronic device (100).

[0061] In an embodiment, the electronic device (100) can obtain an acoustic signal.

[0062] In an embodiment, at least one of the electronic device (100) or the control device may include a microphone capable of receiving an acoustic signal. If the control device is equipped with a microphone, the microphone may receive an analog signal, digitize the analog signal, and transmit the analog signal to the electronic device (100). In an embodiment, the control device may receive an acoustic signal through the microphone, digitize the analog signal, and transmit the digitized acoustic signal to the electronic device (100) using a data transmission communication method such as Bluetooth or Wi-Fi.

[0063] In an embodiment, the electronic device (100) may be equipped with a plurality of microphones spaced apart from each other. In an embodiment, the electronic device (100) may collect an acoustic signal through the plurality of microphones and digitize the same.

[0064] FIG. 1 illustrates, as an example, a case where the electronic device (100) is a TV.

[0065] Figure 1 illustrates a case where multiple users fall asleep while watching an electronic device (100) in the same space as the electronic device (100).

[0066] A user can use the electronic device (100) for its intended purpose in the space where the electronic device (100) is located. For example, a user can use the electronic device (100) in the room, living room, or office where the electronic device (100) is located.

[0067] In an embodiment, the electronic device (100) can obtain status information about a user within a space where the electronic device (100) is located.

[0068] In an embodiment, the electronic device (100) can obtain an acoustic signal pattern from an acoustic signal obtained through a plurality of microphones.

[0069] In an embodiment, the electronic device (100) can synchronize an acoustic signal and a Wi-Fi signal and obtain an acoustic signal pattern from the synchronized acoustic signal.

[0070] In an embodiment, the electronic device (100) can remove noise from an acoustic signal.

[0071] In an embodiment, when the electronic device (100) outputs audio content through a speaker, the electronic device (100) can remove the audio content output through the speaker from the sound signal along with noise.

[0072] In an embodiment, the electronic device (100) can convert a time domain-based audio signal, with audio content and noise filtered, into a signal in the frequency domain.

[0073] In an embodiment, the electronic device (100) can convert a filtered acoustic signal into a Mel-spectrogram image and obtain information about the frequency characteristics of the sound from the Mel-spectrogram image.

[0074] In an embodiment, the frequency characteristics of the sound may include at least one of pitch, timbre, and intensity.

[0075] In an embodiment, the electronic device (100) may acquire an acoustic signal pattern representing information about the frequency characteristics of a sound. In an embodiment, the acoustic signal pattern may include feature information representing a pattern corresponding to a specific event. The feature information may be expressed in the form of a feature vector.

[0076] In an embodiment, the electronic device (100) can obtain status information from an acoustic signal pattern.

[0077] In an embodiment, the status information may be information indicating the user's status.

[0078] In an embodiment, the status information may include information about at least one of the presence or absence of a user, whether the user is breathing, whether the user is moving, and the number of users.

[0079] In an embodiment, the electronic device (100) can use acoustic signal patterns to identify frequency characteristics of various sounds generated when a user breathes, speaks, or performs a specific action.

[0080] Hereinafter, the state information obtained using the acoustic signal pattern will be referred to as first state information.

[0081] In an embodiment, the electronic device (100) can obtain a Wireless Fidelity (Wi-Fi) signal.

[0082] In an embodiment, the electronic device (100) may be located in a place where a Wi-Fi transmitter and a Wi-Fi receiver capable of transmitting and receiving Wi-Fi signals are installed.

[0083] Wi-Fi signals can use radio waves to transmit data. These waves can operate on frequency bands such as 2.4GHz or 5GHz, depending on the Wi-Fi standard used. A router can act as a network hub by being connected to an Internet Service Provider (ISP) via a wired connection, such as DSL, cable, or fiber optic. The router can convert data from a wired Internet connection into wireless signals and broadcast them on a specific channel within a selected frequency band.

[0084] When a router or access point (AP) acts as a Wi-Fi transmitter and transmits Wi-Fi signals, a Wi-Fi receiver or Wi-Fi sensor can receive the Wi-Fi signals.

[0085] In an embodiment, the electronic device (100) can obtain a Wi-Fi signal through a Wi-Fi receiver.

[0086] In an embodiment, the electronic device (100) can synchronize an audio signal and a Wi-Fi signal, and obtain a Wi-Fi signal pattern from the synchronized Wi-Fi signal.

[0087] In an embodiment, the electronic device (100) can collect a pattern trend of a Wi-Fi signal over a certain period of time and identify a pattern corresponding to a specific event therefrom.

[0088] Indoors, Wi-Fi signals can be transmitted as multipath signals through reflection, scattering, and diffraction. Wi-Fi signals tend to change in strength when they reflect off objects or pass near them. Therefore, the signal strength and amplitude between channels vary depending on the presence or absence of users in a space, as well as the number of users within the space.

[0089] In an embodiment, the electronic device (100) can identify status information using a signal pattern that varies depending on the reflection / diffraction of a Wi-Fi signal.

[0090] Hereinafter, the status information obtained from the Wi-Fi signal will be referred to as second status information.

[0091] In an embodiment, the electronic device (100) can identify the user final state based on the first state information and the second state information.

[0092] In some cases, the acoustic signals acquired by the electronic device (100) through multiple microphones may include acoustic signals generated externally rather than in the space where the electronic device (100) is located.

[0093] Additionally, the Wi-Fi signal acquired by the electronic device (100) may be a signal influenced by a user outside the space where the electronic device (100) is located.

[0094] Wi-Fi signals are not only highly sensitive but also operate under non-line-of-sight (NLOS) conditions, allowing them to sense users at greater distances and across a wider angular range. Therefore, Wi-Fi signals can be affected by the presence or activity of other users, even those in neighboring homes, rather than within the same household.

[0095] In an embodiment, the electronic device (100) may obtain an acoustic distance for an acoustic signal to identify whether the status information is status information about a user within a space where the electronic device (100) is located. In an embodiment, the acoustic distance may be a distance from a location where the acoustic signal is generated to the electronic device (100). Specifically, the acoustic distance may be a distance from a location where the acoustic signal is generated to a plurality of microphones provided in the electronic device (100).

[0096] In an embodiment, the electronic device (100) can measure the distance from the location where the sound signal is generated to the microphone by considering the distance between multiple microphones, the speed of sound, the time difference in which each microphone collected the sound signal, etc.

[0097] In an embodiment, the electronic device (100) can identify whether the acoustic distance is within a preset range. The preset range may be information indicating a preset range of a certain distance. The electronic device (100) can identify whether the distance between the electronic device (100) and the point of acoustic signal generation, i.e., the acoustic distance, is within a certain range of distances.

[0098] In an embodiment, the electronic device (100) can determine the final state of the user based on the first state information and the second state information when the acoustic distance is within a preset range.

[0099] In an embodiment, the electronic device (100) can detect multiple users based on at least one of a Wi-Fi signal and an audio signal.

[0100] In an embodiment, the electronic device (100) can identify whether there are multiple users based on at least one of the fluctuation range and signal strength of the Wi-Fi signal.

[0101] In an embodiment, the electronic device (100) can detect multiple users based on at least one of a voice event and a breathing event being detected in an acoustic signal.

[0102] In FIG. 1, the electronic device (100) can identify that there are multiple users based on the strength of a Wi-Fi signal or a user's breathing event.

[0103] In an embodiment, when the electronic device (100) detects multiple users, it can identify a signal corresponding to each of the multiple users from the Wi-Fi signal and the audio signal.

[0104] In an embodiment, the electronic device (100) can identify an audio signal of each of multiple users from audio signals collected using multiple microphones.

[0105] In an embodiment, the electronic device (100) can identify and separate acoustic signals generated due to each user's speech characteristics, breathing characteristics, or each user's movements from acoustic signals of other users.

[0106] In an embodiment, the electronic device (100) can obtain first state information for each user from a separated acoustic signal.

[0107] In FIG. 1, the electronic device (100) can obtain first state information for each of two users.

[0108] In an embodiment, the electronic device (100) may identify a Wi-Fi channel corresponding to a separate acoustic signal for each user, corresponding to the detection of multiple users.

[0109] In an embodiment, the electronic device (100) can identify a Wi-Fi channel corresponding to each user's audio signal by using a neural network that has learned the relationship between the audio signal and the Wi-Fi signal.

[0110] In an embodiment, the electronic device (100) can obtain second status information for each user from the identified Wi-Fi channel.

[0111] In FIG. 1, the electronic device (100) can identify a Wi-Fi channel corresponding to each of the two users based on an acoustic signal corresponding to each of the two users.

[0112] In an embodiment, the electronic device (100) can determine the state weight of each user by using the first state information and the second state information corresponding to each user.

[0113] For example, in FIG. 1, the electronic device (100) can determine the status weight of the first user by using the first status information and the second status information corresponding to the first user among two users, and can determine the status weight of the second user by using the first status information and the second status information corresponding to the second user.

[0114] In an embodiment, the electronic device (100) accumulates the state weight of each user for a certain period of time, and can identify the final state of each user from this.

[0115] In an embodiment, the final state may mean a final state of the user determined by considering first state information based on an acoustic signal and second state information based on a Wi-Fi signal for a certain period of time.

[0116] In FIG. 1, the electronic device (100) can determine that the final state of each of the two users is a sleeping state.

[0117] In an embodiment, the electronic device (100) may operate in a mode corresponding to the user's final state. In an embodiment, a mode may refer to a state in which a specific action or task can be performed. In the present disclosure, a mode may also be referred to as a state.

[0118] In an embodiment, if the electronic device (100) determines that all of the multiple users are in a sleeping state when there are multiple users, the electronic device (100) may operate in a sleep mode accordingly.

[0119] In an embodiment, the sleep mode may be referred to as a sleep state and may be a low power mode / state in which the power of the electronic device (100) is saved.

[0120] Unlike sleep mode, normal mode may be a mode in which power is supplied to all components of the electronic device (100). Normal mode may be referred to as a normal state.

[0121] In an embodiment, when the electronic device (100) detects a user while operating in normal mode and determines that the user's final state is a sleep state, the electronic device (100) may operate by changing the mode / state from normal mode to sleep mode.

[0122] In an embodiment, when multiple users are detected while the electronic device (100) is operating in normal mode and it is determined that the final states of all of the multiple users are in a sleep state, the electronic device (100) may change the mode from normal mode to sleep mode and operate.

[0123] In an embodiment, if the electronic device (100) determines that all detected users are in a sleeping state, it may turn off all functions except for processing Wi-Fi signals and audio signals. The electronic device (100) may prevent video content from being output through the screen and audio content from being output through the speakers.

[0124] In FIG. 1, the electronic device (100) can change the operation mode of the electronic device (100) to the sleep mode, corresponding to the final states of both users being in a sleep state.

[0125] In this way, according to an embodiment, the electronic device (100) can more accurately identify the user's status by using both a Wi-Fi signal and an audio signal.

[0126] In addition, according to an embodiment, the electronic device (100) can accurately identify the status of multiple users, not only when there is one user but also when there are multiple users, by using both Wi-Fi signals and audio signals.

[0127] In addition, according to an embodiment, when a user falls asleep while watching an electronic device (100), the user's sleep state is identified and content output is stopped in response, thereby helping the user get a good night's sleep.

[0128] Figure 2 is a block diagram of an electronic device (100) according to an embodiment.

[0129] The electronic device (100) of FIG. 2 may be an example of the electronic device (100) described in FIG. 1.

[0130] Referring to FIG. 2, the electronic device (100) may include a memory (120) and a processor (110).

[0131] In an embodiment, the memory (120) may store at least one instruction. The memory (220) may store at least one program executed by the processor (110). In addition, the memory (120) may store data input to or output from the electronic device (100).

[0132] The memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk.

[0133] In an embodiment, the memory (120) may store one or more instructions for obtaining first state information from an acoustic signal.

[0134] In an embodiment, the memory (120) may store one or more instructions for outputting audio content.

[0135] In an embodiment, the memory (120) may store one or more instructions for filtering audio content and noise from an audio signal.

[0136] In an embodiment, the memory (120) may store one or more instructions for converting a filtered acoustic signal into a Mel-spectrogram image.

[0137] In an embodiment, the memory (120) may store one or more instructions for obtaining first state information from frequency characteristics of sound obtained from a mel spectrogram image.

[0138] In an embodiment, the frequency characteristics of the sound may include at least one of pitch, timbre, and intensity.

[0139] In an embodiment, the memory (120) may store one or more instructions for obtaining second state information from a Wi-Fi signal.

[0140] In an embodiment, the memory (120) may store one or more instructions for obtaining a Wi-Fi signal pattern from a Wi-Fi signal.

[0141] In an embodiment, the memory (120) may store one or more instructions for obtaining second state information from a pattern trend of a Wi-Fi signal over a certain period of time.

[0142] In an embodiment, the memory (120) may store one or more instructions for identifying a user final state based on the first state information and the second state information.

[0143] In an embodiment, a preset range may be stored in the memory (120).

[0144] In an embodiment, the memory (120) may store one or more instructions for identifying whether the distance to the point of acoustic signal generation is within a preset range.

[0145] In an embodiment, the memory (120) may store one or more instructions for detecting multiple users based on at least one of an acoustic signal and a Wi-Fi signal.

[0146] In an embodiment, the memory (120) may store one or more instructions for detecting at least one of a voice event of multiple users and a breathing event of multiple users in an acoustic signal.

[0147] In an embodiment, the memory (120) may store one or more instructions for detecting multiple users based on at least one of a fluctuation range and signal strength of a Wi-Fi signal.

[0148] In an embodiment, the memory (120) may store one or more instructions for identifying a corresponding signal for each user from the acoustic signal and the Wi-Fi signal, corresponding to the detection of multiple users.

[0149] In an embodiment, the memory (120) may store one or more instructions for separating a first acoustic signal including a speech characteristic of a first user from the acoustic signal collected using a plurality of microphones.

[0150] In an embodiment, the memory (120) may store one or more instructions for obtaining first state information corresponding to a first user from a first acoustic signal.

[0151] In an embodiment, the memory (120) may store one or more instructions for identifying a first Wi-Fi channel corresponding to a first acoustic signal using a neural network that has learned the relationship between an acoustic signal and a Wi-Fi signal.

[0152] In an embodiment, the memory (120) may store one or more instructions for obtaining second state information corresponding to a first user from a first Wi-Fi channel.

[0153] In an embodiment, the memory (120) may store one or more instructions for determining a state weight of the first user using first state information and second state information corresponding to the first user.

[0154] In an embodiment, the memory (120) may store one or more instructions for identifying a final state of the first user based on the state weight of the first user acquired over a certain period of time.

[0155] In an embodiment, the memory (120) may store one or more instructions for operating in a mode corresponding to a user end state.

[0156] In an embodiment, the memory (120) may store one or more instructions for operating in sleep mode if the final state of all detected users is identified as a sleep state.

[0157] A processor (110) according to an embodiment controls the overall operation of an electronic device (100). A processor (101) according to an embodiment may control signal flow between internal components of the electronic device (100) and perform a function of processing data.

[0158] In an embodiment, the processor (101) may control the electronic device (100) to function by executing one or more instructions stored in the memory (103).

[0159] In an embodiment, the processor (101) may include single core, dual core, triple core, quad core and multiples thereof.

[0160] In an embodiment, the processor (101) may be one or more. For example, the processor (101) may include multiple processors. In this case, the processor (101) may be implemented as a main processor and a sub processor.

[0161] In addition, the processor (101) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a VPU (Video Processing Unit). Alternatively, in an embodiment, the processor (101) may be implemented in the form of a SoC (System On Chip) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, in an embodiment, the processor (101) may further include an NPU (Neural Processing Unit).

[0162] In an embodiment, at least one processor (110) can obtain first state information from an acoustic signal.

[0163] In an embodiment, at least one processor (110) may output audio content through a speaker provided in the electronic device (100).

[0164] In an embodiment, at least one processor (110) may filter audio content and noise from an acoustic signal.

[0165] In an embodiment, at least one processor (110) can convert the filtered acoustic signal into a Mel-spectrogram image.

[0166] In an embodiment, at least one processor (110) can obtain first state information from frequency characteristics of sound obtained from a mel spectrogram image.

[0167] In an embodiment, at least one processor (110) can obtain second state information from a Wi-Fi signal.

[0168] In an embodiment, at least one processor (110) can obtain a Wi-Fi signal pattern from a Wi-Fi signal.

[0169] In an embodiment, at least one processor (110) can obtain second state information from a pattern trend of a Wi-Fi signal over a certain period of time.

[0170] In an embodiment, at least one processor (110) may perform synchronization between an audio signal and a Wi-Fi signal.

[0171] In an embodiment, at least one processor (110) may synchronize the audio signal and the Wi-Fi signal by using an interval time between two signals representing an input time difference between the audio signal and the Wi-Fi signal.

[0172] In an embodiment, at least one processor (110) can obtain first state information and second state information from a synchronized audio signal and a Wi-Fi signal, respectively.

[0173] In an embodiment, at least one processor (110) can identify a user state based on the first state information and the second state information.

[0174] In an embodiment, at least one processor (110) can identify whether the distance to the point of acoustic signal generation is within a preset range.

[0175] In an embodiment, at least one processor (110) can detect multiple users based on at least one of an acoustic signal and a Wi-Fi signal.

[0176] In an embodiment, at least one processor (110) can detect at least one of a voice event of multiple users and a breathing event of multiple users in an acoustic signal.

[0177] In an embodiment, at least one processor (110) can detect multiple users based on at least one of a fluctuation range and signal strength of a Wi-Fi signal.

[0178] In an embodiment, at least one processor (110) can identify a corresponding signal for each user from the acoustic signal and the Wi-Fi signal, corresponding to the detection of multiple users.

[0179] In an embodiment, at least one processor (110) can separate a first acoustic signal including a speech characteristic of a first user from an acoustic signal collected using a plurality of microphones.

[0180] In an embodiment, at least one processor (110) can obtain first state information corresponding to a first user from a first acoustic signal.

[0181] In an embodiment, at least one processor (110) can identify a first Wi-Fi channel corresponding to a first acoustic signal using a neural network that has learned a relationship between an acoustic signal and a Wi-Fi signal.

[0182] In an embodiment, at least one processor (110) can obtain second state information corresponding to a first user from a first Wi-Fi channel.

[0183] In an embodiment, at least one processor (110) may determine a state weight of the first user using first state information and second state information corresponding to the first user.

[0184] In an embodiment, at least one processor (110) can identify a final state of the first user based on the state weight of the first user acquired over a period of time.

[0185] In an embodiment, at least one processor (110) may operate in a mode / state corresponding to a user end state.

[0186] In an embodiment, at least one processor (110) may operate in a sleep state if all detected users are identified as being in a sleeping state.

[0187] Figure 3 is a block diagram of an electronic device (100) according to an embodiment.

[0188] The electronic device (100) of FIG. 3 may be an example of the electronic device (100) described in FIG. 2.

[0189] Below, any explanation that overlaps with the explanation in Fig. 2 is omitted.

[0190] Referring to FIG. 3, the electronic device (100) may further include, in addition to the memory (120) and the processor (110), an output unit (130), an audio signal acquisition unit (140), and a Wi-Fi signal acquisition unit (150).

[0191] An output unit (130) according to an embodiment can output content. The output unit (130) can include at least one of a display for outputting video content and a speaker for outputting audio content.

[0192] The output unit (130) can output video and / or audio content while the electronic device (100) is operating in normal mode.

[0193] An acoustic signal acquisition unit (140) according to an embodiment can collect an acoustic signal. The acoustic signal may also be referred to as an audio signal.

[0194] The audio signal acquisition unit (140) may include a microphone array. The audio signal acquisition unit (140) may collect an audio signal through the microphone array.

[0195] The microphone array can capture audio signals including audio content output through speakers, the user's voice or various sounds made by the user, noise, etc., and can digitize analog audio signals.

[0196] In an embodiment, the processor (110) can separate noise from an audio signal collected through an audio signal acquisition unit (140). For example, the audio signal acquisition unit (140) can separate noise from an audio signal using a CNN model such as Wave-U-NET, but is not limited thereto.

[0197] In an embodiment, the processor (110) may separate audio content output to a speaker from an audio signal collected through an audio signal acquisition unit (140). In an embodiment, the processor (110) may capture audio content immediately before being output through a speaker included in an output unit (130). The processor (110) may synchronize the captured audio content with an audio signal collected through a microphone array included in the audio signal acquisition unit (140), and then remove the captured audio content from the audio signal.

[0198] In an embodiment, the processor (110) may filter audio content and noise from an acoustic signal, and then convert the time-domain-based acoustic signal into a signal in the frequency domain. For example, the processor (110) may convert the acoustic signal into a Mel-spectrogram image.

[0199] In an embodiment, the processor (110) can obtain frequency characteristics of sound from a mel spectrogram image. The frequency characteristics of sound can include at least one of pitch, timbre, and intensity.

[0200] In an embodiment, the processor (110) can obtain first state information from the frequency characteristics of the sound.

[0201] In an embodiment, the processor (110) can identify a user's state corresponding to the frequency characteristics of a sound obtained from a current acoustic signal by using a neural network that has previously learned a feature vector representing the frequency characteristics of various sounds.

[0202] For example, the processor (110) is a neural network that has learned in advance the feature vectors of various acoustic signals such as the user's breathing, footsteps, eating, and snack bag sounds, and inputs the currently acquired acoustic signal pattern and can obtain state information corresponding to the acoustic signal pattern from the neural network.

[0203] The Wi-Fi signal acquisition unit (150) according to the embodiment can acquire Wi-Fi signals. Channel indicators that can be measured by the Wi-Fi receiver include RSSI and CSI. For convenience of explanation, the following description will describe a case where the Wi-Fi signal acquisition unit (150) acquires data in the form of CSI, but is not limited thereto.

[0204] In an embodiment, the Wi-Fi signal acquisition unit (150) can receive Wi-Fi CSI channel values. CSI is information indicating the channel frequency response characteristics of a multipath signal. CSI is a sampled value of CFR (Channel Frequency Response) and represents a frequency domain value of CIR (Channel Impulse Response). The Wi-Fi signal acquisition unit (150) can receive CIS raw data in real time.

[0205] Indoors, Wi-Fi signals are transmitted as multipath signals through reflection, scattering, and diffraction. Wi-Fi CSI includes the amplitude, phase, and delay of each multipath component. In an indoor environment, Wi-Fi signals transmitted to a receiver can be expressed in terms of the amplitude, phase, and delay of each multipath component.

[0206] In an embodiment, the processor (110) may preprocess the CIS raw data received through the Wi-Fi signal acquisition unit (150). For example, the processor (110) may exclude null channels from among Wi-Fi signals and group adjacent channels together to obtain an average value between multiple channels.

[0207] In an embodiment, the processor (110) may group subcarriers into adjacent channels, divide them into a predetermined number of groups, and obtain an average value of multiple channel signals included in each group. For example, the processor (110) may group 256 subcarriers into 7 channels, dividing them into a total of 36 groups.

[0208] In an embodiment, the processor (110) may obtain an average value of seven channel signals included in each group and obtain the average value for a predetermined period of time, for example, 25 seconds, to obtain a time series pattern trend of data. In an embodiment, the processor (110) may obtain second state information from the time series pattern trend of Wi-Fi data.

[0209] In an embodiment, the processor (110) may determine the final state of the user based on the first state information and the second state information.

[0210] FIG. 4 illustrates a Wi-Fi signal pattern acquired by the processor (110) of FIG. 3 based on the CSI channel value input through the Wi-Fi signal acquisition unit (150), according to an embodiment.

[0211] Reflection and diffraction of Wi-Fi signals affect the raw CSI values. Therefore, Wi-Fi signal patterns vary depending on the presence or absence of users, the number of users, their breathing, and their movements.

[0212] In Fig. 4, graphs designated by reference numerals 401 and 403 are graphs illustrating the amplitude of subcarriers over time. Reference numeral 401 illustrates the amplitude of a subcarrier when a user is stationary, and reference numeral 403 illustrates that the amplitude of a subcarrier changes depending on the movement of the user when the user stops moving and then starts moving after a certain point in time.

[0213] Depending on the user's movement, the degree of reflection / diffraction of the Wi-Fi signal changes, which affects the CSI raw value and changes the subcarrier value.

[0214] In an embodiment, the processor (110) can perform an ACF (Auto Correlation Function) analysis on a signal acquired through the Wi-Fi signal acquisition unit (150). The processor (110) can check the similarity between a signal transmitted from an AP and a signal received after being reflected from an object to confirm a signal change between two transmitters and receivers. If the change (diffraction) between the two signals is large at time points x1 and x2 over time, the ACF approaches the maximum value, and if the change in the signal is small, the ACF value approaches the minimum value.

[0215] In Fig. 4, graphs 405 and 407 are graphs showing ACF changes over time. Drawing code 405 represents ACF changes when a user is not in a space, and drawing code 407 represents ACF changes that vary depending on movement and stop when a user repeats movement and stop.

[0216] Looking at the graphs of drawings 405 and 407, we can see that the signal between the transmitter and receiver changes differently depending on the presence or absence of a user and whether the user is moving. Therefore, ACF can be used to identify the presence or absence of a user and whether the user is moving.

[0217] FIG. 5 is a graph showing the strength of an ACF signal according to the number of users obtained based on CSI channel values, according to an embodiment.

[0218] The more people in a confined space, such as a room, living room, or office, the more likely it is that Wi-Fi signals will be reflected or diffracted. These reflections and diffractions will alter the CSI raw values, resulting in variations in signal variability and signal strength between channels.

[0219] The graph illustrated in Fig. 5 illustrates that the ACF signal strength varies depending on the number of users. In the graph of Fig. 5, section a represents the ACF signal when there is one user, section b represents the ACF signal when there are three users, and section c represents the ACF signal when there are two users. As illustrated in the graph of Fig. 5, it can be seen that the maximum and minimum values ​​of the ACF signal strength are different in sections a, b, and c. That is, it can be seen that the minimum and maximum values ​​of the ACF signal strength are larger in the order of section b, section c, and section a. This may mean that the strength of the ACF signal increases as the number of users increases.

[0220] In an embodiment, the processor (110) may obtain status information from a Wi-Fi signal using a neural network.

[0221] In an embodiment, the neural network used by the processor (110) may be a deep learning model that has learned changes in Wi-Fi signal patterns that vary depending on the presence or absence of users, the number of users, user movement events, etc.

[0222] In an embodiment, the processor (110) inputs a currently acquired Wi-Fi signal or a pattern of a Wi-Fi signal into a neural network that has completed learning, and can obtain state information corresponding to the pattern of the Wi-Fi signal from the neural network.

[0223] Figure 6 is a graph showing changes in Wi-Fi signals according to the user's movement.

[0224] The reflection and diffraction of Wi-Fi signals can cause changes in signal patterns. For example, the movement of the chest when a user takes a deep breath can cause the Wi-Fi signal to have different patterns.

[0225] Referring to FIG. 6, it can be seen that when a user performs a specific action for a certain period of time, for example, when a user who was standing sits down and then stands up again, the Wi-Fi CSI raw data is acquired as a signal having a certain pattern.

[0226] In an embodiment, the processor (110) can use such a Wi-Fi signal pattern to identify what action the user is taking.

[0227] In an embodiment, the processor (110) can detect various events, such as whether the user is breathing in, breathing out, taking a deep breath, taking a light breath, sitting down, getting up and moving, sitting still, or taking a deep breath while still, using a Wi-Fi signal pattern.

[0228] Figure 7 is an internal block diagram of a processor (110) according to an embodiment.

[0229] The processor (110) of FIG. 7 may be an example of the processor (110) of FIG. 2 or FIG. 3.

[0230] Referring to FIG. 7, the processor (110) may include a distance recognition unit (710), a signal processing unit (720), and a mode change unit (730).

[0231] In an embodiment, a component included in the processor (110) may be a module. In an embodiment, a module may refer to a functional and structural combination of hardware for implementing the technical concepts of the present disclosure and software for operating the hardware. For example, a module may refer to a logical unit of a given code and hardware resources for executing the given code, and is not necessarily limited to physically connected code or a single type of hardware.

[0232] Because Wi-Fi signals can penetrate walls and obstacles and are highly sensitive, they can detect users outside the space where the electronic device (100) is located. In this case, the electronic device (100) may incorrectly change modes based on Wi-Fi signals influenced by users who are not in the same space as the electronic device (100).

[0233] In an embodiment, the electronic device (100) can identify the distance at which the acoustic signal is generated by using an acoustic signal to prevent such problems.

[0234] The distance recognition unit (710) according to the embodiment may be a module that identifies whether the acoustic distance is within a preset range.

[0235] The preset range may also be referred to as the limit distance range.

[0236] The preset range can be used to identify whether the distance from the electronic device (100) to the point of acoustic signal generation is within a certain range. The preset range can be stored in advance in the electronic device (100) as a default value.

[0237] Alternatively, in an embodiment, the electronic device (100) may set a range for each location.

[0238] In an embodiment, the electronic device (100) may output an interface screen that includes instructions for the user to utter a specific sound for a predetermined period of time to set a limit distance range. For example, the interface screen may include a text signal such as "Say 'Ah~' for 5 seconds."

[0239] In an embodiment, the electronic device (100) may output an audio guide signal that includes instructions to pronounce a specific sound.

[0240] When an interface screen or audio signal is displayed instructing the user to pronounce a specific sound for a predetermined period of time, the user can pronounce the specific sound for the predetermined period of time. The user can pronounce the specific sound for the predetermined period of time in a location where the electronic device (100) is primarily used. For example, if the user primarily watches the electronic device (100) from a sofa in the living room, the user can pronounce the specific sound while sitting on the sofa.

[0241] In an embodiment, the electronic device (100) can acquire a user's voice signal when the user utters a specific sound for a predetermined period of time. The electronic device (100) can collect the user's speech through a microphone array provided in the electronic device (100) and measure the distance from the user based on the collected speech.

[0242] If a control device that controls an electronic device (100) has a built-in microphone, the user can press a voice input button provided on the control device and then speak a specific sound. The microphone provided on the control device can receive the user's speech as an analog signal, digitize it, and transmit it to the electronic device (100).

[0243] In an embodiment, the electronic device (100) may measure the distance from the electronic device (100) to the user and then set a limit distance range by applying an error range to the distance to the user. For example, if the distance from the electronic device (100) to the user is 3 m and the error range is 20 cm, the electronic device (100) may set a range of 280 cm, which is the distance to the user minus the error range, to 320 cm, which is the distance to the user plus the error range, as the limit distance range.

[0244] In an embodiment, the electronic device (100) can store the limit distance range as a preset range and then determine whether the distance to the acoustic signal is within the preset range.

[0245] In an embodiment, the distance recognition unit (710) can obtain an acoustic distance when an acoustic signal is input and identify whether the acoustic distance is within a preset range.

[0246] In an embodiment, the distance recognition unit (710) can measure the distance from the location where the acoustic signal is generated to the electronic device (100).

[0247] In an embodiment, the distance recognition unit (710) can estimate the distance and direction from which an acoustic signal is generated by using the characteristics of the microphone array.

[0248] For example, the distance recognition unit (710) can obtain the phase difference of the sound signal that has reached a plurality of different microphones, and can obtain the sound distance between the microphones and the sound source by using the speed of sound and the time difference in which the sound signal has reached the plurality of microphones.

[0249] The distance recognition unit (710) can measure the acoustic distance from the location where the acoustic signal is generated to the microphone by using the time difference in which the acoustic signal generated at the same location reaches the multiple microphone arrays, the speed of sound, the distance between the multiple microphone arrays, i.e., the distance and angle between the microphone arrays, etc.

[0250] In an embodiment, the distance recognition unit (710) may also measure the distance between the user and the electronic device (100) using an ultrasonic sensor. For example, if an ultrasonic sensor is attached to a control device that controls the electronic device (100), the electronic device (100) may receive an ultrasonic signal from the control device and, based on the ultrasonic signal, obtain the distance between the control device and the electronic device (100).

[0251] In an embodiment, the distance recognition unit (710) may obtain the acoustic distance using a neural network that has learned acoustic data for each distance.

[0252] Hereinafter, referring to FIG. 8, the distance recognition unit (710) will be described in detail to obtain the acoustic distance using a neural network that has learned acoustic data for each distance.

[0253] FIG. 8 is a drawing explaining that the distance recognition unit (710) of FIG. 7 obtains an acoustic distance using a neural network that has learned acoustic data for each distance.

[0254] In an embodiment, the manufacturer can acquire the characteristics of sound input to multiple microphones when the same sound source is generated at different distances and angles, and use this as learning data to train a neural network.

[0255] As illustrated in FIG. 8, for example, it is assumed that two microphones (Mic 1, Mic 2) are installed in an electronic device (100) spaced apart by a predetermined distance. A manufacturer of the electronic device (100) can generate the same sound source at positions that are 1 meter, 2 meters, and 3 meters away from the center points of the two microphones installed in the electronic device (100), and at positions when the azimuth angles are 0 degrees, 30 degrees, and 60 degrees, respectively, from the center points of the two microphones.

[0256] A plurality of microphones provided in an electronic device (100) can capture acoustic signals. The acoustic signals acquired by the plurality of microphones may include background noise, etc. The manufacturer can remove noise from the acoustic signals and then obtain feature information in the form of a feature vector from the noise-removed acoustic signals.

[0257] Manufacturers can obtain at least one of various parameters obtained through digitization, frequency conversion, etc. of audio signals, such as Pitch, Formant, LPCC (Linear Predictive Cepstral Coefficient), MFCC (MelFrequency Cepstral Coefficient), PLP (Perceptual Linear Predictive), etc. as feature information. For example, manufacturers can obtain feature information from voice using the MFCC algorithm. The MFCC algorithm is a technique that divides an audio signal into small frames of about 20ms-40ms and analyzes the spectrum of the divided frames to extract features.

[0258] In an embodiment, the manufacturer may train a neural network using feature information obtained from an acoustic signal as learning data together with information about the location of the acoustic signal.

[0259] That is, in an embodiment, the manufacturer can convert the sound signal input by channel from the microphone when a specific sound source is generated by distance / location into a Mel-Spectrogram image, and use the characteristics of the sound source along with the distance or angle label as learning data to train a neural network.

[0260] In an embodiment, the neural network learns distance-related features, such as time delay or sensitivity, from signals in the frequency domain together with distance or angle labels, so as to learn how sound characteristics differ depending on distance and / or angle, and how the characteristics of sounds input to multiple microphones differ when sounds generated at the same location are input to multiple microphones.

[0261] In an embodiment, the neural network trained on distance-based acoustic data may be a CNN-based deep learning model. In an embodiment, the neural network trained on distance-based acoustic data may be a neural network trained to extract frequency-based information from each input spectrogram through convolution and to obtain acoustic distances from feature information obtained from acoustic signals converted to the frequency domain.

[0262] In an embodiment, the distance recognition unit (710) can infer the distance at which audio originated from an acoustic signal using a neural network.

[0263] Again, returning to FIG. 7, the distance recognition unit (710) can notify the signal processing unit (720) when the acoustic distance is within the limit distance range.

[0264] The signal processing unit (720) according to the embodiment may be a module that receives an audio signal and a Wi-Fi signal and obtains status information therefrom.

[0265] In an embodiment, when the signal processing unit (720) acquires an acoustic signal, it can identify whether the acoustic signal is an acoustic signal within a limit distance range. If the acoustic signal is an acoustic signal within the limit distance range, the signal processing unit (720) can acquire first state information from the acoustic signal.

[0266] In an embodiment, the signal processing unit (720) can obtain second state information from a Wi-Fi signal.

[0267] In an embodiment, the signal processing unit (720) can identify the user status based on the first status information and the second status information.

[0268] FIG. 9 is a table explaining that the signal processing unit (720) of FIG. 7 determines the state weight based on the first state information and the second state information.

[0269] Referring to FIG. 9, the signal processing unit (720) can obtain first state information using an acoustic signal pattern obtained from an acoustic signal. The first state information can indicate whether the user is moving, whether the user is breathing, whether the user is absent, etc.

[0270] The signal processing unit (720) can obtain second status information using a Wi-Fi signal pattern obtained from a Wi-Fi signal. The second status information can also indicate whether the user is moving, whether the user is breathing, or whether the user is absent.

[0271] In an embodiment, the signal processing unit (720) may determine the user's state weight by using the first state information and the second state information together.

[0272] In an embodiment, the state weight may be a weight or importance assigned to each individual state to determine the user's final state. In an embodiment, the individual states may include a user absent state, a user sleeping state, a user awake state, and the like.

[0273] In an embodiment, the signal processing unit (720) may store a table or matrix including first state information and second state information in rows and columns, as shown in the table of FIG. 9. Each component of the matrix may be a state weight determined by considering the first state information and the second state information together.

[0274] Referring to the table in Fig. 9, when the first state information is a movement state (Move), weight can be given to the awake state (Awake) except when the second state information is a user absence state (None).

[0275] Wi-Fi signals have distinct signal patterns that differ depending on the presence or absence of a user. Therefore, even if the first state information obtained through the audio signal indicates a motion state, if the second state information obtained through the Wi-Fi signal indicates a user absence state, weighting can be given to the user absence state.

[0276] Since the Wi-Fi signal has the potential to detect a user outside the limit distance range, even if the second state information is a motion state, if the first state information based on the acoustic signal acquired within the limit distance range is a motionless state, the user absence state may be given more weight.

[0277] For the same reason, if the first state information is a user absence state, weighting may be given to the user absence state regardless of the second state information.

[0278] If the second state information is a moving state and the first state information is a breathing state, weight may be given to the state in which the user is awake.

[0279] If both the first state information and the second state information are the same for a user, but the user's movement is not detected or breathing is detected, the sleep state may be weighted.

[0280] In an embodiment, the signal processing unit (720) may determine state weights corresponding to first state information and second state information obtained from the current Wi-Fi signal and audio signal using a pre-stored matrix in the form of FIG. 9.

[0281] In an embodiment, the signal processing unit (720) may receive first state information and second state information for a predetermined period of time and determine a state weight corresponding thereto.

[0282] The signal processing unit (720) can analyze and classify the state weight accumulated over a predetermined period of time to extract the time-series characteristics of the state weight and obtain the final state therefrom.

[0283] In an embodiment, the signal processing unit (720) may utilize a neural network to obtain the final state. For example, the signal processing unit (720) may utilize a Long Short-Term Memory (LSTM) model. LSTM is a type of recurrent neural network (RNN) that can learn long-term dependencies between time steps of sequence data. LSTM receives sequence or time series data as input and can learn long-term dependencies between time steps of the sequence data.

[0284] The signal processing unit (720) can input accumulated data into a neural network and obtain a final state from it. In an embodiment, the final state may be one of a user absent state, a user sleeping state, and a user awake state.

[0285] In an embodiment, the signal processing unit (720) can obtain the user's final state and transmit it to the mode change unit (730).

[0286] Again, returning to FIG. 7, when the signal processing unit (720) transmits the user final state to the mode change unit (730), the mode change unit (730) according to the embodiment can perform an operation of changing the mode based on the user final state. The mode change unit (730) may also be referred to as a state change unit.

[0287] In an embodiment, the electronic device (100) may operate in a sleep mode or a normal mode depending on whether a component included in the electronic device (100) is operating.

[0288] Normal mode may refer to a state in which power is supplied to all components of the electronic device (100). In normal mode, each component of the electronic device (100) operates normally, so the user can normally use the electronic device (100) according to its function, such as watching video content and audio content using the electronic device (100).

[0289] Sleep mode is a low-power mode that may include standby mode.

[0290] In an embodiment, the sleep mode may be a power-saving mode for the electronic device (100). In the sleep mode, power may be supplied only to components that acquire and process Wi-Fi signals or audio signals, while power may be cut off to other components. Since the network function operates even when the electronic device (100) is in sleep mode, the communication unit can acquire Wi-Fi signals. In addition, the electronic device (100) can acquire and process audio signals through the microphone array even when in sleep mode.

[0291] In an embodiment, the mode change unit (730) may operate in sleep mode or normal mode depending on the final state.

[0292] In an embodiment, the mode change unit (730) may continue to operate in normal mode when the user's final state is awake.

[0293] In an embodiment, the mode change unit (730) can change from normal mode to sleep mode when the final state is a sleep state or a user absence state. In an embodiment, the mode change unit (730) can control the display and speaker to turn off so that the display and speaker do not output content.

[0294] In an embodiment, the mode change unit (730) may operate by changing from normal mode to sleep mode corresponding to the user's final state being a sleep state, and when it detects that the user has woken up, it may operate by changing the mode from sleep mode back to normal mode.

[0295] In an embodiment, the mode change unit (730) may operate by changing from normal mode to sleep mode corresponding to the user's final state being a sleep state, and when it detects that the user has woken up, it may operate by changing the mode from sleep mode to music mode or meditation mode.

[0296] For example, meditation mode or music mode may output a low-volume music signal suitable for a user who has just woken up. Meditation mode or music mode may output audio content only, without video content, or may output video content along with music.

[0297] However, this is only one example, and the mode change unit (730) can operate in various modes corresponding to the final state.

[0298] FIG. 10 is a block diagram according to an example of the signal processing unit (720) of FIG. 7.

[0299] Referring to FIG. 10, the signal processing unit (720) may include a multiple user recognition unit (1010), a signal separation unit (1020), and a signal analysis unit (1030) in a module form.

[0300] The multiple user recognition unit (1010) according to the embodiment may be a module that identifies whether there are multiple users.

[0301] In an embodiment, the multiple user recognition unit (1010) can detect that there are multiple users based on at least one of an acoustic signal and a Wi-Fi signal.

[0302] As mentioned above, the more users there are in a Wi-Fi network, the stronger the signal strength tends to be.

[0303] In an embodiment, the multi-user recognition unit (1010) can detect multiple users based on at least one of the fluctuation range and signal strength of the Wi-Fi signal. In an embodiment, the multi-user recognition unit (1010) can identify the number of users by using the fluctuation range of the Wi-Fi signal, i.e., the maximum and minimum strength of the Wi-Fi signal.

[0304] In an embodiment, the multiple user recognition unit (1010) may identify multiple users using an acoustic signal. For example, the multiple user recognition unit (1010) may infer that multiple users are present based on the detection of at least one of a breathing event and a speech / voice event of multiple users in the acoustic signal.

[0305] In an embodiment, the multiple user recognition unit (1010) can infer the number of users from an audio signal that is a mixture of one person's breathing sound or two or three people's breathing sounds by using a CNN-based neural network that has learned the sound in advance.

[0306] In an embodiment, the multiple user recognition unit (1010) can detect voice activity (Voice Activity Detection, VAD) in an audio signal, segment the voice segment, and extract the speaker's features for each segment.

[0307] In an embodiment, the multiple user recognition unit (1010) can adaptively learn a Universal Background Model (UBM) for each speech section and then treat it as a distribution of the speaker's speech characteristics.

[0308] Alternatively, in an embodiment, the multi-user recognition unit (1010) may extract speaker feature vectors based on a deep learning model. The deep learning model used by the multi-user recognition unit (1010) may be trained by defining a distance-based objective function that induces speaker vectors to be close to each other in the case of the same speaker, and to be farther from each other in the case of different speakers. The multi-user recognition unit (1010) may classify speaker feature vectors based on distance.

[0309] FIG. 11 is a diagram illustrating, according to an embodiment, separation of a voice signal into user-specific voice signals.

[0310] The multiple user recognition unit (1010) of FIG. 10 can identify user speech segments in an audio signal and extract voice features from the user speech segments. The multiple user recognition unit (1010) can identify two users by separating signals with similar characteristics, as illustrated in FIG. 11.

[0311] Again, returning to FIG. 10, the multiple user recognition unit (1010) can notify the signal separation unit (1020) if it determines that there are multiple users based on the Wi-Fi signal and / or the audio signal.

[0312] When the signal separation unit (1020) according to the embodiment receives information that there are multiple users from the multiple user recognition unit (1010), it can separate each of the audio signal and the Wi-Fi signal into signals for each user.

[0313] In an embodiment, the signal separation unit (1020) can collect voice signals through multiple microphones. For example, if four microphones are arranged in a row on the front of the electronic device (100) and there are two users around the electronic device (100), in an embodiment, the signal separation unit (1020) can collect voice signals of multiple users using the multiple microphones.

[0314] In an embodiment, the signal separation unit (1020) can identify which user's voice is loudly input into each microphone for each microphone position. In the above example, the sound signal generated from the right user located on the right side of the microphone array is contained largely in the two right microphones among the four microphones, and is contained exponentially less in the remaining two left microphones.

[0315] In an embodiment, the signal separation unit (1020) can separate an audio signal corresponding to a user for each user according to the structural characteristics of the microphone array and the pattern of the audio signal.

[0316] In an embodiment, the signal separation unit (1020) can group channels that exhibit similar characteristics according to the user's location and the characteristics of the audio signal from the user.

[0317] In an embodiment, the signal separation unit (1020) can obtain audio features for each group by amplifying the collected frequencies for each group and then filtering and correcting the frequencies of other groups.

[0318] For example, in the above example, the signal separation unit (1020) can separate the first acoustic signal generated from the first user on the right from the second acoustic signal generated from the second user by amplifying the channel signals acquired by the two microphones on the right that contain many acoustic signal characteristics generated from the right user and removing the remaining signals acquired by the two microphones on the left from the amplified channel signals.

[0319] Similarly, the signal separation unit (1020) can separate a second audio signal generated from a second user on the left from other audio signals by amplifying channel signals acquired by the two microphones on the left among the four microphones and filtering signals acquired by the two microphones on the right.

[0320] In an embodiment, the signal separation unit (1020) can synchronize the WiFi signal acquired simultaneously with the audio information, i.e., the CSI Raw data, with the audio signal.

[0321] In an embodiment, the signal separation unit (1020) can synchronize the audio signal and the CSI Raw data by using the interval time between the two signals representing the input time difference between the audio signal and the Wi-Fi signal.

[0322] In an embodiment, the signal separation unit (1020) can identify a Wi-Fi signal having characteristics similar to an acoustic signal using a neural network.

[0323] In an embodiment, the signal separation unit (1020) can identify a first Wi-Fi channel corresponding to a first acoustic signal and a second Wi-Fi channel corresponding to a second acoustic signal.

[0324] In an embodiment, the signal separation unit (1020) can separate audio signals and Wi-Fi channels for each user, and transmit the separated audio signals and Wi-Fi signals for each user to the signal analysis unit (1030).

[0325] In an embodiment, the signal analysis unit (1030) can receive a signal separated for each user from the signal separation unit (1020) and obtain status information for each user based on the signal.

[0326] In an embodiment, the signal analysis unit (1030) may obtain first status information about the first user from the first acoustic signal, and obtain second status information about the first user from the first Wi-Fi channel corresponding to the first acoustic signal. In an embodiment, the signal analysis unit (1030) may determine a status weight for the first user based on the first status information and the second status information.

[0327] Similarly, the signal analysis unit (1030) may obtain first state information about the second user from the second acoustic signal, and second state information about the second user from the second Wi-Fi channel corresponding to the second acoustic signal. In an embodiment, the signal analysis unit (1030) may determine a state weight for the second user based on the first state information and the second state information.

[0328] FIG. 12 is a diagram illustrating, according to an embodiment, the signal separation unit (1020) of FIG. 10 using a neural network to identify a WiFi channel corresponding to an acoustic signal.

[0329] In an embodiment, the signal separation unit (1020) may utilize artificial intelligence (AI) technology.

[0330] For a neural network to accurately output output data corresponding to its input data, it must be trained. Here, "training" can mean inputting various data into the neural network and training the neural network to discover or learn methods for analyzing the input data, classifying the input data, and / or extracting features necessary for generating output data from the input data.

[0331] Training a neural network means applying a learning algorithm to a large number of training data sets to create an artificial intelligence model with desired characteristics. In some embodiments, this learning may be performed within the electronic device (200) itself, where the artificial intelligence is implemented, or through a separate server / system.

[0332] In an embodiment, the neural network utilized by the signal separation unit (1020) may be a neural network that has learned the characteristics of acoustic signals corresponding to various acoustic events, such as speech characteristics, movement-related sounds, and breathing sounds. Furthermore, the neural network may be a neural network that has learned Wi-Fi signal patterns occurring simultaneously with the corresponding acoustic signals as training data, along with the acoustic signal characteristics.

[0333] In an embodiment, the neural network can learn by using both audio signal patterns and Wi-Fi signal patterns corresponding to the same event as training data.

[0334] For example, a neural network can learn the acoustic signal patterns and Wi-Fi signal patterns of a user conversing as a single set of training data. It can also learn the acoustic signal patterns and Wi-Fi signal patterns of a user sitting down and standing up as a single set. It can also learn the acoustic signal, including breathing sounds, of a user sleeping and the Wi-Fi signal patterns detected at the same time as chest movement as a single set.

[0335] In an embodiment, the neural network can learn the relation weight between the acoustic signal and the Wi-Fi signal by using the acoustic signal pattern and the Wi-Fi signal pattern corresponding to the same event together as learning data, acquiring the features of each of the acoustic signal pattern and the Wi-Fi signal pattern, and learning the relationship between the acquired features.

[0336] In an embodiment, the neural network used by the signal separation unit (1020) may be a neural network trained to obtain a Wi-Fi signal having pattern characteristics similar to an acoustic signal from an acoustic signal.

[0337] In an embodiment, the signal separation unit (1020) inputs the characteristics of a specific sound into a trained neural network, and can infer a Wi-Fi signal pattern highly related to the characteristics of the specific sound from the input.

[0338] In an embodiment, the signal separation unit (1020) can identify a channel having a similar pattern to a WiFi signal pattern inferred using a neural network among a plurality of WiFi channels, and select the identified channel or, if there are multiple identified channels, group the multiple channels.

[0339] Hereinafter, referring to FIG. 12, the signal separation unit (1020) will be described in detail to identify a Wi-Fi channel corresponding to each user using a neural network.

[0340] Referring to FIG. 12, the signal separation unit (1020) can separate an audio signal corresponding to each user according to the structural characteristics of the microphone array and the pattern of the audio signal.

[0341] For example, as shown in FIG. 12, when the microphone array of the electronic device (100) includes four microphones, the signal separation unit (1020) can collect audio signals for each channel of the four microphones. The signal separation unit (1020) can collect audio signals generated from the speech or breathing characteristics of user 1, or the movements of user 1, from the audio signals collected by the first microphone among the four microphones. Similarly, the signal separation unit (1020) can collect audio signals of user 2 and user 3, respectively, by using the second microphone and the third microphone.

[0342] In an embodiment, the signal separation unit (1020) can remove noise such as audio content and background noise output through the electronic device (100) from the user-specific audio signal acquired by each microphone. The signal separation unit (1020) can convert the user-specific audio signal from which the audio content and noise have been removed into a mel spectrogram image (1211, 1212, 1213) in the frequency domain.

[0343] The signal separation unit (1020) can obtain sound frequency characteristics such as pitch, timbre, and intensity as an acoustic signal pattern from a mel spectrogram image (1211).

[0344] In an embodiment, the signal separation unit (1020) can input sound frequency characteristics for each user into a neural network.

[0345] In an embodiment, the signal separation unit (1020) may obtain a Wi-Fi signal synchronized with an audio signal. In an embodiment, the signal separation unit (1020) may receive a Wi-Fi signal synchronized with an audio signal as CIS raw data, preprocess the same, exclude a null channel, and group adjacent channels to obtain an average value among multiple channels. For example, the signal separation unit (1020) may group adjacent channels such as channel groups 0 to 7, channels 8 to 14, channels 15 to 21, and channels 22 to 28.

[0346] In an embodiment, the signal separation unit (1020) can obtain CSI amplitude values ​​for each of multiple channels. The signal separation unit (1020) can use the CSI amplitude values ​​to generate a spectrogram image (1212) of a predetermined size and process it into a data format that can be input into a neural network.

[0347] In an embodiment, the signal separation unit (1020) can input a WiFi feature vector obtained from a spectrogram image (1212) into a neural network.

[0348] In an embodiment, a neural network that has completed learning can obtain a WiFi signal pattern highly related to each user's audio signal pattern based on the correlation between the audio signal pattern and the WiFi signal pattern.

[0349] For example, as illustrated in FIG. 12, the neural network can identify a group of channels 0 to 7 as Wi-Fi channels highly related to the audio signal of user 1, and a group of channels 8 to 14 as Wi-Fi channels highly related to the audio signal of user 2.

[0350] In an embodiment, the signal separation unit (1020) can obtain first state information corresponding to user 1 from the audio signal of user 1, and can also obtain second state information corresponding to user 1 from the Wi-Fi signal included in the channel group from number 0 to number 7.

[0351] Similarly, the signal separation unit (1020) can obtain second state information corresponding to user 2 from the audio signal of user 2, and can also obtain second state information corresponding to user 2 from the Wi-Fi signal included in the channel group from number 8 to number 14.

[0352] In this way, according to an embodiment, the electronic device (100) can obtain a Wi-Fi signal corresponding to an audio signal of each of multiple users by using a neural network that has learned the relationship between an audio signal pattern and a Wi-Fi signal pattern corresponding to the same event.

[0353] Fig. 13 is an internal block diagram of an electronic device according to an embodiment.

[0354] The electronic device (1300) of FIG. 13 may be an example of the electronic device (100) of FIG. 2. Hereinafter, any description overlapping with that described in FIG. 2 will be omitted.

[0355] Referring to FIG. 13, the electronic device (1300) may further include, in addition to the processor (110) and the memory (120), a tuner unit (1310), a communication unit (1320), a detection unit (1330), an input / output unit (1340), a video processing unit (1350), a display unit (1360), an audio processing unit (1370), an audio output unit (1380), and a user input unit (1390).

[0356] The tuner unit (1310) can select and tune only the frequency of the channel to be received by the electronic device (1300) among many radio wave components through amplification, mixing, resonance, etc. of broadcast content received via wire or wireless. The content received through the tuner unit (1310) is decoded and separated into audio, video, and / or additional information. The separated audio, video, and / or additional information can be stored in the memory (120) under the control of the processor (110).

[0357] The communication unit (1320) can connect the electronic device (1300) to peripheral devices, external devices, servers, mobile terminals, etc. under the control of the processor (110). The communication unit (1320) can include at least one communication module capable of performing wireless communication. The communication unit (1320) can include at least one of a wireless LAN module (1321), a Bluetooth module (1322), and a wired Ethernet (1323) depending on the performance and structure of the electronic device (1300).

[0358] The wireless LAN module (1321) can transmit and receive Wi-Fi signals with peripheral devices according to the Wi-Fi communication standard.

[0359] The Bluetooth module (1322) can receive Bluetooth signals transmitted from peripheral devices according to the Bluetooth communication standard. The Bluetooth module (1322) can be a BLE (Bluetooth Low Energy) communication module and can receive BLE signals. The Bluetooth module (1322) can continuously or temporarily scan BLE signals to detect whether a BLE signal is received.

[0360] The detection unit (1330) detects the user's voice, the user's image, or the user's interaction, and may include a microphone (1331), a camera unit (1332), a light receiving unit (1333), and a sensing unit (1334). The microphone (1331) may receive an audio signal including the user's uttered voice or noise, and may convert the received audio signal into an electrical signal and output it to the processor (110).

[0361] The camera unit (1332) includes a sensor (not shown) and a lens (not shown), and can capture an image formed on the screen and transmit it to the processor (110).

[0362] The optical receiver (1333) can receive an optical signal (including a control signal). The optical receiver (1333) can receive an optical signal corresponding to a user input (e.g., touch, press, touch gesture, voice, or motion) from a control device such as a remote control or a mobile phone.

[0363] The sensing unit (1334) can detect the state surrounding the electronic device and transmit the detected information to the communication unit (1320) or the processor (110).

[0364] The input / output unit (1340) can receive video (e.g., a moving image signal or a still image signal), audio (e.g., a voice signal or a music signal), and additional information from a device external to the electronic device (1300) under the control of the processor (110).

[0365] The input / output unit (1340) may include one of an HDMI port (High-Definition Multimedia Interface port, 1341), a component jack (component jack, 1342), a PC port (PC port, 1343), and a USB port (USB port, 1344). The input / output unit (1340) may include a combination of an HDMI port (1341), a component jack (1342), a PC port (1343), and a USB port (1344).

[0366] The video processing unit (1350) processes image data to be displayed by the display unit (1360) and can perform various image processing operations such as decoding, rendering, scaling, noise filtering, frame rate conversion, and resolution conversion for the image data.

[0367] The display unit (1360) can display content received from a broadcasting station, an external server, an external storage medium, etc., on the screen. The content is a media signal and may include a video signal, an image, a text signal, etc.

[0368] When the display unit (1360) is implemented as a touch screen, the display unit (1360) can be used as an input device such as a user interface in addition to an output device. For example, the display unit (1360) can include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display. In addition, depending on the implementation form of the display unit (1360), two or more display units (1360) can be included.

[0369] The audio processing unit (1370) performs processing on audio data. The audio processing unit (1370) can perform various processing such as decoding, amplification, and noise filtering on audio data.

[0370] The audio output unit (1380) can output audio included in content received through the tuner unit (1310) under the control of the processor (110), audio input through the communication unit (1320) or input / output unit (1340), and audio stored in the memory (120). The audio output unit (1380) can include at least one of a speaker (1381), a headphone (1382), or an S / PDIF (Sony / Philips Digital Interface: output terminal) (1383).

[0371] In an embodiment, the audio output unit (1380) can output audio content while the electronic device (1300) operates in normal mode.

[0372] The user input unit (1390) can receive user input for controlling the electronic device (1300). The user input unit (1390) can include various types of user input devices, including, but not limited to, a touch panel that detects a user's touch, a button that receives a user's push operation, a wheel that receives a user's rotation operation, a keyboard, a dome switch, a microphone for voice recognition, a motion detection sensor that senses motion, etc. When a remote control or other mobile terminal controls the electronic device (1300), the user input unit (1390) can receive a control signal received from the mobile terminal.

[0373] Fig. 14 is a flowchart illustrating an operation method of an electronic device according to an embodiment.

[0374] Referring to FIG. 14, the electronic device (100) can obtain first state information from an acoustic signal (step 1410).

[0375] In an embodiment, the electronic device (100) can filter audio content output by the electronic device (100) and other noise from an acoustic signal, and convert the filtered acoustic signal into a Mel-spectrogram image. The electronic device (100) can obtain frequency characteristics of the sound from the Mel-spectrogram image, and obtain first state information from the frequency characteristics of the sound.

[0376] In an embodiment, the electronic device (100) can obtain second state information from a Wi-Fi signal (step 1420).

[0377] In an embodiment, the electronic device (100) can obtain a Wi-Fi signal pattern from a Wi-Fi signal and obtain second state information using the Wi-Fi pattern.

[0378] In an embodiment, the electronic device (100) can identify a user status based on the first status information and the second status information (step 1430).

[0379] In an embodiment, the electronic device (100) may operate in a mode corresponding to a user state (step 1440).

[0380] Fig. 15 is a flowchart illustrating an operation method of an electronic device according to an embodiment.

[0381] Referring to FIG. 15, the electronic device (100) can detect multiple users (step 1510).

[0382] In an embodiment, the electronic device (100) can detect multiple users based on at least one of an acoustic signal and a Wi-Fi signal.

[0383] In an embodiment, the electronic device (100) may separate user-specific audio signals corresponding to the detection of multiple users (step 1520).

[0384] In an embodiment, the electronic device (100) can separate Wi-Fi channels for each user (step 1530).

[0385] In an embodiment, the electronic device (100) can identify a Wi-Fi channel corresponding to an audio signal of a given user by using a neural network that has learned the relationship between an audio signal and a Wi-Fi signal.

[0386] In an embodiment, the electronic device (100) can obtain status information based on user-specific audio signals and Wi-Fi channels (step 1540).

[0387] For example, the electronic device (100) may obtain first state information from an audio signal corresponding to a first user, obtain second state information from a Wi-Fi channel corresponding to the first user, and determine the final state of the first user using the first state information and the second state information.

[0388] The method of operating an electronic device according to some embodiments and the electronic device may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism, and includes any information delivery media.

[0389] In addition, the electronic device and the operating method thereof according to the embodiment of the present disclosure described above may be implemented as a computer program product including a computer-readable recording medium / storage medium having recorded thereon a program for implementing an operating method of the electronic device, the method including a step of obtaining first state information using an acoustic signal pattern obtained from an acoustic signal, a step of obtaining second state information (user presence or absence, user action) using a Wi-Fi signal pattern obtained from a Wi-Fi signal, a step of identifying a user state based on the first state information and the second state information, and a step of operating in a mode corresponding to the user state.

[0390] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0391] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In an electronic device (100), A memory (120) storing at least one instruction; and At least one processor (110) for executing at least one instruction stored in the memory, At least one of the above processors Identify the user's final state based on the first state information acquired according to the acoustic signal pattern acquired from the acoustic signal and the second state information acquired according to the WiFi signal pattern acquired from the WiFi signal. An electronic device that controls operation to a state corresponding to the user's final state.

2. In the first paragraph, at least one processor An electronic device that controls operation to the corresponding state when the distance to the point of occurrence of the above-mentioned acoustic signal is within a preset range.

3. In the first or second paragraph, an output unit (130) including a speaker is further included, At least one of the above processors A Mel-spectrogram image is obtained based on an acoustic signal output through the speaker from which audio content and noise are filtered, Obtaining the first state information from the frequency characteristics of the sound obtained based on the above-mentioned Mel Spectrogram image, A method of operating an electronic device, wherein the frequency characteristics of the sound include at least one of pitch, timbre, and intensity.

4. In any one of clauses 1 to 3, at least one processor Obtaining a WiFi signal pattern from the above WiFi signal, An electronic device that obtains the second state information from the pattern trend of the Wi-Fi signal for a preset period of time.

5. In any one of paragraphs 1 to 4, at least one processor An electronic device that identifies a corresponding signal for each user from the acoustic signal and the Wi-Fi signal when multiple users are identified based on at least one of the acoustic signal and the Wi-Fi signal.

6. In the fifth paragraph, at least one processor An electronic device that identifies the plurality of users based on at least one of a fluctuation range and a signal strength of the Wi-Fi signal and identifies the plurality of users based on at least one of a voice event of the plurality of users and a breathing event of the plurality of users in the acoustic signal.

7. In the fifth or sixth paragraph, at least one processor Obtaining a first acoustic signal including the speech characteristics of the first user from the acoustic signals acquired using multiple microphones, An electronic device that obtains first status information corresponding to the first user from the first acquired acoustic signal.

8. In the 7th paragraph, at least one processor Using a neural network that has learned the relationship between an acoustic signal and a Wi-Fi signal, a first Wi-Fi channel corresponding to the first acoustic signal is identified, An electronic device that obtains second state information corresponding to the first user from the first WiFi channel.

9. In the 8th paragraph, at least one processor Using the first state information and the second state information corresponding to the first user, the state weight of the first user is identified, An electronic device that identifies a final state of the first user based on the state weight of the first user acquired over a certain period of time.

10. In any one of paragraphs 1 to 9, at least one processor An electronic device that controls operation into a sleep state when the final state of all detected users is identified as a sleep state.

11. A step of identifying a user final state based on first state information acquired according to an acoustic signal pattern acquired from an acoustic signal and second state information acquired according to a WiFi signal pattern acquired from a WiFi signal; and A method of operating an electronic device, comprising the step of operating in a state corresponding to the user final state.

12. An operating method of an electronic device, comprising a step of operating in the corresponding state if the distance to the point of occurrence of the acoustic signal in the 11th paragraph is within a preset range.

13. In clause 11 or 12, further comprising a step of outputting audio content, The step of obtaining the above first state information is A step of obtaining a Mel-spectrogram image based on the audio content and noise-filtered acoustic signal; and A step of obtaining the first state information from the frequency characteristics of the sound obtained based on the above-mentioned spectrogram image is included. A method of operating an electronic device, wherein the frequency characteristics of the sound include at least one of pitch, timbre, and intensity.

14. In any one of clauses 11 to 13, the step of obtaining the second state information A step of obtaining a WiFi signal pattern from the above WiFi signal; and An operating method of an electronic device, comprising a step of obtaining the second state information from a pattern trend of the Wi-Fi signal for a preset period of time.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of claims 11 to 14 on a computer.

Images