Electronic device and method for controlling electronic device

Abstract

Provided is an electronic device. The electronic device comprises: a microphone; a memory for storing at least one computer program; and at least one processor communicatively connected to the microphone and the memory. The at least one computer program includes instructions executable by a computer. The instructions, when individually or collectively executed by the at least one processor, enable the electronic device to: store, in the memory, pieces of registration information about respiratory sounds of a plurality of users; acquire pieces of information about the respiratory sounds of the users on the basis of audio signals when receiving the audio signals via the microphone; identify one or more users corresponding to pieces of information about respiratory sounds among the plurality of users by comparing the pieces of information about the respiratory sounds with the pieces of registration information; and, when the at least one user is identified, acquire an analysis result for a sleep state of each of the one or more users on the basis of the information corresponding to each of the one or more users among the pieces of information about the respiratory sounds.

Description

Electronic devices and methods of controlling electronic devices

[0001] The present disclosure relates to electronic devices and methods for controlling such devices. More specifically, the present disclosure relates to providing an electronic device capable of analyzing a user's breathing sounds and a method for controlling the same.

[0002] Recently, with the advancement of fields such as wearable devices, artificial intelligence, and the Internet of Things (IoT), the development of technologies that enable users to effectively manage their health by analyzing their sleep status is accelerating.

[0003] In particular, the technology that analyzes the user's sleep state by analyzing the breathing sound generated by the user's breathing during sleep has the advantages of being able to provide information on the user's sleep state without disturbing the user's sleep in a non-contact manner and not requiring expensive additional sensors.

[0004] However, it has been pointed out that these conventional technologies have limitations in that when multiple users are sleeping together in the same space, it is difficult to clearly distinguish the breathing sounds of multiple users, and therefore it is difficult to accurately analyze the sleeping states of each of the multiple users.

[0005] The above information is provided solely as background information to aid understanding of the present disclosure. No determination has been made, and no claim is made, regarding whether any of the above information constitutes prior art in connection with the present disclosure.

[0006] Aspects of the present disclosure are intended to address at least the problems and / or disadvantages described above and to provide at least the advantages described below.

[0007] Accordingly, one aspect of the present disclosure is to provide an electronic device and a control method thereof capable of clearly distinguishing breathing sounds of multiple users and accurately analyzing the sleep states of each of the multiple users.

[0008] Additional aspects will be set forth in part in the description that follows, and in part will be obvious from the description or may be learned by practice of the embodiments presented. According to one aspect of the present disclosure, an electronic device is disclosed. The electronic device includes a microphone, a memory storing one or more computer programs, and one or more processors in communication with the microphone and the memory, wherein the one or more computer programs include instructions executable by a computer, which instructions, when individually or collectively executed by the one or more processors, cause the electronic device to store registration information on breathing sounds of a plurality of users in the memory, and when an audio signal is received through the microphone, to obtain information on breathing sounds of the users based on the audio signal, to compare the information on breathing sounds with the registration information, to identify at least one user corresponding to the information on breathing sounds among the plurality of users, and when the at least one user is identified, to obtain an analysis result on a sleep state of each of the at least one user based on information corresponding to each of the at least one user among the information on breathing sounds.

[0009] Meanwhile, when the audio signal is received, the one or more processors identify a plurality of segments corresponding to the user's breathing sounds in the audio signal, obtain a plurality of first embedding vectors corresponding to each of the plurality of segments, and identify the at least one user based on comparing each of the plurality of first embedding vectors with a plurality of second embedding vectors corresponding to the registration information.

[0010] Meanwhile, when the plurality of first embedding vectors are obtained, the one or more processors identify distances between the positions of each of the plurality of first embedding vectors in the latent space and the center positions of the embedding vectors corresponding to the first user among the plurality of second embedding vectors, and if the identified distances are less than a preset threshold distance, identify the first user as the at least one user.

[0011] Meanwhile, the one or more processors input the audio signal into a first neural network model trained to distinguish the user's breathing sounds included in the audio signal to obtain information about the plurality of segments, input the plurality of segments into a second neural network model trained to convert the input segments into embedding vectors to obtain the plurality of first embedding vectors, and input information corresponding to each of the at least one user into a third neural network model trained to identify the user's sleep state corresponding to the breathing sounds to obtain the analysis result.

[0012] Meanwhile, the analysis result may include at least one of information on whether the at least one user is sleeping, information on the quality of the at least one user's sleep, and information on the health of the at least one user.

[0013] Meanwhile, the electronic device further includes a communication unit, and the one or more processors control the communication unit to obtain a control signal for controlling an external device and transmit the control signal to the external device when the analysis result indicates that the at least one user is sleeping.

[0014] Meanwhile, the one or more processors update the registration information based on the information about the breathing sound when the at least one user is identified.

[0015] Meanwhile, the electronic device further includes a display, and the one or more processors control the display to display a user interface when the at least one user is not identified, and when a user input for registering information about the breathing sound is received through the user interface, the electronic device adds information about the breathing sound to the registration information.

[0016] Meanwhile, the electronic device further includes a sensor, and the one or more processors, when the at least one user is identified, obtain biometric information about the at least one user through the sensor, and obtain an analysis result about the sleep state of each of the at least one user based on the biometric information and information corresponding to each of the at least one user.

[0017] According to another aspect of the present disclosure, a method performed by an electronic device is disclosed. The method includes the steps of: storing registration information on breathing sounds of a plurality of users; when an audio signal is received, obtaining information on breathing sounds of the users based on the audio signal; comparing the information on breathing sounds with the registration information on breathing sounds of the plurality of users to identify at least one user corresponding to the information on breathing sounds among the plurality of users; and, when the at least one user is identified, obtaining an analysis result on a sleep state of each of the at least one user based on information corresponding to each of the at least one user among the information on breathing sounds.

[0018] Meanwhile, the step of obtaining information about the breathing sound includes, when the audio signal is received, the step of identifying a plurality of segments corresponding to the user's breathing sound in the audio signal and the step of obtaining a plurality of first embedding vectors corresponding to each of the plurality of segments, and the step of identifying the at least one user includes the step of identifying the at least one user based on comparing each of the plurality of first embedding vectors with a plurality of second embedding vectors corresponding to the registration information.

[0019] Meanwhile, the step of identifying the at least one user includes, when the plurality of first embedding vectors are obtained, the step of identifying distances between the positions of each of the plurality of first embedding vectors in the latent space and the center positions of the embedding vectors corresponding to the first user among the plurality of second embedding vectors, and the step of identifying the first user as the at least one user if the identified distances are less than a preset threshold distance.

[0020] Meanwhile, the step of obtaining information on the breathing sound includes a step of obtaining information on the plurality of segments by inputting the audio signal into a first neural network model trained to distinguish the breathing sound of the user included in the audio signal, and a step of obtaining the plurality of first embedding vectors by inputting the plurality of segments into a second neural network model trained to convert the input segments into embedding vectors, and the step of obtaining the analysis result includes a step of obtaining the analysis result by inputting information corresponding to each of the at least one user into a third neural network model trained to identify the sleep state of the user corresponding to the breathing sound.

[0021] Meanwhile, the analysis result may include at least one of information on whether the at least one user is sleeping, information on the quality of the at least one user's sleep, and information on the health of the at least one user.

[0022] Meanwhile, the method further includes a step of obtaining a control signal for controlling an external device and a step of transmitting the control signal to the external device if the analysis result indicates that at least one user is sleeping.

[0023] Meanwhile, the method further includes a step of updating the registration information based on information about the breathing sound when at least one user is identified.

[0024] Meanwhile, the method further includes a step of displaying a user interface if at least one user is not identified, and a step of adding information about the breath sound to the registration information if a user input for registering information about the breath sound is received through the user interface.

[0025] Meanwhile, the method further includes a step of obtaining biometric information about the at least one user when the at least one user is identified, and a step of obtaining an analysis result about the sleep state of each of the at least one user based on information corresponding to each of the at least one user and the biometric information.

[0026] According to another aspect of the present disclosure, one or more non-transitory computer-readable storage media are provided storing one or more computer programs including computer-executable instructions that, when individually or collectively executed by one or more processors of an electronic device, cause the electronic device to perform operations. The operations include: when an audio signal is received, obtaining information about a user's breathing sound based on the audio signal; comparing the information about the breathing sound with registered information about breathing sounds of a plurality of users, thereby identifying at least one user among the plurality of users corresponding to the information about the breathing sound; and when the at least one user is identified, obtaining an analysis result about a sleep state of each of the at least one user based on information corresponding to each of the at least one user among the information about the breathing sound.

[0027] Other aspects, advantages and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the present disclosure taken in conjunction with the accompanying drawings.

[0028] Other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings. Other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the accompanying drawings.

[0029] FIG. 1 is a block diagram briefly illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure;

[0030] FIG. 2 is a block diagram schematically illustrating a plurality of modules according to one or more embodiments of the present disclosure;

[0031] FIG. 3 is a diagram showing information about audio signals and breathing sounds of multiple users according to one or more embodiments;

[0032] FIG. 4 is a diagram illustrating a user identification process according to one or more embodiments;

[0033] FIG. 5 is a block diagram detailing the configuration of an electronic device according to one or more embodiments of the present disclosure;

[0034] FIG. 6 is a block diagram detailing a plurality of modules according to one or more embodiments of the present disclosure, and

[0035] FIG. 7 is a flowchart briefly illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0036] The same reference numbers are used to represent identical elements throughout the drawing.

[0037] The following description, with reference to the attached drawings, is provided to facilitate a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and their equivalents. While it includes numerous specific details to facilitate this understanding, these are to be considered merely exemplary. Accordingly, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0038] The terms and words used in the following description and claims are not limited to their bibliographic meanings, but are used by the inventors solely to facilitate a clear and consistent understanding of the present invention. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the present disclosure as defined by the appended claims and their equivalents.

[0039] Unless the context clearly dictates otherwise, the singular forms “a,” “an,” and “the” should be understood to include plural referents. Thus, for example, reference to “a component surface” includes reference to one or more such surfaces.

[0040] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0041] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0042] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0043] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0044] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0045] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0046] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0047] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0048] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0049] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0050] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0051] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0052] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0053] It should be recognized that the blocks and combinations of flowcharts in each flowchart can be performed by one or more computer programs containing instructions. One or more computer programs may be stored entirely on a single memory device, or one or more computer programs may be divided into different portions stored on multiple different memory devices.

[0054] Any function or operation described in the present disclosure may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and may include circuits such as an application processor (AP, for example, a central processing unit (CPU)), a communication processor (CP, for example, a modem), a graphics processing unit (GPU), a neural processing unit (NPU), an artificial intelligence (AI) chip, a Wi-Fi chip, a Bluetooth® chip, a global positioning system (GPS) chip, a near field communication (NFC) chip, a connection chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on a chip (SoC), an IC, or a similar chip.

[0055] FIG. 1 is a block diagram briefly illustrating the configuration of an electronic device (100) according to one or more embodiments of the present disclosure. FIG. 2 is a block diagram briefly illustrating a plurality of modules according to one or more embodiments of the present disclosure. FIG. 3 is a diagram illustrating information about audio signals and breathing sounds of multiple users according to one or more embodiments, and FIG. 4 is a diagram for explaining a user identification process according to one or more embodiments. The following description will be made with reference to FIGS. 1 to 4 together.

[0056] Referring to FIG. 1, an electronic device (100) according to an embodiment of the present disclosure may include a microphone (110), a memory (120), and a processor (130).

[0057] The microphone (110) can acquire a signal for a sound or voice generated from outside the electronic device (100). Specifically, the microphone (110) can acquire a vibration corresponding to a sound or voice generated from outside the electronic device (100) and convert the acquired vibration into an electrical signal.

[0058] In particular, the microphone (110) according to the present disclosure can acquire a voice signal for a user's voice generated by the user's speech. In addition, the acquired signal can be converted into a digital signal and stored in a memory (120). The microphone (110) can include an A / D converter (Analog to Digital Converter) and can also operate in conjunction with an A / D converter located outside the microphone (110).

[0059] In one or more embodiments, the processor (130) may receive an audio signal via the microphone (110). In particular, the 'audio signal' may be an audio signal received via the microphone (110) while at least one user is sleeping, and may include information about the breathing sounds of the users, as described below. For example, if an audio signal is received via the microphone (110) while users A and B are sleeping, the received audio signal may include information about the breathing sounds of user A and information about the breathing sounds of user B together.

[0060] At least one instruction regarding the electronic device (100) may be stored in the memory (120). In addition, an operating system (O / S) for driving the electronic device (100) may be stored in the memory (120). In addition, various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (120). In addition, the memory (120) may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0061] Specifically, the memory (120) may store various software modules for operating the electronic device (100) according to various embodiments of the present disclosure, and the processor (130) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (120). That is, the memory (120) is accessed by the processor (130), and data reading / recording / modifying / deleting / updating, etc. may be performed by the processor (130).

[0062] Meanwhile, in the present disclosure, the term memory (120) may be used to mean memory (120), read only memory (ROM), random access memory (RAM) in the processor (130), or a memory card (e.g., micro SD (secure digital) card, memory stick) mounted in the electronic device (100).

[0063] In particular, in one or more embodiments, registration information regarding breathing sounds of multiple users may be stored. Here, the term "registration information" refers to information stored in the memory (120) as multiple users input information regarding their breathing sounds. In other words, the term "registration information" is a term used to distinguish and specify information regarding breathing sounds included in audio signals received in real time through the microphone (110) from information regarding breathing sounds previously stored in the memory (120) according to user input.

[0064] 'Information on respiratory sounds' is used as a general term for information on the characteristics of a user's respiratory sounds, and may particularly include information on respiratory sounds generated by the user's breathing during sleep. Specifically, the information on respiratory sounds may include various information, such as frequency characteristics and change patterns of a section corresponding to the respiratory sounds in an audio signal. In addition, the information on respiratory sounds may include at least one of information on a segment corresponding to the respiratory sounds and information on an embedding vector corresponding to the segment.

[0065] In addition, the memory (120) may store various information / data such as audio signals, information about the user's breathing sounds, information about segments corresponding to the breathing sounds, information about embedding vectors corresponding to segments, information about the user's identification results, analysis results about sleep states, control signals, data about neural network models, etc.

[0066] In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (120), and the information stored in the memory (120) may be updated as received from an external device or input by a user.

[0067] The processor (130) controls the overall operation of the electronic device (100). Specifically, the processor (130) is connected to the configuration of the electronic device (100) including a microphone (110) and a memory (120), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (120) as described above.

[0068] The processor (130) may be implemented in various ways. For example, the processor (130) may be implemented as at least one of an application specific integrated circuit (ASIC), an embedded processor, a processor with a microphone (110), hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (DSP). Meanwhile, the term "processor" in the present disclosure may be used to mean a central processing unit (CPU), a graphic processing unit (GPU), and a microprocessor unit (MPU).

[0069] The various operations of the processor (130) may be implemented through multiple modules. Specifically, data for the multiple modules may be stored in the memory (120). Furthermore, the processor (130) may load the data for the multiple modules stored in the memory (120) into the memory (120) or the memory (120) included in the processor (130), and implement various embodiments according to the present disclosure using the multiple modules. The multiple modules may be implemented as software modules or hardware modules.

[0070] Referring to FIG. 2, the plurality of modules may include a breathing sound information acquisition module (1010), a user identification module (1020), and a sleep analysis module (1030). In addition, the breathing sound information acquisition module (1010) may include a segmentation module (1011) and an embedding module (1012).

[0071] In particular, in one or more embodiments, when an audio signal is received through a microphone (110), the processor (130) may obtain information about the user's breathing sound based on the audio signal. The process of obtaining information about the breathing sound may include a segment extraction process and an embedding vector acquisition process, which will be described in detail below.

[0072] When an audio signal is received, the processor (130) can identify multiple segments corresponding to the user's breathing sounds in the audio signal. As illustrated in FIG. 2, the processor (130) inputs the audio signal into the segmentation module (1011) to identify multiple segments corresponding to the user's breathing sounds in the audio signal.

[0073] The processor (130) can identify a section having characteristics corresponding to the user's breathing sound in the audio signal and extract the identified section as a segment. That is, a 'segment' refers to the result of extracting each section having characteristics corresponding to the user's breathing sound in the audio signal, and can be replaced with terms such as 'slice', 'portion', 'subset', etc. For example, the characteristics corresponding to the breathing sound may include not only the user's inhalation and exhalation, but also snoring sounds, sounds indicating symptoms of apnea, etc.

[0074] The processor (130) can identify segments in the audio signal that have characteristics corresponding to the user's breathing sounds using various techniques, such as frequency feature analysis, energy and amplitude analysis, and waveform visualization of the audio signal. In addition, the processor (130) can input the audio signal into a first neural network model trained to distinguish the user's breathing sounds included in the audio signal, thereby obtaining information on multiple segments.

[0075] Image (310) of Fig. 3 illustrates multiple segments corresponding to the user's breathing sounds along with an audio signal, represented by dotted boxes. As illustrated in Fig. 3, the sizes of each of the multiple segments do not necessarily have to be identical.

[0076] When multiple segments are identified, the processor (130) can obtain multiple first embedding vectors corresponding to each of the multiple segments. As illustrated in FIG. 2, when information on multiple segments is obtained through the segmentation module (1011), the processor (130) can input the information on the multiple segments into the embedding module (1012) to obtain multiple first embedding vectors corresponding to each of the multiple segments.

[0077] The 'embedding vector' is a collective term for the result of digitizing each of a plurality of segments so that the characteristics of each segment can be distinguished, and the 'first embedding vector' is an embedding vector corresponding to a segment extracted from an audio signal received through a microphone (110), and is distinguished from the 'second embedding vector' which means an embedding vector corresponding to / included in registration information.

[0078] In the following, we will use the term embedding 'vector' to explain the characteristics of each segment extracted from the audio signal, assuming that they are quantified as a vector of fixed dimensions. However, depending on the embodiment, each of the multiple segments may be converted into the form of a real number, matrix, or tensor.

[0079] Specifically, the processor (130) can obtain a plurality of first embedding vectors corresponding to each of the plurality of segments by using techniques such as Mel-Frequency Cepstral Coefficients (MFCC), spectrogram-based embeddings, etc. In addition, the processor (130) can obtain the first embedding vector by inputting the plurality of segments into a second neural network model trained to convert information about the input segments into embedding vectors.

[0080] The processor (130) can identify at least one user corresponding to the information about the respiratory sound among a plurality of users by comparing the information about the respiratory sound with the registration information. As illustrated in FIG. 2, when the information about the respiratory sound (specifically, the embedding vector) is acquired through the respiratory sound information acquisition module (1010), the processor (130) inputs the information about the respiratory sound and the registration information stored in the memory (120) into the user identification module (1020), thereby identifying at least one user corresponding to the information about the respiratory sound among a plurality of users.

[0081] Specifically, the processor (130) can identify at least one user based on comparing each of the plurality of first embedding vectors with a plurality of second embedding vectors corresponding to the registration information.

[0082] In one or more embodiments, when a plurality of first embedding vectors are obtained, the processor (130) can identify distances between the locations of each of the plurality of first embedding vectors in the latent space and the center locations of the embedding vectors corresponding to the first user among the plurality of second embedding vectors.

[0083] Here, the "latent space" is an abstract space for representing the characteristics of embedding vectors, and each axis of the latent space can represent one of the characteristics of the embedding vectors. Accordingly, the locations of the embedding vectors in the latent space can correspond to the characteristics of the embedding vectors. As an example, Figure 4 shows the locations of each of a plurality of first embedding vectors and a plurality of second embedding vectors in the latent space.

[0084] Referring to Fig. 4, the positions of the first embedding vectors are represented by triangles, and the positions of the second embedding vectors are represented by circles. Specifically, the second embedding vectors of Fig. 4 include embedding vectors (411) corresponding to user A, embedding vectors (412) corresponding to user B, and embedding vectors (413) corresponding to user C. In other words, the second embedding vectors of Fig. 4 refer to embedding vectors registered based on information entered by users A, B, and C about their own breathing sounds.

[0085] Since the breath sounds of each of user A, user B, and user C exhibit different characteristics, the second embedding vectors, the embedding vectors corresponding to user A (411), the embedding vectors corresponding to user B (412), and the embedding vectors corresponding to user C (413), have distinct positions in the latent space.

[0086] Meanwhile, the first embedding vectors (420) of FIG. 4 include an embedding vector (421) corresponding to a first segment extracted from the same audio signal and an embedding vector (422) corresponding to a second segment.

[0087] When the embedding vector (421) corresponding to the first segment is obtained, the processor (130) can map the embedding vector (421) corresponding to the first segment to a latent space and calculate the distance between the embedding vector (421) corresponding to the first segment and the embedding vectors (411) corresponding to user A. Specifically, the processor (130) can calculate only the distance between the position of the embedding vector (421) corresponding to the first segment and the centroid position of the embedding vectors (411) corresponding to user A.

[0088] Here, the "center location" refers to a single location that can represent the embedding vectors. The center location can be determined based on the mean or median of the embedding vectors, or it can be determined based on methods such as PCA (Principal Component Analysis), which uses principal component analysis of the embedding vectors, or GMM (Gaussian Mixture Model), which uses the average of the results of modeling the distribution of the embedding vectors using a Gaussian mixture model.

[0089] The processor (130) can determine a correlation between a plurality of first embedding vectors and a first user based on the distances identified as described above, thereby identifying a user corresponding to information about breathing sounds included in an audio signal.

[0090] In one or more embodiments, if the distances between the positions of each of the plurality of first embedding vectors and the center positions of the embedding vectors corresponding to the first user among the plurality of second embedding vectors are less than a preset threshold distance, the processor (130) can identify the first user as at least one user corresponding to the information about the breathing sound included in the audio signal.

[0091] In the example of FIG. 4, if the distance between the position of the embedding vector (421) corresponding to the first segment and the center position (431) of the embedding vectors (411) corresponding to user A is less than the threshold distance, the processor (130) determines that the correlation between the embedding vector (421) corresponding to the first segment and the embedding vectors (411) corresponding to user A is high, and accordingly, the user corresponding to the information about the breathing sound included in the audio signal can be identified as the registered user A.

[0092] On the other hand, in the example of FIG. 4, if the distance between the position of the embedding vector (421) corresponding to the first segment and the center position (431) of the embedding vectors (411) corresponding to user A is greater than or equal to the threshold distance, the processor (130) determines that the correlation between the embedding vector (421) corresponding to the first segment and the embedding vectors (411) corresponding to user A is low, and accordingly, the user corresponding to the information about the breathing sound included in the audio signal can be identified as not being the registered user A.

[0093] In the above, only the processing process for the embedding vector (421) corresponding to the first segment has been described, but the processing process described above can be performed for the embedding vector vectors corresponding to a plurality of segments included in the audio signal, and as a result, it can be identified whether each of the plurality of segments included in the audio signal corresponds to a certain user.

[0094] The above has described an embodiment of calculating distances between the positions of each of a plurality of first embedding vectors and the center positions of the embedding vectors corresponding to the first user among the plurality of second embedding vectors, but the present disclosure is not limited thereto. For example, the processor (130) may calculate all distances between the positions of the embedding vector (421) corresponding to the first segment and the positions of each of the embedding vectors (411) corresponding to the user A, and may identify at least one user corresponding to information about breathing sounds included in the audio signal based on all of the calculated distances.

[0095] In one or more embodiments, the processor (130) can identify a user corresponding to information about breath sounds included in an audio signal based on a distance between a location of each of the plurality of first embedding vectors and each of the center locations of the second embedding vectors corresponding to the plurality of users.

[0096] In the example of FIG. 4, the processor (130) calculates a first distance (441) between the position of the embedding vector (421) corresponding to the first segment and the center positions of the embedding vectors (411) corresponding to user A, a second distance (442) between the position of the embedding vector (421) corresponding to the first segment and the center positions (432) of the embedding vectors (412) corresponding to user B, and a third distance (443) between the position of the embedding vector (421) corresponding to the first segment and the center positions (432) of the embedding vectors (413) corresponding to user C, and compares the calculated first distance, second distance, and third distance, thereby identifying a user corresponding to information about breathing sounds included in an audio signal.

[0097] For example, as in the example of FIG. 4, if the second distance is shorter than the first distance and the third distance, the processor (130) can identify user B corresponding to the second distance as a user corresponding to the information about breathing sounds included in the audio signal.

[0098] Meanwhile, among the plurality of first embedding vectors, a first embedding vector whose distance from all center positions of the second embedding vectors is greater than a threshold distance may be regarded as an abnormal sample and may be excluded from the user identification process.

[0099] When at least one user is identified, the processor (130) can obtain an analysis result for the sleep state of at least one user based on information corresponding to at least one user among the information on breathing sounds included in the audio signal. As illustrated in FIG. 2, the processor (130) inputs the identification result obtained through the user module and the information on breathing sounds obtained through the breathing sound information obtaining module (1010) into the sleep analysis module (1030), thereby obtaining an analysis result for the sleep state of at least one user.

[0100] The processor (130) can extract information about the breathing sounds of the first user and information about the breathing sounds of the second user by distinguishing them from information about the breathing sounds included in the audio signal. In addition, the processor (130) can obtain analysis results about the sleep state of the first user using only the information about the breathing sounds of the first user, and can obtain analysis results about the sleep state of the second user using only the information about the breathing sounds of the second user.

[0101] Referring to FIG. 3, the processor (130) can identify a plurality of segments (dotted boxes) corresponding to the breathing sounds of a user in an audio signal represented by an image (310), and compare a plurality of first embedding vectors corresponding to each of the plurality of segments with a plurality of second embedding vectors corresponding to registration information, thereby identifying first segments corresponding to the first user and second segments corresponding to the second user among the plurality of segments. Accordingly, the processor (130) can obtain information about the breathing sounds of the first user including the first segments as represented by the image (320), and can obtain information about the breathing sounds of the second user including the second segments as represented by the image (330).

[0102] The analysis results may include at least one of the following: information about whether at least one user is sleeping, information about sleep quality, and information about health. For example, the analysis results may include information about the user's sleep duration, sleep stages (e.g., awake, light sleep, deep sleep, REM (rapid eye movement) sleep), and sleep quality. The analysis results may also include information about the user's health, such as stress levels and breathing conditions.

[0103] The processor (130) may obtain analysis results on the sleep state of at least one user by using the cycle, frequency, and frequency of the breathing sound corresponding to at least one user. In addition, the processor (130) may obtain analysis results by inputting information corresponding to at least one user among the information on breathing sounds included in the audio signal into a third neural network model trained to identify the sleep state of the user corresponding to the breathing sound.

[0104] According to the embodiments described above, the electronic device (100) can clearly distinguish the breathing sounds of multiple users using only information about the breathing sounds of the users in a non-contact manner, and can accurately analyze the sleep states of each of the multiple users accordingly.

[0105] FIG. 5 is a block diagram illustrating in detail the configuration of an electronic device (100) according to one or more embodiments of the present disclosure, and FIG. 6 is a block diagram illustrating in detail a plurality of modules according to one or more embodiments of the present disclosure.

[0106] Referring to FIG. 5, the electronic device (100) may further include a microphone (110), a memory (120), and a processor (130), as well as a communication unit (140), a sensor (150), an input unit (160), and an output unit (170). However, the configurations as shown in FIGS. 1 and 5 are merely exemplary, and it is to be understood that new configurations may be added or some configurations may be omitted in addition to the configurations as shown in FIGS. 1 and 5 when implementing the present disclosure.

[0107] Referring to FIG. 6, the plurality of modules may further include a breathing sound information acquisition module (1010), a user identification module (1020), and a sleep analysis module (1030), as well as a breathing sound information management module (1040), a bio-information acquisition module (1050), and a control signal acquisition module (1060). The configuration of the plurality of modules illustrated in FIGS. 2 and 6 is also merely exemplary.

[0108] The communication unit (140, i.e., transceiver) includes a circuit and can perform communication with an external device. Specifically, the processor (130) can receive various data or information from an external device connected via the communication unit (140), and can also transmit various data or information to the external device.

[0109] The communication unit (140) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and a UWB module (Ultra-Wide Band). Specifically, the WiFi module and the Bluetooth module may each perform communication in the WiFi or Bluetooth manner. When using a WiFi module or a Bluetooth module, various connection information, such as an SSID, may be first transmitted and received, and then communication may be established using this, after which various pieces of information may be transmitted and received.

[0110] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc. And, the NFC module can perform communication in the NFC (Near Field Communication) method using the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, 2.45GHz, etc. In addition, the UWB module can accurately measure ToA (Time of Arrival), which is the time it takes for a pulse to reach a target, and AoA (Ange of Arrival), which is the pulse arrival angle at the transmitting device, through communication between UWB antennas, and accordingly, precise distance and location recognition is possible within an error range of several tens of centimeters indoors.

[0111] In particular, in one or more embodiments, if the analysis result for the sleep state of each of at least one user indicates that at least one user is sleeping, the processor (130) may control the communication unit (140) to obtain a control signal for controlling an external device and transmit the control signal to the external device. Referring to FIG. 6, the processor (130) may input the analysis result obtained through the sleep analysis module (1030) into the control signal acquisition module (1060) to obtain a control signal for controlling the external device.

[0112] In other words, as a result of identifying a user corresponding to information about breathing sounds included in an audio signal, if at least one user is identified and all of the identified at least one user are sleeping, the processor (130) may obtain a control signal for controlling an external device based on the information that all users are sleeping, and control the communication unit (140) to transmit the control signal to the external device. Here, the external device may be a device registered as a device constituting the same IoT (Internet of things) network as the electronic device (100).

[0113] For example, based on information that all users are sleeping, a control signal to turn off the power of lighting devices, speaker devices, TVs, etc. can be obtained, and the communication unit (140) can be controlled to transmit the control signal to an external device.

[0114] In addition, the processor (130) can receive registration information on the breathing sounds of multiple users, information on neural network models, etc. through the communication unit (140), and control the communication unit (140) to transmit the analysis results on the user's sleep state to an external device such as the user's smartphone or smartwatch.

[0115] The sensor (150) can detect various information inside and outside the electronic device (100). Specifically, the sensor (150) may include at least one of a GPS (Global Positioning System) sensor (150), a gyro sensor (150), an acceleration sensor (150), a LiDAR (light detection and ranging sensor), an inertial sensor (150) (Inertial Measurement Unit, IMU), and a motion sensor (150). In addition, the sensor (150) may include various types of sensors (150), such as a temperature sensor (150), a humidity sensor (150), an infrared sensor (150), and a biosensor (150).

[0116] In particular, the sensor (150) according to the present disclosure may include at least one of an image sensor (150) capable of acquiring an image of a user over time, a motion sensor (150) capable of detecting a movement of the user, and a biometric sensor (150) capable of detecting a signal related to the user's biometric information. The biometric sensor (150) may include a sensor (150) such as a heart rate sensor (150), a bioimpedance sensor (150), a blood pressure sensor (150), etc.

[0117] In one or more embodiments, when at least one user is identified, the processor (130) may obtain biometric information about the at least one user through the sensor (150). Then, the processor (130) may obtain analysis results about the sleep state of each of the at least one user based on the information and biometric information corresponding to each of the at least one user.

[0118] Referring to FIG. 6, the processor (130) can input biometric information acquired through the biometric information acquisition module (1050) together with information about the user's breathing sound acquired through the breathing sound information acquisition module (1010) into the sleep analysis module (1030) to obtain analysis results on the user's sleep state.

[0119] For example, if the electronic device (100) includes a sensor (150), the processor (130) can obtain analysis results on the sleep state of each user by using biometric information, such as information on the user's blood pressure and information on the user's heartbeat, together with information on the user's breathing sound.

[0120] Meanwhile, various sensing information such as the user's movements and posture, as well as biometric information, can be used to analyze sleep status.

[0121] The input unit (160) includes a circuit, and the processor (130) can receive a user command to control the operation of the electronic device (100) through the input unit (160). Specifically, the input unit (160) can be configured with a microphone (110), a camera, a remote control signal receiving unit, and the like. In addition, the input unit (160) can be implemented in a form included in a display as a touch screen. In particular, the microphone (110) can receive a voice signal and convert the received voice signal into an electrical signal.

[0122] In particular, in one or more embodiments, the processor (130) may receive, through the input unit (160), a user input for registering information about the user's breathing sound, a user input for updating registration information about the user's breathing sound, etc.

[0123] The output unit (170) includes a circuit, and the processor (130) can output various functions that the electronic device (100) can perform through the output unit (170). In addition, the output unit (170) can include at least one of a display, a speaker, and an indicator.

[0124] The display can output image data under the control of the processor (130). Specifically, the display can output an image previously stored in the memory (120) under the control of the processor (130). In particular, the display according to one embodiment of the present disclosure can also display a user interface stored in the memory (120).

[0125] The display may be implemented as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), etc., and in some cases, the display may also be implemented as a flexible display, a transparent display, etc. However, the display according to the present disclosure is not limited to a specific type.

[0126] The speaker can output audio data under the control of the processor (130).

[0127] The indicator can be lit under the control of the processor (130). Specifically, the indicator can be lit in various colors under the control of the processor (130). For example, the indicator can be implemented using a light emitting diode (LED), a liquid crystal display panel (LCD), a vacuum fluorescent display (VFD), etc., but is not limited thereto.

[0128] In particular, in one or more embodiments, if at least one user is not identified as a result of identifying a user corresponding to information about a breath sound included in an audio signal, the processor (130) may input the identification result into the breath sound information management module (1040) to perform an operation for adding registration information.

[0129] Specifically, if at least one user is not identified as a result of identifying a user corresponding to information about a breathing sound included in an audio signal, the processor (130) may control the display to display a user interface. Here, the user interface may be for requesting the user to register information about the breathing sound. When a user input for registering information about the breathing sound is received through the user interface, the processor (130) may add information about the breathing sound to the registration information.

[0130] For example, when a user input for initiating registration of information about breathing sounds is received through a user interface, the processor (130) may initiate registration of information about breathing sounds. Thereafter, the processor (130) may receive an audio signal corresponding to the user's breathing through a microphone (110) and store information about breathing sounds included in the received audio signal as registration information.

[0131] The process of storing information about breath sounds as registration information may include, as described above, identifying segments in an audio signal and obtaining embedding vectors corresponding to the segments. The processor (130) may also identify segments with distinct characteristics in the audio signal and store only the embedding vectors corresponding to the distinct segments as registration information.

[0132] Meanwhile, the above described embodiment adds registration information when at least one user is not identified as a result of identifying a user corresponding to information about breathing sounds included in an audio signal. Conversely, in the above, when at least one user is identified as a result of identifying a user corresponding to information about breathing sounds included in an audio signal, the processor (130) can input the identification result into the breathing sound information management module (1040) to perform an operation for updating registration information.

[0133] In one or more embodiments, if at least one user is identified as a result of identifying a user corresponding to information about a breathing sound included in an audio signal, the processor (130) may update registration information based on the information about the breathing sound. Referring to FIG. 6, the processor (130) may input information about the breathing sound into a breathing sound management module to obtain updated registration information.

[0134] Specifically, when at least one user is identified, the processor (130) can extract information about the breathing sounds of the first user and information about the breathing sounds of the second user from the information about the breathing sounds included in the audio signal. Then, the processor (130) can update the registration information about the first user based on the information about the breathing sounds of the first user, and can update the registration information about the second user based on the information about the breathing sounds of the second user.

[0135] For example, updating registration information may mean adding a first embedding vector corresponding to a first user to a second embedding vector corresponding to a first user in a latent space as illustrated in FIG. 4, and adding a first embedding vector corresponding to a second user to a second embedding vector corresponding to a second user.

[0136] FIG. 7 is a flowchart briefly illustrating a control method of an electronic device (100) according to one or more embodiments of the present disclosure.

[0137] Referring to FIG. 7, the electronic device (100) can receive an audio signal (S710). Specifically, the electronic device (100) can receive an audio signal through a microphone (110) included in the electronic device (100) and can receive an audio signal from an external device.

[0138] The electronic device (100) can obtain information about the user's breathing sounds based on an audio signal (S720). Specifically, the electronic device (100) can identify multiple segments corresponding to the user's breathing sounds in the audio signal and obtain multiple first embedding vectors corresponding to each of the multiple segments.

[0139] The electronic device (100) can identify at least one user corresponding to the information about the respiratory sounds by comparing the information about the respiratory sounds with the registration information (S730) (S750). Here, the registration information about the respiratory sounds of multiple users may be stored in the memory (120) of the electronic device (100) or may be received from an external device. Specifically, the electronic device (100) can identify at least one user based on comparing each of the plurality of first embedding vectors with the plurality of second embedding vectors corresponding to the registration information.

[0140] When at least one user is identified (S740-Y), the electronic device (100) can obtain analysis results on the sleep status of at least one user based on information corresponding to at least one user among the breathing sound information (S740). The analysis results may include at least one of information on whether at least one user is sleeping, information on sleep quality, and information on health.

[0141] Meanwhile, if at least one user is not identified (S740-N), the electronic device (100) may terminate the operation, provide a user interface to add registration information, and when a user input for registering information about breath sounds is received through the user interface, the electronic device (100) may add information about breath sounds to the registration information.

[0142] Meanwhile, the control method of the electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the control method of the electronic device (100) may be stored and provided in a non-transitory computer readable medium.

[0143] Specifically, in a non-transitory computer-readable recording medium including a program for executing a method for controlling an electronic device (100), the method for controlling an electronic device (100) may include a step of, when an audio signal is received, obtaining information on a user's breathing sound based on the audio signal, a step of comparing the information on the breathing sound with registration information on the breathing sounds of a plurality of users to identify at least one user corresponding to the information on the breathing sound among the plurality of users, and a step of, when at least one user is identified, obtaining an analysis result on a sleep state of each of at least one user based on information corresponding to each of at least one user among the information on the breathing sound.

[0144] In the above, a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100) have been briefly described, but this is only to omit redundant descriptions, and it goes without saying that various embodiments of the electronic device (100) can also be applied to a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100).

[0145] The artificial intelligence-related function according to the present disclosure is operated through the processor (130) and memory (120) of the electronic device (100).

[0146] The processor (130) may be composed of one or more processors (130). At this time, the one or more processors (130) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Processing Unit), but is not limited to the examples of the processors (130) described above.

[0147] The CPU is a general-purpose processor (130) capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layer cache structure allows for the efficient execution of complex programs. The CPU is advantageous in a serial processing method, enabling organic linking of previous and subsequent calculation results through sequential calculations. The general-purpose processor (130) is not limited to the aforementioned examples, except in cases where it is specifically designated as a CPU.

[0148] A GPU is a processor (130) for large-scale operations such as floating-point operations used in graphic processing, and can perform large-scale operations in parallel by integrating a large number of cores. In particular, a GPU may be advantageous compared to a CPU in parallel processing methods such as convolution operations. In addition, a GPU may be used as a co-processor (130) to supplement the functions of a CPU. The processor (130) for large-scale operations is not limited to the examples described above, except in cases where it is specified as a GPU as described above.

[0149] An NPU is a processor (130) specialized in artificial intelligence operations using an artificial neural network, and each layer constituting the artificial neural network can be implemented with hardware (e.g., silicon). At this time, since the NPU is designed specifically according to the required specifications of the company, it has a lower degree of freedom compared to a CPU or GPU, but it can efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor (130) specialized in artificial intelligence operations, the NPU can be implemented in various forms such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), a Vision Processing Unit (VPU), etc. The artificial intelligence processor (130) is not limited to the above-described examples, except in cases where it is specified as the above-described NPU.

[0150] Additionally, one or more processors (130) may be implemented as a SoC (System on Chip). In this case, the SoC may further include, in addition to one or more processors (130), a memory (120), and a network interface such as a bus for data communication between the processor (130) and the memory (120).

[0151] When a plurality of processors (130) are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors (130). For example, the electronic device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors (130). However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor (130).

[0152] In addition, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in one processor (130). In particular, the electronic device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by utilizing multiple cores included in the processor (130).

[0153] One or more processors (130) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in the memory (120). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0154] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0155] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0156] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0157] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0158] According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0159] Each of the components (e.g., modules or programs) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in the various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration.

[0160] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0161] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0162] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored in the storage medium and operating according to the called instructions.

[0163] When the above instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0164] It will be appreciated that the various embodiments of the present disclosure, in accordance with the claims and description of the present disclosure, may be realized in the form of hardware, software, or a combination of hardware and software.

[0165] Such software may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), and the one or more computer programs include computer-executable instructions that, when individually or collectively executed by one or more processors of the electronic device, cause the electronic device to perform the disclosed method.

[0166] Such software may be stored in a volatile or non-volatile storage form, such as, for example, a storage device such as a read only memory (ROM), whether erasable or rewritable, or in a memory form such as a random access memory (RAM), a memory chip, a device or an integrated circuit, or an optical or magnetically readable medium such as, for example, a compact disk (CD), a digital versatile disc (DVD), a magnetic disk or a magnetic tape. It will be appreciated that the storage devices and storage media are various embodiments of non-transitory machine-readable storage suitable for storing a computer program or a computer program comprising instructions that, when executed, implement various embodiments of the present disclosure. Accordingly, various embodiments provide a program comprising code for implementing an apparatus or method as claimed in any of the claims of the present disclosure, and a non-transitory machine-readable storage medium storing such a program.

[0167] While the present disclosure has been illustrated and described with reference to various embodiments, it will be understood by those skilled in the art that various changes may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In electronic devices, mike; Memory that stores one or more computer programs; one or more processors in communication with the microphone and memory; The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: Store registration information on breathing sounds of multiple users in the above memory, When an audio signal is received through the above microphone, information about the user's breathing sound is obtained based on the audio signal, By comparing the information about the above breathing sound with the above registration information, at least one user corresponding to the information about the breathing sound among the plurality of users is identified, An electronic device that, when at least one user is identified, obtains an analysis result on the sleep state of each of the at least one user based on information corresponding to each of the at least one user among the information on the breathing sounds.

2. In paragraph 1, The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: When the above audio signal is received, multiple segments corresponding to the user's breathing sound are identified from the audio signal, Obtaining a plurality of first embedding vectors corresponding to each of the above plurality of segments, An electronic device that identifies at least one user based on comparing each of the plurality of first embedding vectors with a plurality of second embedding vectors corresponding to the registration information.

3. In paragraph 2, The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: When the plurality of first embedding vectors are obtained, the distances between the positions of each of the plurality of first embedding vectors in the latent space and the center positions of the embedding vectors corresponding to the first user among the plurality of second embedding vectors are identified, An electronic device that identifies the first user as at least one user if the identified distances are less than a preset threshold distance.

4. In paragraph 2, The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: Inputting the audio signal into a first neural network model trained to distinguish the user's breathing sounds included in the audio signal, thereby obtaining information about the plurality of segments, By inputting the plurality of segments into a second neural network model trained to convert the input segments into embedding vectors, the plurality of first embedding vectors are obtained. An electronic device that inputs information corresponding to at least one user into a third neural network model trained to identify the user's sleep state corresponding to breathing sounds to obtain the analysis result.

5. In paragraph 1, An electronic device wherein the analysis result includes at least one of information on whether at least one user is sleeping, information on the quality of sleep of the at least one user, and information on the health of the at least one user.

6. In paragraph 1, further comprising a transceiver; The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: If the above analysis results indicate that at least one user is sleeping, a control signal for controlling an external device is obtained, An electronic device that transmits the control signal to the external device.

7. In paragraph 1, The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: An electronic device that updates the registration information based on information about the breathing sound when at least one user is identified.

8. In paragraph 1, display; including more, The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: If at least one of the above users is not identified, display the user interface, An electronic device that, when a user input for registering information about the breath sound is received through the user interface, adds information about the breath sound to the registration information.

9. In paragraph 1, further comprising a sensor; The one or more computer programs comprise instructions executable by a computer, the instructions, when individually or collectively executed by the one or more processors, causing the electronic device to: When at least one user is identified, biometric information about the at least one user is acquired through the sensor, An electronic device that obtains analysis results on the sleep state of each of the at least one user based on information corresponding to each of the at least one user and the biometric information.

10. In a method performed by an electronic device, A step of storing registration information on breathing sounds of multiple users; When an audio signal is received, a step of obtaining information about the user's breathing sound based on the audio signal; A step of comparing the information about the breathing sound with the registration information about the breathing sound of a plurality of users, and identifying at least one user corresponding to the information about the breathing sound among the plurality of users; and A method comprising: a step of obtaining an analysis result on a sleep state of each of the at least one user based on information corresponding to each of the at least one user among information on the breathing sounds, when the at least one user is identified; 11. In clause 10, The step of obtaining information about the above breathing sounds is: When the above audio signal is received, a step of identifying a plurality of segments corresponding to the user's breathing sound in the audio signal; and A step of obtaining a plurality of first embedding vectors corresponding to each of the plurality of segments; comprising: The step of identifying at least one user above comprises: A method comprising: identifying at least one user based on comparing each of the plurality of first embedding vectors with a plurality of second embedding vectors corresponding to the registration information; 12. In paragraph 11, The step of identifying at least one user above comprises: When the plurality of first embedding vectors are obtained, a step of identifying distances between the positions of each of the plurality of first embedding vectors in the latent space and the center positions of the embedding vectors corresponding to the first user among the plurality of second embedding vectors; and A method further comprising: identifying the first user as the at least one user if the identified distances are less than a preset threshold distance; 13. In paragraph 11, The step of obtaining information about the above breathing sounds is: A step of obtaining information about the plurality of segments by inputting the audio signal to a first neural network model learned to distinguish the user's breathing sounds included in the audio signal; and A step of obtaining the plurality of first embedding vectors by inputting the plurality of segments to a second neural network model trained to convert the input segments into embedding vectors; comprising; The steps for obtaining the above analysis results are: A method comprising: a step of obtaining the analysis result by inputting information corresponding to at least one user into a third neural network model learned to identify the user's sleep state corresponding to the breathing sound; 14. In paragraph 10, A method wherein the analysis result includes at least one of information on whether at least one user is sleeping, information on the quality of sleep of the at least one user, and information on the health of the at least one user.

15. In paragraph 10, The above method, If the above analysis result indicates that at least one user is sleeping, a step of obtaining a control signal for controlling an external device; and A method further comprising: a step of transmitting the control signal to the external device;

Images