Method, computing device and computer program for analyzing a user's sleep state through acoustic information

Abstract

To provide a server device on which a sleep analysis model for analyzing a sleep state of a user through sound information is mounted, a device for analyzing sleep state information of the user through sound information and a recording medium recording a program for analyzing sleep state information of the user through sound information.SOLUTION: In a system in which a computing device, a user terminal and an external server mutually transmit and receive data through a network, the computing device for analyzing a sleep state of a user through sound information includes steps of: acquiring sleep sound information related to sleep of the user; performing preprocessing on sleep sound information; and performing analysis on pre-processed sleep sound information and acquiring sleep state information. The sleep state information includes sleep step information related to the depth of sleep of the user.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention is directed to analyzing a user's sleep state, and more specifically, provides analysis information related to sleep stages based on sound information acquired from the user's sleep environment. [Background technology]

[0002] There are various ways to maintain and improve health, such as exercise and diet, but the most important thing is to manage sleep well, which takes up more than 30% of the day.

[0003] However, despite the fact that modern people have been replaced by machines with simpler labor and have more comfortable lives, they are unable to get a good night's sleep due to irregular eating habits, lifestyles and stress, and suffer from sleep disorders such as insomnia, excessive sleep, sleep apnea syndrome, nightmares, night terrors and sleepwalking.

[0004] According to the National Health Insurance Service, the number of people with sleep disorders in Korea increased by an average of 8% per year from 2014 to 2018, and the number of people who received treatment for sleep disorders in Korea in 2018 reached approximately 570,000.

[0005] Deep sleep is recognized as an important factor affecting physical and mental health, and interest in deep sleep is increasing. However, in order to improve sleep disorders, patients must visit specialized medical institutions in person, which requires additional testing fees, and continuous management is difficult, so users are not making enough effort to seek treatment.

[0006] Korean Patent Publication No. 2003-0032529 discloses a sleep induction device and sleep induction method that receives input of a user's physical information and outputs vibrations and / or ultrasound waves in a frequency band detected through repetitive learning according to the user's physical condition during sleep, thereby enabling optimal sleep induction.

[0007] However, conventional technologies can reduce sleep quality due to the inconvenience caused by body-worn devices and require periodic maintenance of the devices (e.g., charging).

[0008] As a result, recent research has been progressing to estimate sleep stages by non-contactly monitoring breathing patterns and the degree of activation of the autonomic nervous system based on body movements during the night.

[0009] However, in order to obtain information on breathing patterns and movements in a non-contact manner at a distance that does not affect movements during sleep, additional equipment (e.g., a radio wave measuring device for sensing movements) must be installed.

[0010] Therefore, there may be a demand for a technology that can easily acquire acoustic information related to a sleep environment through a user terminal (e.g., a mobile terminal) carried by the user without requiring additional equipment, and that can analyze the user's sleep stage based on the acquired acoustic information to detect the sleep state. Summary of the Invention [Problem to be solved by the invention]

[0011] The present invention has been devised in response to the above-mentioned background art, and aims to analyze a user's sleep stage through acoustic information sensed in the user's sleep environment and provide information about the user's sleep state.

[0012] The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0013] In one embodiment of the present invention to solve the above-mentioned problems, a method for analyzing a user's sleep state via acoustic information is disclosed.

[0014] The method may include the steps of acquiring sleep acoustic information related to a user's sleep, performing pre-processing on the sleep acoustic information, and performing analysis on the pre-processed sleep acoustic information to acquire sleep state information, wherein the sleep state information may include sleep stage information related to the depth of the user's sleep.

[0015] In an alternative embodiment, the step of acquiring the sleep sound information may include a step of identifying a singular point where information of a pre-established pattern is detected in the living environment sound information, and a step of acquiring the sleep sound information based on the living environment sound information acquired based on the singular point.

[0016] In an alternative embodiment, the step of performing pre-processing on the sleep acoustic information may include the steps of classifying the sleep acoustic information into one or more acoustic frames having a predetermined time unit, identifying a minimum acoustic frame having a minimum energy level based on the energy levels of each of the one or more acoustic frames, and performing noise reduction on the sleep acoustic information based on the minimum acoustic frame.

[0017] In an alternative embodiment, obtaining the sleep state information may include obtaining the sleep state information using a sleep analysis model that includes one or more network functions.

[0018] In an alternative embodiment, the sleep analysis model may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate the sleep state information.

[0019] In an alternative embodiment, obtaining the sleep state information may include generating a spectrogram corresponding to the sleep acoustic information, and processing the spectrogram as an input to the sleep analysis model to obtain the sleep state information.

[0020] In an alternative embodiment, the method further includes performing data augmentation based on the pre-processed sleep acoustic information, and the data augmentation may include at least one of pitch shifting, Gaussian noise, loudness control, dynamic range control, and spec augmentation.

[0021] In an alternative embodiment, the method may further include receiving sleep plan information from a user terminal, generating external environment creation information based on the sleep plan information, and determining to transmit the external environment creation information as an environment creation module.

[0022] In an alternative embodiment, the method may further include generating external environment creation information based on the sleep state information, and determining to transmit the external environment creation information to an environment creation module.

[0023] In another embodiment of the present invention, a computing device for analyzing a user's sleep state via acoustic information is disclosed, the computing device including: a memory for storing one or more instructions; and a processor for executing the one or more instructions stored in the memory, the processor executing the one or more instructions to perform a method for analyzing a user's sleep state via acoustic information.

[0024] In yet another embodiment of the present invention, a computer program stored in a computer-readable storage medium is disclosed, wherein the computer program may be stored in a computer-readable recording medium so as to be coupled with a computer as hardware to perform the method for analyzing a user's sleep state through acoustic information.

[0025] Other details of the invention are included in the detailed description and drawings. [Effects of the Invention]

[0026] The present invention has been devised in response to the above-mentioned background art, and is capable of providing analytical information on a user's sleep state based on acoustic information acquired from the user's sleep environment.

[0027] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0028] Various aspects are now described with reference to the drawings, wherein like reference numerals are used to generally refer to like elements.

[0029] In the following embodiments, for purposes of explanation, numerous specific details are presented in order to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspects may be practiced without such specific details.

[0030] [Figure 1] FIG. 1 is a conceptual diagram illustrating a system in which various aspects of a computing device for analyzing a user's sleep state through acoustic information, related to an embodiment of the present invention, can be implemented.

[0031] [Figure 2] FIG. 2 illustrates a block diagram of a computing device for analyzing a user's sleep state through acoustic information in accordance with an embodiment of the present invention.

[0032] [Figure 3] FIG. 3 is an exemplary diagram illustrating a process of acquiring sleep sound information from living environment sound information according to an embodiment of the present invention.

[0033] [Figure 4]FIG. 4 is an exemplary diagram illustrating a method for acquiring a spectrogram corresponding to sleep acoustic information according to an embodiment of the present invention.

[0034] [Figure 5] FIG. 5 is an illustrative diagram showing spectrograms associated with various sleep stages in accordance with an embodiment of the present invention.

[0035] [Figure 6] FIG. 6 is an exemplary diagram illustrating a process of acquiring sleep state information through a spectrogram according to an embodiment of the present invention.

[0036] [Figure 7] FIG. 7 is a flow chart illustrating an example of a method for analyzing a user's sleep state through acoustic information according to an embodiment of the present invention.

[0037] [Figure 8] FIG. 8 is a schematic diagram illustrating one or more network functions associated with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0038] The advantages and features of the present invention and the methods for achieving them will become more apparent from the following detailed description of the embodiments taken in conjunction with the accompanying drawings.

[0039] However, the present invention is not limited to the embodiments disclosed below, but may be embodied in various different forms, and the present embodiments are provided merely so that the disclosure of the present invention will be complete and will fully convey the scope of the present invention to those skilled in the art to which the present invention pertains, and the present invention is only defined by the scope of the claims.

[0040] The terms used in this specification are for the purpose of describing embodiments and are not intended to limit the present invention. In this specification, the singular includes the plural unless otherwise specified. The terms "comprises" and / or "comprising" used in this specification do not exclude the presence or addition of one or more other elements other than the elements listed. The same reference numerals refer to the same elements throughout this specification, and "and / or" includes each and every combination of one or more of the listed elements. Although terms such as "first," "second," etc. are used to describe various elements, these elements are not limited by these terms. These terms are used merely to distinguish one element from another. Therefore, a first element referred to below may of course be a second element within the technical spirit of the present invention.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in the sense commonly understood by a person of ordinary skill in the art to which the present invention belongs. Furthermore, commonly used and predefined terms are not to be interpreted ideally or excessively unless expressly defined otherwise.

[0042] The terms "module" and "module" used herein refer to software or hardware components, such as FPGAs or ASICs, that perform a certain function. However, "module" and "module" are not limited to software or hardware. A "module" or "module" may be configured to reside on an addressable storage medium or to execute on one or more processors. Thus, by way of example, a "module" or "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and functionality provided within a "module" or "module" may be combined into fewer components and "modules" or "modules" or further separated into additional components and "modules" or "modules."

[0043] In this specification, the term "computer" refers to any type of hardware device including at least one processor, and may also encompass software configurations operating on such hardware devices according to embodiments. For example, the term "computer" may refer to, but is not limited to, smartphones, tablet PCs, desktops, laptops, and user clients and applications running on each device.

[0044] Those skilled in the art should further recognize that the various illustrative logical blocks, components, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or any combination of both. To clearly illustrate the interchangeability of hardware and software, the various illustrative components, blocks, components, means, logic, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is embodied as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present subject matter.

[0045] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0046] Although each step described in this specification is described as being performed by a computer, the subject of each step is not limited to this, and depending on the embodiment, at least some of each step may be performed by different devices.

[0047] FIG. 1 is a conceptual diagram illustrating a system in which various aspects of a computing device for analyzing a user's sleep state through acoustic information, related to an embodiment of the present invention, can be implemented.

[0048] A system according to an embodiment of the present invention may include a computing device 100, a user terminal 10, an external server 20, and a network. The computing device 100, the user terminal 10, and the external server 20 according to an embodiment of the present invention can mutually transmit and receive data for the system according to an embodiment of the present invention via the network.

[0049] The network according to an embodiment of the present invention can use various wired communication systems such as a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High Speed ​​DSL (VDSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and a Local Area Network (LAN).

[0050] Additionally, the networks presented herein can use a variety of wireless communication systems such as Code Division Multi-Access (CDMA), Time Division Multi-Access (TDMA), Frequency Division Multi-Access (FDMA), Orthogonal Frequency Division Multi-Access (OFDMA), Single Carrier-FDMA (SC-FDMA), and other systems.

[0051] A network according to an embodiment of the present invention may be configured regardless of its communication mode, such as wired or wireless, and may be configured as various communication networks, such as a short-range communication network (PAN: Personal Area Network) or a short-range communication network (WAN: Wide Area Network). The network may be the well-known World Wide Web (WWW), or may use wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. The technology described herein may be used in other networks as well as the networks mentioned above.

[0052] According to one embodiment of the present invention, the user terminal 10 may refer to a terminal carried by a user, which can receive information related to the user's sleep through information exchange with the computing device 100.

[0053] For example, the user terminal 10 may be a terminal associated with a user who wishes to improve his or her health through information related to his or her sleep habits. The user may obtain information related to his or her sleep through the user terminal 10. The information related to sleep may include, for example, sleep state information 421 related to changes in sleep stages during sleep.

[0054] For example, the sleep state information 421 may indicate information indicating whether the user's sleep changed to light sleep, normal sleep, deep sleep, REM sleep, etc. The above-described specific description of the sleep state information 421 is merely an example, and the present invention is not limited thereto.

[0055] The user terminal 10 may refer to any type of entity in a system having a mechanism for communication with the computing device 100. For example, the user terminal 10 may include a personal computer (PC), a notebook computer, a mobile terminal, a smartphone, a tablet PC, an AI speaker, an AI TV, a wearable device, etc., and may include all types of terminals that can be connected to a wired / wireless network. The user terminal 10 may also include any server implemented by at least one of an agent, an application programming interface (API), and a plug-in. The user terminal 10 may also include an application source and / or a client application.

[0056] According to an embodiment of the present invention, the external server 20 may be a server that stores information on a plurality of training data for training a neural network. The plurality of training data may include, for example, health checkup information or sleep checkup information.

[0057] For example, the external server 20 may be at least one of a hospital server and an information server, and may be a server that stores information related to a plurality of sleep polymorphism test records, electronic health records, electronic medical records, etc. For example, the sleep polymorphism test records may include information on the breathing and movements of a sleep examination subject during sleep, and information on corresponding sleep diagnosis results (e.g., sleep stages, etc.). The information stored in the external server 20 can be used as training data, verification data, and test data for training the neural network of the present invention.

[0058] The computing device 100 of the present invention can receive health checkup information, sleep checkup information, etc. from the external server 20 and build a learning dataset based on the information. The computing device 100 may perform learning on one or more network functions through the learning dataset, thereby generating a sleep analysis model for calculating sleep state information 421 corresponding to the sleep acoustic information. A configuration for constructing a learning dataset for neural network learning according to the present invention and a learning method using the learning dataset will be described in detail below with reference to FIG.

[0059] The external server 20 may be a digital device equipped with a processor and memory and having computing capabilities, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone. The external server 20 may be a web server that processes services. The types of servers described above are merely examples, and the present invention is not limited thereto.

[0060] According to an embodiment of the present invention, the computing device 100 may acquire sleep acoustic information related to a user's sleep environment and generate sleep state information 421 related to the user's sleep stage based on the sleep acoustic information. Specifically, the computing device 100 may generate the sleep state information 421 by analyzing the sleep acoustic information using a pre-trained neural network model (e.g., a sleep analysis model) configured to include one or more network functions.

[0061] Here, the sleep acoustic information utilized by the computing device 100 for sleep analysis may be related to acoustic information acquired non-invasively during the user's sleep. For example, the sleep acoustic information may include sounds generated when the user turns over in their sleep, sounds related to muscle movements, or sounds related to the user's breathing during sleep. That is, the sleep acoustic information in the present invention may refer to acoustic information related to the user's movement patterns and breathing patterns during sleep.

[0062] In an embodiment, the sleep sound information may be acquired through a user terminal 10 carried by the user. For example, the sleep sound information related to the user's sleep environment may be acquired through a microphone module provided in the user terminal 10.

[0063] Generally, a microphone module installed in a user terminal 10 carried by a user may be configured with MEMS (Micro-Electro Mechanical Systems) since it must be installed in a relatively small user terminal 10. Such a microphone module can be manufactured in a very small size, but may have a lower signal-to-noise ratio (SNR) than a condenser microphone or a dynamic microphone. A low SNR means that the ratio of noise, which is sound that makes it difficult to identify the sound to be identified, is high, and the sound is difficult to identify (i.e., unknown).

[0064] In the present invention, the information to be analyzed may be acoustic information related to the user's breathing and movements acquired during sleep, i.e., sleep acoustic information. Such sleep acoustic information is information about very small sounds (i.e., sounds that are difficult to distinguish), such as the user's breathing and movements, and is acquired together with other sounds in the sleep environment. Therefore, if it is acquired through the above-mentioned microphone module with a low signal-to-noise ratio, it may be very difficult to detect and analyze.

[0065] According to an embodiment of the present invention, the computing device 100 may provide sleep state information 421 based on sleep audio information acquired through a microphone module configured with MEMS. Specifically, the computing device 100 may convert and / or adjust the sleep audio data, which is unclearly acquired due to a lot of noise, to enable analysis, and may perform learning on an artificial neural network using the converted and / or adjusted data.

[0066] When pre-training of the artificial neural network is completed, the trained neural network (e.g., an acoustic analysis model) can acquire the user's sleep state information 421 based on the acquired (e.g., transformed and / or adjusted) data (e.g., spectrogram) corresponding to the sleep acoustic information. Here, the sleep state information 421 may include information related to changes in the user's sleep stages during sleep.

[0067] As a specific example, the sleep state information 421 may include information that the user was in REM sleep at a first time point and was in light sleep at a second time point different from the first time point. In this case, through the sleep state information 421, it is possible to obtain information that the user was in a relatively deep sleep at the first time point and was in a lighter sleep at the second time point.

[0068] That is, when the computing device 100 acquires sleep acoustic information having a low signal-to-noise ratio through a commonly used user terminal for collecting sound (e.g., an AI speaker, a bedroom IoT device, a mobile phone, etc.), it can process the acquired sleep acoustic information into data suitable for analysis and process the processed data to provide sleep state information 421 related to changes in sleep stages.

[0069] This eliminates the need for a microphone that contacts the user's body to capture clear sound, and allows sleep status to be monitored in a typical home environment simply by updating the software, without the need to purchase a separate additional device with a high signal-to-noise ratio, thereby providing increased convenience.

[0070] In an embodiment, the computing device 100 may be a terminal or a server, and may include any type of device. The computing device 100 may be a digital device equipped with a processor and memory, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone. The computing device 100 may also be a web server that processes services. The types of servers described above are merely examples, and the present invention is not limited thereto.

[0071] According to an embodiment of the present invention, the computing device 100 may be a server that provides a cloud computing service. More specifically, the computing device 100 may be a server that provides a cloud computing service, which is a type of Internet-based computing, in which information is processed by another computer connected to the Internet, other than the user's computer.

[0072] The cloud computing service may be a service that stores materials on the Internet and allows users to use the materials or programs they need anytime and anywhere via an Internet connection without having to install them on their own computers, and allows users to easily share and transmit materials stored on the Internet with simple operations and clicks.

[0073] Furthermore, cloud computing services are not simply a service that stores documents on a server on the Internet, but rather a service that allows users to perform desired tasks by utilizing the functions of applications provided on the web without installing a separate program, and allows multiple people to share documents and work together at the same time.

[0074] The cloud computing service may be implemented in at least one form of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, or a container-based cloud server. That is, the computing device 100 of the present invention may be implemented in at least one form of the above-mentioned cloud computing services.

[0075] The specific description of the cloud computing service mentioned above is merely an example, and may include any platform that builds the cloud computing environment of the present invention.

[0076] The specific configuration, technical features, and effects of the computing device 100 of the present invention will be described below with reference to FIGS. 2 to 8. FIG.

[0077] FIG. 2 illustrates a block diagram of a computing device for analyzing a user's sleep state through acoustic information in accordance with an embodiment of the present invention.

[0078] As shown in FIG. 2, the computing device 100 may include a network unit 110, a memory 120, a sensor unit 130, an environment creation unit 140, and a processor 150. The components included in the computing device 100 described above are merely exemplary, and the scope of the present invention is not limited to the components described above. That is, additional components may be included or some of the components described above may be omitted depending on the implementation manner of the embodiments of the present invention.

[0079] According to an embodiment of the present invention, the computing device 100 may include a network unit 110 that transmits and receives data to and from the user terminal 10 and the external server 20. The network unit 110 may transmit and receive data, etc., for performing a method for analyzing a sleep state based on sleep acoustic information according to an embodiment of the present invention, to and from another computing device, server, etc. That is, the network unit 110 may provide a communication function between the computing device 100, the user terminal 10, and the external server 20.

[0080] For example, the network unit 110 may receive sleep screening records and electronic health records for multiple users from a hospital server. Additionally, the network unit 110 may allow information transmission between the computing device 100 and the user terminal 10 and the external server 20 by calling a procedure in the computing device 100.

[0081] The network unit 110 according to an embodiment of the present invention can use various wired communication systems such as a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High Speed ​​DSL (VDSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and a Local Area Network (LAN).

[0082] In addition, the network unit 110 presented in this specification can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0083] In the present invention, the network unit 110 may be configured regardless of the communication method, such as wired or wireless, and may be configured as various communication networks such as a short-range communication network (PAN: Personal Area Network) or a short-range communication network (WAN: Wide Area Network). In addition, the network may be a public World Wide Web (WWW), or may use wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. The technology described herein may be used in other networks as well as the networks mentioned above.

[0084] According to one embodiment of the present invention, the memory 120 may store a computer program for performing a method for analyzing a sleep state through sleep acoustic information according to one embodiment of the present invention, and the stored computer program may be read and driven by the processor 150.

[0085] In addition, the memory 120 may store any type of information generated or determined by the processor 150 and any type of information received by the network unit 110. In addition, the memory 120 may store data related to the user's sleep.

[0086] For example, the memory 120 may temporarily or permanently store input / output data (eg, sleep acoustic information related to the user's sleep environment, sleep state information 421 corresponding to the sleep acoustic information, etc.).

[0087] According to an embodiment of the present invention, the memory 120 may include at least one type of storage medium selected from the group consisting of flash memory, hard disk, micro multimedia card, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device 100 may also operate in association with web storage that performs the storage function of the memory 120 over the Internet. The above description of memory is for illustrative purposes only, and the present invention is not limited thereto.

[0088] According to one embodiment of the present invention, the sensor unit 130 may include one or more environmental sensing modules for acquiring indoor environment information related to the user's sleep environment, including information on at least one of the user's body temperature, indoor temperature, indoor airflow, indoor humidity, indoor acoustics, and indoor illuminance.

[0089] The indoor environment information may be information related to the user's sleep environment, and may serve as a basis for considering the influence of external factors on the user's sleep through the sleep state related to changes in the user's sleep stage.

[0090] The one or more environmental sensing modules may include at least one sensor module selected from the group consisting of a temperature sensor, an airflow sensor, a humidity sensor, an acoustic sensor, and an illuminance sensor, but are not limited thereto, and may further include various sensors that may affect the user's sleep.

[0091] According to an embodiment of the present invention, the computing device 100 may adjust the user's sleep environment via the environment creation unit 140. Specifically, the environment creation unit 140 may include one or more environment creation modules, and may adjust the user's sleep environment by operating an environment creation module related to at least one of temperature, wind direction, humidity, sound, and illuminance based on the external environment creation information received from the processor 150.

[0092] The external environment creation information may be a signal generated by the processor 150 based on the determination of the sleep state according to the change in the user's sleep stage, and may be, for example, a signal to lower the temperature, increase the humidity, lower the illuminance, or lower the sound level in relation to the user's sleep environment. The above-mentioned specific description of the environmental control signal is merely an example, and the present invention is not limited thereto.

[0093] The one or more environment creation modules may include, for example, at least one of a temperature control module, a wind direction control module, a humidity control module, a sound control module, and a light control module. However, without being limited thereto, the one or more environment creation modules may further include various environment creation modules that can bring about changes in the user's sleep environment. That is, the environment creation unit 140 can adjust the user's sleep environment by driving one or more environment creation modules based on the environment control signal of the processor 150.

[0094] According to another embodiment of the present invention, the environment creating unit 140 may be implemented through connection via the Internet of Things (IoT). Specifically, the environment creating unit 140 may be implemented through connection with various devices that can change the indoor environment in relation to the space where the user is positioned for sleep.

[0095] For example, the environment creation unit 140 may be implemented as a smart air conditioner, a smart heater, a smart boiler, a smart window, a smart humidifier, a smart dehumidifier, a smart lighting, etc., based on connection via the Internet of Things. The specific description of the environment creation unit described above is merely an example, and the present invention is not limited thereto.

[0096] According to one embodiment of the present invention, processor 150 may be configured with one or more cores and may include a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), or other processors for data analysis, deep learning, etc.

[0097] The processor 150 may read a computer program stored in the memory 120 and perform data processing for machine learning according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 150 may perform calculations for neural network training.

[0098] The processor 150 can perform calculations for neural network training, such as processing input data for training in deep learning (DL), extracting features from the input data, calculating errors, and updating the weights of the neural network using backpropagation.

[0099] In addition, at least one of the CPU, GPGPU, and TPU of the processor 150 can process network function learning. For example, the CPU and GPGPU can both process network function learning and data classification using the network function.

[0100] In addition, in one embodiment of the present invention, processors of multiple computing devices can be used together to process network function training and data classification using the network functions. Furthermore, the computer program executed on the computing device according to one embodiment of the present invention can be a CPU, GPGPU, or TPU executable program.

[0101] As used herein, a network function may be used interchangeably with an artificial neural network or a neural network. As used herein, a network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of the one or more neural networks.

[0102] As used herein, a model may include a network function. A model may include one or more network functions, in which case the output of the model may be an ensemble of the outputs of the one or more network functions.

[0103] The processor 150 may read a computer program stored in the memory 120 to provide a sleep analysis model according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 150 may perform calculations to calculate sleep analysis information based on sleep sensing data. According to an embodiment of the present invention, the processor 150 may perform calculations to train the sleep analysis model.

[0104] According to one embodiment of the present invention, the processor 150 may generally handle the overall operation of the computing device 100. The processor 150 may process signals, data, information, etc. input or output via the components detailed above, or may run applications stored in the memory 120 to provide or process appropriate information or functions to the user terminal.

[0105] According to one embodiment of the present invention, processor 150 may acquire sleep acoustic information 210 associated with the user's sleep. According to one embodiment of the present invention, acquiring sleep acoustic information 210 may involve acquiring or loading sleep acoustic information stored in memory 120.

[0106] In addition, the acquisition of sleep acoustic information may involve receiving or loading data onto another storage medium based on wired / wireless communication means from another computing device or a separate processing module within the same computing device.

[0107] The sleep acoustic information may be related to acoustic information acquired non-invasively while the user is sleeping. Specifically, the sleep acoustic information may include sounds generated by the user turning over in their sleep, sounds related to muscle movements, or sounds related to the user's breathing while sleeping. That is, the sleep acoustic information in the present invention may refer to acoustic information related to the user's movement patterns and breathing patterns while sleeping.

[0108] According to one embodiment, living environment acoustic information related to the space in which the user is active may be acquired through the user terminal 10 carried by the user, and the processor 150 may acquire sleep acoustic information from the living environment acoustic information.

[0109] Here, the living environment acoustic information may be acoustic information acquired in the user's daily life, such as acoustic information related to cleaning, cooking, watching TV, etc.

[0110] Specifically, the processor 150 may identify a singular point where information of a pre-established pattern is detected in the living environment sound information, where the pre-established pattern information may be related to breathing and movement patterns associated with sleep.

[0111] For example, when awake, the entire nervous system is active, so breathing patterns may be irregular, there may be a lot of body movement, and there may be very little breathing noise due to the lack of neck muscle relaxation.

[0112] On the other hand, when the user is asleep, the autonomic nervous system is stabilized, breathing becomes regular, body movement may be small, and breathing noise may become louder. That is, the processor 150 may identify, as a singular point, a time point at which sound information of a predetermined pattern associated with regular breathing, small body movement, or small breathing noise is detected in the living environment sound information.

[0113] In addition, the processor 150 may acquire sleep sound information based on the life environment sound information acquired based on the identified singular points. The processor 150 may identify singular points associated with the user's sleep time points from the life environment sound information acquired in time series, and acquire sleep sound information based on the identified singular points.

[0114] 3, processor 150 may identify a singular point 201 associated with a time point at which a previously established pattern is identified from living environment sound information 200. In addition, processor 150 may acquire sleep sound information 210 based on sound information acquired after the identified singular point. The sound-related waveforms and singular points in FIG. 3 are merely examples for understanding the present invention, and the present invention is not limited thereto.

[0115] That is, the processor 150 can extract and acquire only the sleep sound information from a huge amount of sound information by identifying specific points related to the user's sleep from the living environment sound information, which can automate the process of the user recording their sleep time, providing convenience and contributing to improving the accuracy of the acquired sleep sound information.

[0116] According to the embodiment, the sleep sound information can be acquired through the user terminal 10 carried by the user. For example, the sleep sound information related to the user's sleep environment can be acquired through a microphone module provided in the user terminal 10.

[0117] Generally, a microphone module installed in a user terminal 10 carried by a user may be configured with a Micro-Electro Mechanical System (MEMS) since it must be installed in a relatively small user terminal 10. Such a microphone module can be manufactured in a very small size, but may have a lower signal-to-noise ratio (SNR) than a condenser microphone or a dynamic microphone. A low SNR means that the ratio of noise, which is sound that makes it difficult to identify the sound to be identified, is high, and the sound is difficult to identify (i.e., unknown).

[0118] In the present invention, the information to be analyzed may be acoustic information related to the user's breathing and movements acquired during sleep, i.e., sleep acoustic information. Such sleep acoustic information is information about very subtle sounds such as the user's breathing and movements, and is acquired along with other sounds during sleep. Therefore, if it is acquired through the above-mentioned microphone module with a low signal-to-noise ratio, it may be very difficult to detect and analyze. Therefore, when sleep acoustic information with a low signal-to-noise ratio is acquired, the processor 150 may process it into data for processing and / or analysis.

[0119] According to one embodiment, the processor 150 may perform pre-processing on the sleep sound information, which may be pre-processing related to noise removal. Specifically, the processor 150 may classify the sleep audio information into one or more audio frames each having a predetermined time unit. The processor 150 may also identify a minimum audio frame having a minimum energy level based on the energy levels of each of the one or more audio frames. The processor 150 may then perform noise reduction on the sleep audio information based on the minimum audio frame.

[0120] For example, processor 150 may classify 30 seconds of sleep acoustic information into one or more very short acoustic frames each having a magnitude of 40 ms. Processor 150 may also compare the magnitudes of each of the acoustic frames associated with the 40 ms magnitude to identify the smallest acoustic frame having the smallest energy level. Processor 150 may then remove the identified smallest acoustic frame component from the entire sleep acoustic information (i.e., 30 seconds of sleep acoustic information).

[0121] 4, for example, the minimum acoustic frame component may be removed from the sleep acoustic information 210 to obtain the preprocessed sleep acoustic information 211. That is, the processor 150 may identify the minimum acoustic frame as a background noise frame and remove it from the original signal (i.e., the sleep acoustic information), thereby performing preprocessing related to noise removal.

[0122] According to an embodiment of the present invention, the processor 150 may perform an analysis on the pre-processed sleep acoustic information to obtain the sleep state information 421. According to an embodiment, the sleep state information 421 may be information related to sleep stages that change during the user's sleep.

[0123] For example, the sleep state information 421 may indicate information on whether the user's sleep changed to light sleep, normal sleep, deep sleep, or REM sleep at each point during the user's eight hours of sleep last night. The above specific description of the sleep state information 421 is merely an example, and the present invention is not limited thereto.

[0124] Specifically, the processor 150 may generate a spectrogram corresponding to the pre-processed sleep sound information. The process of generating a spectrogram corresponding to the pre-processed sleep sound information will be described below with reference to FIGS. 4 and 5.

[0125] The processor 150 may generate a spectrogram corresponding to the pre-processed sleep audio information 211. The processor 150 may perform a fast Fourier transform on the pre-processed sleep audio information 211 to generate a spectrogram corresponding to the pre-processed sleep audio information 211.

[0126] A spectrogram is a visual representation of sound or waves, and may be a combination of waveform and spectrum characteristics. A spectrogram may show differences in amplitude due to changes in the time and frequency axes, using different print densities or display colors.

[0127] According to an embodiment of the present invention, the spectrogram generated by the processor 150 in response to the sleep acoustic information 210 may include a Mel-Spectrogram. The processor 150 may obtain the Mel-Spectrogram through a Mel-Filter Bank for the spectrogram, as shown in FIG.

[0128] Generally, the parts of the human cochlea that vibrate may differ depending on the frequency of audio data. In addition, the human cochlea has the characteristic of being sensitive to frequency changes in the low frequency band but not sensitive to frequency changes in the high frequency band, so a Mel filter bank can be used to obtain a Mel spectrogram from a spectrogram so that the recognition ability for audio data is similar to the characteristics of the human cochlea.

[0129] That is, the Mel filter bank may apply fewer filter banks to lower frequency bands and wider filter banks to higher frequency bands. In other words, processor 150 can obtain a Mel spectrogram by applying the Mel filter bank to a spectrogram to recognize audio data in a manner similar to the characteristics of the human cochlea. The Mel spectrogram may include frequency components that reflect the characteristics of human hearing.

[0130] That is, in the present invention, the spectrogram that is generated in response to sleep acoustic information and that is the subject of analysis using a neural network may include the above-mentioned mel spectrogram.

[0131] In the present invention, the sleep acoustic information may be very quiet because it is related to breathing and body movement, so the processor 150 can convert the sleep acoustic information into a spectrogram 300 and perform analysis on the acoustics.

[0132] In this case, as described above, the spectrogram 300 contains information showing how the frequency spectrum of the sound changes over time, making it easier to identify relatively small sound-related breathing or movement patterns, thereby improving the efficiency of the analysis.

[0133] For example, as shown in Fig. 5, each spectrogram may be configured to have a frequency spectrum with different density depending on various sleep stages. That is, it may be difficult to predict whether a sleep state is at least one of awake, REM sleep, light sleep, and deep sleep based only on a change in the energy level of the sleep audio information, but by converting the sleep audio information into a spectrogram, changes in the spectrum of each frequency can be easily detected, which may enable analysis corresponding to soft sounds (e.g., breathing and body movements).

[0134] Furthermore, processor 150 may process spectrogram 300 as an input to a sleep analysis model to acquire sleep state information 421. Here, the sleep analysis model is a model for acquiring sleep state information 421 related to changes in the user's sleep stage, and may input sleep acoustic information acquired during the user's sleep and output sleep state information 421. In an embodiment, the sleep analysis model may include a neural network model configured via one or more network functions.

[0135] The sleep analysis model is composed of one or more network functions, which may generally be composed of a collection of interconnected computational units that may be referred to as "nodes." Such "nodes" may also be referred to as "neurons." One or more network functions are composed of at least one or more nodes. The nodes (or neurons) that make up one or more network functions may be interconnected by one or more "links."

[0136] Within a neural network, one or more nodes connected via links can form a relative relationship between input and output nodes. The concepts of input and output nodes are relative, and any node that is in an output node relationship with one node can also be in an input node relationship with another node, and vice versa. As mentioned above, the input node-to-output node relationship can be generated around links. One or more output nodes can be connected to one input node via links, and vice versa.

[0137] In a relationship between an input node and an output node connected via a link, the value of the output node may be determined based on data input to the input node. Here, the node interconnecting the input node and the output node may have a weight. The weight may be variable and may be varied by a user or an algorithm so that the neural network performs a desired function. For example, when one or more input nodes are interconnected to one output node via respective links, the output node may determine its output node value based on the value input to the input node connected to the output node and the weight assigned to the link corresponding to each input node.

[0138] As described above, a neural network has one or more nodes interconnected via one or more links, forming a relationship between input nodes and output nodes within the neural network. The characteristics of a neural network may be determined by the number of nodes and links within the neural network, the correlation between the nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weights between the links, the two neural networks can be recognized as different from each other.

[0139] Some of the nodes constituting a neural network can be configured as a layer based on their distance from the initial input node. For example, a set of nodes whose distance from the initial input node is n can constitute n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach the node from the initial input node. However, this definition of a layer is arbitrary for the purpose of explanation, and the number of layers in a neural network can be defined in a different manner than that described above. For example, a node's layer can be defined by its distance from the final output node.

[0140] An initial input node may refer to one or more nodes in a neural network to which data is directly input without passing through a link in relation to other nodes. Alternatively, it may refer to a node in a neural network that does not have any other input nodes connected to it via a link in relation to other nodes based on the links. Similarly, a final output node may refer to one or more nodes in a neural network that do not have any output nodes in relation to other nodes. Furthermore, a hidden node may refer to a node constituting a neural network that is neither an initial input node nor a final output node. A neural network according to an embodiment of the present invention may have more nodes in an input layer than in a hidden layer closer to the output layer, and the number of nodes may decrease as one moves from the input layer to the hidden layer.

[0141] A neural network may include one or more hidden layers. Hidden nodes in a hidden layer can receive the output of the previous layer and the output of surrounding hidden nodes as input. The number of hidden nodes in each hidden layer may be the same or different. The number of nodes in the input layer may be determined based on the number of data fields in the input data and may be the same or different from the number of hidden nodes. Input data input to the input layer can be operated by hidden nodes in the hidden layer and output by a fully connected layer (FCL), which is the output layer.

[0142] According to an embodiment of the present invention, the sleep analysis model may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep state information 421. A process of obtaining the sleep state information 421 using the sleep analysis model will be described below with reference to FIG. 6.

[0143] According to an embodiment, the feature extraction model 410 can analyze the time-series frequency patterns of the spectrogram 300 to extract features related to respiratory sounds, breathing patterns, and movement patterns. In one embodiment, the feature extraction model 410 can be a neural network model (e.g., an autoencoder) pre-trained via a training dataset, where the training dataset can include multiple spectrograms and multiple sleep stage information corresponding to each spectrogram.

[0144] In one embodiment, the feature extraction model 410 may be configured via an encoder in an autoencoder trained via a training data set. The autoencoder may be trained via an unsupervised learning method. The autoencoder may be trained via the training data set to output output data similar to input data.

[0145] In more detail, only the core feature data (or features) of the spectrogram input through the encoder can be learned through the hidden layer, and the remaining information can be lost. In this case, the output data of the hidden layer during the decoding process through the decoder may be an approximation of the input data (i.e., the spectrogram) rather than a perfect copy. In other words, the autoencoder can learn to adjust the weights so that the output data and the input data are as similar as possible.

[0146] In some embodiments, each of the spectrograms included in the training data set may be tagged with sleep stage information, and each of the spectrograms may be input to an encoder, and the output corresponding to each spectrogram may be matched with the tagged sleep stage information and stored.

[0147] Specifically, when an encoder is used to input a first training dataset (i.e., multiple spectrograms) tagged with first sleep stage information (e.g., light sleep) to the encoder, features associated with the encoder output for that input can be matched and stored with the first sleep stage information.

[0148] In an embodiment, one or more features associated with the encoder output may be represented in a vector space. In this case, feature data output corresponding to each of the first training datasets may be located relatively close to each other in the vector space because they are output via spectrograms associated with the first sleep stage. That is, the encoder may be trained so that multiple spectrograms corresponding to each sleep stage output similar features.

[0149] In the case of an encoder, it can be trained to extract features that allow a decoder to well restore input data. Therefore, the trained autoencoder of the feature extraction model 410 can be implemented through an encoder to extract features (i.e., multiple features) that allow the input data (i.e., spectrogram) to be well restored.

[0150] The encoder that configures the feature extraction model 410 through the above-described learning process can extract features corresponding to a spectrogram (e.g., a spectrogram transformed corresponding to sleep acoustic information) as input.

[0151] In an embodiment, the processor 150 may extract features by processing the spectrogram 300 generated corresponding to the sleep acoustic information 210 as an input to the feature extraction model 410. Here, since the sleep acoustic information 210 is time-series data acquired in a time-series manner while the user is sleeping, the processor 150 may divide the spectrogram 300 into predetermined epochs.

[0152] For example, the processor 150 may divide the spectrogram 300 corresponding to the sleep sound information 210 into 30-second units to obtain a plurality of spectrograms 300n. For example, if sleep sound information is obtained during a user's 7-hour (i.e., 420-minute) sleep, the processor 150 may divide the spectrogram into 30-second units to obtain 140 spectrograms 300n. The specific numerical values ​​for the sleep duration, the time unit for dividing the spectrogram, and the number of divisions described above are merely examples, and the present invention is not limited thereto.

[0153] The processor 150 processes each of the divided spectrograms 300n as an input to the feature extraction model 410, and can extract a plurality of features 411 corresponding to each of the plurality of spectrograms 300n. For example, if the number of spectrograms is 140, the number of features 411 extracted by the feature extraction model 410 may also be 140. The specific numerical values ​​related to the number of spectrograms and features described above are merely examples, and the present invention is not limited thereto.

[0154] 6, the processor 150 may process the plurality of features 411 output through the feature extraction model 410 as input to a feature classification model 420 to obtain sleep state information 421. In an embodiment, the feature classification model 420 may be a neural network model pre-trained to predict sleep stages corresponding to the features.

[0155] For example, the feature classification model 420 may be configured to include a fully connected layer and may classify a feature into at least one of the sleep stages. For example, when a first feature corresponding to a first spectrogram is input, the feature classification model 420 may classify the first feature into light sleep.

[0156] In one embodiment, the feature classification model 420 may perform multi-epoch classification, which uses spectrograms associated with various epochs as input to predict sleep stages for various epochs. The multi-epoch classification does not provide a single sleep stage analysis information corresponding to a spectrogram of a single epoch (i.e., one spectrogram corresponding to 30 seconds), but may use spectrograms corresponding to multiple epochs (i.e., a combination of spectrograms each corresponding to 30 seconds) as input to simultaneously estimate various sleep stages (e.g., changes in sleep stages over time).

[0157] For example, breathing or movement patterns change more slowly than electroencephalogram (EEG) signals or other biological signals, so accurate sleep stage estimation may only be possible by observing how the patterns change between past and future points in time.

[0158] As a specific example, the feature classification model 420 may input 40 spectrograms (e.g., 40 spectrograms each corresponding to 30 seconds) and perform prediction on the central 20 spectrograms. That is, even if spectrograms 1 to 40 are examined in detail, sleep stages may be predicted through classification corresponding to spectrograms 10 to 20. The specific numerical values ​​for the number of spectrograms described above are merely examples, and the present invention is not limited thereto.

[0159] That is, in the process of estimating sleep stages, rather than performing sleep stage prediction for each single spectrogram, spectrograms corresponding to multiple epochs are used as input so that all information related to the past and future can be taken into consideration, thereby improving the accuracy of the output.

[0160] As described above, processor 150 may perform preprocessing on sleep acoustic information acquired with a low signal-to-noise ratio and acquire a spectrogram based on the preprocessed sleep acoustic information. In this case, the spectrogram may be transformed to facilitate analysis of breathing or movement patterns associated with relatively small sounds.

[0161] In addition, the processor 150 may generate sleep state information 421 based on the spectrogram acquired using a sleep analysis model including a feature extraction model 410 and a feature classification model 420. In this case, the sleep analysis model may perform sleep stage prediction using spectrograms corresponding to multiple epochs as input so that information related to both the past and future can be taken into consideration, thereby outputting more accurate sleep state information 421.

[0162] According to an embodiment of the present invention, the processor 150 may perform data augmentation based on the pre-processed sleep acoustic information, such that the sleep analysis model can output the sleep state information 421 robustly even for sounds measured from various domains (e.g., different bedrooms, different microphones, different placement locations, etc.). In embodiments, the data augmentation may include at least one of pitch shifting, gaussian noise, loudness control, dynamic range control, and spec augmentation.

[0163] According to one embodiment, processor 150 may perform data enhancement related to pitch shifting based on the sleep acoustic information. For example, processor 150 may perform data enhancement by adjusting the pitch of the sound, such as raising or lowering the pitch of the sound at predetermined intervals.

[0164] The processor 150 can perform not only pitch shifting but also Gaussian noise, which performs data enhancement through noise-related correction; loudness control, which performs data enhancement by correcting the sound so that the sound quality is maintained even when the volume is changed; dynamic range control, which performs data enhancement by adjusting the dynamic range, which is the ratio between the maximum and minimum amplitudes of the sound measured in dB; and spec augmentation, which is related to the increase in the specifications of the sound.

[0165] That is, the processor 150 can improve the accuracy of sleep stage prediction by enabling the sleep analysis model to perform robust recognition in response to sleep sounds acquired in various environments through data augmentation of the acoustic information (i.e., sleep acoustic information) that forms the basis of the analysis of the present invention.

[0166] According to an embodiment of the present invention, the processor 150 may control the environment creation module. Specifically, the processor 150 may generate external environment creation information related to illuminance adjustment, and may control the illuminance adjustment operation of the environment creation module by determining to transmit the external environment creation information to the environment creation module.

[0167] According to an embodiment, light may be one of the major factors that can affect sleep quality. For example, the illuminance, color, and exposure level of light can have a positive or negative effect on sleep quality. Therefore, the processor 150 can adjust the illuminance to improve the user's sleep quality.

[0168] For example, the processor 150 can monitor the state before and after falling asleep and adjust the illumination to effectively wake up the user. That is, the processor 150 can grasp the sleep state (e.g., sleep stage) and automatically adjust the illumination to maximize the quality of sleep.

[0169] In one embodiment, the processor 150 may receive sleep plan information from the user terminal 10. The sleep plan information is information generated by the user via the user terminal 10 and may include, for example, information about bedtime and wake-up time. The processor 150 may generate external environment creation information based on the sleep plan information.

[0170] For example, the processor 150 may identify the user's bedtime through the sleep plan information and generate external environment creation information based on the bedtime. For example, the processor 150 may generate external environment creation information for providing 3000K white light with an illuminance of 30 lux based on the position of the bed 20 minutes before bedtime. In other words, the processor 150 may generate illuminance that induces the user to fall asleep naturally in relation to bedtime.

[0171] In addition, for example, the processor 150 may identify the user's wake-up time through the sleep plan information and generate external environment creation information based on the wake-up time. For example, the processor 150 may generate external environment creation information that gradually increases the illuminance of 3000K white light from 0 lux to 250 lux based on the position of the bed 30 minutes before the wake-up time. Such external environment creation information may encourage the user to wake up naturally and refreshed according to the desired wake-up time.

[0172] In addition, the processor 150 may determine to transmit the external environment creation information to the environment creation module. That is, the processor 150 may generate external environment creation information that allows the user to easily fall asleep or wake up naturally when going to bed or waking up based on the sleep plan information, thereby improving the quality of the user's sleep.

[0173] In another embodiment, the processor 150 may generate external environment creation information based on the sleep state information 421. In an embodiment, the sleep state information 421 may include information regarding changes in the user's sleep stages, which are acquired over time through analysis of sleep acoustic information.

[0174] For example, when the processor 150 determines that the user has entered a sleep stage (e.g., light sleep) through the user's sleep state information 421, the processor 150 may generate external environment creation information for minimizing illuminance to create a dark room environment without light. That is, by creating optimal illuminance for each sleep stage of the user, i.e., an optimal sleep environment, the efficiency of the user's sleep may be improved.

[0175] In addition, the processor 150 may generate external environment creation information to provide appropriate illumination according to changes in the user's sleep stage during sleep. For example, various external environment creation information may be generated according to changes in the sleep stage, such as providing subtle red light when changing from light sleep to deep sleep, or reducing illumination or providing blue light when changing from REM sleep to light sleep. This may maximize the quality of the user's sleep by automatically considering not only the situation before sleep or immediately after waking up but also the situation during sleep, thereby considering the entire sleep experience rather than just a part of it.

[0176] In a further embodiment, the processor 150 may generate recommended sleep plan information based on the sleep state information 421. Specifically, the processor 150 may obtain information regarding changes in the user's sleep stage (e.g., sleep cycle) through the sleep state information 421, and may set an expected wake-up time based on such information.

[0177] For example, a typical sleep cycle during a day may include light sleep, deep sleep, pre-sleep, and REM sleep stages. The processor 150 may determine that the time after REM sleep is the time when the user can wake up most refreshed, and may generate recommended sleep plan information by determining a wake-up time after the REM time. The processor 150 may also generate external environment creation information based on the recommended sleep plan information and determine to transmit the external environment creation information to the environment creation module. Thus, the user may wake up naturally according to the recommended sleep plan information recommended by the processor 150. This is because the processor 150 recommends a wake-up time for the user according to changes in the user's sleep stage, and this may be a time when the user's fatigue level is minimized, which may have the advantage of improving the user's sleep efficiency.

[0178] FIG. 7 is a flow chart illustrating an example of a method for analyzing a user's sleep state through acoustic information according to an embodiment of the present invention.

[0179] According to one embodiment of the present invention, the method may include a step of acquiring sleep acoustic information associated with a user's sleep (S100).

[0180] According to an embodiment of the present invention, the method may include a step of performing pre-processing on sleep acoustic information (S200).

[0181] According to an embodiment of the present invention, the method may include a step of acquiring sleep state information 421 by performing an analysis on the pre-processed sleep acoustic information (S300).

[0182] 7 may be reordered, and at least one step may be omitted or added, as necessary. That is, the above-described steps are merely one embodiment of the present invention, and the scope of the present invention is not limited thereto.

[0183] FIG. 8 is a schematic diagram illustrating network functions associated with one embodiment of the present invention.

[0184] A deep neural network (DNN) can refer to a neural network that contains multiple hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to understand the latent structures of data, i.e., the latent structures of photos, text, video, audio, and music (e.g., what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the audio, etc.).

[0185] The deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a generative adversarial network (GAN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q-network, a U-network, a Siamese network, etc. The above description of the deep neural network is merely an example, and the present invention is not limited thereto.

[0186] In one embodiment of the present invention, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network for generating output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between the input and output layers.

[0187] The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and may be reduced and expanded symmetrically from the bottleneck layer to the output layer (symmetric to the input layer). The nodes in the dimensionality reduction layer and the dimensionality restoration layer may or may not be symmetric.

[0188] Autoencoders can perform non-linear dimensionality reduction: the number of input and output layers may correspond to the number of sensors remaining after preprocessing of the input data.

[0189] In an autoencoder structure, the number of nodes in the hidden layer included in the encoder may decrease as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, it may not be possible to transmit a sufficient amount of information, so the number of nodes may be kept at a certain number or more (e.g., more than half of the number in the input layer).

[0190] Neural networks can be trained using at least one of supervised learning, unsupervised learning, and semi-supervised learning. The purpose of training a neural network is to minimize the error in the output.

[0191] In neural network training, training data is repeatedly input into the neural network, the error between the neural network output and the target for the training data is calculated, and the error of the neural network is backpropagated from the output layer to the input layer of the neural network in a direction to reduce the error, thereby updating the weight value of each node of the neural network.

[0192] In supervised learning, training data is used in which each training data is labeled with a correct answer (i.e., labeled training data), while in unsupervised learning, each training data may not be labeled with a correct answer.

[0193] For example, in the case of supervised learning for data classification, the training data may be data in which each training data is labeled with a category. The labeled training data is input to a neural network, and an error can be calculated by comparing the output (category) of the neural network with the label of the training data.

[0194] As another example, in unsupervised learning for data classification, input training data can be compared with neural network output to calculate an error, which can then be backpropagated through the neural network in the backward direction (i.e., from the output layer to the input layer) to update the connection weights of each node in each layer of the neural network.

[0195] The amount of change in the connection weight value of each node to be updated may be determined by a learning rate. The neural network calculation for input data and backpropagation of errors may constitute a learning cycle (epoch). The learning rate may be applied differently depending on the number of iterations of the neural network learning cycle.

[0196] For example, in the early stages of neural network learning, a high learning rate can be used to quickly ensure a certain level of performance, thereby increasing efficiency, and in the later stages of learning, a lower learning rate can be used to increase accuracy.

[0197] In neural network training, training data may generally be a subset of actual data (i.e., data to be processed using the trained neural network), and therefore, there may be a training cycle in which the error for the training data decreases but the error for the actual data increases. Overfitting is a phenomenon in which excessive learning on the training data increases the error for the actual data.

[0198] For example, a neural network that has learned cats by showing them yellow cats may be unable to recognize cats that are not yellow. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting.

[0199] To prevent overfitting, methods such as increasing the amount of training data, regularization, and dropout, which omits some of the network nodes during the training process, may be applied.

[0200] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably (hereinafter, the term neural network will be used interchangeably). A data structure may include a neural network. And, a data structure including a neural network may be stored on a computer-readable medium.

[0201] The data structure including the neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for training the neural network. The data structure including the neural network may include any of the components of the disclosed structures.

[0202] That is, a data structure including a neural network may include all or any combination of data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for training the neural network, etc.

[0203] In addition to the above-described configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any type of data that is used or generated in the computational process of the neural network, and is not limited to the above.

[0204] The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a collection of interconnected computational units that may generally be referred to as nodes. Such nodes may also be referred to as neurons. A neural network is composed of at least one or more nodes.

[0205] The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in a software module executed by hardware, or in a combination thereof. The software module may reside in Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable storage medium known in the art to which the present invention pertains.

[0206] Components of the present invention may be embodied as a program (or application) stored on a medium for execution in conjunction with a computer, which is hardware. Components of the present invention may be implemented as software programming or software elements. Similarly, embodiments include various algorithms embodied in a combination of data structures, processes, routines, or other programming constructs, and may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, etc. Functional aspects may be embodied as algorithms executed by one or more processors.

[0207] Those skilled in the art will appreciate that the various illustrative logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be embodied by electronic hardware, various forms of program or design code (for convenience, referred to herein as "software"), or a combination of all of these.

[0208] To clearly illustrate this interoperability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in connection with their functionality. Whether such functionality is embodied as hardware or software depends upon the particular application and design constraints imposed on the overall system.

[0209] Those of ordinary skill in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0210] The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or media accessible by any computer-readable device.

[0211] For example, computer-readable media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, wireless channels and various other media capable of storing, carrying, and / or transmitting instructions and / or data.

[0212] It is understood that the specific order or hierarchy of steps in the processes presented is an example of a sample approach. Based on design priorities, it is understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present invention. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

[0213] The description of the embodiments presented is provided to enable any person skilled in the art to use or practice the invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the invention. The present invention is not intended to be limited to the embodiments presented herein, but rather is to be accorded the widest scope consistent with the principles and novel features disclosed herein. [Explanation of symbols]

[0214] 10: User terminal 20: External server 100: Computing equipment 110: Network Department 120:Memory 130: Sensor unit 140: Environmental Creation Module 150: Processor 200: Living environment acoustic information 201: Singularity related to the point in time when an already established pattern is identified 210: Sleep acoustic information 211: Preprocessed sleep acoustic information 300:Spectrogram 300n: Multiple spectrograms 410: Feature extraction model 411: Multiple Features 420: Feature Classification Model 421: Sleep status information S100: Acquiring sleep acoustic information related to user's sleep S200: Performing Preprocessing on Sleep Acoustic Information S300: Acquiring sleep state information by analyzing pre-processed sleep acoustic information

Claims

1. 1. A method for analyzing a user's sleep state based on sleep acoustic information, comprising: acquiring sleep acoustic information of a user through a microphone module provided in a user terminal; performing pre-processing on the sleep acoustic information by a processor; converting, by the processor, the pre-processed sleep acoustic information into a plurality of spectrograms for visualization; processing, by the processor, the plurality of spectrograms with inputs of a sleep analysis model, including a feature extraction model and a feature classification model, to obtain sleep state information; generating, by the processor, external environment creation information based on the sleep state information; transmitting the external environment creation information to one or more environment creation units by a transmission module; Including, the sleep acoustic information includes sounds associated with breathing and body movements; the one or more environment creation units are configured to create a sleeping environment for the user by operating one or more environment creation modules based on the external environment creation information; the feature extraction model includes an encoder pre-trained via an autoencoder; the feature classification model includes one or more neural networks including a fully connected layer (FCL); the sleep analysis model is trained using training data including a plurality of training spectrograms tagged with sleep stage information; The acquired sleep state information includes sleep stage information related to the depth of the user's sleep, each of the plurality of spectrograms corresponds to a predetermined epoch; the one or more neural networks included in the feature extraction model are configured to extract a plurality of features based on each of the plurality of spectrograms; The one or more neural networks included in the feature classification model are configured to estimate a plurality of sleep stages based on the plurality of features.

2. 2. The method of claim 1 , wherein at least one of the one or more neural networks of the feature extraction model and the one or more neural networks of the feature classification model comprises at least one of a CNN, an RNN, a GAN, an RBM, a Q-network, a U-network, or a Siamese network.

3. 2. The method of claim 1, wherein the plurality of features are extracted from each of the plurality of spectrograms by the one or more neural networks of the feature extraction model based on one or more patterns associated with at least one of respiratory sounds, breathing patterns, and movement patterns.

4. The method of claim 1 , wherein each of the plurality of sleep stages is estimated based on a time series of at least one feature included in the plurality of features.

5. The method of claim 1 , wherein the one or more ambiance creation modules are associated with at least one of temperature, wind direction, humidity, and illuminance of the sleeping environment.

6. A method for analyzing a user's sleep state based on sleep acoustic information acquired by a microphone module provided in a user terminal, receiving the sleep acoustic information by a receiving module; performing pre-processing on the sleep acoustic information by a processor; converting, by the processor, the pre-processed sleep acoustic information into a plurality of spectrograms for visualization; processing, by the processor, the plurality of spectrograms with inputs of a sleep analysis model, including a feature extraction model and a feature classification model, to obtain sleep state information; generating, by the processor, external environment creation information based on the sleep state information; transmitting the external environment creation information to one or more environment creation units by a transmission module; Including, the sleep sound information includes sounds associated with the user's breathing and body movements; the one or more environment creation units are configured to create a sleeping environment for the user by operating one or more environment creation modules based on the external environment creation information; the feature extraction model includes an encoder pre-trained via an autoencoder; the feature classification model includes one or more neural networks including a fully connected layer (FCL); the sleep analysis model is trained using training data including a plurality of training spectrograms tagged with sleep stage information; The acquired sleep state information includes sleep stage information related to the depth of the user's sleep, each of the plurality of spectrograms corresponds to a predetermined epoch; the one or more neural networks included in the feature extraction model are configured to extract a plurality of features based on each of the plurality of spectrograms; The one or more neural networks included in the feature classification model are configured to estimate a plurality of sleep stages based on the plurality of features.

7. 7. The method of claim 6, wherein at least one of the one or more neural networks of the feature extraction model and the one or more neural networks of the feature classification model comprises at least one of a CNN, an RNN, a GAN, an RBM, a Q-network, a U-network, or a Siamese network.

8. 7. The method of claim 6, wherein the plurality of features are extracted from each of the plurality of spectrograms by the one or more neural networks of the feature extraction model based on one or more patterns associated with at least one of respiratory sounds, breathing patterns, and movement patterns.

9. The method of claim 6 , wherein each of the plurality of sleep stages is estimated based on a time series of at least one feature included in the plurality of features.

10. The method of claim 6 , wherein the one or more ambience creation modules are associated with at least one of temperature, wind direction, humidity, and illuminance of the sleeping environment.

11. 1. An apparatus for analyzing a user's sleep state based on sleep acoustic information, comprising: Memory and a transmitting module; a microphone module configured to acquire the sleep acoustic information associated with a user's sleep; a processor; Including, the processor is configured to perform preprocessing on the sleep acoustic information, convert the preprocessed sleep acoustic information into a plurality of spectrograms for visualization, process the plurality of spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to obtain sleep state information, and generate external environment creation information based on the sleep state information; the sleep sound information includes sounds associated with the user's breathing and body movements; The processor is configured to transmit the external environment creation information to one or more environment creation units via the transmission module; the one or more environment creation units are configured to create a sleeping environment for the user by operating one or more environment creation modules based on the external environment creation information; the feature extraction model includes an encoder; the feature classification model includes one or more neural networks including a fully connected layer (FCL); the sleep analysis model is trained using training data including a plurality of training spectrograms tagged with sleep stage information; The acquired sleep state information includes sleep stage information related to the depth of the user's sleep, each of the plurality of spectrograms corresponds to a predetermined epoch; the one or more neural networks included in the feature extraction model are configured to extract a plurality of features based on each of the plurality of spectrograms; The one or more neural networks included in the feature classification model are configured to estimate a plurality of sleep stages based on the plurality of features.

12. 12. The apparatus of claim 11, wherein at least one of the one or more neural networks of the feature extraction model and the one or more neural networks of the feature classification model comprises at least one of a CNN, an RNN, a GAN, an RBM, a Q-network, a U-network, and a Siamese network.

13. 12. The apparatus of claim 11, wherein the plurality of features are extracted from each of the plurality of spectrograms by the one or more neural networks of the feature extraction model based on one or more patterns associated with at least one of respiratory sounds, breathing patterns, and movement patterns.

14. The device of claim 11 , wherein each of the plurality of sleep stages is estimated based on a time series of at least one feature included in the plurality of features.

15. The device of claim 11 , wherein the one or more ambiance creation modules are associated with at least one of temperature, wind direction, humidity, and illuminance of the sleeping environment.

16. A server including a sleep analysis model for analyzing a user's sleep state based on sleep acoustic information acquired through a microphone module provided to a user terminal, a receiving module; a transmitting module; Memory and a processor; Including, the processor is configured to receive the sleep acoustic information from the user terminal via the receiving module; the processor is configured to perform preprocessing on the sleep acoustic information, convert the preprocessed sleep acoustic information into a plurality of spectrograms for visualization, process the plurality of spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to obtain sleep state information, and generate external environment creation information based on the sleep state information; the sleep sound information includes sounds associated with the user's breathing and body movements; The processor is configured to transmit the external environment creation information to one or more environment creation units via the transmission module; the one or more environment creation units are configured to create a sleeping environment for the user by operating one or more environment creation modules based on the external environment creation information; the feature extraction model includes an encoder; the feature classification model includes one or more neural networks including a fully connected layer (FCL); the sleep analysis model is trained using training data including a plurality of training spectrograms tagged with sleep stage information; The acquired sleep state information includes sleep stage information related to the depth of the user's sleep, each of the plurality of spectrograms corresponds to a predetermined epoch; the one or more neural networks included in the feature extraction model are configured to extract a plurality of features based on each of the plurality of spectrograms; The one or more neural networks included in the feature classification model are configured to estimate a plurality of sleep stages based on the plurality of features.

17. 17. The server of claim 16, wherein at least one of the one or more neural networks of the feature extraction model and the one or more neural networks of the feature classification model includes at least one of a CNN, an RNN, a GAN, an RBM, a Q-network, a U-network, and a Siamese network.

18. 17. The server of claim 16, wherein the plurality of features are extracted from each of the plurality of spectrograms by the one or more neural networks of the feature extraction model based on one or more patterns associated with at least one of respiratory sounds, breathing patterns, and movement patterns.

19. The server of claim 16 , wherein each of the plurality of sleep stages is estimated based on a time series of at least one feature included in the plurality of features.

20. The server of claim 16 , wherein the one or more ambiance creation modules are associated with at least one of temperature, wind direction, humidity, and illuminance of the sleeping environment.

21. A computer program stored on a computer-readable recording medium so that the computer program can be combined with a computer that is hardware and can perform any one of the methods of claims 1 to 5.

22. A computer program stored on a computer-readable recording medium so that the computer program can be combined with a computer that is hardware and can perform the method of any one of claims 6 to 10.

Images