Method, device, and system for providing content including sleep state information generated on basis of information acquired from sleep environment of user

Abstract

The present invention relates to a method for providing interpretation content of sleep data of a user, the method comprising the steps of: acquiring sleep information of a user; generating sleep state information of the user on the basis of the acquired sleep information of the user; generating sleep data interpretation content of the user on the basis of the generated sleep state information; and outputting the generated sleep data interpretation content of the user, wherein the sleep data interpretation content of the user is expressed in at least one of a numerical manner and a non-numerical manner.

Description

Method, device and system for providing content including sleep state information generated based on information obtained from a user's sleep environment

[0001] The present invention relates to a method, device, and system for providing content including sleep state information generated based on information obtained from a user's sleep environment.

[0002] There are many ways to maintain and improve health, such as exercise and diet, but the most important thing is to manage sleep well, which takes up about 30% of the day.

[0003] However, despite the simple replacement of labor by machines and the leisurely lifestyle, modern people are unable to get enough sleep due to irregular eating and living habits and stress, and suffer from sleep disorders such as insomnia, hypersomnia, sleep apnea syndrome, nightmares, night terrors, and sleepwalking.

[0004] According to the National Health Insurance Corporation, the number of patients with sleep disorders in Korea increased by an average of approximately 8% annually from 2014 to 2018, and approximately 570,000 patients received treatment for sleep disorders in Korea in 2018.

[0005] As sleep is recognized as an important factor affecting physical and mental health, interest in sleep is increasing. However, to improve sleep disorders, a visit to a specialized medical institution is required, separate examination costs are required, and ongoing management is difficult, so users' efforts in treatment are insufficient.

[0006] Korean Patent Publication No. 10-2023-0012133 discloses an electronic device that provides a user interface based on sleep states and a method for operating the same. It discloses that abnormal REM sleep events can be detected and a user interface based on specified action information provided in response to the detection.

[0007] However, since the conventional technology detects abnormal REM sleep stages by examining the entire sleep stage, there is a concern that the accuracy of the measurement of the sleep stage may be low, and since the basic unit of measurement is composed of minutes, there is a concern that sleep-related events may not be detected in real time.

[0008] Meanwhile, FIGS. 20a to 20e are diagrams showing hypnogram graphs of sleep stage information expressed in a conventional sleep measurement interface.

[0009] However, since the conventional technology expresses the graph representing the sleep stages by expressing the shapes assigned to the first and second sleep stages continuously rather than discretely, there is a concern that this may lead to the misunderstanding that another sleep stage must be passed in the process of entering the second sleep stage from the first sleep stage.

[0010] In addition, as shown in FIG. 20c or FIG. 20d, even if the shape representing the waking stage was displayed as if it were separated from the shapes representing other sleep stages, there was a problem in that it was difficult to clearly understand which sleep stage occurred at that time because the shapes corresponding to the sleep stages were displayed continuously in the areas allocated to other sleep stages.

[0011] Accordingly, research is currently being conducted to intuitively represent graphs of sleep status information, including sleep stages.

[0012] Meanwhile, to accurately measure sleep, polysomnography or home polysomnography is required. However, because human sleep fluctuates significantly from day to day, conventional methods of sleep measurement make it difficult to consistently monitor daily sleep. In particular, because sleep duration and stages can fluctuate due to various factors (e.g., lifestyle habits), sleep analysis across multiple days (or multiple sleep sessions) has been difficult.

[0013] Additionally, there has been a demand for creating or providing a graphical user interface that allows one to analyze sleep over multiple days (or multiple sleep sessions) at a glance.

[0014] The present invention was conceived in consideration of the problems of the prior art described above. The object of the present invention is to provide a graphical user interface that displays information about a user's sleep status using information about the user's sleep obtained through a sleep sensor. Furthermore, the present invention aims to utilize sleep sound information to enhance the accuracy of sleep analysis, thereby providing users with useful sleep-related information.

[0015] The present invention has been conceived in consideration of the problems of the prior art described above. The object of the present invention is to provide a method for analyzing a user's sleep based on at least one of sensing information obtained through a sensor device and the user's sleep information, and to provide a graphical user interface including the analyzed information. Furthermore, the present invention aims to increase the accuracy of sleep analysis and provide useful sleep-related information to the user.

[0016] Another object of the present invention is to provide a method for providing a graphical user interface that displays sleep state information generated based on sleep information acquired during one or more sleep sessions.

[0017] The present invention provides an automated sleep measurement start trigger. Specifically, the present invention provides a system that automatically initiates sleep measurement without any separate user intervention, and can provide a function that automatically initiates sleep measurement based on a broadcast trigger for sleep measurement (e.g., charging start, display OFF, etc.) and a sleep analysis trigger (e.g., decreased movement, cessation of touch input, etc.).

[0018] The present invention can initiate sleep measurement by detecting not only alarms based on time reservations, but also user-initiated events (e.g., swiping) and involuntary events (e.g., decreased movement, cessation of touch).

[0019] The present invention automatically records the start and end points of sleep measurement, thereby enabling the user to collect accurate sleep data without separate operation.

[0020] The present invention solves the problem of delayed alarm or trigger execution due to the battery optimization function of the Android system, and enables execution of a sleep measurement trigger even in low power mode.

[0021] The present invention has been conceived in consideration of the problems of the prior art described above, and the purpose of the present invention is to analyze a user's sleep state based on information acquired from a sleep environment through a sleep sensor, and to generate and provide content related to the user's sleep state information. Furthermore, the present invention also generates and provides content related to the user's sleep state information using generative artificial intelligence. Furthermore, the present invention utilizes sleep information to increase the accuracy of sleep analysis, thereby providing useful sleep-related content to the user and improving the quality of sleep.

[0022] A method for providing a graphical user interface that displays information about a user's sleep according to one embodiment of the present invention may include the steps of: obtaining the user's sleep information; generating the user's sleep state information based on the obtained user's sleep information; generating a sleep state information graph that displays sleep state information over time based on the generated sleep state information; generating a graphical user interface including the sleep state information graph; and outputting a graphical user interface including the generated sleep state information graph.

[0023] The above sleep state information may include at least one of sleep stage information, sleep stage probability information, sleep event information, and sleep event probability information.

[0024] The step of generating the sleep state information graph may further include the step of generating at least one of the stacked sleep stage graphs "G stacked sleep event graphs" based on at least one of the occurrence frequency of sleep stages and the occurrence frequency of sleep events included in the sleep state information generated during a plurality of sleep sessions.

[0025] The graphical user interface may include a plurality of areas, each area corresponding to a different type of sleep state information.

[0026] The above multiple areas can be expressed so as to be distinct from each other, each corresponding to the sleep state information.

[0027] The graphical user interface further includes an area indicating that the sleep session has ended, and the area indicating that the sleep session has ended can be expressed to be distinct from the plurality of areas.

[0028] The step of generating the sleep state information may further include a step of generating the sleep state information for a time corresponding to one or more epochs.

[0029] The above epoch may be set to data corresponding to 30-second units.

[0030] A device for providing a graphical user interface that displays information about a user's sleep according to one embodiment of the present invention includes: an acquisition unit that acquires the user's sleep information; one or more processors; and an output unit, wherein the processor generates sleep state information of the user based on the acquired sleep information of the user, generates a sleep state information graph that displays sleep state information over time based on the generated sleep state information, and generates a graphical user interface including the sleep state information graph, and the output unit can output a graphical user interface including the generated sleep state information graph.

[0031] The above sleep state information may include at least one of sleep stage information, sleep stage probability information, sleep event information, and sleep event probability information.

[0032] The processor may generate at least one of a stacked sleep stage graph "G stacked sleep event graph" based on at least one of a frequency of occurrence of sleep stages and a frequency of occurrence of sleep events included in the sleep state information generated during a plurality of sleep sessions.

[0033] The graphical user interface may include a plurality of areas, each area corresponding to a different type of sleep state information.

[0034] The above multiple areas can be expressed so as to be distinct from each other, each corresponding to the sleep state information.

[0035] The graphical user interface may further include an area indicating that the sleep session has ended.

[0036] The area indicating that the above sleep session has ended may be expressed to be distinct from the above plurality of areas.

[0037] The processor may generate the sleep state information for a period of time corresponding to one or more epochs.

[0038] The above epoch may be set to data corresponding to 30-second units.

[0039] According to one embodiment of the present invention, a method for providing interpretation content of a user's sleep data may include a step of obtaining the user's sleep information, a step of generating the user's sleep state information based on the obtained user's sleep information, a step of generating the user's sleep data interpretation content based on the generated sleep state information, and a step of outputting the generated user's sleep data interpretation content.

[0040] In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the user's sleep data interpretation content can be expressed in at least one of a numerical method and a non-numeric method.

[0041] In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the user's sleep data interpretation content may be generated based on comparing the generated user's sleep state information with at least one of the user's sleep data generated over a predetermined period of time in the past, medical standards, and other user's sleep data generated over a predetermined period of time in the past.

[0042] In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the user's sleep data interpretation content may be generated based on at least one of a lookup table or a large-scale language model.

[0043] In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information may be generated based on combining the acquired user's sleep information and other information into multimodal data.

[0044] In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information may be generated based on a portion of the user's acquired sleep information cut off by a predetermined length on the time axis.

[0045] According to one embodiment of the present invention, an electronic device for providing interpretation content of a user's sleep data includes a sensing unit for acquiring the user's sleep information, a wireless communication unit, one or more processors, and an output unit, wherein the wireless communication unit transmits the acquired sleep information to an external terminal, the external terminal generates sleep state information of the user based on the transmitted sleep information of the user, and generates sleep data interpretation content of the user based on the generated sleep state information, the wireless communication unit receives the generated sleep data interpretation content of the user from the external terminal, and the output unit can output the received sleep data interpretation content of the user.

[0046] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data can be expressed in at least one of a numerical method and a non-numeric method.

[0047] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data may be generated based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated over a predetermined period of time in the past, medical standards, and other user's sleep data generated over a predetermined period of time in the past.

[0048] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data may be generated based on at least one of a lookup table or a large-scale language model.

[0049] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information can be generated based on combining the acquired user's sleep information and other information into multimodal data.

[0050] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information may be generated based on a portion of the user's acquired sleep information cut off by a predetermined length on a time axis.

[0051] According to one embodiment of the present invention, a method for providing interpretation content of a user's sleep data comprises the steps of: obtaining the user's sleep information; generating the user's sleep state information based on the obtained user's sleep information; wherein the sleep state information includes a sleep apnea category, a sleep onset latency category, and a first cycle sleep quality category. A method comprising: generating sleep data belonging to at least one category among a REM Latency category, a REM Ratio category, a Deep Ratio category, a Wake Time after Sleep Onset (WASO) category, a Number of Awakening category, and a Total Sleep Time category; generating user sleep data interpretation content based on the generated sleep state information; and outputting the generated user sleep data interpretation content. The method may further include calculating an importance score between categories of the generated sleep state information; and the outputting of the sleep data interpretation content may further include outputting the sleep data interpretation content belonging to a category having a higher importance score based on the calculated importance score between categories of the sleep state information so that the sleep data interpretation content can be more easily identified than the sleep data interpretation content belonging to another category.

[0052] In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the step of calculating an importance score between categories of the generated sleep state information further includes a step of learning an importance parameter based on human feedback-based reinforcement learning, and the importance score can be calculated based on the learned importance parameter.

[0053] In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the step of calculating an importance score between categories of the generated sleep state information may further include a step of calculating the importance score based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated during a predetermined period in the past, medical standards, and other user's sleep data generated during a predetermined period in the past.

[0054] According to one embodiment of the present invention, an electronic device for providing interpretation content of a user's sleep data includes a sensing unit for obtaining sleep information of the user, a wireless communication unit, one or more processors, and an output unit, wherein the wireless communication unit transmits the obtained sleep information to an external terminal, and the external terminal generates sleep state information of the user based on the transmitted sleep information of the user - the sleep state information includes a sleep apnea category, a sleep onset latency category, and a first cycle sleep quality category. The sleep data belongs to at least one category among the REM Latency category, the REM Ratio category, the Deep Ratio category, the Wake Time after Sleep Onset (WASO) category, the Number of Awakening category, and the Total Sleep Time category, and the user's sleep data interpretation content is generated based on the generated sleep state information, and an importance score between categories of the generated sleep state information is calculated, and the wireless communication unit receives the generated user's sleep data interpretation content and ranking information of the calculated importance score from the external terminal, and the output unit can output the sleep data interpretation content belonging to a category having a higher ranking of the importance score based on the importance score between categories of the received sleep state information so that the sleep data interpretation content can be more easily identified than the sleep data interpretation content belonging to another category.

[0055] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information can be calculated based on an importance parameter learned based on human feedback-based reinforcement learning.

[0056] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information may be calculated based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated during a predetermined period in the past, medical standards, and other user's sleep data generated during a predetermined period in the past.

[0057] According to one embodiment of the present invention, an electronic device for providing interpretation content of a user's sleep data includes a communication module, a processor, and a memory, wherein the communication module receives the user's sleep information obtained from one or more sleep information sensor devices from the sensor devices, and the processor generates the user's sleep state information based on the received user's sleep information, wherein the sleep state information includes a sleep apnea category, a sleep onset latency category, and a first cycle sleep quality category. The sleep data belongs to at least one category among the REM Latency category, the REM Ratio category, the Deep Ratio category, the Wake Time after Sleep Onset (WASO) category, the Number of Awakening category, and the Total Sleep Time category, and the user's sleep data interpretation content is generated based on the generated sleep state information, and an importance score between categories of the generated sleep state information is calculated, and the communication module can transmit the generated user's sleep data interpretation content and ranking information of the calculated importance score to the user terminal.

[0058] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information can be calculated based on an importance parameter learned based on human feedback-based reinforcement learning.

[0059] In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information may be calculated based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated during a predetermined period in the past, medical standards, and other user's sleep data generated during a predetermined period in the past.

[0060] The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; and a sleep data collection tool for activating a sensor module for collecting environmental sensing information of the user terminal based on the collected permissions.

[0061] Additionally, the permission management tool is configured to request at least one of the Overlay permission permission, the foreground service permission, the activation permission of the sensor module, and the time-based alarm permission.

[0062] Additionally, the permission management tool is configured to provide the user with a permission activation notification or guide them to a settings screen when the permission is disabled.

[0063] In addition, the sleep data collection tool is configured to detect an event through the activated sensor module, and a broadcast trigger for sleep measurement of the user terminal is generated based on the detected event, and when the broadcast trigger is generated, the sensor module is further activated for sleep measurement.

[0064] In addition, the detected event is at least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in the accelerometer sensor of the sensor module, switching the display unit of the user terminal to an OFF state, a change in the battery status of the user terminal, connection of the user terminal with another device, and unlocking of the user terminal.

[0065] Additionally, the sleep data collection tool is configured to transmit, when the broadcast trigger occurs, environmental sensing information collected by the additionally activated sensor module for sleep measurement from the time of occurrence of the broadcast trigger to another device for sleep analysis.

[0066] Additionally, the sleep data collection tool is configured to provide a sleep measurement confirmation user interface to the user terminal when the broadcast trigger occurs.

[0067] In addition, the device further includes a time reservation tool for reserving a time for initiating the sleep measurement, wherein the time reservation tool is configured to generate a broadcast trigger for initiating the sleep measurement at the reserved time, and when the broadcast trigger is generated, the device is configured to further activate the sensor module for sleep measurement.

[0068] Additionally, the sleep data collection tool is configured to forcibly generate a broadcast trigger even in the low power mode of the user terminal.

[0069] Additionally, the time reservation tool is configured to forcibly generate a broadcast trigger for initiating the sleep measurement at the reserved time even in the low power mode of the user terminal.

[0070] The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; and a sleep data collection tool for activating a sensor module of the user terminal based on the collected permissions; wherein the sleep data collection tool is configured to detect a first event through the activated sensor module, and a broadcast trigger for sleep measurement of the user terminal is configured to occur based on the detected first event, and when the broadcast trigger is generated, the sensor module is further activated for sleep measurement, and a second event is detected through the additionally activated sensor module for sleep measurement, and a sleep analysis trigger for sleep measurement of the user terminal is configured to occur based on the detected second event, and the detected second event is configured to be detected by a stimulus higher than a threshold by the sensor module.

[0071] In addition, the first event detected is at least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in the accelerometer sensor of the sensor module, switching of the display unit of the user terminal to an OFF state, a change in the battery status of the user terminal, connection of the user terminal with another device, and unlocking of the user terminal.

[0072] In addition, the second event configured to be detected by the sensor module by a stimulus higher than a threshold is at least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in an accelerometer sensor of the sensor module, a change in a battery status of the user terminal, connection of the user terminal with another device, unlocking of the user terminal, swiping a palm with respect to the user terminal, swiping a finger with respect to the user terminal, tapping a finger with respect to the user terminal, voice input with respect to the user terminal, detection of vibration of the user terminal, scrolling a screen wheel with respect to the user terminal, pressing a specific user interface of the user terminal, pressing a display unit of the user terminal for a predetermined period of time or longer, moving two or more fingers simultaneously with respect to the display unit, turning over the user terminal, activating a proximity sensor of the user terminal, detecting a change in temperature of the user terminal, and detecting a change in illumination around the user terminal.

[0073] Additionally, the sleep analysis trigger includes information on when sleep analysis of the measured sleep begins.

[0074] The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; and a sleep data collection tool for activating a sensor module of the user terminal based on the collected permissions; wherein the sleep data collection tool is configured to detect a first event through the activated sensor module, and a broadcast trigger for sleep measurement of the user terminal is configured to occur based on the detected first event, and when the broadcast trigger is generated, the sensor module is further activated for sleep measurement, and a second event is detected through the additionally activated sensor module for sleep measurement, and a sleep analysis trigger for sleep measurement of the user terminal is configured to occur based on the detected second event, and the detected second event is configured to be detected by a stimulus below a threshold by the sensor module.

[0075] In addition, the second event configured to be detected by the sensor module by a stimulus below the threshold is at least one of: when there is no touch input to the display unit of the user terminal for a predetermined period of time or longer; when there is no movement of the accelerometer for the user terminal for a predetermined period of time or longer; when noise around the user terminal decreases and a sound below the threshold is detected for a predetermined period of time or longer; when illuminance around the user terminal decreases and illuminance below the threshold is detected for a predetermined period of time or longer; when the locked state of the user terminal is maintained for a predetermined period of time or longer; when data transmission and reception with a network connected to the user terminal is not performed for a predetermined period of time or longer; when the battery consumption rate of the user terminal falls below the threshold; and when a change in activity data of another device connected to the user terminal is not detected for a predetermined period of time or longer.

[0076] The present invention relates to a server that receives environmental sensing information from a user terminal that generates a broadcast trigger for initiating sleep measurement and / or a sleep analysis trigger including information on a time point for starting sleep analysis of the measured sleep, and performs sleep analysis, the server comprising: a communication module that receives the environmental sensing information from the user terminal; a processor that performs the sleep analysis based on the received environmental sensing information; and a memory that stores a sleep analysis model for performing the sleep analysis; wherein the communication module is configured to receive environmental sensing information acquired by the user terminal from a time point of occurrence of the broadcast trigger generated by the user terminal.

[0077] Additionally, the communication module is configured to receive information on the point in time at which sleep analysis of the measured sleep begins from the user terminal.

[0078] In addition, when the processor performs the sleep analysis based on the received environmental sensing information, the environmental sensing information prior to the point in time at which the sleep analysis of the measured sleep begins among the environmental sensing information is configured to be deleted.

[0079] In addition, when the processor performs the sleep analysis based on the received environmental sensing information, it is configured to perform the sleep analysis only on the environmental sensing information after the point in time when the sleep analysis of the measured sleep starts among the environmental sensing information.

[0080] In addition, when the processor performs the sleep analysis based on the received environmental sensing information, it is configured to acquire sleep sound information only from environmental sensing information after the point in time at which the sleep analysis of the measured sleep starts among the environmental sensing information.

[0081] In addition, when the processor performs the sleep analysis based on the received environmental sensing information, it is configured to acquire sleep state information only from environmental sensing information after the point in time at which the sleep analysis of the measured sleep starts among the environmental sensing information.

[0082] The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; a sleep data collection tool for activating a sensor module of the user terminal based on the collected permissions; and a time reservation tool for storing a time for activating the sensor module; wherein the time reservation tool is configured to detect a first event at the stored time, and a broadcast trigger of sleep measurement of the user terminal is configured to occur based on the detected first event; the sleep data collection tool is configured to detect a second event through the activated sensor module, and a transmission trigger of sleep measurement of the user terminal is configured to occur based on the detected second event, and the detected second event is configured to be detected by a stimulus greater than a threshold by the sensor module.

[0083] A method for providing content related to user's sleep state information according to one embodiment of the present invention includes the steps of: obtaining environmental sensing information through a sensor module of a user terminal; converting sleep sound information included in the obtained environmental sensing information into a spectrogram; processing the spectrogram as an input of a sleep analysis model to generate user's sleep state information; and providing content related to the sleep state information through a content provision model based on the sleep state information, wherein the sleep analysis model is an artificial intelligence model including one or more artificial neural networks, and includes a feature extraction model and a feature classification model, and the content provision model may be a natural language processing-based conversational artificial intelligence model including one or more artificial neural networks.

[0084] In a method for providing content related to sleep state information of a user according to one embodiment of the present invention, the feature extraction model may be a model configured to analyze a frequency pattern of the user's breathing included in the sleep sound information, and the feature classification model may be a model configured to analyze a periodic pattern of the user's breathing included in the sleep sound information.

[0085] In a method for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model may include one or more prompts, and the one or more prompts may include a prompt for specifying a role of the content provision model, a prompt for specifying a purpose of the content provision model, and a prompt for presenting an output format through the content provision model.

[0086] In a method for providing content related to a user's sleep state information according to one embodiment of the present invention, the content provision model includes one or more tools, and the one or more tools include a tool for accessing an external database, and the external database may be a static database or a dynamic database.

[0087] In a method for providing content related to user's sleep state information according to one embodiment of the present invention, at least two of the environmental sensing information, the generated sleep state information, and the conversation information with the user through the content provision model are mapped to each other and configured as metadata, and the metadata can be stored in at least one of a database of a sleep analysis server on which the sleep analysis model is implemented and a database of a content provision server on which the content provision model is implemented.

[0088] In a method for providing content related to a user's sleep state information according to one embodiment of the present invention, the content provision model may be configured to proactively provide content related to the sleep state information through the conversational interface even without a conversation input from the user terminal.

[0089] According to one embodiment of the present invention, a user terminal for providing content related to user's sleep state information includes a sensor module, an input unit, a processor, an output unit, wherein the output unit is configured to output an interactive interface, and a communication module, wherein the interactive interface includes content related to the sleep state information received from the content providing server through the communication module, wherein the content is generated by the content providing server through a content providing model based on the sleep state information received from a sleep analysis server, and wherein the content providing model is an interactive artificial intelligence model based on natural language processing including one or more artificial neural networks, and wherein the sleep state information is obtained by the sleep analysis server processing the transmitted sleep sound information as an input of the sleep analysis model when sleep sound information included in environmental sensing information acquired by the sensor module is transmitted to the sleep analysis server through the communication module, and the sleep analysis model is an artificial intelligence model including one or more artificial neural networks, and may include a feature extraction model and a feature classification model.

[0090] In a user terminal for providing content related to user's sleep state information according to one embodiment of the present invention, the feature extraction model may be a model configured to analyze a frequency pattern of the user's breathing included in the sleep sound information, and the feature classification model may be a model configured to analyze a periodic pattern of the user's breathing included in the sleep sound information.

[0091] In a user terminal for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model may include one or more prompts, and the one or more prompts may include a prompt for specifying a role of the content provision model, a prompt for specifying a purpose of the content provision model, and a prompt for presenting an output format through the content provision model.

[0092] In a user terminal for providing content related to a user's sleep state information according to one embodiment of the present invention, the content provision model includes one or more tools, and the one or more tools include a tool for accessing an external database, and the external database may be a static database or a dynamic database.

[0093] In a user terminal for providing content related to user's sleep state information according to one embodiment of the present invention, at least two of the environmental sensing information, the generated sleep state information, and the conversation information with the user through the content provision model are mapped to each other and configured as metadata, and the metadata can be stored in at least one of a database of a sleep analysis server on which the sleep analysis model is implemented and a database of a content provision server on which the content provision model is implemented.

[0094] In a user terminal for providing content related to a user's sleep state information according to one embodiment of the present invention, the user terminal may be configured to proactively provide content related to the sleep state information through the conversational interface even without a conversation input from the user terminal.

[0095] According to one embodiment of the present invention, a server for providing content related to sleep state information of a user includes a memory, a communication module, and a processor, wherein the processor is configured to generate the content through a content provision model implemented in the memory based on the sleep state information received from a sleep analysis server through the communication module, the processor is configured to generate an interactive interface including the generated content, and the processor is configured to transmit the interactive interface to a user terminal through the communication module, wherein the content provision model is an interactive artificial intelligence model based on natural language processing including one or more artificial neural networks, and the sleep state information is obtained by the sleep analysis server processing the transmitted sleep sound information as an input of the sleep analysis model when sleep sound information included in environmental sensing information acquired by the sensor module is transmitted to the sleep analysis server through the communication module, and the sleep analysis model is an artificial intelligence model including one or more artificial neural networks, and may include a feature extraction model and a feature classification model.

[0096] In a server for providing content related to user's sleep state information according to one embodiment of the present invention, the feature extraction model may be a model configured to analyze a frequency pattern of the user's breathing included in the sleep sound information, and the feature classification model may be a model configured to analyze a periodic pattern of the user's breathing included in the sleep sound information.

[0097] In a server for providing content related to a user's sleep state information according to one embodiment of the present invention, the content provision model may include one or more prompts, and the one or more prompts may include a prompt for specifying a role of the content provision model, a prompt for specifying a purpose of the content provision model, and a prompt for presenting an output format through the content provision model.

[0098] In a server for providing content related to a user's sleep state information according to one embodiment of the present invention, the content provision model includes one or more tools, and the one or more tools include a tool for accessing an external database, and the external database may be a static database or a dynamic database.

[0099] In a server for providing content related to user's sleep state information according to one embodiment of the present invention, at least two of the environmental sensing information, the generated sleep state information, and the conversation information with the user through the content provision model are mapped to each other and configured as metadata, and the metadata can be stored in at least one of a database of a sleep analysis server on which the sleep analysis model is implemented and a database of a content provision server on which the content provision model is implemented.

[0100] In a server for providing content related to a user's sleep state information according to one embodiment of the present invention, the server may be configured to proactively provide content related to the sleep state information through the conversational interface even without a conversation input from the user terminal.

[0101] According to the present invention, by generating and presenting information about a user's sleep through a graphical user interface, the system can contribute to improving the quality of sleep by providing the user with information about their sleep. Furthermore, by presenting sleep status information generated from the start of a sleep session in a chronological order, the system can easily analyze sleep data.

[0102] According to the present invention, by analyzing the user's sleep based on acoustic information, it is possible to monitor the user's sleep state and thereby contribute to improving the quality of the user's sleep.

[0103] In addition, according to the present invention, there is no need to install a microphone in contact with the body to obtain the user's sleep information, and the sleep state can be monitored in a typical home environment through a software update without purchasing a separate additional device, thereby providing the effect of increasing convenience.

[0104] In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, so the size of data used as input for the sleep analysis model can be relatively small, and thus there is an effect that the sleep analysis time can be shortened.

[0105] In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, thereby contributing to improving the quality of sleep of the user by analyzing sleep for a relatively short period of time.

[0106] In addition, according to the present invention, there is an effect that highly accurate sleep analysis is possible by performing sleep analysis in a multimodal manner.

[0107] In addition, according to the present invention, by providing the user's sleep data interpretation content, the user can easily recognize the sleep analysis content, thereby improving convenience and sleep quality.

[0108] In addition, according to the present invention, convenience is improved in that the user can easily recognize the sleep analysis content by providing the user's sleep data interpretation content.

[0109] In addition, according to the present invention, based on the importance scores between categories of sleep information, the user's sleep data interpretation content can be generated or provided so that data of a category of sleep information having a relatively high importance score can be easily identified compared to sleep data belonging to other categories of sleep information, thereby enabling the user to easily recognize the sleep analysis content and improving convenience.

[0110] The present invention can significantly improve user convenience. By automatically starting sleep measurement without requiring the user to initiate a separate sleep measurement, it prevents the loss of sleep data. In particular, it prevents situations where users forget to start sleep measurement even though they intend to, enabling accurate and continuous data collection.

[0111] Furthermore, the present invention combines broadcast triggers and sleep analysis triggers to more precisely record the user's actual sleep onset time. By comprehensively analyzing time reservations, user behavior data, and environmental data, the reliability of sleep data is increased, enabling users to obtain more accurate sleep analysis results.

[0112] The present invention is designed to enable triggers and alarms to operate even in low-power mode of the Android system, thereby resolving functional limitation issues that may arise due to battery optimization, thereby enabling stable and continuous sleep measurement in various environments.

[0113] Furthermore, the present invention provides multifaceted triggering capabilities that utilize user behavior data, as well as time-based triggers. By supporting both user-initiated events (e.g., swiping) and involuntary events (e.g., decreased motion), it can adapt to diverse usage patterns.

[0114] According to the present invention, by analyzing the user's sleep based on acoustic information, it is possible to monitor the user's sleep state and thereby contribute to improving the quality of the user's sleep.

[0115] In addition, according to the present invention, there is no need to install a microphone in contact with the body to obtain the user's sleep information, and the sleep state can be monitored in a typical home environment through a software update without purchasing a separate additional device, thereby providing the effect of increasing convenience.

[0116] In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, so the size of data used as input for the sleep analysis model can be relatively small, and thus there is an effect that the sleep analysis time can be shortened.

[0117] In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, thereby contributing to improving the quality of sleep of a user by analyzing sleep for a relatively short period of time.

[0118] In addition, according to the present invention, there is also an effect that accurate sleep analysis is possible by performing sleep analysis in a multimodal manner.

[0119] In addition, the present invention is effective in that it provides content related to the user's sleep state, thereby enabling the user to easily recognize sleep analysis results and easily obtain information for improving sleep quality.

[0120] FIG. 1A is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface generating devices (100) representing information about a user's sleep according to one embodiment of the present invention can be implemented.

[0121] FIG. 1b is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface providing devices (200) for displaying information about a user's sleep according to one embodiment of the present invention can be implemented.

[0122] FIG. 2a is a conceptual diagram illustrating a system in which, according to another embodiment of the present invention, generation and / or provision of one or more graphical user interfaces representing information about a user's sleep is implemented in a user terminal (10).

[0123] FIG. 2b is a conceptual diagram illustrating a system in which various aspects of various electronic devices related to another embodiment of the present invention can be implemented.

[0124] Figure 3a is a block diagram for explaining a user terminal (10) according to one embodiment of the present invention.

[0125] FIG. 3b is a block diagram showing the configuration of one or more graphical user interface generating devices (100) / providing devices (200) that display information about a user's sleep according to one embodiment of the present invention.

[0126] Figure 3c is a block diagram illustrating an external server (20) according to one embodiment of the present invention.

[0127] FIG. 4A is a diagram illustrating a graphical user interface including a hypnogram according to one embodiment of the present invention.

[0128] FIG. 4b is a graph showing the time ratio of each sleep stage measured according to one embodiment of the present invention.

[0129] Figure 4c is a drawing showing a respiratory stability graph according to an embodiment of the present invention.

[0130] FIG. 4d is a diagram illustrating a graphical user interface including a hypnogram according to another embodiment of the present invention.

[0131] FIG. 4e is a diagram illustrating a graphical user interface including a description display for respiratory instability according to one embodiment of the present invention.

[0132] FIGS. 5A and 5B are diagrams illustrating a graphical user interface including statistical information of sleep state information according to embodiments of the present invention.

[0133] FIGS. 6A and 6B are diagrams illustrating a graphical user interface including sleep status information acquired over a week according to embodiments of the present invention.

[0134] FIGS. 7A and 7B are diagrams illustrating a graphical user interface including sleep state information acquired over a predetermined period of time according to embodiments of the present invention.

[0135] FIGS. 8A through 8E are black and white drawings of a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

[0136] Figure 9a is a drawing for explaining a process of obtaining sleep sound information in a sleep analysis method according to the present invention.

[0137] FIG. 9b is a drawing for explaining a method for obtaining a spectrogram corresponding to sleep sound information in a sleep analysis method according to the present invention.

[0138] FIG. 10 is a flowchart of a method for generating and providing one or more graphical user interfaces representing information about a user's sleep according to one embodiment of the present invention.

[0139] FIG. 11 is a schematic diagram illustrating one or more network functions for performing a sleep analysis method according to the present invention.

[0140] Figure 12 is a drawing for explaining sleep stage analysis using a spectrogram in a sleep analysis method according to the present invention.

[0141] Figure 13 is a drawing for explaining sleep disorder determination using a spectrogram in a sleep analysis method according to the present invention.

[0142] Figure 14 is a drawing showing an experimental process for verifying the performance of a sleep analysis method according to the present invention.

[0143] FIG. 15a and FIG. 15b are drawings for explaining the overall structure of a sleep analysis model according to one embodiment of the present invention.

[0144] FIG. 16 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention.

[0145] Figures 17a and 17b are graphs verifying the performance of the sleep analysis method according to the present invention, and are drawings comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention.

[0146] Fig. 18 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a diagram comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention in relation to sleep apnea and hypopnea.

[0147] FIGS. 19A to 19C are diagrams showing how a graphical user interface according to embodiments of the present invention is displayed on various display units.

[0148] Figures 20a to 20e are diagrams showing hypnogram graphs of sleep stage information expressed in a conventional sleep measurement interface.

[0149] FIG. 21 is a diagram illustrating a graphical user interface that displays a graph representing sleep stage information and a graph representing sleep event information in parallel according to one embodiment of the present invention.

[0150] FIGS. 22A through 22E are diagrams illustrating a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

[0151] FIGS. 23A through 23D are diagrams illustrating a graphical user interface including a Stacked Sleep Stage Graph generated based on a larger number of sleep sessions compared to FIGS. 22A through 22E, according to embodiments of the present invention.

[0152] FIG. 24a and FIG. 24b are conceptual diagrams illustrating a system in which various aspects of a sleep data interpretation content creation device or a sleep data interpretation content provision device based on user sleep information according to one embodiment of the present invention can be implemented.

[0153] Figure 24c is a conceptual diagram illustrating a system in which sleep data interpretation content creation and provision based on user sleep information according to one embodiment of the present invention is implemented in a user terminal (300).

[0154] Figure 24d is a conceptual diagram illustrating a system of a sleep data interpretation content creation device (100a) based on user sleep information according to one embodiment of the present invention.

[0155] Figure 24e is a conceptual diagram illustrating a system of a sleep data interpretation content providing device (100b) based on user sleep information according to one embodiment of the present invention.

[0156] FIG. 24F is a conceptual diagram illustrating a system in which various aspects of various electronic devices according to embodiments of the present invention can be implemented.

[0157] FIG. 25a and FIG. 25b are block diagrams illustrating a computing device (100) according to one embodiment of the present invention.

[0158] Figure 25c is a block diagram for explaining an external terminal (200) according to one embodiment of the present invention.

[0159] FIG. 25d is a block diagram illustrating a user terminal (300) according to one embodiment of the present invention.

[0160] Figures 26a and 26b are graphs verifying the performance of the sleep analysis method according to the present invention, and are drawings comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention.

[0161] Figure 26c is a graph verifying the performance of the sleep analysis method according to the present invention, and is a drawing comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention in relation to sleep apnea and hypopnea.

[0162] Figure 27 is a drawing showing an experimental process for verifying the performance of a sleep analysis method according to the present invention.

[0163] FIG. 28 is an exemplary diagram illustrating a process of obtaining sleep sound information from environmental sensing information according to one embodiment of the present invention.

[0164] FIG. 29a is an exemplary diagram illustrating a method for obtaining a spectrogram corresponding to sleep sound information according to one embodiment of the present invention.

[0165] Figure 29b is a drawing for explaining sleep stage analysis using a spectrogram in a sleep analysis method according to the present invention.

[0166] Figure 29c is a drawing for explaining sleep event determination using a spectrogram in a sleep analysis method according to the present invention.

[0167] FIG. 30a is a schematic diagram illustrating one or more network functions according to one embodiment of the present invention.

[0168] FIG. 30b is a diagram for explaining the structure of a sleep analysis model utilizing deep learning to analyze a user's sleep according to one embodiment of the present invention.

[0169] FIG. 31A is a diagram for explaining a method for generating interpretation content of sleep data based on a lookup table according to one embodiment of the present invention.

[0170] FIG. 31b is a diagram illustrating a method for generating interpretation content of sleep data based on a large-scale language model, according to one embodiment of the present invention.

[0171] FIG. 31c is a flowchart of a method for providing non-numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

[0172] FIG. 31d is a flowchart of a method for providing numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

[0173] FIG. 32 is a flowchart illustrating a method for learning importance parameters that serve as the basis for calculating importance scores of sleep categories according to embodiments of the present invention.

[0174] FIG. 33A is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

[0175] FIG. 33b is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

[0176] FIG. 33c is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

[0177] FIG. 33d is an exemplary diagram illustrating one or more graphical user interfaces representing numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

[0178] FIG. 33e is an exemplary diagram illustrating one or more graphical user interfaces representing numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

[0179] FIGS. 34 and 35 are graphs illustrating a method for calculating importance scores of categories of sleep information based on medical criteria according to embodiments of the present invention.

[0180] FIG. 36 is a graph illustrating a method for calculating an importance score of a category of sleep information based on information compared to sleep data of a user generated over a predetermined period of time in the past according to embodiments of the present invention.

[0181] Figure 37 is a diagram for explaining consistency training according to one embodiment of the present invention.

[0182] FIG. 38 is a flowchart illustrating a method for analyzing sleep state information, including a process of combining sleep sound information and sleep environment information into multimodal data according to one embodiment of the present invention.

[0183] FIG. 39 is a flowchart illustrating a method for analyzing sleep state information, including a step of combining each of inferred sleep sound information and sleep environment information into multimodal data according to one embodiment of the present invention.

[0184] FIG. 40 is a flowchart illustrating a method for analyzing sleep state information, including a step of combining inferred sleep sound information with sleep environment information and multimodal data according to one embodiment of the present invention.

[0185] FIG. 41 is a diagram for explaining a linear regression analysis function used to analyze AHI, an index of sleep apnea occurrence, through sleep events occurring during sleep according to one embodiment of the present invention.

[0186] Figures 42a to 42c are conceptual diagrams showing exemplary systems in which services according to the present invention can be implemented.

[0187] FIGS. 42d and 42e are conceptual diagrams illustrating an exemplary system in which a service for providing content related to a user's sleep status information according to the present invention can be implemented.

[0188] FIG. 42f is a conceptual diagram illustrating a system in which various aspects of various electronic devices according to embodiments of the present invention can be implemented.

[0189] FIG. 43a and FIG. 43b are block diagrams illustrating a computing device (100) according to one embodiment of the present invention.

[0190] Figure 43c is a block diagram illustrating a server (200) according to one embodiment of the present invention.

[0191] FIG. 43d is a block diagram illustrating a user terminal (300) according to one embodiment of the present invention.

[0192] Figure 44 is a schematic diagram of a data set for sleep analysis according to one embodiment of the present invention.

[0193] Figure 45a is a drawing for explaining noise reduction according to one embodiment of the present invention.

[0194] Figure 45b is a diagram for explaining the process of preprocessing and converting data according to one embodiment of the present invention.

[0195] Figure 45c is a diagram for explaining a data conversion process according to one embodiment of the present invention.

[0196] Figure 46 is a drawing for explaining a method for obtaining a spectrogram corresponding to sleep sound information in a sleep analysis method according to the present invention.

[0197] Figure 47 illustrates a flowchart exemplarily showing a method for analyzing a user's sleep state through acoustic information according to one embodiment of the present invention.

[0198] FIG. 48a and FIG. 48b are drawings for explaining the overall structure of a sleep analysis model according to one embodiment of the present invention.

[0199] FIG. 49 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention.

[0200] Figure 50 is a drawing for explaining in detail the operation of a sleep analysis model according to one embodiment of the present invention.

[0201] FIG. 51a and FIG. 51b are diagrams for explaining the performance of sleep event determination and noise addition using a spectrogram in a sleep analysis method according to the present invention.

[0202] Figure 52 is a drawing for explaining sleep stage analysis using a spectrogram in a sleep analysis method according to the present invention.

[0203] Figure 53 is a drawing for explaining sleep event determination using a spectrogram in a sleep analysis method according to the present invention.

[0204] Figure 54 is a drawing showing an experimental process for verifying the performance of a sleep analysis method according to the present invention.

[0205] Figure 55 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention.

[0206] Figure 56 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention in relation to sleep apnea and hypopnea.

[0207] Figure 57 is a drawing for explaining the configuration of an automatic measurement system according to one embodiment of the present invention.

[0208] FIG. 58 is a diagram showing a screen in which an app requests and manages specific permissions in an Android system according to one embodiment of the present invention.

[0209] Figure 59a is a diagram illustrating the main screen of a user interface for automatic sleep measurement via an app according to one embodiment of the present invention. Figure 59b is a diagram illustrating an Auto Tracking settings screen for setting an automatic measurement schedule according to one embodiment of the present invention. Figure 59c is a diagram illustrating a user interface for displaying results after automatic sleep measurement is completed according to one embodiment of the present invention.

[0210] FIG. 60 is a diagram for explaining a process for setting up automation of sleep measurement using a Shortcuts app in an iOS environment according to one embodiment of the present invention.

[0211] FIG. 61 is a diagram illustrating a process for setting up automation in an iOS Shortcuts App according to one embodiment of the present invention.

[0212] FIG. 62 is a diagram illustrating a process by which a user adds a new action (e.g., starting sleep measurement) to an automated task to be executed at a specific time in a shortcut app according to one embodiment of the present invention.

[0213] FIG. 63 is a drawing showing a screen that sets a state in which a sleep measurement termination task according to one embodiment of the present invention can be automatically registered and executed through an App Intent.

[0214] FIG. 64 is a drawing showing an embodiment of providing an activation notification of a sensor module when a broadcast trigger occurs according to the present invention.

[0215] Figure 65 is a diagram explaining the structure of the Transformer model that forms the basis of the Large Language Model.

[0216] FIG. 66 is a diagram for explaining an inverted diffusion model (Inverter Model) of a DIFFUSION model in a content-generating artificial intelligence according to one embodiment of the present invention.

[0217] FIG. 67 is a diagram for explaining a generator and a discriminator of a GAN (Generative Adversarial Network) in a content-generating artificial intelligence according to one embodiment of the present invention.

[0218] FIG. 68 is a diagram illustrating a method for generating interpretation content of sleep data based on a large-scale language model, according to one embodiment of the present invention.

[0219] FIG. 69a is a flowchart of a method for providing non-numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

[0220] FIG. 69b is a flowchart of a method for providing numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

[0221] The following description sets forth exemplary methods, parameters, and the like. However, it should be recognized that this description is not intended to limit the scope of the present invention, but rather serves as a description of exemplary embodiments.

[0222] There is a need for a method for providing one or more graphical user interfaces that efficiently display information about a user's sleep. Efficiently presenting information about a user's sleep can provide information that is medically relevant and user-friendly, while also providing more relevant information to the user.

[0223] Although the following description uses terms like "first," "second," etc. to describe various elements, these elements should not be limited by these terms. These terms are used to distinguish one element from another.

[0224] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device may include a portable communication device (e.g., a mobile phone, a smartwatch) that includes other functions, such as PDA and music player functions. Alternatively, embodiments may include an electronic device that includes a touch-sensitive surface and display.

[0225] Additionally, in some embodiments of the present invention, an electronic device including a display (or touch-sensitive surface) may be included.

[0226] Overall composition

[0227] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0228] Although each step described in this specification is described as being performed by a computer, the subject of each step is not limited thereto, and at least some of each step may be performed by different devices depending on the embodiment.

[0229] User terminal (10)

[0230] Figure 3a is a block diagram for explaining a user terminal (10) according to one embodiment of the present invention.

[0231] According to one embodiment of the present invention, a user terminal (10) is a terminal that can receive information related to the user's sleep through information exchange with at least one of various electronic devices (e.g., a graphical user interface generation device, a provision device, or an external server) according to embodiments of the present invention, and may refer to a terminal carried by the user. For example, the user terminal (10) may be a terminal related to a user who wishes to improve his or her health through information related to his or her sleep habits. The user may obtain monitoring information related to his or her sleep through the user terminal (10). The monitoring information related to sleep may include, for example, sleep state information related to the time the user fell asleep, the time the user slept, the time the user woke up, etc., or sleep stage information related to changes in sleep stages during sleep. For a specific example, the sleep stage information may refer to information on changes in the user's sleep such as light sleep, normal sleep, deep sleep, or REM sleep at each point during the user's 8 hours of sleep the previous night. The specific description of the above-described sleep stage information is merely an example, and the present invention is not limited thereto.

[0232] The user terminal (10) may include a mobile phone, a smart phone, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, a head mounted display (HMD)), etc.

[0233] The above user terminal (10) may include a wireless communication unit (11), an input unit (12), a sensing unit (14), an output unit (15), an interface unit (16), a memory (17), a control unit (18), and a power supply unit (19). The components illustrated in FIG. 2f are not essential for implementing the user terminal, and thus the user terminal described in this specification may have more or fewer components than the components listed above.

[0234] Wireless Communication Department (11)

[0235] The wireless communication unit (11) may include one or more modules that enable wireless communication between a user terminal (10) and a wireless communication system, between the user terminal (10) and a graphical user interface generating device (100), a graphical user interface providing device (200), or between the user terminal (10) and an external server (20). In addition, the wireless communication unit (11) may include one or more modules that connect the user terminal (10) to one or more networks.

[0236] This wireless communication unit (11) may include at least one of a broadcast reception module (311), a mobile communication module (312), a wireless Internet module (313), a short-range communication module (314), and a location information module (315).

[0237] Input section (12)

[0238] According to the present invention, the input unit (12) may include a camera (321) or an image input unit for inputting an image signal, a microphone (322) or an audio input unit for inputting an audio signal, and a user input unit (323, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (12) may be analyzed and processed into a user's control command.

[0239] Sensing unit (14)

[0240] According to the present invention, the sensing unit (14) may include one or more sensors for sensing at least one of information within the user terminal, information about the surrounding environment surrounding the user terminal, and user information. For example, the sensing unit (14) may include at least one of a proximity sensor (341), an illumination sensor (342), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera (see 321)), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the user terminal disclosed in this specification can utilize information sensed by at least two of these sensors in combination.

[0241] Output section (15)

[0242] According to the present invention, the output unit (15) is for generating output related to visual, auditory or tactile sensations, and may include at least one of a display unit (351), an audio output unit (352), a haptic module (353), and an optical output unit (354).

[0243] According to the present invention, the display unit (351) can be formed as a touch screen by forming a mutual layer structure with the touch sensor or by forming an integral structure. This touch screen can function as a user input unit (323) that provides an input interface between the user terminal (10) and the user, and at the same time, provide an output interface between the user terminal (10) and the user.

[0244] Interface section (16)

[0245] According to the present invention, the interface unit (16) serves as a passageway for various types of external devices connected to the user terminal (10). This interface unit (16) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. In the user terminal (10), in response to the external device (e.g., an electronic device) being connected to the interface unit (16), appropriate control related to the connected external device can be performed.

[0246] Memory (17)

[0247] According to the present invention, the memory (17) stores data supporting various functions of the user terminal (10). The memory (17) can store a plurality of application programs (or applications) running on the user terminal (10), data, commands, or instructions for the operation of the user terminal (10). At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the user terminal (10) from the time of shipment for the basic functions of the user terminal (10) (e.g., call receiving, calling, message receiving, calling). Meanwhile, the application programs can be stored in the memory (17), installed on the user terminal (10), and driven by the control unit (18) to perform the operations (or functions) of the user terminal. The memory (17) can store instructions for the operations of the control unit (18).

[0248] Control unit (18)

[0249] According to the present invention, the control unit (18) controls the overall operation of the user terminal (10) in addition to the operations related to the application program. The control unit (18) processes signals, data, information, etc. input or output through the components discussed above, or operates the application program stored in the memory (17), thereby providing or processing appropriate information or functions to the user.

[0250] According to the present invention, the control unit (18) can control at least some of the components discussed with reference to FIG. 2f to drive an application program stored in the memory (17). Furthermore, the control unit (18) can operate at least two or more of the components included in the user terminal (10) in combination to drive the application program.

[0251] Power supply unit (19)

[0252] According to the present invention, the power supply unit (19) receives external power and internal power under the control of the control unit (18) and supplies power to each component included in the user terminal (10). This power supply unit (19) includes a battery, and the battery may be a built-in battery or a replaceable battery.

[0253] At least some of the above components may cooperate with each other to implement the operation, control, or control method of the user terminal according to the various embodiments described below. In addition, the operation, control, or control method of the user terminal may be implemented on the user terminal by driving at least one application program stored in the memory (17).

[0254] Graphical User Interface Generation Device (100)

[0255] A graphical user interface generation device (100) according to the present invention can generate a graphical user interface for transmitting information to a user based on the user's sleep information (e.g., environmental sensing information, etc.).

[0256] Alternatively, the graphical user interface generation device (100) according to embodiments of the present invention may generate a graphical user interface for conveying information to the user based on sleep state information generated based on the user's sleep information.

[0257] FIG. 1A is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface generation devices (100) representing information about a user's sleep according to one embodiment of the present invention may be implemented. As illustrated in FIG. 1A, a system according to embodiments of the present invention may include a graphical user interface generation device (100), a user terminal (10), an external server (20), and a network.

[0258] Here, a system in which one or more graphical user interface generating devices (100) representing information about a user's sleep as shown in FIG. 1a are implemented is according to one embodiment, and its components are not limited to the embodiment shown in FIG. 1a, and may be added, changed, or deleted as needed.

[0259] graphical user interface providing device (200)

[0260] The graphical user interface providing device (200) according to the present invention can output a graphical user interface generated based on the user's sleep information (e.g., environmental sensing information, etc.) or sleep state information to provide the user with the graphical user interface.

[0261] FIG. 1B is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface providing devices (200) for displaying information about a user's sleep according to one embodiment of the present invention can be implemented. As illustrated in FIG. 1B, a system according to embodiments of the present invention may include a graphical user interface providing device (200), a user terminal (10), an external server (20), and a network.

[0262] Here, the system in which one or more graphical user interface providing devices (200) for displaying information about the user's sleep as shown in FIG. 1b is implemented is according to one embodiment, and its components are not limited to the embodiment shown in FIG. 1b, and may be added, changed, or deleted as needed.

[0263] Meanwhile, FIG. 2A is a conceptual diagram illustrating a system in which, according to another embodiment of the present invention, one or more graphical user interfaces representing information about a user's sleep are generated and / or provided in a user terminal (10). As illustrated in FIG. 2A, one or more graphical user interfaces representing information about a user's sleep may be generated and / or provided in a user terminal (10) without a separate generating device (100) and / or a separate providing device (200).

[0264] Additionally, FIG. 2b illustrates a conceptual diagram of a system in which various aspects of various electronic devices related to another embodiment of the present invention can be implemented.

[0265] The electronic devices illustrated in FIG. 2b can perform at least one of the operations performed by various devices according to embodiments of the present invention.

[0266] For example, operations performed by various electronic devices according to embodiments of the present invention may include operations of acquiring environmental sensing information, operations of performing learning for sleep analysis, operations of performing inference for sleep analysis, and operations of acquiring sleep state information.

[0267] Alternatively, for example, it may include an operation of providing information related to the user's sleep, transmitting or receiving environmental sensing information, determining environmental sensing information, extracting acoustic information from environmental sensing information, processing or manipulating data, processing a service, providing a service, constructing a learning data set based on environmental sensing information or information related to the user's sleep, storing acquired data or a plurality of data that become inputs to a neural network, transmitting or receiving various pieces of information, transmitting and receiving data for a system according to embodiments of the present invention through a network, generating one or more graphical user interfaces representing information related to the user's sleep, or providing one or more graphical user interfaces representing information related to the user's sleep.

[0268] External Server (20)

[0269] Figure 3c is a block diagram illustrating an external server (20) according to one embodiment of the present invention.

[0270] According to the present invention, the external server (20) may include a processor (21), memory (22), and a communication module (27). The external server (20) may be a cloud server or an edge server. The external server (20) may be a digital device equipped with a processor, memory, and computing power, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone. The external server (20) may be a web server that processes services. The types of servers described above are merely examples, and the present invention is not limited thereto.

[0271] Processor (21)

[0272] According to the present invention, the processor (21) controls the external server (20) as a whole. The processor (21) may include an AI processor (215).

[0273] The AI ​​processor (215) can learn a neural network using a program stored in the memory (22). In particular, the AI ​​processor (215) can learn a neural network for recognizing data related to the operation of the user terminal (10). Here, the neural network can be designed to simulate the human brain structure (e.g., the neuron structure of a human neural network) on a computer. The neural network can include an input layer, an output layer, and at least one hidden layer. Each layer includes at least one neuron having a weight, and the neural network can include a synapse connecting neurons. In the neural network, each neuron can output an input signal input through a synapse as a function value of an activation function for a weight and / or a bias.

[0274] Multiple network modes can exchange data according to their respective connection relationships, simulating the synaptic activity of neurons exchanging signals through synapses. Here, the neural network may include a deep learning model developed from a neural network model. In a deep learning model, multiple network nodes are located in different layers and can exchange data according to convolutional connection relationships. Examples of neural network models include various deep learning techniques such as deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks, restricted Boltzmann machines, deep belief networks, and deep Q-networks, and can be applied in fields such as vision recognition, speech recognition, natural language processing, and speech / signal processing.

[0275] Meanwhile, the processor (21) performing the function described above may be a general-purpose processor (e.g., CPU), but may be an AI-only processor for artificial intelligence learning (e.g., GPU, TPU).

[0276] Memory (22)

[0277] According to the present invention, the memory (22) can store various programs and data required for the operation of the user terminal (10) and / or the external server (20). The memory (22) is accessed by the AI ​​processor (215), and data reading / recording / modifying / deleting / updating, etc. can be performed by the AI ​​processor (215). In addition, the memory (22) can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data classification / recognition. Furthermore, the memory (22) can store not only the learning model (221), but also input data, learning data, learning history, etc.

[0278] Meanwhile, the AI ​​processor (215) may include a data learning unit (215a) that learns a neural network for data classification / recognition. The data learning unit (215a) may learn criteria regarding which learning data to use to determine data classification / recognition and how to classify and recognize data using the learning data. The data learning unit (215a) may acquire learning data to be used for learning and apply the acquired learning data to the deep learning model, thereby learning the deep learning model.

[0279] The data learning unit (215a) may be manufactured in the form of at least one hardware chip and mounted on an external server (20). For example, the data learning unit (215a) may be manufactured in the form of a dedicated hardware chip for artificial intelligence, and may be manufactured as a part of a general-purpose processor (CPU) or a graphics processor (GPU) and mounted on the external server (20). In addition, the data learning unit (215a) may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored on a non-transitory computer readable medium that can be read by a computer. In this case, at least one software module may be provided to an operating system (OS) or provided by an application.

[0280] The data learning unit (215a) can use the acquired learning data to learn the neural network model so that it has judgment criteria on how to classify / recognize certain data. At this time, the learning method by the model learning unit can be classified into supervised learning, unsupervised learning, and reinforcement learning. Here, supervised learning refers to a method of training an artificial neural network in a state where labels for the learning data are given, and the labels can mean the correct answer (or result value) that the artificial neural network must infer when the learning data is input to the artificial neural network. Unsupervised learning can refer to a method of training an artificial neural network in a state where labels for the learning data are not given. Reinforcement learning can refer to a method of training an agent defined in a specific environment to select an action or action sequence that maximizes the cumulative reward in each state. In addition, the model learning unit can train the neural network model using a learning algorithm including error backpropagation or gradient descent. Once a neural network model is trained, the trained neural network model can be referred to as a training model (221). The training model (221) is stored in memory (22) and can be used to infer results for new input data other than training data.

[0281] According to the present invention, the AI ​​processor (215) may further include a data preprocessing unit (215b) and / or a data selection unit (215c) to improve the analysis results using the learning model (221) or to save resources or time required for generating the learning model (221).

[0282] According to the present invention, the data preprocessing unit (215b) can preprocess the acquired data so that the acquired data can be used for learning / inference for situational judgment. For example, the data preprocessing unit (215b) can extract feature information as preprocessing for input data received through the communication module (27), and the feature information can be extracted in the format of a feature vector, feature point, or feature map.

[0283] According to the present invention, the data selection unit (215c) can select data required for learning from among the learning data or learning data preprocessed by the preprocessing unit. The selected learning data can be provided to the model learning unit. For example, the data selection unit (215c) can detect a specific area in an image acquired through a camera of an electronic device, thereby selecting only data regarding objects included in the specific area as learning data. In addition, the data selection unit (215c) can also select data required for inference from among input data acquired through an input device or input data preprocessed by the preprocessing unit.

[0284] In addition, the AI ​​processor (215) may further include a model evaluation unit (215d) to improve the analysis results of the neural network model. The model evaluation unit (215d) inputs evaluation data into the neural network model, and if the analysis results output from the evaluation data do not satisfy a predetermined standard, it may cause the model learning unit to relearn. In this case, the evaluation data may be preset data for evaluating the learning model (221). For example, the model evaluation unit (215d) may evaluate that the predetermined standard is not satisfied if the number or ratio of evaluation data for which the analysis results are inaccurate among the analysis results of the learned neural network model for the evaluation data exceeds a preset threshold.

[0285] Communication module (27)

[0286] The communication module (27) can transmit and receive data and / or information to and from at least one of the user terminal (10), the graphical user interface generation device (100), the graphical user interface provision device (200), and other electronic devices according to one embodiment of the present invention. For example, the communication module (27) can receive environmental sensing information from the user terminal (10) and transmit the AI ​​processing result by the AI ​​processor (215) to the user terminal (10). The manner in which data is transmitted and / or received is not limited thereto.

[0287] Various embodiments of the present invention

[0288] The electronic devices illustrated in FIG. 2b may individually perform operations performed by various electronic devices according to embodiments of the present invention, but may also perform one or more operations simultaneously or in a time series manner. At least one of the electronic devices illustrated in FIG. 2b may be a user terminal (10). At least one of the electronic devices illustrated in FIG. 2b may be a graphical user interface generating device (100). At least one of the electronic devices illustrated in FIG. 2b may be a graphical user interface providing device (200). At least one of the electronic devices illustrated in FIG. 2b may be an external server (20).

[0289] Referring to FIG. 2B, the electronic devices (10a to 10d) illustrated in FIG. 2B may be electronic devices within the range of an area (11a) capable of acquiring environmental sensing information. Hereinafter, for convenience, the area (11a) capable of acquiring environmental sensing information will be referred to as "area (11a)."

[0290] Meanwhile, referring to FIG. 2b, the electronic devices (10a and 10d) may be devices formed by a combination of two or more electronic devices.

[0291] Meanwhile, referring to FIG. 2b, the electronic devices (10a and 10b) may be electronic devices connected to a network within an area (11a).

[0292] Meanwhile, referring to FIG. 2b, the electronic devices (10c and 10d) may be electronic devices that are not connected to a network within the area (11a).

[0293] Meanwhile, referring to FIG. 2b, the electronic devices (20a and 20b) may be electronic devices outside the range of the area (11a).

[0294] Meanwhile, referring to FIG. 2b, there may be a network that interacts with electronic devices within the scope of area (11a), and there may be a network that interacts with electronic devices outside the scope of area (11a).

[0295] Here, a network interacting with electronic devices within the scope of area (11a) can play a role in transmitting and receiving information for controlling smart home appliances.

[0296] Additionally, the network interacting with electronic devices within the scope of area (11a) may be, for example, a short-range network or a local network. Here, the network interacting with electronic devices within the scope of area (11a) may be, for example, a long-range network or a global network.

[0297] Since the specific description of the operation of the networks illustrated in Fig. 2b is the same as that described above, redundant description will be omitted.

[0298] Meanwhile, referring to FIG. 2b, there may be one or more electronic devices connected via a network outside the scope of area (11a), and in this case, the electronic devices may perform distributed data processing or perform one or more operations separately. Here, the electronic devices connected via a network outside the scope of area (11a) may include a server device.

[0299] Alternatively, if there is one or more electronic devices connected via a network outside the scope of area (11a), the electronic devices may perform various operations independently of each other.

[0300] As illustrated in FIGS. 1A and 1B, one or more graphical user interface generating devices (100) representing information about a user's sleep and one or more graphical user interface providing devices (200) representing information about a user's sleep can mutually transmit and receive data for a system according to embodiments of the present invention to and from a user terminal (10) via a network.

[0301] As illustrated in FIG. 2a, even if one or more graphical user interface generating devices (100) representing information about the user's sleep and one or more graphical user interface providing devices (200) representing information about the user's sleep are not separately provided, the user terminal (10) can perform the role of one or more graphical user interface generating devices (100) representing information about the user's sleep and / or one or more graphical user interface providing devices (200) representing information about the user's sleep through a network, thereby mutually transmitting and receiving data for the system according to embodiments of the present invention.

[0302] network

[0303] As illustrated in FIG. 2b, various electronic devices according to the present invention can mutually transmit and receive data for a system according to embodiments of the present invention through a network.

[0304] The network according to embodiments of the present invention may use various wired communication systems such as Public Switched Telephone Network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and local area networks (LANs). In addition, the network presented herein may use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0305] The network according to embodiments of the present invention can be configured regardless of the communication mode, such as wired or wireless, and can be configured as various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA) or Bluetooth. The technologies described herein can be used not only in the networks mentioned above, but also in other networks.

[0306] FIG. 3b is a block diagram showing the configuration of one or more graphical user interface generating devices (100) / providing devices (200) that display information about a user's sleep according to one embodiment of the present invention.

[0307] According to one embodiment of the present invention, one or more graphical user interface generating devices (100) / providing devices (200) for displaying information about a user's sleep may include a display (120) / (220), a memory (140) / (240) for storing one or more programs configured to be executed by one or more processors, and one or more processors (160) / (260).

[0308] According to one embodiment of the present invention, the memory (140) / (240) storing one or more programs includes a high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid state storage devices. In addition, the memory may store instructions for performing a method of providing one or more graphical user interfaces that display information about a user's sleep.

[0309] According to one embodiment of the present invention, the processor (160) / (260) may be configured as one or more. In addition, the processor may execute a memory storing one or more programs.

[0310] Sleep information

[0311] According to one embodiment of the present invention, sleep information may be acquired from one or more sleep information sensor devices to generate one or more graphical user interfaces representing information about the user's sleep. The sleep information may include sleep sound information obtained noninvasively during the user's activity or sleep. Additionally, the sleep information may include the user's lifestyle information and log data.

[0312] Meanwhile, in the present invention, sleep information may include environmental sensing information and the user's lifestyle information. The user's lifestyle information may include information that affects the user's sleep. Specifically, information that affects the user's sleep may include the user's age, gender, medical condition, occupation, bedtime, wake-up time, heart rate, electrocardiogram, and sleep duration. For example, if the user's sleep duration is less than a standard amount, it may affect the user's sleep the next day by requiring more sleep. Conversely, if the user's sleep duration is more than the standard amount, it may affect the user's sleep the next day by requiring less sleep.

[0313] One or more sleep information sensor devices

[0314] According to one embodiment of the present invention, one or more sleep sensor devices may include a microphone module, a camera, and a light sensor provided in a user terminal (10).

[0315] For example, information related to the user's activities in a work space can be obtained through a microphone module equipped in the user terminal (10).

[0316] In addition, since the microphone module must be equipped in a relatively small user terminal (10), it may be configured as a MEMC (Micro-electro Mechanical System).

[0317] Environmental sensing information

[0318] In an embodiment, environmental sensing information of the present invention may be acquired through a user terminal (10). Environmental sensing information may refer to sensing information acquired in the space where the user is located. Environmental sensing information may be sensing information acquired in a non-contact manner related to the user's activity or sleep.

[0319] According to one embodiment of the present invention, it may mean sensing information acquired in an area (11a) capable of acquiring environmental sensing information, as illustrated in FIG. 2b, but is not limited thereto.

[0320] For example, environmental sensing information may be sleep sound information obtained in a bedroom where a user is sleeping. According to an embodiment, environmental sensing information obtained through the user terminal (10) may serve as the basis for obtaining user sleep state information in the present invention. For example, sleep state information regarding whether the user is before, during, or after sleep can be obtained through environmental sensing information obtained in relation to the user's activity.

[0321] For another example, environmental sensing information may include sounds generated by the user tossing and turning during sleep, sounds related to muscle movement, or sounds related to the user's breathing during sleep. Sleep sound information may refer to sound information related to movement patterns and breathing patterns occurring during the user's sleep. Environmental sensing information related to the user's activities in a space may be acquired through a microphone provided in the user terminal (10).

[0322] Alternatively, the environmental sensing information may include information on the user's breathing and movement. The user terminal (10) may include a radar sensor as a motion sensor. The user terminal (10) may process the user's movement and distance measured by the radar sensor to generate a discrete waveform (respiration information) corresponding to the user's breathing. Quantitative indicators may be obtained based on the discrete waveform and movement.

[0323] Environmental sensing information may include measurements obtained from sensors measuring the temperature, humidity, and lighting levels of the user's sleeping space. For this purpose, the user terminal (10) may be equipped with sensors that measure the temperature, humidity, and lighting levels of the bedroom.

[0324] According to one embodiment of the present invention, a graphical user interface generating device (100) or a graphical user interface providing device (200) can obtain sleep state information based on environmental sensing information obtained through a microphone module composed of MEMS.

[0325] Specifically, the graphical user interface generating device (100) or the graphical user interface providing device (200) can convert environmental sensing information that is acquired unclearly, including a lot of noise, into data that can be analyzed, and can perform learning on an artificial neural network by utilizing the converted data.

[0326] According to one embodiment of the present invention, when pre-training for an artificial neural network is completed, the trained neural network can acquire information about the user's sleep state based on a spectrogram acquired in response to sleep sound information. Specifically, the trained neural network may be an artificial intelligence sound analysis model, but is not limited thereto.

[0327] That is, the graphical user interface generation device (100) / providing device (200) according to embodiments of the present invention can obtain sound information having a low signal-to-noise ratio through a user terminal (e.g., artificial intelligence speaker, bedroom IoT device, mobile phone, wearable device, etc.) that is generally widely used to collect sound.

[0328] Alternatively, acoustic information or sleep acoustic information may be obtained through a microphone module provided in the graphical user interface generating device (100) / providing device (200) according to embodiments of the present invention.

[0329] When a graphical user interface generating device (100) or a graphical user interface providing device (200) or a user terminal (10) obtains sound information or sleep sound information having a low signal-to-noise ratio, it can process the data into data suitable for analysis and process the processed data to provide sleep state information.

[0330] Acoustic information or sleep acoustic information according to an embodiment of the present invention may be obtained through a microphone module configured to be in contact with the user's body, but may also be obtained through a microphone module that is not configured to be in contact with the user's body.

[0331] This can provide increased convenience by eliminating the need for a microphone to be placed in contact with the user's body to obtain clear sound, and by enabling monitoring of sleep status in a typical home environment through a software update without purchasing a separate additional device with a high signal-to-noise ratio.

[0332] Although the graphical user interface generation device (100) is expressed as a separate entity from the user terminal (10) in FIG. 1a, according to an embodiment of the present invention, as shown in FIG. 2a, the graphical user interface generation device (100) may be included within the user terminal (10) to perform the functions of measuring sleep status and generating a graphical user interface in a single integrated device.

[0333] Similarly, although the graphical user interface providing device (200) in FIG. 1b is expressed as a separate entity from the user terminal (10), according to an embodiment of the present invention, as shown in FIG. 2a, the graphical user interface providing device (200) may be included in the user terminal (10) to perform the functions of measuring sleep status and providing a graphical user interface in a single integrated device.

[0334] The user terminal (10) may refer to any type of entity(ies) in a system having a mechanism for communicating with an external server (20) or a separate electronic device. For example, the user terminal (10) may include a personal computer (PC), a notebook, a mobile terminal, a smart phone, a tablet PC, an artificial intelligence (AI) speaker, an artificial intelligence TV, and a wearable device, and may include all types of terminals capable of connecting to a wired / wireless network. In addition, the user terminal (10) may include any server implemented by at least one of an agent, an application programming interface (API), and a plug-in. In addition, the user terminal (10) may include an application source and / or a client application.

[0335] When an external server (20) according to one embodiment of the present invention receives sleep sound information, the received sleep sound information can be processed into appropriate data so that sleep state information can be generated based on the received sleep sound information.

[0336] According to one embodiment of the present invention, the external server (20) may be a server that stores information on a plurality of training data for training a neural network. The plurality of training data may include, for example, health checkup information or sleep checkup information. For example, the external server (20) may be at least one of a hospital server and an information server, and may be a server that stores information on a plurality of polysomnography records, electronic health records, electronic medical records, etc. For example, the polysomnography records may include information on breathing and movement during sleep of a sleep test subject and information on sleep diagnosis results (e.g., sleep stages, etc.) corresponding to the information. The information stored in the external server (20) may be utilized as training data, verification data, and test data for training the neural network in the present invention.

[0337] At least one of the user terminal (10), the graphical user interface generating device (100), or the graphical user interface providing device (200) according to embodiments of the present invention may receive health checkup information or sleep checkup information from an external server (20) and construct a learning data set based on the information. In addition, at least one of the user terminal (10), the graphical user interface generating device (100), or the graphical user interface providing device (200), or the external server (20) according to embodiments of the present invention may perform learning on one or more network functions through the learning data set, thereby generating a sleep analysis model for obtaining sleep state information corresponding to environmental sensing information, or may obtain sleep state information through the implemented sleep analysis model. A specific description of a configuration for constructing a learning data set for neural network learning of the present invention and a learning method utilizing the learning data set will be described later.

[0338] Acquisition of environmental sensing information

[0339] Environmental sensing information or sleep information according to embodiments of the present invention may be obtained from one or more sensor devices. Furthermore, a sensor device according to one embodiment of the present invention may be implemented in the form of a user terminal (10).

[0340] Environmental sensing information may refer to sensing information acquired in the space where the user is located. Environmental sensing information may be sensing information acquired in the space where the user is located using a non-contact method.

[0341] For example, environmental sensing information may be acoustic information acquired in a bedroom where a user is sleeping. In an embodiment, environmental sensing information acquired through the user terminal (10) may serve as the basis for acquiring user sleep status information in the present invention. For example, sleep status information regarding whether the user is before, during, or after sleep can be acquired through environmental sensing information acquired in relation to the user's activity.

[0342] In addition, the environmental sensing information may be at least one of noise information commonly occurring in daily life (sound information related to cleaning, sound information related to cooking, sound information related to watching TV, cat sounds, dog sounds, bird sounds, car sounds, wind sounds, rain sounds, etc.) or other biometric information (electrocardiogram, brain waves, pulse information, information related to muscle movement, etc.).

[0343] The user terminal (10) may refer to any type of entity(ies) in a system having a mechanism for communicating with the graphical user interface generating device (100) / providing device (200). For example, the user terminal (10) may include a personal computer (PC), a notebook, a mobile terminal, a smart phone, a tablet PC, an artificial intelligence (AI) speaker, an artificial intelligence TV, a wearable device, etc., and may include all types of terminals that can connect to a wired / wireless network. In addition, the user terminal (10) may include any server implemented by at least one of an agent, an application programming interface (API), and a plug-in. In addition, the user terminal (10) may include an application source and / or a client application.

[0344] According to one embodiment, the processor (160) can obtain environmental sensing information. Specifically, the environmental sensing information can be obtained through a user terminal (10) carried by the user. For example, environmental sensing information related to the space in which the user is active can be obtained through the user terminal (10) carried by the user, and the processor (160) can receive the corresponding environmental sensing information from the user terminal (10).

[0345] Additionally, an external server according to one embodiment of the present invention may record an artificial intelligence model for analyzing sleep state information. In this case, by acquiring environmental sensing information from a user terminal (10) or the like and transmitting the acquired environmental sensing information to an external server (20), the external server (20) can generate sleep state information based on the environmental sensing information through the embedded artificial intelligence model.

[0346] Alternatively, according to one embodiment of the present invention, environmental sensing information may be acquired from a user terminal (10), sleep sound information may be acquired through preprocessing of the environmental sensing information from the user terminal (10), the user terminal (10) may transmit the acquired sleep sound information to an external server, and the external server may generate sleep state information based on the received sleep sound information.

[0347] A graphical user interface generation device (100) / providing device (200) according to one embodiment of the present invention can receive health checkup information or sleep checkup information, etc. from an external server and construct a learning data set based on the information.

[0348] A graphical user interface generation device (100) / providing device (200) according to one embodiment of the present invention can generate a sleep analysis model for acquiring sleep state information based on environmental sensing information by performing learning on one or more network functions through a learning data set. A detailed description of the configuration for constructing a learning data set for neural network learning of the present invention and a learning method utilizing the learning data set will be described below.

[0349] An external server may be a digital device equipped with a processor, memory, and computing power, such as a laptop, notebook, desktop computer, web pad, or mobile phone. The external server may be a web server that processes services. The types of servers described above are merely examples and the present invention is not limited thereto.

[0350] According to one embodiment of the present invention, a graphical user interface generating device (100) / providing device (200) can obtain user's sleep state information, and generate and / or provide one or more graphical user interfaces that display information about the user's sleep based on the user's sleep state information.

[0351] Specifically, the graphical user interface generating device (100) / providing device (200) according to one embodiment of the present invention can obtain sleep state information related to whether the user is before, during, or after sleep based on environmental sensing information, and can generate and / or provide one or more graphical user interfaces that display information about the user's sleep based on the obtained sleep state information of the user.

[0352] Meanwhile, the specific description related to the aforementioned sleep state information is only an example, and the present invention is not limited thereto.

[0353] According to one embodiment of the present invention, one or more sleep sensor devices may include a microphone module, a camera, and a light sensor provided in a user terminal (10).

[0354] For example, information related to a user's activities in a space can be acquired through a microphone module equipped in a user terminal (10). In addition, when sensing information through a microphone module equipped in a user terminal (10), the microphone module must be equipped in a relatively small-sized user terminal (10), and thus may be configured as a MEMS (Micro-electro Mechanical System).

[0355] The microphone module according to embodiments of the present invention can be manufactured very compactly, but may have a lower signal-to-noise ratio (SNR) than a condenser microphone or a dynamic microphone. A low signal-to-noise ratio may mean that the ratio of noise, which is a sound that is not intended to be identified, to the sound to be identified is high, making it difficult to identify the sound (i.e., unclear).

[0356] Furthermore, the environmental sensing information analyzed in the present invention may include acoustic information related to the user's breathing and movements acquired during sleep. This acoustic information relates to extremely small sounds (i.e., sounds difficult to distinguish) such as the user's breathing and movements, and is acquired along with other sounds during sleep. Therefore, if acquired through a microphone module with a low signal-to-noise ratio, such as the aforementioned one, detection and analysis can be extremely difficult.

[0357] In such a case, an electronic device according to an embodiment of the present invention can convert and / or adjust environmental sensing information that is acquired unclearly, including a lot of noise, into data that can be analyzed, and can perform learning for an artificial neural network using the converted and / or adjusted data. When pre-learning for the artificial neural network is completed, the learned neural network (e.g., an acoustic analysis model) can acquire information on the user's sleep state based on data (e.g., a spectrogram) acquired (e.g., converted and / or adjusted) corresponding to sleep sound information.

[0358] In an embodiment, the sleep state information may include not only information regarding whether the user is sleeping, but also sleep stage information regarding changes in the user's sleep stage during sleep. For example, the sleep state information may include sleep stage information indicating that the user was in REM sleep at a first time point and in light sleep at a second time point, which is different from the first time point. In this case, the sleep state information may be used to determine that the user was in relatively deep sleep at the first time point and in lighter sleep at the second time point.

[0359] That is, when the graphical user interface generating device (100) / providing device (200) according to embodiments of the present invention obtains sleep sound information having a low signal-to-noise ratio through a user terminal (e.g., an artificial intelligence speaker, a bedroom IoT device, a mobile phone, a wearable device, etc.) that is generally widely used to collect sound, it can process the data into data suitable for analysis and provide sleep state information related to changes in sleep stages by processing the processed data.

[0360] In an embodiment, the graphical user interface generating device (100) / providing device (200) may be a terminal or a server, and may include any type of device. The graphical user interface generating device (100) / providing device (200) may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, which has a computing capability equipped with a processor and memory. Alternatively, the graphical user interface generating device (100) / providing device (200) may be a web server that processes a service.

[0361] According to one embodiment of the present invention, the graphical user interface generation device (100) / providing device (200) may be a server that provides a cloud computing service. More specifically, the graphical user interface generation device (100) / providing device (200) may be a server that provides a cloud computing service, which is a type of Internet-based computing that processes information using another computer connected to the Internet rather than the user's computer.

[0362] The aforementioned cloud computing service can be a service that stores data online and allows users to access it anytime, anywhere via an internet connection without having to install the necessary data or programs on their own computers. It also allows users to easily share and transfer data stored online with simple operations and clicks. Furthermore, cloud computing services go beyond simply storing data on an internet server. Users can perform desired tasks using the functions of web-based applications without the need for separate program installation. It can also be a service that allows multiple people to simultaneously share and work on documents.

[0363] In addition, cloud computing services can be implemented in the form of at least one of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, and a container-based cloud server. That is, the graphical user interface generation device (100) / providing device (200) of the present invention can be implemented in the form of at least one of the above-described cloud computing services. The specific description of the above-described cloud computing service is merely an example, and may include any platform that constructs the cloud computing environment of the present invention. Meanwhile, the types of servers described above are merely examples, and the present invention is not limited thereto.

[0364] Sleep sound information

[0365] Meanwhile, in the present invention, sleep information may include environmental sensing information and user lifestyle information. The environmental sensing information may be acoustic information related to the user's sleep. One or more sleep information sensor devices may collect raw data regarding sounds generated during sleep for sleep analysis. The raw data regarding sounds generated during sleep may be time-domain information.

[0366] Meanwhile, in the present invention, by performing preprocessing on sound information acquired by a sleep information sensor device, it is also possible to acquire sleep sound information containing information on breathing and / or movement patterns.

[0367] Specifically, sleep sound information may imply information about the user's breathing and / or movement patterns related to sleep. For example, during a wakeful state, the entire nervous system is activated, so breathing patterns may be irregular and there may be a lot of body movement. Additionally, because the neck muscles do not relax, breathing sounds may be very small. Conversely, when the user is sleeping, the autonomic nervous system stabilizes, so breathing changes regularly, body movement may also decrease, and breathing sounds may become louder. Furthermore, if apnea occurs during sleep, loud breathing sounds may occur immediately after the apnea as a compensatory mechanism. Because sleep sound information implicitly contains such information about breathing and / or movement, raw data about sleep can be collected to acquire sleep sound information, and sleep analysis can be performed based on the acquired sleep sound information.

[0368] Information in the time domain

[0369] Meanwhile, in the present invention, sleep information may include at least one of environmental sensing information, sleep environment information, or user lifestyle information. The environmental sensing information or sleep environment information may be acoustic information for analyzing the user's sleep. One or more sleep information sensor devices may collect raw data regarding sounds generated during sleep for sleep analysis. The raw data regarding sounds generated during sleep may be time-domain information. The term "raw acoustic information" described below refers to raw data regarding sounds generated during sleep.

[0370] Preprocessing of information in the time domain

[0371] According to embodiments of the present invention, acquired environmental sensing information and acoustic information are information in the time domain and may undergo a preprocessing process of noise reduction.

[0372] In the noise reduction process, noise (e.g., white noise) contained in raw data is removed or reduced. The noise reduction process can be performed using algorithms such as spectral gating and spectral subtraction to remove or reduce background noise. Furthermore, in the present invention, the process of removing or reducing noise can be performed using a deep learning-based noise reduction algorithm. The deep learning-based noise reduction algorithm can utilize a noise reduction algorithm specialized for the user's breathing or respiration sounds, in other words, a noise reduction algorithm learned through the user's breathing or respiration sounds.

[0373] Preprocessing like the above can be performed during the learning process of sleep state information or during the inference process.

[0374] Spectral noise gating

[0375] Spectral gating, or spectral noise gating, is a preprocessing method for audio information. While noise reduction can be performed on the entire acquired audio information, it can also be performed by splitting the audio information at regular time intervals (e.g., every 5 minutes), and then performing noise reduction on each of the split audio information. To perform noise reduction on audio information split at regular time intervals, a method may first include calculating a spectrum for each frame.

[0376] Among each spectrum frame produced, the frame having the frequency spectrum with the lowest energy can be identified.

[0377] It may include a method of assuming that a frame having a frequency spectrum with the lowest energy among each spectrum frame is static noise, and attenuating the frequency of the frequency spectrum frame assumed to be static noise from the spectrum frame.

[0378] Deep learning-based noise reduction

[0379] According to the present invention, a deep learning-based noise reduction method that performs noise reduction preprocessing on raw acoustic information in the time domain, rather than the frequency domain, may be utilized. For deep learning-based noise reduction, information such as sleep sound information, which is required as input to a sleep analysis model, may be maintained, while other sounds are attenuated.

[0380] According to the present invention, noise reduction can be performed not only on acoustic information obtained through PSG test results, but also on acoustic information obtained through a microphone built into a user terminal such as a smartphone.

[0381] Generation / conversion and preprocessing of information or spectrograms in the frequency domain

[0382] In one embodiment of the present invention, raw time-domain information may be converted into frequency-domain information to analyze sleep sound information. Alternatively, according to embodiments of the present invention, raw time-domain information may be converted into information including changes in frequency components of the raw time-domain information along the time axis.

[0383] According to an embodiment of the present invention, raw sound information from which noise has been removed or reduced can be converted into information in the frequency domain or information including changes in frequency components of the raw sound information along the time axis. Preferably, it can be converted into a spectrogram. In this case, a method of converting the raw sound information into a spectrogram based only on amplitude, excluding phase, can be used, and through this method, not only privacy can be protected, but also processing speed can be improved by reducing data capacity. However, in another embodiment, it is also possible to generate a spectrogram using both phase and amplitude.

[0384] One embodiment of the present invention can create a sleep analysis model using a spectrogram (SP) converted based on sleep sound information (SS).

[0385] One embodiment of the present invention removes or reduces noise in sleep sound information using the above-described method, converts it into a spectrogram, and trains the spectrogram to create a sleep analysis model, thereby reducing the amount of computation and computation time and even protecting an individual's privacy.

[0386] For example, among the sound information acquired through a microphone, etc., the sleep sound information (e.g., the user's breathing sound, etc.) required for sleep stage analysis may be relatively smaller than other noises, but when converted to a spectrogram, the identification of the sleep sound information may be relatively superior compared to other surrounding noises.

[0387] Meanwhile, when converting to a spectrogram according to an embodiment of the present invention, personal information cannot be identified by converting the resolution of the frequency domain to a low level. In the case of configuring a frequency resolution (frequency bins) of less than a certain number (e.g., 20), personal information cannot be identified from the restored signal.

[0388] Additionally, according to an embodiment of the present invention, a method for converting acquired acoustic information into a spectrogram in real time may be included.

[0389] Additionally, since the compression of the frequency resolution of the spectrogram can be performed on the user's smartphone rather than on a server or cloud, it can also prevent leakage of personal information.

[0390] Meanwhile, the spectrogram according to an embodiment of the present invention may be a mel spectrogram to which a mel scale is applied.

[0391] Method for converting information or spectrograms in the frequency domain

[0392] Figure 9a is a drawing for explaining a process of obtaining sleep sound information in a sleep analysis method according to the present invention.

[0393] FIG. 9b is a drawing for explaining a method for obtaining a spectrogram corresponding to sleep sound information in a sleep analysis method according to the present invention.

[0394] The processor (160) can generate a spectrogram (SP) corresponding to sleep sound information (SS), as illustrated in FIG. 9B. Raw data (raw sound information in the time domain) that serves as the basis for generating the spectrogram (SP) can be input. The raw data according to the present invention can be collected through a polysomnography (PSG) in a hospital environment, or can be collected by a user in a home environment, for example, through a microphone built into a user terminal such as a wearable device or a smartphone.

[0395] In addition, raw data may be acquired through the user terminal (10) from the start time input by the user to the end time, or may be acquired from the time when the user operates the device (e.g., setting an alarm) to the time corresponding to the device operation (e.g., alarm setting time), or may be acquired by automatically selecting a time point based on the user's sleep pattern, or may be acquired by automatically determining the time point based on the user's intended sleep time point based on sound (user's voice, breathing sound, sound of peripheral devices (TV, washing machine), etc.) or change in illumination, etc. Meanwhile, the intended sleep time point according to one embodiment of the present invention may be calculated from the user's intended sleep time point.

[0396] According to the present invention, the processor (160) can perform a fast Fourier transform on sleep sound information (SS) to generate a sleep spectrogram (SP). A spectrogram (SP) is intended to visualize and understand sounds or waves, and may be a combination of waveform and spectrum characteristics. The spectrogram (SP) may represent the difference in amplitude according to changes in the time axis and frequency axis as a difference in print density or display color.

[0397] According to the present invention, preprocessed sound-related raw data can be divided into 30-second units and converted into a spectrogram. Accordingly, the 30-second spectrogram has a dimension of 20 frequency bins x 1201 time steps. In the present invention, in order to change a rectangular spectrogram into a shape close to a square, various methods such as reshaping, resizing, and split-cat can be used, thereby converting it into a shape close to a square. Alternatively, by using these methods, it is possible to relatively preserve the amount of information.

[0398] Meanwhile, the present invention can utilize a method to simulate breathing sounds measured in various home environments by adding various noises generated in the home environment to clean breathing sounds. Because sounds have additive properties, they can be added to each other. However, adding original audio signals such as MP3 or PCM and converting them into spectrograms consumes a significant amount of computing resources. Therefore, the present invention proposes a method to convert breathing sounds and noise into spectrograms and then add them. Through this, breathing sounds measured in various home environments can be simulated and utilized for AI model training, thereby ensuring the robustness of the AI ​​model to information from various home environments.

[0399] Preprocessing of information or spectrograms in the frequency domain

[0400] The purpose of converting data into a spectrogram according to one embodiment of the present invention is to use the data as input to a sleep analysis model and infer, through a learned model, which sleep state or stage the pattern in the spectrogram corresponds to. However, several preprocessing steps may be required before using the data as input to the sleep analysis model. These preprocessing steps may be performed only during the learning process, or may be performed both during the learning process and the inference process, or may be performed only during the inference process.

[0401] The preprocessing process of the spectrogram according to embodiments of the present invention may include data augmentation preprocessing techniques such as a pitch shifting preprocessing method that inflates the amount of data by adding Gaussian noise to the data, a pitch shifting preprocessing method that slightly raises or lowers the pitch of the overall sound, a so-called TUT (Tile UnTile) augmentation method that converts a spectrogram or mel spectrogram into a vector during the learning process, randomly cuts (tiles) the converted vector at the input stage of one node (neuron), and recombines (untiles) it after the output of the node (neuron).

[0402] In addition, the data augmentation preprocessing method according to an embodiment of the present invention may include a noise addition augmentation method that adds noise generated in various environments other than Gaussian noise (e.g., external sounds, natural sounds, fan sounds, door opening or closing sounds, animal sounds, human conversation sounds, movement sounds, etc.).

[0403] In order to shorten the learning time when using a spectrogram as input to a learning model, the noise augmentation according to an embodiment of the present invention may include a method of artificially adding noise information on top of the sleep sound information and the spectrogram after converting noise information into a spectrogram. In this case, there may not be a significant difference between a spectrogram obtained by converting the entirety of the original sound information domain into which noise information is added to the sleep sound information, and a spectrogram obtained by adding noise information on a domain in which each of the sleep sound information and noise information is converted into a spectrogram.

[0404] In addition, the noise augmentation according to an embodiment of the present invention protects the user's privacy by making it difficult to convert back from the spectrogram to the original signal, by maintaining only the amplitude among the amplitude and phase in the spectrogram of each of the sleep sound information and noise information, and adding the phase by making it an arbitrary phase, thereby making it difficult to convert back from the spectrogram to the original signal.

[0405] Alternatively, the noise augmentation according to an embodiment of the present invention may include a method of adding sound information on a domain converted into a spectrogram, as well as a method of adding sound information on a domain converted into a mel spectrogram to which a mel scale is applied.

[0406] In addition, according to the method of adding in the Mel scale according to an embodiment of the present invention, the time required for hardware to process data can be shortened.

[0407] Additionally, a preprocessing method may be performed to convert information in the transformed frequency domain, information including changes in frequency components along the time axis, or a spectrogram into a form close to a square.

[0408] Sleep status information

[0409] In one embodiment, the sleep state information may include information regarding whether the user is sleeping. Specifically, the sleep state information may include at least one of first sleep state information indicating that the user is before sleeping, second sleep state information indicating that the user is sleeping, and third sleep state information indicating that the user is after sleeping. In other words, when the first sleep state information is inferred with respect to the user, the processor (160) may determine that the user is in a state before sleeping (i.e., before going to bed), when the second sleep state information is inferred, the processor (160) may determine that the user is in a state during sleeping, and when the third sleep state information is obtained, the processor (160) may determine that the user is in a state after sleeping (i.e., waking up).

[0410] Additionally, the sleep state information may include, in addition to information related to the user's sleep stage, information about at least one of sleep apnea, snoring, tossing and turning, coughing, sneezing, or teeth grinding (e.g., sleep event information).

[0411] In order to learn or infer sleep stage information according to embodiments of the present invention, acoustic information acquired over a long time interval may be required.

[0412] On the other hand, in order to learn or predict sleep state information (e.g., snoring or apnea information, etc.) other than sleep stage information according to embodiments of the present invention, acoustic information acquired during a relatively short time interval (e.g., 1 minute) before and after the time when the corresponding sleep state occurs may be required.

[0413] Such sleep state information may be characterized as being acquired based on environmental sensing information. The environmental sensing information may include sensing information acquired in a non-contact manner from the space where the user is located.

[0414] According to the present invention, the processor (160) can obtain sleep state information based on at least one of acoustic information, actigraphy, biometric information, environmental sensing information, and sleep information obtained from the user terminal (10).

[0415] Meanwhile, the processor (160) according to one embodiment of the present invention can identify singular points in acoustic information. Here, the singular points in the acoustic information may be related to breathing and movement patterns associated with sleep. For example, in a wakeful state, since the entire nervous system is activated, breathing patterns may be irregular and there may be a lot of body movement. Additionally, because the neck muscles do not relax, breathing sounds may be very small. On the other hand, when the user is sleeping, the autonomic nervous system stabilizes, breathing may become regular, body movement may also decrease, and breathing sounds may also increase. In other words, the processor (160) can identify the point in time when acoustic information related to patterns such as regular breathing, minimal body movement, or minimal breathing sounds is detected as a singular point in the acoustic information. Furthermore, the processor (160) can acquire sleep sound information based on the acoustic information acquired based on the identified singular points. The processor (160) can identify singular points related to the user's sleep time in the acoustic information acquired over time and acquire sleep sound information based on the singular points.

[0416] FIG. 9A is a diagram illustrating a process of acquiring sleep sound information in a sleep analysis method according to the present invention. Referring to FIG. 9A, the processor (160) can identify a singular point (P) related to a point in time when sound information of a pattern related to regular breathing, little body movement, or little breathing sound is detected from the sound information (E). The processor (160) can acquire sleep sound information (SS) based on sound information acquired after the identified singular point (P). The waveform and singular point related to sound in FIG. 4 are merely examples for understanding the present invention, and the present invention is not limited thereto.

[0417] That is, the processor (160) can identify a singular point (P) related to the user's sleep from the acoustic information, and can thereby extract and acquire only sleep sound information (SS) from a large amount of environmental sensing information (i.e., acoustic information) based on the singular point (P). This can provide convenience by automating the process of the user recording his or her sleep time, and can also contribute to improving the accuracy of the acquired sleep sound information.

[0418] In addition, in the embodiment, the processor (160) can obtain sleep state information related to whether the user is before or during sleep based on a singular point (P) identified from the acoustic information (E). Specifically, if the singular point (P) is not identified, the processor (160) can determine that the user is before sleep, and if the singular point (P) is identified, the processor can determine that the user is asleep after the singular point (P). In addition, after the singular point (P) is identified, the processor (160) can identify a point in time (e.g., a time of waking up) at which the identified pattern is no longer observed, and if the point in time is identified, the processor can determine that the user is after sleep, i.e., has woken up.

[0419] That is, the processor (160) can obtain sleep state information regarding whether the user is before, during, or after sleep based on whether a singular point (P) is identified in the acoustic information (E) and whether a preset pattern is continuously detected after the singular point is identified.

[0420] Meanwhile, the processor (160) can acquire sleep status information based on actigraphy or biometric information, rather than acoustic information (E). It may be advantageous to acquire the user's movement information through a sensor unit in contact with the body. Since the present invention uses actigraphy or biometric information to determine the user's sleep status in advance during the primary sleep analysis, the reliability of sleep status analysis can be further improved.

[0421] According to one embodiment of the present invention, the processor (160) can obtain environmental sensing information. Alternatively, according to one embodiment of the present invention, the environmental sensing information can be obtained through a user terminal (10) carried by the user. For example, environmental sensing information related to a space in which the user is active can be obtained through the user terminal (10) carried by the user, and the processor (160) can receive the corresponding environmental sensing information from the user terminal (10). According to one embodiment of the present invention, the processor (160) can obtain sleep sound information based on the environmental sensing information.

[0422] According to the present invention, environmental sensing information may be acoustic information acquired in a non-contact manner during the user's daily life. For example, environmental sensing information may include various acoustic information acquired according to the user's daily life, such as acoustic information related to cleaning, acoustic information related to cooking, acoustic information related to TV watching, and sleep acoustic information acquired during sleep. In an embodiment, sleep acoustic information acquired during the user's sleep may include sounds generated by the user tossing and turning during sleep, sounds related to muscle movement, or sounds related to the user's breathing during sleep. That is, sleep acoustic information in the present invention may refer to acoustic information related to movement patterns and breathing patterns associated with the user during sleep.

[0423] Alternatively, according to one embodiment of the present invention, at least one of the user terminal (10) or the external server (20) can generate or infer sleep state information. When first sleep state information is inferred with respect to the user, the processor provided in the user terminal (10) or the external server (20) can determine that the user is in a state before sleeping (i.e., before going to bed), when second sleep state information is inferred, the processor can determine that the user is in a state during sleep, and when third sleep state information is obtained, the processor can determine that the user is in a state after sleeping (i.e., waking up). Meanwhile, although the operation of generating sleep state information has been described as the operation of the processor (160), the above-described operation may also be performed by at least one of the processors of various electronic devices according to embodiments of the present invention.

[0424] Sleep stage information

[0425] According to one embodiment, the processor (160) can extract sleep stage information. The sleep stage information can be extracted based on the user's environmental sensing information. Sleep stages can be divided into NREM (non-REM) sleep and REM (Rapid eye movement) sleep, and NREM sleep can be further divided into multiple stages (e.g., two stages of Light and Deep, and four stages of N1 to N4). Sleep stage settings can be defined as general sleep stages, but can also be arbitrarily set to various sleep stages depending on the designer. Through sleep stage analysis, not only sleep quality related to sleep but also sleep disorders (e.g., sleep apnea) and their underlying causes (e.g., snoring) can be predicted. Meanwhile, although the operation of extracting sleep stage information has been described as an operation of the processor (160), the above-described operation may also be performed by at least one processor of various electronic devices according to embodiments of the present invention.

[0426] In addition, the processor (160) according to one embodiment of the present invention can generate environment creation information based on sleep stage information. For example, when the sleep stage is in the Light stage or the N1 stage, environment creation information for controlling environment creation devices (lighting, air purifier, etc.) to induce deep sleep or REM sleep can be generated. Meanwhile, the operation of generating environment creation information has been described as the operation of the processor (160), but the above-described operation may also be performed by at least one of the processors of various electronic devices according to embodiments of the present invention.

[0427] According to one embodiment of the present invention, when the word indicating a light sleep stage is displayed in Korean, it may be displayed as "light sleep" or "normal sleep." For users unfamiliar with sleep stages, the term "light sleep" may mislead them into thinking they haven't slept properly. In contrast, displaying the word "normal sleep" has the potential to reduce this misunderstanding.

[0428] Hypnogram

[0429] According to the present invention, a hypnogram is a graph that represents sleep stage information as a function of time. Through a hypnogram graph, sleep can be divided into REM sleep and non-REM sleep, from the time of falling asleep to the time of waking. Alternatively, a hypnogram can be used to represent sleep stage information for a total of four stages: REM sleep, deep sleep, light sleep, and wakefulness.

[0430] Figures 17a and 17b are graphs verifying the performance of the sleep analysis method according to the present invention, and are drawings comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention.

[0431] As illustrated in FIG. 17b, the sleep stage information obtained according to the present invention is not only highly consistent with polysomnography, but also includes more precise and meaningful information related to sleep stages (Wake, Light, Deep, REM).

[0432] The hypnodensity graph illustrated at the bottom of Fig. 17a is a graph representing sleep stage probability information that represents the probability of which sleep stage it belongs to among the four sleep stage classes. According to an embodiment of the present invention, when predicting sleep stage information through the hypnodensity graph, it is possible to represent the probability (i.e., sleep stage probability information) of which sleep stage class it belongs to among the four classes (Wake, Light, Deep, REM) in units of cycles according to one or more epochs, and furthermore, it is possible to represent the probability of which sleep stage class it belongs to among the five classes (Wake, N1, N2, N3, REM), the probability of which sleep stage class it belongs to among the three classes (Wake, Non-REM, REM), and the probability of which sleep stage class it belongs to among the two classes (Wake, Sleep). Here, the sleep stage probability information may mean a numerical representation of the proportion of a given sleep stage in a given epoch when classifying sleep stages.

[0433] The hypnogram, a graph depicted above the hypnodensity graph, can be obtained by determining the sleep stage with the highest probability from the hypnodensity graph, according to one embodiment of the present invention. As illustrated in Figure 17b, the sleep analysis results obtained according to the present invention showed very consistent performance when compared to labeled data obtained through polysomnography.

[0434] Meanwhile, FIG. 4A is a drawing for explaining another example of a hypnogram indicating a sleep stage within a user's sleep period according to one embodiment of the present invention.

[0435] Hypnograms can generally be obtained through electroencephalograms (EEGs), electrooculography (EOGs), electromyography (EMGs), and polysomnography (PSGs).

[0436] As shown in Figure 4a, the hypnogram can represent sleep stages divided into REM sleep and non-REM sleep. For example, it can be represented as four stages: REM sleep, deep sleep, light sleep, and wakefulness. Details on the graphical user interface for displaying the hypnogram will be described later.

[0437] Sleep intention information

[0438] According to one embodiment of the present invention, the processor (160) can obtain sleep intention information based on environmental sensing information. According to one embodiment, the processor (160) can identify the type of sound included in the environmental sensing information.

[0439] Additionally, the processor (160) can calculate sleep intention information based on the number of types of identified sounds. The processor (160) can calculate lower sleep intention information as the number of types of sounds increases, and can calculate higher sleep intention information as the number of types of sounds decreases.

[0440] For example, if there are three types of sounds included in the environmental sensing information (e.g., vacuum cleaner sound, TV sound, and user voice), the processor (160) can calculate sleep intention information as two points. Also, for example, if there is one type of sound included in the environmental sensing information (e.g., washing machine), the processor (160) can calculate sleep intention information as six points.

[0441] That is, the processor (160) can obtain sleep intention information related to the degree to which the user intends to sleep based on the number of types of sounds included in the environmental sensing information. For example, the more types of sounds are identified, the lower the user's sleep intention information (i.e., sleep intention information with a low score) can be output.

[0442] The specific numerical descriptions of the types of sounds and sleep intention information included in the aforementioned environmental sensing information are merely examples, and the present invention is not limited thereto. Furthermore, while the operation of calculating sleep intention information has been described as the operation of the processor (160), the above-described operation may also be performed by a processor included in another electronic device disclosed in the present invention (e.g., a processor of a user terminal (10) or an external server (20), etc.).

[0443] Obtaining sleep intention information using the intention score table method

[0444] In addition, in an embodiment, the processor (160) may create or record an intention score table by pre-matching different intention scores to each of a plurality of pieces of acoustic information. For example, a first piece of acoustic information related to a washing machine may be pre-matched with an intention score of 2 points, a second piece of acoustic information related to the sound of a humidifier may be pre-matched with an intention score of 5 points, and a third piece of acoustic information related to a voice may be pre-matched with an intention score of 1 point. The processor (160) may pre-match a relatively high intention score to acoustic information related to the user's sleep (e.g., sounds generated by the user's activities, such as vacuum cleaners, dishwashing, and voice sounds), and may pre-match a relatively low intention score to acoustic information unrelated to the user's sleep (e.g., sounds unrelated to the user's activities, such as vehicle noise and rain sounds) to create an intention score table. The specific numerical description of the intention scores matched to each piece of acoustic information described above is merely an example, and the present invention is not limited thereto.

[0445] According to the present invention, the processor (160) can obtain sleep intention information based on environmental sensing information and an intention score table. Specifically, the processor (160) can record an intention score matched to an identified sound in response to a point in time when at least one of a plurality of sounds included in the intention score table in the environmental sensing information is identified. For example, when a vacuum cleaner sound is identified in response to a first point in time in a process of acquiring environmental sensing information in real time, the processor (160) can record two intention scores matched to the vacuum cleaner sound by matching them to the first point in time. In the process of acquiring environmental sensing information, the processor (160) can record an intention score matched to the identified sound by matching it to the corresponding point in time each time various sounds are identified.

[0446] In an embodiment, the processor (160) may obtain sleep intention information based on the sum of intention scores acquired over a predetermined period of time (e.g., 10 minutes). For example, a higher intention score acquired over a 10-minute period may indicate higher sleep intention information, while a lower intention score acquired over a 10-minute period may indicate lower sleep intention information. The specific numerical description of the aforementioned predetermined period of time is merely an example, and the present invention is not limited thereto.

[0447] That is, the processor (160) can acquire sleep intention information related to the degree to which the user intends to sleep based on the characteristics of the sound included in the environmental sensing information. For example, the more sounds related to the user's activity are identified, the lower the user's sleep intention information (i.e., sleep intention information with a low score) can be output.

[0448] Meanwhile, although the above operation has been described as an operation of a processor (160), the above operation may also be performed by a processor provided in another electronic device disclosed in the present invention (e.g., a processor of a user terminal (10) or an external server (20), etc.).

[0449] Sleep event information

[0450] Sleep events according to one embodiment of the present invention include various events that may occur during sleep, such as snoring, sleep breathing (e.g., including information related to sleep apnea), and teeth grinding.

[0451] According to one embodiment of the present invention, sleep event information indicating that a given sleep event has occurred, or sleep event probability information indicating the probability of determining that a given sleep event has occurred, may be generated. Sleep respiration information, an example of sleep event information, will be described below.

[0452] Fig. 18 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a diagram comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI ​​algorithm according to the present invention in relation to sleep apnea and hypopnea.

[0453] The probability graph shown at the bottom of Figure 18 shows the probability of which of the two diseases (sleep apnea, hypopnea) a user belongs to in 30-second units when predicting sleep disorders by inputting user sleep sound information.

[0454] Among the three graphs shown in Figure 18, the first graph shown in the AI ​​result can be obtained by determining the disease with the highest probability from the probability graph shown below it.

[0455] Using sleep analysis according to the present invention, as illustrated in Figure 18, the sleep state information obtained according to the present invention demonstrated performance highly consistent with polysomnography. Furthermore, the sleep analysis demonstrated performance that included more precise analysis information related to apnea and hypopnea.

[0456] According to the present invention, by analyzing a user's sleep in real time, the point at which a sleep disorder (sleep apnea, hyperventilation, or hypoventilation) occurs can be identified. Providing the user with a stimulus (tactile, auditory, olfactory, etc.) at the moment the sleep disorder occurs can temporarily alleviate the sleep disorder. In other words, the present invention can stop a user's sleep disorder and reduce its frequency based on accurate event detection related to sleep disorder.

[0457] In a probability graph according to one embodiment of the present invention, when predicting a sleep disorder by inputting user sleep sound information, the probability of which of two disorders (sleep apnea, hypopnea) belongs to each disorder can be displayed in 30-second units, but is not limited to 30 seconds.

[0458] Sleep analysis model

[0459] FIG. 15a and FIG. 15b are drawings for explaining the overall structure of a sleep analysis model according to one embodiment of the present invention.

[0460] According to one embodiment of the present invention, the sleep analysis model may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep stage information.

[0461] In the present invention, sleep state information can be obtained through a sleep analysis model that analyzes the user's sleep stage based on acoustic information (sleep acoustic information).

[0462] In the present invention, sleep sound information (SS) is a sound related to breathing and body movement acquired during the user's sleep time, so it can be a very small sound. Accordingly, the present invention can perform an analysis of the sound by converting the sleep sound information (SS) into a spectrogram (SP) as described above. In this case, since the spectrogram (SP) contains information showing how the frequency spectrum of the sound changes over time, breathing or movement patterns related to relatively small sounds can be easily identified, thereby improving the efficiency of the analysis. Specifically, it may be difficult to predict whether the user is in at least one of an awake state, a REM sleep state, a light sleep state, and a deep sleep state based solely on changes in the energy level of sleep sound information. However, by converting sleep sound information into a spectrogram, changes in the spectrum of each frequency can be easily detected, thereby enabling analysis corresponding to small sounds (e.g., breathing and body movement).

[0463] According to the present invention, the processor (160) can acquire sleep state information by processing a spectrogram (SP) as input to a sleep analysis model. Here, the sleep analysis model is a model for acquiring sleep state information related to changes in the user's sleep stage, and can output sleep state information by inputting sleep sound information acquired during the user's sleep. In an embodiment, the sleep analysis model may include a neural network model configured through one or more network functions.

[0464] According to one embodiment of the present invention, a sleep analysis model (symbol A) utilizing deep learning for analyzing a user's sleep disclosed in FIG. 15b can perform sleep information inference (symbol E) through a feature extraction model (symbol B), an intermediate layer (symbol C), and a feature classification model (symbol D). The sleep analysis model (symbol A) utilizing deep learning configured through the feature extraction model (symbol B), the intermediate layer (symbol C), and the feature classification model (symbol D) performs both time-series feature learning and feature learning for multiple images, and the sleep analysis model (symbol A) utilizing deep learning learned through this can infer sleep stages for the entire sleep time and infer sleep events occurring in real time.

[0465] Network functions and neural networks

[0466] FIG. 11 is a schematic diagram illustrating one or more network functions for performing a sleep analysis method according to the present invention.

[0467] According to the present invention, a sleep analysis model is composed of one or more network functions, and the one or more network functions may be composed of a set of interconnected computational units, which may generally be referred to as "nodes." These "nodes" may also be referred to as "neurons." The one or more network functions are composed of at least one node. The nodes (or neurons) constituting the one or more network functions may be interconnected by one or more "links."

[0468] According to the present invention, within a neural network, one or more nodes connected through links can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, and any node in an output node relationship with respect to one node can also be in an input node relationship with respect to another node, and vice versa. As described above, the relationship between input nodes and output nodes can be created based on links. One or more output nodes can be connected to one input node through links, and vice versa.

[0469] In a relationship between input nodes and output nodes connected through a single link, the value of the output node can be determined based on the data input to the input node. Here, the node interconnecting the input node and the output node can have a weight. The weight can be variable and can be varied by the user or the algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values ​​input to the input nodes connected to the output node and the weight set for the link corresponding to each input node.

[0470] As described above, a neural network is a network in which one or more nodes are interconnected through one or more links, forming input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks have the same number of nodes and links but different weight values ​​between the links, the two neural networks can be recognized as different from each other.

[0471] Some of the nodes that make up a neural network can form a layer based on their distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be passed to reach the node from the initial input node. However, this definition of a layer is arbitrary for explanatory purposes, and the order of layers within a neural network can be defined in a different way than described above. For example, a layer of nodes can also be defined by its distance from the final output node.

[0472] The initial input node may refer to one or more nodes in a neural network into which data is directly input without going through a link in its relationship with other nodes. Alternatively, in a neural network, in terms of the relationship between nodes based on links, it may refer to nodes that do not have other input nodes connected by links. Similarly, the final output node may refer to one or more nodes in a neural network that do not have an output node in its relationship with other nodes. In addition, a hidden node may refer to nodes that constitute a neural network other than the initial input node and the final output node. A neural network according to one embodiment of the present invention may be a neural network in which the number of nodes in the input layer may be greater than that in the hidden layer closer to the output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer.

[0473] A neural network may include one or more hidden layers. Hidden nodes in a hidden layer can receive the output of the previous layer and the output of surrounding hidden nodes as input. The number of hidden nodes in each hidden layer may be the same or different. The number of nodes in the input layer may be determined based on the number of data fields in the input data and may be the same as or different from the number of hidden nodes. Input data input to the input layer may be computed by the hidden nodes in the hidden layer and output by the output layer, a fully connected layer (FCL).

[0474] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. DNNs can be used to identify latent structures in data. Specifically, they can identify the latent structures of images, text, videos, audio, and music (e.g., what objects are in a photo, the content and emotion of a text, the content and emotion of a voice, etc.).

[0475] Deep neural networks may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siamese networks, transformers, vision transformers (ViTs), mobile vision transformers (Mobile ViTs), and the like. The description of the above-described deep neural networks is merely an example and the present invention is not limited thereto.

[0476] In the present invention, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network that outputs output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be positioned between the input and output layers.

[0477] The number of nodes in each layer can be reduced from the number of nodes in the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to the output layer (symmetrical to the input layer).

[0478] Nodes in the dimensionality reduction and dimensionality restoration layers may or may not be symmetrical. Autoencoders can perform nonlinear dimensionality reduction. The number of input and output layers may correspond to the number of sensors remaining after preprocessing the input data.

[0479] In an autoencoder architecture, the number of nodes in the hidden layer within the encoder can decrease as it moves away from the input layer. However, if the number of nodes in the bottleneck layer (the layer with the fewest nodes between the encoder and decoder) is too small, it may not convey sufficient information, so it may be maintained above a certain number (e.g., more than half of the input layer).

[0480] Neural networks can be trained using at least one of supervised learning, unsupervised learning, and semi-supervised learning. The goal of neural network training is to minimize output errors. Training involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network in a direction that reduces the error. Supervised learning uses training data where the correct answer is labeled for each training data (i.e., labeled training data). Unsupervised learning, on the other hand, may not include the correct answer label for each training data. For example, in the case of supervised learning for data classification, the training data may be data where each training data category is labeled. Labeled training data is input to a neural network, and the error can be calculated by comparing the output (categories) of the neural network with the labels of the training data.

[0481] As another example, in unsupervised learning for data classification, errors can be calculated by comparing input training data with the output of a neural network. The calculated errors are backpropagated through the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated according to the learning rate. The amount of change in the updated connection weights of each node can be determined by the learning rate. The neural network's calculation of the input data and backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network training to quickly achieve a certain level of performance, thereby improving efficiency. A lower learning rate can be used in the later stages of training to improve accuracy.

[0482] In neural network training, training data can typically be a subset of real-world data (i.e., the data to be processed using the trained neural network). Therefore, there can be a learning cycle where errors on the training data decrease but errors on the real-world data increase. Overfitting is a phenomenon where excessive training on the training data leads to increased errors on the real-world data. For example, a neural network trained on yellow cats may fail to recognize non-yellow cats as cats, a phenomenon that can be a form of overfitting. Overfitting can increase errors in AI algorithms. Various optimization methods can be used to prevent overfitting. These methods include increasing the training data, regularization, and dropout, which omits some nodes from the network during the training process.

[0483] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. (Hereinafter, they are collectively referred to as neural networks.) A data structure may include a neural network. And a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and a loss function for learning the neural network. A data structure including a neural network may include any of the components disclosed above.

[0484] That is, a data structure including a neural network may be configured to include all or any combination of data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for training the neural network. In addition to the aforementioned configurations, a data structure including a neural network may include any other information that determines the characteristics of the neural network.

[0485] Additionally, the data structure may include any form of data used or generated in the computational process of the neural network, and is not limited to the aforementioned. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. The neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is composed of at least one node.

[0486] Feature extraction model and feature classification model

[0487] FIG. 16 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention.

[0488] The sleep analysis model used in the present invention may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep state information. The feature extraction model may extract features related to breathing sounds, breathing patterns, and movement patterns by analyzing the time-series frequency pattern of a spectrogram (SP). In one embodiment, the feature extraction model may be configured through a portion of a neural network model that has been pre-trained through a learning data set.

[0489] The sleep analysis model used in the present invention may include a feature extraction model and a feature classification model. The feature extraction model may be a deep learning model based on a natural language processing model capable of learning time-series correlations of given data. The feature classification model may be a learning model based on a natural language processing model capable of learning time-series correlations of given data. Here, the deep learning models based on a natural language processing model capable of learning time-series correlations may include, but are not limited to, Tarnsformer, ViT, MobileViT, and MobileViT2.

[0490] A learning data set according to one embodiment of the present invention may be composed of data in the frequency domain and a plurality of pieces of sleep state information corresponding to each piece of data.

[0491] Alternatively, a learning data set according to one embodiment of the present invention may be composed of a plurality of spectrograms and a plurality of sleep state information corresponding to each spectrogram.

[0492] Alternatively, a learning data set according to one embodiment of the present invention may be composed of a plurality of Mel spectrograms and a plurality of sleep state information corresponding to each Mel spectrogram.

[0493] For convenience of explanation below, the configuration and execution of a sleep analysis model according to one embodiment of the present invention will be described in detail based on a data set of spectrograms. However, the learning data utilized in the sleep analysis model of the present invention is not limited to spectrograms, and information in the frequency domain, information including changes in frequency components of information in the time domain along the time axis, spectrograms, or mel spectrograms can be utilized as learning data.

[0494] Among the sleep analysis models according to embodiments of the present invention, the feature extraction model may be pre-trained by a one-to-one proxy task that inputs one spectrogram and learns to predict sleep state information corresponding to one spectrogram. When a CNN deep learning model is employed in the feature extraction model according to embodiments of the present invention, learning may be performed by adopting a structure of a fully connected layer (FC) or a fully connected neural network (FCN). When a MobileViTV2 deep learning model is employed in the feature extraction model according to embodiments of the present invention, learning may be performed by adopting a structure of an intermediate layer.

[0495] Among the sleep analysis models according to an embodiment of the present invention, a feature classification model can be trained to input a plurality of consecutive spectrograms, predict sleep state information of each spectrogram, and analyze a sequence of a plurality of consecutive spectrograms to predict or classify overall sleep state information.

[0496] In addition, according to an embodiment of the present invention, after pre-training is performed through a one-to-one proxy task for a feature extraction model, fine tuning can be performed through a many-to-many task for the pre-trained feature extraction model and feature classification model. For example, a sequence of 40 consecutive spectrograms can be input into multiple feature extraction models trained through a one-to-one proxy task, and 20 pieces of sleep state information can be output, thereby inferring sleep stages. The specific numerical descriptions related to the number of spectrograms, the number of feature extraction models, and the number of sleep state information described above are merely examples, and the present invention is not limited thereto.

[0497] As described above, an inference model is created to extract the user's sleep state and stage through deep learning of environmental sensing information. Briefly, environmental sensing information, including acoustic information, is converted into a spectrogram, and an inference model is created based on the spectrogram.

[0498] The inference model can be built into the graphical user interface generation device (100) / providing device (200), as described above.

[0499] Thereafter, environmental sensing information including user sound information acquired through the user terminal (10) is input into the corresponding inference model, and sleep state information and / or sleep stage information is output as a result value. At this time, learning and inference may be performed in the same entity, but learning and inference may be performed in separate entities. That is, both learning and inference may be performed by the graphical user interface generation device (100) of FIG. 1A or the graphical user interface provision device (200) of FIG. 1B, and learning may be performed in the graphical user interface generation device (100) of FIG. 1A or the graphical user interface provision device (200) of FIG. 1B, while inference may be performed in the user terminal (10). Alternatively, at least one of learning and inference may be performed in an external server (20).

[0500] Alternatively, as in FIG. 2a, when the graphical user interface generation device (100) and the graphical user interface provision device (200) are implemented as a user terminal (10), both learning and inference may be performed by the user terminal (10).

[0501] Alternatively, as illustrated in FIG. 2b, when the present invention is implemented through a system comprising various electronic devices, learning or inference may be performed by at least one of the electronic devices illustrated in FIG. 2b. For example, learning or inference may be performed by an external server (20).

[0502] Hereinafter, a feature extraction model and a feature classification model according to an embodiment of the present invention will be described in detail.

[0503] Feature extraction model

[0504] According to the present invention, the feature extraction model can be comprised of a unique deep learning model trained through a training data set. The feature extraction model can be trained through supervised or unsupervised learning. The feature extraction model can be trained through the training data set to output data similar to the input data. Specifically, only the core characteristic data (or features) of the input spectrogram can be trained through a hidden layer. In this case, during the decoding process through the decoder, the output data of the hidden layer may be an approximation of the input data (i.e., the spectrogram) rather than a perfect copy.

[0505] According to the present invention, each of a plurality of spectrograms included in a learning data set can be tagged with sleep state information. Each of the plurality of spectrograms can be input to a feature extraction model, and the output corresponding to each spectrogram can be stored in a manner matching the tagged sleep state information. Specifically, when first learning data sets (i.e., a plurality of spectrograms) tagged with first sleep state information (e.g., shallow sleep) are input, features related to the output for the corresponding input can be stored in a manner matching the first sleep state information. In an embodiment, one or more features related to the output can be represented in a vector space. In this case, since the feature data output corresponding to each of the first learning data sets is an output through a spectrogram related to the first sleep stage, they can be located at a relatively close distance in the vector space. In other words, learning can be performed so that the plurality of spectrograms output similar features corresponding to each sleep stage.

[0506] According to the present invention, when a feature extraction model through the aforementioned learning process takes a spectrogram (e.g., a spectrogram converted to correspond to sleep sound information) as input, it can extract a feature corresponding to the spectrogram.

[0507] In an embodiment, the processor (160) may process a spectrogram (SP) generated in response to sleep sound information (SS) as input to a feature extraction model to extract features. Here, since the sleep sound information (SS) is time-series data acquired over time during the user's sleep, the processor (160) may divide the spectrogram (SP) into predetermined epochs. For example, the processor (160) may divide the spectrogram (SP) corresponding to the sleep sound information (SS) into 30-second units to obtain multiple spectrograms. For example, if sleep sound information is acquired during the user's 7-hour (i.e., 420 minutes) sleep, the processor (160) may divide the spectrogram into 30-second units to obtain 840 spectrograms. The specific numerical descriptions of the sleep time, the spectrogram division time unit, and the number of divisions described above are merely examples, and the present invention is not limited thereto.

[0508] According to the present invention, the processor (160) can process each of the plurality of segmented spectrograms as input to a feature extraction model to extract a plurality of features corresponding to each of the plurality of spectrograms. For example, if the number of the plurality of spectrograms is 840, the number of the plurality of features extracted by the feature extraction model corresponding thereto can also be 840. The specific numerical descriptions related to the number of spectrograms and the plurality of features described above are merely examples, and the present invention is not limited thereto.

[0509] Meanwhile, the feature extraction model according to an embodiment of the present invention can be trained using a one-to-one proxy task. Furthermore, during the process of learning to extract sleep state information for a single spectrogram, the feature extraction model can be trained to extract sleep state information by combining it with another NN (Neural Network).

[0510] According to an embodiment of the present invention, if learning is performed through a simple pre-learned Neural Network, the learning time of a feature extraction model can be shortened or the learning efficiency can be increased.

[0511] For example, according to one embodiment of the present invention, a spectrogram divided into 30-second units may be trained to output sleep state information by using the output vector as an input to a feature extraction model and using the output vector as an input to another NN.

[0512] Although sleep stages are predicted by using sleep sound information as input, sleep stage analysis and inference can be performed by using sleep sound information as input, such as information in the frequency domain, information including changes in frequency components of sleep sound information along the time axis, spectrogram, or mel spectrogram. Therefore, according to embodiments of the present invention, sleep sound information is converted into information in the frequency domain, information including changes in frequency components of sleep sound information along the time axis, spectrogram, or mel spectrogram and used as input for a sleep analysis model, so that, unlike existing sleep analysis models, sleep stages can be sensed or acquired in real time through analysis of the specificity of sleep patterns.

[0513] A graphical user interface that displays information about sleep

[0514] According to the present invention, a graphical user interface that displays information about a user's sleep can be provided.

[0515] FIGS. 19A to 19C are diagrams showing how a graphical user interface according to embodiments of the present invention is displayed on various display units.

[0516] As illustrated in FIGS. 19a to 19c, according to embodiments of the present invention, a graphical user interface indicating information about a user's sleep may be displayed on an electronic device or user terminal (10) implemented with various types of displays.

[0517] Hereinafter, a graphical user interface including a graph representing information about sleep according to embodiments of the present invention will be described using drawings.

[0518] Graphical user interface including hypnogram

[0519] FIG. 4A is a diagram illustrating a graphical user interface including a hypnogram according to one embodiment of the present invention.

[0520] A graphical user interface including a hypnogram according to an embodiment of the present invention may display the phrase “sleep stage” together, as indicated by reference numeral 101.

[0521] According to one embodiment of the present invention, the hypnogram may include a shape corresponding to each sleep stage information, with the x-axis representing time (reference number 114) and the y-axis representing sleep stage information.

[0522] According to FIG. 4A, sleep stage information according to one embodiment of the present invention may include a total of four stages, and may include multiple areas assigned to each sleep stage information. For example, it may be expressed by dividing into an area assigned to the wake stage (reference number 102), an area assigned to the REM sleep stage (reference number 103), an area assigned to the light sleep (or 'normal sleep') stage (reference number 104), and an area assigned to the deep sleep stage (reference number 105).

[0523] According to one embodiment of the present invention, the colors of the shapes corresponding to each sleep stage information may be expressed differently. For example, the color of the shape corresponding to the deep sleep stage may be relatively darker than the other shapes (reference number 113). Alternatively, the color of the shape corresponding to the REM sleep stage may be relatively brighter than the other shapes (reference number 111).

[0524] Alternatively, according to embodiments of the present invention, the colors of the shapes and backgrounds corresponding to each sleep stage information may be colors expressed at different locations in the color space.

[0525] For example, according to one embodiment of the present invention, the background may be expressed in a black color, a shape corresponding to a deep sleep stage may be expressed in a relatively dark blue color, a shape corresponding to a light sleep or normal sleep stage may be expressed in a relatively light blue color (reference number 112), a shape corresponding to a REM sleep stage may be expressed in a purple color, and a shape corresponding to a wake-up stage may be expressed in a yellow color.

[0526] According to embodiments of the present invention, shapes corresponding to sleep stage information may be shapes expressed discretely from each other.

[0527] In the present invention, the term 'discretely expressed figure' may mean that each figure is not connected to each other by a solid line or dotted line, etc.

[0528] In addition, in accordance with one embodiment of the present invention, since the shapes are expressed discretely, unlike in the prior art, there may be cases where shapes corresponding to different sleep stages share only one intersection point.

[0529] Additionally, according to one embodiment of the present invention, the fact that a shape is discretely expressed may include the meaning that it includes a plurality of shapes, each of which corresponds to a plurality of sleep stages, and at least one shape among the plurality of shapes is isolated from the remaining shapes.

[0530] Figures 20a to 20e are diagrams showing hypnogram graphs of sleep stage information expressed in a conventional sleep measurement interface.

[0531] As shown in FIGS. 20a to 20e, in the hypnogram graph included in the conventional sleep measurement interface, even if the shapes representing the sleep stages are separated from each other, there are cases where each shape is displayed as being connected to each other by a line such as a dotted line or solid line between the shapes.

[0532] For example, as can be seen from FIGS. 20a to 20e, in conventional techniques, when expressing a graph representing sleep stages, there were cases where changes between the first sleep stage and the second sleep stage were expressed continuously rather than discretely.

[0533] In addition, as shown in Fig. 20e, in order to express that there is continuity between the shapes even though the intersections of the shapes do not exist, there were cases where a line (dotted line or solid line) was connected between the point indicating the end of the first shape and the point indicating the start of the second shape.

[0534] However, the representation of hypnograms through these continuous graphs raises concerns that transitioning from stage 1 to stage 2 requires passing through another sleep stage. Transition patterns between sleep stages are crucial in sleep stage analysis, and the representation of continuous hypnograms may struggle to accurately represent these transition patterns.

[0535] According to the present invention, unlike the prior art, when a shape corresponding to each sleep stage information is discretely expressed according to one embodiment of the present invention, there is an effect that the misunderstanding that a different sleep stage must be passed through in the process of transitioning from the first sleep stage to the second sleep stage can be reduced.

[0536] According to the present invention, in the process of transitioning from the first sleep stage to the second sleep stage, it is not necessary to necessarily go through another sleep stage, so when a shape corresponding to each sleep stage information is discretely expressed, there is an advantage in that the transition between sleep stages can be intuitively understood.

[0537] For example, referring to the figure indicated by reference number 110 in FIG. 4a (a figure corresponding to the wake-up stage), the transition occurs from the light sleep (normal sleep) stage to the wake-up stage and then back to the light sleep (normal sleep) stage. If this transition or change in sleep stage is expressed as a 'continuous' hypnogram graph, it may cause a misunderstanding that the sleep stage must pass through the REM sleep stage when switching, but if it is expressed as a 'discrete' hypnogram graph according to the present invention, such misunderstanding can be reduced.

[0538] Furthermore, when representing each sleep stage discretely in a hypnogram, the frequency of each sleep stage is clearly expressed, allowing for direct confirmation of the frequency of sleep stages within a specific interval and facilitating the identification of the relative proportion of each sleep stage. This can be helpful in identifying characteristics or abnormalities in sleep patterns.

[0539] Meanwhile, according to one embodiment of the present invention, the shape corresponding to the sleep stage information may be expressed as at least one shape selected from the group consisting of a trapezoid, an isosceles trapezoid, a kite, a parallelogram, a rhombus, a rectangle, a square, and other general quadrilaterals. Preferably, it may be expressed as a rectangle.

[0540] As illustrated in FIG. 4A, multiple sleep stage information may be represented by multiple rectangles. At this time, at least one of the multiple rectangles may be isolated from the remaining rectangles, such as the rectangle indicated by reference numeral 110 in FIG. 4A. In the embodiment illustrated in FIG. 4A, the rectangle (110) corresponding to the wake stage of the sleep stage information is isolated from the remaining rectangles.

[0541] When performing sleep analysis, the wake stage may appear at least once during the sleep period. When expressing sleep stage information as a hypnogram according to embodiments of the present invention, a rectangle representing a wake stage can be displayed separately from rectangles representing other sleep stages. In this display, there is an advantage in that sleep can be analyzed by clearly distinguishing the occurrence of a wake stage during sleep. According to the interface of the prior art, when information on wake stages during sleep is frequently obtained, each sleep stage and the wake stage during sleep are connected with a line (solid line or dotted line), so there is a concern that the hypnogram may be expressed relatively unclearly, and there is a concern that the occurrence of a wake stage may not be clearly distinguished.

[0542] However, when sleep stage information is represented as a discrete and / or separate representation according to the present invention, the frequency of occurrence of sleep-wake stages can be clearly confirmed, thereby resolving the unclear errors of the prior art.

[0543] Additionally, according to one embodiment of the present invention, areas allocated to the wake stage, REM sleep stage, light sleep stage, and deep sleep stage may be displayed in order from top to bottom (see reference numbers 102 to 105).

[0544] Meanwhile, as shown in FIG. 20c or FIG. 20d, even if the shape representing the waking stage was displayed as if it were separated from the shapes representing other sleep stages, there was a problem in that it was difficult to clearly understand which sleep stage occurred at that time because the shapes corresponding to the sleep stage were displayed continuously in the area allocated to the other sleep stages.

[0545] For example, referring to FIG. 20c, although multiple graphs representing the wake stage during the entire sleep period are displayed as if they are separated from the graphs corresponding to other sleep stages, there is a problem in that it is unclear whether the person is sleeping or awake at that time because graphs are also displayed in areas representing the REM sleep stage, Light sleep stage, or Deep sleep stage at that time.

[0546] On the other hand, in an interface according to embodiments of the present invention, if at least one shape among shapes corresponding to multiple sleep stages is isolated from the remaining shapes, there is an effect that it is possible to clearly understand which sleep stage has occurred at that point in time by ensuring that no shapes exist in areas allocated to other sleep stages.

[0547] Additionally, in interfaces according to embodiments of the present invention, shapes corresponding to multiple sleep stages may be expressed as shapes of the same shape (e.g., rectangles), and at least one of the shapes may be isolated from the remaining shapes. In such embodiments, the shape is prevented from existing in areas allocated to other sleep stages represented by shapes of the same shape, thereby enabling a clear understanding of which sleep stage has occurred at a given time.

[0548] Additionally, among the shapes corresponding to a plurality of sleep stages included in the interface according to embodiments of the present invention, at least one shape corresponding to a sleep stage may have the same shape as a shape corresponding to another sleep stage.

[0549] Additionally, according to embodiments of the present invention, a shape corresponding to at least one sleep stage may be displayed separately from other shapes corresponding to the same sleep stage. Furthermore, a shape corresponding to at least one sleep stage may be displayed separately from shapes corresponding to other sleep stages.

[0550] According to an embodiment of the present invention, when a wake-up stage occurs, as shown in reference numeral 110 in FIG. 4A, a shape corresponding to the sleep stage is displayed only in the reference numeral 102 region at that time, and is not displayed in the regions allocated to the remaining sleep stages (reference numerals 103 to 105), so that a user can clearly understand which sleep stage (specifically, the wake-up stage) has occurred at that time just by looking at the hypnogram graph according to the present invention.

[0551] Furthermore, as illustrated in FIG. 4A, according to one embodiment of the present invention, when a word indicating a light sleep stage is displayed in Korean, it may be displayed as "light sleep" or "normal sleep." Users who are not expertly familiar with sleep stages may misunderstand that they did not sleep properly during the sleep period due to the word "light sleep." In contrast, when the word "normal sleep" is displayed, such misunderstandings are likely to be reduced (reference number 104).

[0552] According to one embodiment of the present invention, the shapes corresponding to each sleep stage can be displayed only in multiple areas allocated to each sleep stage information (see reference numbers 102 to 105). This display method allows for the clear distinction and presentation of sleep stage information during a sleep period, thereby enabling users to distinguish sleep stage information during a sleep period and accurately understand the graph representing sleep stage information.

[0553] According to one embodiment of the present invention, boundaries may exist between multiple regions assigned to each sleep stage information, and these boundaries may be indicated by dotted or solid lines (reference numeral 108). The lines representing the boundaries may be straight or curved.

[0554] Additionally, according to one embodiment of the present invention, the lines delineating the boundaries of the multiple regions assigned to each sleep stage information may be displayed with different colors or brightnesses. Specifically, the boundary line separating the wake stage region from the remaining sleep stage regions may be displayed with a relatively brighter color.

[0555] According to one embodiment of the present invention, each of the plurality of areas allocated to each piece of sleep stage information may have a height allocated thereto.

[0556] Additionally, according to one embodiment of the present invention, the heights assigned to the multiple areas assigned to each sleep stage information may be the same or different. Specifically, the height assigned to the area assigned to the wake stage information may be different from the heights assigned to other areas (see reference number 102).

[0557] According to one embodiment of the present invention, there may be a pitch assigned between a plurality of regions assigned to each sleep stage information.

[0558] According to one embodiment of the present invention, the shape corresponding to each sleep stage information can be discretely represented based on the pitch assigned between multiple regions assigned to each sleep stage information. This discrete representation allows the user to distinguish sleep stage information during a sleep period, allowing them to accurately understand the graph representing the sleep stage information.

[0559] As illustrated in FIG. 4A, according to one embodiment of the present invention, at least one piece of information from the time of falling asleep and the time of waking up may be displayed together in a hypnogram. In this case, at least one piece of information from the time of falling asleep and the time of waking up may be displayed as time information on the x-axis (reference numbers 106 and 107). In this case, when at least one piece of information from the time of falling asleep and the time of waking up is displayed together in a hypnogram, the time interval from the time of falling asleep to the time of waking up can be easily identified by looking at only the hypnogram. According to one embodiment of the present invention, when at least one piece of information from the time of falling asleep and the time of waking up is displayed as time information on the x-axis, it may be displayed by connecting a shape and a line corresponding to each sleep stage information. According to embodiments of the present invention, the line connecting at least one piece of information from the time of falling asleep and the time of waking up to the shape corresponding to the sleep stage information may be displayed as a solid line, a dotted line, a straight line, or a curved line.

[0560] Additionally, according to one embodiment of the present invention, as illustrated in FIG. 4a, comment information based on sleep stage information or sleep state information may be displayed together with a hypnogram graph representing sleep stages (reference number 109).

[0561] FIG. 4d is a diagram illustrating a graphical user interface including a hypnogram according to another embodiment of the present invention.

[0562] According to one embodiment of the present invention, in the case where sleep measurement has not been performed at all after the start of sleep measurement, only the figure corresponding to the waking stage may be displayed in the hypnogram, as shown in FIG. 4d.

[0563] According to one embodiment of the present invention, a hypnogram graph can be generated in real time during a sleep period. According to the present invention, sleep information, including the user's sleep sounds, is acquired in real time, and the acquired sleep sound information can be immediately converted into a spectrogram. The converted spectrogram can be used as input to a sleep analysis model to immediately analyze sleep stages. Accordingly, a hypnogram graph representing sleep stage information can be generated in real time simultaneously with the sleep stage analysis.

[0564] A graph showing the percentage of time spent in each sleep stage

[0565] FIG. 4b is a graph showing the time ratio of each sleep stage measured according to one embodiment of the present invention.

[0566] A graphical user interface including a graph displaying the time ratio of each sleep stage according to an embodiment of the present invention may be accompanied by the phrase "My Sleep at a Glance", as indicated by reference numeral 201.

[0567] The time ratio for each sleep stage according to the present invention can be calculated as the ratio of the time corresponding to each sleep stage to the total sleep period. Specifically, the total sleep time can be calculated as the time from the moment of falling asleep to the moment of waking up. The time corresponding to each sleep stage can be calculated based on the sleep stage inferred by the artificial intelligence model according to an embodiment of the present invention.

[0568] For example, as illustrated in reference numeral 203 of FIG. 4B, if the total sleep time from the time of falling asleep to the time of waking up is 7 hours and 22 minutes, the period inferred as the general sleep (light sleep) stage is 4 hours and 34 minutes, the period inferred as the REM sleep stage is 1 hour and 37 minutes, the period inferred as the deep sleep stage is 58 minutes, and the period inferred as the wake-up stage is 13 minutes, the ratio of each sleep stage can be displayed as a percentile value obtained by dividing the time corresponding to each period by the total sleep time. However, the numerical values ​​for the specific times described above are merely examples, and the present invention is not limited thereto.

[0569] In addition, in the graphical user interface according to an embodiment of the present invention, time information corresponding to each sleep stage and the ratio of each sleep stage can be displayed side by side in parallel on the left and right (reference number 202). In addition, as illustrated in reference numbers 204 to 207 of FIG. 4b, the time ratio corresponding to each sleep stage can be visually represented using a predetermined shape.

[0570] Here, the color of the shape corresponding to each sleep stage can be displayed in the same way as the color of the shape corresponding to the sleep stage information shown in the hypnogram of Fig. 4a.

[0571] In addition, according to one embodiment of the present invention, as illustrated in reference numerals 202 and 204 to 207 of FIG. 4b, the figures corresponding to each sleep stage may be displayed in order of increasing time ratio corresponding to each sleep stage; however, according to other embodiments, the order of the displayed sleep stages may be changed.

[0572] For example, the deep sleep stage may be configured to be displayed first and the wake stage last. Alternatively, the wake stage may be configured to be displayed first and the deep sleep stage last. Alternatively, the REM sleep stage may be configured to be displayed first and the normal sleep stage may be configured to be displayed first. This order may be directly selected by the user or may be automatically configured by an algorithm. The above-described order is merely an example, and the present invention is not limited thereto.

[0573] Respiratory stability graph

[0574] Figure 4c is a drawing showing a respiratory stability graph according to an embodiment of the present invention.

[0575] A graphical user interface including a respiratory stability graph according to an embodiment of the present invention may display the phrase “respiratory stability” together, as indicated by reference numeral 301.

[0576] Through sleep analysis according to one embodiment of the present invention, it is possible to predict sleep disorders (e.g., sleep apnea) and their underlying causes (e.g., snoring).

[0577] According to an embodiment of the present invention, the stability of breathing can be determined by criteria such as breathing cycle, breathing frequency fluctuation, and breathing pattern.

[0578] If your breathing pattern is irregular or changes suddenly during sleep, it may be considered unstable breathing.

[0579] Alternatively, breathing may be considered unstable if the frequency of breathing fluctuates significantly or shows irregular changes during sleep.

[0580] According to one embodiment of the present invention, when breathing is determined to be unstable, the breathing stability during that period may be indicated as "respiratory instability" (reference number 303). Furthermore, the breathing stability during a period not corresponding to "respiratory instability" may be indicated as "respiratory stability" (reference number 302).

[0581] As indicated by reference numerals 302 and 303 in Fig. 4c, respiratory stability can be represented as a graph over time. In this case, if the respiratory stability during sleep is judged to be unstable, the phrase "unstable" may also be displayed (reference numeral 306).

[0582] The apnea-hypopnea index (AHI) is calculated by dividing the total number of apneas and hypopneas recorded during sleep through polysomnography and other sleep analyses by the total sleep time. An AHI of less than 5 is considered normal, 5 to 15 is considered mild, 15 to 30 is considered moderate, and 30 or more is considered severe.

[0583] According to one embodiment of the present invention, an AHI value can be determined through sleep analysis using an artificial intelligence model, and a breathing stability ratio can be calculated and displayed on an interface based on the determined AHI value (reference numbers 304 and 305).

[0584] Additionally, according to one embodiment of the present invention, the color of the shape corresponding to a point where breathing is determined to be unstable can be displayed brighter. This display has the advantage of allowing for better detection of events that pose a problem in terms of breathing stability.

[0585] FIG. 21 is a diagram illustrating a graphical user interface that displays a graph representing sleep stage information and a graph representing sleep event information in parallel according to one embodiment of the present invention.

[0586] According to one embodiment of the present invention, as illustrated in FIG. 21, a hypnogram graph representing sleep stage information and a graph representing respiratory stability can be displayed side by side, one above the other, on the same time axis. This display allows the user to see at a glance the level of respiratory stability in each sleep stage, making it effective for analyzing sleep.

[0587] For example, if a breathing instability event occurs during the REM sleep stage, it can be interpreted as more serious. If the hypnogram graph representing sleep stage information and the graph representing breathing stability are displayed side by side on the same time axis, as shown above, the occurrence of such an event can be easily identified at a glance, making it easier to interpret and / or judge sleep information.

[0588] In addition, in cases where a hypnogram graph representing sleep stage information and a graph representing respiratory stability are displayed together in parallel on the same time axis in the embodiments of the present invention, there is an advantage in that such information can be utilized for providing various services, or such graphic image information can be utilized for training a deep learning model.

[0589] In addition, there is an advantage in that the correlation between sleep information identified from graphs displayed in parallel as above can be used to generate sleep evaluation information.

[0590] FIG. 4e is a diagram illustrating a graphical user interface including a description display for respiratory instability according to one embodiment of the present invention.

[0591] A graphical user interface including a description display for respiratory instability according to an embodiment of the present invention may be displayed together with the phrase “What is respiratory instability?”, as indicated by reference numeral 401.

[0592] As illustrated in Figure 4e, a graphical user interface may display a description of respiratory instability (reference numeral 402). Furthermore, if a user clicks on a portion of a designated area, indicated by reference numeral 403, the user may be directed to an external website describing respiratory instability or to a screen displaying information with more detailed information.

[0593] As illustrated in FIG. 4e, when a description of respiratory instability is displayed according to one embodiment of the present invention, a graphical user interface located in the background portion may be expressed relatively dark (reference numeral 404).

[0594] Screen showing sleep statistics

[0595] FIGS. 5A and 5B are diagrams illustrating a graphical user interface including statistical information of sleep state information according to embodiments of the present invention.

[0596] A graphical user interface including statistical information of sleep state information according to an embodiment of the present invention may display the phrase “sleep statistics” together, as indicated by reference numeral 501.

[0597] According to an embodiment of the present invention, daily sleep time can be expressed as a bar graph. Specifically, the date on which sleep status information was acquired can be displayed on the x-axis (reference number 503), and the sleep time value can be displayed on the y-axis (reference number 504).

[0598] According to an embodiment of the present invention, the sleep time calculated based on the sleep state information acquired on the day corresponding to the date on the x-axis can be displayed in the form of a bar graph (reference number 502).

[0599] Additionally, according to one embodiment of the present invention, information on the day of the week on which the corresponding sleep state information was acquired may be displayed in the upper part of the bar graph (reference number 505).

[0600] Additionally, according to one embodiment of the present invention, information indicating the average actual sleep time may be displayed at the bottom of the bar graph (reference numbers 506 to 508). Specifically, at least one of the following may be displayed: the average sleep time obtained over a given period, the average sleep time measured on weekdays, or the average sleep time measured on weekends.

[0601] Additionally, according to one embodiment of the present invention, the average of sleep time obtained over a given period, the average of sleep time measured during a weekday, or the average of sleep time measured during a weekend may all be displayed together.

[0602] Furthermore, according to one embodiment of the present invention, a graphical user interface may be provided that displays the average sleep time measured during a weekday and the average sleep time measured during a weekend as a line between the average values, as indicated by reference number 509, in order to compare the average sleep time measured during a weekday and the average sleep time measured during a weekend with the average sleep time measured during a predetermined period. In this case, the line for displaying the distinction may be a dotted line or a solid line, and may be expressed as a curve or a straight line.

[0603] According to one embodiment of the present invention, if sleep status information is acquired only on weekdays and sleep analysis is not performed on weekends, and thus sleep status information is not acquired, the graphical user interface may be displayed as illustrated at reference numeral 508 of FIG. 5A. In addition, in this case, bar graphs corresponding to the 11th (Saturday) and the 12th (Sunday) of FIG. 5A may not be formed. The specific description of the days of the week and dates described above is merely an example and is not limited thereto. For example, if sleep analysis is not performed on the 9th (Thursday), a bar graph corresponding to the 9th (Thursday) may not be formed.

[0604] Additionally, as illustrated in FIG. 5b, according to one embodiment of the present invention, when a specific bar graph is clicked among the bar graphs, sleep time information acquired on the date corresponding to the bar graph may be displayed (reference number 513). In this case, the corresponding bar graph may be displayed brightly (reference number 512), while other bar graphs may be displayed relatively darkly.

[0605] Additionally, as illustrated in FIGS. 5A and 5B, the average sleep time information obtained over a predetermined period may be displayed along with a bar graph. For example, if the average sleep time obtained over a predetermined period is 6 hours and 26 minutes, the average sleep time may be displayed as a line (e.g., a dotted line) or other shape at a position corresponding to 6 hours and 26 minutes on the y-axis of the bar graph (reference number 510).

[0606] In a graphical user interface including statistical information of sleep state information according to one embodiment of the present invention, as indicated by reference number 511, the phrase “sleep statistics” and information on the period in which the sleep state information was acquired (e.g., period information “2022.6.6 ~ 6.12”) may be displayed together.

[0607] Additionally, a graphical user interface including statistical information on sleep status information according to one embodiment of the present invention may display a brief description of the graph (e.g., "I slept this much this week" or "The more consistent the height of the sleep bar, the better"), as indicated by reference numeral 511. The brief description of the graph described above is merely an example, and the present invention is not limited thereto.

[0608] Sleep status information obtained over a week

[0609] FIGS. 6A and 6B are diagrams illustrating a graphical user interface including sleep status information acquired over a week according to embodiments of the present invention.

[0610] As illustrated in FIGS. 6A and 6B, sleep status information acquired over a week according to embodiments of the present invention may be expressed as a bar graph in which the x-axis represents the day of the week (reference number 601) and the y-axis represents time information. Here, the time information on the y-axis may be expressed in 2-hour units (reference number 602). In addition, a line (solid line or dotted line) corresponding to each time on the y-axis may also be displayed (reference number 603).

[0611] As illustrated in FIG. 6a, according to one embodiment of the present invention, the bar graph may be expressed entirely in a single color (reference number 604).

[0612] In addition, as illustrated in FIG. 6b, according to one embodiment of the present invention, the bar graph may display portions corresponding to sleep stages of REM sleep, deep sleep, light sleep, and wakefulness separately (reference numbers 605 to 608). Specifically, when the colors of the shapes corresponding to each sleep stage information in the hypnogram graph are expressed differently, the color of the shape corresponding to each sleep stage information in the bar graph for the corresponding date may be displayed in proportion to the corresponding sleep stage.

[0613] In addition, in the case where the parts corresponding to each sleep stage are displayed separately according to one embodiment of the present invention, the part corresponding to the wake stage may be arranged at the top of the bar graph (reference number 608), and the part corresponding to the deep sleep may be arranged at the bottom of the bar graph (reference number 605). Alternatively, the part corresponding to the sleep stage with a higher time ratio may be arranged at the bottom of the bar graph, and the part corresponding to the sleep stage with a lower time ratio may be arranged at the top of the bar graph. Meanwhile, this arrangement order is merely an example, and the present invention is not limited thereto. For example, the part corresponding to the deep sleep stage, the part corresponding to the light sleep stage, the part corresponding to the REM sleep stage, and the part corresponding to the wake stage may be arranged in that order from the bottom.

[0614] Sleep status information acquired over a specified period of time

[0615] FIGS. 7A and 7B are diagrams illustrating a graphical user interface including sleep state information acquired over a predetermined period of time according to embodiments of the present invention.

[0616] As illustrated in FIGS. 7a and 7b, sleep status information acquired over a week according to embodiments of the present invention can be expressed as a bar graph in which the x-axis represents the date (reference numbers 701a and 701b) and the y-axis represents time information.

[0617] Here, referring to reference numbers 701a and 701b of Fig. 7a, when sleep state information is obtained through sleep analysis performed from the date displayed at the top to the date displayed at the bottom, a bar graph can be displayed based on the information on the time of falling asleep and the information on the time of waking up included in the obtained sleep state information. For example, if it is determined that the person fell asleep after the evening of the 12th and woke up after the dawn of the 13th, it can be expressed in the shape of the leftmost bar graph among the bar graphs shown in Fig. 7a. At this time, as shown in Figs. 7a and 7b, the two ends of the bar graph of sleep state information obtained over a predetermined period may correspond to the time of falling asleep and the time of waking up, respectively.

[0618] Here, the time information on the y-axis may be expressed in units of time, but may also be expressed as words such as noon, dawn, midnight, evening, and noon, as illustrated in FIGS. 7A and 7B (reference number 702). In addition, a line (solid or dotted line) corresponding to each time on the y-axis may also be displayed (reference number 703).

[0619] As illustrated in FIG. 7a, in a case where lines corresponding to each time on the y-axis are displayed together according to one embodiment of the present invention, the line corresponding to midnight time can be distinguished by being displayed as a relatively thicker or brighter line than the lines corresponding to other times.

[0620] As illustrated in FIG. 7a, according to one embodiment of the present invention, the bar graph may be expressed entirely in a single color (reference number 704).

[0621] In addition, as illustrated in FIG. 7b, according to one embodiment of the present invention, the bar graph may display portions corresponding to sleep stages of REM sleep, deep sleep, light sleep, and wakefulness. Specifically, when the colors of the shapes corresponding to each sleep stage information in the hypnogram graph are expressed differently, the color of the shape corresponding to each sleep stage information in the bar graph for the corresponding date may be displayed in proportion to the corresponding sleep stage.

[0622] In addition, as illustrated in FIG. 7b, in a case where a portion corresponding to the sleep stages of REM sleep, deep sleep, light sleep, and wakefulness is displayed separately on a bar graph according to one embodiment of the present invention, the portion corresponding to each sleep stage can be arranged in the order of the time at which the corresponding sleep stage was detected.

[0623] Graphical user interface including a stacked sleep stage graph

[0624] Hereinafter, a stacked sleep stages graph will be described using FIGS. 8a to 8e, FIGS. 22a to 22e, and FIGS. 23a to 23d. Stacked sleep stage information refers to information that, when sleep state information is acquired during multiple sleep sessions, represents the frequency of occurrence of sleep stages included in the sleep state information acquired in each sleep session in a time-series manner.

[0625] FIGS. 8A through 8E are black and white drawings of a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

[0626] FIGS. 22A through 22E are diagrams illustrating a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

[0627] FIGS. 23A to 23D are diagrams illustrating a graphical user interface including a stacked sleep stage graph generated based on a larger number of sleep sessions compared to FIGS. 22A to 22E, according to embodiments of the present invention. According to one embodiment of the present invention, a method is disclosed for generating user sleep state information based on environmental sensing information acquired during one or more sleep sessions, and generating and providing a graph representing the frequency of a predetermined sleep state over time along a time axis based on the generated sleep state information. Here, a sleep session may be a term referring to a period from the start of sleep measurement to the end of sleep measurement. One sleep session means that sleep measurement is started once and ended once.

[0628] Meanwhile, according to one embodiment of the present invention, one or more sleep sessions that serve as the basis for generating the stacked sleep stage graph may or may not be consecutive sleep sessions.

[0629] For example, sleep state information can be generated over 30 sleep sessions, and a sleep stage graph can be generated based on all of the sleep state information generated over the 30 sleep sessions.

[0630] As another example, a sleep stage graph could be generated by selecting some of the 30 sleep sessions and generating a sleep stage graph based on the sleep state information generated during those sessions. Some of the sleep sessions could be consecutive or non-consecutive.

[0631] When generating a stacked sleep stage graph based on discontinuous sleep sessions, it is also possible to generate a stacked sleep stage graph based on sleep sessions measured during a specific period. For example, only sleep sessions measured on weekdays can be aggregated to generate a stacked sleep stage graph based on sleep state information generated on weekdays. Alternatively, only sleep sessions measured on weekends can be aggregated to generate a stacked sleep stage graph based on sleep state information generated on weekends. Alternatively, only sleep sessions measured during a specific period can be aggregated to generate a stacked sleep stage graph, or only sleep sessions measured under specific conditions can be aggregated.

[0632] By selecting these discrete sleep sessions and generating a stacked sleep stage graph based on the sleep state information generated from the selected sleep sessions, a so-called conditioned stacked sleep stage graph can be generated. This creates a graph that allows for easy visualization of sleep state information across multiple sleep sessions under specific conditions, and provides a graphical user interface including the generated graph.

[0633] According to embodiments of the present invention, a graphical user interface including a stacked sleep stage graph such as FIGS. 8A to 8E can be output to a display screen provided on various electronic devices such as a user terminal (10), a graphical user interface generating device (100), or a graphical user interface providing device (200).

[0634] The x-axis of each graph illustrated in FIGS. 8A to 8E represents the flow of time, and may be expressed in units of epochs, for example. According to one embodiment of the present invention, each epoch may be set with data corresponding to a unit of 30 seconds. For example, 200 epochs represent data corresponding to 10 minutes (i.e., 600 seconds), 400 epochs represent data corresponding to 20 minutes (i.e., 1200 seconds), ..., 1000 epochs represent data corresponding to 500 minutes (i.e., 30000 seconds). Meanwhile, the unit time of the epoch is only an example and is not limited thereto. For example, each epoch may be set with data corresponding to a unit of 60 seconds, in which case, the x-axis scale of the graphs may be expressed differently when compared to the graphs illustrated in FIGS. 8A to 8E.

[0635] In addition, the unit of ticks displayed on the x-axis of the stacked sleep stage graph according to embodiments of the present invention may be 200 epochs as shown in FIGS. 8A to 8E and FIGS. 22A to 22E, or 60 epochs as shown in FIGS. 23A to 23D, but this is merely an example and is not limited thereto.

[0636] Meanwhile, the y-axis of each graph shown in FIGS. 8a to 8e represents the frequency with which the corresponding sleep state (e.g., sleep stage) appears.

[0637] Referring to FIGS. 8A to 8E, in a graphical user interface including a stacked sleep stage graph, one or more areas corresponding to sleep stages may be expressed. Here, the sleep stage information may include information indicating that the sleep stage corresponds to a Wake sleep stage, information indicating that the sleep stage corresponds to a Light sleep stage, information indicating that the sleep stage corresponds to a Deep sleep stage, or information indicating that the sleep stage corresponds to a REM sleep stage.

[0638] In the stacked sleep stage graph according to embodiments of the present invention, the order in which areas indicating information corresponding to each sleep stage are displayed may be changed, and the order may be set arbitrarily.

[0639] For example, referring to the graphical user interfaces illustrated in FIGS. 8A to 8E and FIGS. 22A to 22E, in the stacked sleep stage graph, an area indicating information corresponding to the Wake sleep stage may be displayed at the bottom of the graph, and above that, an area indicating information corresponding to the Light sleep stage, an area indicating information corresponding to the Deep sleep stage, and an area indicating information corresponding to the REM sleep stage may be displayed in that order.

[0640] As another example, referring to the graphical user interface illustrated in FIGS. 23a to 23d, an area indicating information corresponding to the Wake sleep stage may be displayed at the very bottom of the graph, and above that, an area indicating information corresponding to the REM sleep stage, an area indicating information corresponding to the Light sleep stage, and an area indicating information corresponding to the DEEP sleep stage may be displayed in that order.

[0641] Meanwhile, according to one embodiment of the present invention, an area that is not measured further after sleep measurement in a sleep session is terminated may be expressed as an NA (Not Applicable) area in the stacked sleep stage graph. Meanwhile, the NA area may be an area where no value exists at all, or may be an area that is displayed in a way that is distinct from the area expressing the sleep stage by using different colors, brightness, etc. In the stacked sleep stage graph according to embodiments of the present invention, the total length of the x-axis is formed based on the longest sleep session among multiple sleep sessions. Since sleep state information to be displayed in the stacked sleep stage graph is not generated after sleep measurement is terminated in the remaining sleep sessions (i.e., sleep sessions having a shorter sleep session length than the longest sleep session), such an area may be expressed as NA. The NA area may be an area indicating that a sleep session has ended.

[0642] Referring to FIGS. 8A to 8E , in a graphical user interface including a stacked sleep stage graph, the classes of each sleep stage can be displayed in a distinct manner. For example, the Wake sleep stage can be displayed in yellow, the Light sleep stage in light blue, the Deep sleep stage in dark blue, and the REM sleep stage in red. Alternatively, the areas representing each sleep stage can be expressed as lines. For example, if each of the graphical user interfaces illustrated in FIGS. 8A to 8E is colored, it can be expressed correspondingly to each of the graphical user interfaces illustrated in FIGS. 22A to 22E .

[0643] According to one embodiment of the present invention, a user can be provided with accumulated sleep stage information through a graphical user interface including an accumulated sleep stage graph.

[0644] Hereinafter, a detailed description will be given of the accumulated sleep stage graph using FIGS. 8A and 22A. According to one embodiment of the present invention, it is assumed that the graphs illustrated in FIGS. 8A and 22A are graphs obtained by measuring a user's sleep 100 times. In the present invention, the fact that a user measured sleep 100 times is referred to as "100 sleep sessions." Furthermore, it is assumed that one epoch is set as data corresponding to 30 seconds. The specific figures for the number of measured sleep sessions and the time corresponding to each epoch are merely examples and are not limited thereto.

[0645] In the graphs shown in FIG. 8a and FIG. 22a, the area from 0 to 100 of the y-axis is filled with the Wake sleep stage while the value of the x-axis increases from 0 to 10. This confirms that, among the 100 sleep sessions of this user, during the time corresponding to the first 10 epochs (i.e., 300 seconds from the start of each sleep session), the sleep stage of this user was detected as the Wake stage for all 100 sleep sessions.

[0646] Next, a graph was displayed in which the area from 0 to 41 of the y-axis was filled with the Wake sleep stage while the values ​​on the x-axis increased from 10 to 20, but the area from 41 to 100 of the y-axis was filled with the Light sleep stage. Through this, it can be confirmed that among the 100 sleep sessions of this user, there were a total of 41 sleep sessions that detected the Wake sleep stage during the time corresponding to the 10th to the 20th epoch (i.e., during the time from 300 seconds to 600 seconds from the start of each sleep session), and there were a total of 59 (=100-41) sleep sessions that detected the Light sleep stage.

[0647] When the value of the x-axis is 30, a graph is displayed in which the area from 0 to 32 of the y-axis is filled with the Wake sleep stage, the area from 32 to 90 of the y-axis is filled with the Light sleep stage, and the area from 90 to 100 of the y-axis is filled with the Deep sleep stage. Through this, it can be confirmed that among the 100 sleep sessions of this user, at the time corresponding to the 30th epoch (i.e., when 900 seconds had passed from the start of each sleep session), there were a total of 32 sleep sessions that detected the Wake sleep stage, a total of 58 (=90-32) sleep sessions that detected the Light sleep stage, and a total of 10 (100-90) sleep sessions that detected the Deep sleep stage.

[0648] When the value of the x-axis is 60, the graph is displayed in which the area from 0 to 15 of the y-axis is filled with the Wake sleep stage, the area from 15 to 62 of the y-axis is filled with the Light sleep stage, the area from 62 to 95 of the y-axis is filled with the Deep sleep stage, and the area from 95 to 100 of the y-axis is filled with the REM sleep stage. Through this, we can confirm that among the 100 sleep sessions of this user, at the time corresponding to the 60th epoch (i.e., when 1800 seconds had passed from the start of each sleep session), there were a total of 15 sleep sessions that detected the Wake sleep stage, a total of 47 (= 62-15) sleep sessions that detected the Light sleep stage, a total of 33 (= 95-62) sleep sessions that detected the Deep sleep stage, and a total of 5 (100-95) sleep sessions that detected the REM sleep stage.

[0649] Meanwhile, if a cumulative sleep stage graph is generated based on sleep analysis over a sufficient number of sleep sessions, even if the user does not measure their sleep, it has the effect of allowing the user to estimate the probability of entering a certain sleep stage at a given point in time after sleeping simply by looking at the cumulative sleep stage graph.

[0650] Here, when a stacked sleep stage graph is generated based on sleep analysis over a sufficient number of sleep sessions, the sleep sessions may be continuous or discontinuous. When generating a stacked sleep stage graph based on discontinuous sleep sessions, a stacked sleep stage graph can also be generated based on sleep sessions measured over a specific period. Only sleep sessions measured on weekdays can be collected separately, or only sleep sessions measured over weekends can be collected separately. Alternatively, only sleep sessions measured over a certain period can be collected separately, or only sleep sessions measured under specific conditions can be collected separately.

[0651] By selecting a sufficient number of discrete sleep sessions and generating a stacked sleep stage graph based on the sleep state information generated from the selected sleep sessions, a conditional stacked sleep stage graph can be generated. Even if the user does not measure their sleep, simply viewing the previously generated conditional stacked sleep stage graph can provide an estimate of the probability of entering a given sleep stage at a given point in time after sleep under certain conditions.

[0652] For example, if a sufficient number of sleep sessions measured on a public holiday are aggregated to generate a conditional stacked sleep stage graph, then even if sleep is not measured directly on the public holiday, it is possible to estimate the probability of which sleep stage a person will be in at a specific point after falling asleep during the public holiday based on the previously generated conditional stacked sleep stage graph.

[0653] Furthermore, the accumulated sleep stage graph according to the present invention can also be used to estimate the sleep cycle during a user's sleep session. For example, as illustrated in FIGS. 8A and 22A, the boundary between the REM sleep stage region and the Deep sleep stage region cyclically increases and then decreases over time, confirming that the frequency of REM sleep stages repeatedly increases and then decreases during the user's sleep session.

[0654] The accumulated sleep stage graph according to the present invention has the effect of allowing the user's sleep duration to be determined over multiple sessions. As illustrated in the graphs of Figures 8a and 22a, a vertical boundary line exists between the NA region and the regions occupied by each sleep stage (Wake, Light, Deep, REM). The location of this boundary line on the graph and its length allow the duration of each sleep session to be determined.

[0655] For example, in the graphs illustrated in FIG. 8a and FIG. 22a, the values ​​on the x-axis at which a vertical boundary (y-axis direction) exists between the sleep stage region and the NA region correspond to x=80, 100 680, 740, 760,..., 920, respectively. For example, when the value of the x-axis is 680, a vertical boundary exists between the NA region and the sleep stage (REM) region, and the minimum y-value of this vertical boundary is 85. Through this, it can be seen that among the user's 100 sleep sessions, the number of times the user's sleep session ended at a time corresponding to the 680th epoch (i.e., a time point after 20400 (=680×30) seconds have passed from the start of each sleep session) was 15 (=100-85) times in total.

[0656] In addition, when the value of the x-axis is 760, there is a vertical boundary line between the NA region and the sleep stage (REM, Light) region, and the minimum y-value of this vertical boundary line is 75. Through this, we can see that among the user's 100 sleep sessions, the number of times the user's sleep session ended at the point corresponding to the 760th epoch (i.e., 22800 (=760×30) seconds after the start of each sleep session) was 25 (=100-75) times in total.

[0657] In the stacked sleep stage graph according to embodiments of the present invention, since the end of a sleep session is indicated by an NA area, the area corresponding to the sleep stage can form a form in which the y value monotonically decreases according to the passage of time on the x-axis.

[0658] Meanwhile, the end of a sleep session can mean that the user has finished measuring data for sleep analysis. It can also mean the user has completely awakened from sleep. The accumulated sleep stage graph has the advantage of easily identifying the frequency with which sleep sessions ended across multiple sleep sessions. In other words, it provides information on the epoch at which sleep ended. Furthermore, it allows for a quick overview of the distribution of sleep duration (or sleep session length) across multiple sleep sessions.

[0659] Recently, research is being conducted to intuitively represent graphs of sleep status information, including sleep stages.

[0660] To accurately measure sleep, polysomnography or home polysomnography must be used. However, because human sleep fluctuates greatly from day to day, conventional sleep measurement methods make it difficult to continuously monitor daily sleep. Sleep duration and stages can fluctuate due to various factors (e.g., lifestyle habits). Therefore, conventional methods have had the problem of making it difficult to analyze sleep across multiple days (or multiple sleep sessions). Furthermore, even when sleep analysis across multiple sleep sessions was performed, it was difficult to easily understand the results of the sleep analysis across multiple sleep sessions at a glance. As described using Figures 8a to 8e, the present invention effectively solves the aforementioned problems by enabling a user's sleep analysis across multiple sleep sessions to be understood at a glance through a stacked sleep stage graph.

[0661] Ease of understanding sleep through accumulated sleep stage graphs

[0662] Meanwhile, the sleep stage appears from the beginning of the sleep session and shows periodicity or a certain pattern as the sleep session progresses, so by presenting the sleep state information generated from the start time of the sleep session in a time series, it has the effect of making it easy to understand the information analyzed for sleep.

[0663] For normal sleep

[0664] For example, referring to the graphs illustrated in FIGS. 8b and 22b, it can be easily seen at a glance that the time from the start of the user's sleep session to the time of falling asleep was short as a result of analyzing sleep during multiple sleep sessions, and that the Wake stage did not appear much after sleep began. In addition, it can be seen at a glance that the Deep sleep stage was often detected in the early part of each sleep session after sleep began, and that the frequency of Deep sleep decreased over time and the REM sleep stage tended to be detected. In addition, since the sleep time identified from the position of the borderline with the NA area was observed to be more consistent than the sleep time in FIG. 8a, it can be easily seen that FIGS. 8b and 22b are graphs analyzing the sleep of a healthy person.

[0665] In addition, as shown in Fig. 23a, if the results of analyzing sleep over a larger number of sleep sessions than the graph shown in Fig. 22b are presented as a stacked sleep stage graph, the periodicity according to the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

[0666] Referring to the graph illustrated in Figure 23a, it can be seen that in multiple sleep sessions, the Deep sleep stage initially increases and then decreases in a cycle of approximately 1 to 1.5 hours after sleep begins. Furthermore, in the first Deep sleep stage cycle, the frequency of Deep sleep decreases around epoch 60, while the frequency of REM sleep increases. Then, around epoch 150, the frequency of Deep sleep increases and the frequency of REM sleep decreases.

[0667] Additionally, referring to the graph shown in Figure 23a, it can be seen that over time in each sleep session, the periodicity of REM sleep tends to weaken as REM sleep lasts longer.

[0668] For those suffering from insomnia,

[0669] Referring to the graphs shown in Figures 8c and 22c, the Wake phase was detected frequently throughout the sleep sessions, and the NA region occupies nearly half of the entire graph, indicating that each sleep session was short. This could indicate a short sleep time, making it easy to understand that Figures 8c and 22c are graphs analyzing the sleep of a person suffering from insomnia.

[0670] Additionally, as shown in Fig. 23b, if the results of analyzing sleep over a larger number of sleep sessions than the graph shown in Fig. 22c are presented as a stacked sleep stage graph, the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

[0671] Referring to the graph shown in Figure 23b, it can be seen that the Wake stage is detected frequently throughout sleep in multiple sleep sessions, and the NA region also appears frequently, so it can be determined that this is a graph analyzing the sleep of a person suffering from insomnia.

[0672] For sleep apnea

[0673] Referring to the graphs shown in Figs. 8d and 22d, it can be seen that the Light sleep stage was detected frequently throughout the sleep sessions, the Deep sleep stage was detected less frequently, and the Wake stage was consistently detected at every point throughout each sleep session. In addition, it can be seen that the Deep sleep stage was detected less frequently at the beginning of each sleep session, and the periodicity of the Deep sleep stage was not distinct. It is known that people with sleep apnea usually do not sleep deeply and the periodicity of the Deep sleep stage is not distinct. In light of this, it can be inferred that the graphs in Figs. 8d and 22d indicate that before transitioning from the Light sleep stage to the Deep sleep stage, there are frequent awakenings during sleep, deep sleep is not achieved, and the Light sleep ratio is high due to obstructive apnea caused by muscle relaxation.

[0674] In addition, as shown in Fig. 23c, if the results of analyzing sleep for a larger number of sleep sessions than the graph shown in Fig. 22d are displayed as a stacked sleep stage graph, the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

[0675] Referring to the graph shown in Figure 23c, it can be seen that the frequency of occurrence of Deep sleep stages does not appear to be cyclical across multiple sleep sessions. Furthermore, the frequency of occurrence of Light sleep stages is high throughout sleep. In light of this, it can be inferred that the graph shown in Figure 23c analyzes the sleep of a person suffering from sleep apnea.

[0676] For sleepers who suffer from both insomnia and sleep apnea

[0677] Referring to the graphs shown in Figures 8e and 22e, it can be seen that both the Wake and Light stages were frequently detected throughout the sleep sessions, while the Deep sleep stage was rarely detected. Therefore, considering that Figure 8e exhibits all the characteristics found in Figures 8c and 8d, it is easy to see that this graph analyzes the sleep of a person suffering from both sleep apnea and insomnia.

[0678] Additionally, as shown in Fig. 23d, if the results of analyzing sleep over a larger number of sleep sessions than the graph shown in Fig. 22e are presented as a stacked sleep stage graph, the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

[0679] Referring to the graph illustrated in Figure 23d, it can be seen that the frequency of occurrence of the Wake sleep stage is high throughout sleep across multiple sleep sessions, while the frequency of occurrence of the Deep sleep stage is not clearly periodic. Furthermore, the frequency of occurrence of the Light sleep stage is high throughout sleep. In light of this, it can be seen that the graph illustrated in Figure 23d is a graph analyzing the sleep of a person suffering from both sleep apnea and insomnia.

[0680] Meanwhile, according to one embodiment of the present invention, to more easily understand sleep analysis results, auxiliary lines representing various information may be displayed on the accumulated sleep stage graph. For example, an auxiliary line representing average sleep time may be displayed on the graph. Alternatively, an auxiliary line representing average sleep onset delay may be displayed on the graph. The examples of auxiliary lines are for illustrative purposes only and are not limiting; it is conceivable that auxiliary lines representing information belonging to at least one of various categories for sleep interpretation could be used.

[0681] The method for generating and providing a graphical user interface including a sleep stage graph based on sleep state information acquired during multiple sleep sessions has been described above, but it is not limited to sleep stage information, and the frequency with which sleep event information appears may also be generated in a similar manner to the stacked sleep stage graphs shown in FIGS. 8A to 8E. For example, the frequency of occurrence of sleep events during multiple sessions may be analyzed, and stacked sleep event information may be acquired based on the analysis.

[0682] Stacked sleep stage information refers to information that represents the frequency of occurrence of sleep events included in the sleep state information obtained from multiple sleep sessions in a time-series manner. Furthermore, a stacked sleep event graph based on this stacked sleep event information can be generated in a format identical to or similar to that shown in Figures 8a to 8e.

[0683] Alternatively, two graphs can be displayed in parallel by setting certain conditions. Alternatively, graphs can be created for comparison by overlapping them using methods such as adjusting shading.

[0684] For example, a stacked sleep stage graph and a stacked sleep event graph can be displayed in parallel, or the two graphs can be overlaid. By comparing the frequency of sleep stages and the frequency of sleep events over time across multiple sleep sessions, this approach allows for a multifaceted sleep analysis.

[0685] In addition, the results of analyzing at least one of the stacked sleep stage graph and the stacked sleep event graph according to the present invention can be labeled and used as learning data for an artificial intelligence model (e.g., an artificial intelligence model based on image processing).

[0686] Alternatively, the results of analyzing at least one of the accumulated sleep stage graph and the accumulated sleep event graph may be used as learning data for an artificial intelligence model (e.g., an artificial intelligence model based on natural language processing), thereby outputting the results of analyzing the graph by artificial intelligence.

[0687] Flowchart of the method for creating and providing a graphical user interface

[0688] According to the present invention, a graphical user interface that displays information about a user's sleep can be provided.

[0689] Hereinafter, using FIG. 10, a flowchart of a method for generating and providing a graphical user interface that displays information about a user's sleep according to embodiments of the present invention will be described.

[0690] FIG. 10 is a flowchart of a method for generating and providing one or more graphical user interfaces representing information about a user's sleep according to one embodiment of the present invention.

[0691] As illustrated in FIG. 10, according to one embodiment of the present invention, a method for generating one or more graphical user interfaces representing information about sleep may include a step of acquiring sleep information (S120), a step of converting sleep information acquired in the time domain into information in the frequency domain (S140), a step of generating a graphical user interface (S160), and a step of providing a graphical user interface (S180).

[0692] The sleep information acquired in the step of acquiring sleep information here may include environmental sensing information or sleep sound information.

[0693] According to one embodiment of the present invention, after the step (S140) of converting sleep information acquired in the time domain into information in the frequency domain, a step (not shown) of acquiring sleep state information may be further included.

[0694] In addition, according to one embodiment of the present invention, a method for generating one or more graphical user interfaces representing information about sleep may further include a sleep log storage step of storing sleep log information related to an account assigned to a user in a memory. Here, the sleep log information may be stored in at least one of various electronic devices according to embodiments of the present invention. For example, the sleep log information may be sleep state information stored in at least one of a user terminal (10), a graphical user interface generating device (100), a graphical user interface providing device (200), and an external server (20). In a case where a device (a first electronic device) in which sleep log information is stored and a device (a second electronic device) in which a graphical user interface is generated are different devices, the second electronic device may receive specific sleep state information among sleep state information recorded in the sleep log information stored in the first electronic device during a plurality of sleep sessions, generate a sleep state information graph based on the received sleep state information, and generate a graphical user interface including the generated sleep state information graph.

[0695] In addition, according to one embodiment of the present invention, the step (S140) of converting sleep information acquired in the time domain into information in the frequency domain may include a step of performing preprocessing on raw sound information in the time domain or information in the frequency domain.

[0696] Alternatively, according to one embodiment of the present invention, the step (S140) of converting sleep information acquired in the time domain into information in the frequency domain may include a step of converting acoustic information into spectrogram information in the frequency domain. In this case, the step of converting the spectrogram into a mel spectrogram by applying a mel scale may further include.

[0697] Meanwhile, according to one embodiment of the present invention, a step of converting sleep information obtained in the time domain into information including changes in frequency components along the time axis may be included.

[0698] In addition, a step (not shown) of obtaining sleep state information according to one embodiment of the present invention may include a step of extracting sleep state information corresponding to each piece of information in the frequency domain, spectrogram, or mel spectrogram divided into 30-second units.

[0699] And, according to one embodiment of the present invention, the step (S160) of generating a graphical user interface may include a hypnogram graph generation step of generating a graph for sleep stages within a user's sleep period based on sleep information.

[0700] In addition, according to one embodiment of the present invention, the sleep information that serves as a basis for generating a hypnogram graph may be at least one of sleep information obtained in the time domain, information in the frequency domain, information including changes in frequency components of the sleep information along the time axis, a spectrogram, or a mel spectrogram to which a mel scale is applied.

[0701] Additionally, the step (S160) of generating a graphical user interface according to one embodiment of the present invention may include a step of generating a graph of the user's sleep stability within the user's sleep period based on sleep information.

[0702] The step (S180) of providing a graphical user interface according to one embodiment of the present invention may further include a step of outputting the graphical user interface to a display unit provided in various electronic devices such as a user terminal, a graphical user interface generating device (100), or a graphical user interface providing device (200).

[0703] Meanwhile, according to one embodiment of the present invention, a method for providing a graphical user interface that displays information about a user's sleep may include a step of obtaining the user's sleep information, a step of generating the user's sleep state information based on the obtained user's sleep information, a step of generating a sleep state information graph over time based on the generated sleep state information, and a step of outputting a graphical user interface including the generated sleep state information graph.

[0704] Sleep state information according to one embodiment of the present invention may include at least one of sleep stage information, sleep stage probability information, accumulated sleep stage information, sleep event information, sleep event probability information, and accumulated sleep event information.

[0705] A sleep state information graph according to one embodiment of the present invention may be at least one of a graph representing sleep stage information, a graph representing sleep stage probability information, a graph representing sleep event information, and a graph representing sleep event probability information.

[0706] Meanwhile, a method for providing a graphical user interface that displays information about a user's sleep according to an embodiment of the present invention may further include a sleep log information storage step for storing the user's sleep analysis information for a plurality of sleep sessions. At least one of the accumulated sleep stage graph or the accumulated sleep event graph according to an embodiment of the present invention may be generated based on sleep state information recorded in sleep log information stored for a plurality of sleep sessions. Here, the sleep log information may be stored in at least one of various electronic devices according to embodiments of the present invention. For example, the sleep log information may be sleep state information stored in at least one of a user terminal (10), a graphical user interface generating device (100), a graphical user interface providing device (200), and an external server (20). When a device (a first electronic device) that stores sleep log information according to an embodiment of the present invention and a device (a second electronic device) that generates a graphical user interface are different devices, the second electronic device may receive specific sleep state information from among the sleep state information recorded in the sleep log information stored in the first electronic device for a plurality of sleep sessions, generate a sleep state information graph based on the received sleep state information, and generate a graphical user interface including the generated sleep state information graph.

[0707] FIG. 24a and FIG. 24b are conceptual diagrams illustrating a system in which various aspects of a sleep data interpretation content creation device or a sleep data interpretation content provision device based on user sleep information according to one embodiment of the present invention can be implemented.

[0708] As illustrated in FIG. 24a, a system according to embodiments of the present invention may include a computing device (100-2), a user terminal (300-2), an external server (20-2), and a network. Here, as illustrated in FIG. 24a, a sleep data analysis content creation device (100a-2) based on user sleep information and / or a sleep data analysis content provision device (100b-2) may be implemented as the computing device (100-2).

[0709] As illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information according to an embodiment of the present invention may be implemented as a user terminal (300-2), and a device for providing sleep data interpretation content based on user sleep information may be implemented as an external server (20-2). Alternatively, as illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information may be implemented as an external server (20-2), and a device for providing sleep data interpretation content based on user sleep information may be implemented as a user terminal (300-2). Alternatively, as illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information and a device for providing sleep data interpretation content based on user sleep information may both be implemented as a user terminal (300-2). Alternatively, as illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information and a device for providing sleep data interpretation content based on user sleep information may both be implemented as an external server (20-2).

[0710] Meanwhile, according to one embodiment of the present invention, a device for generating sleep data interpretation content based on user sleep information can generate sleep data interpretation content using generative artificial intelligence. Furthermore, according to one embodiment of the present invention, a device for providing sleep data interpretation content based on user sleep information can provide sleep data interpretation content generated using generative artificial intelligence.

[0711] Alternatively, according to one embodiment of the present invention, a device for generating sleep data interpretation content based on user sleep information may generate sleep data interpretation content based on a lookup table. Furthermore, according to one embodiment of the present invention, a device for providing sleep data interpretation content based on user sleep information may provide sleep data interpretation content generated based on a lookup table.

[0712] Figure 24c is a conceptual diagram illustrating a system in which sleep data interpretation content creation and provision based on user sleep information according to one embodiment of the present invention is implemented in a user terminal (300-2).

[0713] As illustrated in FIG. 24c, sleep data interpretation content may be generated and provided based on user sleep information in a user terminal (300-2) without a separate generation device (100a-2) and / or a separate provision device (100b-2). Meanwhile, according to one embodiment of the present invention, a device for generating sleep data interpretation content based on user sleep information may generate sleep data interpretation content using generative artificial intelligence. Furthermore, according to one embodiment of the present invention, a device for providing sleep data interpretation content based on user sleep information may provide sleep data interpretation content generated using generative artificial intelligence.

[0714] As illustrated in FIG. 24c, even if the sleep data interpretation content creation device (100a-2) and the providing device (100b-2) based on user sleep information according to embodiments of the present invention are not separately provided, the user terminal (300-2) can perform the role of the sleep data interpretation content creation device (100a-2) and / or the providing device (100b-2) based on user sleep information through a network, thereby mutually transmitting and receiving data for the system according to embodiments of the present invention.

[0715] According to one embodiment of the present invention, a device (100a-2) for generating sleep data interpretation content based on user sleep information may include at least one of a display (not shown), a memory (not shown) storing one or more programs configured to be executed by one or more processors, and one or more processors (not shown).

[0716] In addition, according to one embodiment of the present invention, a device (100b-2) that provides sleep data interpretation content based on user sleep information may include at least one of a display (not shown), a memory (not shown) that stores one or more programs configured to be executed by one or more processors, and one or more processors (not shown).

[0717] FIG. 24d is a conceptual diagram illustrating a system of a sleep data interpretation content creation device (100a-2) based on user sleep information according to one embodiment of the present invention. As illustrated in FIG. 24d, the system according to one embodiment of the present invention may include a sleep data interpretation content creation device (100a-2) based on user sleep information, a user terminal (300-2), and a network.

[0718] Figure 24e is a conceptual diagram illustrating a system of a sleep data interpretation content providing device (100b-2) based on user sleep information according to one embodiment of the present invention.

[0719] As illustrated in FIG. 24e, a system according to one embodiment of the present invention may include a sleep data interpretation content providing device (100b-2) based on user sleep information, a user terminal (300-2), and a network.

[0720] As illustrated in FIGS. 24d and 24e, in a system according to embodiments of the present invention, at least one of a sleep information-based sleep data analysis content creation device (100a-2) and a sleep information-based sleep data analysis content provision device (100b-2) can mutually transmit and receive data for the system according to embodiments of the present invention with a user terminal (300-2) through a network.

[0721] FIG. 24F is a conceptual diagram illustrating a system in which various aspects of various electronic devices according to embodiments of the present invention can be implemented. As illustrated in FIG. 24F, the electronic devices illustrated in FIG. 24F can perform at least one of the operations performed by various devices according to embodiments of the present invention.

[0722] For example, operations performed by various electronic devices according to embodiments of the present invention may include operations of acquiring environmental sensing information and sleep information, operations of performing learning for sleep analysis, operations of performing inference for sleep analysis, and operations of acquiring sleep state information. Alternatively, for example, operations may include receiving information related to the user's sleep, transmitting or receiving at least one of environmental sensing information and sleep information, performing preprocessing on the environmental sensing information, determining the environmental sensing information and sleep information, extracting acoustic information from the environmental sensing information and sleep information, processing or manipulating data, processing a service, providing a service, constructing a learning data set based on the environmental sensing information or the user's sleep information, storing acquired data or a plurality of data that become inputs to a neural network, transmitting or receiving various pieces of information, mutually transmitting and receiving data for a system according to embodiments of the present invention through a network, generating or providing sleep data interpretation content based on the user's sleep information, generating or providing imagery induction information, generating or providing sleep images, generating or providing sleep data interpretation content using generative artificial intelligence based on the user's sleep information, and the like.

[0723] The electronic devices illustrated in FIG. 24f may individually perform operations performed by various electronic devices according to embodiments of the present invention, but may also perform one or more operations simultaneously or in time series.

[0724] Referring to FIG. 24f, the electronic devices (1a-2 to 1d-2) illustrated in FIG. 24f may be electronic devices within the range of an area (or, sleep detection area) (11a-2) capable of acquiring environmental sensing information. Hereinafter, for convenience, the area (or, sleep detection area) (11a-2) capable of acquiring environmental sensing information will be referred to as “area (11a-2).”

[0725] Meanwhile, referring to FIG. 24f, the electronic devices (1a-2 and 1d-2) may be devices formed by a combination of two or more electronic devices.

[0726] Meanwhile, referring to FIG. 24F, electronic devices (1a-2 and 1b-2) may be electronic devices connected to a network within the area (11a-2). Furthermore, electronic devices (1c-2 and 1d-2) may be electronic devices not connected to a network within the area (11a-2). Furthermore, electronic devices (2a-2 to 2b-2) may be electronic devices outside the range of the area (11a-2). Furthermore, there may be a network that interacts with electronic devices within the range of the area (11a-2), and there may be a network that interacts with electronic devices outside the range of the area (11a-2). Here, the network that interacts with electronic devices within the range of the area (11a-2) may serve to transmit and receive information for controlling smart home appliances.

[0727] In addition, the network interacting with the electronic devices within the scope of the area (11a-2) may be, for example, a short-range network or a local network. Here, the network interacting with the electronic devices within the scope of the area (11a-2) may be, for example, a long-range network or a global network. In addition, there may be one or more electronic devices connected through a network outside the scope of the area (11a-2), and in this case, the electronic devices may perform distributed processing of data or perform one or more operations separately. Here, the electronic devices connected through a network outside the scope of the area (11a-2) may include a server device. Alternatively, when there is one or more electronic devices connected through a network outside the scope of the area (11a-2), the electronic devices may perform various operations independently of each other.

[0728] Computing Device (100-2)

[0729] FIG. 25a and FIG. 25b are block diagrams illustrating a computing device (100-2) according to one embodiment of the present invention.

[0730] Referring to FIG. 25b, the computing device (100-2) may include at least one of a processor (110-2), a memory (120-2), an output device (130-2), an input device (140-2), an input / output interface (150-2), a sensor module (160-2), a communication module (170-2), and a network unit (180-2).

[0731] According to the present invention, the computing device (100-2) can obtain sleep state information related to whether the user is before, during, or after sleep based on environmental sensing information. Specifically, the environmental sensing information may include sound information obtained non-invasively regarding the user's activities in a space or during sleep. For specific examples, the environmental sensing information may include sounds generated by the user tossing and turning during sleep, sounds related to muscle movements, or sounds related to the user's breathing during sleep. According to an embodiment, the environmental sensing information may include sleep sound information, and the sleep sound information may refer to sound information related to movement patterns and breathing patterns that occur during the user's sleep.

[0732] In one embodiment of the present invention, environmental sensing information can be acquired through a user terminal (300-2) carried by the user. For example, environmental sensing information related to the user's activities in a space can be acquired through a microphone module equipped in the user terminal (300-2).

[0733] According to the present invention, the microphone module equipped in the user terminal (300-2) carried by the user must be equipped in a relatively small-sized user terminal (300-2), and thus may be configured as a MEMS (Micro-Electro-Mechanical Systems). Such a microphone module can be manufactured very small, but may have a lower signal-to-noise ratio (SNR) than a condenser microphone or a dynamic microphone. A low signal-to-noise ratio may mean that the ratio of noise, which is a sound that is not to be identified, to the sound that is to be identified is high, making it difficult to identify the sound (i.e., unclear).

[0734] The environmental sensing information analyzed in the present invention may include sleep sound information, i.e., acoustic information related to the user's breathing and movements acquired during sleep. This sleep sound information relates to very small sounds (i.e., sounds that are difficult to distinguish), such as the user's breathing and movements, and is acquired along with other sounds during sleep. Therefore, if acquired through a microphone module with a low signal-to-noise ratio, such as the aforementioned one, detection and analysis may be very difficult.

[0735] According to one embodiment of the present invention, the computing device (100-2) can acquire sleep state information based on environmental sensing information acquired through a microphone module configured as a MEMS. Specifically, the computing device (100-2) can convert and / or adjust environmental sensing information acquired unclearly, including a lot of noise, into data that can be analyzed, and can perform learning for an artificial neural network using the converted and / or adjusted data. When pre-learning for the artificial neural network is completed, the learned neural network (e.g., an acoustic analysis model) can acquire sleep state information for the user based on data (e.g., a spectrogram) acquired (e.g., converted and / or adjusted) corresponding to sleep sound information.

[0736] According to the present invention, sleep state information may include not only information regarding whether a user is sleeping, but also sleep stage information regarding changes in the user's sleep stage during sleep. For example, the sleep state information may include sleep stage information indicating that the user was in REM sleep at a first time point and in light sleep at a second time point, which is different from the first time point. In this case, the sleep state information may be used to determine that the user was in relatively deep sleep at the first time point and in a lighter sleep at the second time point.

[0737] According to the present invention, the computing device (100-2) may be a terminal or a server, and may include any type of device. The computing device (100-2) may be a digital device equipped with a processor and memory, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, and may be a digital device with computing power. The computing device (100-2) may be a web server that processes services.

[0738] According to the present invention, the computing device (100-2) can perform an operation of generating sleep data interpretation content based on user sleep information and / or an operation of providing sleep data interpretation content based on user sleep information. In this case, a sleep data interpretation content generation and provision system equipped with the computing device (100-2) can be implemented without separately providing a device (100a-2) for generating sleep data interpretation content based on user sleep information and / or a device (100b-2) for providing sleep data interpretation content based on user sleep information.

[0739] Processor (110-2)

[0740] According to the present invention, the processor (110-2) may include one or more application processors (APs), one or more communication processors (CPs), or at least one artificial intelligence processor (AI processor). The application processors, communication processors, or AI processors may be contained within different integrated circuit (IC) packages, or may be contained within a single IC package.

[0741] According to the present invention, the application processor can control a number of hardware or software components connected to the application processor by running an operating system or application program, and perform various data processing / operations, including multimedia data. For example, the application processor may be implemented as a system on chip (SoC). The processor (110-2) may further include a graphics processing unit (GPU, not shown).

[0742] According to the present invention, the communication processor can perform functions such as managing data links and converting communication protocols in communications between the computing device (100-2) and other computing devices connected to a network. For example, the communication processor can be implemented as an SoC. The communication processor can perform at least a portion of the multimedia control functions. In addition, the communication processor can control the data transmission and reception of the communication module (170-2). The communication processor can also be implemented to be included as at least a portion of an application processor.

[0743] According to the present invention, an application processor or a communication processor can load commands or data received from at least one of the non-volatile memory or other components connected thereto into volatile memory and process them. Furthermore, the application processor or communication processor can store data received from at least one of the other components or generated by at least one of the other components in non-volatile memory.

[0744] According to the present invention, when the computer program is loaded into the memory (120-2), it may include one or more instructions that cause the processor (110-2) to perform methods / operations according to various embodiments of the present invention. That is, the processor (110-2) may perform methods / operations according to various embodiments of the present invention by executing one or more instructions.

[0745] In one embodiment, the computer program may include one or more instructions for performing a method for creating a sleep environment based on sleep state information, the method including the steps of obtaining sleep state information of a user, generating environment creation information based on the sleep state information, and transmitting the environment creation information to an environment creation device.

[0746] According to one embodiment of the present invention, the processor (110-2) may be configured with one or more cores and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device.

[0747] According to one embodiment of the present invention, the processor (110-2) can read a computer program stored in the memory (120-2) and perform data processing for machine learning according to one embodiment of the present invention. In addition, the processor (110-2) can perform operations for neural network learning. The processor (110-2) can perform calculations for neural network learning, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation.

[0748] In addition, at least one of the CPU, GPGPU, and TPU of the processor (110-2) can process network function learning. For example, the CPU and GPGPU can jointly process network function learning and data classification using the network function. In addition, in one embodiment of the present invention, processors of multiple computing devices can be used together to process network function learning and data classification using the network function. In addition, a computer program executed in a computing device according to one embodiment of the present invention can be a CPU, GPGPU, or TPU executable program.

[0749] In this specification, the term "network function" may be used interchangeably with "artificial neural network" or "neural network." In this specification, the network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of one or more neural networks.

[0750] In this specification, a model may include a network function. The model may also include one or more network functions, in which case the output of the model may be an ensemble of the outputs of one or more network functions.

[0751] According to the present invention, the processor (110-2) can read a computer program stored in the memory (120-2) to provide a sleep analysis model according to an embodiment of the present invention. According to an embodiment of the present invention, the processor (110-2) can perform calculations to derive environmental composition information based on sleep state information. According to an embodiment of the present invention, the processor (110-2) can perform calculations to train the sleep analysis model. The sleep analysis model will be described in more detail below. Based on the sleep analysis model, sleep information related to the quality of the user's sleep can be inferred. Environmental sensing information acquired from the user in real time or periodically is input as an input value to the sleep analysis model to output data related to the user's sleep.

[0752] The learning of such a sleep analysis model and the inference based thereon can be performed by the computing device (100-2). That is, it can be designed so that both learning and inference are performed by the computing device (100-2). However, in another embodiment, learning may be performed in the computing device (100-2), but inference may be performed in the user terminal (300-2). In addition, learning may be performed in the computing device (100-2), but inference may be performed in the external terminal (200-2). Conversely, learning may be performed in the user terminal (300-2), but inference may be performed in the computing device (100-2). In addition, learning may be performed in the external terminal (200-2), but inference may be performed in the computing device (100-2). Alternatively, both learning and inference may be performed in the external terminal (200-2), or both learning and inference may be performed in the user terminal (300-2).

[0753] According to one embodiment of the present invention, the processor (110-2) can typically process the overall operation of the computing device (100-2). In addition, the processor (110-2) can process signals, data, information, etc. input or output through the components described above, or run application programs stored in the memory (120-2), thereby providing or processing appropriate information or functions to the user terminal.

[0754] According to one embodiment of the present invention, the processor (110-2) can acquire the user's sleep state information. According to one embodiment of the present invention, acquiring the sleep state information may involve acquiring or loading sleep state information stored in the memory (120-2). Furthermore, acquiring sleep sound information may involve receiving or loading data from another storage medium, another computing device, or a separate processing module within the same computing device, based on a wired / wireless communication means.

[0755] FIG. 25a is a block diagram illustrating a computing device (100-2) according to one embodiment of the present invention.

[0756] As illustrated in FIG. 25A, the computing device (100-2) may include a network unit (180-2), a memory (120-2), and a processor (110-2). The components included in the aforementioned computing device (100-2) are not limited. That is, additional components may be included or some of the aforementioned components may be omitted depending on the implementation aspects of the embodiments of the present invention.

[0757] Memory (120-2)

[0758] According to the present invention, the memory (120-2) may include built-in memory or external memory. The built-in memory may include at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.) or non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, NAND flash memory, NOR flash memory, etc.). According to one embodiment, the built-in memory may take the form of a solid state drive (SSD). The external memory may further include a flash drive, for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), or a memory stick.

[0759] According to one embodiment of the present invention, the memory (120-2) can store a computer program for performing a method for creating a sleep environment according to sleep state information according to one embodiment of the present invention, and the stored computer program can be read and operated by the processor (110-2). In addition, the memory (120-2) can store any type of information generated or determined by the processor (110-2) and any type of information received by the network unit (180-2). In addition, the memory (120-2) can store data related to the user's sleep. For example, the memory (120-2) can temporarily or permanently store input / output data (e.g., environmental sensing information related to the user's sleep environment, sleep state information corresponding to the environmental sensing information, or environment creation information according to the sleep state information, etc.).

[0760] Output device (130-2)

[0761] The output device (130-2) may include at least one of a display module and / or a speaker. Specifically, the output device (130-2) may display or output various data, including multimedia data, text data, voice data, etc., to the user as sound.

[0762] Input device (140-2)

[0763] According to the present invention, the input device (140-2) may include a touch panel, a digital pen sensor, a key, or an ultrasonic input device. For example, the input device (140) may be an input / output interface (150-2).

[0764] According to the present invention, the touch panel can recognize touch input in at least one of electrostatic, pressure-sensitive, infrared, or ultrasonic methods. In addition, the touch panel may further include a controller (not shown). In the case of electrostatic, not only direct touch but also proximity recognition is possible. The touch panel may further include a tactile layer. In this case, the touch panel can provide a tactile response to the user. The digital pen sensor can be implemented using the same or similar method as that for receiving the user's touch input, or using a separate recognition layer. The key can be a keypad or a touch key. The ultrasonic input device is a device that can detect micro-sound waves in a terminal and confirm data through a pen that generates an ultrasonic signal, and is capable of wireless recognition. The computing device (100-2) can also receive user input from an external device (e.g., a network, a computer, or a server) connected thereto using the communication module (170-2).

[0765] According to the present invention, the input device (140-2) may further include a camera module and / or a microphone. The camera module is a device capable of capturing images and videos, and may include one or more image sensors, an image signal processor (ISP), or a flash LED. The microphone may receive a voice signal and convert it into an electrical signal.

[0766] Input / Output Interface (150-2)

[0767] According to the present invention, the input / output interface (150-2) can transmit a command or data input from a user through an input device (140-2) or an output device (130-2) to a processor (110-2), a memory (120-2), a communication module (170-2), etc., via a bus (not shown). For example, the input / output interface (150-2) can provide data regarding a user's touch input input through a touch panel to the processor (110-2). For example, the input / output interface (150-2) can output a command or data received from the processor (110-2), the memory (120-2), the communication module (170-2), etc., via a bus through the output device (130-2). For example, the input / output interface (150-2) can output voice data processed by the processor (110-2) to the user through a speaker.

[0768] Sensor module (160-2)

[0769] According to the present invention, the sensor module (160-2) may include at least one of a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, an RGB (red, green, blue) sensor, a biosensor, a temperature / humidity sensor, an illuminance sensor, or an UV (ultra violet) sensor. The sensor module (160) may measure a physical quantity or detect the operating status of the computing device (100-2) and convert the measured or detected information into an electrical signal. In addition, the sensor module (160-2) may include an olfactory sensor (E-nose sensor), an electromyography sensor (EMG sensor), an electroencephalogram sensor (EEG sensor, not shown), an electrocardiogram sensor (ECG sensor), a photoplethysmography sensor (PPG sensor), a heart rate monitor sensor (HRM sensor), a perspiration sensor, or a fingerprint sensor. The sensor module (160) may further include a control circuit for controlling at least one sensor included therein.

[0770] Communication module (170-2)

[0771] According to the present invention, the communication module (170-2) may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS, or NFC. For example, the wireless communication module may provide a wireless communication function using radio frequencies. Additionally or alternatively, the wireless communication module may include a network interface or modem, etc., for connecting the computing device (100-2) to a network (e.g., the Internet, LAN, WAN, telecommunication network, cellular network, satellite network, POTS, or 5G network, etc.).

[0772] According to the present invention, the RF module can be responsible for transmitting and receiving data, for example, transmitting and receiving RF signals or called electronic signals. For example, the RF module can include a transceiver, a power amp module (PAM), a frequency filter, or a low noise amplifier (LNA). In addition, the RF module can further include components for transmitting and receiving electromagnetic waves in free space in wireless communication, for example, a conductor or a wire.

[0773] According to the present invention, the computing device (100-2) may include at least one of a server, a TV, a smart TV, a refrigerator, an oven, a clothing styler, a robot vacuum cleaner, a drone, an air conditioner, an air purifier, a PC, a speaker, a home CCTV, a light, a washing machine, and a smart plug. Since the components of the computing device (100-2) described in FIG. 25d are examples of components generally provided in a computing device, the computing device (100-2) is not limited to the aforementioned components, and certain components may be omitted and / or added as needed.

[0774] Network Department (180-2)

[0775] The network unit (180-2) according to one embodiment of the present invention can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0776] In addition, the network unit (180-2) presented in this specification can use various wireless communication systems that can be realized now and in the future, such as mobile communication systems such as 4G and 5G (LTE), and satellite communication systems such as Starlink.

[0777] External terminal (200-2)

[0778] Figure 25c is a block diagram for explaining an external terminal (200-2) according to one embodiment of the present invention.

[0779] According to the present invention, an external terminal (200-2) may include a processor (210-2), a memory (220-2), and a communication module (270-2). Specifically, the external terminal (200-2) may be an external server (20-2) or a cloud server. In addition, the external server (20-2) may be a digital device equipped with a processor, memory, and computing power, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone. The external server (20-2) may be a web server that processes services. The types of servers described above are merely examples, and the present invention is not limited thereto.

[0780] According to one embodiment of the present invention, the external terminal (200-2) may be a server providing a cloud computing service. More specifically, the external terminal (200-2) may be a server providing a cloud computing service, a type of Internet-based computing in which information is processed by another computer connected to the Internet rather than by the user's computer. The cloud computing service may be a service that stores data on the Internet and allows the user to access it anytime and anywhere via an Internet connection without having to install the necessary data or programs on their own computer. Data stored on the Internet can be easily shared and transmitted through simple operations and clicks.

[0781] According to one embodiment of the present invention, the external server (20-2) may be a server that stores information on a plurality of training data for training a neural network. The plu...

Claims

1. A method for providing interpretation content of a user's sleep data, Step of obtaining user's sleep information; A step of generating user's sleep state information based on the user's acquired sleep information; A step for generating user's sleep data interpretation content based on the generated sleep state information; and A step of outputting the generated user's sleep data interpretation content; Including, The above user's sleep data interpretation content is expressed in at least one of a numerical or non-numerical manner, A method for providing interpretation content of a user's sleep data.

2. In paragraph 1, The above user's sleep data interpretation content is generated based on comparing the generated user's sleep state information with at least one of the user's sleep data generated over a predetermined period of time in the past, medical standards, and other user's sleep data generated over a predetermined period of time in the past. A method for providing interpretation content of a user's sleep data.

3. In paragraph 2, The above user's sleep data interpretation content is generated based on at least one of a lookup table or a large-scale language model. A method for providing interpretation content of a user's sleep data.

4. In any one of paragraphs 1 to 3, The above sleep status information is generated based on combining the acquired user's sleep information and other information into multimodal data. A method for providing interpretation content of a user's sleep data.

5. In any one of paragraphs 1 to 3, The above sleep status information is generated based on a portion of the user's acquired sleep information cut off by a predetermined length on the time axis. A method for providing interpretation content of a user's sleep data.

6. In a system for initiating sleep measurement through a user terminal, A permission management tool that collects the permissions required to perform sleep measurements; and A sleep data collection tool that activates the sensor module of the user terminal based on the collected authority; The above sleep data collection tool is, configured to detect a first event via the above activated sensor module, The broadcast trigger of the sleep measurement of the user terminal is configured to occur based on the detected first event, When the above broadcast trigger occurs, the sensor module is further activated for sleep measurement, configured to detect a second event via an additional activated sensor module for the above sleep measurement, The sleep analysis trigger of the sleep measurement of the user terminal is configured to occur based on the detected second event, The second event detected above is configured to be detected by the sensor module by a stimulus above a threshold, A system for initiating sleep measurement through a user terminal.

7. In paragraph 6, The above permission management tool is Configured to request at least one of the Overlay permission permission, the Foreground service permission, the Activation permission of the sensor module and the Time-based alarm permission. A system for initiating sleep measurement through a user terminal.

8. In paragraph 6, The first event detected above is, At least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in the accelerometer sensor of the sensor module, switching the display unit of the user terminal to the OFF state, a change in the battery status of the user terminal, connection of the user terminal with another device, and unlocking of the user terminal. A system for initiating sleep measurement through a user terminal.

9. In paragraph 6, The second event, which is configured to be detected by a stimulus above a threshold by the sensor module, At least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of change in an accelerometer sensor of the sensor module, change in a battery status of the user terminal, connection of the user terminal with another device, unlocking of the user terminal, palm swiping for the user terminal, finger swiping for the user terminal, finger tapping for the user terminal, voice input for the user terminal, vibration detection of the user terminal, screen wheel scrolling for the user terminal, pressing a specific user interface of the user terminal, pressing a display unit of the user terminal for a predetermined period of time or longer, moving two or more fingers simultaneously for the display unit, flipping the user terminal, activating a proximity sensor of the user terminal, detection of a temperature change of the user terminal, and detection of a change in illumination around the user terminal. A system for initiating sleep measurement through a user terminal.

10. In paragraph 6, The above sleep analysis trigger is Including information on the start time of sleep analysis of the measured sleep, A system for initiating sleep measurement through a user terminal.

11. In paragraph 6, The above sleep data collection tool is, Configured to transmit environmental sensing information collected by the sensor module from the time of occurrence of the above broadcast trigger to another device for sleep analysis. A system for initiating sleep measurement through a user terminal.

12. In paragraph 6, The above sleep data collection tool is, When the above broadcast trigger occurs, Configured to provide a sleep measurement confirmation user interface to the above user terminal, A system for initiating sleep measurement through a user terminal.

13. In paragraph 6, The above sleep data collection tool is, Configured to force broadcast trigger generation even in low power mode of the above user terminal, A system for initiating sleep measurement through a user terminal.

14. A method for providing content related to the user's sleep status information, A step of obtaining environmental sensing information through a sensor module of a user terminal; A step of converting sleep sound information included in the acquired environmental sensing information into a spectrogram; A step of processing the above spectrogram as input to a sleep analysis model to generate information on the user's sleep state; and A step of providing content related to the sleep state information through a content provision model based on the above sleep state information; Including, The above sleep analysis model is an artificial intelligence model including one or more artificial neural networks, and includes a feature extraction model and a feature classification model. The above content provision model is a conversational artificial intelligence model based on natural language processing that includes one or more artificial neural networks. A method for providing content related to a user's sleep state information.

15. In paragraph 14, The above feature extraction model is a model configured to analyze the frequency pattern of the user's breathing included in the above sleep sound information, The above feature classification model is a model configured to analyze the periodic pattern of the user's breathing included in the above sleep sound information. A method for providing content related to a user's sleep state information.

16. In paragraph 14, The above content delivery model includes one or more prompts, The one or more prompts include a prompt for specifying a role of the content delivery model, a prompt for specifying a purpose of the content delivery model, and a prompt for presenting an output format through the content delivery model. A method for providing content related to a user's sleep state information.

17. In paragraph 14, The above content delivery model includes one or more tools, One or more of the above tools include tools for accessing external databases, The above external database is a static database or a dynamic database. A method for providing content related to a user's sleep state information.

18. In any one of paragraphs 14 to 17, At least two of the above environmental sensing information, the above generated sleep state information, and the conversation information with the user through the content provision model are mapped to each other and configured as metadata. The above metadata is stored in at least one of the database of the sleep analysis server where the sleep analysis model is implemented and the database of the content provision server where the content provision model is implemented. A method for providing content related to a user's sleep state information.

19. In any one of paragraphs 14 to 17, The above content provision model is: Even if there is no dialogue input from the user terminal, it is configured to proactively provide content related to the sleep state information through the conversational interface. A method for providing content related to a user's sleep state information.

20. For a user terminal to provide content related to the user's sleep status information, sensor module; Input section; processor; Output section - said output section being configured to output an interactive interface; and Communication module; Including, The above interactive interface, Includes content related to the sleep state information received from the content providing server through the communication module, The above content is generated by the content providing server through a content providing model based on the sleep status information received from the sleep analysis server. The above content provision model is a conversational artificial intelligence model based on natural language processing that includes one or more artificial neural networks. The above sleep state information is obtained by the sleep analysis server processing the transmitted sleep sound information as input to the sleep analysis model when the sleep sound information included in the environmental sensing information acquired by the sensor module is transmitted to the sleep analysis server through the communication module. The above sleep analysis model is an artificial intelligence model including one or more artificial neural networks, including a feature extraction model and a feature classification model. A user terminal for providing content related to the user's sleep status information.

Images