Method for creating an environment for an object, smart lighting, and electronic device for controlling the smart lighting

Abstract

The present invention discloses an AI-based non-contact sleep analysis system and method. The sleep analysis system includes a smartphone that downloads a sleep analysis app from a server, collects a user's sleep sound information in real time, transmits it to the server, and receives a sleep analysis result report learned by artificial intelligence from the server, and at least one smart home appliance that is located remotely around the user, simultaneously collects the sleep sound information, transmits it to the smartphone, and provides a customized sleep environment to the user in response to the control of the smartphone. According to the present invention, the wake time, which is the basis of all sleep treatments, can be accurately set, and various types of user's sleep can be conveniently and accurately analyzed at home regardless of time and place. Also, electronic devices or home appliances that can create an environment based on the analyzed sleep state information can be controlled.

Description

[Technical Field]

[0001] The present invention relates to an AI-based non-contact sleep analysis method for performing sleep analysis and a method for creating a real-time sleep environment. [Background technology]

[0002] True healthcare requires 24 / 7 monitoring and management, because health monitoring and management is not a simple one-to-one match, but rather a complex interrelationship of all factors.

[0003] There are various ways to maintain and improve health, such as exercise and diet, but the most important thing is to manage sleep well, which takes up more than 30% of the day.

[0004] However, despite the fact that modern people have been replaced by machines with simpler labor and have more comfortable lives, they are unable to get a good night's sleep due to irregular eating habits, lifestyles and stress, and suffer from sleep disorders such as insomnia, excessive sleep, sleep apnea syndrome, nightmares, night terrors and sleepwalking.

[0005] According to the National Health Insurance Service, the number of people with sleep disorders in Korea increased by an average of 8% per year from 2014 to 2018, and the number of people who received treatment for sleep disorders in Korea in 2018 reached approximately 570,000.

[0006] Deep sleep is recognized as an important factor affecting physical and mental health, and interest in deep sleep is increasing. However, in order to improve sleep disorders, patients must visit specialized medical institutions in person, which requires additional testing fees, and continuous management is difficult, so users are not making enough effort to seek treatment.

[0007] As sleep problems become more serious, the need for sleep health management is increasing, and the sleep tech market, which aims to solve sleep problems with technology, is also growing rapidly.

[0008] Korean Patent Publication No. 2003-0032529 discloses a sleep induction device and sleep induction method that receives input of a user's physical information and outputs vibrations and / or ultrasound waves in a frequency band detected through repetitive learning according to the user's physical condition during sleep, thereby enabling optimal sleep induction.

[0009] However, conventional technologies can reduce sleep quality due to the inconvenience caused by body-worn devices and require periodic maintenance of the devices (e.g., charging).

[0010] In addition, conventional sleep analysis methods using wearable devices have the problem that sleep analysis is impossible if the wearable device is not in proper contact with the user's body or if the user is not wearing the wearable device.

[0011] Furthermore, when multiple users sleep in the same space, the movements of those not wearing the wearable device can interfere with sleep analysis of those wearing the wearable device, and it is impossible to perform sleep analysis on those not wearing the wearable device.

[0012] In addition, conventional sleep analysis methods using wearable devices or non-contact sleep management research use the Heart Rate Variability (HRV) variance or EEG change values ​​between the asleep and awake states, but the difference between these is not large, meaning that they have the limitation of not being able to accurately time the awake state, which is the basis of all sleep treatments.

[0013] In particular, when using changes in brain waves to treat sleep disorders such as snoring, it is not possible to detect precursor symptoms of snoring from changes in brain waves at all, so it cannot be used to prevent snoring.In addition, since it detects changes in brain waves that occur after the patient snores, it has the limitation of only being used to diagnose snoring.

[0014] As a result, research has been progressing recently into a non-contact method of estimating sleep stages by monitoring breathing patterns and the degree of activation of the autonomic nervous system based on body movements during the night, and creating a sleep environment for the user based on the estimated sleep state.

[0015] In particular, many papers that have studied the relationship between sleep and the sleep environment, such as air quality, temperature, and humidity, have confirmed that the sleep environment, such as air quality, temperature, and humidity, has a decisive impact on sleep quality, which means that the sleep environment needs to be optimized in order to improve sleep quality. Summary of the Invention [Problem to be solved by the invention]

[0016] An object of the present invention is to provide a sleep analysis system and method that can conveniently and accurately analyze the sleep of various types of users in real time, regardless of time or place, without the need to purchase or wear a separate wearable device.

[0017] Another object of the present invention is to provide a sleep analysis system and method that can in-depth analyze a user's sleep through artificial intelligence learning instead of conventional various biological signals using only the user's breathing by simultaneously using a smart home appliance with a built-in microphone and a smartphone.

[0018] The present invention also provides various home appliances for providing an optimal sleeping environment related to various factors such as air quality, temperature and / or humidity of the sleeping environment based on sleep state information sensed in the user's sleeping environment.

[0019] The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0020] Other details of the invention are included in the detailed description and drawings.

[0021] To achieve the above object, the AI-based non-contact sleep analysis system according to the present invention includes a smartphone that downloads a sleep analysis app from a server, collects a user's sleep acoustic information in real time, transmits the information to the server, and receives a sleep analysis result report learned by AI from the server; and at least one smart home appliance that is located remotely around the user, simultaneously collects the sleep acoustic information, transmits the information to the smartphone, and provides a customized sleep environment for the user in response to control of the smartphone.

[0022] To achieve the above object, the smart home appliance of the AI-based non-contact sleep analysis system according to the present invention includes a sensor unit that collects a user's sleep acoustic information through a built-in microphone module, a memory that stores a program for performing sleep analysis, a processor that reads the program stored in the memory to extract a sleep analysis model and performs a sleep analysis of the user based on the sleep acoustic information using the sleep analysis model, and an alarm unit that delivers a tactile or auditory stimulus to the user if a sleep disorder occurs during the sleep analysis.

[0023] To achieve the above object, the sleep analysis result report of the AI-based non-contact sleep analysis system according to the present invention is characterized by including the time of going to bed, the delay in falling asleep, the duration of sleep, and the time it takes to wake up after an alarm.

[0024] To achieve the above object, the processor of the AI-based non-contact sleep analysis system according to the present invention is characterized by monitoring the user's sleep apnea in real time based on the sleep analysis.

[0025] To achieve the above object, the AI-based non-contact sleep analysis system according to the present invention further includes a communication unit that transmits and receives data to and from the smartphone and the server via a wireless communication network.

[0026] To achieve the above object, the processor of the AI-based non-contact sleep analysis system according to the present invention converts raw data of a user's sleep acoustic information into a spectrogram, and then inputs the spectrogram into a sleep analysis model modeled through deep learning to perform secondary sleep analysis.

[0027] To achieve the above object, the AI-based non-contact sleep analysis method according to the present invention includes the steps of: a smartphone downloading a sleep analysis app from a server; at least one smart home appliance collecting a user's sleep acoustic information in real time and transmitting the information to the server; the smartphone simultaneously collecting the user's sleep acoustic information in real time and transmitting the information to the server; the server transmitting an AI-learned sleep analysis result report to the smartphone; the smartphone outputting a control signal for controlling the operation of the at least one smart home appliance; and the at least one smart home appliance providing a customized sleep environment to the user.

[0028] In order to achieve the above object, the server of the AI-based non-contact sleep analysis method according to the present invention is an artificial intelligence server.

[0029] To achieve the above object, the AI-based non-contact sleep analysis method according to the present invention includes: (a) a step (S7000) of determining whether at least one smart home device has a built-in microphone; (b) a step (S7100) of downloading a sleep analysis app from a server by the smartphone if the determination result of step (a) is positive; (c) a step (S8000) of determining whether the smart home device can create a sleep environment if the sleep analysis app has been downloaded; (d) a step (S9000) of determining whether the smart home device is capable of providing data based on sleep analysis if the determination result of step (c) is negative; and (e) a step (S9100) of activating the sleep analysis app if the determination result of step (d) is positive.

[0030] To achieve the above object, step (b) of the AI-based non-contact sleep analysis method according to the present invention is characterized by further including a step (S7200) of linking the sleep analysis app to an app already installed on the smartphone if the determination result of step (a) is negative.

[0031] To achieve the above object, step (d) of the AI-based non-contact sleep analysis method according to the present invention is characterized by further including a step (S8100) in which, if the judgment result of step (c) is positive, the SleepTrack app is activated and a study interaction is generated.

[0032] To achieve the above object, step (d) of the AI-based non-contact sleep analysis method according to the present invention is characterized in that it further includes a step (S9000) of determining whether the smart home appliance is a device that can provide data based on sleep analysis through a user interface if the determination result of step (c) is negative.

[0033] In step (c) of the AI-based non-contact sleep analysis method according to the present invention to achieve the above object, the sleep environment includes one or more of temperature, humidity, light, sound, head and body position, and scent.

[0034] To achieve the above object, the smart home appliances that reach the step (S8100) of the AI-based non-contact sleep analysis method according to the present invention are characterized by including at least one of an air conditioner, air purifier, humidifier, dehumidifier, blinds, curtains, lamp, smart speaker, smart bed, smart diffuser, and smart device with a healthcare app installed.

[0035] To achieve the above object, the smart home appliances that reach step S9100 of the AI-based non-contact sleep analysis method according to the present invention are characterized by including at least one of a TV, a clothes management machine, a robot vacuum cleaner, a washing machine, a dryer, a refrigerator, and a smart appliance with a healthcare app installed.

[0036] To achieve the above object, the air purifier according to the present invention includes a network unit that receives environmental sensing information from a user terminal, a processor that acquires sleep state information based on the environmental sensing information and generates environment creation information using the sleep state information, and a driving unit that adjusts the air quality in a sleeping space based on the environment creation information.

[0037] The sleep environment control system may further include a measuring unit for measuring air components in the sleep space, and the processor may generate the environment creation information based on the measured air components and the environment sensing information.

[0038] To achieve the above object, the air conditioner according to the present invention includes a network unit that receives environmental sensing information from a user terminal, a processor that acquires sleep state information based on the environmental sensing information and generates environment creation information using the sleep state information, and a driving unit that adjusts the temperature and / or humidity in the sleeping space based on the environment creation information.

[0039] The sleep space may further include a measurement unit for measuring temperature and / or humidity in the sleep space, and the processor may generate the environment creation information based on the measured temperature and / or humidity and the environmental sensing information. [Effects of the Invention]

[0040] According to one embodiment of the present invention, it is possible to predict the time when a user is awake and / or sleep state information, and to conveniently and accurately analyze the sleep of various users at home regardless of time and place.

[0041] Furthermore, the user does not need to wear a wearable device during sleep analysis, which allows the user to have more freedom of movement during sleep.

[0042] In addition, by collecting the results of multi-dimensional sleep tests worldwide, sleep sound data can be created, and acoustic AI can be used to establish a new standard for home environment sleep tracking, validating diverse races, ages, genders, and even measurement environments.

[0043] In addition, it can learn various ambient noises, including routine noises and abnormal or intermittent noises, in the space surrounding the user's sleep environment to build an AI sleep stage analysis model.

[0044] In addition, sound AI and wireless communication sensing clinical datasets can be constructed by utilizing smartphone sound data and smart speaker sound data collected over a long period of time, along with multidimensional sleep tests of many clinicians.

[0045] In addition, it is possible to perform in-depth analysis of the user's sleep using smart home appliances and smartphones, and it is possible to perform sleep analysis of not only one person but also many people.

[0046] In addition, if a user experiences a sleep disorder, the sleep disorder can be appropriately alleviated, and if multiple people are sleeping in the same space, an alarm for alleviating the sleep disorder can be transmitted only to the user who has experienced the sleep disorder, thereby preventing the sleep of others from being disturbed.

[0047] Additionally, smart home appliances and / or smartphones can be used to monitor a user's physical activity status in real time 24 hours a day.

[0048] In addition, according to one embodiment of the present invention, an optimized sleeping environment for improving the quality of a user's sleep can be provided through sleep state information sensed in relation to the user's sleeping environment.

[0049] In particular, the quality of sleep can be significantly improved by creating an optimal sleeping environment that takes into account various factors such as air quality, temperature and / or humidity of the sleeping environment.

[0050] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0051] [Figure 1(a)] FIG. 1(a) is a conceptual diagram illustrating a system in which various aspects of a computing device for creating a sleep environment based on sleep state information according to an embodiment of the present invention can be implemented. [Figure 1(b)] FIG. 1(b) is a conceptual diagram showing a system in which various aspects of a sleep environment adjusting device related to yet another embodiment of the present invention can be implemented. [Figure 1(c)] FIG. 1(c) is a conceptual diagram showing a system in which various aspects of various electronic devices related to yet another embodiment of the present invention can be implemented. [Figure 2]FIG. 2 illustrates a block diagram of a computing device for creating a sleep environment based on sleep state information in accordance with an embodiment of the present invention. [Figure 3] 4 is a diagram comparing the results of polysomnography (PSG) in FIG. 3 with the analysis results (AI results) using the AI ​​algorithm according to the present invention. [Figure 4] FIG. 4 is a graph comparing the results of polysomnography (PSG) in relation to sleep apnea and hypopnea with the analysis results using the AI ​​algorithm according to the present invention. [Figure 5] FIG. 5 is an exemplary diagram illustrating a process of acquiring sleep sound information from environmental sensing information according to an embodiment of the present invention. [Figure 6(a)] FIG. 6(a) is an exemplary diagram illustrating a method for acquiring a spectrogram corresponding to sleep acoustic information according to an embodiment of the present invention. [Figure 6(b)] FIG. 6(b) is a conceptual diagram illustrating a privacy protection method using mel-spectrogram conversion for sleep acoustic information extracted from a user in the sleep analysis method according to the present invention. [Figure 7] FIG. 7 is an exemplary view illustrating environment creation information for each time point according to a user's sleep state, according to an embodiment of the present invention. [Figure 8] FIG. 8 illustrates an exemplary flow chart for providing a method for creating a sleep environment based on sleep state information according to an embodiment of the present invention. [Figure 9] FIG. 9 is a schematic diagram illustrating one or more network functions associated with an embodiment of the present invention. [Figure 10] FIG. 10 shows an exemplary block diagram of a sleep environment adjusting device in accordance with an embodiment of the present invention. [Figure 11(a)] FIG. 11(a) shows an exemplary block diagram of a receiving module and a transmitting module associated with one embodiment of the present invention. [Figure 11(b)] FIG. 11(b) is a block diagram showing the configuration of smart home appliances in the AI-based non-contact sleep analysis system according to the present invention. [Figure 12] FIG. 12 is an exemplary view illustrating a second sensor unit that detects whether a user is located in a pre-set area according to an embodiment of the present invention. [Figure 13] FIG. 13 is a flowchart illustrating an example of a process for generating sleep state information through an automatic sleep measurement mode according to an embodiment of the present invention. [Figure 14] FIG. 14 is a flowchart illustrating an example of a process for creating an environment that induces a user to fall asleep, according to an embodiment of the present invention. [Figure 15] FIG. 15 is a flowchart illustrating an example of a process for changing a user's sleep environment during sleep and immediately before waking up, according to an embodiment of the present invention. [Figure 16(a)] FIG. 16(a) is a conceptual diagram for explaining the operation of the air conditioner according to one embodiment of the present invention. [Figure 16(b)] FIG. 16(b) is a conceptual diagram for explaining the operation of the air conditioner according to one embodiment of the present invention. [Figure 16(c)] FIG. 16(c) is a conceptual diagram for explaining the operation of the air purifier according to one embodiment of the present invention. [Figure 16(d)] FIG. 16(d) is a conceptual diagram for explaining the operation of the air purifier according to one embodiment of the present invention. [Figure 17(a)] FIG. 17(a) is a block diagram showing the configuration of an air conditioner according to one embodiment of the present invention. [Figure 17(b)] FIG. 17(b) is a block diagram showing the configuration of an air purifier according to one embodiment of the present invention. [Figure 18] FIG. 18 is a diagram for explaining an example of the air conditioner shown in FIGS. 16 and 17. In FIG. [Figure 19] FIG. 19 is a diagram for explaining another example of the air conditioner shown in FIGS. 16 and 17. In FIG. [Figure 20]FIG. 20 is a diagram for explaining yet another example of the air conditioner shown in FIGS. [Figure 21] FIG. 21 is a diagram for explaining yet another example of the air conditioner shown in FIGS. [Figure 22] FIG. 22 is a diagram for explaining yet another example of the air conditioner shown in FIGS. [Figure 23(a)] FIG. 23(a) is a diagram for explaining a method of operating the indoor unit 500'' in a sleep mode via the display unit 570'' of the indoor unit 500'' shown in FIGS. 20 and 21. [Figure 23(b)] FIG. 23(b) is a diagram for explaining a method of operating the indoor unit 500'' in a sleep mode via the display unit 570'' of the indoor unit 500'' shown in FIGS. 20 and 21. [Figure 23(c)] FIG. 23(c) is a diagram of the display unit 4000 for explaining the sleep mode of the air purifier 700' according to an embodiment of the present invention. [Figure 23(d)] FIG. 23(d) is a diagram of the display unit 4000 for explaining the sleep mode of the air purifier 700' according to an embodiment of the present invention. [Figure 24(a)] Figure 24(a) is a diagram illustrating a method of operating the indoor units 500', 500'', 500'''', and 500'''' shown in Figures 18 to 22 in sleep mode via a remote control 600 according to one embodiment of the present invention. [Figure 24(b)] FIG. 24(b) is a diagram showing an example of a display unit 4000 of an air purifier 700' according to an embodiment of the present invention. [Figure 25(a)] Figure 25(a) is a diagram illustrating a method of operating the indoor units 500', 500'', 500'''', and 500'''' shown in Figures 18 to 22 in sleep mode via a user terminal 10 according to one embodiment of the present invention. [Figure 25(b)]Figure 25(b) is a diagram illustrating a method of operating the indoor units 500', 500'', 500'''', and 500'''' shown in Figures 18 to 22 in sleep mode via a user terminal 10 according to one embodiment of the present invention. [Figure 25(c)] FIG. 25(c) is a diagram showing a screen of a first application for remotely controlling an air purifier 700'' in a user terminal 10 according to an embodiment of the present invention. [Figure 25(d)] FIG. 25(d) is a diagram showing a screen of an application for controlling the sleep mode of the air purifier 700'' according to an embodiment of the present invention. [Figure 26] FIG. 26 is a diagram for explaining a method for automatically operating the indoor unit or air purifier in sleep mode. [Figure 27] FIG. 27 is a diagram illustrating the timing of the sleep mode operation shown in FIG. [Figure 28] FIG. 28 is a diagram illustrating the timing of the sleep mode operation shown in FIG. [Figure 29] (a) and (b) of FIG. 29 are diagrams for explaining an example of the air purifier shown in FIG. 16 and FIG. [Figure 30] FIG. 30 is a view showing the air purifier 700' shown in FIG. 29 with some parts of the covers 1100 and 2100 removed. [Figure 31(a)] FIG. 31(a) is a diagram for explaining another example of the air purifier shown in FIGS. [Figure 31(b)] FIG. 31(b) is a diagram illustrating the time point at which the air purifier 700''' shown in FIG. 26 operates in sleep mode. [Figure 32(a)] FIG. 32(a) is a diagram for explaining sleep stage analysis using a spectrogram in the sleep analysis method according to the present invention. [Figure 32(b)] FIG. 32(b) is a diagram for explaining the determination of a sleep disorder using a spectrogram in the sleep analysis method according to the present invention. [Figure 33(a)]FIG. 33(a) is a diagram showing an experimental process for verifying the performance of the sleep analysis method according to the present invention. [Figure 33(b)] Figure 33(b) is a graph verifying the performance of the sleep analysis method according to the present invention, which compares the results of a sleep polymorphism test (PSG result) with the analysis results (AI result) using the AI ​​algorithm according to the present invention. [Figure 34] FIG. 34 is a table verifying the accuracy of the sleep analysis method according to the present invention, and shows experimental result data analyzed according to age, sex, BMI, and presence or absence of disease. [Figure 35] FIG. 35 is a conceptual diagram illustrating an embodiment of the sleep analysis method according to the present invention, in which a smart speaker and a smartphone are used for easy understanding. [Figure 36(a)] FIG. 36(a) is a flowchart illustrating a method for preventing and alleviating sleep disorders using an AI-based non-contact sleep analysis system according to an embodiment of the present invention. [Figure 36(b)] FIG. 36(b) is a flowchart illustrating a method for preventing and alleviating sleep disorders using an AI-based non-contact sleep analysis system according to another embodiment of the present invention. [Figure 37] FIG. 37 is a diagram illustrating a traffic handling method when the sleep analysis method according to the present invention is performed in the cloud. [Figure 38] FIG. 38 is a conceptual diagram for explaining the sleep analysis of one person and the sleep analysis of many people in the sleep analysis method according to the present invention. [Figure 39] FIG. 39 is a flowchart illustrating the operation of the AI-based non-contact sleep analysis method according to the present invention. [Figure 40] FIG. 40 is a flow chart showing various embodiments of smart home appliances that can be used in the sleep analysis method according to the present invention. [Figure 41] FIG. 41 is a table illustrating an example of an operation of a sleep preparation stage among specific scenarios of a plurality of smart home appliances that operate in time series according to a user's sleep stages using the sleep analysis method according to the present invention. [Figure 42]FIG. 42 is a table showing an example of the operations in the stages from after falling asleep to before deep sleep, which are linked chronologically to FIG. 41 in the scenario. [Figure 43] FIG. 43 is a table showing an example of the operation of the stage from after deep sleep to before wake-up detection, which is linked chronologically to FIG. 42 in the scenario. [Figure 44] FIG. 44 is a table showing an example of the wake-up stage operation, which is linked to FIG. 43 in chronological order among the scenarios. [Figure 45] FIG. 45 is a conceptual diagram showing a training method in which only microphone data S from a multi-dimensional sleep test is used in a hospital environment according to a conventional sleep analysis method, in order to compare the sleep analysis method of the present invention with the conventional technology. [Figure 46] FIG. 46 is a conceptual diagram of a method for generating an AI sleep analysis model by incorporating various sounds in a home environment in accordance with the sleep analysis method of the present invention into the training method shown in FIG. [Figure 47] FIG. 47 is a table verifying the performance of the sleep analysis method according to the present invention after training by dividing the subjects into nine groups according to the type of residential noise. [Figure 48] FIG. 48 is a schematic diagram illustrating a 24-hour monitoring process of a user using an AI-based non-contact sleep analysis system and sleep analysis method according to the present invention. [Figure 49] FIG. 49 is a table showing the mean per class results of comparing the smart home appliances and sleep analysis method according to the present invention with products and devices from existing world-leading sleep tech companies. [Figure 50] FIG. 50 is a block diagram illustrating the operation of an AI-based non-contact sleep analysis system according to an embodiment of the present invention. [Figure 51] FIG. 51 is a block diagram illustrating the operation between components of an AI-based non-contact sleep analysis system according to an embodiment of the present invention. [Figure 52]FIG. 52 is a table showing exemplary operations in the sleep mode and wake-up mode, whether or not the environment creating device is activated based on the sleep state information, and the detailed product type. DETAILED DESCRIPTION OF THE INVENTION

[0052] The advantages and features of the present invention, and methods for achieving them, will become apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various different forms. The present embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully convey the scope of the present invention to those skilled in the art. The present invention is defined only by the scope of the claims.

[0053] When describing the embodiments disclosed herein, if it is determined that a specific description of the related publicly disclosed technology may obscure the gist of the embodiments disclosed herein, the detailed description will be omitted. Furthermore, the attached drawings are merely provided to facilitate understanding of the embodiments disclosed herein, and the technical ideas disclosed herein should not be limited by the attached drawings, and should be understood to include all modifications, equivalents, or alternatives within the spirit and technical scope of the present invention.

[0054] The terms used in this specification are intended to describe the embodiments and are not intended to limit the present invention.

[0055] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in the sense commonly understood by those of ordinary skill in the art to which the present invention pertains. Furthermore, commonly used and predefined terms are not to be interpreted ideally or excessively unless expressly defined otherwise.

[0056] In this specification, the singular includes the plural unless otherwise specified. The terms "comprises" and / or "comprising" used in this specification do not exclude the presence or addition of one or more other elements other than the elements being mentioned. The same reference numerals refer to the same elements throughout this specification, and "and / or" includes each and every combination of one or more of the elements being mentioned. Although terms such as "first," "second," etc. are used to describe various elements, these elements are not limited by these terms. These terms are used merely to distinguish one element from another. Therefore, a first element referred to below may of course be a second element within the technical spirit of the present invention.

[0057] The terms "module" and "module" used herein refer to software or hardware components, such as FPGAs or ASICs, that perform a certain function. However, "module" and "module" are not limited to software or hardware. A "module" or "module" may be configured to reside on an addressable storage medium or to execute on one or more processors. Thus, by way of example, a "module" or "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and functionality provided within a "module" or "module" may be combined into fewer components and "modules" or "modules" or further separated into additional components and "modules" or "modules."

[0058] In this specification, the term "computer" refers to any type of hardware device including at least one processor, and may also encompass software configurations operating on such hardware devices according to embodiments. For example, the term "computer" may refer to, but is not limited to, smartphones, tablet PCs, desktops, laptops, and user clients and applications running on each device.

[0059] Furthermore, the "smart home appliances" described below are devices with built-in microphones that can detect the user's breathing sounds and collect acoustic data, and may include smart speakers, smart TVs, smart lighting, smart mattresses, etc.

[0060] In addition, a "sleep track app" may refer to an application that transmits a user's sleep report to a smartphone using a PUI, VUI, or GUI and operates smart home appliances according to the results of the report.

[0061] In addition, "research interaction" may mean that new products to improve users' sleep quality are researched and developed in the relevant categories, such as fragrances, cosmetics, food, health functional foods, and hormones.

[0062] In addition, "Sleep Track App Research Interaction" may mean that sleep environment creation services and new products for improving sleep quality will be developed based on sleep analysis performed by the Sleep Track App.

[0063] Additionally, "sleep management app interaction" may refer to interaction between the traditional sleep industry, sports, hotels, cram schools, military, and other related industries that enable sleep storytelling, and sleep management apps that enable sleep analysis without a hardware solution.

[0064] Additionally, "interaction from research interaction to sleep management app" may refer to the interaction between a new product that does not have a digital product and a sleep management app that is capable of sleep analysis without a hardware solution.

[0065] Those skilled in the art should further recognize that the various illustrative logical blocks, components, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or any combination of both. To clearly illustrate the interchangeability of hardware and software, the various illustrative components, blocks, components, means, logic, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is embodied as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present subject matter.

[0066] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0067] Although each step described in this specification is described as being performed by a computer, the subject of each step is not limited to this, and depending on the embodiment, at least some of each step may be performed by different devices.

[0068] Overall structure

[0069] 1(a) is a conceptual diagram illustrating a system in which various aspects of a computing device for creating a sleeping environment based on sleep state information, related to an embodiment of the present invention, can be implemented. The system according to an embodiment of the present invention may include a computing device 100, a user terminal 10, an external server 20, an environment creation device 30, and a network. Here, the system for implementing a method for creating a sleeping environment based on sleep state information shown in FIG. 1(a) is according to one embodiment, and its components are not limited to those of the embodiment shown in FIG. 1, and may be added, changed, or deleted as necessary.

[0070] Meanwhile, FIG. 1(b) is a conceptual diagram showing a system in which various aspects of a sleep environment adjusting device related to yet another embodiment of the present invention can be implemented.

[0071] A system according to an embodiment of the present invention may include a sleep environment adjusting device 400, a user terminal 10, an external server 20, and a network. Here, the system for implementing the method for creating a sleep environment based on sleep state information shown in Fig. 1(b) is according to one embodiment, and its components are not limited to the embodiment shown in Fig. 1(b), and may be added, changed, or deleted as necessary.

[0072] First, a system according to the embodiment shown in FIG. 1(a) will be described.

[0073] As shown in FIG. 1(a), in the present invention, a computing device 100, a user terminal 10, an external server 20, and an environment creation device 30 can mutually transmit and receive data for a system according to an embodiment of the present invention via a network.

[0074] Networks according to embodiments of the present invention can use various wired communication systems such as Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High Speed ​​DSL (VDSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN). In addition, the networks presented herein can use various wireless communication systems such as Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), Single Carrier-FDMA (SC-FDMA), and other systems.

[0075] A network according to an embodiment of the present invention may be configured regardless of its communication mode, such as wired or wireless, and may be configured as various communication networks such as a short-range communication network (PAN: Personal Area Network) or a short-range communication network (WAN: Wide Area Network). The network may be the well-known World Wide Web (WWW), or may use wireless transmission technologies used for short-range communication, such as infrared data association (IrDA) or Bluetooth (registered trademark). The technology described herein may be used in other networks as well as the networks mentioned above.

[0076] According to an embodiment of the present invention, the user terminal 10 may refer to a terminal carried by a user that can receive information related to the user's sleep through information exchange with the computing device 100. For example, the user terminal 10 may be a terminal associated with a user who wishes to improve their health through information related to their sleep habits. The user can obtain information related to their sleep through the user terminal 10. The sleep-related monitoring information may include, for example, sleep state information related to the time when the user fell asleep, the time when they fell asleep, the time when they woke up, etc., or sleep stage information related to changes in sleep stages during sleep. For example, the sleep stage information may refer to information on whether the user's sleep changed to light sleep, normal sleep, deep sleep, REM sleep, etc., for each time point during the user's eight hours of sleep last night. The specific description of the sleep stage information described above is merely exemplary, and the present invention is not limited thereto.

[0077] Meanwhile, FIG. 1(c) is a conceptual diagram showing a system in which various aspects of various electronic devices related to yet another embodiment of the present invention can be implemented.

[0078] The electronic device shown in FIG. 1(c) may perform at least one of the operations performed by various devices according to embodiments of the present invention.

[0079] For example, the operations performed by various devices according to embodiments of the present invention may include an operation of acquiring environmental sensing information, an operation of learning a sleep analysis model, an operation of inferring a sleep analysis model, an operation of acquiring sleep state information, an operation of controlling an electronic device, an operation of displaying sleep state information, and an operation of displaying environmental creation information.

[0080] Or, for example, it may include operations such as receiving information related to the user's sleep, sending or receiving environmental sensing information, determining the environmental sensing information, processing or manipulating data, processing services, providing services, analyzing the sleep state, constructing a learning data set based on the information related to the user's sleep, storing information on acquired data or multiple learning data for neural network learning, generating environment creation information, determining the environment creation information, operating an environment creation module based on the environment creation information, sending or receiving various information, and mutually transmitting and receiving data for the system according to an embodiment of the present invention via a network.

[0081] The electronic device shown in FIG. 1(c) may individually perform the operations performed by the various devices according to the embodiments of the present invention, or may perform one or more operations simultaneously or sequentially.

[0082] Referring to (c) of FIG. 1, the electronic devices 1a to 1d may be electronic devices within a predefined area 11a, which is an area where object state information such as information about the user's movements or breathing can be acquired.

[0083] On the other hand, referring to FIG. 1(c), the electronic devices 1a and 1d may be a device made up of a combination of two or more electronic devices.

[0084] Meanwhile, referring to FIG. 1(c), the electronic devices 1a and 1b may be electronic devices connected to a network within a pre-established area 11a.

[0085] Meanwhile, referring to FIG. 1(c), electronic devices 1c and 1d may be electronic devices that are not connected to the network within the already established area 11a.

[0086] On the other hand, referring to FIG. 1(c), the electronic devices 2a to 2b may be electronic devices that are outside the range of the already set area 11a.

[0087] On the other hand, referring to (c) of FIG. 1, there may be a network that interacts with electronic devices within the range of the already set area 11a, and there may be a network that interacts with electronic devices outside the range of the already set area 11a.

[0088] Here, the network interacting with electronic devices within the range of the pre-established area 11a can perform the role of transmitting and receiving information for controlling smart home appliances.

[0089] Furthermore, the network that interacts with the electronic device within the range of the pre-established area 11a may be, for example, a short-range network or a local network, whereas the network that interacts with the electronic device within the range of the pre-established area 11a may be, for example, a long-range network or a global network.

[0090] The specific explanation of the operation of the network shown in Figure 1(c) is the same as that explained through the drawings of Figure 1(a) or Figure 1(b), so duplicated explanation will be omitted.

[0091] On the other hand, referring to (c) of FIG. 1, there may be one or more electronic devices connected via a network outside the range of the previously set area 11a, and in this case, the electronic devices may process data in a distributed manner or perform one or more operations separately.

[0092] Alternatively, if there are one or more electronic devices connected via a network outside the range of the previously set area 11a, the electronic devices may perform operations independently of each other.

[0093] Hereinafter, various aspects according to embodiments of the present invention will be described with reference to FIG. 1(c), but the present invention is not limited thereto.

[0094] For example, according to one embodiment of the present invention, in an electronic device having environmental sensing and control functions implemented therein, steps may be performed: acquiring environmental sensing information; performing preprocessing on the acquired environmental sensing information; converting acoustic information contained in the preprocessed environmental sensing information into a spectrogram; generating sleep state information based on the converted spectrogram; and controlling the electronic device to create an environment based on the generated sleep state information.

[0095] Alternatively, according to one embodiment of the present invention, the steps of acquiring environmental sensing information, performing preprocessing on the acquired environmental sensing information, converting acoustic information included in the preprocessed environmental sensing information into a spectrogram, and transmitting the converted spectrogram to an AI server 310 may be performed within an electronic device having environmental sensing and control functions. If the AI ​​server 310 generates sleep state information through learning or inference based on the transmitted spectrogram, the electronic device may receive the sleep state information generated by the AI ​​server 310, and control the electronic device to create an environment based on the received sleep state information.

[0096] Alternatively, according to one embodiment of the present invention, there may be an electronic device for controlling a home appliance for creating an environment, and the electronic device may perform the steps of acquiring environmental sensing information, performing preprocessing on the acquired environmental sensing information, converting acoustic information included in the preprocessed environmental sensing information into a spectrogram, and generating sleep state information based on the converted spectrogram, and the electronic device may perform the steps of controlling the home appliance so that the home appliance can create an environment based on the generated sleep state information.

[0097] Alternatively, according to one embodiment of the present invention, there may be an electronic device for controlling a home appliance for creating an environment, and the electronic device may perform the steps of acquiring environmental sensing information, performing preprocessing on the acquired environmental sensing information, converting acoustic information included in the preprocessed environmental sensing information into a spectrogram, and transmitting the converted spectrogram to an AI server 310. If the AI ​​server 310 generates sleep state information through learning or inference based on the transferred spectrogram, the electronic device may receive the sleep state information generated by the AI ​​server 310, and control the home appliance to create an environment based on the received sleep state information.

[0098] Alternatively, according to an embodiment of the present invention, there may be an electronic device for controlling a home appliance for creating an environment, and another electronic device may acquire environmental sensing information, convert acoustic information included in the acquired environmental sensing information into a spectrogram, and generate sleep state information based on the converted spectrogram. The electronic device may then perform a step of receiving sleep state information from the other electronic device and a step of controlling the home appliance to create an environment based on the received sleep state information. Here, the other electronic device may be a device different from the electronic device controlling the home appliance and may correspond to one or more other electronic devices. If there are multiple other electronic devices, the steps of acquiring environmental sensing information, converting acoustic information included in the environmental sensing information into a spectrogram, and generating sleep state information may be performed independently.

[0099] For example, according to one embodiment of the present invention, there is an electronic device for controlling a home appliance for creating an environment, and if another electronic device acquires environmental sensing information, converts acoustic information included in the acquired environmental sensing information into a spectrogram, and transmits the converted spectrogram to an AI server 310, the AI ​​server 310 generates sleep state information based on the transmitted spectrogram, and the electronic device receives the sleep state information generated by the AI ​​server 310 and controls the home appliance to create an environment based on the received sleep state information. Here, the description of the other electronic device is the same as that described above, and therefore, a repeated description will be omitted.

[0100] The various embodiments of the present invention described above are examples to illustrate that various operations such as acquiring environmental sensing information, preprocessing the environmental sensing information, converting spectrograms, generating sleep state information, and controlling electronic devices or home appliances (e.g., smart home appliances) do not necessarily occur within the same electronic device, but can occur in different devices, and this can occur in chronological order, simultaneously, or independently and individually. Therefore, the present invention is not limited to the various embodiments described above.

[0101] Hereinafter, various operations according to the present invention will be described using specific examples. However, as described above, the examples of electronic devices used in the following description are merely examples for the purpose of clarity and are not intended to limit the electronic devices that perform specific operations.

[0102] Acquiring environmental sensing information

[0103] In an embodiment, the environmental sensing information of the present invention may be acquired via an electronic device (e.g., the user terminal 10, etc.). The environmental sensing information may refer to sensing information acquired in a space where a user is located. The environmental sensing information may be sensing information acquired in relation to the user's activity or sleep using a non-contact method.

[0104] For example, the environmental sensing information may be sleep acoustic information acquired in a bedroom where the user sleeps. According to an embodiment, the environmental sensing information acquired through the user terminal 10 may be information that serves as a basis for acquiring the user's sleep state information in the present invention. As a specific example, sleep state information related to whether the user is before sleep, during sleep, or after sleep may be acquired through the environmental sensing information acquired in relation to the user's activity.

[0105] For example, the environmental sensing information may include information about the user's breathing and movement. To this end, the user terminal 10 may include a radar sensor as a motion sensor. The user terminal 10 may perform signal processing on the user's movement and distance measured through the radar sensor to generate a discrete waveform (breathing information) corresponding to the user's breathing.

[0106] For example, the environmental sensing information may include measurements obtained via sensors measuring the temperature, humidity, and lighting levels in a bedroom. To this end, the user terminal 10 may be equipped with sensors measuring the temperature, humidity, and lighting levels in a bedroom.

[0107] The user terminal 10 may refer to any type of entity in a system having a mechanism for communication with the computing device 100. For example, the user terminal 10 may include a personal computer (PC), a notebook computer, a mobile terminal, a smartphone, a tablet PC, an AI speaker, an AI TV, a wearable device, etc., and may include all types of terminals that can be connected to a wired / wireless network. The user terminal 10 may also include any server implemented by at least one of an agent, an application programming interface (API), and a plug-in. The user terminal 10 may also include an application source and / or a client application.

[0108] According to an embodiment of the present invention, the external server 20 may be a server that stores information on a plurality of training data for training a neural network. The plurality of training data may include, for example, health checkup information or sleep screening information. For example, the external server 20 may be at least one of a hospital server and an information server, and may be a server that stores information on a plurality of sleep screening test records, electronic health records, electronic medical records, etc. For example, the sleep screening test records may include information on the breathing and movements of a sleep screening subject during sleep, and information on corresponding sleep diagnosis results (e.g., sleep stages, etc.). The information stored in the external server 20 may be used as training data, verification data, and test data for training the neural network of the present invention.

[0109] The computing device 100 of the present invention can receive health checkup information or sleep screening information from the external server 20 and construct a training dataset based on the received information. The computing device 100 can generate a sleep analysis model for acquiring sleep state information corresponding to environmental sensing information by performing training on one or more network functions through the training dataset. A configuration for constructing a training dataset for neural network training of the present invention and a training method using the training dataset will be described in detail below.

[0110] The external server 20 may be a digital device equipped with a processor, memory, and computing power, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone. The external server 20 may be a web server that processes services. The above-mentioned types of servers are merely examples, and the present invention is not limited thereto.

[0111] According to an embodiment of the present invention, the environment creation device 30 may adjust a user's sleep environment. Specifically, the environment creation device 30 may include one or more environment creation modules, and may adjust the user's sleep environment by operating an environment creation module related to at least one of air quality, illuminance, temperature, wind direction, humidity, and sound in a space where the user is located based on environment creation information received from the computing device 100.

[0112] Also, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform the above-described operations.

[0113] The environment creating device 30 may be embodied as a TV that provides images and videos and generates sound, an air purifier that can control air quality, a lighting device that can control light intensity (illuminance), a cooler / heater that can control temperature, an air conditioner that can adjust temperature and humidity, a humidifier / dehumidifier that can control humidity, an audio / speaker that can control sound, a styler that can manage clothes, blinds or curtains, a robot or vacuum cleaner, a washing machine or dryer, a water purifier, an oven or range, etc.

[0114] The environment creation information may be a signal generated from the computing device 100 based on the determination of the user's sleep state information. For example, the environment creation information may include information on lowering or increasing illuminance. If the environment creation device 30 is a lighting device, the environment creation information may include control information for gradually increasing the illuminance of 3000K white light from 0 lux to 250 lux 30 minutes before the predicted time of waking up.

[0115] For example, if the environment creating device 30 is an air purifier or an air conditioner, the environment creating information may include various information related to temperature and / or humidity adjustment, fine dust (fine dust, ultrafine dust, and ultra-ultrafine dust) removal, harmful gas removal, allergy care activation, deodorization / sterilization activation, dehumidification / humidification adjustment, airflow intensity adjustment, air purifier or air conditioner operation noise adjustment, LED lighting, smog-causing substance (SO2, NO2) management, household odor removal, etc. If the environment creating device 30 is an air conditioner, the environment creating information may include sleep space temperature and humidity adjustment, airflow intensity adjustment, operation noise adjustment, LED lighting, etc., based on the user's real-time sleep state.

[0116] For example, the environment creation information may include control information for adjusting at least one of temperature, humidity, wind direction, and sound. The above-described specific descriptions of the environment creation information are merely examples, and the present invention is not limited thereto.

[0117] The one or more environment creation modules included in the environment creation device 30 may include, for example, at least one of a light control module, a temperature control module, a wind direction control module, a humidity control module, and a sound control module. However, without being limited thereto, the one or more environment creation modules may further include various environment creation modules that can bring about changes in the user's sleep environment. That is, the environment creation device 30 can adjust the user's sleep environment by driving one or more environment creation modules based on an environment control signal from the computing device 100.

[0118] According to an embodiment of the present invention, the computing device 100 may acquire sleep state information of a user and adjust the user's sleep environment based on the sleep state information. Specifically, the computing device 100 may acquire sleep state information related to whether the user is about to fall asleep, asleep, or asleep based on environmental sensing information, and adjust the sleep environment of the space in which the user is located based on the sleep state information. For example, if the computing device 100 acquires sleep state information indicating that the user is about to fall asleep, the computing device 100 may generate environment creation information related to light intensity and illuminance (e.g., 3000K white light, 30 lux illuminance) and air quality (fine dust concentration, toxic gas concentration, air humidity, air temperature, etc.) for inducing sleep based on the sleep state information. The computing device 100 may transmit the environment creation information related to light intensity and illuminance and air quality for inducing sleep to the environment creation device 30. In this case, the environment creation device 30 may adjust the light intensity and illuminance of the space where the user is located to an appropriate intensity and illuminance for inducing sleep (e.g., 3000K white light with an illuminance of 30 lux) based on the environment creation information received from the computing device 100. That is, the environment creation information generated in the computing device 100 may be transmitted to a lighting device, which is one embodiment of the environment creation device 30, to adjust the illuminance in the sleeping space.

[0119] In addition, the computing device 100 can generate environment creation information such as fine dust removal, harmful gas removal, allergy care activation, deodorization / sterilization activation, dehumidification / humidification adjustment, airflow intensity adjustment, operating noise adjustment of the environment creation device 30, and various information related to LED lighting based on the user's sleep state information.

[0120] In addition, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform the above-described operations.

[0121] For example, the environment creation information generated in the computing device 100 may be transmitted to an air purifier or air conditioner, which is an embodiment of the environment creation device 30, to adjust the temperature, humidity, or air quality in a room, vehicle, or sleeping space.

[0122] Hereinafter, when describing the operation of smart home appliances, the terms "sleep mode" and "wake-up mode" will be used for convenience. The "sleep mode" is a concept that includes the operation modes of smart home appliances when the user is preparing to go to bed, when the user is falling asleep, and when the user is asleep, respectively. The "wake-up mode" is a concept that includes the operation modes of smart home appliances when the user is about to wake up, when the user is waking up, and when the user is waking up, respectively.

[0123] 52 is a table showing whether or not an environment creation device is activated based on sleep state information, and exemplary operations in sleep mode and wake-up mode, for each location where the environment creation device is placed and each detailed product. Specifically, the table shows whether or not an environment creation device 30 is activated based on sleep state information (going to sleep, falling asleep, sleeping, before waking up, waking up, after waking up), and exemplary operations in sleep mode and wake-up mode, for each location where the environment creation device 30 is placed and each detailed product of the environment creation device 30. The environment creation information may include control information for enabling the activation and operations in sleep mode and wake-up mode to be performed for each product.

[0124] The above-described specific descriptions regarding the sleep state information and the environment creation information are merely examples, and the present invention is not limited thereto.

[0125] According to an embodiment of the present invention, the environmental sensing information utilized by the computing device 100 for analyzing the sleep state may include information acquired non-invasively during a user's activity in a space or sleep. For specific examples, the environmental sensing information may include sounds generated when the user turns over in their sleep, sounds associated with muscle movements, or sounds associated with the user's breathing during sleep. Alternatively, the environmental sensing information may include movement and distance information associated with the user's movement during sleep and breathing information generated based thereon.

[0126] According to an embodiment, the environmental sensing information may include sleep acoustic information, which may refer to acoustic information related to movement and breathing patterns occurring during the user's sleep, or may include sleep movement information, which may refer to information related to movement and breathing patterns occurring during the user's sleep.

[0127] In an embodiment, the environmental sensing information may be acquired through a user terminal 10 carried by a user. For example, the environmental sensing information related to the user's activities in a space may be acquired through a microphone module provided in the user terminal 10. Alternatively, the environmental sensing information related to the user's activities in a space may be acquired through a radar sensor provided in the user terminal 10.

[0128] Generally, a microphone module installed in a user terminal 10 carried by a user may be configured with a Micro-Electro Mechanical System (MEMS) since it must be installed in a relatively small user terminal 10. Such a microphone module can be manufactured in a very small size, but may have a lower signal-to-noise ratio (SNR) than a condenser microphone or a dynamic microphone. A low SNR means that the ratio of noise, which is sound that makes it difficult to identify the sound to be identified, is high, and the sound is difficult to identify (i.e., unknown).

[0129] In the present invention, the environmental sensing information to be analyzed may include acoustic information related to the user's breathing and movements acquired during sleep, i.e., sleep acoustic information. Such sleep acoustic information is information about very small sounds (i.e., sounds that are difficult to distinguish), such as the user's breathing and movements, and is acquired together with other sounds in the sleep environment. Therefore, if it is acquired through the above-mentioned microphone module with a low signal-to-noise ratio, it may be very difficult to detect and analyze.

[0130] According to an embodiment of the present invention, the computing device 100 may acquire sleep state information based on environmental sensing information acquired from the user terminal 10. Specifically, the computing device 100 may convert and / or adjust the environmental sensing information, which is acquired unclearly due to a large amount of noise, to enable analysis, and may perform training on the artificial neural network using the converted and / or adjusted data. Once pre-training of the artificial neural network is complete, the trained neural network (e.g., an acoustic analysis model) may acquire sleep state information of the user based on data (e.g., a spectrogram) acquired (e.g., converted and / or adjusted) corresponding to the sleep acoustic information. In an embodiment, the sleep state information may include not only information related to whether the user is sleeping, but also sleep stage information related to changes in the user's sleep stage during sleep. For example, the sleep state information may include sleep stage information indicating that the user was in REM sleep at a first time point and was in light sleep at a second time point different from the first time point. In this case, through the sleep state information, it is possible to obtain information that the user entered a relatively deep sleep at the first time point and entered a lighter sleep at the second time point.

[0131] That is, when the computing device 100 acquires sleep sound information having a low signal-to-noise ratio through many commonly used user terminals for collecting sound (e.g., AI speakers, bedroom IoT devices, mobile phones, etc.), it processes the acquired data into data suitable for analysis and can provide sleep state information related to changes in sleep stages by processing the processed data. This eliminates the need for a microphone that contacts the user's body to acquire clear sound, and also allows the monitoring of sleep states in a typical home environment with just a software update without the need to purchase a separate additional device with a high signal-to-noise ratio, thereby providing an effect of increased convenience.

[0132] Although the computing device 100 and the environment creation device 30 are shown as separate entities in FIG. 1(a), in accordance with an embodiment of the present invention, the environment creation device 30 may be included within the computing device 100, and the sleep state measurement and environment adjustment operation functions may be performed as a single integrated device.

[0133] In addition, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform the above-described operations.

[0134] In an embodiment, the computing device 100 may be a terminal or a server, and may include any type of device. The computing device 100 may be a digital device equipped with a processor, memory, and computing capabilities, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone. The computing device 100 may also be a web server that processes services. The types of servers described above are merely examples, and the present invention is not limited thereto.

[0135] According to an embodiment of the present invention, the computing device 100 may be a server that provides a cloud computing service. More specifically, the computing device 100 may be a server that provides a cloud computing service, a type of Internet-based computing, in which information is processed by another computer connected to the Internet, rather than the user's computer. The cloud computing service may store data on the Internet and allow users to access the data or programs they need anytime and anywhere via an Internet connection without having to install them on their own computers. The cloud computing service may also allow users to easily share and transfer data stored on the Internet with simple operations and clicks. The cloud computing service may not only simply store data on an Internet server, but also allow users to perform desired tasks using the functions of web-based applications without installing additional programs, and may allow various users to share documents and work simultaneously. The cloud computing service may be implemented in at least one form among Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, and a container-based cloud server. That is, the computing device 100 of the present invention may be implemented in at least one form among the above-mentioned cloud computing services. The specific description of the cloud computing service mentioned above is merely an example and may include any platform that builds the cloud computing environment of the present invention.

[0136] The overall configuration of the computing device

[0137] The specific configuration, technical features, and effects of the technical features of the computing device 100 of the present invention will be described with reference to the accompanying drawings.

[0138] FIG. 2 illustrates a block diagram of a computing device for creating a sleep environment based on sleep state information in accordance with an embodiment of the present invention.

[0139] 2, the computing device 100 may include a network unit 110, a memory 120, and a processor 130. The computing device 100 is not limited to the components described above. That is, additional components may be included, or some of the components described above may be omitted, depending on the implementation of the present invention.

[0140] According to an embodiment of the present invention, the computing device 100 may include a network unit 110 that transmits and receives data to and from the user terminal 10, the external server 20, and the environment creation device 30. The network unit 110 may transmit and receive data, etc., for performing the method for creating a sleep environment using sleep state information according to an embodiment of the present invention, to and from other computing devices, servers, etc. That is, the network unit 110 may provide a communication function between the computing device 100, the user terminal 10, the external server 20, and the environment creation device 30. For example, the network unit 110 may receive sleep examination records and electronic health records for multiple users from a hospital server. As another example, the network unit 110 may receive environmental sensing information related to a space in which a user is active from the user terminal 10. As another example, the network unit 110 may transmit environment creation information to the environment creation device 30 to adjust the environment of the space in which the user is located. Additionally, the network unit 110 may allow information transfer between the computing device 100 and the user terminal 10 and the external server 20 by calling a procedure in the computing device 100 .

[0141] The network unit 110 according to an embodiment of the present invention can use various wired communication systems such as a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High Speed ​​DSL (VDSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and a Local Area Network (LAN).

[0142] In addition, the network unit 110 presented in this specification can use various wireless communication systems that can be realized now and in the future, such as mobile communication systems such as 4G and 5G (LTE), and satellite communication systems such as Starlink.

[0143] In the present invention, the network unit 110 may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a short-range communication network (PAN: Personal Area Network) or a short-range communication network (WAN: Wide Area Network). In addition, the network may be the well-known World Wide Web (WWW), or may use wireless transmission technology used for short-range communication, such as infrared data association (IrDA) or Bluetooth (Bluetooth®). The technology described herein can be used not only with the networks mentioned above, but also with other networks.

[0144] According to an embodiment of the present invention, the memory 120 may store a computer program for performing a method for creating a sleep environment based on sleep state information according to an embodiment of the present invention, and the stored computer program may be read and driven by the processor 130. The memory 120 may also store any type of information generated or determined by the processor 130 and any type of information received by the network unit 110. The memory 120 may also store data related to the user's sleep. For example, the memory 120 may temporarily or permanently store input / output data (e.g., environmental sensing information related to the user's sleep environment, sleep state information corresponding to the environmental sensing information, or environment creation information based on the sleep state information).

[0145] According to an embodiment of the present invention, the memory 120 may include at least one type of storage medium selected from the group consisting of flash memory, hard disk, micro multimedia card, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device 100 may also operate in association with web storage that performs the storage function of the memory 120 over the Internet. The above description of memory is for illustrative purposes only, and the present invention is not limited thereto.

[0146] The computer program may include one or more instructions that, when loaded into memory 120, cause processor 130 to perform methods / operations according to various embodiments of the present invention. That is, processor 130 may perform methods / operations according to various embodiments of the present invention by executing one or more instructions.

[0147] In one embodiment, the computer program may include one or more instructions for performing a method for creating a sleep environment based on sleep state information, the method including acquiring sleep state information of a user, generating environment creation information based on the sleep state information, and transmitting the environment creation information to an environment creation device.

[0148] According to one embodiment of the present invention, processor 130 may be configured with one or more cores and may include a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), or other processors for data analysis, deep learning, etc.

[0149] The processor 130 may read a computer program stored in the memory 120 and perform data processing for machine learning according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 130 may perform calculations for neural network training. The processor 130 may perform calculations for neural network training, such as processing input data for training in deep learning (DL), extracting features from the input data, calculating errors, and updating weights of the neural network using backpropagation.

[0150] In addition, at least one of the CPU, GPGPU, and TPU of the processor 130 may process network function learning. For example, the CPU and GPGPU may both process network function learning and data classification using the network function.

[0151] For example, both the CPU and GPGPU can process network function learning and data classification using network functions.

[0152] In addition, in one embodiment of the present invention, processors of a plurality of computing devices may be used together to process network function learning and data classification using the network function. In one embodiment of the present invention, processors of a plurality of computing devices may be used together to process network function learning and data classification using the network function. In addition, a computer program executed in a computing device according to one embodiment of the present invention may be a CPU, GPGPU, or TPU executable program.

[0153] As used herein, a network function may be used interchangeably with an artificial neural network or a neural network. As used herein, a network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of the one or more neural networks.

[0154] As used herein, a model may include a network function. A model may include one or more network functions, in which case the output of the model may be an ensemble of the outputs of the one or more network functions.

[0155] The processor 130 may read a computer program stored in the memory 120 to provide a sleep analysis model according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 130 may perform calculations to calculate environment creation information based on sleep state information. According to an embodiment of the present invention, the processor 130 may perform calculations to train the sleep analysis model. The sleep analysis model will be described in more detail below. Sleep information related to the user's sleep quality may be inferred based on the sleep analysis model. Environmental sensing information acquired from the user in real time or periodically is input as an input value to the sleep analysis model, and data related to the user's sleep is output.

[0156] The learning of the sleep analysis model and the inference based thereon may be performed by the computing device 100 of FIG. 1(a). That is, both the learning and the inference may be designed to be performed by the computing device 100. However, in another embodiment, the learning may be performed by the computing device 100, but the inference may be performed by the user terminal 10. Also, the learning may be performed by the computing device 100, but the inference may be performed by the environment creation device 30 embodied in smart home appliances (various home appliances such as air conditioners, TVs, lighting, refrigerators, air purifiers, etc.). Also, in another embodiment, the learning may be performed by the sleep environment control device 400 of FIG. 1(b). That is, both the learning and the inference may be performed by the sleep environment control device 400.

[0157] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0158] According to an embodiment of the present invention, the processor 130 may generally process the overall operation of the computing device 100. The processor 130 may process signals, data, information, etc. input or output via the components described in detail above, or may run applications stored in the memory 120, thereby providing or processing appropriate information or functions to the user terminal.

[0159] According to an embodiment of the present invention, processor 130 may acquire sleep state information of the user. Acquisition of sleep state information according to an embodiment of the present invention may involve acquiring or loading sleep state information stored in memory 120. Acquisition of sleep state information may also involve receiving or loading data from another storage medium, another computing device, or another processing module within the same computing device via wired or wireless communication.

[0160] Also, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0161] Sleep status information

[0162] In one embodiment, the sleep state information may include information related to whether the user is asleep. Specifically, the sleep state information may include at least one of first sleep state information indicating that the user is before sleeping, second sleep state information indicating that the user is asleep, and third sleep state information indicating that the user has fallen asleep. In other words, if first sleep state information is inferred related to a user, processor 130 may determine that the user is in a pre-sleep (i.e., before falling asleep) state; if second sleep state information is inferred, processor 130 may determine that the user is in a sleeping state; and if third sleep state information is obtained, processor 130 may determine that the user is in a post-sleep (i.e., wake-up) state.

[0163] The sleep state information may be acquired based on environmental sensing information, which may include sensing information acquired in a space where the user is located in a non-contact manner.

[0164] According to an embodiment, the processor 130 may acquire environmental sensing information. Specifically, the environmental sensing information may be acquired via the user terminal 10 carried by the user. For example, environmental sensing information related to a space in which the user is active may be acquired via the user terminal 10 carried by the user, and the processor 130 may receive the environmental sensing information from the user terminal 10. The environmental sensing information may be acoustic information acquired in a non-contact manner during the user's daily life. For example, the environmental sensing information may include various acoustic information acquired during the user's daily life, such as acoustic information related to cleaning, acoustic information related to cooking food, acoustic information related to TV viewing, and sleep acoustic information acquired during sleep. In an embodiment, the sleep acoustic information acquired during the user's sleep may include acoustic information generated by the user turning over in bed during sleep, acoustic information related to muscle movement, or acoustic information related to the user's breathing during sleep. That is, the sleep acoustic information in the present invention may refer to acoustic information related to the user's movement patterns and breathing patterns during sleep.

[0165] Sleep analysis information and sleep stage information

[0166] Sleep analysis analyzes various information such as time to fall asleep, time to wake up, and total sleep time. According to an embodiment, the processor 130 can extract sleep stage information. The sleep stage information can be extracted based on environmental sensing information of the user. Sleep stages can be classified into non-REM (NREM) sleep and rapid eye movement (REM) sleep, and NREM sleep can be further classified into multiple stages (e.g., two stages: light and deep, and four stages: N1 to N4). Sleep stages can be defined as general sleep stages, or various sleep stages can be arbitrarily set by the designer. Sleep stage analysis can predict not only sleep quality but also sleep disorders (e.g., sleep apnea) and their underlying causes (e.g., snoring).

[0167] In sleep analysis, changes in sleep stages can be analyzed and a hypnogram can be generated to enable changes in the analyzed sleep stages to be identified, thereby allowing the user's sleep cycles to be identified.

[0168] FIG. 3 is a graph comparing the results of polysomnography (PSG) and the results of analysis using the AI ​​algorithm according to the present invention.

[0169] As shown in Figure 3, the sleep stage information obtained by the present invention not only closely matches that of a multidimensional sleep test, but also includes more precise and meaningful information related to sleep stages (Wake, Light, Deep, REM). The hypnogram shown at the bottom of Figure 3 shows the probability of which of four classes (Wake, Light, Deep, REM) the user will fall into in 30-second increments when predicting a sleep stage based on input user audio information. Here, the four classes represent the awake state, light sleep state, deep sleep state, and REM sleep state, respectively.

[0170] Figure 4 compares the results of a polysomnography (PSG) test (PSG result) and the analysis results (AI result) using the AI ​​algorithm according to the present invention in relation to sleep apnea and hypopnea. The hypnogram shown at the bottom of Figure 4 shows the probability of the user being diagnosed with one of two disorders (sleep apnea or hypopnea) in 30-second intervals when predicting a sleep disorder based on input of user audio information.

[0171] Using the sleep stage information according to the present invention, as shown in FIG. 4, the sleep stage information obtained according to the present invention is not only highly consistent with the sleep polymorphism test, but also includes more precise analytical information related to apnea and hypopnea.

[0172] The processor 130 may generate environment creation information according to the sleep stage information. For example, if the sleep stage is the Light stage or the N1 stage, the processor 130 may generate environment creation information for controlling environment creation devices (such as an air conditioner, lighting, or air purifier) ​​to induce deep sleep.

[0173] For example, some smart home appliances 800 according to embodiments of the present invention may be configured to sound an alarm if REM sleep is detected within 30 minutes of the wake-up time set by the user.

[0174] This is because if an alarm sounds during REM sleep, the user will wake up more refreshed. The sleep management app of the present invention can detect REM in real time while the user is sleeping and deliver auditory or tactile stimuli to the user to wake them up within the specified time.

[0175] In addition, some smart home appliances 800 according to embodiments of the present invention may detect periods of unstable breathing based on sleep acoustic information while the user is sleeping, and may provide vibrotactile stimulation to the user to encourage the user to return to stable breathing.

[0176] Generally, if sleep apnea persists, the sympathetic nervous system becomes activated, which may later lead to cardiovascular diseases. Therefore, when an unstable breathing period is detected in real time during the user's sleep through the sleep management app of the present invention, auditory and tactile stimuli can be transmitted to the user through a part of the smart home appliance 800 according to an embodiment of the present invention to interrupt the user's unstable breathing.

[0177] Obstructive sleep apnea can be screened in stages based on body movement information or user posture information.

[0178] The sleep analysis analyzes the quality of sleep, sleep stages, and the presence or absence of sleep apnea based on sleep acoustic information. The sleep acoustic information may refer to acoustic information related to breathing generated during a user's sleep.

[0179] Sleep analysis analyzes the user's sleep stages through pre-processing of the user's sleep acoustic information and AI algorithms. The specific analysis method will be explained in more detail below.

[0180] Identifying singular points by detecting pre-defined patterns

[0181] According to an embodiment of the present invention, the processor 130 may acquire sleep state information based on the environmental sensing information. Specifically, the processor 130 may identify a singular point at which pre-defined pattern information is detected in the environmental sensing information. Here, the pre-defined pattern information may be related to breathing and movement patterns associated with sleep. For example, in a wakeful state, the entire nervous system is activated, resulting in irregular breathing patterns and frequent body movements. Furthermore, the neck muscles may not be relaxed, resulting in very little breathing noise. On the other hand, when the user is sleeping, the autonomic nervous system is stabilized, resulting in regular breathing, less body movements, and louder breathing noise. That is, the processor 130 may identify a time point at which pre-defined pattern sound information associated with regular breathing, less body movements, or less breathing noise is detected in the environmental sensing information as a singular point. Furthermore, the processor 130 may acquire sleep sound information based on the environmental sensing information acquired based on the identified singular point. The processor 130 may identify a specific point associated with the user's sleep time point in the environmental sensing information acquired in a time series manner, and acquire sleep sound information based on the specific point.

[0182] FIG. 5 is an exemplary diagram illustrating a process of acquiring sleep sound information 210 from environmental sensing information 200 according to an embodiment of the present invention.

[0183] 5, the processor 130 may identify a singular point 201 associated with a point in time at which a previously established pattern is identified from the environmental sensing information 200. The processor 130 may acquire sleep sound information 210 based on sound information acquired after the identified singular point. The sound-related waveforms and singular points in FIG. 5 are merely examples for understanding the present invention, and the present invention is not limited thereto.

[0184] That is, the processor 130 can identify specific points related to the user's sleep from the environmental sensing information, and extract and acquire only the sleep sound information from a huge amount of sound information (i.e., the environmental sensing information) based on the specific points. This can automate the process of the user recording their sleep time, providing convenience and contributing to improving the accuracy of the acquired sleep sound information.

[0185] In addition, in an embodiment, the processor 130 may acquire sleep state information related to whether the user is asleep or not based on the singular point 201 identified from the environmental sensing information 200. Specifically, the processor 130 may determine that the user is asleep if the singular point 201 is not identified, and may determine that the user is asleep after the singular point 201 is identified. Furthermore, the processor 130 may identify a time point (e.g., a wake-up time) at which a previously set pattern is not observed after the singular point 201 is identified, and may determine that the user has fallen asleep, i.e., woken up, if the time point is identified.

[0186] That is, the processor 130 can acquire sleep state information related to whether the user is before sleep, during sleep, or after sleep based on whether a singular point 201 is identified in the environmental sensing information 200 and whether a previously set pattern is continuously sensed after the singular point is identified.

[0187] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0188] In addition, the processor 830 included in the smart home device 800 according to an embodiment of the present invention may identify a singular point 201 associated with a point in time at which a pre-established pattern is identified from the acoustic information 200.

[0189] The processor 830 may acquire the sleep acoustic information 210 based on the acoustic information acquired after the identified singular point 201, with the singular point 201 as a reference.

[0190] The acoustic waveforms and singularities in FIG. 5 are merely examples for understanding the present invention, and the present invention is not limited thereto.

[0191] That is, the processor 830 included in the smart home appliance 800 according to an embodiment of the present invention can identify singular points 201 related to the user's sleep from the acoustic information, and extract and acquire only the sleep acoustic information 210 from a vast amount of environmental sensing information (i.e., acoustic information) based on the singular points 201.

[0192] This provides convenience by automating the process of recording the user's sleep time, and can also contribute to improving the accuracy of the acquired sleep acoustic information.

[0193] In addition, in an embodiment, the processor 830 may acquire sleep state information related to whether the user is about to fall asleep or is asleep based on the singular point 201 identified from the environmental sensing information 200. Specifically, the processor 830 may determine that the user is about to fall asleep if the singular point 201 is not identified, and may determine that the user is asleep after the singular point 201 is identified.

[0194] In addition, after identifying singular point 201, processor 830 identifies a time point (e.g., the time of waking up) at which the already set pattern is not observed, and if that time point is identified, it can determine that the user has fallen asleep, i.e., woken up.

[0195] That is, the processor 830 can acquire sleep state information related to whether the user is before sleep, during sleep, or after sleep based on whether a singular point 201 is identified in the environmental sensing information 200 and whether a previously set pattern is continuously sensed after the singular point is identified.

[0196] On the other hand, the processor 830 can obtain the sleep state information based on the sleep acoustic information rather than the environmental sensing information 200 .

[0197] In the present invention, the sleep state information of the user is grasped in advance using sleep acoustic information during the primary sleep analysis, so that the reliability of the analysis of the sleep state can be further improved.

[0198] The sleep analysis method according to the present invention generates an inference model through deep learning of environmental sensing information, and the inference model extracts the user's sleep state and sleep stage.

[0199] To put it simply again, environmental sensing information (acoustic information) including sleep acoustic information is converted into a spectrogram, and an inference model is generated based on the spectrogram.

[0200] In this case, in sleep analysis using acoustic information, the protection of user privacy cannot be overlooked, and the present invention uses a process of preprocessing environmental sensing information (acoustic information) to protect user privacy.

[0201] As described above, an inference model for extracting a user's sleep state and sleep stage is generated through deep learning of environmental sensing information. Briefly again, environmental sensing information including acoustic information is converted into a spectrogram, and an inference model can be generated based on the spectrogram.

[0202] As described above, the inference model can be built in the computing device 100 shown in FIG. 1(a) or the sleep environment adjustment device 400 shown in FIG. 1(b).

[0203] Thereafter, environmental sensing information including user acoustic information acquired through the user terminal is input to the inference model, and sleep state information and / or sleep stage information is output as a result value. At this time, learning and inference may be performed by the same entity, or may be performed by separate entities. That is, both learning and inference may be performed by the computing device 100 of FIG. 1(a) or the environmental control device 400 of FIG. 1(b). Learning may be performed by the computing device 100, but inference may be performed by the user terminal 10. Learning may be performed by the computing device 100, but inference may be performed by the environment creation device 30 embodied in smart home appliances (various home appliances such as air conditioners, TVs, lighting, refrigerators, air purifiers, etc.).

[0204] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the operations described above.

[0205] Sleep analysis model and sleep analysis method

[0206] According to an embodiment of the present invention, the sleep stage information may be obtained through a sleep analysis model that analyzes the sleep stages of a user based on environmental sensing information. That is, the sleep stage information of the present invention may be obtained through a sleep analysis model.

[0207] According to an embodiment of the present invention, processor 130 or processor 830 may acquire environmental sensing information and acquire sleep sound information based on the environmental sensing information. In this case, the sleep sound information is information related to sounds acquired while the user is sleeping, and may include, for example, sounds generated by the user turning over in their sleep, sounds related to muscle movements, or sounds related to the user's breathing while sleeping.

[0208] Hereinafter, a sleep analysis method according to an embodiment of the present invention will be described with reference to the drawings.

[0209] FIG. 32(a) is a diagram for explaining sleep stage analysis using a spectrogram in the sleep analysis method according to the present invention.

[0210] FIG. 32(b) is a diagram for explaining the determination of a sleep disorder using a spectrogram in the sleep analysis method according to the present invention.

[0211] FIG. 33(a) is a diagram showing an experimental process for verifying the performance of the sleep analysis method according to the present invention.

[0212] Figure 33(b) is a graph verifying the performance of the sleep analysis method according to the present invention, comparing the results of a sleep polymorphism test (PSG result) with the analysis results (AI result) using the AI ​​algorithm according to the present invention.

[0213] As shown in FIG. 32(a), when the sleep sound information of the user is input, the corresponding sleep stage (Wake, REM, Light, Deep) may be immediately inferred.

[0214] Furthermore, secondary analysis based on sleep acoustic information can extract the time points at which sleep disorders (sleep apnea, hyperventilation) and snoring occur through the singular points of the Mel spectrum corresponding to sleep stages.

[0215] As shown in Figure 32(b), if a breathing pattern is analyzed in one melspectrogram and characteristics corresponding to a sleep apnea or hyperpnea event are detected, the time point can be determined as the time point when a sleep disorder occurs. In this case, a step of classifying the event as snoring rather than sleep apnea or hyperpnea through frequency analysis may be further included.

[0216] As shown in Figure 33(a), a user's sleep video and sleep sounds are acquired in real time, and the acquired sleep sound information is immediately converted into a spectrogram. At this time, a pre-processing process of the sleep sound information may be performed. The spectrogram is input into a sleep analysis model, and the sleep stages are immediately analyzed.

[0217] When compared with the results of polysomnography (PSG), it was confirmed that the results of the sleep analysis model using sleep acoustic information as input were highly accurate.

[0218] The hypnogram at the bottom of Figure 33(a) shows the probability of which of four classes (Wake, Light, Deep, REM) the user will fall into in 30-second increments when predicting their sleep stage based on the input of the user's sleep acoustic information. Here, the four classes represent the awake state, light sleep state, deep sleep state, and REM sleep state, respectively.

[0219] As shown in Figure 33(b), the sleep analysis results obtained by the present invention are not only highly consistent with the sleep polymorphism test, but also contain more precise and meaningful information related to sleep stages (Wake, Light, Deep, REM).

[0220] Spectrogram generation and acquisition

[0221] FIG. 6(a) is an exemplary diagram illustrating a method for acquiring a spectrogram corresponding to sleep acoustic information according to an embodiment of the present invention.

[0222] According to the present invention, a sleep analysis model can be generated using a spectrogram generated based on sleep audio information. If sleep audio information expressed as audio data were used directly, the amount of information would be very large, resulting in a significant increase in the amount of calculation and time required. Furthermore, the inclusion of unwanted signals would reduce calculation accuracy. Furthermore, if all of a user's audio signals were transmitted to a server, privacy could be infringed. The present invention removes noise from the sleep audio information, converts it into a spectrogram (Mel spectrogram), and then trains the spectrogram to generate a sleep analysis model. This reduces the amount of calculation and time required, and even protects personal privacy.

[0223] The processor 130 or the processor 830 according to the embodiment of the present invention can generate a spectrogram 300 corresponding to the sleep acoustic information 210 as shown in FIG. 6(a).

[0224] Raw data (sleep acoustic information) that is the basis for generating the spectrogram 300 can be input. The raw data can be acquired from the start time to the end time entered by the user via a user terminal, or from the time when the user operates the terminal (e.g., setting an alarm) to the time corresponding to the terminal operation (e.g., the alarm setting time). The raw data can also be acquired by automatically selecting a time based on the user's sleep pattern. Alternatively, the user's intended sleep time can be automatically determined and acquired based on sounds (such as the user's voice, breathing sounds, sounds from peripheral devices (TV, washing machine), etc.) or changes in illumination.

[0225] Although not shown in FIG. 6(a), a preprocessing process of input raw data may be further included. The preprocessing process includes a noise reduction process of the raw data. In the noise reduction process, noise (e.g., white noise) contained in the raw data is removed. The noise reduction process may be performed using algorithms such as spectral gating and spectral subtraction to remove background noise. Furthermore, in the present invention, the noise reduction process may be performed using a deep learning-based noise reduction algorithm. That is, a noise reduction algorithm specialized for the user's breathing and respiratory sounds may be used through deep learning. In particular, the present invention may generate a spectrogram based only on the amplitude of the raw data, excluding the phase, but is not limited thereto. This not only protects privacy but also reduces data volume and improves processing speed.

[0226] According to an embodiment of the present invention, the processor 130 or the processor 830 may perform a fast Fourier transform on the sleep acoustic information 210 to generate a spectrogram 300 corresponding to the sleep acoustic information 210 .

[0227] Spectrogram 300 is used to visualize and understand sounds and waves, and may be a combination of waveform and spectrum characteristics. Spectrogram 300 may show differences in amplitude according to changes in the time axis and frequency axis as differences in print density or display color.

[0228] The preprocessed acoustic-related raw data may be cut into 30-second units and converted into a mel spectrogram. As a result, a 30-second mel spectrogram may have dimensions of 20 frequency bins x 1201 time steps. In the present invention, a split-cut method is used to convert a rectangular mel spectrogram into a square form, thereby enabling the storage of information.

[0229] The present invention can simulate breathing measured in various home environments by adding various noises generated in home environments to clean breathing. Because sounds have an additive nature, they can be added to each other. However, adding an original audio signal such as MP3 or PCM and converting it into a mel spectrogram can consume a lot of computing resources.

[0230] Therefore, the present invention proposes a method for converting breathing and noise into mel spectrograms and adding them to the mel spectrograms, thereby simulating breathing measured in various home environments and utilizing the simulated results for deep learning model training, thereby ensuring robustness in various home environments.

[0231] In the present invention, the sleep sound information 210 may be very quiet because it is related to sounds associated with breathing and body movements acquired during the user's sleep. Accordingly, the processor 130 or 830 can convert the sleep sound information into a spectrogram 300 to perform analysis on the sounds. In this case, as described above, the spectrogram 300 includes information indicating how the frequency spectrum of the sound changes over time, so that breathing or movement patterns associated with relatively quiet sounds can be easily identified, thereby improving the efficiency of the analysis.

[0232] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform the above-described operations.

[0233] According to an embodiment, each spectrogram may be configured to have a frequency spectrum with different density according to various sleep stages. Specifically, it may be difficult to predict whether a sleeper is in at least one of an awake state, a REM sleep state, a light sleep state, and a deep sleep state based solely on a change in the energy level of the sleep sound information. However, by converting the sleep sound information into a spectrogram, it may be possible to easily detect changes in the spectrum of each frequency, thereby enabling analysis corresponding to small sounds (e.g., breathing and body movements).

[0234] Furthermore, processor 130 or processor 830 may process spectrogram 300 as an input to a sleep analysis model to acquire sleep stage information. Here, the sleep analysis model is a model for acquiring sleep stage information related to changes in a user's sleep stage, and may input sleep acoustic information acquired during the user's sleep and output sleep stage information. In an embodiment, the sleep analysis model may include a neural network model configured via one or more network functions.

[0235] Network Functions and Neural Networks

[0236] In an embodiment of the present invention, the sleep analysis model may include a neural network model configured via one or more network functions. The sleep analysis model is configured by one or more network functions, and the one or more network functions may be configured by a set of interconnected computational units that may generally be referred to as "nodes." Such "nodes" may also be referred to as "neurons." The one or more network functions are configured to include at least one or more nodes. The nodes (or neurons) that configure the one or more network functions may be interconnected by one or more "links."

[0237] FIG. 9 is a schematic diagram illustrating one or more network functions associated with an embodiment of the present invention.

[0238] A deep neural network (DNN) can refer to a neural network that contains multiple hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to understand the latent structures of data.

[0239] That is, it is possible to grasp the latent structure of a photo, text, video, audio, or music (e.g., what object is in the photo, what is the content and emotion of the text, what is the content and emotion of the audio, etc.). The deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a generative adversarial network (GAN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q-network, a U-network, a Siamese network, etc. The above description of the deep neural network is merely an example, and the present invention is not limited thereto.

[0240] In one embodiment of the present invention, the network function may include an autoencoder, which may be a type of artificial neural network for generating output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between the input and output layers.

[0241] The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and may be reduced and expanded symmetrically from the bottleneck layer to the output layer (symmetric to the input layer). The nodes in the dimensionality reduction layer and the dimensionality restoration layer may be symmetric or asymmetric.

[0242] An autoencoder according to an embodiment of the present invention may perform nonlinear dimensionality reduction. The number of input layers and output layers may correspond to the number of sensors remaining after preprocessing of input data. The autoencoder structure may have a structure in which the number of nodes in hidden layers included in the encoder decreases with increasing distance from the input layer.

[0243] If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, it may not be able to convey a sufficient amount of information, so it may be kept above a certain number (e.g., more than half of the number in the input layer).

[0244] Neural networks can be trained using at least one of supervised learning, unsupervised learning, and semi-supervised learning. The purpose of training a neural network is to minimize the error in the output.

[0245] In neural network training, training data is repeatedly input into the neural network, the error between the neural network output and the target for the training data is calculated, and the error of the neural network is backpropagated from the output layer to the input layer of the neural network in a direction to reduce the error, thereby updating the weight values ​​of each node of the neural network.

[0246] In supervised learning, training data in which each training data is labeled with a correct answer (i.e., labeled training data) is used, while in unsupervised learning, each training data may not be labeled with a correct answer. That is, for example, in the case of supervised learning for data classification, the training data may be data in which each training data is labeled with a category.

[0247] Labeled training data is input to a neural network, and an error can be calculated by comparing the neural network's output (category) with the label of the training data. As another example, in unsupervised learning for data classification, the input training data can be compared with the neural network output to calculate the error.

[0248] The calculated error is backpropagated from the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated by backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate.

[0249] The neural network calculations on the input data and backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network learning cycle.

[0250] For example, in the early stages of neural network learning, a high learning rate can be used to quickly ensure a certain level of performance, thereby increasing efficiency, and in the later stages of learning, a lower learning rate can be used to increase accuracy.

[0251] In training a neural network, the training data may generally be a subset of the actual data (i.e., the data to be processed using the trained neural network), and therefore there may be training cycles in which the error decreases for the training data but increases for the actual data.

[0252] Overfitting is a phenomenon in which excessive learning from training data leads to an increase in errors in real data. For example, a neural network that has learned to recognize cats by showing them yellow cats may not be able to recognize cats when it sees a cat that is not yellow.

[0253] Overfitting can increase the error of machine learning algorithms. Various optimization methods can be used to prevent overfitting. To prevent overfitting, methods such as increasing the amount of training data, regularization, and dropout, which omits some nodes of the network during the training process, can be applied.

[0254] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably (hereinafter, they will be referred to as neural networks). The data structure may include a neural network.

[0255] The data structure including the neural network may then be stored on a computer-readable medium. The data structure including the neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and a loss function for training the neural network.

[0256] A data structure including a neural network may include any of the components disclosed above, such as data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, a loss function for training the neural network, and any combination thereof. In addition to the above-described components, a data structure including a neural network may include any other information that determines the characteristics of the neural network.

[0257] Furthermore, the data structure may include all types of data used or generated in the computational process of a neural network, and is not limited to the above. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a collection of interconnected computational units that may generally be referred to as nodes. Such nodes may also be referred to as neurons. A neural network is composed of at least one or more nodes.

[0258] Within a neural network, one or more nodes connected via links can form a relative relationship of input node and output node. The concepts of input node and output node are relative, and any node that is in an output node relationship with one node can also be in an input node relationship with another node, and vice versa.

[0259] As mentioned above, the relationship between input nodes and output nodes may be created around links. As shown in Figure 8, one input node may be connected to one or more output nodes via links, and vice versa.

[0260] In a relationship between an input node and an output node connected via a link, the value of the output node may be determined based on data input to the input node, and the node interconnecting the input node and the output node may have a weight.

[0261] The weights may be variable and may be varied by a user or an algorithm so that the neural network can perform a desired function. For example, if one or more input nodes are interconnected to one output node by respective links, the output node may determine its output node value based on the values ​​input to the input nodes connected to the output node and the weights set for the links corresponding to each input node.

[0262] As described above, a neural network has one or more nodes interconnected via one or more links to form input and output node relationships within the neural network. The characteristics of a neural network may be determined by the number of nodes and links within the neural network, the relationship between the nodes and links, and the weights assigned to each link.

[0263] For example, if two neural networks have the same number of nodes and links but different weights between the links, the two neural networks can be recognized as different from each other.

[0264] Some of the nodes that make up the neural network can be organized into a layer based on their distance from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers.

[0265] The distance from the initial input node can be defined as the minimum number of links that must be traversed to reach the node from the initial input node.

[0266] However, this definition of layers is arbitrary for illustrative purposes, and the number of layers within a neural network may be defined in a manner different from that described above. For example, a layer of nodes may be defined by its distance from the final output node.

[0267] The initial input node may refer to one or more nodes in a neural network to which data is directly input without passing through a link in relation to other nodes, or may refer to a node in a neural network that does not have other input nodes connected to it via a link in relation to nodes based on a link.

[0268] Similarly, a final output node may refer to one or more nodes in a neural network that do not have an output node relative to other nodes. A hidden node may refer to a node that constitutes a neural network and is neither the first input node nor the last output node. A neural network according to an embodiment of the present invention may have more nodes in the input layer than in the hidden layer closer to the output layer, and the number of nodes may decrease as one moves from the input layer to the hidden layer.

[0269] A neural network may include one or more hidden layers. Hidden nodes in a hidden layer can receive inputs from the outputs of previous layers and surrounding hidden nodes. The number of hidden nodes in each hidden layer may be the same or different.

[0270] The number of nodes in the input layer may be determined based on the number of data fields in the input data, and may be the same as or different from the number of hidden nodes. The input data input to the input layer can be operated on by the hidden nodes in the hidden layer and output by a fully connected layer (FCL), which is the output layer.

[0271] Feature extraction and classification models

[0272] According to one embodiment of the present invention, the sleep analysis model may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep stage information.

[0273] According to an embodiment, a feature extraction model can analyze the time series frequency patterns of the spectrogram 300 to extract features related to breathing sounds and patterns.

[0274] In one embodiment, the feature extraction model may be constructed as part of a neural network model (e.g., an autoencoder) that has been pre-trained via a training dataset, where the training dataset may comprise a plurality of spectrograms and a plurality of sleep stage information corresponding to each spectrogram.

[0275] In one embodiment, the feature extraction model may be constructed via a proprietary deep learning model (e.g., an autoencoder) trained via a training dataset. The feature extraction model may be trained via map learning or non-map learning methods. The feature extraction model may be trained via the training dataset to output output data similar to input data.

[0276] In more detail, only the core feature data (or features) of the spectrogram input through the encoder can be learned through the hidden layer, and the remaining information can be lost. In this case, the output data of the hidden layer during the decoding process through the decoder may be an approximation of the input data (i.e., the spectrogram) rather than a perfect copy. In other words, the autoencoder can learn to adjust the weights so that the output data and the input data are as similar as possible.

[0277] Each of the spectrograms included in the training data set may be tagged with sleep stage information, and each of the spectrograms may be input to an encoder, and the output corresponding to each spectrogram may be matched with the tagged sleep stage information and stored.

[0278] Specifically, when a first training data set (i.e., a plurality of spectrograms) tagged with first sleep stage information (e.g., light sleep) is used as input, features associated with the input can be matched and stored with the first sleep stage information. In an embodiment, one or more features associated with the output can be represented in a vector space.

[0279] In this case, the feature data output corresponding to each of the first training datasets may be located relatively close to each other in the vector space because they are output via spectrograms associated with the first sleep stage. That is, the encoder may be trained so that multiple spectrograms output similar features corresponding to each sleep stage.

[0280] In the case of an encoder, it can be trained to extract features that allow a decoder to well restore input data. Therefore, the trained autoencoder of the feature extraction model 410 can be implemented through an encoder to extract features (i.e., multiple features) that allow the input data (i.e., spectrogram) to be well restored.

[0281] The encoder that configures the feature extraction model 410 through the above-described learning process can extract features corresponding to a spectrogram (e.g., a spectrogram transformed corresponding to sleep acoustic information) as input.

[0282] In an embodiment, the processor 130 or the processor 830 may extract features by processing the spectrogram 300 generated corresponding to the sleep audio information 210 as an input of a feature extraction model. Here, since the sleep audio information 210 is time-series data acquired in a time-series manner during a user's sleep, the processor 130 or the processor 830 may divide the spectrogram 300 corresponding to the sleep audio information 210 into predetermined epochs. For example, the processor 130 or the processor 830 may divide the spectrogram 300 corresponding to the sleep audio information 210 into 30-second intervals to acquire a plurality of spectrograms. For example, if sleep audio information is acquired during a user's 7-hour (i.e., 420-minute) sleep, the processor 130 or the processor 830 may divide the spectrogram into 30-second intervals to acquire 840 spectrograms.

[0283] Alternatively, in the case of an embodiment such as that shown in Fig. 1(c), at least one of the electronic devices shown in Fig. 1(c) may perform at least one of the above-described operations. The specific numerical values ​​for the sleep time, the division time unit of the spectrogram, and the number of divisions are merely examples, and the present invention is not limited thereto.

[0284] According to an embodiment of the present invention, processor 130 or processor 830 processes each of the divided spectrograms as an input of a feature extraction model to extract a plurality of features corresponding to each of the plurality of spectrograms. For example, if the number of spectrograms is 840, the number of features extracted by the feature extraction model may also be 840. The specific numerical values ​​related to the number of spectrograms and features described above are merely examples, and the present invention is not limited thereto.

[0285] Additionally, processor 130 or processor 830 may process the plurality of features output through the feature extraction model with the input of a feature classification model to obtain sleep stage information. In an embodiment, the feature classification model may be a neural network model modeled to predict sleep stages corresponding to the features.

[0286] For example, the feature classification model may include a fully connected layer and classify features into at least one of the sleep stages. For example, when a first feature corresponding to a first spectrogram is input, the feature classification model may classify the first feature as light sleep.

[0287] The feature classification model may perform multi-epoch classification, which predicts sleep stages of various epochs by inputting spectrograms related to various epochs. The multi-epoch classification does not provide one piece of sleep stage analysis information corresponding to a spectrogram of a single epoch (i.e., one spectrogram corresponding to 30 seconds), but may input spectrograms corresponding to multiple epochs (i.e., a combination of spectrograms each corresponding to 30 seconds) to simultaneously estimate various sleep stages (e.g., changes in sleep stages over time).

[0288] For example, because breathing patterns change more slowly than electroencephalograms or other biological signals, accurate sleep stage estimation may be possible only by observing how the patterns change between past and future points in time. For example, a feature classification model may input 40 spectrograms (e.g., 40 spectrograms, each corresponding to 30 seconds) and perform prediction on the central 20 spectrograms. That is, by closely examining all spectrograms 1 to 40, sleep stages may be predicted through classification corresponding to spectrograms 10 to 20. The specific numerical values ​​for the number of spectrograms described above are merely examples, and the present invention is not limited thereto.

[0289] That is, in the process of estimating sleep stages, rather than performing sleep stage prediction for each single spectrogram, spectrograms corresponding to multiple epochs are used as input so that all information related to the past and future can be taken into consideration, thereby improving the accuracy of the output.

[0290] As described above, the processor 130 or the processor 830 may acquire a spectrogram based on the sleep acoustic information. In this case, the spectrogram may be converted to facilitate analysis of breathing or movement patterns associated with relatively small sounds. The processor 130 or the processor 830 may also generate sleep stage information based on the acquired spectrogram using a sleep analysis model including a feature extraction model and a feature classification model. In this case, the sleep analysis model may input spectrograms corresponding to multiple epochs to perform sleep stage prediction so that information related to both the past and future can be considered, thereby outputting more accurate sleep stage information.

[0291] That is, the processor 130 or the processor 830 may utilize the sleep analysis model as described above to output sleep stage information corresponding to the sleep acoustic information.

[0292] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform the above-described operations.

[0293] According to an embodiment, the sleep stage information may be information related to sleep stages that change during the user's sleep. For example, the sleep stage information may indicate whether the user's sleep changed to light sleep, normal sleep, deep sleep, REM sleep, etc. at each point during the user's eight hours of sleep last night. The specific description of the sleep stage information described above is merely an example, and the present invention is not limited thereto.

[0294] How we protect your privacy

[0295] FIG. 6(b) is a conceptual diagram illustrating a privacy protection method using mel-spectrogram conversion for sleep acoustic information extracted from a user in the sleep analysis method according to the present invention.

[0296] As shown in Figure 6(b), the audio information extracted from the user or the raw data of sleep audio information extracted therefrom undergoes a noise reduction preprocessing process, in which noise (e.g., white noise) contained in the raw data is removed.

[0297] The noise reduction process may be performed using algorithms such as spectral gating and spectral subtraction to remove background noise.

[0298] Furthermore, in the present invention, the noise removal process can be performed using a deep learning-based noise reduction algorithm. The deep learning-based noise reduction algorithm can be a noise reduction algorithm specialized for a user's breathing and respiratory sounds, in other words, a noise reduction algorithm learned through a user's breathing and respiratory sounds.

[0299] Then, the raw data from which noise has been removed is converted into a Mel-Spectrogram, which is a simplified vector sequence of the given input text in the frequency domain.

[0300] In this case, a method of generating a mel spectrogram based only on the amplitude obtained by removing the phase from raw data can be used, which not only protects privacy but also reduces data volume and improves processing speed. However, in other embodiments, a mel spectrogram can be generated using both the phase and amplitude.

[0301] In the present invention, a sleep analysis model is generated using the mel spectrogram 300 generated based on the sleep acoustic information 210. If the sleep acoustic information expressed as audio data were used as is, the amount of information would be very large, which would significantly increase the amount of calculation and the calculation time. Furthermore, since unwanted signals would be included, the calculation accuracy would be reduced. Furthermore, if all of the user's audio signals were transferred to the external server 20 or the AI ​​server 310, there would be a risk of privacy violation.

[0302] In the present invention, after noise is removed from sleep acoustic information using the above-mentioned method, the information is converted into a Mel spectrogram, and the Mel spectrogram is trained to generate a sleep analysis model, thereby reducing the amount of calculation and calculation time, and even protecting personal privacy.

[0303] In this case, the de-identification of sound data may be performed on natural language and respiratory sounds, which may be converted into natural language converted Mel spectrograms and respiratory sound converted Mel spectrograms, respectively. In the sleep analysis according to the present invention, only information required for the analysis model is used, thereby improving the calculation speed and reducing the calculation load.

[0304] Accuracy verification and embodiment of sleep analysis method

[0305] FIG. 34 is a table verifying the accuracy of the sleep analysis method according to the present invention, showing experimental result data analyzed according to age, sex, BMI, and presence or absence of disease.

[0306] FIG. 34 is a conceptual diagram for easy understanding showing a case where a smart speaker and a smartphone are used as one embodiment of the sleep analysis method according to the present invention.

[0307] Unlike the polysomnography test method used in hospitals, the sleep analysis method according to the present invention allows the lighting to be turned on / off during the test, and the room temperature and humidity to be freely adjusted.

[0308] In other words, sleep analysis can be performed conveniently and flexibly in various non-hospital environments using only the smart home appliance 800 and smartphone 900, which allows for verification in various real-world situations due to the ultra-low latency, going beyond verification in a fixed hospital environment.

[0309] As a result, as shown in Figure 34, experimental results showed consistently high accuracy even for subjects with a wide range of ages, genders, BMIs, sleep apnea, and limb movement disorders.

[0310] 34, for ease of understanding, the smart home appliance 800 is assumed to be a smart speaker, but is not limited thereto. That is, the smart home appliance 800 may be embodied as a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop personal computer (PC), a laptop personal computer, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, a wearable device (e.g., smart glasses, a head-mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic toy, a smart watch), a smart mirror, a kiosk, etc.

[0311] Furthermore, the smart home appliance 800 may be embodied as a smart home appliance such as a TV, a digital video disk (DVD) player, an audio system, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box, a game console, an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame, various medical devices, a home robot, or an internet of things device (e.g., a light bulb, various sensors, an electric or gas meter, a sprinkler system, a fire alarm, a thermostat, a street light, a toaster, exercise equipment, a hot water tank, a heater, a boiler, etc.). Furthermore, the smart home appliance 800 may be embodied as a piece of furniture or part of a building / structure, an electronic board, an electronic signature receiving device, a projector, etc., or may be a combination of one or more of the various devices mentioned above.

[0312] Alternatively, for example, in an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may correspond to one or more combinations of the various devices described above.

[0313] Therefore, the sleep analysis method according to the present invention enables a user's deep sleep to be analyzed conveniently and easily through a smart home appliance 800 such as a smartphone 900 or a smart speaker, regardless of time and place, even outside of a hospital.

[0314] Non-contact sleep analysis system configuration

[0315] FIG. 50 is a block diagram illustrating the operation of the AI-based non-contact sleep analysis system according to the present invention, which includes one or more smart home appliances 800, a sleep track app, an autonomous vehicle 801, and a living space 802.

[0316] FIG. 51 is a block diagram illustrating the operation between components of an AI-based non-contact sleep analysis system according to the present invention, and includes a smart home appliance 800, a smartphone 900, and an AI server 310.

[0317] As shown in FIG. 50, the smart home appliance 800 according to the present invention can perform more versatile and precise sleep analysis by acquiring the user's sleep acoustic information using a built-in microphone and performing sleep analysis (non-contact sleep analysis) using the acquired information.

[0318] In other words, it can verify various real-world situations beyond environmental verification such as hospital-based sleep testing, accurately detect not only insomnia but also sleep apnea and sleep hypopnea events in real time, and provide sleep diagnosis solutions for a wide range of ages, genders, races, BMIs, and the presence or absence of diseases.

[0319] 51, a smart home appliance 800 and a smartphone 900 work together to perform sleep analysis of a user. The smart home appliance 800 and the smartphone 900 may be paired via Bluetooth or other wireless communication methods.

[0320] The smartphone 900 may perform sleep analysis based on the sleep acoustic information of the user acquired from the smart home appliance 800.

[0321] At this time, the user's sleep sound information may be acquired from the smart home appliance 800 and transmitted to the smartphone 900, or may be acquired independently through a microphone built into the smartphone 900.

[0322] 51, the sleep stage analysis is performed in a non-contact manner via the smart home appliance 800 and the smartphone 900. The user can check the sleep stage analysis results derived from the smartphone 900 on the screen of the smartphone 900.

[0323] Thus, even if the user is not wearing the smart home appliance 800, the smart home appliance 800 needs to be appropriately positioned around the user to receive at least a portion of the input signal for the sleep analysis (e.g., body movement information) or the input signal for the sleep analysis (sleep acoustic information).

[0324] In particular, in order to extract information about body movements, it is preferable to place the sensor in an area where it can sense at least the user's movements (e.g., under the pillow, on top of the mattress, etc.).

[0325] On the other hand, since sound is transmitted in a radial direction, when only sleep sound information is used, it has the advantage that information can be collected and analyzed regardless of the user's position or the distance or angle between the user and the smart home appliance 800.

[0326] Therefore, the smart home appliance 800 of the present invention does not necessarily have to be worn by the user, but can perform the sleep stage analysis described above as long as it is properly placed within a predetermined radius (e.g., 4 to 5 m) in the user's sleep space, regardless of the user's position, distance, or angle to the user. Specific numerical values ​​for the radius are merely examples, and the present invention is not limited thereto.

[0327] In one embodiment, when the smart home appliance 800 is not worn by a user, the smart home appliance 800 may transmit a predetermined signal to prompt the user to place the smart home appliance 800 closer to the user so that the smart home appliance 800 can receive an input signal (sleep acoustic information) for sleep analysis. The predetermined signal may be a vibration, an alarm, a text, an LED, or the like.

[0328] The radius between the user and the smart home appliance 800 may be extracted by the smart home appliance 800 or by the smartphone 900.

[0329] That is, since the user's sleeping space is fixed, the location of the smart home appliance 800 can be tracked to determine whether the smart home appliance 800 is placed in an appropriate location.

[0330] Meanwhile, the smart home appliance 800 may be a sleep product (device) used for a user's sleep, rather than a wearable device. For example, a smart speaker may be used as one of the smart home appliances 800. The smart speaker may include an acoustic sensor therein to measure various acoustic information.

[0331] The smart speaker may perform primary sleep analysis using acoustic information acquired through an acoustic sensor. The smart speaker may be paired with the smartphone 900, and information measured by the smart speaker or primary sleep analysis results analyzed by the smart speaker may be transmitted to the smartphone 900. In this case, the smart speaker may include a communication module.

[0332] A smart mattress may be used as one of the smart home appliances 800. The smart mattress may include an acoustic sensor therein to measure various acoustic information.

[0333] The smart mattress can perform primary sleep analysis using acoustic information. The smart mattress can be paired with the smartphone 900 and transmit information measured by the smart mattress or primary sleep analysis results analyzed by the smart mattress to the smartphone 900. In this case, the smart mattress may include a communication module.

[0334] Meanwhile, the smart mattress may include various modules for adjusting temperature (temperature adjustment module, infrared irradiation module, cooling module), and the temperature may be adjusted based on the final sleep stage analysis result, which improves the quality of the user's sleep.

[0335] Meanwhile, the smart speaker or smart mattress mentioned above may include a vibration module or an alarm module to alleviate and improve sleep disorders, as described below. That is, if sleep apnea, snoring, sleep hyperpnea, REM sleep, or the like is detected, the vibration module or alarm module of the smart speaker or smart mattress can be activated to transmit tactile or auditory stimulation to the user.

[0336] In addition, in an autonomous vehicle 801 or a recently constructed residential space 802, one or more smart devices can be linked to the Sleep Track app to build and operate an AI-based non-contact sleep analysis system according to the present invention.

[0337] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0338] The above-described types of smart home appliances and spaces are merely examples, and the present invention is not limited thereto.

[0339] Smart home appliances in a sleep analysis system

[0340] FIG. 11(b) is a block diagram showing the configuration of smart home appliances in the AI-based non-contact sleep analysis system according to the present invention.

[0341] The smart home device 800 according to the present invention includes a communication unit 810, a sensor unit 820, a processor 830, a memory 840, and an alarm unit 850. Various components for performing other functions of the smart home device 800 may also be included.

[0342] That is, depending on the implementation of the embodiments of the present invention, additional components may be included, or some of the components may be omitted, or two or more components may be integrated into one component.

[0343] The communication unit 810 transmits and receives data to and from the smartphone 900 and the AI ​​server 310 via a wireless communication network. The wireless communication network may include short-range wireless communication networks such as Z-wave, Zigbee (registered trademark), Wi-Fi (wifi), Bluetooth (registered trademark), LTE-M, LoRa (long range), Narrowband Internet of Things (NB-IoT), and Infrared Data Association (IrDA). Furthermore, the wireless communication network may include, but is not limited to, a 2G mobile communication network such as a wireless LAN (WLAN), wireless broadband (Wibro), wireless fidelity (Wifi), WiMax (world interoperability for microwave access), a global system for mobile communication (GSM) or code division multiple access (CDMA), a 3G mobile communication network such as a wideband code division multiple access (WCDMA) or CDMA2000, a 3.5G mobile communication network such as a high speed downlink packet access (HSDPA) or high speed uplink packet access (HSUPA), a 4G, 5G, or 6G mobile communication network such as a long term evolution (LTE) network or an LTE-Advanced network.

[0344] The sensor unit 820 may include a microphone module for extracting sleep acoustic information of the user. The microphone module may be configured with MEMS (Micro-Electro-Mechanical Systems) for application to small devices. Such a microphone module can be manufactured in a very small size and may have a much lower SNR (Signal-Noise Ratio) than a condenser microphone or a dynamic microphone.

[0345] In this case, the sleep acoustic information is information on acoustic signals during sleep, and interacts closely with sleep itself, so that it can be acquired without the need to wear a separate wearable device such as a smart watch or smart ring.

[0346] The sensor unit 820 may include an air pressure sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a temperature sensor, a humidity sensor, and an illuminance sensor.

[0347] The memory 840 may store a computer program for performing sleep analysis, and the stored computer program may be read and executed by the processor 830, which will be described later. The memory 840 may also store any type of information generated or determined by the processor 830 and any type of information received by the communication unit 810. The memory 840 may also store data related to the user's sleep.

[0348] For example, memory 840 may provide temporary or permanent storage of input / output data.

[0349] The memory 840 may be embodied as at least one type of storage medium including, but not limited to, a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0350] The computer program may include one or more instructions that, when loaded into memory 840, cause processor 830 to perform methods / operations according to various embodiments of the present invention. That is, processor 830 may perform methods / operations according to various embodiments of the present invention by executing one or more instructions.

[0351] The processor 830 may be configured with one or more cores and may include a central processing unit (CPU) of a smart home device, a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), or other processors for data analysis and deep learning.

[0352] The processor 830 can read a computer program stored in the memory 840 to perform data processing for machine learning according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 830 can perform calculations for neural network training.

[0353] The processor 830 can perform calculations for neural network training, such as processing input data for training in deep learning (DL), extracting features from the input data, calculating errors, and updating neural network weights using backpropagation.

[0354] In addition, at least one of the CPU, GPGPU, and TPU of the processor 830 can process the learning of the network function.

[0355] For example, a CPU and a GPGPU can both process network function learning and data classification using the network function. Also, in one embodiment of the present invention, processors of multiple smart home appliances can be used together to process network function learning and data classification using the network function.

[0356] Furthermore, the computer program executed by the smart home device according to an embodiment of the present invention may be a CPU, GPGPU, or TPU executable program.

[0357] A network function may be used interchangeably with an artificial neural network or a neural network. A network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of the one or more neural networks.

[0358] A model (inference model) may include a network function. A model may include one or more network functions, in which case the output of the model may be an ensemble of the outputs of the one or more network functions.

[0359] The processor 830 may provide a sleep analysis model according to an embodiment of the present invention by reading a computer program stored in the memory 840. According to an embodiment of the present invention, the processor 830 may perform sleep analysis of a user based on sleep acoustic information using the sleep analysis model.

[0360] In other words, a user's breathing during sleep contains a lot of information for analyzing sleep, not only about body movements and breathing sounds during sleep, but also about various sleep disorders (e.g., sleep apnea, sleep hypopnea, snoring), etc., so when artificial intelligence (AI) is used, high accuracy can be expected.

[0361] As shown in Figure 32(b), during the sleep stage, the user's breathing pattern and regularity, sounds of movement during sleep, and respiration may be measured, including recovery sounds after an apnea event and unstable breath sounds between hypopnea events.

[0362] Furthermore, if the frequency patterns of breathing sounds are analyzed, it will be possible to make fundamental predictions about the causes of snoring and sleep apnea.

[0363] In particular, sleep respiratory sounds are the breathing of a user while sleeping, and as shown in Figure 35, this information can be conveniently measured outside of a hospital via various smart home appliances 800 such as a smartphone 900 or a smart speaker.

[0364] The processor 830 may perform calculations for training a sleep analysis model. Based on the sleep analysis model, sleep information related to the user's sleep stages, sleep quality, occurrence of sleep disorders, etc. may be inferred. Sleep acoustic information acquired from the user in real time or periodically is input as an input value to the sleep analysis model, which outputs data related to the user's sleep (data related to sleep stages, sleep quality, occurrence of sleep disorders, etc.).

[0365] Meanwhile, the smart home appliance 800 according to the present invention may further include an alarm unit 850. The alarm unit 850 is a means for providing tactile or auditory feedback to the user when a sleep disorder such as sleep apnea occurs during primary and secondary sleep analysis.

[0366] For example, the alarm unit 850 may be implemented as an actuator, a vibration module, or a haptic module that generates vibrations, or may be implemented as a speaker module that generates sounds.

[0367] Meanwhile, in the present invention, the sleep state information may be information related to whether or not the user is asleep. Specifically, the sleep state information may include at least one of first sleep state information indicating that the user is before sleeping, second sleep state information indicating that the user is sleeping, and third sleep state information indicating that the user has fallen asleep.

[0368] In other words, if first sleep state information is inferred in relation to a user, processor 830 can determine that the user is in a pre-sleep (i.e., pre-bedtime) state, if second sleep state information is inferred, processor 830 can determine that the user is in a sleeping state, and if third sleep state information is obtained, processor 830 can determine that the user is in a post-sleep (i.e., wake-up) state.

[0369] The sleep state information may be acquired based on environmental sensing information, which may be sensing information acquired in a space where the user is located in a non-contact manner.

[0370] For example, the processor 830 can extract sleep state information based on environmental sensing information acquired by the sensor unit 820 (acoustic information related to cleaning, acoustic information related to cooking food, acoustic information related to watching TV, sleep acoustic information acquired during sleep, etc.).

[0371] In this case, the sleep sound information acquired during the user's sleep may include sounds generated when the user turns over in sleep, sounds related to muscle movements, breathing sounds during sleep, etc. That is, the sleep sound information in the present invention may refer to sound information related to the breathing pattern associated with the user's sleep.

[0372] Sleep stages may be classified into NREM (non-REM) sleep and REM (rapid eye movement) sleep, and NREM sleep may be further classified into multiple stages (e.g., two stages of Light and Deep, and four stages of N1 to N4). The sleep stages may be defined based on commonly used sleep stages, or may be arbitrarily set by a designer in various ways.

[0373] Through sleep stage analysis, it is possible to predict not only sleep quality but also sleep disorders (e.g., sleep apnea) and their underlying causes (e.g., snoring).

[0374] The processor 830 may acquire sleep state information based on the acoustic information acquired from the smart home appliance 800. Specifically, the processor 830 may identify a singular point where information of a pre-established pattern is detected in the acoustic information.

[0375] Here, the preset pattern information may be related to sleep-related breathing patterns, for example, in a wakeful state, the entire nervous system is activated, leading to irregular breathing patterns and frequent body movements.

[0376] In addition, breathing noise may be very quiet because the neck muscles are not relaxed. On the other hand, when the user is asleep, the autonomic nervous system stabilizes, breathing becomes regular, and breathing noise may become louder.

[0377] That is, the processor 830 may identify a point in time when acoustic information of a predetermined pattern associated with regular breathing, small breathing sounds, etc. is detected as a singular point in the acoustic information. In addition, the processor 830 may acquire sleep acoustic information based on the acoustic information acquired with reference to the identified singular point.

[0378] The processor 830 may identify a specific point associated with the user's sleep time point from the time-series acquired acoustic information, and acquire sleep acoustic information based on the specific point.

[0379] Furthermore, according to one embodiment of the present invention, for example, in an embodiment such as (c) of FIG. 1, at least one of the electronic devices shown in (c) of FIG. 1 may perform at least one of the above-described operations.

[0380] Comparison of conventional sleep analysis methods with the sleep analysis method of the present invention

[0381] FIG. 45 is a conceptual diagram showing a training method using only microphone data S from a multi-dimensional sleep test in a hospital environment according to a conventional sleep analysis method, in order to compare the sleep analysis method of the present invention with the conventional technology.

[0382] FIG. 46 is a conceptual diagram of a method for generating an AI sleep analysis model by incorporating various sounds in a home environment in accordance with the sleep analysis method of the present invention into the training method shown in FIG.

[0383] Here, waveform (a) is the waveform of sleep multivariate test microphone data S in a hospital environment, waveform (b) is the waveform of various noise data N generated in a home environment, and waveform (c) is a waveform that combines waveforms (a) and (b).

[0384] FIG. 47 is a table verifying the performance of the sleep analysis method according to the present invention after training by dividing the subjects into nine groups according to the type of residential noise, and shows experimental result data tested on groups 1 to 9 (group 0 to group 8).

[0385] The types of residential noise are as follows: Group 1 is the sound of rain and wind; Group 2 is the sound of electric fans and air conditioners; Group 3 is the sound of TVs, telephones, and video recorders; Group 4 is the sound of cars, motorcycles, and other vehicles; Group 5 is the sound of clocks; Group 6 is the sound of people talking and voices; Group 7 is the sound of electronic products; Group 8 is noise between rooms / floor areas; and Group 9 is the sound of pets.

[0386] As shown in Figure 45, in a conventional training method using only sleep multivariate test microphone data S in a hospital environment, the sleep multivariate test microphone data S collected at the hospital is input and output through a first AI sleep analysis model, and sleep analysis and diagnosis labels reflecting classification loss are generated and fed back.

[0387] On the other hand, the training method when using home multi-dimensional sleep test microphone data H is as follows.

[0388] First, as shown in FIG. 46, various noise data N generated in a home environment are combined with the sleep polymorphism test microphone data S used in the training method (a) in which only the sleep polymorphism test microphone data S is used in a conventional hospital environment and input.

[0389] If this combined data S+N is input and output through a second AI sleep analysis model, a consistency loss occurs.

[0390] When this consistency loss is added to the classification loss generated by the training method shown in Figure 20, a third AI sleep analysis model is generated.

[0391] At this time, the first and second AI sleep analysis models add correlation between each other's output data.

[0392] Comparison of user 24-hour monitoring processors and mean results per class

[0393] FIG. 48 is a schematic diagram illustrating a 24-hour monitoring process of a user using an AI-based non-contact sleep analysis system and sleep analysis method according to the present invention.

[0394] FIG. 49 is a table showing the mean per class results of comparing the smart home appliances and sleep analysis method according to the present invention with products and devices from existing world-leading sleep tech companies.

[0395] In the past, when analyzing a user's activity, rest, sleep, and other patterns using only a conventional smartwatch, there was a problem that the sleep analysis was interrupted if the smartwatch was removed during sleep. The present invention uses a smartphone 900 linked to a smart home appliance 800 to seamlessly monitor all of a user's activities in real time, even when the smartwatch is removed during sleep.

[0396] For example, by automatically activating the smartphone 900 when the smartwatch is removed, plugged into a charger, placed on a charging pad, etc., it is possible to continuously analyze the user's activity, rest, sleep, etc. In this case, the smartphone 900 can be activated when it is time to sleep even when not adjacent to the smart home appliance 800.

[0397] In this manner, continuity of measurement of the user's activities, including sleep, can be ensured. For example, as shown in Fig. 48, 24-hour data can be obtained via the smartphone 900. The data can then be processed into various reports and provided to the user.

[0398] The user touches the screen of the smartphone 900 to start recording sleep, and receives a sleep analysis result report (such as bedtime, time to fall asleep, sleep duration, and time taken to wake up after the alarm) analyzed in the manner described above. Alarms (such as alarms with gradually increasing volume according to individual sleep stages) may be automatically generated according to the sleep stage, and all-day care services such as user profiling (such as sleep information, preferred content, and content recommendations by age group / gender / occupation group) and recommendations for customized sleep / exercise / diet / cosmetics / behavioral guidelines optimized for individual sleep patterns may be provided.

[0399] The present invention displays weight / blood pressure and sleep apnea, insomnia, or exercise and insomnia as a sleep measurement record, which can motivate users to change their behavior to improve their health. In other words, the present invention can improve the user's compliance with behavioral changes in a very natural way.

[0400] For example, if a user is overweight, sleep apnea is not uncommon, and weight loss can help reverse sleep apnea, so the present invention can be integrated with diet, exercise, and weight tracking in a healthcare app.

[0401] That is, the sleep apnea history allows for real-time sleep apnea detection and accurate behavioral intervention of the present invention.

[0402] Furthermore, since sleep apnea can cause high blood pressure, if periods of unstable breathing are regular, the present invention can be used to track and manage blood pressure.

[0403] In other words, weight loss helps lower blood pressure in the human body, so when weight loss is successful, the PSQI can be used to objectively compare sleep quality before and after.

[0404] Additionally, exercise (except within three hours before bedtime) helps alleviate insomnia, so outdoor activities increase the amount of time spent in natural light, improving the user's mood.

[0405] It will also be possible to receive recommendations for various exercise programs from the health care app.

[0406] In addition, the present invention can show the user correlations such as stress levels and sleep, or premenstrual syndrome and insomnia, which can help the user to re-evaluate their own health condition.

[0407] In other words, by showing the correlation between stress levels and sleep quality, it adds an interesting element and makes it possible to create psychiatric questionnaires to be provided in healthcare apps based on the user's stress levels and level of depression.

[0408] In addition, if a user complains of insomnia as one of the symptoms of premenstrual syndrome, the calendar in the menstrual cycle tracking function can be used to compare sleep data and record sleep efficiency, allowing the user to receive a health check related to their physiological phenomena.

[0409] Meanwhile, one of the important aspects of sleep stage analysis is to determine whether a user wakes up during sleep or whether a true wake-up has occurred. In other words, the WAKE stage, which is the stage at which a user wakes up, must be analyzed properly, and sleep sound signals are a very useful factor in detecting whether the user is in the true WAKE stage.

[0410] While conventional EEG measurement in sleep multidimensional analysis merely confirms changes in EEG when the user is awake, the sleep audio signal used in the sleep stage analysis of the present invention shows precursor signals (sound patterns, movement patterns, etc.) before the user wakes up (before reaching the WAKE stage), and through this, the WAKE stage can be predicted and detected.

[0411] The AI ​​sleep stage analysis model, trained through a large amount of data, will be able to more accurately determine the wake stage, especially based on sleep sound signals. Furthermore, whether a user wakes up from sleep is influenced by their body's biorhythm or external factors (such as ambient noise), the model can also be used to determine the wake stage.

[0412] The present invention builds an AI sleep stage analysis model by learning the user's sleep environment, i.e., various ambient noises such as routine noises and abnormal or intermittent noises in the surrounding space, thereby enabling more clear and reliable prediction and detection of the wake stage.

[0413] In fact, as shown in Figure 49, when compared with solutions from existing world-leading sleep tech companies, the wake accuracy was calculated to be 43% higher than existing wearables and 52% higher than existing contactless systems, and the average wake / sleep accuracy was calculated to be 16% higher than existing wearables and 20% higher than existing contactless systems.

[0414] In addition, the average accuracy of Wake / NREM / REM (3C) was calculated to be 15% higher than existing wearable devices and 25% higher than existing contactless devices.

[0415] In addition, the sleep analysis of the present invention using sleep acoustic information is highly versatile and can be applied to a variety of devices, as anyone can perform sleep analysis as long as they have a device including a microphone.

[0416] The sleep analysis method, sleep disorder alleviation and prevention method, sleep disorder improvement method, and sleep monitoring method according to the present invention may be provided by a server that provides a cloud computing service. More specifically, the sleep analysis method, sleep disorder alleviation and prevention method, sleep disorder improvement method, and sleep monitoring method according to the present invention may be performed by a server that provides a cloud computing service, which is a type of Internet-based computing and processes information on a computer connected to the Internet other than the user's computer.

[0417] That is, in the embodiment shown in Figures 50 and 51, various sleep acoustic information acquired by the smart home appliance 800 and the smartphone 900 is transmitted to the AI ​​server 310, and the AI ​​server 310 performs sleep analysis using the information and then transmits the results back to the smart home appliance 800 and the smartphone 900.

[0418] According to another embodiment of the present invention, various sleep acoustic information acquired by the smartphone 900 may be converted into a spectrogram by the smartphone 900 and transmitted to the AI ​​server 300. In this case, the AI ​​server 310 may perform sleep analysis using the spectrogram.

[0419] According to yet another embodiment of the present invention, various sleep acoustic information acquired by the smartphone 900 is converted into a spectrogram by the AI ​​server 310, and the AI ​​server 310 can perform sleep analysis using the spectrogram.

[0420] A cloud computing service may be a service that stores materials on the Internet and allows users to access them anytime and anywhere via an Internet connection without having to install the necessary materials or programs on their own computers, and allows materials stored on the Internet to be easily shared and transmitted with simple operations and clicks. A cloud computing service may not only simply store materials on an Internet server, but also allow users to perform desired tasks using application functions provided on the web without installing additional programs, and may allow various people to share documents and work on them simultaneously.

[0421] In addition, the cloud computing service may be implemented in at least one form of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, or a container-based cloud server. That is, the smart home appliance 800 of the present invention may be implemented in at least one form of the above-mentioned cloud computing services. The specific descriptions of the above-mentioned cloud computing services are merely examples, and may include any platform that builds a cloud computing environment of the present invention.

[0422] The sleep analysis method, sleep disorder alleviation and prevention method, sleep disorder improvement method, and monitoring method according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention, or those publicly known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0423] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0424] Generating environmental creation information

[0425] According to an embodiment of the present invention, the processor 130 or the processor 830 may generate environment creation information based on the sleep state information and / or the sleep stage information.

[0426] The sleep state information is information related to whether the user is sleeping or not, and may include at least one of first sleep state information indicating that the user is about to fall asleep, second sleep state information indicating that the user is sleeping, and third sleep state information indicating that the user has fallen asleep. Hereinafter, the step of generating environment creation information will be described in detail using the processor 130 as an example.

[0427] According to the embodiment, the processor 130 may generate the first environment creation information based on the first sleeping state information. Specifically, when the processor 130 acquires the first sleeping state information indicating that the user is before sleeping, the processor 130 may generate the first environment creation information based on the first sleeping state information.

[0428] According to an embodiment, the first environment creation information may be information regarding light intensity and illuminance for inducing natural sleep. Specifically, the first environment creation information may be control information for supplying 3000K white light at an illuminance of 30 lux from a sleep induction point until the second sleep state information is acquired.

[0429] According to an embodiment, the sleep induction time point may be determined by the processor 130. Specifically, the processor 130 may determine the sleep induction time point through information exchange with the user's user terminal 10. For a specific example, the user may set the time point at which they intend to fall asleep via the user terminal 10 and transmit the time point to the processor 130. The processor 130 may determine the sleep induction time point based on the time point at which the user intends to fall asleep from the user terminal 10. For example, the processor 130 may determine the sleep induction time point to be 20 minutes before the time point at which the user intends to fall asleep. For a specific example, if the time point at which the user intends to fall asleep set by the user is 11:00, the processor 130 may determine 10:40 as the sleep induction time point. The specific numerical values ​​for the above-mentioned time points are merely examples, and the present invention is not limited thereto.

[0430] According to an embodiment, the processor 130 may acquire the user's sleep intention information based on the environmental sensing information and determine a sleep induction time point based on the sleep intention information. The sleep intention information may be information that quantitatively indicates the user's intention to sleep. For example, the higher the user's sleep intention, the closer to 10 the calculated sleep intention information may be, and the lower the user's sleep intention, the closer to 0 the calculated sleep intention information may be.

[0431] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0432] The specific numerical values ​​for the sleep intention information described above are merely examples, and the present invention is not limited thereto.

[0433] Acquisition of sleep intention information

[0434] The processor 130 or the processor 830 may acquire the sleep intention information based on the environmental sensing information. Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may acquire the sleep intention information. Hereinafter, the step of acquiring the sleep intention information will be described in detail using the processor 130 as an example.

[0435] According to an embodiment, processor 130 may identify types of sounds included in the environmental sensing information. Processor 130 may calculate sleep intention information based on the number of identified types of sounds. Processor 130 may calculate a lower sleep intention information as the number of types of sounds increases, and may calculate a higher sleep intention information as the number of types of sounds decreases. For example, if there are three types of sounds included in the environmental sensing information (e.g., the sound of a vacuum cleaner, the sound of a TV, and a user's voice), processor 130 may calculate the sleep intention information as 2 points. Also, if there is one type of sound included in the environmental sensing information (e.g., a washing machine), processor 130 may calculate the sleep intention information as 6 points. The specific numerical values ​​related to the types of sounds included in the environmental sensing information and the sleep intention information described above are merely examples, and the present invention is not limited thereto.

[0436] That is, the processor 130 may acquire sleep intention information related to the user's intention to sleep according to the number of types of sounds included in the environmental sensing information. For example, the more types of sounds are identified, the lower the sleep intention information indicating the user's low sleep intention (i.e., sleep intention information with a low score) may be output.

[0437] In addition, in an embodiment, the processor 130 may pre-match different intention scores for each of a plurality of pieces of sound information to generate or record an intention score table. For example, first sound information related to a washing machine may be pre-matched with an intention score of 2, second sound information related to a humidifier may be pre-matched with an intention score of 5, and third sound information related to voice may be pre-matched with an intention score of 1. The processor 130 may pre-match relatively high intention scores for sound information related to the user's sleep (e.g., sounds generated by the user's activities, such as vacuuming, washing dishes, and voices) and relatively low intention scores for sound information unrelated to the user's sleep (e.g., sounds unrelated to the user's activities, such as vehicle noise and the sound of falling rain) to generate the intention score table. The specific numerical values ​​for the intention scores matched to each piece of sound information described above are merely examples, and the present invention is not limited thereto.

[0438] The processor 130 may acquire sleep intention information based on the environmental sensing information and the intention score table. Specifically, the processor 130 may record an intention score corresponding to a time point at which at least one of a plurality of sounds included in the intention score table is identified in the environmental sensing information, and the intention score may be matched to the identified sound. For example, if the sound of a vacuum cleaner is identified corresponding to a first time point in the process of acquiring the environmental sensing information in real time, the processor 130 may match two intention scores to the sound of the vacuum cleaner and record the matched score at the first time point. Each time various sounds are identified in the process of acquiring the environmental sensing information, the processor 130 may match the intention score to the identified sound and record the matched score at the corresponding time point.

[0439] In an embodiment, the processor 130 may acquire sleep intention information based on the sum of intention scores acquired during a predetermined time period (e.g., 10 minutes). For example, the higher the intention score acquired during the 10 minutes, the higher the sleep intention information may be acquired, and the lower the intention score acquired during the 10 minutes, the lower the sleep intention information may be acquired. The specific numerical values ​​for the predetermined time period described above are merely examples, and the present invention is not limited thereto.

[0440] That is, the processor 130 may acquire sleep intention information related to the user's intention to sleep according to the characteristics of the sound included in the environmental sensing information. For example, the more sounds related to the user's activity are identified, the lower the sleep intention information indicating the user's low sleep intention (i.e., sleep intention information with a low score) may be output.

[0441] Determining environmental information and operating smart home appliances

[0442] According to an embodiment of the present invention, the processor 130 or the processor 830 may determine the environment creation information based on the sleep state information and / or the sleep intention information.

[0443] In addition, various smart home appliances 800 according to embodiments of the present invention can be operated based on the environment creation information.

[0444] Alternatively, in the case of an embodiment such as that shown in Fig. 1(c), at least one of the electronic devices shown in Fig. 1(c) may perform at least one of the above-described operations. Hereinafter, the determination of environment creation information and the operation of smart home appliances will be described in detail with reference to the drawings.

[0445] Overall Operation

[0446] FIG. 8 illustrates an exemplary flow chart for providing a method for creating a sleep environment based on sleep state information according to an embodiment of the present invention.

[0447] According to one embodiment of the present invention, the method may include a step of acquiring sleep state information of a user (S100).

[0448] According to an embodiment of the present invention, the method may include generating environment creation information based on sleep state information (S200).

[0449] According to an embodiment of the present invention, the method may include a step of transmitting environment creation information to the environment creation device 30 (S300).

[0450] 8 may be reordered, and at least one step may be omitted or added, as necessary. That is, the above-described steps are merely one embodiment of the present invention, and the scope of the present invention is not limited thereto.

[0451] FIG. 39 is a flowchart illustrating the operation of the AI-based non-contact sleep analysis method according to the present invention.

[0452] FIG. 40 is a flow chart showing various embodiments of smart home appliances that can be used in the sleep analysis method according to the present invention.

[0453] The overall operation of the AI-based non-contact sleep analysis method according to the present invention will be briefly described below with reference to FIGS. 50, 51, and 39.

[0454] A sleep analysis app can be downloaded to the smartphone 900 (S1000).

[0455] At least one smart home appliance 800 can collect the user's sleep sound information in real time and transmit it to the server 310 (S2000).

[0456] The smartphone 900 can simultaneously collect the user's sleep acoustic information in real time and transmit it to the server 310 (S3000).

[0457] The server 310 may transmit the AI-trained sleep analysis result report to the smartphone 900 (S4000).

[0458] The smartphone 900 may output a control signal for controlling the operation of at least one smart home appliance 800 (S5000).

[0459] At least one smart home appliance 800 can provide a customized sleep environment to the user (S6000).

[0460] Next, the detailed operation of the AI-based non-contact sleep analysis method according to the present invention will be described with reference to FIGS. 50, 51, and 40.

[0461] First, it can be determined whether the smart home appliance 800 has a built-in microphone (S7000).

[0462] If the answer is yes, the sleep analysis app (hereinafter referred to as the sleeptrack app) according to the present invention can be downloaded to the smartphone 900 (S7100), and if no, the sleeptrack app can be linked to apps already installed on the smartphone 900 (S7200).

[0463] Here are the features of the Sleep Track app:

[0464] As a sleep analysis app that detects users' real-time sleep stages and breathing instability periods, it uses a database that can store weekly and monthly sleep quality indicators and sleep environment, and a dashboard that can derive service insights through usage sessions, sleep statistics, etc., to calculate a highly accurate overnight sleep stage graph (hypnogram), sleep evaluation indicators, and breathing instability indicators.

[0465] In addition, the SleepTrack app allows for seamless monitoring and data collection between daily life and sleep in a contactless manner without the need for a separate wearable device.

[0466] This not only increases the degree of freedom of the body during sleep, but also allows for accurate timing of the wake time, which is the basis of all sleep therapy, and allows for convenient and accurate analysis of various types of user sleep at home, regardless of time or place.

[0467] The Sleep Track app can be used for the following purposes:

[0468] It not only provides an individual sleep pattern analysis report for each user to create an optimal sleeping environment based on the sleep analysis results by intervening in the user's sleep based on real-time sleep tracking, but also provides alarms tailored to individual sleep stages, sleep hygiene guides, and sound content for falling asleep / waking up.

[0469] In addition, it can recommend content that can help create behavioral correction and sleep routines, such as customized exercise and eating patterns, optimized for individual sleep patterns, as well as user profiles such as user-specific sleep information, preferred content, sleep BTI, and responsiveness to recommended content, according to age group, gender, and occupation.

[0470] Meanwhile, in step S7100, it is determined whether the smart home appliance can create a sleep environment (S8000). Here, the sleep environment may include temperature, humidity, light, sound, head and body position, scent, etc.

[0471] In step (S8000), if the answer is yes, the SleepTrack app may be activated and a research interaction may be generated (S810), and if no, it may be determined whether the device is capable of providing customer value, i.e., data, based on sleep analysis via various user interfaces (e.g., PUI, VUI, and / or GUI) (S9000).

[0472] If the result of step S9000 is affirmative, the sleep track application is activated (S9100), and if the result is negative, the operation may be terminated since the introduction of the sleep track application is meaningless.

[0473] For example, smart home appliances that reach step (S8100) may include air conditioners and / or air purifiers that adjust temperature, humidifiers and / or dehumidifiers that adjust humidity, blinds and / or curtains that adjust light, smart speakers that adjust lights and sounds, smart beds that adjust the position of the user's head and body, smart diffusers that adjust fragrances, smart devices with healthcare apps installed, etc.

[0474] In addition, smart home appliances that reach step (S9100) may include TVs, clothing management machines, robot vacuum cleaners, washing machines and / or dryers, refrigerators, smart appliances with healthcare apps installed, etc.

[0475] In addition, application fields that can reach "sleep management app interaction" other than step (S8100) and step (S9100) may be industries related to fragrance, cosmetics, health functions, traditional sleep industry, sports, hotels, cram schools, fire departments, and government agencies.

[0476] At this time, a "sleep management app" refers to a type of sleep management app that can perform sleep analysis without a hardware solution.

[0477] In addition, the "sleep track app" may refer to a sleep analysis app that transmits a user's sleep report to the user's smartphone 900 in real time via a PUI, VUI, and / or GUI, and operates the smart home appliance 800 according to the report results.

[0478] The steps shown in the above drawings may be reordered as necessary, and at least one step may be omitted or added. That is, the above steps are merely one embodiment of the present invention, and the scope of the present invention is not limited thereto.

[0479] Hereinafter, the step of determining the environment creation information will be described in detail by dividing it into sleep states and sleep stages using the processor 130 as an example. Also, the step will be described in detail by taking an example of a smart home appliance 800 that operates according to the environment creation information. However, the present invention is not limited to the following embodiments.

[0480] Determining the time point for sleep induction

[0481] According to an embodiment, the processor 130 may determine the sleep induction time point based on the sleep intention information. Alternatively, in an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may determine the sleep induction time point. Specifically, the processor 130 may identify a time point at which the sleep intention information exceeds a predetermined threshold as the sleep induction time point. That is, when high sleep intention information is obtained, the processor 130 may identify this as a time point suitable for sleep induction, i.e., a sleep induction time point.

[0482] As described above, the processor 130 may determine the sleep induction time point of the user. According to an embodiment, when the processor 130 acquires first sleep state information indicating that the user is about to fall asleep, the processor 130 may generate first environment creation information (supplying 3000K white light at an illuminance of 30 lux) for adjusting light based on the sleep induction time point until the time when the second sleep state information is acquired.

[0483] First environment creation information and smart home appliance operation based on pre-sleep state

[0484] According to one embodiment of the present invention, when the user's state is a pre-sleep state, the processor 130 may generate first environment creation information that adjusts light from the time when the user is predicted to prepare for sleep (e.g., the sleep induction time) to the time when the user falls asleep (i.e., the time when the second sleep state information is acquired), and may decide to transmit the first environment creation information to the environment creation device 30.

[0485] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0486] This allows 3000K white light to be supplied at 30 lux from 20 minutes before the user falls asleep (for example, the sleep induction point) until the moment the user falls asleep. This light is excellent for secreting melatonin before the user falls asleep, and by inducing natural sleep, it can improve the user's sleep efficiency.

[0487] According to an embodiment, when the user is in a pre-sleep state, the processor 130 may generate first environment creation information for controlling smart home appliances from a time when the user is predicted to prepare for sleep (e.g., a sleep induction time) to a time when the user falls asleep (i.e., a time when the second sleep state information is acquired). Specifically, the first environment creation information may be generated, such as removing fine dust and harmful gases in advance until a predetermined time (e.g., 20 minutes) before the user falls asleep, or controlling the room temperature and humidity for falling asleep. The first environment creation information may also include information for controlling smart home appliances to induce noise (white noise) sufficient to induce sleep just before falling asleep, adjusting the airflow intensity of smart home appliances such as air purifiers or air conditioners to a level below a preset intensity, lowering the intensity of LEDs, or switching from direct airflow to indirect airflow. The first environment creation information may also include information for controlling smart home appliances to dehumidify / humidify based on temperature and humidity information within the sleeping space. In addition, the first environment creation information may include control information for adjusting personalized temperature, humidity, airflow strength, noise, etc. according to the operation history of smart home appliances such as air purifiers or air conditioners and the acquired sleep state (sleep quality).

[0488] According to an embodiment of the present invention, when a user is in a pre-sleep state, a smart home appliance may operate according to the first environment creation information from the time when the user is predicted to prepare for sleep (e.g., the sleep induction time) to the time when the user falls asleep (i.e., the time when the second sleep state information is acquired). Hereinafter, the operation of various smart home appliances will be described with examples.

[0489] For example, when a user is preparing to go to bed, such as when the user is predicted to get ready to sleep or when the user intends to sleep, a built-in motion sensor in a light installed in a bedroom, living room, kitchen, bathroom, etc. can detect whether the user is present in the room, and a healthcare app can start measuring the user's sleep.

[0490] A TV according to an embodiment of the present invention may provide user-optimized sleep content or set a screen-off time, where the user-optimized sleep content may include mindfulness, guided imagery, ASMR, counting numbers backwards, and counting quantities.

[0491] An air conditioner and / or air purifier according to an embodiment of the present invention can adjust the room temperature to help a user fall asleep, and can also switch the type of air provided to indirect air.

[0492] The humidifier and / or dehumidifier according to an embodiment of the present invention can be activated in a low noise state and can maintain an appropriate humidity level.

[0493] A refrigerator according to an embodiment of the present invention can recommend foods that help with sleep (e.g., warm milk, chamomile, etc.) based on an analysis of the user's personal bedtime, or can encourage the user to avoid eating a late-night snack.

[0494] The clothing management machine according to an embodiment of the present invention can be switched to a low noise mode or set to start sleeping immediately upon waking up.

[0495] According to an embodiment of the present invention, blinds and / or curtains may be automatically closed, and the sleep lights may be dimmed, while all other lights may be set to be turned off.

[0496] Furthermore, according to an embodiment of the present invention, when the user falls asleep, the healthcare app can recognize that the user has fallen asleep, and the TV can be set to continue providing sound-related content among the optimized sleep content for the user, but turn off the screen.

[0497] Second environment creation information and smart home appliance operation based on the second sleep state

[0498] According to an embodiment of the present invention, the processor 130 may generate second environment creation information based on the second sleeping state information. For example, the processor 130 may determine the time when the user falls asleep, i.e., the time when the user falls asleep, through the second sleeping state information, and may generate the second environment creation information based on the time.

[0499] For example, the processor 130 may generate second environment creation information that minimizes light or controls smart home appliances into a sleep mode to optimize temperature and humidity to create a quiet, dark room-like atmosphere as shown in Fig. 7. Such second environment creation information helps the user fall into a deep sleep, thereby improving the quality of sleep.

[0500] In an embodiment, the processor 130 may generate external environment creation information based on the sleep stage information. In an embodiment, the sleep stage information may include information regarding changes in the user's sleep stages, which are acquired over time through analysis of sleep acoustic information.

[0501] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform the above operation.

[0502] The second environment creation information may be control information for minimizing illuminance to create a dark room environment. For example, if there is light interference during sleep, the probability of fragmented sleep increases, making it difficult to get a good night's sleep.

[0503] In addition, the processor 130 may generate second environment creation information for controlling the smart home appliance to lower the brightness of the display of the smart home appliance to a predetermined brightness, turn off the display, operate at a noise level below a predetermined level, adjust the airflow intensity to a level below a predetermined level, adjust the airflow temperature to within a predetermined range, maintain the humidity in the sleeping space at a predetermined temperature, or maintain indirect airflow based on the second sleep state information.

[0504] The second environment creation information may include control information for operating smart home appliances, such as improving the air quality in the indoor space or optimizing the temperature and humidity, since there is less risk of waking up when the user is in deep sleep according to the sleep stage.

[0505] That is, when the processor 130 detects that the user has entered sleep (or a sleep stage) (when the processor 130 acquires the second sleep state information), it can generate second environment creation information that can block the supply of light or control the operation of smart home appliances, thereby increasing the probability that the user will fall into a deep sleep and improving the quality of their sleep.

[0506] As a specific example, when the processor 130 determines that the user has entered a sleep stage (e.g., light sleep) based on the user's sleep stage information, it may generate external environment creation information that optimizes the room temperature and humidity, minimizes illuminance to create a dark room environment without light, or controls smart home appliances to remove fine dust / harmful gases, adjust the air temperature and humidity, turn on LEDs, adjust the driving noise level, and blow air, etc., to enable deep sleep. In other words, by creating optimal illuminance for each user's sleep stage, i.e., an optimal sleep environment, the user's sleep efficiency can be improved.

[0507] In addition, the processor 130 may generate environmental creation information to provide appropriate illumination or adjust air quality according to changes in the user's sleep stage during sleep. For example, various external environment creation information may be generated according to changes in sleep stage, such as providing subtle red light when changing from light sleep to deep sleep, or reducing illumination or providing blue light when changing from REM sleep to light sleep. This automatically considers not only the situation before sleep or immediately after waking up but also the situation during sleep, thereby maximizing the quality of the user's sleep by considering the entire sleep experience rather than just a part of it.

[0508] Hereinafter, various smart home appliance operations based on the second sleep state information will be described with examples.

[0509] A healthcare application according to an embodiment of the present invention can analyze a user's breathing sounds in real time and provide stimuli such as vibrations or alarms when the user stops breathing.

[0510] A TV according to an embodiment of the present invention can turn off the screen and the sound.

[0511] An air conditioner and / or air purifier according to an embodiment of the present invention can maintain an appropriate indoor temperature and indirect airflow, and can adjust the temperature when detecting light sleep due to temperature changes.

[0512] The humidifier and / or dehumidifier according to an embodiment of the present invention can maintain a low noise mode and an appropriate humidity level.

[0513] The door lock according to an embodiment of the present invention can check the lock state.

[0514] An outlet and / or switch according to an embodiment of the present invention can be switched into a low power mode.

[0515] Among the lamps according to an embodiment of the present invention, the sleep lamp can be turned off at a preset time (for example, 15 to 25 minutes) after it is recognized that the user has fallen asleep.

[0516] Meanwhile, the sleep stage of the sleep mode may again be categorized into options of a basic sleep mode, a sleep personalization mode, and a special care mode.

[0517] The basic sleep mode can provide a comfortable sleep environment by setting the environment (air, temperature, humidity, light, fragrance, etc.) as the sleep mode basic value.

[0518] The personalized sleep mode can provide a customized sleep mode according to the user's sleep quality based on accumulated user data.

[0519] Special care modes can be developed and provided as customized sleep modes optimized for specific users who have sleep difficulties due to itching, being overweight, etc.

[0520] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0521] Third environment creation information and smart home appliance operation based on wake-up guidance time

[0522] According to an embodiment of the present invention, the processor 130 may generate the third environment creation information based on the wake-up induction time point. Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may generate the third environment creation information.

[0523] For example, the processor 130 may identify the user's wake-up time through the sleep plan information, generate a predicted wake-up time based on the wake-up time, and generate environment creation information accordingly. For example, as shown in Fig. 7, the processor 130 may generate third environment creation information that gradually increases the illuminance of 3000K white light from 0 lux to 250 lux based on the position of the bed, starting 30 minutes before the predicted wake-up time. This third environment creation information may guide the user to wake up naturally and refreshed according to the desired wake-up time.

[0524] The processor 130 may also determine to transmit the environment creation information to the environment creation device 30. That is, the processor 130 may improve the quality of the user's sleep by generating external environment creation information that allows the user to easily fall asleep or wake up naturally when going to bed or waking up based on the sleep plan information.

[0525] In a further embodiment, the processor 130 may generate recommended sleep plan information based on the sleep stage information. Specifically, the processor 130 may obtain information regarding changes in the user's sleep stages (e.g., sleep cycles) through the sleep stage information and may set an expected wake-up time based on such information.

[0526] For example, a typical sleep cycle during a day may include light sleep, deep sleep, pre-sleep, and REM sleep stages. The processor 130 may determine that the time after REM sleep is the time when the user can wake up most refreshed and may determine a wake-up time after the REM time, thereby generating recommended sleep plan information. The processor 130 may also generate environment creation information according to the recommended sleep plan information and determine to transmit the environment creation information to the environment creation device 30. Thus, the user may wake up naturally according to the recommended sleep plan information recommended by the processor 130. This has the advantage of improving the user's sleep efficiency because the processor 130 recommends a wake-up time for the user according to changes in the user's sleep stages, which may be a time when the user's fatigue level is minimized.

[0527] As described above, the third environment creation information may be control information for gradually increasing the illuminance of 3000K white light from 0 lux to 250 lux from the wake-up induction time to the wake-up time. For example, the third environment creation information may be control information related to gradually increasing the illuminance from 30 minutes before the user wakes up (i.e., the wake-up induction time). Here, the wake-up induction time may be determined based on the predicted wake-up time.

[0528] In one embodiment, the wake-up guidance time may be determined based on a predicted wake-up time. The predicted wake-up time may be information about the time when the user is expected to wake up. For example, the predicted wake-up time may be 7:00 AM for the first user. The specific descriptions of the predicted wake-up times or values ​​described above are merely examples, and the present invention is not limited thereto.

[0529] The third environment creation information may include information for controlling smart home appliances to induce wake-up by increasing or decreasing at least one of indoor temperature, humidity, airflow strength, noise, and vibration at the time of waking up. The third environment creation information may also include control information for controlling smart home appliances to generate white noise to induce wake-up gradually.

[0530] The third environment creation information may include control information for controlling the noise of the smart home appliances to be maintained below a preset level after waking up.

[0531] The third environment creation information may include control information for controlling smart home appliances in conjunction with a predicted wake-up time and a recommended wake-up time. The recommended wake-up time may be a time automatically selected according to a user's sleep pattern, and the predicted wake-up time will be described in detail later.

[0532] Hereinafter, the wake-up mode operations of various smart home appliances based on the wake-up induction time and the wake-up time will be described as examples.

[0533] In the pre-wake stage, the healthcare app according to an embodiment of the present invention may perform sleep analysis of the user to recognize the user's sleep patterns.

[0534] An air conditioner and / or air purifier according to an embodiment of the present invention can adjust the environment, such as the quality of air, temperature, or humidity, in a room to wake up a user.

[0535] During the wake-up stage, the healthcare app according to an embodiment of the present invention can activate a smart alarm installed in the app if the user's rapid eye movement (REM) sleep is detected or a change in the user's body temperature is detected.

[0536] The humidifier and / or dehumidifier according to an embodiment of the present invention can be switched to a normal operation mode.

[0537] The clothing management machine according to an embodiment of the present invention can start operating according to the wake-up alarm time that has already been set during the bedtime preparation stage.

[0538] The blinds and / or curtains according to an embodiment of the present invention can be opened automatically.

[0539] The washing machine according to an embodiment of the present invention can start a washing operation.

[0540] Additionally, the dryer according to an embodiment of the present invention can start a drying operation.

[0541] In the post-wake-up stage, the healthcare app according to one embodiment of the present invention can display the analyzed sleep report of the user on the user's smartphone 900 and provide the user with optimized content such as today's weather, major news, etc.

[0542] The clothing care machine according to an embodiment of the present invention can complete tasks such as removing dust and wrinkles from clothing, removing odors, sterilizing, and drying according to a preset time when the user is going out.

[0543] At this stage, the robot vacuum cleaner according to an embodiment of the present invention may transmit a report to the user's smartphone 900 as needed, and collect, analyze, and reflect the user data, and after completing the washing operation of the washing machine and the drying operation of the dryer before the user leaves the house, collect, analyze, and reflect the user data.

[0544] A water purifier according to an embodiment of the present invention can collect, analyze, and reflect user data after automatically dispensing customized water that reflects the user's preferences.

[0545] The refrigerator according to an embodiment of the present invention can display a list of recommended and non-recommended breakfast menus and a list of recommended morning exercises on a display unit installed on the front based on analyzed sleep and health data of the user.

[0546] According to an embodiment of the present invention, when one or more breakfast recommended menu items are clicked on in the refrigerator, the oven / microwave oven automatically preheats the menu items and then acquires, analyzes, and reflects user data.

[0547] In addition, the user data of the previous smart home appliance 800 may be collected, analyzed, and reflected to recommend the optimal sleeping environment (temperature, humidity, air quality, illuminance, etc.) based on personal data.

[0548] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0549] Determining the predicted time to wake up

[0550] In one embodiment, the predicted wake-up time may be determined in advance through information exchange between the user and the user terminal 10. For example, the user may set the time at which they intend to wake up through the user terminal 10 and transmit the time to the processor 130. That is, the processor 130 may acquire the predicted wake-up time based on the time set by the user of the user terminal 10. For example, when the user sets an alarm time through the user terminal 10, the processor 130 may determine the set alarm time as the predicted wake-up time.

[0551] In another embodiment, the predicted wake-up time may be determined based on the time the user falls asleep identified through the second sleep state information. Specifically, the processor 130 may determine the time the user falls asleep through the second sleep state information indicating that the user is asleep. The processor 130 may determine the predicted wake-up time based on the time the user falls asleep identified through the second sleep state information. For example, the processor 130 may determine the predicted wake-up time to be a time 8 hours or later, which is the appropriate sleep time, from the time the user falls asleep. As a specific example, if the time the user falls asleep is 11:00 PM, the processor 130 may determine the predicted wake-up time to be 7:00 AM. The specific numerical values ​​for each time point described above are merely examples, and the present invention is not limited thereto. That is, the processor 130 may determine the predicted wake-up time based on the time the user falls asleep.

[0552] In another embodiment, the recommended wake-up time may be determined based on the user's sleep stage information. For example, a user can wake up most refreshed if they wake up in the REM sleep stage. During a night's sleep, a user may have a sleep cycle consisting of light sleep, deep sleep, light sleep, and REM sleep in that order, and can wake up most refreshed when they wake up in the REM sleep stage. Preferably, the recommended sleep time may be determined taking into account the user's appropriate or desired sleep time while minimally satisfying the appropriate or desired sleep time.

[0553] Thus, the processor 130 can determine the predicted wake-up time of the user based on sleep stage information related to the sleep stage of the user. For example, the processor 130 can determine the time when the user changes from the REM stage to another sleep stage (preferably, the time immediately before the transition from the REM stage to another sleep stage) as the recommended wake-up time based on the sleep stage information. That is, the processor 130 can determine the predicted wake-up time based on the sleep stage information (i.e., the REM sleep stage) from which the user can wake up most refreshed.

[0554] As described above, the processor 130 may determine the predicted wake-up time of the user based on at least one of the user setting, the time at which the user falls asleep, and the sleep stage information. Furthermore, if the processor 130 determines the predicted wake-up time as the time at which the user intends to wake up, it may determine the wake-up induction time based on the predicted wake-up time. For example, the processor 130 may determine the wake-up induction time to be 30 minutes before the time at which the user intends to wake up. For example, if the time at which the user intends to wake up (i.e., the predicted wake-up time) set by the user is 7:00 AM, the processor 130 may determine 6:30:00 AM as the wake-up induction time. The above-described specific examples of the time are merely examples, and the present invention is not limited thereto.

[0555] That is, the processor 130 may determine a wake-up guidance time by grasping a wake-up prediction time when the user is predicted to wake up, and may generate third environment creation information for supplying 3000K white light with gradually increasing illuminance from 0 lux to 250 lux from the wake-up guidance time to the wake-up time (e.g., until the user actually wakes up). The processor 130 may determine to transmit the third environment creation information to the environment creation device 30, and the environment creation device 30 may then perform a light-related adjustment operation in the space where the user is located based on the third environment creation information.

[0556] Alternatively, in the case of an embodiment such as that shown in Fig. 1(c), at least one of the electronic devices shown in Fig. 1(c) may perform at least one of the above-described operations. For example, the environment creating device 30 may control the light supply module to gradually increase the illuminance of 3000K white light from 0 lux to 250 lux 30 minutes before waking up. The above numerical values ​​are merely examples, and the present invention is not limited thereto.

[0557] 3. Fourth environment creation information and smart home appliance operation based on sleep state information

[0558] According to an embodiment of the present invention, the processor 130 may acquire fourth environment creation information based on the third sleep state information. Specifically, the processor 130 may acquire user sleep disorder information. In one embodiment, the sleep disorder information may include delayed sleep phase syndrome. Delayed sleep phase syndrome may be a sleep disorder symptom in which a user is unable to fall asleep at a desired time and their ideal sleep time shifts later. According to an embodiment, blue-light therapy is one of the methods for treating delayed sleep phase syndrome and may involve providing blue light for approximately 30 minutes after the user wakes up at their desired wake-up time. If this blue light is provided every morning, it may restore the circadian rhythm and prevent the user from becoming sleepy later in the night compared to a normal person.

[0559] Thus, the processor 130 may generate fourth environment creation information based on the sleep disorder information and the third sleep state information. For example, when the processor 130 acquires sleep disorder information indicating that the user has a delayed sleep phase syndrome and third sleep state information indicating that the user has fallen asleep (i.e., woken up) via the user terminal 10, the processor 130 may generate fourth environment creation information. In this case, the fourth environment creation information may be control information for supplying blue light with an illuminance of 300 lux, a chromaticity of 221 degrees, a saturation of 100%, and a brightness of 56% for a preset time period from the time of waking up. In one embodiment, the blue light with an illuminance of 300 lux, a chromaticity of 221 degrees, a saturation of 100%, and a brightness of 56% may represent blue light for treating delayed sleep phase syndrome. For example, if a user with delayed sleep phase syndrome wakes up at 7:00 AM, the processor 130 may determine the wake-up time as 7:00 AM based on the third sleep state information and generate fourth environment creation information to supply blue light with an illuminance of 300 lux, a chromaticity of 221 degrees, a saturation of 100%, and a brightness of 56% from the wake-up time of 7:00 AM to a preset time (e.g., 7:30 AM). This allows the user's circadian rhythm to be adjusted to a range of a normal person (e.g., falling asleep around midnight and waking up around 7:00 AM). In other words, generating the fourth environment creation information may improve the quality of sleep for a user with a specific sleep disorder.

[0560] According to an embodiment of the present invention, the processor 130 may determine to transmit the environment creation information to the environment creation device 30. Specifically, the processor 130 may generate environment creation information related to illuminance adjustment, and may control the illuminance adjustment operation of the environment creation device 30 by determining to transmit the environment creation information to the environment creation device 30.

[0561] According to an embodiment, light may be one of the major factors that can affect sleep quality. For example, the illuminance, color, and degree of exposure of light can have a positive or negative effect on sleep quality. Therefore, the processor 130 can adjust the illuminance to improve the user's sleep quality. For example, the processor 130 can monitor the situation before and after falling asleep and adjust the illuminance accordingly to effectively wake the user. That is, the processor 130 can grasp the sleep state (e.g., sleep stage) and automatically adjust the illuminance to maximize sleep quality.

[0562] Alternatively, in the case of an embodiment such as that shown in Fig. 1(c), at least one of the electronic devices shown in Fig. 1(c) may perform at least one of the above-described operations. The above numerical values ​​and timings are merely examples, and the present invention is not limited thereto.

[0563] Environmental creation information based on detected events

[0564] An embodiment of the present invention may generate ambiance creation information for controlling an ambiance creation device according to at least one detected event.

[0565] The generation of the environment creation information according to an embodiment of the present invention may be performed by the computing device 100 shown in FIG. 1(a) or the sleep environment adjusting device 400 shown in FIG. 1(b).

[0566] The event may be preset in various ways, such as one or more events. For example, the event may include one or more of the following events A to H.

[0567] A. In room B. In bed C. Fall Asleep D. Apnea E.Deep sleep F. Wake up during sleep G. REM near alarm H.Wake up

[0568] Each event will be explained in detail below.

[0569] The above event A is an event that means that the user has entered a space where they sleep, such as a bedroom.

[0570] The A event may be sensed via a presence detection sensor.

[0571] The presence detection sensor is also called a human body detection sensor, and includes, for example, a radar sensor, a PIR motion sensor, a Wi-Fi sensing sensor, a camera sensor, and an ultrasonic sensor.

[0572] The presence detection sensor may be attached to the environment creating device 30, or may be separately attached to a bedroom and connected to the environment creating device 30 by wire or wirelessly. Alternatively, the presence detection sensor may be connected to the network of Fig. 1(a) or 1(b) and transmit a detection signal to the computing device 100, the sleep environment controlling device 400, the user terminal 10, or the environment creating device 30. Alternatively, the presence detection sensor may be connected to the user terminal 10 via short-range communication and transmit a detection signal to the user terminal 10.

[0573] Alternatively, when the present invention is embodied in the embodiment of FIG. 1(c), the presence detection sensor may be present in at least one of the electronic devices shown in FIG. 1(c).

[0574] When the event A occurs, i.e., when the event A is detected by the presence sensor, A-th environment creation information may be generated to automatically turn on the environment creation device 30. Here, the A-th environment creation information may include control information for automatically turning on the environment creation device 30 and changing the environment creation device 30 to a specific operation mode.

[0575] The above event B indicates that the user has lain down on the bed. The event B may be detected through a piezoelectric sensor. The piezoelectric sensor may be attached to the bed on which the user sleeps. However, the present invention is not limited to this, and the piezoelectric sensor may also be attached to a sofa, massage chair, or the like on which the user sleeps.

[0576] The piezoelectric sensor may be connected to the environment creating device 30 by wire or wirelessly. Alternatively, the piezoelectric sensor may be connected to the network of FIG. 1(a) or 1(b) to transmit a sensing signal to the computing device 100, the sleep environment controlling device 400, the user terminal 10, or the environment creating device 30. Alternatively, the piezoelectric sensor may be connected to the user terminal 10 via short-range communication to transmit a sensing signal to the user terminal 10.

[0577] When the event B occurs, i.e., when the event B is detected by the piezoelectric sensor, environment creation information B may be generated to operate the environment creation device 30 in a sleep mode. The environment creation information B may include control information for the environment creation device 30 to help the user fall asleep. The control information may include information for reducing noise and light generated by the environment creation device 30. For example, if the environment creation device 30 is an air conditioner, the control information may include information for changing the airflow rate to a specific strength or lowering the current airflow rate to a specific strength or lower, switching direct airflow to indirect airflow or no airflow, or lowering the brightness of the display unit to a predetermined brightness or lower. The control information may also include information for turning off lights installed in the bedroom or lowering the brightness to a predetermined brightness or lower. The control information may also include information for closing curtains or blinds installed in the bedroom to eliminate external sleep disturbances. The control information may also include information for turning on a sound device installed in the bedroom to play a specific sound source or turning off the sound device. The control information may also include information for changing the motion of a motion bed installed in the bedroom to a specific motion that is convenient for reading or watching media before falling asleep. The information may also include information to operate a scent generator installed in the bedroom to emit a scent that helps the user fall asleep.

[0578] The above C event is an event that indicates that the user has fallen asleep (or is falling asleep). As described above, the C event can be determined by the computing device 100 or the sleep environment adjusting device 400 that receives environmental sensing information sensed in the user terminal 10. As shown in FIG. 7, the time when the user falls asleep (or is falling asleep) can be determined through the environmental sensing information sensed in the user terminal 10.

[0579] When the C event occurs, i.e., when the C event is detected, C-th environment creation information for operating the environment creating device 30 in sleep mode may be generated. The C-th environment creation information may include control information for creating an optimal bedroom sleep environment. Here, the optimal bedroom sleep environment may be optimal environment information obtained based on paired data (temperature and / or humidity and sleep quality) acquired over a predetermined period of time (e.g., one week or one month). For example, based on quantitative data indicating the user's sleep quality over the past week and the bedroom temperature and humidity data during the period in which the quantitative data was acquired, the bedroom temperature and humidity at which the user slept best may be determined as the optimal bedroom sleep environment. The control information may set the bedroom temperature and humidity of the environment creating device 30 to the optimal temperature and humidity. The control information may also include information to turn off lights installed in the bedroom, information to turn off sound devices installed in the bedroom, and information to change the motion of a motion bed installed in the bedroom to a specific motion that is favorable for deep sleep. The information may also include information to turn on or off a fragrance generator installed in the bedroom to emit a fragrance that helps deep sleep.

[0580] The above D event is an event that indicates that sleep apnea or hypopnea occurs while the user is sleeping. As described above, the D event can be determined by the computing device 100 or the sleep environment controlling device 400 that receives environmental sensing information sensed in the user terminal 10. As shown in Fig. 4, the time point when sleep apnea or hypopnea occurs can be determined through the environmental sensing information sensed in the user terminal 10.

[0581] When the D event occurs, i.e., when the D event is detected, D-th environment creation information for operating the environment creating device 30 in sleep mode may be generated. The D-th environment creation information may include control information for alleviating sleep apnea or hypopnea or quickly converting stopped or weak breathing to normal breathing. For example, if the environment creating device 30 is an air conditioner, the control information may include information for increasing the set humidity or temperature or switching direct or indirect airflow to no airflow in order to protect the airway and neck of a user experiencing sleep apnea. Alternatively, if the environment creating device 30 includes a humidification function, the control information may include information for activating the humidification function. Alternatively, if the environment creating device 30 includes a vibration function, the control information may include information for activating the vibration function. Furthermore, the control information may include information for illuminating a light installed in the bedroom at a specific brightness and color temperature. Alternatively, the control information may include information for turning on a sound device installed in the bedroom. Alternatively, the control information may include information for changing the motion of a motion bed installed in the bedroom to a specific motion that assists the user's breathing. The information may also include information for emitting a scent from a scent generator installed in the bedroom to generate a scent that can alleviate sleep apnea.

[0582] The above E event is an event that indicates that the user has entered deep sleep. As described above, the E event can be determined by the computing device 100 or the sleep environment controlling device 400 that receives environmental sensing information sensed in the user terminal 10. As shown in Fig. 3, the time point at which the user has entered deep sleep can be determined through the environmental sensing information sensed in the user terminal 10.

[0583] If the E event occurs, i.e., if the E event is detected, E environment creation information for operating the environment creating device 30 in a sleep mode may be generated. The E environment creation information may include control information for changing the temperature or humidity to an optimized temperature or humidity for a deep sleep stage. For example, if the environment creating device 30 is an air conditioner, the control information may include information for changing the current bedroom temperature or humidity to the optimized temperature or humidity. Here, the optimized temperature or humidity may be determined to be a specific temperature or humidity at which the user maintained deep sleep the longest using a quantitative sleep report acquired over a previous predetermined period. The E environment creation information may also include information for turning off lights installed in the bedroom or reducing brightness to the minimum. The E environment creation information may also include information for turning off a sound device installed in the bedroom. The E environment creation information may also include information for changing the motion of a motion bed installed in the bedroom to a specific motion favorable for deep sleep. The E environment creation information may also include information for turning on a fragrance generator installed in the bedroom to emit a fragrance that helps maintain deep sleep.

[0584] The above F event is an event that indicates when a user wakes up during sleep. As described above, the F event can be determined by the computing device 100 or the sleep environment adjusting device 400 that receives environmental sensing information sensed in the user terminal 10. As shown in Fig. 3, the time when the user wakes up after sleeping can be determined through the environmental sensing information sensed in the user terminal 10.

[0585] If the F event occurs, i.e., if the F event is detected, F-th environment creation information for operating the environment creating device 30 in a sleep mode may be generated. The F-th environment creation information may include control information for helping a user fall asleep again. For example, if the environment creating device 30 is an air conditioner, the control information may include information for changing the set temperature and humidity of the air conditioner to a preferred temperature or humidity that the user previously set when falling asleep. Alternatively, the control information may include information for changing the set temperature or humidity of the environment creating device 30 to a specific temperature or humidity at which the user fell asleep the shortest, based on a past quantitative sleep report. The control information may also include information for lighting a bedroom at a specific brightness and color temperature that helps the user fall asleep. The control information may also include information for turning on and off a sound device installed in the bedroom. The control information may also include information for changing the motion of a motion bed installed in the bedroom to a specific motion that helps the user fall asleep again. The control information may also include information for activating a fragrance generator installed in the bedroom to emit a fragrance that helps the user fall asleep again.

[0586] The G event is an event that indicates that rapid eye movement (REM) sleep occurs near a preset alarm time. As described above, the G event can be determined by the computing device 100 or the sleep environment controlling device 400 that receives environmental sensing information sensed in the user terminal 10. As shown in FIG. 3, the REM sleep time point can be determined through the environmental sensing information sensed in the user terminal 10.

[0587] If the G event occurs, i.e., if the G event is detected, G-th environment creation information for operating the environment creation device 30 in a sleep mode may be generated. The G-th environment creation information may include control information for helping a user wake up. For example, if the environment creation device 30 is an air conditioner, the control information may include information for changing the set temperature or humidity of the air conditioner to a specific temperature or humidity that allows the user to wake up naturally or most refreshed. Alternatively, the control information may include information for changing the set temperature or humidity of the air conditioner to a specific temperature or humidity that the user most prefers using a past quantitative sleep report. The control information may also include information for lighting a bedroom with a specific brightness and color temperature specialized for waking up. The control information may also include information for opening curtains or blinds installed in the bedroom. The control information may also include information for turning on a sound device installed in the bedroom and playing a specific sound source. The control information may also include information for changing the motion of a motion bed installed in the bedroom to a specific motion that is useful for waking up. The information may also include information for causing a fragrance generator installed in the bedroom to emit a fragrance that can provide a refreshing wake-up.

[0588] The above H event is an event that indicates the time when the user wakes up. As described above, the H event can be determined by the computing device 100 or the sleep environment controlling device 400 that receives the environmental sensing information sensed in the user terminal 10. As shown in Fig. 5, the wake-up time can be determined by determining whether a predetermined pattern is continuously sensed after a singular point 201 is identified in the environmental sensing information sensed in the user terminal 10.

[0589] If the H event occurs, i.e., if the H event is detected, H environment creation information for operating the environment creating device 30 in a sleep mode may be generated. The H environment creation information may include control information for setting the temperature of the bedroom where the user slept to an optimal temperature after waking up. If the environment creating device 30 is an air conditioner, the control information may include information for changing the set temperature or humidity of the air conditioner to a temperature or humidity preferred by the user based on the user's past history when the user wakes up. Alternatively, the control information may include suggestion information for changing the set temperature or humidity of the air conditioner to a recommended temperature or humidity after waking up via the user terminal. The control information may also include information for lighting the bedroom with a specific brightness and color temperature. The control information may also include information for turning on a sound device installed in the bedroom to display specific media or play a specific sound source. The control information may also include information for opening a window installed in the bedroom to wake up. The control information may also include information for changing the motion of a motion bed installed in the bedroom to a specific motion that helps the user wake up. The information may also include information for emitting a scent that can help the user move around after waking up by emitting a scent from a scent generator installed in the bedroom.

[0590] Furthermore, if an embodiment of the present invention is, for example, an embodiment such as (c) of FIG. 1, at least one of the electronic devices shown in (c) of FIG. 1 may perform at least one of the above-described operations.

[0591] Sleep Planning Information

[0592] In one embodiment, the processor 130 may receive sleep plan information from the user terminal 10. The sleep plan information is information generated by the user via the user terminal 10 and may include, for example, information regarding bedtime and wake-up time. The processor 130 may generate external environment creation information based on the sleep plan information. For example, the processor 130 may identify the user's bedtime through the sleep plan information and generate external environment creation information based on the bedtime. Alternatively, in an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0593] For example, the processor 130 may generate first environment creation information for providing 3000K white light with an illuminance of 30 lux based on the bed position 20 minutes before bedtime, as shown in Fig. 7. That is, an illuminance that induces the user to fall asleep naturally in relation to bedtime may be created. The above-described values ​​and time points are merely examples, and the present invention is not limited thereto.

[0594] Embodiment 1 of the method for preventing and alleviating sleep disorders

[0595] FIG. 36(a) is a flowchart illustrating a method for preventing and alleviating sleep disorders using an AI-based non-contact sleep analysis system according to an embodiment of the present invention.

[0596] The present invention can analyze a user's sleep in real time to identify when a sleep disorder (sleep apnea, sleep hyperpnea, sleep hypopnea) occurs. At the moment a sleep disorder occurs, the sleep disorder can be temporarily alleviated by providing the user with a stimulus (tactile stimulus, auditory stimulus, olfactory stimulus, etc.).

[0597] That is, the present invention can interrupt a user's sleep disorder and reduce the frequency of sleep disorders based on accurate event detection related to sleep disorders.

[0598] Referring to FIG. 36(a), the method for preventing and alleviating sleep disorders using a smart home device 800 according to the present invention collects sleep acoustic information of a user and performs primary and secondary sleep analyses based on the collected information.

[0599] The primary sleep analysis is a sleep analysis based on the user's sleep acoustic information, and the secondary sleep analysis is an analysis based on the results of the primary sleep analysis and the sleep acoustic information, and the specific analysis method is the same as that described above.

[0600] When it is determined that the user has experienced sleep apnea as a result of the primary sleep analysis and the secondary sleep analysis, the smart home device 800 may generate at least one of haptic feedback and auditory feedback. The smart home device 800 may further include an alarm unit 850 for providing feedback, which may be implemented as an actuator, a vibration module, or a haptic module that generates vibrations, or a speaker module that generates sounds.

[0601] The vibrations transmitted to the body part that the smart home appliance 800 comes into contact with (e.g., the whole body in the case of a smart mat) and the sounds or acoustics that resonate near the ears (e.g., smart speakers, smartphones, smart TVs, etc.) stimulate the user's brain, thereby alleviating sleep apnea relatively quickly. If this process continues while the user is sleeping, it can be seen that the frequency of sleep apnea is also significantly reduced.

[0602] In this case, not only a single sleep apnea event can be detected, but also a cluster of consecutive sleep apnea events can be predicted in advance. To this end, the sleep analysis learning model can perform learning to predict a cluster of consecutive sleep apnea events.

[0603] That is, as described above, input information based on the user's sleep acoustic information is input to the input layer after undergoing the preprocessing process and mel-spectrogram conversion process, and the sleep analysis learning model that learns this information can predict the cluster of consecutive occurrences of sleep apnea events.

[0604] If clusters of consecutive sleep apnea events can be predicted in advance, sleep apnea can be prevented in advance or mitigated or improved by vibrating the smartphone 900 once or several times at the predicted time, not just at the moment when a sleep apnea event is detected.

[0605] That is, the present invention makes it possible to analyze sleep stages based on sleep acoustic information signals and alleviate and improve sleep apnea.

[0606] The patterns of tactile and auditory feedback provided to the user may be adapted to maintain the user's deep sleep and reduce the frequency of sleep apnea, and these patterns may be adjusted in real time based on the results of an analysis of the user's sleep stages.

[0607] Such patterns may also be inferred through a deep learning model trained based on big data regarding the user's sleep stage analysis results and big data regarding the frequency of sleep apnea.

[0608] The above mentions sleep disorders such as sleep apnea and hyperventilation, but in order to improve the quality of sleep, if it is determined that the user is in the REM sleep stage, stimulation can be transmitted to the user via the smart home appliance 800.

[0609] REM sleep is a sleep stage in which brain waves accelerate and autonomic nervous activity, like heart rate and breathing, is irregular, accompanied by mild involuntary muscle spasms and rapid eye movements. It generally occurs 3-4 times every 80-120 minutes, but in severe cases, it can develop into REM sleep disorder, which can affect the quality of sleep.

[0610] Therefore, the user can be stimulated not only during sleep disorders such as sleep apnea, hyperventilation, and snoring, but also during REM sleep through the smart home device 800. That is, when it is determined that the user has entered the REM sleep stage as a result of the primary sleep analysis and the secondary sleep analysis, the smart home device 800 can generate at least one of haptic feedback and auditory feedback.

[0611] Alternatively, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0612] Method for preventing and alleviating sleep disorders, embodiment 2

[0613] FIG. 36(b) is a flowchart illustrating a method for preventing and alleviating sleep disorders using an AI-based non-contact sleep analysis system according to another embodiment of the present invention.

[0614] The embodiment shown in FIG. 36(b) is based on the assumption that sleep analysis is performed using a smart home appliance 800 and a smartphone 900.

[0615] The sleep analysis result may include sleep state information, sleep stage information, sleep disorder occurrence information, time information, etc. The smartphone 900 performs sleep analysis based on sleep sound information acquired through a built-in microphone module. Hereinafter, a method in which the smartphone 900 derives a final sleep analysis result using the sleep sound information will be described.

[0616] First, the smartphone 900 may derive a final sleep analysis result using a weighting value. Specifically, the smartphone 900 may derive a second sleep analysis result by applying the same weighting value to the first sleep analysis result and the sleep analysis result using sleep acoustic information.

[0617] In another embodiment, the smartphone 900 may determine that the user has entered a corresponding sleep stage and derive a final sleep analysis result only when the sleep stages in the first and second sleep analysis results completely match. In another embodiment, the smartphone 900 may first perform a second sleep analysis using sleep sound information using an AI sleep analysis model (described later), and then additionally extract an AI confidence factor for the sleep stage for each time period.

[0618] If the extracted confidence level is equal to or lower than a preset value, the sleep stage for that time period is determined by the sleep stage results derived by the primary sleep analysis.

[0619] That is, by additionally adopting the primary sleep analysis result in addition to the secondary sleep analysis result, it is possible to derive a more reliable sleep analysis result.

[0620] In another embodiment, the smartphone 900 first obtains statistics for portions of the AI ​​sleep analysis model (described later) that are inconsistent with the actual analysis results. The statistics may be input by the user or may be obtained independently based on multiple user data. The smartphone 900 may additionally use the primary sleep analysis results for portions of the obtained statistics that are inconsistent with the actual analysis results, centering on the secondary sleep analysis results (sound-based analysis).

[0621] The learning method of the AI ​​sleep analysis model will be described in more detail below, but in brief, by inputting two pieces of information (primary sleep analysis results and sleep acoustic information) into the deep learning input layer, an AI sleep analysis model that performs sleep analysis based on two factors may be generated.

[0622] This is merely an example embodiment, and the final sleep analysis result may be derived in various ways.

[0623] If the smartphone 900 determines that sleep apnea has occurred as a result of the secondary sleep analysis, the sensor unit can immediately transmit sleep apnea occurrence information to a processor built into the smartphone 900. The sleep apnea occurrence information corresponds to a trigger signal for the smart home appliance 800 to generate at least one of haptic feedback and auditory feedback, and when the smart home appliance 800 receives the sleep apnea occurrence information, it can stimulate the user through vibration, sound, audio, etc. The stimulation can quickly alleviate the user's sleep apnea, and the user's sleep apnea can be prevented or alleviated through continuous monitoring and stimulation.

[0624] 34 to 36, the primary sleep analysis may be omitted, and sleep analysis may be performed only by the smartphone 900. That is, if the sleep analysis of the user is performed in the above-mentioned manner based on the sleep acoustic information of the user, and sleep apnea is detected as a result of the sleep analysis, sleep apnea occurrence information may be immediately transmitted to the smart home appliance 800 (e.g., smart mat, smart speaker, etc.) linked to the smartphone 900, thereby causing the smart home appliance 800 to generate a vibration or an alarm (sound, audio).

[0625] Meanwhile, the correlation between air quality and sleep is as follows:

[0626] According to research results, if a pregnant woman is exposed to bad air during the first to eighth weeks of pregnancy, the baby's sleep efficiency will decrease, and if the period is between 31 and 35 weeks, the baby's sleep time will decrease. There are also research results that show that the quality of sleep during the growth period is closely related to the ability to acquire knowledge and growth and development.

[0627] Research has also shown that exposure to PM10 in the summer increases irregular breathing during sleep, which is associated with increased cardiovascular disease and mortality in humans.

[0628] On the other hand, the correlation between temperature / humidity and sleep is as follows:

[0629] In a comparative experiment comparing sleep conditions in an environment with a carbon dioxide concentration of 800 ppm and 17,000 ppm, one study found that the air felt more stuffy and hot when the carbon dioxide concentration was 800 ppm. Another study found that sleeping in a chamber with a temperature of 28 degrees Celsius resulted in lower sleep efficiency and work efficiency the next day compared to sleeping in a chamber with a temperature of 24 degrees Celsius.

[0630] Additionally, research has shown that when sleeping in an environment with a humidity level of 80% and a temperature of 32 degrees Celsius, compared to sleeping in an environment with a humidity level of 50% and a temperature of 26 degrees Celsius, the frequency of awakenings during sleep increases and the rate of deep sleep decreases.

[0631] Such stimuli to prevent or alleviate the user's sleep disorder may be generated by other environment-creating devices other than the smartphone 900 or smart speaker.

[0632] Here, the other environment creating devices may be lighting, air purifiers, humidifiers, speakers (audio), clothing management machines, TVs, clocks, PCs, motion beds, mattresses, smart pillows, blinds, curtains, robots, vacuum cleaners, washing machines, dryers, water purifiers, refrigerators, ovens / ranges, etc. Information on the occurrence of a sleep disorder by the user may be transmitted to the various environment creating devices mentioned above, and the environment creating devices may generate a stimulus source to stimulate the user.

[0633] For example, information on the occurrence of a sleep disorder can stimulate the user by controlling lighting (electric lights) to increase brightness, making an air purifier make a running sound, turning on a TV, activating a clock alarm, turning on a PC, controlling a motion bed to change the angle of the bed, controlling a smart pillow or smart mattress to provide tactile changes or movements, or activating various home appliances to make sounds, thereby interrupting or alleviating the sleep disorder.

[0634] Also, in the case of an embodiment such as that shown in FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may perform at least one of the above-described operations.

[0635] Traffic response method and single / multiple person sleep analysis

[0636] FIG. 37 is a diagram illustrating a traffic handling method when the sleep analysis method according to the present invention is performed in the cloud.

[0637] 38 is a conceptual diagram illustrating sleep analysis for one person and sleep analysis for multiple people in the sleep analysis method according to the present invention. To facilitate understanding, the following description will be given assuming that the smart home appliance 800 is a smart speaker. However, this is merely an explanation for facilitating understanding, and the smart home appliance of the present invention is not limited thereto.

[0638] The sleep analysis method according to the present invention can be provided to users via the cloud of Amazon Web Services (AWS). Since the sleep analysis method according to the present invention is performed mainly from dusk to dawn, traffic may occur during this time.

[0639] Therefore, the sleep analysis method according to the present invention may further include the steps of analyzing a time period in which a lot of traffic occurs, predicting an event that will occur during that time period, and automatically adjusting (adding, relocating, etc.) the AI ​​server 310 when the event occurs. This allows the present invention to flexibly handle traffic that may occur at a specific time.

[0640] First, when one person is sleeping as shown in (a) of FIG. 38, the smart speaker and smartphone 900 are all located in the same sleep space in the sleep analysis of one person. That is, the smart speaker can acquire sleep acoustic information, sleep environment information, etc. of one user, and the smartphone 900 can acquire sleep acoustic information, sleep environment information (illuminance, etc.) of one user. In such a sleep environment of one person, the sleep analysis method described above may be applied as is.

[0641] However, when many people are sleeping as shown in (b) of FIG. 38, the sleep acoustic information acquired by the smart speaker or smartphone 900 may include sleep information of many people, such as user 1 and user 2.

[0642] Therefore, when multiple users sleep in the same sleep space, sleep analysis will be performed more precisely. The method for analyzing the sleep of multiple people has been described above with reference to Figures 39 to 46, so a duplicated description will be omitted.

[0643] Sleep environment control device

[0644] The sleep environment adjusting device shown in Fig. 1(b) will be described in more detail below. As shown in Fig. 1(b), the sleep environment adjusting device 400, the user terminal 10, and the external server 20 can mutually transmit and receive data for the system according to an embodiment of the present invention via a network.

[0645] The network according to the embodiment of the present invention is the same as that described in detail above, so a repeated description will be omitted.

[0646] According to the present embodiment, the user terminal 10 may refer to a terminal carried by a user that can receive information related to the user's sleep through information exchange with the sleep environment controlling device 400. The general configuration and functions of the user terminal 10 may be the same as those described above. The user terminal 10 may acquire acoustic information related to a space in which the user is located. For example, the acoustic information may refer to acoustic information acquired in the space in which the user is located. The acoustic information may be acquired in relation to the user's activity or sleep in a contactless manner.

[0647] For example, the acoustic information may be acquired in the space while the user is sleeping. According to an embodiment, the acoustic information acquired through the user terminal 10 may be information that serves as a basis for acquiring the user's sleep state information in the present invention. For a specific example, sleep state information related to whether the user is before, during, or after sleep may be acquired through acoustic information acquired in relation to the user's movement or breathing. Furthermore, for example, information regarding changes in the user's sleep stage during sleep time may be acquired through the acoustic information.

[0648] The sleep environment adjusting device 400 of the present invention can receive health checkup information or sleep examination information from the external server 20 and create a learning data set based on the received information. The external server 20 has been described in detail above, and therefore, a detailed description thereof will be omitted here.

[0649] According to an embodiment, the acoustic information utilized by the sleep environment adjusting device 400 for sleep state analysis may be non-invasively acquired during a user's activity or sleep in a space. For specific examples, the acoustic information may include sounds generated by the user turning over in their sleep, sounds related to muscle movements, or sounds related to the user's breathing during sleep. According to an embodiment, the environmental sensing information may include sleep acoustic information, which may refer to sounds related to the user's movement patterns and breathing patterns generated during the user's sleep.

[0650] In an embodiment, the acoustic information may be acquired through at least one of the user terminal 10 carried by the user and the acoustic collection unit 414. For example, environmental sensing information related to the user's activity in a space may be acquired through a microphone module provided in the user terminal 10 and the acoustic collection unit 414.

[0651] The configuration of the microphone module provided in the user terminal 10 or the sound collecting unit 414 is the same as that described above.

[0652] The acoustic information to be analyzed in the present invention is information related to the user's breathing and movements acquired during sleep, and is information about very small sounds (i.e., sounds that are difficult to distinguish). Since it is acquired along with other sounds during sleep, it can be very difficult to detect and analyze if it is acquired through the above-mentioned microphone module with a low signal-to-noise ratio.

[0653] According to an embodiment of the present invention, the sleep environment controlling device 400 may acquire sleep state information based on acoustic information acquired through a microphone module configured with MEMS. Specifically, the sleep environment controlling device 400 may convert and / or adjust unclearly acquired acoustic information containing a lot of noise into data that can be analyzed, and may perform training on an artificial neural network using the converted and / or adjusted data. Once pre-training of the artificial neural network is complete, the trained neural network (e.g., an acoustic analysis model) may acquire sleep state information of a user based on data (e.g., a spectrogram) acquired (e.g., converted and / or adjusted) corresponding to the acoustic information. In an embodiment, the sleep state information may include not only information related to whether the user is sleeping, but also sleep stage information related to changes in the user's sleep stage during sleep. For example, the sleep state information may include sleep stage information indicating that the user was in REM sleep at a first time point and was in light sleep at a second time point different from the first time point. In this case, information may be obtained through the sleep state information that the user fell into a relatively deep sleep at a first time point and fell into a lighter sleep at a second time point.

[0654] That is, when the sleep environment adjusting device 400 acquires sleep acoustic information having a low signal-to-noise ratio from a commonly used user terminal for collecting acoustics (e.g., an AI speaker, a bedroom IoT device, a mobile phone, etc.) or through the acoustic collecting unit 414, it processes the acquired sleep acoustic information into data suitable for analysis, and processes the processed data to provide information on whether the user is before sleep, during sleep, or after sleep, and sleep state information related to changes in sleep stages.

[0655] In an embodiment, the sleep environment adjusting device 400 may be a terminal or a server, and may include any type of device. The sleep environment adjusting device 400 may be a digital device equipped with a processor, memory, and computing capabilities, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone. The sleep environment adjusting device 400 may be a web server that processes services. The types of servers described above are merely examples, and the present invention is not limited thereto.

[0656] According to an embodiment of the present invention, the sleep environment adjusting device 400 may be a server that provides cloud computing services, which has been described in detail above and will not be described here.

[0657] Alternatively, for example, in the embodiment of FIG. 1(c), at least one of the electronic devices shown in FIG. 1(c) may be embodied in the sleep environment adjusting device 400.

[0658] FIG. 10 shows an exemplary block diagram of a sleep environment adjusting device in accordance with an embodiment of the present invention.

[0659] As shown in FIG. 10, the sleep environment adjusting device 400 may include a receiving module 410 and a transmitting module 420.

[0660] According to an embodiment of the present invention, the sleep environment adjusting device 400 may include a transmitting module 420 that transmits a wireless signal and a receiving module 410 that receives the transmitted wireless signal. In one embodiment, the wireless signal may refer to an orthogonal frequency division multiplexing (OFDM) signal. For example, the wireless signal may be a Wi-Fi-based OFDM sensing signal. The transmitting module 420 may be implemented as a laptop, smartphone, tablet PC, smart speaker (AI speaker), etc., and the receiving module 410 may be implemented as a Wi-Fi receiver. According to an embodiment, the receiving module 410 may be implemented as various computing devices such as a laptop, smartphone, tablet PC, etc. For example, the transmitting module 420 and the receiving module 410 may be equipped with a wireless chip conforming to Wi-Fi 802.11n, 802.11ac, or other standards supporting OFDM. In other words, the sleep environment adjusting device 400 that reliably acquires object state information may be implemented using relatively low-cost equipment.

[0661] In one embodiment, the transmitting module 420 can transmit a wireless signal in one direction to where the object is located, and the receiving module 410 is separated from the transmitting module 420 by a predetermined distance and can receive the wireless signal transmitted from the transmitting module 420. Such a wireless signal can be transmitted or received via multiple subcarriers by using an orthogonal frequency division multiplexing method.

[0662] The transmitting module 420 and the receiving module 410 may be provided with a predetermined separation distance. In this case, the predetermined separation distance may refer to a space in which an object is active or located. In a specific embodiment, the transmitting module 420 and the receiving module 410 may be provided at positions facing each other based on a pre-defined area. Here, the pre-defined area 11a may be, for example, an area related to a position where a user sleeps, such as an area where a bed is located, as shown in FIG. 12. Alternatively, it may refer to an area from which object state information, such as information related to the user's movements or breathing, can be obtained. Here, the object state information is not limited to information related to the user's movements or breathing, but may also include various information related to the user, such as audio information or visual information.

[0663] The transmitting module 420 and the receiving module 410 may be provided on both sides of the bed where the user sleeps. In this case, the sleep environment controlling device 400 of the present invention can acquire object state information, such as information on whether the user is located in a pre-defined area and information on the user's movement or breathing, based on the Wi-Fi-based OFDM signals transmitted and received through the transmitting module 420 and the receiving module 410.

[0664] According to one embodiment, the transmitting module 420 and the receiving module 410 can transmit and receive wireless signals (e.g., OFDM signals) via one or more antennas. For example, if the transmitting module 420 and the receiving module 410 each have three antennas, channel state information related to a total of 192 (i.e., 3×64) channels can be acquired for each frame via the three antennas and 64 subcarriers. The specific numerical values ​​for the antennas and subcarriers described above are for illustrative purposes only, and the present disclosure is not limited thereto.

[0665] According to an embodiment, a plurality of transmitting modules 420 and receiving modules 410 may be provided. More specifically, three transmitting modules and four receiving modules may be provided at predetermined intervals. In this case, the wireless signals transmitted and received by the plurality of transmitting modules and receiving modules may be different from each other.

[0666] In an embodiment, the wireless signal received via the receiving module 410 may be a wireless signal that has passed through a channel corresponding to a pre-established area and may include information indicating characteristics of the channel. The receiving module 410 may acquire channel state information from the wireless signal. The channel state information is information indicating characteristics related to a channel associated with a space where a user is located, and may be calculated based on the wireless signal transmitted from the transmitting module 420 and the wireless signal received via the receiving module.

[0667] Specifically, the wireless signal transmitted from the transmitting module 420 may pass through a specific channel (i.e., the space where the user is located) and be received via the receiving module 410. In this case, the wireless signal may be transmitted via multiple subcarriers corresponding to each multipath. As a result, the wireless signal received via the receiving module 410 may be a signal reflecting the user's movement in the pre-defined area 11a. The processor may acquire channel state information related to the channel characteristics experienced by the wireless signal as it passes through the channel (i.e., the space where the user is located) via the received wireless signal. Such channel state information may be composed of amplitude and phase. That is, the sleep environment controlling device 400 may acquire channel state information related to the characteristics of the space (i.e., the pre-defined area) between the transmitting module 420 and the receiving module 410 based on the wireless signal transmitted from the transmitting module 420 and the wireless signal received via the receiving module 410 (i.e., the signal reflecting the object's movement).

[0668] According to an embodiment, when the receiving module 410 receives a wireless signal transmitted from the transmitting module 420, the receiving module 410 may detect user movement based on the received wireless signal. The receiving module 410 may acquire information regarding whether the user is located in a pre-defined area through a change in channel state information. According to an embodiment, during the process of transmitting and receiving a wireless signal via the transmitting module 420 and the receiving module 410, channel state information acquired when the user is located or not between the transmitting module 420 and the receiving module 410 may differ. According to a specific embodiment, the transmitting module 420 and the receiving module 410 may be arranged such that a difference between channel state information acquired when the user is located in the area between the transmitting module 420 and the receiving module 410 (i.e., the pre-defined area) and when the user is not located therein is maximized. According to an additional embodiment, a directional patch antenna may be provided corresponding to each of the transmitting module 420 and the receiving module 410. Here, the directional patch antenna may be an antenna module configured with m×n patches (i.e., m horizontal patches and n vertical patches). For example, the antenna beam may be preset to increase the signal difference when the user is positioned between the transmitting module 420 and the receiving module 410. The antenna beam width may be preset to be optimal, and the transmitting module 420 and the receiving module 410 may be positioned so that the user is lying down in the direction of transmitting and receiving signals using such directional patch antennas. That is, a wireless link that directly ensures line-of-sight may be formed between the directional patch antennas of the transmitting module 420 and the receiving module 410. With this configuration, the antennas of each module may be operated as directional antennas, thereby forming a wireless link corresponding to a smaller area (e.g., a pre-defined area).

[0669] That is, when a wireless link is formed between the antennas of the transmitting module 420 and the receiving module 410 and a user is positioned between the wireless link, the user's body blocks the wireless link, distorting the wireless link and significantly changing the signal level (i.e., channel state information). In an embodiment, the change in the signal level can be detected through changes in the received signal strength indicator (RSSI) and channel state information (CSI), and the receiving module 410 can determine whether the user is positioned in the previously set area 11a based on these changes.

[0670] In an embodiment, information regarding whether the user is located in the previously set area 11a can be used to determine whether to activate the environment creation unit 415 or to understand the user's sleep intention.

[0671] According to an embodiment of the present invention, the receiving module 410 can calculate sleep state information of a user and adjust the user's sleep environment based on the sleep state information. Specifically, the receiving module 410 can acquire sleep state information related to whether the user is about to fall asleep, asleep, or asleep based on the acquired sensing information, and can adjust the sleep environment of the space where the user is located according to the sleep state information. For example, if the receiving module 410 acquires sleep state information indicating that the user is about to fall asleep, the receiving module 410 can generate environment creation information related to light intensity and illuminance for inducing sleep (e.g., 3000K white light, 30 lux illuminance) based on the sleep state information. Furthermore, the receiving module 410 can adjust the light intensity and illuminance of the space where the user is located to an appropriate intensity and illuminance for inducing sleep (e.g., 3000K white light, 30 lux illuminance) based on the environment creation information related to the light intensity and illuminance for inducing sleep.

[0672] In addition, when the receiving module 410 acquires sleep state information indicating that the user is about to fall asleep, it can generate environment creation information for controlling smart home appliances from the time when the user is predicted to prepare for sleep (e.g., the sleep induction time) to the time when the user falls asleep (i.e., the time when the second sleep state information is acquired).

[0673] Specifically, it can generate environmental creation information such as removing fine dust and harmful gases in advance up to a specified time (e.g., 20 minutes) before the user goes to sleep, controlling the indoor temperature and humidity to optimize it according to the season and the user, and controlling the lighting.

[0674] In addition, the environmental creation information may include information such as controlling various smart home appliances to induce noise (white noise) that is sufficient to induce sleep just before falling asleep, adjusting the airflow strength to a level lower than the preset strength, lowering the LED strength, and switching direct airflow to indirect airflow.

[0675] In addition, the first environment creation information may include control information for adjusting at least one of various environments such as a personalized indoor temperature, indoor humidity, airflow strength, or noise level according to the operation history of the smart home appliance and the acquired sleep state (e.g., sleep quality).

[0676] The above-described specific descriptions regarding the sleep state information and the environment creation information are merely examples, and the present invention is not limited thereto.

[0677] FIG. 11(a) shows an exemplary block diagram of a receiving module and a transmitting module associated with one embodiment of the present invention.

[0678] 11(a), the receiving module 410 may include a network unit 411, a memory 412, a sensor unit 413, a sound collecting unit 414, an environment creating unit 415, and a reception control unit 416. The receiving module 410 is not limited to the above-mentioned components. That is, additional components may be included, or some of the above-mentioned components may be omitted, depending on the implementation of the present invention.

[0679] 11(a), the transmission module 420 may include a transmitter 421 that transmits a wireless signal and a transmission control unit 422 that controls a wireless signal transmission operation of the transmitter 421. In the embodiment, the transmission control unit 422 may determine a time point at which the wireless signal is transmitted via the transmitter 421. For example, the transmission control unit 422 may control the transmitter 421 in response to a time point at which the sleep measurement mode is started, thereby transmitting the wireless signal.

[0680] According to an embodiment of the present invention, the receiving module 410 may include a network unit 411 that transmits and receives data to and from the transmitting module 420, the user terminal 10, and the external server 20. The network unit 411 may transmit and receive data, etc., for performing the method for adjusting a sleep environment based on sleep state information according to an embodiment of the present invention, to and from other computing devices, servers, etc. That is, the network unit 411 may provide a communication function between the receiving module 410, the transmitting module 420, the user terminal 10, and the external server 20.

[0681] For example, the network unit 411 may receive sleep examination records and electronic health records for a plurality of users from a hospital server. As another example, the network unit 411 may receive acoustic information related to the space in which the user is active from the user terminal 10. As another example, the network unit 411 may transmit environment creation information for adjusting the environment of the space in which the user is located to the environment creation unit 415. Furthermore, the network unit 411 may allow information transmission between the sleep environment adjusting device 400, the user terminal 10, and the external server 20 by calling a procedure in the sleep environment adjusting device 400.

[0682] The network unit 411 according to an embodiment of the present invention may be configured using any one of the various wired and wireless communication systems described above, or a combination thereof.

[0683] According to an embodiment of the present invention, the memory 412 may store a computer program for performing a sleep environment adjusting method based on sleep state information according to an embodiment of the present invention, and the stored computer program may be read and driven by the reception control unit 416. The memory 412 may also store any type of information generated or determined by the reception control unit 416 and any type of information received by the network unit 411. The memory 412 may also store data related to the user's sleep. For example, the memory 412 may temporarily or permanently store input / output data (e.g., acoustic information related to the user's sleep environment, sleep state information corresponding to the acoustic information, or environment creation information based on the sleep state information).

[0684] According to an embodiment of the present invention, the memory 412 may include at least one type of storage medium selected from the group consisting of flash memory, hard disk, micro multimedia card, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PEOM), magnetic memory, magnetic disk, and optical disk. The sleep environment controlling device 400 may also operate in association with web storage that performs the storage function of the memory 412 over the Internet. The above description of the memory is for illustrative purposes only, and the present invention is not limited thereto.

[0685] The computer program may include one or more instructions that, when loaded into memory 412, cause reception control unit 416 to perform methods / operations according to various embodiments of the present invention. That is, reception control unit 416 can perform methods / operations according to various embodiments of the present invention by executing one or more instructions.

[0686] According to an embodiment of the present invention, the receiving module 410 may include a sensor unit 413 that acquires one or more pieces of sensing information related to a space. In the present invention, a space refers to a space where a user lives, such as a bedroom where the user sleeps.

[0687] According to an embodiment, the sensor unit 413 may include a first sensor unit that detects the movement of a user in a space. The first sensor unit may include at least one of a PIR sensor (Passive Infrared Sensor) and an ultrasonic sensor. The PIR sensor can detect the movement of a user within a detection range by detecting changes in infrared rays emitted from the user's body. For example, the PIR sensor can identify infrared rays of 8 μm to 14 μm emitted from the user's body and detect the movement of a user in a bedroom. The ultrasonic sensor can detect the movement of an object by generating sound waves and detecting the signals reflected by a specific object. For example, the ultrasonic sensor can generate sound waves within the space of a bedroom and detect the movement of a user in the bedroom through the sound waves reflected by the user's body when the user enters the bedroom.

[0688] In addition, in an embodiment, the sensor unit 413 may include a second sensor unit that detects whether the user is located in a pre-defined area of ​​the one space based on a wireless signal. The second sensor unit may receive a wireless signal transmitted from the transmission module 420 and detect whether the user is located in a pre-defined area based on the received wireless signal. In an embodiment, the pre-defined area is related to an area within the one space where the user lies down to sleep, and may refer to an area where a bed is provided, for example. For a specific example, in the present invention, the one space may refer to a space within a bedroom, and the pre-defined area may refer to a space where the bed is located.

[0689] In an embodiment, the second sensor unit may be provided at a position facing the transmitting module 420 based on the preset area. For example, the transmitting module 420 and the second sensor unit may be provided on both sides of the bed where the user sleeps. In this case, the sleep environment adjusting device 400 of the present invention can obtain object state information, which is information on whether the user is located in the preset area and information on the user's movement or breathing, based on the Wi-Fi-based OFDM signal transmitted and received via the transmitting module 420 and the receiving module ().

[0690] According to one embodiment, the receiving module 410 may allow the operation of the environment creation unit 415 when it is determined through the second sensor that the user is located in a previously set area. In other words, the receiving module 410 may allow the operation of the environment creation unit 415 only when it is detected that the user is located in a previously set area 11a. That is, the receiving module 410 may generate environment creation information only when the user is located in a previously set area, thereby controlling the operation of the environment creation unit 415 that performs an environment adjustment operation. The environment creation unit 415 may not perform an operation to change the sleeping environment if the user is not located in a specific position.

[0691] In an additional embodiment, the sensor unit 413 may include one or more environmental sensing modules for acquiring indoor environment information related to the user's sleep environment, such as the user's body temperature, indoor temperature, indoor airflow, indoor humidity, and indoor illuminance. The indoor environment information may be information related to the user's sleep environment, such as information that serves as a reference for considering the influence of external factors on the user's sleep through a sleep state related to changes in the user's sleep stage. The one or more environmental sensing modules may include, for example, at least one sensor module selected from the group consisting of a temperature sensor, an airflow sensor, a humidity sensor, an acoustic sensor, and an illuminance sensor. However, the sensor unit 413 is not limited thereto, and may further include various sensors that can measure the external environment that may affect the user's sleep.

[0692] According to an embodiment of the present invention, the receiving module 410 may include an acoustic collector 414. The acoustic collector 414 may include a small microphone module and may acquire information about acoustics generated in a space where a user sleeps. According to an embodiment, the microphone module included in the acoustic collector 414 may be a relatively small-sized MEMS (Micro-Electro-Mechanical Systems). Such a microphone module is advantageous in terms of cost and can be manufactured in a very small size, but may have a lower signal-to-noise ratio (SNR) than a condenser microphone or a dynamic microphone. A low SNR means that the ratio of noise, which is an acoustic that cannot be identified, is high compared to the acoustic ratio that is intended to be identified, making it difficult to identify the acoustic (i.e., unclear). The information to be analyzed in the present invention may be acoustic information related to the user's breathing and movement acquired during sleep, i.e., sleep acoustic information. Since such sleep sound information is information about very minute sounds such as the user's breathing and movements and is acquired together with other sounds during sleep, it may be very difficult to detect and analyze if acquired through the microphone module having a low signal-to-noise ratio. Therefore, when sleep sound information having a low signal-to-noise ratio is acquired, the reception control unit 416 can process it into data for processing and / or analysis.

[0693] According to an embodiment of the present invention, the receiving module 410 may include an environment creation unit 415. The environment creation unit 415 may adjust the user's sleep environment. Specifically, the environment creation unit 415 may adjust at least one of air quality, illuminance, temperature, wind direction, humidity, and sound in the space where the user is located based on the environment creation information. The environment creation information may be a signal generated by the reception control unit 416 based on the determination of the user's sleep state information. For example, the environment creation information may include information on lowering or increasing illuminance. As a more specific example, the environment creation information may include control information for gradually increasing 3000K white light from 0 lux to 250 lux 30 minutes before the predicted wake-up time. As a further example, the environment creation information may include control information for adjusting at least one of temperature, humidity, wind direction, and sound. The environment creation information may include various information related to fine dust removal, harmful gas removal, allergy care activation, deodorization / sterilization activation, room temperature adjustment, dehumidification adjustment, humidification adjustment, airflow intensity adjustment, selection and adjustment of airflow direction type, driving noise adjustment, vibration adjustment, LED lighting adjustment, etc. The specific descriptions of the environment creation information described above are merely examples, and the present invention is not limited thereto.

[0694] The environment creating unit 415 may perform at least one of illumination control, temperature control, wind direction control, humidity control, and sound control. However, without being limited thereto, the environment creating unit may further perform various control operations that may bring about changes in the user's sleep environment. That is, the environment creating unit 415 may adjust the user's sleep environment by performing various control operations based on the environment control signal from the reception control unit 416.

[0695] In an additional embodiment, the environment creation unit 415 may be implemented through connection via the Internet of Things (IOT). Specifically, the environment creation unit 415 may be implemented through connection with various devices that can change the indoor environment in relation to the space where the user is located. For example, the environment creation unit 415 may be implemented as various smart home appliances such as a smart air conditioner, a smart heater, a smart air purifier, a smart boiler, smart windows, a smart humidifier, a smart dehumidifier, and smart lighting, based on connection via the Internet of Things. The above-described specific description of the environment creation unit is merely an example, and the present invention is not limited thereto.

[0696] According to one embodiment of the present invention, the reception control unit 416 may be configured with one or more cores and may include a processor for data analysis and deep learning, such as a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0697] The reception control unit 416 can read a computer program stored in the memory 412 and perform data processing for machine learning according to an embodiment of the present invention.

[0698] According to an embodiment of the present invention, the receiving control unit 416 may perform calculations for neural network learning, such as processing input data for learning in deep learning (DL), extracting features from the input data, calculating errors, and updating weights of the neural network using backpropagation.

[0699] In addition, at least one of the CPU, GPGPU, and TPU of the reception control unit 416 may process network function learning. For example, the CPU and GPGPU may both process network function learning and data classification using the network function. In addition, in an embodiment of the present invention, processors of multiple computing devices may be used together to process network function learning and data classification using the network function. In addition, a computer program executed in a computing device according to an embodiment of the present invention may be a CPU-, GPGPU-, or TPU-executable program.

[0700] As used herein, the term network function may be used interchangeably with the terms artificial neural network and neural network. As used herein, the term network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of the one or more neural networks.

[0701] As used herein, a model may include a network function. A model may include one or more network functions, in which case the output of the model may be an ensemble of the outputs of the one or more network functions.

[0702] The reception control unit 416 may read a computer program stored in the memory 412 to provide a sleep analysis model according to an embodiment of the present invention. According to an embodiment of the present invention, the reception control unit 416 may perform calculations to calculate environment creation information based on sleep state information. According to an embodiment of the present invention, the reception control unit 416 may perform calculations to train the sleep analysis model.

[0703] According to an embodiment of the present invention, the reception control unit 416 may generally process the overall operation of the sleep environment adjusting device 400. The reception control unit 416 may process signals, data, information, etc. input or output via the components described in detail above, or may run applications stored in the memory 412 to provide or process appropriate information or functions to the user terminal.

[0704] According to an embodiment of the present invention, the reception control unit 416 may acquire acoustic information related to the space in which the user sleeps. Acquiring the acoustic information according to an embodiment of the present invention may involve acquiring or loading acoustic information stored in the memory 412. Acquiring the acoustic information may also involve receiving or loading data from another storage medium, another computing device, or a separate processing module within the same computing device via wired or wireless communication.

[0705] According to an embodiment, the reception control unit 416 may acquire sleep sound information from the living environment sound information. Here, the living environment sound information may be sound information acquired in the user's daily life. For example, the living environment sound information may include various sound information acquired in the user's daily life, such as sound information related to cleaning, sound information related to cooking, sound information related to watching TV, etc.

[0706] Specifically, the reception control unit 416 may identify a singular point at which pre-set pattern information is detected in the living environment sound information. Here, the pre-set pattern information may be related to breathing and movement patterns associated with sleep. For example, in a wakeful state, the entire nervous system is activated, resulting in irregular breathing patterns and frequent body movements. Furthermore, the neck muscles may not be relaxed, resulting in very little breathing noise. On the other hand, when the user sleeps, the autonomic nervous system stabilizes, resulting in regular breathing, less body movements, and louder breathing noise. That is, the reception control unit 416 may identify a time point at which pre-set pattern information associated with regular breathing, less body movements, or less breathing noise is detected in the living environment sound information as a singular point. Furthermore, the reception control unit 416 may acquire sleep sound information based on the living environment sound information acquired based on the identified singular point. The reception control unit 416 may identify a singular point related to the user's sleep time point in the life environment sound information acquired in time series, and acquire sleep sound information based on the singular point.

[0707] For example, the reception control unit 416 may identify a singular point associated with a point in time at which a previously established pattern is identified from the living environment acoustic information, and may acquire sleep acoustic information based on acoustic information acquired after the identified singular point.

[0708] That is, the reception control unit 416 can extract and acquire only sleep sound information from a huge amount of sound information by identifying singular points related to the user's sleep from the living environment sound information. In other words, it can acquire only sounds related to sleep (i.e., sleep sound information) from sounds generated in a space. This automates the process of the user recording their sleep time, providing convenience and contributing to improving the accuracy of the acquired sleep sound information.

[0709] According to the embodiment, the reception control unit 416 may calculate the sleep state information based on the acoustic information. Specifically, the reception control unit 416 may calculate the sleep state information based on the sleep acoustic information of the user acquired via the acoustic collection unit 414.

[0710] In one embodiment, the sleep state information may include information related to whether the user is asleep. Specifically, the sleep state information may include at least one of first sleep state information indicating that the user is before falling asleep, second sleep state information indicating that the user is asleep, and third sleep state information indicating that the user has fallen asleep. In other words, when first sleep state information is acquired related to a user, the reception control unit 416 can determine that the user is in a pre-sleep state (i.e., before falling asleep), when second sleep state information is acquired, the reception control unit 416 can determine that the user is in a sleeping state, and when third sleep state information is acquired, the reception control unit 416 can determine that the user is in a post-sleep state (i.e., woken up).

[0711] The sleep state information may be acquired based on sleep acoustic information, which may include acoustic information acquired while the user is sleeping in a space where the user is located in a non-contact manner.

[0712] According to one embodiment, the reception control unit 416 may calculate sleep state information based on the collected acoustic information (S140). In this embodiment, the reception control unit 416 may acquire sleep state information related to whether the user is about to fall asleep or is asleep based on a singular point identified from the acoustic information. Specifically, the reception control unit 416 may determine that the user is about to fall asleep if no singular point is identified, and may determine that the user is asleep after the singular point if a singular point is identified. Furthermore, the reception control unit 416 may identify a time point (e.g., a wake-up time) at which a previously established pattern is not observed after the singular point is identified, and may determine that the user has fallen asleep, i.e., woken up, if the time point is identified.

[0713] That is, the reception control unit 416 can acquire sleep state information related to whether the user is before sleep, during sleep, or after sleep based on whether a singular point is identified in the acoustic information and whether a previously set pattern is continuously detected after the singular point is identified.

[0714] According to an embodiment of the present invention, the reception control unit 416 may generate environment creation information based on sensing information and sleep state information. Specifically, the reception control unit 416 may generate the environment creation information based on sensing information acquired via the sensor unit 413 and sleep state information acquired as a result of acoustic analysis. The reception control unit 416 generates the environment creation information based on the sensing information and the sleep state information, and transmits the generated environment creation information to the environment creation unit 415, thereby controlling the sleep environment changing operation of the environment creation unit 415.

[0715] In an embodiment, the reception control unit 416 may generate environment creation information based on the sleep state information. The sleep state information is information related to whether the user is sleeping or not, and may include at least one of first sleep state information indicating that the user is about to fall asleep, second sleep state information indicating that the user is sleeping, and third sleep state information indicating that the user has fallen asleep.

[0716] More specifically, the reception control unit 416 may generate the first environment creation information based on the first sleeping state information. Specifically, when the reception control unit 416 acquires first sleeping state information indicating that the user is about to fall asleep, the reception control unit 416 may generate the first environment creation information based on the first sleeping state information. That is, when the user's sleeping state is about to fall asleep, the reception control unit 416 may generate the first environment creation information for supplying a preset white light for a certain period of time.

[0717] According to an embodiment, the sleep induction time point may be determined by the reception control unit 416. Specifically, the reception control unit 416 may determine the sleep induction time point through information exchange with the user's user terminal 10. As a specific example, the user may generate sleep plan information by setting the time at which the user intends to go to sleep and the time at which the user intends to wake up via the user terminal 10, and transmit the generated sleep plan ...

Claims

1. A method for creating an environment for an object through smart lighting, comprising: a sensor; a processor; and a lighting unit, acquiring environmental sensing information in real time in an environment of the object via the sensor; performing pre-processing on the acquired environmental sensing information via the processor to reduce noise; converting sleep sound information, including at least one of information regarding a user's breathing sound or a user's movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms via the processor; generating sleep stage information of the user in real time by processing the converted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model via the processor; generating environment creation information for controlling the lighting unit in real time based on the acquired environment sensing information and the generated sleep stage information through the processor; Including, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less from the lighting unit within a preset period from the time when the sleep stage information is information indicating that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; A method for creating an environment for an object via smart lighting.

2. A method for creating an environment for an object through smart lighting, the method comprising: a sensor; a processor; and a lighting unit, determining a time point for acquiring environmental sensing information based on at least one of an operation input to the user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of the user via the processor; acquiring, via the processor, the environmental sensing information in the object's environment from the determined acquisition time point; performing pre-processing on the acquired environmental sensing information via the processor to reduce noise; converting sleep sound information, including at least one of information regarding a user's breathing sound or a user's movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms via the processor; generating sleep stage information for the user by processing the transformed spectrograms as inputs to a sleep analysis model including a feature extraction model and a feature classification model via the processor; generating environment creation information for controlling the lighting unit based on the acquired environment sensing information and the generated sleep stage information through the processor; Including, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less from the lighting unit within a preset period from the time when the sleep stage information is information indicating that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; A method for creating an environment for an object via smart lighting.

3. A method for creating an environment for an object through smart lighting, comprising: a sensor; a processor; and a lighting unit, determining a time point for acquiring environmental sensing information based on at least one of an operation input to the user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of the user via the processor; acquiring the environmental sensing information in real time in the environment of the object from the determined acquisition time point via the sensor; performing pre-processing on the acquired environmental sensing information via the processor to reduce noise; converting sleep sound information, including at least one of information regarding a user's breathing sound or a user's movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms via the processor; generating sleep stage information of the user in real time by processing the converted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model via the processor; generating environment creation information for controlling the lighting unit in real time based on the acquired environment sensing information and the generated sleep stage information through the processor; Including, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less from the lighting unit within a preset period from the time when the sleep stage information is information indicating that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; A method for creating an environment for an object via smart lighting.

4. each of the plurality of features is extracted by the one or more artificial neural networks included in the feature extraction model based on one or more patterns associated with at least one of a respiratory sound, a respiratory pattern, and a movement pattern of each of the plurality of spectrograms; A method for creating an ambience of an object through smart lighting according to any one of claims 1 to 3.

5. The one or more artificial neural networks included in the feature classification model are configured to simultaneously estimate changes in a plurality of sleep stages during the user's sleep as the sleep stage information by performing multi-epoch classification on at least some of the plurality of features. A method for creating an ambience of an object through smart lighting according to any one of claims 1 to 3.

6. At least some of the features on which the multi-epoch classification is performed are features corresponding to a plurality of epochs in time series.

6. The method for creating an ambience of an object through smart lighting according to claim 5.

7. preprocessing the transformed spectrograms through a deep learning-based noise reduction algorithm via the processor; further comprising the sleep stage information is generated by processing the pre-processed spectrograms as inputs to the sleep analysis model. A method for creating an ambience of an object through smart lighting according to any one of claims 1 to 3.

8. A computer program stored on a computer-readable recording medium so as to be coupled with a computer that is hardware and capable of performing the method of any one of claims 1 to 3.

9. In smart lighting to create an environment for objects, a sensor configured to acquire environmental sensing information in real time in an environment of the object; a processor; Lighting department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; The converted spectrograms are processed as inputs to a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information for the user in real time; and generating environment creation information for controlling the lighting unit in real time based on the acquired environment sensing information and the generated sleep stage information, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; Smart lighting to create an environment for your objects.

10. In smart lighting to create an environment for objects, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in an environment of the object from the determined acquisition time point; Lighting department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; processing the transformed spectrograms as inputs to a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information for the user; and generating environment creation information for controlling the lighting unit based on the acquired environment sensing information and the generated sleep state information, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; Smart lighting to create an environment for your objects.

11. In smart lighting to create an environment for objects, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in real time in an environment of the object from the determined acquisition time; Lighting department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; The converted spectrograms are processed as inputs to a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information for the user in real time; and generating environment creation information for controlling the lighting unit in real time based on the acquired environment sensing information and the generated sleep stage information, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; Smart lighting to create an environment for your objects.

12. In smart lighting to create an environment for objects, a sensor configured to acquire environmental sensing information in real time in an environment of the object; The network department and a processor; Lighting department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; transmitting the converted spectrograms to a server via the network unit; The server processes the transmitted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information in real time, and receives the generated sleep stage information via the network unit; and generating environment creation information for controlling the lighting unit in real time based on the acquired environment sensing information and the received sleep stage information, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; Smart lighting to create an environment for your objects.

13. In smart lighting to create an environment for objects, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in an environment of the object from the determined acquisition time point; The network department and Lighting department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; transmitting the converted spectrograms to a server via the network unit; the server processes the transmitted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information of the user, and receives the generated sleep stage information via the network unit; and generating environment creation information for controlling the lighting unit based on the acquired environment sensing information and the received sleep stage information, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; Smart lighting to create an environment for your objects.

14. In smart lighting to create an environment for objects, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in real time in an environment of the object from the determined acquisition time; The network department and Lighting department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; transmitting the converted spectrograms to a server via the network unit; The server processes the transmitted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information in real time, and receives the generated sleep stage information via the network unit; and generating environment creation information for controlling the lighting unit in real time based on the acquired environment sensing information and the received sleep stage information, each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; Smart lighting to create an environment for your objects.

15. each of the plurality of features is extracted by the one or more artificial neural networks included in the feature extraction model based on one or more patterns associated with at least one of a respiratory sound, a respiratory pattern, and a movement pattern of each of the plurality of spectrograms; Smart lighting for creating an environment for an object according to any one of claims 9 to 14.

16. The one or more artificial neural networks included in the feature classification model are configured to simultaneously estimate changes in a plurality of sleep stages during the user's sleep as the sleep stage information by performing multi-epoch classification on at least some of the plurality of features. Smart lighting for creating an environment for an object according to any one of claims 9 to 14.

17. At least some of the features on which the multi-epoch classification is performed are features corresponding to a plurality of epochs in time series.

17. Smart lighting for creating an ambience of an object according to claim 16.

18. The processor is further configured to preprocess the transformed spectrograms via a deep learning-based noise reduction algorithm; the sleep stage information is generated by processing the pre-processed spectrograms with inputs of the sleep analysis model by the processor. Smart lighting for creating an environment for an object according to any one of claims 9 to 14.

19. An electronic device for controlling smart lighting to create an environment for an object, a sensor configured to acquire environmental sensing information in real time in an environment of the object; Processor and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; The converted spectrograms are processed as inputs to a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information for the user in real time; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the generated sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

20. An electronic device for controlling smart lighting to create an environment for an object, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in an environment of the object from the determined acquisition time point; Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; processing the transformed spectrograms as inputs to a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information for the user; and configured to generate environment creation information for controlling a lighting unit of the smart lighting based on the acquired environment sensing information and the generated sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

21. An electronic device for controlling smart lighting to create an environment for an object, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in real time in an environment of the object from the determined acquisition time; Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; The converted spectrograms are processed as inputs to a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information for the user in real time; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the generated sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

22. An electronic device for controlling smart lighting to create an environment for an object, a sensor configured to acquire environmental sensing information in real time in an environment of the object; The network department and Processor and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; transmitting the converted spectrograms to a server via the network unit; The server processes the transmitted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information in real time, and receives the generated sleep stage information via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

23. An electronic device for controlling smart lighting to create an environment for an object, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in an environment of the object from the determined acquisition time point; Network Department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; transmitting the converted spectrograms to a server via the network unit; the server processes the transmitted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate sleep stage information of the user, and receives the generated sleep stage information via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

24. An electronic device for controlling smart lighting to create an environment for an object, a processor configured to determine a time point for acquiring environmental sensing information based on at least one of an operation input to a user terminal, a start time point input to the user terminal, or a previously stored sleep pattern of a user; a sensor configured to acquire the environmental sensing information in real time in an environment of the object from the determined acquisition time; Network Department and Including, The processor: performing pre-processing on the acquired environmental sensing information to reduce noise; converting sleep sound information, including at least one of information regarding the user's breathing sound or movement sound, included in the pre-processed environmental sensing information into a plurality of spectrograms; transmitting the converted spectrograms to a server via the network unit; The server processes the transmitted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information in real time, and receives the generated sleep stage information via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

25. An electronic device for controlling smart lighting to create an environment for an object, The network department and Processor and Including, The processor: the user terminal acquires environmental sensing information in real time in the environment of the object, converts sleep sound information including at least one of information about the user's breathing sound or movement sound included in the acquired environmental sensing information into a plurality of spectrograms, processes the converted spectrograms as inputs to a sleep analysis model including a feature extraction model and a feature classification model, and generates user sleep stage information in real time; and receives the sleep stage information from the user terminal via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

26. An electronic device for controlling smart lighting to create an environment for an object, The network department and Processor and Including, The processor: the user terminal determines an acquisition time point of environmental sensing information based on at least one of an operation input to the user terminal, a start time point input to the user terminal, or a user's sleep pattern already stored; acquires the environmental sensing information in the environment of the object from the determined acquisition time point; converts sleep sound information including at least one of information regarding the user's breathing sound or movement sound included in the acquired environmental sensing information into a plurality of spectrograms; processes the converted spectrograms as input to a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information; and receives the sleep stage information from the user terminal via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

27. An electronic device for controlling smart lighting to create an environment for an object, The network department and Processor and Including, The processor: the user terminal determines an acquisition time point of environmental sensing information based on at least one of an operation input to the user terminal, a start time point input to the user terminal, or a user's sleep pattern that has already been stored; acquires the environmental sensing information in real time in the environment of the object from the determined acquisition time point; converts sleep sound information including at least one of information regarding the user's breathing sound or movement sound included in the acquired environmental sensing information into a plurality of spectrograms; processes the converted spectrograms as input to a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information in real time; and receives the sleep stage information from the user terminal via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

28. An electronic device for controlling smart lighting to create an environment for an object, The network department and Processor and Including, The processor: the user terminal acquires environmental sensing information in real time in the environment of the object, converts sleep sound information including at least one of information about the user's breathing sound or movement sound included in the acquired environmental sensing information into a plurality of spectrograms, transmits the converted spectrograms to a server, and the server processes the transmitted spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information in real time, and receives the generated sleep stage information via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

29. An electronic device for controlling smart lighting to create an environment for an object, The network department and Processor and Including, The processor: the user terminal determines an acquisition time point of environmental sensing information based on at least one of an operation input to the user terminal, a start time point input to the user terminal, or a user's sleep pattern that has already been stored; acquires the environmental sensing information in the environment of the object from the determined acquisition time point; converts sleep sound information including at least one of information regarding the user's breathing sound or movement sound included in the acquired environmental sensing information into a plurality of spectrograms; transmits the converted plurality of spectrograms to a server; and the server processes the transmitted plurality of spectrograms as input to a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information; and receives the generated sleep stage information via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

30. An electronic device for controlling smart lighting to create an environment for an object, The network department and Processor and Including, The processor: the user terminal determines an acquisition time point of environmental sensing information based on at least one of an operation input to the user terminal, a start time point input to the user terminal, or a user's sleep pattern that has already been stored, acquires the environmental sensing information in real time in the environment of the object from the determined acquisition time point, converts sleep sound information including at least one of information regarding the user's breathing sound or movement sound included in the acquired environmental sensing information into a plurality of spectrograms, transmits the converted plurality of spectrograms to a server, and the server processes the transmitted plurality of spectrograms as inputs of a sleep analysis model including a feature extraction model and a feature classification model to generate user sleep stage information in real time, and receives the generated sleep stage information via the network unit; and configured to generate environment creation information for controlling a lighting unit of the smart lighting in real time based on the acquired environment sensing information and the received sleep stage information; each of the plurality of spectrograms corresponds to a predetermined epoch; the feature extraction model includes one or more artificial neural networks configured to extract a respective plurality of features in response to input of each of the plurality of spectrograms, each of the plurality of features being based on a respective one of the plurality of spectrograms; the feature classification model includes one or more artificial neural networks configured to, in response to an input of the extracted plurality of features, estimate information related to changes in sleep stages during the user's sleep using the sleep stage information; The environmental creation information is (a) first environment creation information for preventing light from being emitted from the lighting unit or for emitting light having at least one of a first illuminance value or less and a first color temperature value or less within a preset period from the time when the sleep stage information indicates that the user has fallen asleep; and (b) the sleep stage information includes at least one of second environment creation information for emitting light having at least one property of a second illuminance value or more and a second color temperature value or more from the lighting unit when the sleep stage information is information that the user has woken up and the information that the user has woken up is generated within a predetermined period from a desired wake-up time that the user has previously set, and the second illuminance value is greater than the first illuminance value, and the second color temperature value is greater than the first color temperature value; An electronic device for controlling smart lighting to create an environment for an object.

31. each of the plurality of features is extracted by the one or more artificial neural networks included in the feature extraction model based on one or more patterns associated with at least one of a respiratory sound, a respiratory pattern, and a movement pattern of each of the plurality of spectrograms; An electronic device for controlling smart lighting to create an environment of an object according to any one of claims 19 to 30.

32. The one or more artificial neural networks included in the feature classification model are configured to simultaneously estimate changes in a plurality of sleep stages during the user's sleep as the sleep stage information by performing multi-epoch classification on at least some of the plurality of features. An electronic device for controlling smart lighting to create an environment of an object according to any one of claims 19 to 30.

33. At least some of the features on which the multi-epoch classification is performed are features corresponding to a plurality of epochs in time series.

33. An electronic device for controlling smart lighting to create an environment of an object according to claim 32.

34. The processor is further configured to preprocess the transformed spectrograms via a deep learning-based noise reduction algorithm; the sleep stage information is generated by processing the pre-processed spectrograms with inputs of the sleep analysis model by the processor. An electronic device for controlling smart lighting to create an environment of an object according to any one of claims 19 to 24.

35. The user terminal is configured to preprocess the converted spectrograms through a deep learning-based noise reduction algorithm; the sleep stage information is generated by processing the pre-processed spectrograms with inputs of the sleep analysis model by the processor. An electronic device for controlling smart lighting to create an environment of an object according to any one of claims 25 to 27.

36. The server is configured to pre-process the transmitted spectrograms through a deep learning-based noise reduction algorithm; the sleep stage information is generated by processing the pre-processed spectrograms with inputs of the sleep analysis model by the processor. An electronic device for controlling smart lighting to create an environment of an object according to any one of claims 28 to 30.

Images