System, program, computer device and method

Abstract

An object of the present invention is to provide a system capable of identifying silent intervals. A system including at least one computer device, the system includes a first identification means for identifying occurrence of snoring sounds or breathing sounds while a user is sleeping based on the sound information while the user is sleeping or the spectrogram into which the sound information is converted, using a prediction model that is machine-learned using input data of sound information including snoring sounds or breathing sounds while a person is sleeping or a spectrogram into which the sound information is converted, and output data of information regarding occurrence of snoring sounds or breathing sounds while the user is sleeping, and a second identification means for identifying whether or not a silent section between adjacent snoring sounds, a silent section between adjacent snoring sounds and breathing sounds, and / or a silent section between adjacent breathing sounds satisfy a predetermined first condition.

Description

[Technical field]

[0001] The present invention relates to a system, a program, a computer device and a method. [Background technology]

[0002] Sleep apnea syndrome, a condition in which breathing stops repeatedly during sleep, increases the incidence of ischemic lung diseases such as hypertension, stroke, and myocardial infarction due to stress caused by hypoxia during sleep and daytime sleepiness. Sleep apnea syndrome is also known to cause complications such as diabetes and hyperlipidemia.

[0003] The evaluation of the user's breathing state during sleep is generally performed by polysomnography (PSG). However, PSG requires hospitalization, which places a heavy burden on the user. In addition, it is necessary to wear various devices, and the user sleeps in a different environment than usual, making it difficult to reproduce normal sleep conditions.

[0004] In view of this, for example, Patent Document 1 discloses a method for evaluating the breathing state of a user by acquiring sounds of a subject while sleeping using a microphone of a mobile device. Patent Document 1 describes that an apnea state can be determined from sounds acquired by the microphone of the mobile device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2013-236925 A Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention can solve, for example, any of the following problems. A first object of the present invention is to provide a system capable of identifying silent intervals. A second object of the present invention is to provide a system capable of identifying the occurrence of apnea or hypopnea during a user's sleep by utilizing sound information. A third object of the present invention is to provide a program that enables a user to recognize the occurrence of apnea or hypopnea during sleep. A fourth object of the present invention is to provide a program that can transmit acquired sound information to a second computer device. [Means for solving the problem]

[0007] The object of the present invention is to [1] A system including at least one computer device, the system including: a first identification means for identifying the occurrence of snoring sounds or breathing sounds while a user is sleeping based on the sound information while the user is sleeping or the spectrogram into which the sound information is converted, using a machine-learned prediction model with input data being sound information including snoring sounds or breathing sounds while a user is sleeping or output data being information regarding the occurrence of snoring sounds or breathing sounds while the user is sleeping, and a second identification means for identifying whether or not a silent interval between adjacent snoring sounds, a silent interval between adjacent snoring sounds and breathing sounds, and / or a silent interval between adjacent breathing sounds satisfy a predetermined first condition; [2] The system according to [1], further comprising a removal means for removing noise from sound information of the user while sleeping, and the first identification means identifies the occurrence of breathing sounds or snoring sounds while the user is sleeping based on the sound information from which the noise has been removed or a spectrogram obtained by converting the sound information from which the noise has been removed; [3] The system according to [1] or [2], wherein the first identification means identifies the occurrence of snoring sounds or breathing sounds based on sound information for a time period that satisfies a predetermined second condition or a spectrogram obtained by converting the sound information; [4] The system according to any one of [1] to [3] above, further comprising a third identification means for identifying the occurrence of apnea or hypopnea in a silent section satisfying a predetermined condition, and a calculation means for calculating the total, average or maximum duration of apnea and / or hypopnea, the total number of apneas and / or hypopneas, the number of apneas and / or hypopneas per unit time, and / or the total or average duration of apnea and / or hypopnea per unit time, based on information on the silent section in which the identified apnea or hypopnea is occurring; [5] The system according to any one of [1] to [4], wherein the snoring or breathing sounds during the person's sleep occur immediately before an apnea or hypopnea occurs, or immediately after an apnea or hypopnea occurs; [6] A computer device, comprising: a first identification means for identifying the occurrence of snoring sounds or breathing sounds while a user is sleeping, based on the sound information while the user is sleeping, or a spectrogram obtained by converting the sound information, using a machine-learned prediction model with input data being sound information including snoring sounds or breathing sounds while the user is sleeping, and output data being information regarding the occurrence of snoring sounds or breathing sounds while the user is sleeping, based on the sound information while the user is sleeping, or a spectrogram obtained by converting the sound information; and a second identification means for identifying whether or not a silent interval between adjacent snoring sounds, a silent interval between adjacent snoring sounds and breathing sounds, and / or a silent interval between adjacent breathing sounds satisfy a predetermined condition. to act as,program; [7] A method executed on at least one computer device, comprising: a first identification step of identifying occurrences of snoring sounds or breathing sounds while a user is sleeping based on the sound information while the user is sleeping or a spectrogram into which the sound information is converted, using a machine-learned prediction model with input data being sound information including snoring sounds or breathing sounds while a user is sleeping or output data being information regarding the occurrences of snoring sounds or breathing sounds while the user is sleeping, based on the sound information while the user is sleeping or a spectrogram into which the sound information is converted; and a second identification step of identifying whether or not a silent interval between adjacent snoring sounds, a silent interval between adjacent snoring sounds and breathing sounds, and / or a silent interval between adjacent breathing sounds satisfy a predetermined condition; [8] A system including at least one computer device, comprising: a means for identifying occurrences of apnea or hypopnea during a user's sleep based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a machine-learned prediction model that uses as input data sound information during a person's sleep or a spectrogram obtained by converting the sound information, and as output data information regarding occurrences of apnea or hypopnea during the sleep.; [9] A program that causes a computer device to function as an identification means for identifying the occurrence of apnea or hypopnea during a user's sleep based on the sound information during the user's sleep or a spectrogram converted from the sound information, using a machine-learned prediction model that uses as input data sound information during a person's sleep or a spectrogram converted from the sound information, and as output data information regarding the occurrence of apnea or hypopnea during the sleep;

[10] A method executed on at least one computer device, comprising: a step of identifying occurrences of apnea or hypopnea during a user's sleep based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a machine-learned prediction model that uses as input data sound information during the user's sleep or a spectrogram obtained by converting the sound information, and as output data information regarding occurrences of apnea or hypopnea during the user's sleep;

[11] A program that causes a computer device to function as a display means for displaying an area corresponding to when apnea or hypopnea occurs within an area corresponding to when the user is sleeping;

[12] The program according to

[11] , wherein the display means displays the area corresponding to the user's sleep in an arc-like or band-like shape;

[13] The program according to

[11] or

[12] , wherein the display means displays the area corresponding to the occurrence of apnea or hypopnea in different display modes depending on the duration of the apnea and / or hypopnea and / or the number of apnea and / or hypopnea per unit time or the total or average duration of the apnea and / or hypopnea per unit time;

[14] The program according to any one of

[11] to

[13] , wherein the display means further displays information indicating the total, average or longest duration of apnea and / or hypopnea, the total number of apneas and / or hypopneas, and / or the number of apneas and / or hypopneas per unit time or the total or average duration of apneas and / or hypopneas per unit time;

[15] A computer device including a display means for displaying an area corresponding to when apnea or hypopnea occurs within an area corresponding to when a user is sleeping;

[16] A method, executed on at least one computer device, comprising: displaying an area corresponding to when an apnea or hypopnea is occurring within an area corresponding to when the user is sleeping;

[17] A program for making a computer device function as an acquisition means for acquiring sound information of a user while sleeping, a generation means for generating sound files based on the acquired sound information, and a transmission means for transmitting the generated sound files to a second computer device, in which the generation means generates a plurality of sound files having different times at which the sound information was acquired, based on the sound information of a single time during which the user sleeps;

[18] The program according to

[17] , wherein the transmitting means transmits the generated sound file in association with information capable of identifying the order in which the sound information was acquired;

[19] The program described in

[17] or

[18] , wherein the transmitting means transmits the generated sound file when a specified communication line is detected, when the duration of the generated sound file is less than a specified time, or when the data amount of the sound file is less than a specified amount.

[20] The program according to any one of

[17] to

[19] , further comprising a notification means for notifying the second computer device of notification information indicating the presence of an unsent sound file in response to detection of a predetermined communication line when an unsent sound file exists in the second computer device;

[21] The program according to any one of

[17] to

[20] , wherein the generating means generates a plurality of sound files according to the duration of the sound files or the data amount of the sound files;

[22] The program according to any one of

[17] to

[21] , wherein the sound information is used to identify the occurrence of apnea or hypopnea during the user's sleep;

[23] The program according to any one of

[17] to

[22] , wherein the acquisition means acquires sound information for generating sound files to be generated after the one sound file in parallel with the generation of the one sound file by the generation means and / or the transmission of the one sound file by the transmission means;

[24] The program according to any one of

[17] to

[23] , wherein the generating means generates sound files of different durations or amounts of data depending on the computer device that executes the generating means;

[25] A computer device comprising: an acquisition means for acquiring sound information of a user while sleeping; a generation means for generating a sound file based on the acquired sound information; and a transmission means for transmitting the generated sound file to a second computer device, wherein the generation means generates a plurality of sound files having different times at which the sound information was acquired based on the sound information of the user during one sleep period;

[26] A method executed in at least one computer device, the method comprising: an acquisition step of acquiring sound information of a user while sleeping; a generation step of generating a sound file based on the acquired sound information; and a transmission step of transmitting the generated sound file to a second computer device, the generation step generating a plurality of sound files in which the sound information was acquired at different times based on the sound information of the user during one period of sleep;

[27] A system comprising a first computer device and a second computer device, wherein the first computer device comprises an acquisition means for acquiring sound information of a user while sleeping, a generation means for generating a sound file based on the acquired sound information, and a transmission means for transmitting the generated sound file to the second computer device, wherein the generation means generates a plurality of sound files having different times at which the sound information was acquired based on the sound information of the user during one sleep session, and the second computer device comprises an order identification means for identifying the order of the sound files based on information capable of identifying the order in which the sound information was acquired;

[28] A system having at least one computer device, the system comprising: a first identification means for identifying whether a silent section between adjacent snoring sounds, between adjacent snoring sounds and breathing sounds, and / or between adjacent breathing sounds in sound information of a user while sleeping satisfies a predetermined condition; and a second identification means for identifying the occurrence of apnea or hypopnea during the user's sleep in the silent section identified by the first identification means as satisfying the predetermined condition based on the sound information of the user while sleeping or the spectrogram converted from the sound information using a machine-learned prediction model with input data being sound information of a person while sleeping or a spectrogram converted from the sound information and output data being information regarding the occurrence of apnea or hypopnea during the sleep;

[29] A program that causes a computer device to function as a first identification means for identifying whether a silent section between adjacent snoring sounds, between adjacent snoring sounds and breathing sounds, and / or between adjacent breathing sounds in sound information of a user while sleeping satisfies a predetermined condition, and a second identification means for identifying the occurrence of apnea or hypopnea during the user's sleep, based on the sound information of the user while sleeping or a spectrogram converted from the sound information, in a silent section identified by the first identification means as satisfying the predetermined condition, using a machine-learned prediction model that uses sound information of a person while sleeping or a spectrogram converted from the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data;

[30] A method executed on at least one computer device, the method comprising: a first identification step of identifying whether a silent section between adjacent snoring sounds, between adjacent snoring sounds and breathing sounds, and / or between adjacent breathing sounds in sound information of a user while sleeping satisfies a predetermined condition; and a second identification step of identifying occurrences of apnea or hypopnea during the user's sleep in the silent section identified in the first identification step as satisfying the predetermined condition based on the sound information of the user while sleeping or the spectrogram converted from the sound information, using a machine-learned prediction model with input data being sound information of a person while sleeping or a spectrogram converted from the sound information, and output data being information regarding occurrences of apnea or hypopnea during the sleep; This can be solved by: Effect of the Invention

[0008] The present invention has, for example, any of the following effects. According to the present invention, it is possible to provide a system capable of identifying silent intervals. According to the present invention, it is possible to provide a system capable of identifying the occurrence of apnea or hypopnea during a user's sleep by utilizing sound information. According to the present invention, it is possible to provide a program that enables a user to recognize the occurrence of apnea or hypopnea during sleep. According to the present invention, it is possible to provide a program that is capable of transmitting acquired sound information to a second computer device. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of a system according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram showing a hardware configuration of a user terminal according to an embodiment of the present invention. FIG. [Diagram 3] 2 is a block diagram showing a hardware configuration of a server device according to an embodiment of the present invention. FIG. [Figure 4] FIG. 4 is a flowchart illustrating a transmission process according to an embodiment of the present invention. [Diagram 5] FIG. 4 is a flowchart illustrating an analysis process according to an embodiment of the present invention. [Figure 6] 1 is a diagram showing a flowchart of an apnea etc. identification process according to an embodiment of the present invention. [Figure 7] 11 is a diagram for explaining an apnea etc. identification process according to an embodiment of the present invention. FIG. [Figure 8] 11 is a diagram for explaining an apnea etc. identification process according to an embodiment of the present invention. FIG. [Figure 9] FIG. 13 is a flowchart illustrating another example of the apnea, etc. identification process according to the embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating an example of a display screen according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following describes an embodiment of the present invention, but the present invention is not limited to the following embodiment, as long as it does not go against the spirit of the present invention. The order of each process constituting the flowchart described below is random as long as there is no contradiction or inconsistency in the process content, and it is also possible to omit a part of each process constituting the flowchart or add a new process to each process constituting the flowchart, as long as there is no contradiction or inconsistency in the process content. In addition, the device that executes each process constituting the flowchart can be changed to another device, as long as it does not go against the spirit of the present invention. In that case, the process content can be changed so that there is no contradiction or inconsistency in the process content.

[0011] [System configuration] FIG. 1 is a block diagram showing a configuration of a system according to an embodiment of the present invention. The system 10 includes at least one computer device. The system 10 includes a user terminal 1 and a server device 2. The user terminal 1 is operated by a user. The server device 2 may be managed by an administrator who manages the system 10. Although not shown, the system 10 may include an administrator terminal operated by an administrator who manages the server device 2. The system 10 may also include a computer device that builds a prediction model, which will be described later. The user terminal 1 is an example of a first computer device, and the server device 2 is an example of a second computer device.

[0012] The user terminal 1 and the server device 2 are communicatively connected to each other via a communication network 3. In the system 10, either the user terminal 1 or the server device 2 can function as an information processing device. When either the user terminal 1 or the server device 2 functions as an information processing device, transmission and reception of information is performed between the user terminal 1 and the server device 2 as necessary.

[0013] The system 10 may include two or more user terminals 1. The system 10 may include two or more server devices 2. The server devices 2 may function in a distributed manner across a plurality of computer devices. For example, instead of the server devices 2, a distributed ledger technology such as a blockchain may be used.

[0014] [System Overview] The system 10 according to this embodiment can include a transmission process in the user terminal 1 that acquires sounds made by the user while the user is sleeping as sound information and transmits the sound information to the server device 2, and an analysis process in the server device 2 that analyzes the transmitted sound information.

[0015] The sound information is information obtained by converting sounds around the user terminal 1 collected by the acquisition unit 14 of the user terminal 1 into electrical signals. The sounds around the user terminal 1 include not only the voices emitted by the user but also environmental sounds (noises) such as the operating sounds of electronic devices such as air conditioners.

[0016] The sound information acquired by the acquisition unit 14 is transmitted to the server device 2 as a sound file. The sound information acquired in one measurement may be divided into two or more sound files. A sound file is a collection of data that allows the acquired sound information (electrical signals) to be managed and stored. The file format of the sound file is WAVE, AIFF, MP3, AAC, FLAC, etc. Sound files will be described later.

[0017] In the analysis process, the server device 2 can remove noise such as environmental sounds from the sound information included in the sound file, and can identify the occurrence of breathing sounds and snoring sounds from the sound information included in the sound file. In addition, in the analysis process, the server device 2 can identify the occurrence of apnea or hypopnea during the user's sleep from the sound information included in the sound file. In addition, in the analysis process, the server device 2 can identify the type of apnea or hypopnea during the user's sleep from the sound information included in the sound file.

[0018] The analysis process also includes a display process for displaying the analysis results on the user terminal 1. The display process allows the user to visually recognize the occurrence of apnea or hypopnea while the user is sleeping.

[0019] In this embodiment, the occurrence of "apnea" refers to the user's cessation of breathing during sleep for a predetermined period of time (e.g., 10 seconds) or more, and the occurrence of "hypopnea" refers to the oxygen saturation (SpO2) during sleep decreasing by a predetermined percentage (e.g., 3%) or more from the average oxygen saturation of a single user during a single sleep period.

[0020] The occurrence status of apnea or hypopnea (also called apnea, etc.) includes, for example, information on the time when apnea or hypopnea occurred, the duration of apnea or hypopnea, whether apnea or hypopnea occurred, and / or the type of apnea or hypopnea. The occurrence status of apnea, etc. can be obtained from the occurrence of apnea, etc. during the user's sleep identified in the analysis process and / or the type of apnea, etc.

[0021] [User device] The user terminal 1 is not particularly limited as long as it is a computer device equipped with an acquisition device capable of acquiring sound information and a display device. Examples of the user terminal 1 include conventional mobile phones, tablet terminals, smartphones, etc. In addition, the user terminal 1 may be, for example, a computer device that can be placed next to the user while the user is sleeping.

[0022] 2 is a block diagram showing a hardware configuration of a user terminal according to an embodiment of the present invention. The user terminal 1 includes a control unit 11, a RAM 12, a storage unit 13, an acquisition unit 14, an input unit 15, a display unit 16, and a communication interface 17, which are connected to each other via an internal bus.

[0023] The control unit 11 is composed of a CPU and a ROM. The control unit 11 executes programs stored in the storage unit 13 and controls the user terminal 1. The RAM 12 is a work area for the control unit 11. The storage unit 13 is a storage area for saving programs and data. In other words, the storage unit 13 functions as a recording medium that stores programs. The control unit 11 performs arithmetic processing based on the programs and data read from the RAM 12, the acquisition unit 14, and data input by the input unit 15.

[0024] The acquisition unit 14 converts sounds around the user terminal 1 into electrical signals and acquires them as sound information. The acquisition unit 14 has, for example, a microphone. The acquisition unit 14 is an acquisition device capable of acquiring sound information. Note that the acquisition unit 14 may be provided in the user terminal 1 by attaching an external microphone.

[0025] The display unit 16 has a display screen. The control unit 11 outputs a video signal for displaying an image on the display screen according to the result of the arithmetic processing. Here, the display screen of the display unit 16 may be a touch panel equipped with a touch sensor. In this case, the touch panel functions as the input unit 15.

[0026] The communication interface 17 can be connected to the communication network 3 wirelessly or via a wire, and can transmit and receive data to and from other computer devices via the communication network 3. Data received via the communication interface 17 is loaded into the RAM 12, and the control unit 11 performs arithmetic processing on the data.

[0027] [Server device] 3 is a block diagram showing a hardware configuration of a server device according to an embodiment of the present invention. The server device 2 includes at least a control unit 21, a RAM 22, a storage unit 23, and a communication interface 24, which are connected to each other via an internal bus.

[0028] The control unit 21 is composed of a CPU and a ROM, and executes programs stored in the storage unit 23 to control the server device 2. The control unit 21 also has an internal timer that measures time. The RAM 22 is the work area of ​​the control unit 21. The storage unit 23 is a storage area for saving programs and data. In other words, the storage unit 23 functions as a recording medium that stores programs. The control unit 21 reads out the programs and data from the RAM 22, and performs program execution processing based on information received from the user terminal 1, etc.

[0029] The program may be stored in a recording medium such as a CD-ROM. In this case, the program stored in the recording medium may be installed in the user terminal 1 or the server device 2 to execute a predetermined function.

[0030] Alternatively, the program may be distributed from a computer device outside the system. In this case, the program distributed from the computer device outside the system may be installed in the user terminal 1 or the server device 2 to execute a predetermined function.

[0031] [Initial Settings] In the system 10 of this embodiment, information about a user who uses the system 10 is registered in advance in the server device 2. When a user logs in to the system 10 for the first time, the user operates the user terminal 1 and performs initial settings by inputting information about the user, etc.

[0032] First, the user starts an application program (hereinafter, referred to as a dedicated app) downloaded to the user terminal 1 and accesses the server device 2, thereby logging in to the system 10. Alternatively, the user may log in to the system 10 by accessing the server device 2 from the user terminal 1 via a web browser.

[0033] When logging in to the system 10, the user inputs user information about the user, such as user name, password, gender, date of birth, medical history, occupation, company, etc., as initial settings. Next, the input user information is sent to the server device 2, where the user is registered. It is preferable that the registered user is given user identification information (hereinafter also referred to as a user ID) for identifying the user. The user ID may be stored in association with the user information. Furthermore, in the system 10, a page dedicated to the registered user is created. Even if the user uses multiple terminals, the same page can be displayed using the same user ID.

[0034] When logging in to the system 10, the user may be required to input a pre-registered user ID and password. Alternatively, when logging in to the system 10, the user may be required to input a pre-registered email address and password.

[0035] [Transmission process] Next, a transmission process related to the system 10 of this embodiment will be described. In the transmission process, when transmitting sound information of the user during sleep to the server device 2, the user terminal 1 generates a sound file based on the acquired sound information and transmits the generated sound file to the server device 2. The transmitted sound information is used in an analysis process described later to identify the occurrence of apnea or hypopnea during the user's sleep.

[0036] In the transmission process, the user performs the following operations. Before going to sleep, the user starts a dedicated application from the user terminal 1. Next, the user inputs a command to start acquiring sound information to the user terminal 1 and starts sleeping. Then, when the user wakes up, the user inputs a command to end acquiring sound information to the user terminal 1.

[0037] 4 is a diagram showing a flowchart of a transmission process according to an embodiment of the present invention. A user operates the input unit 15 of the user terminal 1 to start a dedicated application. The user terminal 1 accesses the server device 2 to log in to the system 10. Logging in to the system 10 may also be performed by accessing the server device 2 from the user terminal 1 via a web browser.

[0038] When logging in to the system 10, the user inputs the start of acquisition of sound information on the user terminal 1. For example, the user can input the start of acquisition of sound information by selecting a "start recording" icon displayed on the user terminal 1. When the user terminal 1 accepts the input to start acquisition, the acquisition unit 14 acquires sound information of the user while he or she is sleeping (step S1).

[0039] After the user terminal 1 starts acquiring sound information, if a predetermined condition is satisfied (step S2), a sound file is generated in the user terminal 1 based on the acquired sound information (step S3). The generated sound file is transmitted from the user terminal 1 to the server device 2 (step S4). The server device 2 receives the sound file (step S5) and stores the sound file (step S6). In step S6, the sound file is stored in the storage unit 23 of the server device 2 in association with information capable of identifying the order in which the sound information was acquired. The sound file may also be stored in association with a user ID.

[0040] As described above, the sound file is a collection of data that can manage and store the sound information acquired in step S1. In step S3, the sound file may be generated so that the file has a predetermined time or a predetermined amount of data. The sound files are generated in order starting from the file that includes the sound information acquired the earliest.

[0041] Furthermore, the first sound file and the second sound file generated immediately after the first sound file may contain sound information acquired at different times, or may contain sound information acquired at the same part of the time. Specifically, a first sound file containing sound information from 23:30 to 0:00 and a second sound file containing sound information from 0:00 to 0:30 may be generated, or a first sound file containing sound information from 23:30 to 0:00 and a second sound file containing sound information from 23:55 to 0:25 may be generated.

[0042] The predetermined condition in step S2 includes, for example, that a predetermined time has passed since the start of acquisition of sound information, that a predetermined time has passed since the final time of sound information in a sound file that has been generated or is to be generated, that the data amount of acquired sound information has exceeded a predetermined amount, that the data amount of sound information that has not yet been converted into a sound file has exceeded a predetermined data amount, and / or that an operation to end acquisition of sound information has been performed on the user terminal 1. In step S3, specifically, a sound file may be generated every 30 minutes after sound information is acquired by the user terminal 1, and the sound file may be generated so that the data amount of the sound file is within 15 MB. In step S3, the system 10 of this embodiment can generate multiple sound files according to the time of the sound file or the data amount of the sound file.

[0043] The predetermined conditions for generating a sound file can be set appropriately, but may be set in advance by an administrator inputting into the administrator terminal or a user inputting into the user terminal 1.

[0044] An operation for ending acquisition of sound information in the user terminal 1 is, for example, an operation in which the user selects a "stop recording" icon displayed on the user terminal 1 after waking up. This operation allows input of the end of acquisition of sound information. When input of the end of acquisition of sound information is input, a sound file can be generated for the sound information being acquired even before a predetermined time has elapsed or before the data amount exceeds a predetermined data amount. Therefore, the time or data amount of the last sound file generated may differ from the time or data amount of other sound files. Note that end information may be input to the user terminal 1 by accepting an input of the end of acquisition of sound information at the user terminal 1. At this time, the end information is included in the sound information.

[0045] The process of steps S2 to S6 is repeated until all sound information during the user's sleep time is processed. The system 10 of this embodiment can generate multiple sound files with different times when the sound information was acquired based on the sound information during one sleep time of the user. The transmission process is completed after steps S1 to S6.

[0046] Here, the process of step S4 will be described. The user terminal 1 may transmit the generated sound file in association with information capable of identifying the order in which the sound information was acquired. The information capable of identifying the order in which the sound information was acquired is, for example, time information relating to the time when the sound information included in the sound file was acquired, order information relating to the order in which the sound file was generated, time information when the sound file was transmitted to the server device 2 in step S4, etc.

[0047] Alternatively, the information capable of identifying the order in which the sound information was acquired may be, for example, time information at which the server device 2 received the sound file in step S5. In this case, in step S4, the generated sound file does not need to be transmitted in association with information capable of identifying the order in which the user terminal 1 acquired the sound information.

[0048] In addition, regarding the process of acquiring sound information in step S1, the user terminal 1 may acquire sound information for generating sound files to be generated after the first sound file in parallel with the process of generating one sound file in step S3 and / or the process of transmitting one sound file in step S4. Also, the process of generating sound files in step S3 may be executed according to the time series in which the sound information was acquired.

[0049] For example, if acquisition of sound information starts at 23:00 and sound files are set to be generated every 30 minutes, at 23:30, the user terminal 1 generates a sound file (called the first sound file) containing sound information from 23:00 to 23:30. The user terminal 1 also acquires sound information from 23:30 onwards in parallel with generating the first sound file. Furthermore, once the first sound file is generated, the user terminal 1 also acquires sound information from 23:30 onwards in parallel with transmitting the first sound file.

[0050] Furthermore, when the time reaches 0:00 while the first sound file is being generated, the user terminal 1 may generate a sound file (referred to as a second sound file) including sound information from 23:30 to 0:00 in parallel with the generation of the first sound file. Also, when the first sound file is generated while the second sound file is being generated, the user terminal 1 may transmit the first sound file in parallel with the generation of the second sound file. That is, regarding the process of generating a sound file in step S3, the user terminal 1 may generate a sound file that is generated after the first sound file in parallel with the process of generating one sound file in step S3 and / or the process of transmitting one sound file in step S4. Also, regarding the process of transmitting a sound file in step S4, the user terminal 1 may transmit multiple sound files simultaneously.

[0051] Alternatively, after the acquisition of sound information is completed, the system 10 may generate a plurality of sound files based on the acquired sound information, the sound files having different acquisition times.

[0052] In addition, the sound file transmitted to the server device 2 in step S4 may be deleted from the user terminal 1 or may be stored in the storage unit 13 of the user terminal 1.

[0053] In addition, in step S3, the system 10 of the present embodiment may generate sound files of different durations or data amounts depending on the user terminal 1 and / or server device 2 that execute the process of generating the sound file. Specifically, if the data capacity of the user terminal 1 that executes the generation process is less than a predetermined capacity, a sound file of shorter duration or a sound file of smaller data amount than normal may be generated. Also, if the processing speed of the server device 2 that receives the sound file is slower than a predetermined processing speed, a sound file of shorter duration or a sound file of smaller data amount than normal may be generated. In other words, the predetermined condition in step S2 may be changed depending on the user terminal 1 and / or server device 2 that executes the generation process. Note that information regarding the user terminal 1 and / or server device 2 may be stored in the user terminal 1 in advance, or may be acquired by the user terminal 1 before the transmission process is executed.

[0054] Furthermore, in step S3, the system 10 of the present embodiment may generate sound files of different durations or amounts of data depending on the time period in which the generation process is executed. Specifically, the system 10 may generate sound files that are shorter than normal or have a smaller amount of data than normal during times when the lines are congested (e.g., 10:00-12:00), and generate sound files that are longer than normal or have a larger amount of data than normal during times when the lines are not congested (e.g., 12:00-5:00). In other words, the predetermined conditions in step S2 may be changed depending on the time period in which the generation process is executed.

[0055] Note that, before starting to acquire the sound information in step S1, instruction information for the user may be displayed. For example, the instruction information may include, for the user terminal 1, a caution regarding the location where the user terminal 1 is placed, a caution regarding environmental sounds such as turning off the TV in the room, a caution regarding the state of the user terminal 1 such as connecting the user terminal 1 to a power source, dimming the brightness of the display screen of the user terminal 1, or checking the data capacity of the user terminal 1. In addition, the system 10 of this embodiment may control the brightness of the display screen of the user terminal 1 to be dimmed when the acquisition of the sound information in step S1 is started. At this time, the system 10 may control the brightness of the display screen of the user terminal 1 to be the dimmest, or may control it to be dimmed depending on the brightness of the room.

[0056] In step S4, the user terminal 1 may transmit the sound file in response to satisfying a predetermined transmission condition, such as detection of a predetermined communication line, the duration of the generated sound file being equal to or less than a predetermined time, the amount of data in the sound file being equal to or less than a predetermined amount, or the user inputting an operation into the user terminal 1.

[0057] Detecting a predetermined communication line means, for example, detecting a line with no data capacity limit such as Wi-Fi. If the predetermined transmission condition is that a predetermined communication line has been detected, the user terminal 1 may be configured not to generate a sound file in step S3, and to transmit the acquired sound information as one sound file in step S4 without dividing the acquired sound information.

[0058] In addition, in the transmission process, steps S2 and S3 may be repeatedly executed, and the user terminal 1 may transmit the sound file when the above-mentioned predetermined transmission condition is satisfied.

[0059] Furthermore, before executing step S4, the user terminal 1 may output instruction information prompting the user to transmit a sound file in response to detecting a predetermined communication line. In response to an operation input by the user to the user terminal 1, the user terminal 1 may transmit the sound file.

[0060] Furthermore, when there is a sound file that has not been transmitted to the server device 2, the system 10 may notify notification information indicating the presence of an untransmitted sound file in response to a predetermined notification condition being satisfied. The predetermined notification condition may be, for example, that a predetermined communication line has been detected or that a predetermined time (for example, one day) has passed since the sound information was obtained.

[0061] When the system 10 detects a specified communication line, it can send a sound file or indicate that there is an unsent sound file, thereby encouraging the user to send a sound file without increasing the data usage by the user terminal 1 that is borne by the user by sending the sound file.

[0062] [Analysis process] Next, the analysis process according to the system 10 of the present embodiment will be described. The analysis process is executed after all sound files of the user during sleep are stored in the server device 2 by the transmission process. Alternatively, the analysis process may be executed every time a sound file is stored in the server device 2.

[0063] The analysis process in the server device 2 can specify, for example, the length of time that one snoring sound lasted, the total or longest length of time that one snoring sound lasted, the total number of times that snoring sounds occurred, the number of times that snoring sounds occurred per unit time or the total or average length of time that they lasted per unit time, the length of time that one breathing sound lasted, the total number of times that breathing sounds occurred, the number of times that breathing sounds occurred per unit time or the total or average length of time that they lasted per unit time, the time that one apnea or hypopnea occurred, the length of time that one apnea or hypopnea lasted, the total or longest length of time that apnea or hypopnea lasted, the total number of apneas or hypopneas, the number of apneas or hypopneas per unit time or the total or average length of time that they lasted per unit time, etc. The above information specified by the analysis process is also called an analysis result.

[0064] 5 is a diagram showing a flowchart of an analysis process according to an embodiment of the present invention. The server device 2 identifies the order of the sound files (step S11). Next, the server device 2 removes noise from the sound information included in one sound file (step S12).

[0065] Here, noise is a sound other than the voice uttered by the user. For example, noise is environmental sound such as the operating sound of electronic devices such as an air conditioner, the naturally occurring sounds of rain and wind, and the sound of a user turning over in bed. Noise may be a sound other than the voice uttered by the user that has a specific pattern (a certain frequency or periodicity). Voice uttered by the user is, for example, breathing sounds such as breathing in sleep and snoring sounds. Furthermore, even voices uttered by the user, sneezing and talking in their sleep may be considered as noise.

[0066] The noise removal method can employ a method for removing known noise components, and can be performed, for example, as follows. The server device 2 removes environmental sounds from the sound information contained in the sound file. The environmental sounds can be removed by a method for suppressing known noise components. For example, filtering such as spectral subtraction, spectral restoration, human voice model, and the like can be employed. When using spectral subtraction, for example, Wiener filtering can be used.

[0067] Next, the server device 2 identifies events that are candidates for breathing sounds and snoring sounds from the sound information from which the environmental sounds have been removed (step S13). Step S13 is a process of identifying candidates for time periods (sections) corresponding to the breathing sounds and snoring sounds of the user from the sound information. Step S13 is also a process of identifying time periods (sections) that satisfy a predetermined condition (also referred to as a predetermined second condition) from the sound information from which the environmental sounds have been removed. In this embodiment, a time period in the sound information that satisfies the predetermined second condition is called an event. The predetermined second condition is, for example, that the volume of the sound information from which the environmental sounds have been removed is equal to or greater than a predetermined volume, and a time period in which the volume is equal to or greater than the predetermined volume can be defined as an event. For example, a predetermined volume is set in advance in the server device 2, and an event can be identified for sound information from which the environmental sounds have been removed.

[0068] In addition, the volume of the same breathing sound and the volume of the same snoring sound are different when the user terminal 1 acquires a voice uttered by the user at a position close to the user and when the user terminal 1 acquires a voice uttered by the user at a position far from the user. Even if the user terminal 1 is placed in the same position and acquires a voice, the position of the user changes due to reasons such as turning over in bed. Therefore, events that are candidates for breathing sounds and snoring sounds may be identified based on the relative magnitude of the volume. Furthermore, events that are candidates for breathing sounds and snoring sounds may be identified based on not only the volume of the sound information from which environmental sounds have been removed, but also other elements constituting the sound information, such as frequency and sound duration.

[0069] Next, the server device 2 executes an apnea etc. identification process for identifying the occurrence of apnea or hypopnea (step S14). Then, information relating to the identified occurrence of apnea etc. is stored in the server device 2 (step S15). Information relating to the occurrence of apnea etc. may be stored in the server device 2 in association with a user ID. The processes of steps S12 to S15 are repeatedly executed until all sound files acquired while the user was sleeping have been processed.

[0070] When all sound files have been processed, the server device 2 generates display information for displaying the analysis results on the user terminal 1 (step S16), and transmits the display information to the user terminal 1 (step S17). When the display information is received by the user terminal 1 (step S18), the display information is displayed on the user terminal 1 (step S19). The analysis process is completed by the above steps S11 to S19.

[0071] Here, the process of identifying the order of the sound files in step S11 will be described. In step S6, the sound files and information capable of identifying the order in which the sound information was acquired are stored in association with each other in the server device 2, so the server device 2 can identify the order of the sound files based on the information capable of identifying the order in which the sound information was acquired. The sound files whose order is identified are sound files corresponding to the sound information acquired during one sleep period of the user. Specifically, the sound files corresponding to the sound information acquired during one sleep period of the user are sound files corresponding to the sound information acquired from the start of the acquisition of the sound information by the user terminal 1 until the end of the acquisition of the sound information is received.

[0072] The order of the sound files is determined by the chronological order in which the sound information was acquired. For example, if sound information from 23:00 to 1:00 is acquired and sound files are generated every 30 minutes, four sound files will be generated. In this case, the order of the sound files will be as follows: sound files from 23:00 to 23:30, sound files from 23:30 to 0:00, sound files from 0:00 to 0:30, and sound files from 0:30 to 1:00.

[0073] Therefore, if the information capable of identifying the order in which sound information was acquired is time information related to the time when sound information included in the sound files was acquired, the order of the sound files can be identified based on the time when the sound information was acquired. Also, if the information capable of identifying the order in which sound information was acquired is order information related to the order in which the sound files were generated, and the sound files are generated in the order in which the sound information was acquired, the order of the sound files can be identified based on the order in which the sound files were generated. Also, if the information capable of identifying the order in which sound information was acquired is time information when the sound files were sent to the server device 2 in step S4, and the sound files are generated in the order in which the sound information was acquired and sent to the server device 2 immediately after generation, the order of the sound files can be identified based on the time when the sound files were sent to the server device 2.

[0074] Furthermore, if the information capable of identifying the order in which the sound information was acquired is time information relating to the time when the sound files were received by the server device 2, the order of the sound files can be identified based on the time when the sound files were received.

[0075] In the system 10 of the present embodiment, in step S11, the server device 2 identifies that a plurality of sound files transmitted from the same user terminal 1 during the same sleep period (also referred to as a single sleep time) from night to morning on the same day are a single related and coherent file. As a result, the analysis results of these plurality of sound files are generated as a single piece of display information in step S16, and are displayed as a single piece of display information in step S19. The generated display information may display the analysis results in chronological order based on the order identified in step S11 and / or time information corresponding to any one of the states of breathing sounds, snoring sounds, apnea, and hypopnea.

[0076] Note that the noise removal process in step S12 and / or the event identification process in step S13 may be omitted. Also, the processes of steps S12 to S15 may be performed for some sound files during one sleep period of the user. Also, if the sound information of the sound files includes time information related to the acquisition time, the process of identifying the order of the sound files in step S11 may be omitted. Also, after performing the process of identifying the order of the sound files in step S11, one sound file that combines multiple sound files may be generated again, and the processes of steps S12 to S15 may be performed for the sound file.

[0077] [Apnea etc. identification processing] Next, the apnea etc. identifying process in step S14 will be described. Fig. 6 is a diagram showing a flowchart of the apnea etc. identifying process according to the embodiment of the present invention. The apnea etc. identifying process is executed by the server device 2.

[0078] First, the server device 2 identifies the breathing sounds and / or snoring sounds of the user from the sound information in the sound file (step S21).

[0079] The breathing sounds and / or snoring sounds of a user can be identified, for example, as follows: For example, a prediction model (also referred to as a first prediction model) trained by machine learning can be used with sound information including snoring sounds or breathing sounds during a person's sleep or a spectrogram converted from the sound information as input data and information regarding the occurrence of snoring sounds or breathing sounds during the sleep as output data.

[0080] The information on the occurrence of snoring or breathing sounds includes, for example, information on whether snoring or breathing sounds have occurred, information on when snoring or breathing sounds have occurred, information on whether either snoring or breathing sounds have occurred, and / or information on the type of snoring or breathing sounds. Using the above-mentioned first prediction model, it is possible to determine whether breathing sounds have occurred and / or whether snoring sounds have occurred in the event determined in step S13 based on the sound information during the user's sleep or a spectrogram converted from the sound information. In other words, in step S21, it is possible to determine whether the event determined in step S13 as a candidate for snoring or breathing sounds is a snoring sound or breathing sound.

[0081] The system 10 of the present embodiment may specify the occurrence status of the snoring sound or breathing sound for the snoring sound or breathing sound specified in step S21. The occurrence status of the snoring sound or breathing sound includes, for example, information regarding the time when the breathing sound or the snoring sound occurred, the duration of the occurrence of the breathing sound or the snoring sound, whether the breathing sound or the snoring sound occurred, and / or the type of the breathing sound or the snoring sound.

[0082] The process of step S21 will be described. For example, a first prediction model is stored in the storage unit 23 of the server device 2. The machine learning algorithm is not particularly limited and any known algorithm can be used, such as linear regression, multiple regression analysis, support vector machine, decision tree, random forest, and deep learning using a multilayer neural network.

[0083] A multilayer neural network has an input layer, an output layer, and multiple intermediate layers. Weights are set for the edges connecting the nodes in each layer. Weights corresponding to each input to the node are set for the edges, and the edges are multiplied by the weights corresponding to each input to the node, and the values ​​obtained by multiplying these weights are added to a bias. The value obtained by the addition is nonlinearly transformed using an activation function to calculate an activation value. The calculated activation value becomes the input value passed to the node in the next layer. The number of intermediate layers can be designed as appropriate. The weights are optimized using the above-mentioned training data.

[0084] The input data, which is sound information including snoring or breathing sounds of a person while sleeping, or a spectrogram obtained by converting the sound information, includes, for example, time information, sound frequency, and / or sound volume. The spectrogram used in the input data is a result of calculating a frequency spectrum by passing the sound information through a window function. The spectrogram is a so-called voiceprint. In this embodiment, the sound information may be converted into a Mel frequency spectrogram or MFCC (Mel Frequency Cepstral Coefficients). When a spectrogram is used for the input data, in step S21, the server device 2 converts the sound information into a spectrogram. The sound information that the server device 2 converts into a spectrogram may be only a portion corresponding to a time period (event) that satisfies a predetermined second condition.

[0085] A spectrogram is a time-based display of the spectra of each audio data segment (frame), extracted from the waveform of an audio file's electrical signal by Fourier transforming the frequency and amplitude components. A spectrogram is displayed as a three-dimensional graph (time, frequency, and signal component strength). The horizontal axis of a spectrogram represents time, and the vertical axis represents frequency. A spectrogram also visualizes the strength of the signal component (amplitude) at each time and frequency component by displaying different colors or shades. A spectrogram is image information.

[0086] A Mel frequency spectrogram (also called a Mel spectrogram) is calculated by applying a Mel filter bank that extracts only certain frequency bands that are equally spaced in the Mel scale, which is based on the frequency perception of the human ear (it is sensitive to low frequency sounds and insensitive to high frequency sounds).

[0087] When the input data is sound information for a time period corresponding to an event or a spectrogram obtained by converting the sound information, the output data can be information on whether or not a snoring sound or a breathing sound occurred during the time period. Alternatively, when the input data is sound information having a predetermined length or a spectrogram obtained by converting the sound information, the output data can be information on which section of the sound information the snoring sound or the breathing sound occurred. The snoring sound or the breathing sound may occur immediately before apnea or hypopnea occurs during a person's sleep, or immediately after apnea or hypopnea occurs.

[0088] The output data may be, for example, data obtained by a person listening to a sound corresponding to sound information and judging whether it is a snoring sound, whether it is a breathing sound, or whether it is a breathing sound or a snoring sound. The output data may be data obtained by judging whether a breathing sound or a snoring sound has occurred based on the frequency components and / or the volume of the sound included in the sound information. The output data may be data obtained by using a portable sleep apnea testing device, PSG, a doctor's judgment, etc. to determine that a candidate (event) for a snoring sound or a breathing sound immediately before or immediately after a time period in which the occurrence of apnea or hypopnea is identified is a snoring sound or a breathing sound. The output data may be a combination of the above. A prediction model machine-learned using one of the above output data may be additionally trained using a different output data.

[0089] For example, the input information and output information used in the first prediction model can be obtained as follows. For example, sound information of the user A during sleep is obtained. The sound information is subjected to noise removal and event identification. This is the same process as step S12 and step S13. Next, the administrator who constructs the first prediction model compares the result of identifying the occurrence of apnea or hypopnea during sleep of the user A in a time series with the event. If apnea or hypopnea occurs immediately before or immediately after the occurrence of the event, the event is determined to be a snoring sound or breathing sound related to apnea or hypopnea. The labeled evaluation is the output information used in the first prediction model. In addition, sound information corresponding to the event determined to be a snoring sound or breathing sound related to apnea or hypopnea, or a spectrogram (image information) obtained by converting the sound information, can be used as the input information used in the first prediction model.

[0090] In this way, sound information including snoring sounds or breathing sounds by user A while sleeping or a spectrogram obtained by converting the sound information is used as input information, and information regarding the occurrence of snoring sounds or breathing sounds while sleeping is used as output information for the first prediction model. In the same manner, input information and output information are obtained for a plurality of users other than user A, and machine learning is performed based on these plurality of pieces of input information and output information.

[0091] Returning to the description of the process of step S21, when the server device 2 accepts input of sound information contained in a sound file, the control unit 21 of the server device 2 uses the first prediction model to identify information on the occurrence of snoring sounds or breathing sounds corresponding to the accepted sound information. In step S21, the data input to the first prediction model corresponds to the input data when the first prediction model is trained. For example, the server device 2 inputs sound information corresponding to a time period (event) that satisfies a predetermined second condition or a spectrogram obtained by converting the sound information to the first prediction model. Then, the server device 2 can identify whether or not snoring sounds or breathing sounds are occurring for a time period (event) that satisfies the predetermined second condition based on the output data.

[0092] This allows the server device 2 to identify the occurrence status of the snoring sound or breathing sound identified in step S21. For example, the server device 2 can identify that the sound from 0:10:00 to 0:10:05 is a breathing sound, and the sound from 0:20:00 to 0:20:08 is a snoring sound.

[0093] Furthermore, when the first prediction model identifies only the occurrence of snoring sounds among the time periods (events) that satisfy the second predetermined condition, the server device 2 can determine the events that do not identify the occurrence of snoring sounds as breathing sounds among the events. Alternatively, when the first prediction model identifies only the occurrence of breathing sounds among the events, the server device 2 can determine the events that do not identify the occurrence of breathing sounds as snoring sounds among the events.

[0094] Alternatively, if the first prediction model identifies the occurrence of snoring or breathing sounds that occurred immediately before or immediately after the occurrence of apnea or hypopnea among the time periods (events) that satisfy a specified second condition, the server device 2 can identify events among which the first prediction model did not identify the occurrence of snoring or breathing sounds as breathing sounds unrelated to apnea, etc.

[0095] When the breathing sound and the snoring sound of the user are identified from the sound information included in the sound file, a silent section is identified (step S22). A silent section is a time period in which no breathing sound or snoring sound occurs in the sound information. One silent section is a section from when a breathing sound or a snoring sound occurs until the next breathing sound or a snoring sound occurs. In the present embodiment, the silent section may be a time period in which the occurrence of a breathing sound or a snoring sound cannot be identified in the sound information. The silent section does not have to be a completely silent section, and may be a section in which a sound occurs. For example, even if the breathing sound is too small and is not detected from the sound information in step S21, it may be a silent section. The silent section may actually include a breathing sound. In this case, there is a possibility that hypopnea occurs in the silent section, not apnea.

[0096] The silent section specified in step S22 may be a silent section that satisfies a predetermined judgment condition. The predetermined judgment condition is, for example, that the silent section is a silent section between two preceding and succeeding snoring sounds, a silent section between two preceding and succeeding snoring sounds and breathing sounds, and / or a silent section between two preceding and succeeding breathing sounds. The predetermined judgment condition may also be, for example, that the silent section is a silent section between a snore sound or breathing sound (hereinafter also referred to as a specific snore sound, etc.) specified by the first prediction model in step S21 when the snore sound or breathing sound exists before and after the snore sound or breathing sound. The predetermined judgment condition may also be that the silent section is a silent section between a preceding and succeeding specific snore sound, etc. and breathing sounds unrelated to apnea, etc., and / or a silent section between a preceding and succeeding breathing sounds unrelated to apnea, etc. and breathing sounds unrelated to apnea, etc.

[0097] Here, the term "snoring sound and a snoring sound before and after" can be defined as two snoring sounds when there is only one silent section between the occurrence of a snoring sound and the occurrence of the next snoring sound. For example, if a breathing sound occurs between the occurrence of a snoring sound and the occurrence of the next snoring sound, there will be two silent sections between the snoring sounds, and these two snoring sounds are not "snoring sound and a snoring sound before and after". The relationship between the snoring sounds before and after and the snoring sounds is also called "continuous snoring sounds". Similarly, the term "snoring sound and a breathing sound before and after" can be defined as a snoring sound and a breathing sound when there is only one silent section between the occurrence of a snoring sound and the occurrence of the next breathing sound, or between the occurrence of a breathing sound and the occurrence of the next snoring sound. The term "snoring sound and a breathing sound before and after" can be defined as two breathing sounds when there is only one silent section between the occurrence of a breathing sound and the occurrence of the next breathing sound.

[0098] In step S22, when a specific snore sound or the like exists before and after the specific snore sound or the like, the server device 2 can identify the interval between the specific snore sounds or the like as a silent interval satisfying the predetermined judgment condition. For example, when specific snore sounds or the like occur continuously, the server device 2 can identify the interval between the specific snore sounds or the like as a silent interval satisfying the predetermined judgment condition. On the other hand, when the specific snore sounds or the like occur in the order of a breathing sound unrelated to apnea or the like, and the specific snore sounds or the like, the server device 2 does not need to identify the two silent intervals existing between the specific snore sounds or the like as silent intervals satisfying the predetermined judgment condition.

[0099] Alternatively, when it is determined in step S21 that the event is a snoring sound and a breathing sound, in step S22, the server device 2 can determine, as a silent section that satisfies a predetermined determination condition, a section between two adjacent snoring sounds with no breathing sound therebetween. Also, the server device 2 can determine, as a silent section, a section between two adjacent breathing sounds with no snoring sound therebetween.

[0100] Regarding the process of step S22, for example, the sound information includes breathing sounds and snoring sounds, but when the snoring sounds and the snoring sounds occur consecutively, the server device 2 can identify the section between the snoring sounds and the snoring sounds as a silent section that satisfies the predetermined judgment condition. On the other hand, when the snoring sounds, breathing sounds, and snoring sounds occur in this order, the server device 2 does not identify the section between the snoring sounds and the breathing sounds and the section between the breathing sounds and the snoring sounds as a silent section that satisfies the predetermined judgment condition. Similarly, when the breathing sounds and the breathing sounds occur consecutively, the server device 2 can identify the section between the breathing sounds and the breathing sounds as a silent section that satisfies the predetermined judgment condition. On the other hand, when the breathing sounds, snoring sounds, and the breathing sounds occur in this order, the server device 2 does not identify the section between the breathing sounds and the snoring sounds and the section between the snoring sounds and the breathing sounds as a silent section that satisfies the predetermined judgment condition.

[0101] When silent intervals are identified in step S22, silent intervals that satisfy a predetermined condition (also referred to as a first predetermined condition) are identified from those silent intervals (step S23). The first predetermined condition is, for example, that the length of the silent interval is within a predetermined length range. The predetermined length range can be designed appropriately, and specifically, may be 8 seconds or more, or 10 seconds or more. The predetermined length range may be within 4 minutes, or within 3 minutes. The predetermined length range may be 8 seconds to 4 minutes, or 10 seconds to 3 minutes.

[0102] Next, it is determined whether or not apnea or hypopnea occurs during the user's sleep (step S24). In step S24, the server device 2 may determine that apnea or hypopnea occurs during the user's sleep in the silent section that satisfies the predetermined first condition, which is determined in step S23. Alternatively, the server device 2 may use a machine-learned prediction model to determine only the occurrence of apnea during the user's sleep, or only the occurrence of hypopnea, or even both the occurrence of apnea and the occurrence of hypopnea, in the silent section that is determined in step S23 to satisfy the predetermined first condition, based on sound information during the user's sleep. The prediction model is machine-learned using sound information during a person's sleep or a spectrogram obtained by converting the sound information as input data, and information regarding the occurrence of apnea or hypopnea during the sleep as output data.

[0103] Here, the information regarding the occurrence of apnea or hypopnea includes, for example, information regarding whether apnea or hypopnea has occurred and / or information regarding the type of apnea or hypopnea. Note that, although the types of apnea or hypopnea include, for example, obstructive and central, the types of apnea or hypopnea machine-learned as output data are not limited thereto and may be more finely classified.

[0104] The process of identifying whether apnea or hypopnea occurs using a prediction model in step S24 will be described. For example, the storage unit 23 of the server device 2 stores a prediction model (also called a second prediction model) that is machine-learned using sound information during a person's sleep or a spectrogram obtained by converting the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data. The machine learning algorithm is not particularly limited, and a known algorithm can be used, as with the first prediction model, and examples of the algorithm include linear regression, multiple regression analysis, support vector machine, decision tree, random forest, and deep learning using a multilayer neural network.

[0105] A multilayer neural network has an input layer, an output layer, and multiple intermediate layers. Weights are set for the edges connecting the nodes in each layer. Weights corresponding to each input to the node are set for the edges, and the edges are multiplied by the weights corresponding to each input to the node, and the values ​​obtained by multiplying these weights are added to a bias. The value obtained by the addition is nonlinearly transformed using an activation function to calculate an activation value. The calculated activation value becomes the input value passed to the node in the next layer. The number of intermediate layers can be designed as appropriate. The weights are optimized using the above-mentioned training data.

[0106] The input data, which is sound information during a person's sleep or a spectrogram obtained by converting the sound information, includes, for example, time information, sound frequency, and / or sound loudness. The spectrogram used in the input data is similar to that described in the first prediction model above, and is a result of passing the sound information through a window function and calculating a frequency spectrum. When a spectrogram is used for the input data, in step S24, the server device 2 converts the sound information into a spectrogram. Alternatively, when the sound information is converted into a spectrogram in step S21, the spectrogram converted in step S21 may be used.

[0107] For example, in the second prediction model, when the input data is sound information corresponding to a silent section satisfying the first predetermined condition specified in step S23 or a spectrogram obtained by converting the sound information, the output data can be information regarding whether or not apnea or hypopnea has occurred in the silent section. Alternatively, in the second prediction model, when the input data is sound information having a predetermined length or a spectrogram obtained by converting the sound information, the output data can be information regarding in which section of the sound information apnea or hypopnea has occurred.

[0108] The output data used may be, for example, data indicating the occurrence of apnea or hypopnea identified using a portable sleep apnea testing device, PSG, a doctor's judgment, or the like.

[0109] When the prediction model is used in step S24, when the server device 2 receives input of sound information of the sound file, the control unit 21 of the server device 2 uses the second prediction model to determine whether or not apnea or hypopnea has occurred in the silent section identified in step S23. The data input to the second prediction model in step S24 corresponds to the input data when the second prediction model is trained. For example, the server device 2 inputs sound information corresponding to the silent section that satisfies a predetermined first condition or a spectrogram obtained by converting the sound information into the second prediction model. Then, the server device 2 can determine whether or not apnea or hypopnea has occurred in the silent section based on the output data. By using the second prediction model to check whether apnea or hypopnea has occurred in the identified silent section, the system 10 of this embodiment can accurately detect the occurrence of apnea or hypopnea.

[0110] By the process of step S24, the occurrence status of apnea or hypopnea can be identified for the silent section identified in step S23. For example, the server device 2 can identify that apnea occurs in the silent section from 00:10:05 to 00:10:25, and that hypopnea occurs in the silent section from 00:22:00 to 00:22:12. Note that, for example, the start time of the silent section may be the end time of the event immediately preceding the silent section, and the end time of the silent section may be the start time of the event immediately following the silent section. Furthermore, the section in which apnea or hypopnea occurs may or may not coincide with the silent section identified in step S23.

[0111] When it is determined that the user is experiencing apnea or hypopnea during sleep, the type of apnea or hypopnea is identified (step S25). If the type of apnea or hypopnea is identified in the process of step S24, the server device 2 can use the identified information in step S25. The apnea etc. identification process is completed through steps S21 to S25.

[0112] Here, the types of apnea or hypopnea include, for example, obstructive and central apnea. For example, apnea or hypopnea occurring between snoring sounds may be identified as obstructive apnea, and apnea or hypopnea occurring between breathing sounds may be identified as central apnea.

[0113] The system 10 of the present embodiment can identify the type of apnea or hypopnea depending on whether the silent interval identified in step S23 is a silent interval between snoring sounds or a silent interval between breathing sounds. For example, if a silent interval between snoring sounds is identified as apnea in step S24, the apnea is identified as obstructive apnea in step S25.

[0114] Also, for example, in the process of step S25, if snoring sounds are present before and after the time when hypopnea occurs, the server device 2 may identify the hypopnea as obstructive hypopnea. Also, if breathing sounds are present before and after the time when hypopnea occurs, the server device 2 may identify the hypopnea as hypopnea caused by a nasal disease such as central or allergic rhinitis or sinusitis.

[0115] In this embodiment, the server device 2 may determine the silent section satisfying the predetermined judgment condition in step S22 as the silent section between the preceding and succeeding snoring sounds, and may determine the silent section between the preceding and succeeding breathing sounds as the non-analysis section in which analysis is not performed. Alternatively, the server device 2 may perform the processes in steps S23 to S25 by regarding the silent section between the preceding and succeeding breathing sounds as the silent section satisfying the predetermined condition. In this case, the apnea or hypopnea identified as occurring during the user's sleep in step S24 can be identified as central apnea or hypopnea in step S25.

[0116] Here, an overview of apnea etc. identified in step S23 will be described. FIG. 7 is a diagram for explaining the apnea etc. identification process according to an embodiment of the present invention. FIG. 7(A) is a diagram showing the waveform of the electrical signal (sound information) of a sound file from which noise has been removed, and is a diagram showing the waveform of the electrical signal of the sound file from which the process of step S12 has been executed. The horizontal axis of FIG. 7(A) indicates the time when the sound information was acquired, and the vertical axis indicates the amplitude. FIG. 7(B) is a diagram of the electrical signal (sound information) of the sound file converted into a spectrogram. The horizontal axis of FIG. 7(B) indicates the time when the sound information was acquired, the vertical axis indicates the frequency, and the shading indicates the intensity of the amplitude.

[0117] When step S13 is executed, an event is identified in a time period (section) indicated by a solid line in FIG. 7. In step S21, sound information of a section corresponding to the event or a spectrogram obtained by converting the sound information is input to the first prediction model. As a result, a snoring sound or a breathing sound is identified for the section by the first prediction model. In FIG. 7, an event corresponding to a snoring sound or a breathing sound (specific snoring sound, etc.) identified by the first prediction model is shown by a square. On the other hand, an event not identified as a snoring sound or a breathing sound by the first prediction model is shown by a circle as a breathing sound unrelated to apnea, etc. When the silent section identified in step S22 is a section in which the snoring sound or the breathing sound identified by the first prediction model continues, when step S22 is executed, the section indicated by the dashed line in FIG. 7 is identified as a silent section between specific snoring sounds, etc. In step S23, it is determined whether the silent section indicated by the dashed line in FIG. 7 satisfies a predetermined first condition. When the predetermined first condition is satisfied, in step S24, the server device 2 can determine that hypopnea or apnea is occurring in the silent section. In Fig. 7, the section determined to be the occurrence of hypopnea or apnea is indicated by a triangle.

[0118] If it is determined in step S21 that the time period (event) that satisfies the second predetermined condition is a snoring sound or a breathing sound, the silent section determined in step S22 can be a section during which the snoring sound continues or a section during which the breathing sound continues. FIG. 8 corresponds to the apnea etc. identifying process when it is determined in step S21 that the event is a snoring sound or a breathing sound. FIG. 8 is a diagram for explaining the apnea etc. identifying process according to an embodiment of the present invention. FIG. 8(A) is a diagram showing the waveform of the electrical signal (sound information) of a sound file from which noise has been removed, and is a diagram showing the waveform of the electrical signal of the sound file from which the process of step S12 has been executed. In addition, the horizontal axis of FIG. 8(A) to (D) indicates the time when the sound information is acquired, and the vertical axis indicates the amplitude.

[0119] Fig. 8(B) shows a time period (section) in which the occurrence of apnea or hypopnea was identified from the sound information contained in the sound file shown in Fig. 8(A). Fig. 8(C) shows a time period (section) in which the occurrence of breathing sounds was identified from the sound information contained in the sound file shown in Fig. 8(A). Fig. 8(D) shows a time period (section) in which the occurrence of snoring sounds was identified from the sound information contained in the sound file shown in Fig. 8(A).

[0120] As shown in FIG. 8(B), when the silent interval between the preceding and succeeding snoring sounds shown in FIG. 8(D) is within a predetermined length range, it is identified as a silent interval that satisfies a predetermined first condition (step S23). If the predetermined first condition is satisfied, the server device 2 can identify that hypopnea or apnea has occurred in the silent interval. The server device 2 may use a second prediction model to identify that apnea or hypopnea has occurred in the silent interval (step S24). Note that FIG. 8(E) shows a silent interval in which no snoring sound was detected between breathing sounds from the sound information of the sound file shown in FIG. 8(A). The silent interval may be a non-analysis interval.

[0121] The process of step S21 may be executed as follows. First, the server device 2 specifies an event that satisfies a predetermined third condition for specifying a snoring sound or a predetermined fourth condition for specifying a breathing sound for the event specified in step S13. The predetermined third condition and the predetermined fourth condition can be appropriately set as known conditions for specifying a snoring sound or a breathing sound. For example, the predetermined third condition is that the volume of the sound information included in the event is equal to or greater than a predetermined volume. For example, the predetermined fourth condition is that the volume of the sound information included in the event is equal to or less than a predetermined volume or falls below a predetermined volume. The server device 2 can specify that an event that satisfies the predetermined third condition is a snoring sound, and that an event that satisfies the predetermined fourth condition is a breathing sound. As a result, the server device 2 can specify that a snoring sound or a breathing sound is occurring for the event specified in step S13.

[0122] The process of step S25 may be omitted. In addition, in steps S24 and S25, only the occurrence and type of apnea may be identified, or only the occurrence and type of hypopnea may be identified.

[0123] Next, another example of the apnea etc. identifying process in step S14 will be described. Fig. 9 is a diagram showing a flowchart of another example of the apnea etc. identifying process according to the embodiment of the present invention. The apnea etc. identifying process is executed by the server device 2.

[0124] First, it is determined whether or not apnea or hypopnea occurs during the user's sleep (step S31). In step S31, the occurrence of apnea or hypopnea during the user's sleep can be identified based on the sound information during the user's sleep using a second prediction model machine-learned using sound information during a person's sleep or a spectrogram obtained by converting the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data.

[0125] When it is determined that the user is experiencing apnea or hypopnea during sleep, the type of hypopnea is identified (step S32). Step S32 is the same process as step S25. After steps S31 and S32, the apnea etc. identification process ends.

[0126] Here, the second prediction model will be described. When apnea occurs, the snoring sound or breathing sound before and after the silent section where apnea occurs has characteristics in terms of the frequency and loudness of the sound contained in the snoring sound and breathing sound. When hypopnea occurs, the snoring sound or breathing sound in the section where hypopnea occurs has characteristics in terms of the frequency and loudness of the sound contained in the snoring sound and breathing sound.

[0127] For example, the spectrogram of the sound included in the snoring sound or breathing sound near the occurrence of apnea or the like may have a similar pattern. As described above, the spectrogram is image information with the vertical axis representing frequency and the horizontal axis representing time. In addition, the frequency component of the sound included in the snoring sound or breathing sound near the occurrence of apnea or the like may include a higher frequency than the frequency of the sound included in other snoring sounds or breathing sounds. In addition, for example, the loudness of the sound included in the snoring sound or breathing sound near the occurrence of apnea or the like may be louder than the loudness of the sound included in other snoring sounds or breathing sounds. The waveform of the electrical signal indicating the snoring sound or breathing sound near the occurrence of apnea or the like may have a similar shape. Therefore, the second prediction model can be constructed by machine learning.

[0128] For the second prediction model, the input data may be sound information having a predetermined length or a spectrogram obtained by converting the sound information, and the output data may be information regarding which section of the sound information an apnea or hypopnea occurred. The sound information having a predetermined length used as the input data or the spectrogram obtained by converting the sound information may be appropriately designed, but preferably includes a time corresponding to at least one time period during which an apnea or hypopnea occurred. When a spectrogram is used as the input data, in step S31, the server device 2 converts the sound information into a spectrogram.

[0129] The second prediction model may be a plurality of prediction models. For example, a prediction model trained by machine learning using sound information during a person's sleep as input data and information regarding the occurrence of apnea during the sleep as output data, and a prediction model trained by machine learning using sound information during a person's sleep as input data and information regarding the occurrence of hypopnea during the sleep as output data may be constructed.

[0130] The server device 2 can calculate the analysis result based on the information on the occurrence of snoring or breathing sounds specified in step S21, the information on the silent intervals in which apnea or hypopnea occurs specified in steps S22 and S23, and / or the information on the occurrence or type of apnea or hypopnea specified in steps S24, S25, S31, and S32. The information on the silent intervals includes, for example, time information when the silent interval occurs.

[0131] [Display information] Next, the display information displayed on the user terminal 1 in step S19 will be described. Fig. 10 is an example of the user terminal 1 on which the display information is displayed in step S19. On the display screen 100, an analysis result 101 for October 3, 2023 and analysis information 110 (110a to 110d) are displayed. In addition, on the display screen 100, a description 111 and a plurality of icons 120 (120a, 120b) may be displayed.

[0132] In the analysis result 101, an area 102 corresponding to when the user is awake and an area 103 corresponding to when the user is asleep are displayed. In addition, in the analysis result 101, an area 104 corresponding to when apnea or hypopnea occurs is displayed within the area 103 corresponding to when the user is asleep.

[0133] Regions 102 and 103 are displayed in a circumferential shape with the center of the circle being the upper center of display screen 100. Regions 102 and 103 have a width in the radial direction of the circle. The width of regions 102 and 103 may be constant or may differ between regions 102 and 103. Regions 102 or 103 may be displayed in an arc-like or band-like shape.

[0134] A scale 105 is displayed inside the circularly displayed areas 102 and 103. The scale 105 indicates the time. On the inside of the circumference, a scale mark "12" is provided at the topmost position on the display screen 100, and the scale mark "12" indicates midnight. A scale mark "3" provided at the right end of the inside of the circumference indicates 3am. The scale 105 has thick scale marks every predetermined time (1 hour) and thin scale marks every predetermined time (12 minutes). The time of day on which the scale is provided, the thickness of the lines, and / or the length of the lines are set appropriately.

[0135] In FIG. 10, area 102 corresponding to the user's awake time and area 103 corresponding to the user's asleep time correspond to a total of 12 hours. Scale 105 corresponds to an actual clock capable of indicating 12 hours. Areas 102 and 103 are displayed so that the time and hour of the user's sleeping time correspond to scale 105. That is, in FIG. 10, the user can recognize that he / she started sleeping after 10:36 p.m. and ended sleeping after 7 a.m. the following day. The time when the user started sleeping corresponds to the time when the acquisition of sound information in step S1 started, and the time when the user ended sleeping corresponds to the time when the acquisition of sound information ended.

[0136] The time displayed in the analysis result 101 may be 12 hours or 24 hours, or may only be the time corresponding to the area 103 corresponding to the user's sleep time. The scale 105 does not have to be displayed in the analysis result 101. The area 102 does not have to be displayed in the analysis result 101.

[0137] Areas 102 and 103 are displayed continuously. In Fig. 10, area 102 is displayed in light gray, area 103 is displayed in black, and area 104 is displayed in white. That is, areas 102, 103, and 104 are displayed in different display modes. The different display modes are not particularly limited, and examples thereof include different colors, different patterns, and different blinking patterns. Note that areas 102 and 104 may be displayed in the same display mode.

[0138] An area 104 corresponding to when apnea or the like occurs is displayed as a slit (gap) in an area 103 corresponding to when the user is sleeping. Also, the area 104 is displayed with the time and duration when apnea or the like occurred corresponding to a scale 105. For example, the area 104 corresponding to the period from 3:35 to 3:36 is displayed in a color different from that of the area 103. The area 104 indicates that apnea or hypopnea occurred between 3:35 and 3:36.

[0139] The color of region 104 is not particularly limited as long as it is different from that of region 103, and may be displayed in the same color as the background color of display screen 100, for example. It is also preferable that region 103 and region 104 are displayed in a manner that clearly distinguishes them from each other. For example, region 103 and region 104 may be displayed in colors with different saturations, such as white and black, or in colors that have a strong contrast and are opposite colors, such as blue and yellow.

[0140] In addition, region 104 may be displayed in different manners depending on the length of time that apnea and / or hypopnea lasts, and / or the number of apnea and / or hypopnea events per unit time or the total or average length of time that apnea and / or hypopnea lasts per unit time.

[0141] The region 104 may be displayed in a different manner depending on the length of time that apnea, etc. continues. For example, if apnea, etc. continues for a predetermined time or longer (e.g., one minute), the width of the slit may be made wider, and if apnea, etc. continues for less than the predetermined time, the width of the slit may be made narrower or no slit may be displayed.

[0142] The region 104 may be displayed in a different display mode depending on the number of occurrences of apnea, etc. per unit time, for example, when the number of occurrences of apnea, etc. in a predetermined unit time (e.g., 10 minutes) is more than a predetermined number (e.g., 2 occurrences), a slit may be provided in the predetermined unit time, and when the number of occurrences of apnea, etc. in the predetermined unit time is equal to or less than the predetermined number, a slit may not be provided in the predetermined unit time. Alternatively, the server device 2 may store the number of occurrences of apnea, etc. in a predetermined unit time and the width of the slit in a table in association with each other, and determine the width of the slit by referring to the table.

[0143] The area 104 may be displayed in a different manner depending on the total or average duration of apnea, etc. per unit time. For example, if the total duration of apnea, etc. in a predetermined unit time (e.g., 10 minutes) is a predetermined time or longer (e.g., 3 minutes), a slit may be provided in the predetermined unit time, and if the total duration of apnea, etc. is shorter than the predetermined time, a slit may not be provided in the predetermined unit time.

[0144] Furthermore, if the average duration of apnea, etc. in a predetermined unit time is equal to or longer than a predetermined time (e.g., one minute), the server device 2 may provide a slit in the predetermined unit time, and if the total duration of apnea, etc. is equal to or shorter than the predetermined time, the server device 2 may not provide a slit in the predetermined unit time. Alternatively, the server device 2 may store the total or average duration of apnea, etc. in a predetermined unit time and the width of the slit in a table in association with each other, and determine the width of the slit by referring to the table.

[0145] The predetermined unit time can be set appropriately. For example, the predetermined unit time may be the shortest time for which a slit can be provided, or may be a time corresponding to the time marked on the scale.

[0146] The different display modes are not limited to only the change in the width of the slit. The different display modes are, for example, different colors, different patterns, different blinking patterns, etc. When combining the above different display modes, it is preferable to adopt a different display mode for each indicator. For example, when the display mode of the region 104 is changed according to the number of apnea, etc. per unit time and the total length of the continuous time of apnea, etc. per unit time, the server device 2 may change the display mode of the region 104 by changing the width of the slit for the number of apnea, etc. per unit time and changing the color for the total length of the continuous time of apnea, etc. per unit time.

[0147] For example, the region 104 may be displayed with slits in an area corresponding to 2:13 to 2:23, or may be displayed with slits in an area corresponding to 4:02 to 4:25. By making the region 104 with slits of different widths rather than slits corresponding to the times when apnea or the like occurred, the visibility of the analysis result 101 can be improved.

[0148] Explanation 111 displays information explaining what area 103 and area 104 of analysis result 101 indicate. Explanation 111a displays the same color as area 103, and explanation 111b displays the same color as area 104. Explanation 111b is displayed as a slit within the area of ​​explanation 111a. Therefore, the user can understand that explanation 111a indicates area 103, and explanation 111b indicates area 104.

[0149] Explanation 111c indicates that the area of ​​explanation 111b is "a time when breathing is thought to be stopped." Therefore, the user can recognize that area 104 of analysis result 101 indicates a time when breathing is thought to be stopped. Note that system 10 of the present embodiment may express "the occurrence of apnea or hypopnea" as "a time when breathing is thought to be stopped."

[0150] The analysis information 110 indicates information on snoring and / or information on apnea or hypopnea during sleep of the user identified in the analysis process. The system 10 of the present embodiment may display, as the analysis information 110, information indicating the total, average, or longest duration of apnea and / or hypopnea, the total number of apnea and / or hypopnea, and / or the number of apnea and / or hypopnea per unit time or the total or average duration of apnea and / or hypopnea per unit time.

[0151] The analysis information 110a indicates the total number of times that apnea or hypopnea occurred during one sleep period of the user. The analysis information 110b indicates the sleeping time during one sleep period of the user. The sleeping time is the time from when the user terminal 1 receives an input to start acquiring sound information to when it receives an input to end acquiring sound information during the transmission process.

[0152] The analysis information 110c indicates the average number of times per unit time that the user is thought to not be breathing during one sleep period, and the analysis information 110d indicates the longest number of seconds that the user is thought to not be breathing during one sleep period.

[0153] The explanation 106 indicates an explanation for the user. The content of the explanation 106 may be changed depending on the analysis result. For example, if the analysis result shows that the occurrence of apnea or the like (for example, the number of occurrences or duration of apnea or the like) is within a predetermined range, the explanation 106 may say "There is no problem with your sleep state." If the occurrence of apnea or the like is outside the predetermined range, the explanation 106 may say "We recommend that you consult a medical institution."

[0154] The icon 120a is an icon for displaying another display screen. When the user selects the icon 120a, the display screen 100 is switched. The display screen to be switched to may be set arbitrarily, but may be, for example, a list of analysis results and dates and times of analysis of the user's sleep up to now. When the user selects a displayed analysis result or date and time, a display screen including the same information as the display screen 100 for the selected analysis result or date and time is displayed on the user terminal 1.

[0155] Icon 120b is an icon for displaying details of the analysis result. When the user selects icon 120b, the display screen 100 is switched. The display screen to be switched to may be set arbitrarily, and may display, for example, details of the analysis result 101 on the display screen 100. In the switched display screen, the acquired sound information or the sound information from which noise has been removed may be replayed as sound, and the area 104 may display the analysis result 101 displayed in perfect correspondence with the time and duration of the occurrence of apnea or the like.

[0156] The area 103 corresponding to the user's sleep may be displayed in the form of a graph to indicate the depth of the user's sleep. In the graph, the depth of the user's sleep can be displayed on the vertical axis, and time can be displayed on the horizontal axis.

[0157] In the above embodiment, a sound file is transmitted to the server device 2, and the server device 2 executes the process of analyzing the sound file. However, the user terminal 1 may execute the process of analyzing the sound file. In this case, the process of transmitting the sound file in step S4 and the process of transmitting the display information in step S17 may be omitted. Also, the process of generating the sound file in step S3 may be omitted.

[0158] When the processes of steps S11 to S16 are executed in the user terminal 1, a necessary program may be stored in the user terminal 1 in advance, or may be transmitted from the server device 2. The system 10 may include an analysis process in the user terminal 1 that acquires sounds made by the user while sleeping as sound information and analyzes the sound information.

[0159] In the above embodiment, a sound file is transmitted to the server device 2, and the server device 2 executes the process of analyzing the sound file. However, a third party such as a medical professional may analyze the sound file. In this case, the system 10 is accessed from a third party terminal operated by the third party via the server device 2. Then, the sound file from which the noise removal process in step S12 has been executed is transmitted from the server device 2 to the third party terminal. When the third party finishes analyzing the sound information of the sound file, the analysis result may be transmitted from the third party terminal to the user terminal 1 via the server device 2 by the third party's operational input.

[0160] In the above embodiment, the processes of acquiring and transmitting sound information in steps S1 to S4 and the processes of displaying display information in steps S18 and S19 are performed by the same user terminal 1. However, the processes of steps S1 to S4 and steps S18 and S19 may be performed by different user terminals 1. In this case, the user terminal 1 that executes steps S1 to S4 may be provided with an acquisition device. Also, the user terminal 1 that executes steps S18 and S19 may be provided with a display device.

[0161] Furthermore, the process of displaying the display information in steps S18 and S19 may be executed at any timing of the user. For example, when the user logs in to the system 10, the display information may be transmitted from the server device 2 and displayed on the user terminal 1.

[0162] In the analysis process, the system 10 of the present embodiment may evaluate whether the user has or may have a predetermined disease such as sleep apnea syndrome. For example, if the total number of occurrences or duration of the stored apnea or hypopnea during sleep exceeds a predetermined threshold, the server device 2 can determine that the user has or may have a predetermined disease.

[0163] In the above, a case where sound information is mainly acquired in the transmission process has been described, but the information acquired in step S1 is not particularly limited. The information acquired in step S1 may be, for example, video information. The video information acquired in step S1 may be information about the user, and may include a specific part of the user (for example, face, neck, arms, legs, body, etc.). When video information is acquired, the above transmission process can be applied with the sound information as video information and the sound file as a video file. For example, in step S3, a video file is generated. A video file is a collection of data that allows the acquired video information to be managed and stored. The file format of the video file is MP4, AVI, MOV, MPG, etc.

[0164] The information acquired and transmitted in the transmission process is analyzed by the server device 2 to identify information about the user. In the above, the analysis process is used to identify the occurrence of apnea or hypopnea while the user is sleeping, but the content of the analysis is not limited to this. The information acquired and transmitted in the transmission process can be changed as appropriate depending on the content of the analysis. For example, when a video file is transmitted, the server device 2 may perform an analysis to evaluate the user's movements. Also, for example, when a sound file including the user's vocalization is transmitted, the server device 2 may perform an analysis to evaluate the user's vocalization.

[0165] According to the present invention, a machine-learned prediction model is used that uses sound information including snoring sounds or breathing sounds during a person's sleep or a spectrogram converted from the sound information as input data, and information regarding the occurrence of snoring sounds or breathing sounds during the sleep as output data, and identifies the occurrence of snoring sounds or breathing sounds during the user's sleep based on the sound information during the user's sleep or a spectrogram converted from the sound information, thereby making it possible to preferably identify snoring sounds or breathing sounds during the user's sleep.

[0166] According to the present invention, the system includes a first specifying means for specifying the occurrence of snoring or breathing sounds during a user's sleep based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a machine-learned prediction model with input data being sound information including snoring sounds or breathing sounds during a person's sleep or a spectrogram obtained by converting the sound information, and output data being information regarding the occurrence of snoring sounds or breathing sounds during the sleep, and a second specifying means for specifying whether or not a silent interval between two adjacent snoring sounds, a silent interval between two adjacent snoring sounds and a breathing sound, and / or a silent interval between two adjacent breathing sounds and a breathing sound satisfy a predetermined first condition. Therefore, the system can specify the occurrence of apnea or hypopnea using the specified result. In addition, even if the accuracy of directly specifying a silent interval or the occurrence of apnea or hypopnea using a machine-learned prediction model (so-called AI) is insufficient, the silent interval or the occurrence of apnea or hypopnea can be specified with high accuracy by specifying snoring sounds and breathing sounds as described above.

[0167] According to the present invention, the system includes a removal means for removing noise from the sound information of the user while sleeping, and the first identification means identifies the occurrence of breathing sounds or snoring sounds while the user is sleeping based on the sound information from which the noise has been removed or a spectrogram obtained by converting the sound information from which the noise has been removed, so that even sound information containing noise can be analyzed. The user terminal can obtain sound information near the user's bedding, for example, and therefore obtain sound information of the user during daily sleep.

[0168] In addition, since the first identifying means identifies the occurrence of snoring sounds or breathing sounds based on sound information in a time period that satisfies a predetermined second condition or a spectrogram obtained by converting the sound information, the burden of identifying the occurrence of snoring sounds or breathing sounds can be reduced compared to identifying the occurrence of snoring sounds or breathing sounds from the entire sound information or a spectrogram obtained by converting the sound information.In addition, since the system includes a third identifying means for identifying the occurrence of apnea or hypopnea in a silent section that satisfies a predetermined condition, and a calculating means for calculating the total, average, or maximum length of time that apnea and / or hypopnea continues, the total number of apnea and / or hypopnea, the number of apnea and / or hypopnea per unit time, and / or the total or average length of time that apnea and / or hypopnea continues per unit time based on information on the silent section in which the identified apnea or hypopnea occurs, information on apnea or hypopnea can be calculated. Furthermore, since the snoring or breathing sounds made by the person while sleeping occur immediately before apnea or hypopnea occurs, or immediately after apnea or hypopnea occurs, it is possible to identify the occurrence of snoring or breathing sounds related to apnea or hypopnea.

[0169] According to the present invention, the system uses sound information during a person's sleep or a spectrogram converted from the sound information as input data, and information regarding the occurrence of apnea or hypopnea during sleep as output data, using a machine-learned predictive model to identify the occurrence of apnea or hypopnea during a user's sleep based on the sound information during the user's sleep or a spectrogram converted from the sound information, thereby making it possible to identify the occurrence of apnea or hypopnea.

[0170] According to the present invention, an area corresponding to when apnea or hypopnea occurs is displayed within an area corresponding to when the user is sleeping, so that the user can visually recognize when apnea or hypopnea occurs during the user's sleep. Also, according to the present invention, an area corresponding to when the user is sleeping is displayed in an arc-like or band-like shape, so that the user can intuitively recognize when apnea or hypopnea occurs during the user's sleep.

[0171] According to the present invention, the area corresponding to when apnea or hypopnea occurs is displayed in different display modes depending on the length of time that apnea and / or hypopnea continues and / or the number of apnea and / or hypopnea events per unit time or the total or average length of time that they continue per unit time, thereby improving the visibility of the analysis results regarding the user's breathing state during sleep.

[0172] The present invention further displays information indicating the total, average or longest duration of apnea and / or hypopnea, the total number of apnea and / or hypopnea episodes, and / or the number of apnea and / or hypopnea episodes per unit time or the total or average duration of apnea and / or hypopnea per unit time, allowing the user to check the analysis results regarding the occurrence of apnea, etc. during the user's sleep.

[0173] According to the present invention, a plurality of sound files obtained at different times are generated based on sound information from a user's sleep period, and the generated sound files are transmitted to a second computer device, so that the load of transmitting sound information from a user's sleep period to a second computer device (server device) can be reduced compared to transmitting sound information from a user's sleep period as a single sound file. Also, the sound information of the sound file can be used in the second computer device to identify whether or not apnea or hypopnea occurs during the user's sleep.

[0174] According to the present invention, the generated sound file is transmitted in association with information that can identify the order in which the sound information was acquired, so that the sound information during one sleep of the user can be identified in the chronological order in which it was acquired. Furthermore, according to the present invention, when a predetermined communication line is detected, if the time of the generated sound file is equal to or less than a predetermined time, or if the data amount of the sound file is equal to or less than a predetermined amount, the generated sound file is transmitted, so that the data amount borne by the user when transmitting the sound file can be prevented from becoming excessive.

[0175] Furthermore, according to the present invention, when there is a sound file that has not been transmitted to the second computer device, notification information indicating that there is an untransmitted sound file is sent in response to detection of a specific communication line, so that the user can be informed of the existence of an untransmitted sound file. This makes it possible to prevent a state in which a sound file remains untransmitted. Furthermore, according to the present invention, multiple sound files are generated according to the time of the sound file or the data amount of the sound file, so that transmission failures due to the data amount of the sound file can be prevented, and sound information during one sleep of the user can be transmitted to the second computer device.

[0176] According to the present invention, the generated sound file is transmitted in association with information that can identify the order in which the sound information was acquired, so that the sound information during one sleep session of the user can be identified in chronological order. Also, according to the present invention, in parallel with the generation and / or transmission of one sound file, sound information for generating sound files to be generated after the one sound file is acquired, so that the time required to generate and transmit the sound files can be reduced.

[0177] According to the present invention, sound files are generated for different times or amounts of data depending on the computer device that executes the generation of the sound files, so the load on the computer device is small when the sound files are generated and transmitted. [Explanation of symbols]

[0178] 1 User terminal 2 Server device 3 Communication network 10. System 11 control section 12 RAM 13 storage section 14 Acquisition unit 15 Input unit 16 Display unit 17 Communication interface 21 control section 22 RAM 23 storage section 24 Communication Interface 100 Display screen 101 Analysis result 102 Area corresponding to user's wakefulness 103 Area corresponding to user's sleep 104 Areas corresponding to when apnea or hypopnea occurs 105 Scale 106 Explanation 110 Analysis Information 111 Description 120 Icon

Claims

1. A system comprising at least one computing device, a first identification means for identifying the occurrence of snoring or breathing sounds while a user is sleeping based on the sound information while the user is sleeping or a spectrogram obtained by converting the sound information, using a prediction model machine-learned with input data being sound information including snoring or breathing sounds while a user is sleeping or output data being information regarding the occurrence of snoring or breathing sounds while the user is sleeping, based on the sound information while the user is sleeping or a spectrogram obtained by converting the sound information; a second specifying means for specifying whether or not a silent interval between adjacent snoring sounds, a silent interval between adjacent snoring sounds and breathing sounds, and / or a silent interval between adjacent breathing sounds satisfy a predetermined first condition based on information regarding the occurrence of snoring sounds or breathing sounds specified by the first specifying means; A system comprising:

2. A means for removing noise from sound information while the user is sleeping Equipped with The system according to claim 1 , wherein the first identification means identifies the occurrence of breathing sounds or snoring sounds during the user's sleep based on the sound information from which noise has been removed or a spectrogram converted from the sound information from which noise has been removed.

3. 3. The system according to claim 1, wherein the first identification means identifies the occurrence of snoring sounds or breathing sounds based on sound information from a time period that satisfies a predetermined second condition or a spectrogram obtained by converting the sound information.

4. a third identification means for identifying that apnea or hypopnea has occurred in a silent section that satisfies a predetermined condition; a calculation means for calculating, based on information on the silent intervals in which the identified apnea or hypopnea occurs, the total, average or maximum duration of apnea and / or hypopnea, the total number of apneas and / or hypopnea, the number of apneas and / or hypopnea per unit time, and / or the total or average duration of apnea and / or hypopnea per unit time. The system according to claim 1 or 2, comprising:

5. 3. The system of claim 1 or 2, wherein the snoring or breathing sounds during the person's sleep occur immediately before an apnea or hypopnea occurs, or immediately after an apnea or hypopnea occurs.

6. A computer device, a first identification means for identifying the occurrence of snoring or breathing sounds while a user is sleeping based on the sound information while the user is sleeping or a spectrogram obtained by converting the sound information, using a prediction model machine-learned with input data being sound information including snoring or breathing sounds while a user is sleeping or output data being information regarding the occurrence of snoring or breathing sounds while the user is sleeping, based on the sound information while the user is sleeping or a spectrogram obtained by converting the sound information; a second specifying means for specifying whether or not a silent interval between adjacent snoring sounds, a silent interval between adjacent snoring sounds and breathing sounds, and / or a silent interval between adjacent breathing sounds satisfy a predetermined condition based on information regarding the occurrence of snoring sounds or breathing sounds specified by the first specifying means A program that functions as a

7. A method executed on at least one computing device, comprising: a first identification step of identifying the occurrence of snoring or breathing sounds while a user is sleeping based on the sound information while the user is sleeping or the spectrogram obtained by converting the sound information, using a machine-learned prediction model that uses as input data sound information including snoring or breathing sounds while a user is sleeping or a spectrogram obtained by converting the sound information, and as output data information regarding the occurrence of snoring or breathing sounds while the user is sleeping; a second specifying step of specifying whether or not a silent section between adjacent snoring sounds, a silent section between adjacent snoring sounds and breathing sounds, and / or a silent section between adjacent breathing sounds satisfy a predetermined condition based on information regarding the occurrence of snoring sounds or breathing sounds specified in the first specifying step; The method comprising:

8. A system comprising at least one computing device, A means for identifying the occurrence of apnea or hypopnea during a user's sleep based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a prediction model machine-learned with input data being sound information during a person's sleep or a spectrogram obtained by converting the sound information, and output data being information regarding the occurrence of apnea or hypopnea during the sleep A system comprising:

9. A type identification means for identifying the type of apnea or hypopnea identified by the identification means. The system of claim 8 , comprising:

10. Computer equipment A means for identifying the occurrence of apnea or hypopnea during a user's sleep based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a prediction model machine-learned with input data being sound information during a person's sleep or a spectrogram obtained by converting the sound information, and output data being information regarding the occurrence of apnea or hypopnea during the sleep A program that functions as a

11. A method executed on at least one computing device, comprising: A step of identifying the occurrence of apnea or hypopnea during the user's sleep based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information, using a prediction model machine-learned with the input data being sound information during the user's sleep or a spectrogram obtained by converting the sound information, and the output data being information regarding the occurrence of apnea or hypopnea during the user's sleep. The method comprising:

12. A computer device, A display means for displaying an area corresponding to when the user is sleeping in an arc shape, and displaying an area corresponding to when apnea or hypopnea occurs within the area corresponding to when the user is sleeping. A program that functions as a

13. The program according to claim 12, wherein the display means displays the area corresponding to when apnea or hypopnea occurs in different display modes depending on the length of time that apnea and / or hypopnea continues and / or the number of apnea and / or hypopnea per unit time or the total or average length of time that apnea and / or hypopnea continues per unit time.

14. 14. The program of claim 12 or 13, wherein the display means further displays information indicating the total, average or longest duration of apnea and / or hypopnea, the total number of apneas and / or hypopnea, and / or the number of apneas and / or hypopnea per unit time or the total or average duration of apnea and / or hypopnea per unit time. A display means for displaying an area corresponding to when the user is sleeping in an arc shape, and displaying an area corresponding to when apnea or hypopnea occurs within the area corresponding to when the user is sleeping. A computer device comprising:

16. A method executed on at least one computing device, comprising: A display step of displaying an area corresponding to when the user is sleeping in an arc shape, and displaying an area corresponding to when apnea or hypopnea occurs within the area corresponding to when the user is sleeping. The method comprising:

17. A computer device, An acquisition means for acquiring sound information of a user while sleeping; A generating means for generating a sound file based on the acquired sound information; a transmission means for transmitting the generated sound file to a second computer device when a predetermined communication line is detected, when the duration of the generated sound file is equal to or less than a predetermined duration, or when the data amount of the sound file is equal to or less than a predetermined amount; Function as a A program in which a generating means generates a plurality of sound files having different times at which the sound information was acquired based on sound information during one sleep session of a user.

18. 18. The program according to claim 17, wherein the transmitting means transmits the generated sound file in association with information capable of identifying an order in which the sound information was acquired.

19. a notification means for notifying the second computer device of notification information indicating the existence of an unsent sound file in response to detection of a predetermined communication line when the second computer device has an unsent sound file; The program according to claim 17 or 18, comprising:

20. 19. The program according to claim 17, wherein the generating means generates a plurality of sound files according to a duration of the sound file or a data amount of the sound file.

21. 19. A program according to claim 17 or 18, wherein the sound information is used to identify occurrences of apnea or hypopnea during the user's sleep.

22. The program according to claim 17 or 18, wherein the acquisition means acquires sound information for generating sound files to be generated after the one sound file in parallel with the generation of the one sound file by the generation means and / or the transmission of the one sound file by the transmission means.

23. 19. The program according to claim 17 or 18, wherein the generating means generates sound files of different durations or amounts of data depending on the computer device on which the generating means is executed.

24. An acquisition means for acquiring sound information of a user while sleeping; A generating means for generating a sound file based on the acquired sound information; a transmitting means for transmitting the generated sound file to a second computer device when a predetermined communication line is detected, when the duration of the generated sound file is equal to or less than a predetermined duration, or when the data amount of the sound file is equal to or less than a predetermined amount; Equipped with A computer device in which a generating means generates a plurality of sound files having different times at which the sound information was acquired based on sound information during one sleep session of a user.

25. A method executed on at least one computing device, comprising: An acquisition step of acquiring sound information of a user while sleeping; A generation step of generating a sound file based on the acquired sound information; a transmitting step of transmitting the generated sound file to a second computer device when a predetermined communication line is detected, when the duration of the generated sound file is equal to or less than a predetermined duration, or when the data amount of the sound file is equal to or less than a predetermined amount; having A method in which the generating step generates a plurality of sound files having different times at which the sound information was acquired, based on sound information during one sleep session of the user.

26. A system comprising a first computing device and a second computing device, a first computing device, An acquisition means for acquiring sound information of a user while sleeping; A generating means for generating a sound file based on the acquired sound information; a transmitting means for transmitting the generated sound file to a second computer device when a predetermined communication line is detected, when the duration of the generated sound file is equal to or less than a predetermined duration, or when the data amount of the sound file is equal to or less than a predetermined amount; Equipped with A generating means generates a plurality of sound files obtained at different times based on sound information obtained during one sleep period of the user, a second computing device comprising: an order specifying means for specifying the order of the sound files based on information capable of specifying the order in which the sound information was acquired; A system comprising:

Images