System, program, computer device, and method

Abstract

To provide a program capable of transmitting acquired sound information to a second computer device.SOLUTION: Provided is a program that causes a computer device to function as: acquisition means that acquires sound information during sleeping of a user; generation means that generates a plurality of sound files on the basis of the acquired sound information; and transmission means that transmits the generated sound file to a second computer device in association with information that can identify the order of acquiring the sound information. Each of the plurality of generated sound files includes pieces of sound information acquired at different times.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a system, a program, a computer device, and a method.

Background Art

[0002] Sleep apnea syndrome, which repeats apnea during sleep, increases the occurrence of ischemic lung diseases such as hypertension, stroke, and myocardial infarction due to stress such as hypoxia during sleep and daytime sleepiness. In addition, sleep apnea syndrome is known to cause complications such as diabetes and hyperlipidemia.

[0003] The evaluation of a user's breathing state during sleep is generally performed by a polysomnogram (PSG). However, hospitalization is required to perform a PSG, which places a heavy burden on the user. In addition, it is necessary to wear various instruments, and since the user sleeps in an environment different from usual, it has been difficult to reproduce the normal sleep state.

[0004] Therefore, for example, Patent Document 1 discloses that the sound of a subject during sleep is acquired by a microphone of a mobile terminal to evaluate the breathing state of the user. Patent Document 1 describes that an apnea state can be determined from the sound acquired by the microphone of the mobile terminal.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present invention can solve, for example, any of the following problems. A first object of the present invention is to provide a system capable of identifying a silent section. A second object of the present invention is to provide a system capable of identifying the occurrence of apnea or hypopnea during a user's sleep using sound information. A third object of the present invention is to provide a program for a user to recognize the occurrence status of apnea or hypopnea during sleep. A fourth object of the present invention is to provide a program capable of transmitting acquired sound information to a second computer device.

Means for Solving the Problems

[0007] The problems of the present invention are [1] A system including at least one computer device, which uses, as input data, sound information including snoring sounds or breathing sounds during a person's sleep or a spectrogram obtained by converting the sound information, and uses a prediction model that has been machine-learned with information regarding the occurrence of snoring sounds or breathing sounds during the sleep as output data, to identify the occurrence of snoring sounds or breathing sounds during the user's sleep based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information, and a second specifying means for specifying whether a silent section between snoring sounds before and after and between snoring sounds, a silent section between a snoring sound before and after and a breathing sound, and / or a silent section between breathing sounds before and after satisfies a predetermined first condition; [2] The system according to [1] above, further including a removing means for removing noise from the sound information during the user's sleep, wherein the first specifying means identifies the occurrence of breathing sounds or snoring sounds during the user's sleep based on the sound information from which the noise has been removed or the spectrogram obtained by converting the sound information from which the noise has been removed; [3] The system according to [1] or [2] above, wherein the first specifying means identifies the occurrence of snoring sounds or breathing sounds based on the sound information in a time period that satisfies a predetermined second condition or the spectrogram obtained by converting the sound information; [4] A system according to any one of [1] to [3] above, comprising: a third specifying means for specifying that apnea or hypopnea has occurred in a silent interval satisfying a predetermined condition; and based on information regarding the silent interval in which the specified apnea or hypopnea has occurred, calculating means for calculating the total, average, or longest length of the time during which apnea and / or hypopnea has continued, the total number of times of apnea and / or hypopnea, the number of times of apnea and / or hypopnea per unit time, and / or the total or average length of the time during which apnea and / or hypopnea has continued per unit time; [5] A system according to any one of [1] to [4] above, wherein the snoring sound or breathing sound during the person's sleep is generated immediately before or immediately after apnea or hypopnea occurs; [6] A program for causing a computer device to function as: a first specifying means for specifying the occurrence of a snoring sound or breathing sound during a user's sleep, using, as input data, sound information including the snoring sound or breathing sound during the person's sleep or a spectrogram obtained by converting the sound information, and using a prediction model machine-learned with information regarding the occurrence of the snoring sound or breathing sound during the sleep as output data; and a second specifying means for specifying whether a silent interval between successive snoring sounds, a silent interval between a snoring sound and a breathing sound, and / or a silent interval between successive breathing sounds satisfies a predetermined condition; [7] A method executed in at least one computer device, the method comprising: a first specifying step of specifying the occurrence of a snoring sound or breathing sound during a user's sleep, using, as input data, sound information including the snoring sound or breathing sound during the person's sleep or a spectrogram obtained by converting the sound information, and using a prediction model machine-learned with information regarding the occurrence of the snoring sound or breathing sound during the sleep as output data; and a second specifying step of specifying whether a silent interval between successive snoring sounds, a silent interval between a snoring sound and a breathing sound, and / or a silent interval between successive breathing sounds satisfies a predetermined condition; [8]A system comprising at least one computer device, which uses, as input data, sound information during a person's sleep or a spectrogram obtained by converting the sound information, and uses a prediction model that has been machine-learned with information regarding the occurrence of apnea or hypopnea during the sleep as output data, to identify the occurrence of apnea or hypopnea during the user's sleep based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information, the system comprising identification means.; [9]A program that causes a computer device to function as identification means for identifying the occurrence of apnea or hypopnea during a user's sleep, using, as input data, sound information during the person's sleep or a spectrogram obtained by converting the sound information, and using a prediction model that has been machine-learned with information regarding the occurrence of apnea or hypopnea during the sleep as output data, based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information.;

[10] A method executed in at least one computer device, the method comprising: using, as input data, sound information during a person's sleep or a spectrogram obtained by converting the sound information, and using a prediction model that has been machine-learned with information regarding the occurrence of apnea or hypopnea during the sleep as output data, to identify the occurrence of apnea or hypopnea during the user's sleep based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information, the method having an identification step.;

[11] A program that causes a computer device to function as display means for displaying a region corresponding to the time when apnea or hypopnea occurs within a region corresponding to the user's sleep.;

[12] The program according to

[11] , wherein the display means displays the region corresponding to the user's sleep in an arc shape or a band shape.;

[13] The program according to

[11] or

[12] , wherein the display means displays the region corresponding to the time when apnea or hypopnea occurs in different display modes according to the length of time that apnea and / or hypopnea continues, and / or the number of times per unit time of apnea and / or hypopnea or the total or average length of the time that apnea and / or hypopnea continues per unit time.;

[14] The display means further displays information indicating the total, average, or longest length of the duration of apnea and / or hypopnea, the total number of apnea and / or hypopnea, and / or the number of apnea and / or hypopnea per unit time or the total or average length of the duration per unit time, the program according to any one of

[11] to

[13] above;

[15] A computer device comprising display means for displaying a region corresponding to the occurrence of apnea or hypopnea within a region corresponding to the user's sleep;

[16] A method executed in at least one computer device, the method having a display step of displaying a region corresponding to the occurrence of apnea or hypopnea within a region corresponding to the user's sleep;

[17] A program that causes a computer device to function as acquisition means for acquiring sound information during the user's sleep, generation means for generating a sound file based on the acquired sound information, and transmission means for transmitting the generated sound file to a second computer device, and the generation means generates a plurality of sound files with different acquisition times of the sound information based on the sound information during one sleep of the user;

[18] The program according to

[17] above, wherein the transmission means transmits the generated sound file in association with information capable of specifying the order in which the sound information was acquired;

[19] The program according to

[17] or

[18] above, wherein the transmission means transmits the generated sound file when a predetermined communication line is detected, when the time of the generated sound file is less than or equal to a predetermined time, or when the data amount of the sound file is less than or equal to a predetermined amount;

[20] The program according to any one of

[17] to

[19] above, comprising notification means for notifying notification information indicating the existence of untransmitted sound files in response to the detection of a predetermined communication line when there are sound files that have not been transmitted to the second computer device;

[21] The program according to any one of

[17] to

[20] above, wherein the generation means generates a plurality of sound files according to the time of the sound file or the data amount of the sound file;

[22] The program according to any one of

[17] to

[21] , wherein the sound information is used to identify the occurrence of apnea or hypopnea during the user's sleep;

[23] The program according to any one of

[17] to

[22] , wherein the acquisition means acquires sound information for generating a sound file generated after the one sound file by the generation means and / or transmitted by the transmission means in parallel with the generation of the one sound file by the generation means and / or the transmission of the one sound file by the transmission means;

[24] The program according to any one of

[17] to

[23] , wherein the generation means generates sound files with different times or data amounts according to the computer device that executes the generation means;

[25] A computer device comprising an acquisition means for acquiring sound information during the user's sleep, a generation means for generating a sound file based on the acquired sound information, and a transmission means for transmitting the generated sound file to a second computer device, wherein the generation means generates a plurality of sound files with different times at which the sound information was acquired based on the sound information during one sleep of the user;

[26] A method executed in at least one computer device, the method having an acquisition step of acquiring sound information during the user's sleep, a generation step of generating a sound file based on the acquired sound information, and a transmission step of transmitting the generated sound file to a second computer device, wherein the generation step generates a plurality of sound files with different times at which the sound information was acquired based on the sound information during one sleep of the user;

[27] A system comprising a first computer device and a second computer device, wherein the first computer device comprises an acquisition means for acquiring sound information during the user's sleep, a generation means for generating a sound file based on the acquired sound information, and a transmission means for transmitting the generated sound file to the second computer device, the generation means generates a plurality of sound files with different times at which the sound information was acquired based on the sound information during one sleep of the user, and the second computer device comprises an order specifying means for specifying the order of the sound files based on information capable of specifying the order in which the sound information was acquired;

[28] A system comprising at least one computer device, the system comprising: first specifying means for specifying whether a silent interval between snoring sounds before and after, a silent interval between a snoring sound and a breathing sound before and after, and / or a silent interval between breathing sounds before and after in sound information during a user's sleep satisfies a predetermined condition; and second specifying means for specifying the occurrence of apnea or hypopnea during the user's sleep in a silent interval specified as satisfying the predetermined condition by the first specifying means, based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a prediction model that is machine-learned with the sound information during a person's sleep or the spectrogram obtained by converting the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data;

[29] A program for causing a computer device to function as: first specifying means for specifying whether a silent interval between snoring sounds before and after, a silent interval between a snoring sound and a breathing sound before and after, and / or a silent interval between breathing sounds before and after in sound information during a user's sleep satisfies a predetermined condition; and second specifying means for specifying the occurrence of apnea or hypopnea during the user's sleep in a silent interval specified as satisfying the predetermined condition by the first specifying means, based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a prediction model that is machine-learned with the sound information during a person's sleep or the spectrogram obtained by converting the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data; A method executed in at least one computer device, comprising: a first specifying step of specifying whether a predetermined condition is satisfied in a silent interval between snoring sounds before and after, between a snoring sound and a breathing sound before and after, and / or between breathing sounds before and after in sound information during a user's sleep; and a second specifying step of specifying the occurrence of apnea or hypopnea during the user's sleep in a silent interval specified as satisfying the predetermined condition in the first specifying step, based on the sound information during the user's sleep or a spectrogram obtained by converting the sound information, using a prediction model that is machine-learned with the sound information during a person's sleep or the spectrogram obtained by converting the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data; which can be solved by.

Advantages of the Invention

[0008] The present invention can achieve, for example, any of the following advantages. According to the present invention, a system capable of specifying a silent interval can be provided. According to the present invention, a system capable of specifying the occurrence of apnea or hypopnea during a user's sleep using sound information can be provided. According to the present invention, a program for enabling a user to recognize the occurrence status of apnea or hypopnea during sleep can be provided. According to the present invention, a program capable of transmitting the acquired sound information to a second computer device can be provided.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described. However, the present invention is not limited to the following embodiments as long as it does not contravene the gist of the present invention. The order of each process constituting the flowchart described below may be in any order as long as there is no contradiction or inconsistency in the process content. Also, within the range where there is no contradiction or inconsistency in the process content, it is possible to omit a part of each process constituting the flowchart or add a new process to each process constituting the flowchart. Further, the device that is the main body for executing each process constituting the flowchart can be changed to another device as long as it does not contravene the gist of the present invention. In that case, it is possible to change the process content so that there is no contradiction or inconsistency in the process content.

[0011] [Configuration of the System] FIG. 1 is a block diagram showing the configuration of a system according to an embodiment of the present invention. System 10 includes at least one computer device. System 10 includes a user terminal 1 and a server device 2. The user terminal 1 is operated by a user. The server device 2 may be managed by an administrator who manages the system 10. Although not shown, the system 10 may include an administrator terminal operated by an administrator who manages the server device 2. Further, the system 10 may include a computer device that constructs a prediction model described later. The user terminal 1 is an example of a first computer device, and the server device 2 is an example of a second computer device.

[0012] The user terminal 1 and the server device 2 are communicably connected to each other via a communication network 3. In the system 10, either the user terminal 1 or the server device 2 can function as an information processing device. When either the user terminal 1 or the server device 2 functions as an information processing device, information is transmitted and received between the user terminal 1 and the server device 2 as necessary.

[0013] Note that the system 10 may include two or more user terminals 1. Further, the system 10 may include two or more server devices 2. Further, the server device 2 may be distributed and function in a plurality of computer devices. For example, instead of the server device 2, a distributed ledger technology such as a blockchain may be used.

[0014] [Overview of the System] The system 10 according to the present embodiment can include a transmission process of acquiring, at the user terminal 1, sound during the user's sleep as sound information and transmitting the sound information to the server device 2, and an analysis process of analyzing the transmitted sound information at the server device 2.

[0015] The sound information is information obtained by converting the sound around the user terminal 1 collected by the acquisition unit 14 of the user terminal 1 into an electrical signal. The sound around the user terminal 1 includes not only the sound emitted by the user but also environmental sound (noise) such as the operating sound of electronic devices such as an air conditioner.

[0016] The sound information acquired by the acquisition unit 14 is transmitted to the server device 2 as a sound file. The sound information acquired in one measurement may be divided into two or more sound files. A sound file is a collection of one piece of data that can manage and store the acquired sound information (electrical signal). The file format of the sound file is WAVE, AIFF, MP3, AAC, FLAC, etc. The sound file will be described later.

[0017] In the analysis process, the server device 2 can remove noise such as environmental sound from the sound information contained in the sound file, and can identify the occurrence of breathing sounds and snoring sounds from the sound information contained in the sound file. Also, in the analysis process, the server device 2 can identify the occurrence of apnea or hypopnea during the user's sleep from the sound information contained in the sound file. Further, in the analysis process, the server device 2 can identify the type of apnea or hypopnea during the user's sleep from the sound information contained in the sound file.

[0018] Also, the analysis process includes a display process for displaying the analysis result on the user terminal 1. By the display process, the user can visually recognize the occurrence status of apnea or hypopnea during the user's sleep.

[0019] In this embodiment, the occurrence of "apnea" means that the user's breathing stops for a predetermined time (for example, 10 seconds) or more during sleep. Also, the occurrence of "hypopnea" means that the oxygen saturation (SpO 2 ) during sleep decreases by a predetermined ratio (for example, 3%) or more from the average value of the oxygen saturation of one user during one sleep.

[0020] The occurrence status of apnea or hypopnea (also referred to as apnea, etc.) includes, for example, the time when apnea or hypopnea occurred, the duration of the time when apnea or hypopnea occurred, whether either apnea or hypopnea occurred, and / or information regarding the type of apnea or hypopnea. The occurrence status of apnea, etc. can be obtained from the occurrence of apnea, etc. during the user's sleep identified in the analysis process and / or the type of apnea, etc.

[0021] [User terminal] The user terminal 1 is not particularly limited as long as it is a computer device including an acquisition device capable of acquiring sound information and a display device. Examples of the user terminal 1 include a conventional mobile phone, a tablet terminal, a smartphone, etc. Further, the user terminal 1 may be, for example, a computer device that can be placed beside the user during the user's sleep.

[0022] FIG. 2 is a block diagram showing the hardware configuration of the user terminal according to an embodiment of the present invention. The user terminal 1 includes a control unit 11, a RAM 12, a storage unit 13, an acquisition unit 14, an input unit 15, a display unit 16, and a communication interface 17, which are connected by an internal bus, respectively.

[0023] The control unit 11 is composed of a CPU and a ROM. The control unit 11 executes the program stored in the storage unit 13 to control the user terminal 1. The RAM 12 is a work area of the control unit 11. The storage unit 13 is a storage area for storing programs and data. That is, the storage unit 13 functions as a recording medium storing the program. The control unit 11 performs arithmetic processing based on the program and data read from the RAM 12, the acquisition unit 14, and the data input by the input unit 15.

[0024] The acquisition unit 14 converts the sound around the user terminal 1 into an electrical signal and acquires it as sound information. The acquisition unit 14 has, for example, a microphone. The acquisition unit 14 is an acquisition device capable of acquiring sound information. Note that the acquisition unit 14 may be provided in the user terminal 1 by attaching an external microphone.

[0025] The display unit 16 has a display screen. The control unit 11 outputs a video signal for displaying an image on the display screen according to the result of arithmetic processing. Here, the display screen of the display unit 16 may be a touch panel provided with a touch sensor. In this case, the touch panel functions as the input unit 15.

[0026] The communication interface 17 can be connected to the communication network 3 wirelessly or by wire, and can transmit and receive data with other computer devices via the communication network 3. The data received via the communication interface 17 is loaded into the RAM 12, and arithmetic processing is performed by the control unit 11.

[0027] [Server device] FIG. 3 is a block diagram showing the hardware configuration of the server device according to an embodiment of the present invention. The server device 2 includes at least a control unit 21, a RAM 22, a storage unit 23, and a communication interface 24, which are connected by an internal bus respectively.

[0028] The control unit 21 is composed of a CPU and a ROM, executes the program stored in the storage unit 23, and controls the server device 2. Also, the control unit 21 includes an internal timer for measuring time. The RAM 22 is a work area of the control unit 21. The storage unit 23 is a storage area for storing programs and data. That is, the storage unit 23 functions as a recording medium storing the program. The control unit 21 reads the program and data from the RAM 22, and performs program execution processing based on information received from the user terminal 1 and the like.

[0029] Also, the program may be stored in a recording medium such as a CD-ROM. In this case, the program stored in the recording medium may be installed in the user terminal 1 or the server device 2 to execute a predetermined function.

[0030] Alternatively, the program may be distributed from a computer device outside the system. In this case, the program distributed from the computer device outside the system may be installed in the user terminal 1 or the server device 2 to execute a predetermined function.

[0031] [Initial Settings] In the system 10 of the present embodiment, information about the user who uses the system 10 is registered in the server device 2 in advance. When the user logs in to the system 10 for the first time, the user performs initial settings by operating the user terminal 1 and inputting information about the user.

[0032] First, the user starts an application program (hereinafter referred to as a dedicated app) downloaded to the user terminal 1 and accesses the server device 2 to log in to the system 10. Also, the user may log in to the system 10 by accessing the server device 2 from the user terminal 1 via a web browser.

[0033] When logging in to the system 10, as initial settings, the user inputs user information about the user, such as the user name, password, gender, date of birth, medical history, occupation, and affiliated company. Next, by sending the input user information to the server device 2, the user is registered in the server device 2. It is preferable that the registered user is given user identification information (hereinafter also referred to as a user ID) for identifying the user. The user ID may be stored in association with the user information. Also, in the system 10, a page dedicated to the registered user is created. Even if the user uses a plurality of terminals, the same page can be displayed with the same user ID.

[0034] When logging in to the system 10, it may be required to input a pre-registered user ID and password. Alternatively, when logging in to the system 10, it may be required to input a pre-registered email address and password.

[0035] [Transmission Process] Next, the transmission process according to the system 10 of the present embodiment will be described. In the transmission process, when the user terminal 1 transmits the sound information during the user's sleep to the server device 2, a sound file is generated based on the acquired sound information, and the generated sound file is transmitted to the server device 2. The transmitted sound information is used to identify the occurrence of apnea or hypopnea during the user's sleep in the analysis process described later.

[0036] In the transmission process, the user performs the following operations. Before going to sleep, the user starts a dedicated application from the user terminal 1. Next, the user inputs to the user terminal 1 to start the acquisition of sound information and then starts to sleep. When the user wakes up, the user inputs to the user terminal 1 to end the acquisition of sound information.

[0037] FIG. 4 is a diagram showing a flowchart of the transmission process according to the embodiment of the present invention. The user operates the input unit 15 of the user terminal 1 to start a dedicated application. By accessing the server device 2 from the user terminal 1, the system 10 is logged in. Logging in to the system 10 may be performed by accessing the server device 2 from the user terminal 1 via a web browser.

[0038] After logging in to the system 10, the user inputs to the user terminal 1 to start the acquisition of sound information. For example, the user can input to start the acquisition of sound information by selecting the "Start Recording" icon displayed on the user terminal 1. When the user terminal 1 receives the input to start the acquisition, the acquisition unit 14 acquires the sound information during the user's sleep (step S1).

[0039] After the acquisition of sound information is started in the user terminal 1, when a predetermined condition is satisfied (step S2), a sound file is generated in the user terminal 1 based on the acquired sound information (step S3). The generated sound file is transmitted from the user terminal 1 to the server device 2 (step S4). At the server device 2, the sound file is received (step S5) and stored (step S6). In step S6, the sound file is stored in the storage unit 23 of the server device 2 in association with information that can identify the order in which the sound information was acquired. Also, the sound file may be stored in association with the user ID.

[0040] As described above, the sound file is a collection of one piece of data capable of managing and storing the sound information acquired in step S1. In step S3, the sound file may be generated such that the file reaches a predetermined time or a predetermined data volume. The sound files are generated in order from those containing the sound information with an earlier acquisition time.

[0041] Also, one sound file and a second sound file generated immediately after the one sound file may include sound information with different acquisition times, or may include sound information with a part of the acquisition time being the same. Specifically, a first sound file including sound information from 23:30 to 0:00 and a second sound file including sound information from 0:00 to 0:30 may be generated, or a first sound file including sound information from 23:30 to 0:00 and a second sound file including sound information from 23:55 to 0:25 may be generated.

[0042] The predetermined conditions in step S2 include, for example, that a predetermined time has elapsed since the start of acquiring sound information, that a predetermined time has elapsed since the final time of the sound information in the generated or to-be-generated sound file, that the data amount of the acquired sound information exceeds a predetermined amount, that the data amount of the sound information not yet converted into a sound file exceeds a predetermined data amount, and / or that an operation for ending the acquisition of sound information has been performed on the user terminal 1. In step S3, specifically, a sound file may be generated every 30 minutes after the sound information is acquired by the user terminal 1, or the sound file may be generated so that the data amount of the sound file is within 15 MB. The system 10 of the present embodiment can generate a plurality of sound files according to the time of the sound file or the data amount of the sound file in step S3.

[0043] The predetermined conditions for generating a sound file can be set as appropriate, but may be set in advance by input from the administrator terminal by the administrator or input from the user terminal 1 by the user.

[0044] The operation for ending the acquisition of sound information on the user terminal 1 is, for example, an operation in which the user selects the "Stop Recording" icon displayed on the user terminal 1 after waking up. By this operation, an input for ending the acquisition of sound information can be made. When the end of the acquisition of sound information is input, a sound file can be generated even before a predetermined time has elapsed or before the data amount exceeds a predetermined data amount for the sound information being acquired. Therefore, the time or data amount of the last generated sound file may be different from the time or data amount of other sound files. Note that by accepting the input for ending the acquisition of sound information on the user terminal 1, end information may be input to the user terminal 1. At this time, the end information is included in the sound information.

[0045] The processes in steps S2 to S6 are repeatedly executed until the processing of all sound information during the user's sleep time is completed. The system 10 of the present embodiment can generate a plurality of sound files with different times when the sound information was acquired, based on the sound information during one sleep of the user. Through steps S1 to S6, the transmission process ends.

[0046] Here, the process of step S4 will be described. The user terminal 1 may transmit the generated sound file in association with information that can specify the order in which the sound information was acquired. Information that can specify the order in which the sound information was acquired is, for example, time information regarding the time when the sound information included in the sound file was acquired, order information regarding the order in which the sound file was generated, time information when the sound file was transmitted to the server device 2 in step S4, and the like.

[0047] Alternatively, the information that can specify the order in which the sound information was acquired may be, for example, the time information when the server device 2 received the sound file in step S5. In this case, in step S4, the user terminal 1 does not necessarily need to transmit the generated sound file in association with the information that can specify the order in which the sound information was acquired.

[0048] Regarding the process of acquiring sound information in step S1, the user terminal 1 may acquire sound information for generating sound files generated after the next of the one sound file in parallel with the process of generating the one sound file in step S3 and / or the process of transmitting the one sound file in step S4. Also, the process of generating the sound file in step S3 may be executed in accordance with the time series in which the sound information was acquired.

[0049] For example, when the acquisition of sound information starts at 23:00 and the setting is to generate a sound file every 30 minutes, in the user terminal 1, when it reaches 23:30, a sound file (referred to as the first sound file) containing the sound information from 23:00 to 23:30 is generated. Also, in the user terminal 1, in parallel with the generation of the first sound file, the acquisition of sound information after 23:30 is executed. Furthermore, in the user terminal 1, when the first sound file is generated, in parallel with the transmission of the first sound file, the acquisition of sound information after 23:30 is executed.

[0050] Furthermore, when it reaches 0:00 during the generation of the first sound file, in the user terminal 1, in parallel with the generation of the first sound file, the generation of a sound file (referred to as the second sound file) containing the sound information from 23:30 to 0:00 may be executed. Also, when the first sound file is generated during the generation of the second sound file, in the user terminal 1, in parallel with the generation of the second sound file, the transmission of the first sound file may be executed. That is, regarding the process of generating the sound file in step S3, the user terminal 1 may generate the sound files generated after the first sound file in parallel with the process of generating 1 sound file according to step S3 and / or the process of transmitting 1 sound file according to step S4. Also, the user terminal 1 may transmit a plurality of sound files simultaneously regarding the process of transmitting the sound file in step S4.

[0051] Alternatively, after the acquisition of the sound information is completed, the system 10 may generate a plurality of sound files with different times when the sound information was acquired based on the acquired sound information.

[0052] Note that in step S4, the sound file transmitted to the server device 2 may be deleted from the user terminal 1 or may be stored in the storage unit 13 of the user terminal 1.

[0053] Note that in step S3, the system 10 of the present embodiment may generate audio files with different times or data amounts according to the user terminal 1 and / or the server device 2 that execute the process of generating the audio file. Specifically, when the data capacity of the user terminal 1 that executes the generation process is less than a predetermined capacity, an audio file with a shorter time than normal or an audio file with a smaller data amount than normal may be generated. Also, when the processing speed of the server device 2 that receives the audio file is slower than a predetermined processing speed, an audio file with a shorter time than normal or an audio file with a smaller data amount than normal may be generated. That is, the predetermined conditions in step S2 may be changed according to the user terminal 1 and / or the server device 2 that execute the generation process. Note that information regarding the user terminal 1 and / or the server device 2 may be stored in the user terminal 1 in advance, or may be acquired by the user terminal 1 before executing the transmission process.

[0054] In addition, the system 10 of the present embodiment may generate audio files with different times or data amounts according to the time zone in which the generation process is executed in step S3. Specifically, the system 10 may generate an audio file with a shorter time than normal or an audio file with a smaller data amount than normal during a time zone when the line is congested (for example, from 10:00 to 12:00), and may generate an audio file with a longer time than normal or an audio file with a larger data amount than normal during a time zone when the line is not congested (for example, from 12:00 to 5:00). That is, the predetermined conditions in step S2 may be changed according to the time zone in which the generation process is executed.

[0055] Note that before starting to acquire the sound information in step S1, instruction information for the user may be displayed. The instruction information may include, for example, cautions regarding the location where the user terminal 1 is placed, cautions regarding environmental sounds such as turning off the TV in the room, connecting the user terminal 1 to the power supply, dimming the brightness of the display screen of the user terminal 1, or cautions regarding the state of the user terminal 1 such as checking the data capacity of the user terminal 1. Further, when the acquisition of the sound information in step S1 is started, the system 10 of the present embodiment may control to dim the brightness of the display screen of the user terminal 1. At this time, the system 10 may control to make the brightness of the display screen of the user terminal 1 the darkest, or may control to dim it according to the brightness of the room.

[0056] Note that in step S4, the user terminal 1 may transmit the sound file in response to satisfying a predetermined transmission condition. The predetermined transmission condition is, for example, detecting a predetermined communication line, the time of the generated sound file being equal to or less than a predetermined time, the data amount of the sound file being equal to or less than a predetermined amount, or a user operation input being made to the user terminal 1.

[0057] Detecting a predetermined communication line means, for example, detecting a line without data capacity limitation such as Wi-Fi. When the predetermined transmission condition is detecting a predetermined communication line, the user terminal 1 may be configured to, in step S3, not generate a sound file and, in step S4, transmit the acquired sound information as one sound file without dividing the acquired sound information.

[0058] Further, in the transmission process, steps S2 and S3 may be repeatedly executed, and the user terminal 1 may transmit the sound file in response to satisfying the above-described predetermined transmission condition.

[0059] Also, before executing step S4, the user terminal 1 may output instruction information for prompting the user to transmit a sound file in response to detecting a predetermined communication line. In response to a user's operation input to the user terminal 1, the user terminal 1 may transmit the sound file.

[0060] Also, when there is a sound file that has not been transmitted to the server device 2, the system 10 may notify notification information indicating the existence of the untransmitted sound file in response to satisfying a predetermined notification condition. The predetermined notification condition is, for example, detecting a predetermined communication line or elapse of a predetermined time (e.g., one day) since acquisition of sound information.

[0061] When the system 10 detects a predetermined communication line, it can prompt the user to transmit the sound file without increasing the data usage amount by the user terminal 1 borne by the user due to the transmission of the sound file by transmitting the sound file or indicating the existence of an untransmitted sound file.

[0062] [Analysis process] Next, the analysis process according to the system 10 of the present embodiment will be described. The analysis process is executed after all sound files during the user's sleep are stored in the server device 2 by the transmission process. Alternatively, the analysis process may be executed each time a sound file is stored in the server device 2.

[0063] By the analysis process in the server device 2, for example, the length of time during which a snoring sound of 1 continues, the total or longest length of the time during which the snoring sound continues, the total number of times the snoring sound occurs, the number of times per unit time of the snoring sound or the total or average length of the continuous time per unit time, the length of time during which a breathing sound of 1 continues, the total or longest length of the time during which the breathing sound continues, the total number of times the breathing sound occurs, the number of times per unit time of the breathing sound or the total or average length of the continuous time per unit time, the time when apnea or hypopnea of 1 occurs, the length of time during which apnea or hypopnea continues, the total or longest length of the time during which apnea or hypopnea continues, the total number of times of apnea or hypopnea, the number of times per unit time of apnea or hypopnea or the total or average length of the continuous time per unit time, etc. can be specified. The above information specified by the analysis process is also referred to as the analysis result.

[0064] FIG. 5 is a diagram showing a flowchart of the analysis process according to an embodiment of the present invention. In the server device 2, the order of the sound files is specified (step S11). Next, in the server device 2, noise is removed from the sound information included in one sound file (step S12).

[0065] Here, the noise is a sound other than the voice emitted by the user. For example, the noise is an environmental sound such as the operating sound of an electronic device such as an air conditioner, a sound naturally generated by rain or wind, or a sound when the user turns over. The noise may be a sound other than the voice emitted by the user and having a specific pattern (a certain frequency or periodicity). The voice emitted by the user is, for example, a breathing sound such as a snoring sound. Also, even if it is a voice emitted by the user, a sneeze or a sleep talk may be regarded as noise.

[0066] The noise removal method can adopt a technique for removing known noise components and can be executed as follows, for example. The server device 2 removes ambient noise from the sound information included in the sound file. For removing ambient noise, a technique for suppressing known noise components can be adopted. For example, filtering such as the spectral subtraction method, spectral restoration, and methods such as a human voice model can be adopted. When using the spectral subtraction method, for example, the Wiener filtering method can be used.

[0067] Next, the server device 2 identifies an event that is a candidate for breathing sounds and snoring sounds from the sound information from which the ambient noise has been removed (step S13). Step S13 is a process of identifying candidates for time zones (intervals) corresponding to the user's breathing sounds and snoring sounds from the sound information. Step S13 is also a process of identifying a time zone (interval) that satisfies a predetermined condition (also referred to as a predetermined second condition) from the sound information from which the ambient noise has been removed. In the present embodiment, among the sound information, a time zone that satisfies the predetermined second condition is referred to as an event. The predetermined second condition is, for example, that the volume of the sound information from which the ambient noise has been removed is equal to or greater than a predetermined volume or exceeds the predetermined volume, and the time zone equal to or greater than the predetermined volume or the time zone exceeding the predetermined volume can be defined as an event. For example, in the server device 2, a predetermined volume is set in advance, and an event can be identified for the sound information from which the ambient noise has been removed.

[0068] Also, when the user terminal 1 acquires the voice emitted by the user at a position close to the user and when the user terminal 1 acquires the voice emitted by the user at a position far from the user, even for the same breathing sound, the volume is different, and even for the same snoring sound, the volume is different. Even when the user terminal 1 is placed at the same position to acquire voice, the position of the user changes for reasons such as turning over in bed. Therefore, an event that is a candidate for breathing sounds and snoring sounds may be identified based on the relative magnitude of the volume. Furthermore, an event that is a candidate for breathing sounds and snoring sounds may be identified based not only on the volume of the sound information from which the ambient noise has been removed but also on other elements constituting the sound information such as frequency and sound duration.

[0069] Next, at the server device 2, an apnea / similar event identification process for identifying the occurrence of apnea or hypopnea is executed (step S14). Then, information regarding the identified occurrence status of apnea or similar events is stored in the server device 2 (step S15). The information regarding the occurrence status of apnea or similar events may be stored in the server device 2 in association with the user ID. The processes of steps S12 to S15 are repeatedly executed until the processing of all the sound files acquired during the user's sleep is completed.

[0070] When the processing of all the sound files is completed, at the server device 2, display information for displaying the analysis result on the user terminal 1 is generated (step S16), and the display information is transmitted to the user terminal 1 (step S17). When the user terminal 1 receives the display information (step S18), the display information is displayed on the user terminal 1 (step S19). Through the above processes of steps S11 to S19, the analysis process ends.

[0071] Here, the process of specifying the order of the sound files in step S11 will be described. Since, in step S6, information capable of specifying the order in which the sound file and the sound information were acquired is stored in the server device 2 in association, the server device 2 can specify the order of the sound files based on the information capable of specifying the order in which the sound information was acquired. Note that the sound files for which the order is to be specified are the sound files corresponding to the sound information acquired during one sleep of the user. Specifically, the sound files corresponding to the sound information acquired during one sleep of the user are the sound files acquired at the user terminal 1 until the input for starting the acquisition of the sound information is received and then the input for ending the acquisition of the sound information is received.

[0072] The order of the audio files is determined by the time series in which the audio information is acquired. For example, when acquiring audio information from 23:00 to 1:00 and generating an audio file every 30 minutes, four audio files are generated. At this time, the order of the audio files is the order of the audio file from 23:00 to 23:30, the audio file from 23:30 to 0:00, the audio file from 0:00 to 0:30, and the audio file from 0:30 to 1:00.

[0073] Therefore, if the information that can specify the order in which the audio information is acquired is time information regarding the time when the audio information included in the audio file is acquired, the order of the audio files can be specified based on the time when the audio information is acquired. Also, if the information that can specify the order in which the audio information is acquired is order information regarding the order in which the audio files are generated, and the audio files are generated in the order in which the audio information is acquired, the order of the audio files can be specified based on the order in which the audio files are generated. Further, if the information that can specify the order in which the audio information is acquired is time information regarding the time when the audio file is transmitted to the server device 2 in step S4, and the audio files are generated in the order in which the audio information is acquired and the audio file is transmitted to the server device 2 immediately after generation, the order of the audio files can be specified based on the time when the audio file is transmitted to the server device 2.

[0074] Also, if the information that can specify the order in which the audio information is acquired is time information regarding the time when the audio file is received by the server device 2, the order of the audio files can be specified based on the time when the audio file is received.

[0075] In step S11, the system 10 of the present embodiment determines that a plurality of sound files transmitted from the same user terminal 1 during the night to morning of the same day, that is, within the same one sleep period (also referred to as one sleep time), are one related and coherent file in the server device 2. As a result, the analysis results of these plurality of sound files are generated as one display information in step S16 and displayed as one display information in step S19. Note that the generated display information may display the analysis results in time series based on the order specified in step S11 and / or time information corresponding to any of the states of breathing sound, snoring sound, apnea, and hypopnea.

[0076] Note that the noise removal process in step S12 and / or the event identification process in step S13 may be omitted. Also, the processes of steps S12 to S15 may be executed for some of the sound files during one sleep time of the user. Further, when time information regarding the time when the sound information of the sound file was acquired is included, the process of specifying the order of the sound files in step S11 may be omitted. Also, after executing the process of specifying the order of the sound files in step S11, one sound file obtained by combining the plurality of sound files may be generated again, and the processes of steps S12 to S15 may be executed for this sound file.

[0077] [Apnea and other specific processing] Next, the apnea and other specific processing in step S14 will be described. FIG. 6 is a diagram showing a flowchart of the apnea and other specific processing according to an embodiment of the present invention. The apnea and other specific processing is executed by the server device 2.

[0078] First, in the server device 2, the breathing sound and / or snoring sound of the user is identified from the sound information in the sound file (step S21).

[0079] The breathing sound and / or snoring sound of the user can be identified as follows, for example. For example, sound information including the snoring sound or breathing sound during a person's sleep or a spectrogram obtained by converting the sound information is used as input data, and a prediction model (also referred to as a first prediction model) that has been machine-learned with information regarding the occurrence of the snoring sound or breathing sound during the sleep as output data can be used.

[0080] Information regarding the occurrence of the snoring sound or breathing sound includes, for example, information regarding whether the snoring sound or breathing sound has occurred, information regarding when the snoring sound or breathing sound has occurred, information regarding whether either the snoring sound or breathing sound has occurred, and / or information regarding the type of the snoring sound or breathing sound. Using the first prediction model described above, it is possible to identify that a breathing sound is occurring and / or that a snoring sound is occurring in the event specified in step S13 based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information. That is, in step S21, it is possible to identify whether the event specified as a candidate for the snoring sound or breathing sound in step S13 is a snoring sound or a breathing sound.

[0081] The system 10 of the present embodiment may identify the occurrence status of the snoring sound or breathing sound for the snoring sound or breathing sound identified in step S21. The occurrence status of the snoring sound or breathing sound includes, for example, the time when the breathing sound or snoring sound has occurred, the length of time when the breathing sound or snoring sound has occurred, whether either the breathing sound or snoring sound has occurred, and / or information regarding the type of the breathing sound or snoring sound.

[0082] The process of step S21 will be described. For example, the first prediction model is stored in the storage unit 23 of the server device 2. The machine learning algorithm is not particularly limited, and a known algorithm can be used. Examples thereof include linear regression, multiple regression analysis, support vector machine, decision tree, random forest, and deep learning using a multi-layer neural network.

[0083] A multi-layer neural network has an input layer, an output layer, and a plurality of intermediate layers. Weights are set for the edges connecting the nodes of each layer. For each edge, weights corresponding to each input to the node are set, and the weights corresponding to each input to the node are multiplied, and the value obtained by multiplying these weights and the bias are added. The value obtained by the addition is non-linearly transformed using an activation function to calculate an activation value. The calculated activation value becomes the value of the input passed to the nodes of the next layer. The number of intermediate layers can be appropriately designed. The weights are optimized by the above teacher data.

[0084] The input data, which is sound information including snoring sounds or breathing sounds during a person's sleep or a spectrogram obtained by converting the sound information, includes, for example, time information, the frequency of the sound, and / or the loudness of the sound. The spectrogram used as the input data is the result of calculating the frequency spectrum by passing the sound information through a window function. The spectrogram is a so-called voiceprint. In the present embodiment, the sound information may be converted into a mel-frequency spectrogram or may be converted into MFCC (Mel Frequency Cepstral Coefficients). When using a spectrogram as the input data, in step S21, the server device 2 converts the sound information into a spectrogram. The sound information that the server device 2 converts into a spectrogram may be only the part corresponding to a time period (event) that satisfies a predetermined second condition.

[0085] A spectrogram is a spectrum of each audio data segment (frame) obtained by extracting frequency components and amplitude components from the waveform of the electrical signal of an audio file by Fourier transform, arranged along the time axis. A spectrogram is represented by a three-dimensional graph (time, frequency, strength of the signal component). The horizontal axis of the spectrogram represents time, and the vertical axis represents frequency. Also, a spectrogram visualizes the strength of the signal component (amplitude) in each time component and frequency component by differences in display color or shading. A spectrogram is image information.

[0086] A mel-frequency spectrogram (also called a mel spectrogram) is calculated by applying a mel filter bank that extracts only specific frequency bands that are equally spaced on the mel scale. The mel scale is a scale based on the frequency perception of the human ear (sensitive to low-frequency sounds and insensitive to high-frequency sounds).

[0087] When the input data is the sound information in the time zone corresponding to the event or the spectrogram obtained by converting the sound information, the output data can be information regarding whether or not snoring sounds or breathing sounds have occurred in that time zone. Alternatively, when the input data is the sound information having a predetermined length or the spectrogram obtained by converting the sound information, the output data may be information regarding in which section of this sound information snoring sounds or breathing sounds have occurred. The above snoring sounds or breathing sounds may occur immediately before or immediately after apnea or hypopnea occurs during a person's sleep.

[0088] As the output data, for example, data obtained by a person listening to the sound corresponding to the sound information and determining whether it is a snoring sound, a breathing sound, or either a breathing sound or a snoring sound may be used. Also, as the output data, data determined based on the frequency components and / or the loudness of the sound included in the sound information as to whether or not a breathing sound has occurred or whether or not a snoring sound has occurred may be used. Further, as the output data, candidates (events) for snoring sounds or breathing sounds immediately before or after the time zone in which apnea or hypopnea has been identified by using a portable in-sleep apnea detection device, a PSG, a doctor's judgment, etc. and determined to be snoring sounds or breathing sounds may be used. Also, the output data may be a combination of the above. Further, a prediction model machine-learned by one of the above output data may be additionally learned by different output data.

[0089] For example, the input information and output information used in the first prediction model can be obtained as follows. For example, sound information during the sleep of user A is acquired. Regarding the sound information, noise removal and event identification are performed. This is the same process as steps S12 and S13. Next, the administrator who constructs the first prediction model compares the result of identifying the occurrence of apnea or hypopnea during user A's sleep in time series with the event. If apnea or hypopnea occurred immediately before or after the event occurred, the event is considered to be a snoring sound or a breathing sound related to apnea or hypopnea. The labeled result of this evaluation becomes the output information used in the first prediction model. Also, the sound information corresponding to the event when it is considered to be a snoring sound or a breathing sound related to apnea or hypopnea, or the spectrogram (image information) obtained by converting the sound information, can be used as the input information for the first prediction model.

[0090] In this way, sound information including snoring sounds or breathing sounds during the sleep of user A, or the spectrogram obtained by converting the sound information, is used as the input information for the first prediction model, and information regarding the occurrence of snoring sounds or breathing sounds during the sleep is used as the output information. For a plurality of other users different from user A, input information and output information are obtained in the same way, and machine learning is performed based on these plurality of input information and output information.

[0091] Returning to the description of the process of step S21. When the server device 2 receives the input of the sound information included in the sound file, the control unit 21 of the server device 2 uses the first prediction model to identify information regarding the occurrence of snoring sounds or breathing sounds corresponding to the received sound information. In step S21, the data input to the first prediction model corresponds to the input data when the first prediction model was trained. For example, the server device 2 inputs the sound information corresponding to the time period (event) that satisfies a predetermined second condition, or the spectrogram obtained by converting the sound information, to the first prediction model. Then, based on the output data, the server device 2 can identify whether a snoring sound or a breathing sound has occurred for the time period (event) that satisfies the predetermined second condition.

[0092] Accordingly, regarding the snoring sound or breathing sound to be specified in step S21, the generation status of the snoring sound or breathing sound can be specified. For example, the server device 2 can specify that the sound from 0:10:00 to 0:10:05 is a breathing sound, and the sound from 0:20:00 to 0:20:08 is a snoring sound.

[0093] Furthermore, among the time zones (events) that satisfy a predetermined second condition, when the first prediction model specifies only the occurrence of a snoring sound, the server device 2 can regard an event in which the occurrence of a snoring sound is not specified among the events as a breathing sound. Alternatively, among the events, when the first prediction model specifies only the occurrence of a breathing sound, the server device 2 can regard an event in which the occurrence of a breathing sound is not specified among the events as a snoring sound.

[0094] Alternatively, among the time zones (events) that satisfy a predetermined second condition, when the first prediction model specifies the occurrence of a snoring sound or breathing sound that occurs immediately before or immediately after the occurrence of apnea or hypopnea, the server device 2 can regard an event in which the occurrence of a snoring sound or breathing sound is not specified in the first prediction model among the events as a breathing sound that has nothing to do with apnea or the like.

[0095] When the breathing sound and snoring sound of the user are specified from the sound information included in the sound file, a silent interval is specified (step S22). The silent interval is a time zone in which no breathing sound or snoring sound occurs in the sound information. One silent interval is an interval from when a breathing sound or snoring sound occurs until the next breathing sound or snoring sound occurs. In the present embodiment, the silent interval may be a time zone in which the occurrence of a breathing sound or snoring sound cannot be specified in the sound information. The silent interval does not have to be a completely silent interval and may be an interval in which sound is occurring. For example, even when the breathing sound is too small to be detected from the sound information in step S21, it may be regarded as a silent interval. In fact, this silent interval may include a breathing sound. At this time, in the silent interval, there may be hypopnea instead of apnea.

[0096] The silent interval specified in step S22 may be a silent interval that satisfies a predetermined determination condition. The predetermined determination condition is, for example, that the silent interval is a silent interval between snoring sounds before and after, a silent interval between a snoring sound and a breathing sound before and after, and / or a silent interval between breathing sounds before and after. Also, the predetermined determination condition may be, for example, that the silent interval is a silent interval between the snoring sound or breathing sound (hereinafter also referred to as the specified snoring sound, etc.) specified by the first prediction model in step S21 when they exist before and after. Note that the predetermined determination condition may be that the silent interval is a silent interval between the specified snoring sound, etc. before and after and a breathing sound unrelated to apnea, etc., and / or a silent interval between breathing sounds unrelated to apnea, etc. before and after and breathing sounds unrelated to apnea, etc.

[0097] Here, "snoring sounds before and after" can be defined as these two snoring sounds when only one silent interval exists between the occurrence of a snoring sound and the occurrence of the next snoring sound. For example, if a breathing sound occurs between the occurrence of a snoring sound and the occurrence of the next snoring sound, two silent intervals exist between the snoring sounds, so these two snoring sounds are not snoring sounds before and after. The relationship between snoring sounds before and after is also referred to as consecutive snoring sounds. Similarly, "snoring sound and breathing sound before and after" can be defined as this snoring sound and breathing sound when only one silent interval exists between the occurrence of a snoring sound and the occurrence of the next breathing sound, or between the occurrence of a breathing sound and the occurrence of the next snoring sound. "Breathing sounds before and after" can be defined as these two breathing sounds when only one silent interval exists between the occurrence of a breathing sound and the occurrence of the next breathing sound.

[0098] In step S22, when specific snoring sounds or the like exist before and after, the server device 2 can identify the interval therebetween as a silent interval that satisfies a predetermined determination condition. For example, when specific snoring sounds or the like are continuously generated, the server device 2 can identify the interval between the specific snoring sounds or the like as a silent interval that satisfies a predetermined determination condition. On the other hand, when specific snoring sounds or the like, breath sounds unrelated to apnea or the like, and specific snoring sounds or the like are generated in this order, the server device 2 does not necessarily have to identify the two silent intervals existing between the specific snoring sounds or the like as silent intervals that satisfy a predetermined determination condition.

[0099] Alternatively, in step S21, when it is specified whether the event is a snoring sound and whether the event is a breath sound, in step S22, as a silent interval that satisfies a predetermined determination condition, the server device 2 can identify the interval between the snoring sounds before and after where no breath sound exists as a silent interval. Also, the server device 2 can identify the interval between the breath sounds before and after where no snoring sound exists as a silent interval.

[0100] Regarding the process of step S22, for example, although the sound information includes breath sounds and snoring sounds, when snoring sounds are continuously generated, the server device 2 can identify the interval between the snoring sounds as a silent interval that satisfies a predetermined determination condition. On the other hand, when the snoring sound, breath sound, and snoring sound are generated in this order, the server device 2 does not identify the intervals between the snoring sound and the breath sound and between the breath sound and the snoring sound as silent intervals that satisfy a predetermined determination condition. Similarly, when breath sounds are continuously generated, the server device 2 can identify the interval between the breath sounds as a silent interval that satisfies a predetermined determination condition. On the other hand, when the breath sound, snoring sound, and breath sound are generated in this order, the server device 2 does not identify the intervals between the breath sound and the snoring sound and between the snoring sound and the breath sound as silent intervals that satisfy a predetermined determination condition.

[0101] When a silent period is identified in step S22, a silent period that satisfies a predetermined condition (also referred to as a predetermined first condition) is identified from those silent periods (step S23). The predetermined first condition is, for example, that the length of the silent period is within a predetermined length range. The predetermined length range can be designed as appropriate. Specifically, it may be 8 seconds or more, or 10 seconds or more. Also, the predetermined length range may be within 4 minutes, or within 3 minutes. The predetermined length range may be from 8 seconds to 4 minutes, or from 10 seconds to 3 minutes.

[0102] Next, it is determined whether apnea or hypopnea occurs during the user's sleep (step S24). In step S24, the server device 2 may determine that apnea or hypopnea occurs during the user's sleep in the silent period that satisfies the predetermined first condition identified in step S23. Alternatively, the server device 2 may use a machine-learned prediction model to determine only the occurrence of apnea, only the occurrence of hypopnea, or both the occurrence of apnea and the occurrence of hypopnea in the silent period determined to satisfy the predetermined first condition in step S23 based on the sound information during the user's sleep. The prediction model is machine-learned with sound information during a person's sleep or a spectrogram obtained by converting the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data.

[0103] Here, the information regarding the occurrence of apnea or hypopnea includes, for example, whether either apnea or hypopnea occurs and / or information regarding the type of apnea or hypopnea. Note that the types of apnea or hypopnea include, for example, obstructive and central, but the types of apnea or hypopnea machine-learned as output data are not limited thereto and may be more finely classified.

[0104] In step S24, a process of identifying whether apnea or hypopnea has occurred using a prediction model will be described. For example, in the storage unit 23 of the server device 2, a prediction model (also referred to as a second prediction model) that is machine-learned with sound information during a person's sleep or a spectrogram obtained by converting the sound information as input data and information regarding the occurrence of apnea or hypopnea during the sleep as output data is stored. The machine learning algorithm is not particularly limited, and similar to the first prediction model, a known algorithm can be used. Examples thereof include linear regression, multiple regression analysis, support vector machine, decision tree, random forest, and deep learning using a multi-layer neural network.

[0105] The multi-layer neural network has an input layer, an output layer, and a plurality of intermediate layers. Weights are set for the edges connecting the nodes of each layer. Weights corresponding to each input to the node are set for the edge, and the weights corresponding to each input to the node are multiplied, and the value obtained by multiplying these weights and the bias are added. The value obtained by the addition is non-linearly transformed using an activation function to calculate an activation value. The calculated activation value becomes the value of the input passed to the nodes of the next layer. The number of intermediate layers can be appropriately designed. The weights are optimized by the above-described teacher data.

[0106] The sound information during a person's sleep or the spectrogram obtained by converting the sound information, which is the input data, includes, for example, time information, the frequency of the sound, and / or the loudness of the sound. The spectrogram used as the input data is the same as that described in the first prediction model above and is the result of calculating the frequency spectrum by passing the sound information through a window function. When using a spectrogram as the input data, in step S24, the server device 2 converts the sound information into a spectrogram. Alternatively, if the sound information has been converted into a spectrogram in step S21, the spectrogram converted in step S21 may be used.

[0107] For example, in the second prediction model, when the input data is sound information corresponding to a silent section that satisfies a predetermined first condition specified in step S23 or a spectrogram obtained by converting the sound information, the output data can be information regarding whether apnea or hypopnea has occurred in the silent section. Alternatively, in the second prediction model, when the input data is sound information having a predetermined length or a spectrogram obtained by converting the sound information, the output data may be information regarding in which section of this sound information apnea or hypopnea has occurred.

[0108] The output data used is, for example, data for which the occurrence of apnea or hypopnea has been specified by a portable sleep apnea detection device, a PSG, a doctor's judgment, or the like.

[0109] When using the prediction model in step S24, when the server device 2 receives an input of sound information of a sound file, the control unit 21 of the server device 2 uses the second prediction model to determine whether apnea or hypopnea has occurred in the silent section specified in step S23. In step S24, the data input to the second prediction model corresponds to the input data when the second prediction model was trained. For example, the server device 2 inputs sound information corresponding to a silent section that satisfies a predetermined first condition or a spectrogram obtained by converting the sound information to the second prediction model. Then, the server device 2 can determine whether apnea or hypopnea has occurred in the silent section based on the output data. By using the second prediction model to confirm whether apnea or the like has occurred in the specified silent section, the system 10 of the present embodiment can accurately detect the occurrence of apnea or hypopnea.

[0110] By the process of step S24, it is possible to identify the occurrence status of apnea or hypopnea for the silent interval identified in step S23. For example, the server device 2 can identify that apnea has occurred in the silent interval from 0:10:05 to 0:10:25, and hypopnea has occurred in the silent interval from 0:22:00 to 0:22:12. Note that, for example, the start time of the silent interval may be the end time of the event immediately before the silent interval, and the end time of the silent interval may be the start time of the event immediately after the silent interval. Also, the interval during which apnea or hypopnea has occurred may or may not coincide with the silent interval identified in step S23.

[0111] When it is identified that apnea or hypopnea is occurring during the user's sleep, the type of the apnea or the hypopnea is identified (step S25). When the type of apnea or hypopnea is identified in the process of step S24, in step S25, the server device 2 can utilize the identified information. By steps S21 to S25, the specific process for apnea etc. ends.

[0112] Here, the types of apnea or hypopnea are, for example, obstructive and central. For example, apnea or hypopnea between snores may be identified as obstructive apnea, and apnea or hypopnea between breath sounds may be identified as central apnea.

[0113] The system 10 of the present embodiment can identify the type of apnea or hypopnea according to whether the silent interval identified in step S23 is a silent interval between snores or a silent interval between breath sounds. For example, for a silent interval between snores, if apnea is identified in step S24, in step S25, the apnea is identified as obstructive apnea.

[0114] Further, for example, in the process of step S25, if snoring sounds exist before and after the time when hypopnea occurs, the server device 2 may identify the hypopnea as obstructive hypopnea. Also, if breathing sounds exist before and after the time when hypopnea occurs, the server device 2 may identify the hypopnea as hypopnea resulting from nasal diseases such as central or allergic rhinitis and sinusitis.

[0115] In this embodiment, the server device 2 may regard a silent interval that satisfies the predetermined determination condition in step S22 as a silent interval between the snoring sounds before and after, and regard a silent interval between the breathing sounds before and after as a non-analysis interval where no analysis is performed. Alternatively, the server device 2 may also perform the processes of steps S23 to S25 on the silent interval between the breathing sounds before and after as a silent interval that satisfies the predetermined condition. In this case, in step S24, the apnea or hypopnea identified as occurring during the user's sleep can be identified as central apnea or hypopnea in step S25.

[0116] Here, an outline of apnea and the like identified in step S23 will be described. FIG. 7 is a diagram for explaining the apnea and the like identification process according to an embodiment of the present invention. FIG. 7(A) is a diagram showing the waveform of the electrical signal (sound information) of the sound file from which noise has been removed, and is a diagram showing the waveform of the electrical signal of the sound file on which the process of step S12 has been executed. The horizontal axis in FIG. 7(A) represents the time when the sound information was acquired, and the vertical axis represents the amplitude. FIG. 7(B) is a diagram obtained by converting the electrical signal (sound information) of the sound file into a spectrogram. The horizontal axis in FIG. 7(B) represents the time when the sound information was acquired, the vertical axis represents the frequency, and the shading represents the intensity of the amplitude.

[0117] When step S13 is executed, an event that is a time period (interval) indicated by the solid line in FIG. 7 is specified. In step S21, sound information in the interval corresponding to the event or a spectrogram obtained by converting the sound information is input to the first prediction model. As a result, the first prediction model specifies snoring sounds or breathing sounds for the interval. In FIG. 7, an event corresponding to the snoring sound or breathing sound (specified snoring sound, etc.) specified by the first prediction model is indicated by a square. On the other hand, an event not specified as a snoring sound or breathing sound by the first prediction model is indicated by a circle as a breathing sound unrelated to apnea or the like. When the silent interval specified in step S22 is an interval in which the snoring sound or breathing sound specified by the first prediction model continues, when step S22 is executed, the interval indicated by the dashed line in FIG. 7 is specified as the silent interval between the specified snoring sounds, etc. In step S23, it is determined whether or not the silent interval indicated by the dashed line in FIG. 7 satisfies a predetermined first condition. When the predetermined first condition is satisfied, in step S24, the server device 2 can specify that hypopnea or apnea has occurred in the silent interval. In FIG. 7, the interval specified as having hypopnea or apnea occurring is indicated by a triangle.

[0118] In addition, when it is specified whether a time period (event) satisfying a predetermined second condition in step S21 is a snoring sound or a breathing sound, the silent interval specified in step S22 can be an interval while the snoring sound continues or an interval while the breathing sound continues. FIG. 8 corresponds to the apnea or the like specifying process when it is specified whether an event is a snoring sound or a breathing sound in step S21. FIG. 8 is a diagram for explaining the apnea or the like specifying process according to an embodiment of the present invention. FIG. 8(A) is a diagram showing the waveform of an electrical signal (sound information) of a sound file from which noise has been removed, and is a diagram showing the waveform of the electrical signal of the sound file for which the process of step S12 has been executed. In addition, the horizontal axis of FIGS. 8(A) to 8(D) represents the time when the sound information was acquired, and the vertical axis represents the amplitude.

[0119] FIG. 8(B) shows the time period (interval) in which apnea or hypopnea is identified from the sound information included in the sound file shown in FIG. 8(A). FIG. 8(C) shows the time period (interval) in which breath sounds are identified from the sound information included in the sound file shown in FIG. 8(A). FIG. 8(D) shows the time period (interval) in which snoring sounds are identified from the sound information included in the sound file shown in FIG. 8(A).

[0120] As shown in FIG. 8(B), when the silent interval between the snoring sounds before and after shown in FIG. 8(D) is within a range of a predetermined length, it is identified as a silent interval satisfying a predetermined first condition (step S23). When the predetermined first condition is satisfied, the server device 2 can identify that apnea or hypopnea has occurred during that silent interval. The server device 2 may use a second prediction model to identify that apnea or hypopnea has occurred during that silent interval (step S24). Note that FIG. 8(E) shows a silent interval in which no snoring sound is detected between breath sounds in the sound information of the sound file shown in FIG. 8(A). This silent interval may be a non-analyzed interval.

[0121] Note that the process of step S21 may be executed as follows. First, at the server device 2, for the event identified in step S13, an event that satisfies a predetermined third condition for identifying a snoring sound or a predetermined fourth condition for identifying a breath sound is identified. The predetermined third condition and the predetermined fourth condition can be appropriately set with known conditions for identifying a snoring sound or a breath sound. The predetermined third condition is, for example, that the volume of the sound information included in the event is equal to or greater than a predetermined volume or exceeds the predetermined volume. The predetermined fourth condition is, for example, that the volume of the sound information included in the event is equal to or less than a predetermined volume or is below the predetermined volume. The server device 2 can identify that an event satisfying the predetermined third condition is a snoring sound and that an event satisfying the predetermined fourth condition is a breath sound. Thereby, the server device 2 can identify that a snoring sound or a breath sound has occurred for the event identified in step S13.

[0122] Note that the process of step S25 may be omitted. Also, in steps S24 and S25, only the occurrence and type of apnea may be specified, or only the occurrence and type of hypopnea may be specified.

[0123] Next, another example of the apnea and the like identification process in step S14 will be described. FIG. 9 is a diagram showing a flowchart of another example of the apnea and the like identification process according to an embodiment of the present invention. The apnea and the like identification process is executed by the server device 2.

[0124] First, it is determined whether apnea or hypopnea occurs during the user's sleep (step S31). In step S31, using the sound information during a person's sleep or the spectrogram obtained by converting the sound information as input data, and using the second prediction model that has been machine-learned with information regarding the occurrence of apnea or hypopnea during the sleep as output data, based on the sound information during the user's sleep, it is possible to identify the occurrence of apnea or hypopnea during the user's sleep.

[0125] When it is determined that apnea or hypopnea occurs during the user's sleep, the type of the hypopnea is identified (step S32). Step S32 is the same process as step S25. By steps S31 and S32, the apnea and the like identification process ends.

[0126] Here, the second prediction model will be described. When apnea occurs, the snoring sound or breathing sound before and after the silent period where apnea occurs has characteristics regarding the frequency and volume of the sounds included in the snoring sound and breathing sound. Also, when hypopnea occurs, the snoring sound or breathing sound in the period where hypopnea occurs has characteristics regarding the frequency and volume of the sounds included in the snoring sound and breathing sound.

[0127] For example, the spectrograms of the snoring sounds or breathing sounds in the vicinity where apnea or the like has occurred may have similar patterns. As described above, the spectrogram is image information with the vertical axis representing frequency and the horizontal axis representing time. Also, the frequency components of the sounds included in the snoring sounds or breathing sounds in the vicinity where apnea or the like has occurred may include frequencies higher than the frequencies of the sounds included in other snoring sounds or breathing sounds. Further, for example, the magnitude of the sounds included in the snoring sounds or breathing sounds in the vicinity where apnea or the like has occurred may be greater than the magnitude of the sounds included in other snoring sounds or breathing sounds. The waveforms of the electrical signals indicating the snoring sounds or breathing sounds in the vicinity where apnea or the like has occurred may have similar shapes. Therefore, the second prediction model can be constructed by machine learning.

[0128] Regarding the second prediction model described above, the input data may be sound information having a predetermined length or a spectrogram obtained by converting the sound information, and the output data may be information regarding in which section of this sound information apnea or hypopnea has occurred. The portion of the sound information having a predetermined length or the spectrogram obtained by converting the sound information used for the input data can be designed as appropriate, but preferably includes the time corresponding to the time zone in which at least one apnea or hypopnea has occurred. When using a spectrogram for the input data, in step S31, the server device 2 converts the sound information into a spectrogram.

[0129] Note that the second prediction model may be a plurality of prediction models. For example, a prediction model that is machine-learned with sound information during a person's sleep as input data and information regarding the occurrence of apnea during the sleep as output data, and a prediction model that is machine-learned with sound information during a person's sleep as input data and information regarding the occurrence of hypopnea during the sleep as output data may be constructed.

[0130] Note that the server device 2 can calculate the analysis result based on the information regarding the generation of the snoring sound or breathing sound specified in step S21, the information regarding the silent section where apnea or hypopnea occurs specified in steps S22 and S23, and / or the information regarding the occurrence or type of apnea or hypopnea specified in steps S24, S25, S31, and S32. The information regarding the silent section includes, for example, the time information when the silent section occurred.

[0131] [Display information] Next, the display information displayed on the user terminal 1 in step S19 will be described. FIG. 10 is an example of the user terminal 1 on which the display information is displayed in step S19. On the display screen 100, the analysis result 101 on October 3, 2023 and the analysis information 110 (110a to 110d) are displayed. Also, an explanation 111 and a plurality of icons 120 (120a, 120b) may be displayed on the display screen 100.

[0132] In the analysis result 101, an area 102 corresponding to the user's awake state and an area 103 corresponding to the user's sleep state are displayed. Also, in the analysis result 101, an area 104 corresponding to the time when apnea or hypopnea occurs is displayed within the area 103 corresponding to the user's sleep state.

[0133] The area 102 and the area 103 are displayed in a circular shape centered on the upper center of the display screen 100. Also, the area 102 and the area 103 have a width in the radial direction of the circle. The thickness of the width of the area 102 and the area 103 may be constant or different between the area 102 and the area 103. Also, the area 102 or the area 103 may be displayed in an arc shape or a band shape.

[0134] Inside the circumferentially displayed areas 102 and 103, a scale 105 is displayed. The scale 105 indicates time. Inside the circumference, a scale of "12" is provided at the uppermost position with respect to the display screen 100, and the scale of "12" indicates 0:00 am. The scale of "3" provided at the right end inside the circumference indicates 3:00 am. For the scale 105, thick scales are provided every predetermined time (1 hour), and thin scales are provided every predetermined time (12 minutes). The time at which the scale is provided, the thickness of the line, and / or the length of the line are set as appropriate.

[0135] In FIG. 10, the area 102 corresponding to the user's waking time and the area 103 corresponding to the user's sleeping time together correspond to a time period of 12 hours. The scale 105 corresponds to an actual clock that can indicate 12 hours. Also, the areas 102 and 103 are displayed with the time of the user's sleeping time corresponding to the scale 105. That is, in FIG. 10, the user can recognize that sleep started after 10:36 pm and ended after 7:00 am the next day. Note that the time when sleep started corresponds to the time when the acquisition of the sound information in step S1 was started, and the time when sleep ended corresponds to the time when the acquisition of the sound information was ended.

[0136] Note that the time displayed in the analysis result 101 may be 12 hours, 24 hours, or only the time corresponding to the area 103 corresponding to the user's sleeping time. Note that the scale 105 may not be displayed in the analysis result 101. Also, the area 102 may not be displayed in the analysis result 101.

[0137] The areas 102 and 103 are displayed continuously. In FIG. 10, the area 102 is displayed in light gray, the area 103 is displayed in black, and the area 104 is displayed in white. That is, the areas 102, 103, and 104 are each displayed in a different display mode. The different display modes are not particularly limited, and examples include different colors, different patterns, different blinking patterns, etc. Note that the areas 102 and 104 may be displayed in the same display mode.

[0138] The area 104 corresponding to the occurrence of apnea or the like is displayed as a slit (gap) within the area 103 corresponding to the user's sleep time. Also, the area 104 is displayed with the time and moment when apnea or the like occurred corresponding to the scale 105. For example, the area 104 displays the area corresponding to the period from 3:35 to 3:36 in a color different from that of the area 103. The area 104 indicates that apnea or hypopnea occurred between 3:35 and 3:36.

[0139] The area 104 is not particularly limited as long as it has a color different from that of the area 103. For example, it may be represented by the same color as the background color of the display screen 100. Also, it is preferable that the area 103 and the area 104 are displayed in a manner that clearly distinguishes them. For example, the area 103 and the area 104 may be displayed in colors with different saturations such as white and black, or in colors with a strong contrast relationship such as blue and yellow, which are opposite colors.

[0140] Also, the area 104 may be displayed in different display modes according to the length of time that apnea and / or hypopnea continued, and / or the number of times of apnea and / or hypopnea per unit time or the total or average length of the time that apnea and / or hypopnea continued per unit time.

[0141] That the area 104 is displayed in different display modes according to the length of time that apnea or the like continued means, for example, that when the time that apnea or the like continued is equal to or more than a predetermined time (for example, 1 minute), the width of the slit is widened, and when the time that apnea or the like continued is less than the predetermined time, the width of the slit is narrowed or the slit is not displayed.

[0142] The fact that the area 104 is displayed in different display modes according to the number of occurrences of apnea or the like per unit time means, for example, that in a predetermined unit time (e.g., 10 minutes), if the number of occurrences of apnea or the like is more than a predetermined number (e.g., 2 times), a slit is provided in the predetermined unit time, and if the number of occurrences of apnea or the like in the predetermined unit time is less than or equal to the predetermined number, it may be possible not to provide a slit in the predetermined unit time. Alternatively, the server device 2 may store the number of occurrences of apnea or the like in a predetermined unit time and the width of the slit in association with each other in a table, and may determine the width of the slit by referring to the table.

[0143] The fact that the area 104 is displayed in different display modes according to the total or average length of the continuous time of apnea or the like per unit time means, for example, that in a predetermined unit time (e.g., 10 minutes), if the total continuous time of apnea or the like is equal to or more than a predetermined time (e.g., 3 minutes), a slit is provided in the predetermined unit time, and if the total continuous time of apnea or the like is less than the predetermined time, it may be possible not to provide a slit in the predetermined unit time.

[0144] Also, in a predetermined unit time, if the average of the continuous time of apnea or the like is equal to or more than a predetermined time (e.g., 1 minute), a slit is provided in the predetermined unit time, and if the total continuous time of apnea or the like is less than the predetermined time, the server device 2 may not provide a slit in the predetermined unit time. Alternatively, the server device 2 may store the total or average length of the continuous time of apnea or the like in a predetermined unit time and the width of the slit in association with each other in a table, and may determine the width of the slit by referring to the table.

[0145] Note that the predetermined unit time can be set as appropriate. The predetermined unit time may be, for example, the shortest time in which a slit can be provided as the unit time, or a time corresponding to the time provided with a scale as the unit time.

[0146] Note that different display modes are not limited to only changes in the width of the slit. Different display modes include, for example, different colors, different patterns, different blinking patterns, etc. When combining the above different display modes, it is preferable that different display modes are adopted for each respective index. For example, when varying the display mode of region 104 according to the number of times per unit time of apnea or the like and the total length of the continuous time per unit time of apnea or the like, the number of times per unit time of apnea or the like can be changed as the width of the slit, and the total length of the continuous time per unit time of apnea or the like can be changed as the color, so that the server device 2 may vary the display mode of region 104.

[0147] For example, region 104 may be displayed with slits for the region corresponding to from 2:13 to 2:23, or may be displayed with slits for the region corresponding to from 4:02 to 4:25. By setting the width of the slit for region 104 to be different from the slit corresponding to the time when apnea or the like occurred, the visibility of the analysis result 101 can be enhanced.

[0148] Explanation 111 displays information explaining what the regions 103 and 104 of the analysis result 101 indicate. In explanation 111a, the same color as region 103 is displayed, and in explanation 111b, the same color as region 104 is displayed. Also, explanation 111b is displayed as a slit within the region of explanation 111a. Therefore, the user can understand that explanation 111a indicates region 103 and explanation 111b indicates region 104.

[0149] Explanation 111c indicates that the region of explanation 111b is "the time when it is considered that breathing is not occurring". Therefore, the user can recognize that region 104 of the analysis result 101 indicates the time when it is considered that breathing is not occurring. Note that the system 10 of the present embodiment may represent "the occurrence of apnea or hypopnea" as "the time when it is considered that breathing is not occurring".

[0150] The analysis information 110 indicates information regarding the user's snoring during sleep and / or information regarding apnea or hypopnea identified in the analysis process. The system 10 of the present embodiment may display, for example, as the analysis information 110, the total, average, or longest length of the time during which apnea and / or hypopnea continued, the total number of occurrences of apnea and / or hypopnea, and / or information indicating the number of occurrences per unit time of apnea and / or hypopnea or the total or average length of the time during which apnea and / or hypopnea continued per unit time.

[0151] The analysis information 110a indicates the total number of occurrences of apnea or hypopnea during one sleep of the user. The analysis information 110b indicates the sleep time during one sleep of the user. The sleep time is the time from when the user terminal 1 received the input for starting the acquisition of sound information to when the user terminal 1 received the input for ending the acquisition of sound information in the transmission process.

[0152] The analysis information 110c indicates the average number of occurrences per unit time during which the user is considered not to be breathing during one sleep. The analysis information 110d indicates the longest number of seconds during which the user is considered not to be breathing during one sleep.

[0153] The explanation 106 indicates an explanation for the user. The explanation 106 may change its content according to the analysis result. For example, as a result of the analysis, when the occurrence status of apnea or the like (for example, the number of occurrences or the duration of apnea or the like) is within a predetermined range, the explanation 106 of "There is no problem with the sleep state." may be displayed, and when the occurrence status of apnea or the like is outside the predetermined range, the explanation 106 of "It is recommended that you consult a medical institution." may be displayed.

[0154] Icon 120a is an icon for displaying other display screens. When the user selects icon 120a, the display screen 100 switches. The switching display screen can be arbitrarily set. For example, the analysis results of the user's previous sleep and the date and time may be displayed in a list. When the user selects the displayed analysis results or date and time, on the user terminal 1, a display screen containing information similar to that of display screen 100 for the selected analysis results or date and time is displayed.

[0155] Icon 120b is an icon for displaying the details of the analysis results. When the user selects icon 120b, the display screen 100 switches. The switching display screen can be arbitrarily set. For example, the details of the analysis result 101 of the display screen 100 are displayed. In the switched display screen, the acquired or noise-removed sound information may be made playable as sound, and the analysis result 101 in which the area 104 is completely corresponding to the time when apnea or the like occurred and the time may be displayed.

[0156] Note that the area 103 corresponding to the user's sleep may be displayed in a graph shape for indicating the depth of the user's sleep. In this graph, the depth of the user's sleep can be displayed on the vertical axis and the time on the horizontal axis.

[0157] Note that in the above embodiment, the mode in which the sound file is transmitted to the server device 2 and the server device 2 executes the process of analyzing the sound file has been described. However, on the user terminal 1, the process of analyzing the sound file may be executed. In this case, the process of transmitting the sound file in step S4 and the process of transmitting the display information in step S17 can be omitted. Also, the process of generating the sound file in step S3 may be omitted.

[0158] In the user terminal 1, when the processes of steps S11 to S16 are executed, the necessary programs may be stored in the user terminal 1 in advance, or may be transmitted from the server device 2. The system 10 may include an analysis process that acquires, as sound information, the sound during the user's sleep in the user terminal 1 and analyzes the sound information.

[0159] In addition, in the above embodiment, the mode in which the sound file is transmitted to the server device 2 and the server device 2 executes the analysis process of the sound file has been described. However, a third party such as a medical staff may analyze the sound file. In this case, access is made to the system 10 via the server device 2 from a third-party terminal operated by the third party. Then, the sound file for which the noise removal process in step S12 has been executed is transmitted from the server device 2 to the third-party terminal. When the analysis of the sound information of the sound file by the third party is completed, the analysis result may be transmitted from the third-party terminal to the user terminal 1 via the server device 2 by the operation input of the third party.

[0160] In addition, in the above embodiment, the mode in which the processes of acquiring and transmitting the sound information in steps S1 to S4 and the process of displaying the display information in steps S18 and S19 are executed on the same user terminal 1 has been described. However, the processes of steps S1 to S4 and steps S18 and S19 may be executed on different user terminals 1. In this case, the user terminal 1 that executes steps S1 to S4 only needs to be equipped with an acquisition device. Also, the user terminal 1 that executes steps S18 and S19 only needs to be equipped with a display device.

[0161] Also, the process of displaying the display information in steps S18 and S19 may be executed at an arbitrary timing of the user. For example, when the user logs in to the system 10, the display information may be transmitted from the server device 2 and the display information may be displayed on the user terminal 1.

[0162] Note that, in the analysis process, the system 10 of the present embodiment may evaluate whether the user has a predetermined disease such as sleep apnea syndrome or the possibility of having the disease. For example, when the total number of occurrences and the duration during sleep of apnea or hypopnea stored exceed a predetermined threshold, the server device 2 can determine that the user has or may have a predetermined disease.

[0163] Note that, in the above, the case of mainly acquiring sound information in the transmission process has been described, but the information acquired in step S1 is not particularly limited. The information acquired in step S1 may be, for example, video information. The video information acquired in step S1 may be information about the user and may include a predetermined part of the user (for example, face, neck, arm, leg, body, etc.). When video information is acquired, the above transmission process can be applied with video information instead of sound information and video files instead of sound files. For example, in step S3, a video file is generated. A video file is a collection of one piece of data that can manage and store the acquired video information. The file format of the video file is MP4, AVI, MOV, MPG, etc.

[0164] Note that the information acquired and transmitted in the transmission process is analyzed by the server device 2 to identify information about the user. In the above, it has been used to identify the occurrence of apnea or hypopnea during the user's sleep in the analysis process, but the content to be analyzed is not limited to this. The information acquired and transmitted in the transmission process can be appropriately changed according to the content to be analyzed. For example, when a video file is transmitted, the server device 2 may execute an analysis for evaluating the user's movement. Also, for example, when a sound file including the user's voice is transmitted, the server device 2 may execute an analysis for evaluating the user's voice.

[0165] According to the present invention, using, as input data, sound information including snoring sounds or breathing sounds during a person's sleep or a spectrogram obtained by converting the sound information, and a prediction model that has been machine-learned with information related to the occurrence of snoring sounds or breathing sounds during the sleep as output data, based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information, in order to identify the occurrence of snoring sounds or breathing sounds during the user's sleep, it is possible to suitably identify the snoring sounds or breathing sounds during the user's sleep.

[0166] According to the present invention, a system includes: a first specifying means for specifying the occurrence of snoring sounds or breathing sounds during a user's sleep, using, as input data, sound information including snoring sounds or breathing sounds during a person's sleep or a spectrogram obtained by converting the sound information, and a prediction model that has been machine-learned with information related to the occurrence of snoring sounds or breathing sounds during the sleep as output data, based on the sound information during the user's sleep or the spectrogram obtained by converting the sound information; and a second specifying means for specifying whether or not a silent interval between snoring sounds before and after, a silent interval between a snoring sound and a breathing sound before and after, and / or a silent interval between breathing sounds before and after satisfies a predetermined first condition. Therefore, it is possible to specify a silent interval that satisfies the predetermined first condition. The system can specify the occurrence of apnea or hypopnea using the specified result. Further, even when the accuracy of directly specifying the occurrence of a silent interval, apnea, or hypopnea using a machine-learned prediction model (so-called AI) is not sufficient, as described above, by specifying snoring sounds and breathing sounds, it is possible to accurately specify the occurrence of a silent interval, apnea, or hypopnea.

[0167] According to the present invention, a system includes a removing means for removing noise from sound information during the user's sleep, and the first specifying means can analyze sound information containing noise because it specifies the occurrence of breathing sounds or snoring sounds during the user's sleep based on the sound information from which noise has been removed or a spectrogram obtained by converting the sound information from which noise has been removed. Since the user terminal can acquire sound information, for example, beside the user's bedding, it can acquire sound information during the user's daily sleep.

[0168] In addition, the first specifying means can reduce the load of specifying the occurrence of snoring or breathing sounds by specifying the occurrence of snoring or breathing sounds based on the sound information in a time period that satisfies a predetermined second condition or the spectrogram obtained by converting the sound information, as compared with the case of specifying from the entire sound information or the spectrogram obtained by converting the sound information. Further, the system includes a third specifying means for specifying that apnea or hypopnea has occurred in a silent section that satisfies a predetermined condition, and based on the information regarding the silent section in which the specified apnea or hypopnea has occurred, calculates the total, average, or longest length of the time during which apnea and / or hypopnea has continued, the total number of times of apnea and / or hypopnea, the number of times of apnea and / or hypopnea per unit time, and / or the total or average length of the time during which apnea and / or hypopnea has continued per unit time. Therefore, information regarding apnea or hypopnea can be calculated. Further, since the snoring or breathing sound during the sleep of the person occurs immediately before or immediately after the occurrence of apnea or hypopnea, the occurrence of the snoring or breathing sound related to apnea or hypopnea can be specified.

[0169] According to the present invention, since the system includes a specifying means for specifying the occurrence of apnea or hypopnea during the sleep of the user based on the sound information during the sleep of the user or the spectrogram obtained by converting the sound information, using a prediction model that is machine-learned with the sound information during the sleep of the user or the spectrogram obtained by converting the sound information as input data and the information regarding the occurrence of apnea or hypopnea during the sleep as output data, the occurrence of breathing or hypopnea can be specified.

[0170] According to the present invention, since the area corresponding to the time of the user's sleep is displayed with an area corresponding to the occurrence of apnea or hypopnea, the user can visually recognize the time when apnea or hypopnea occurs during the user's sleep. Further, according to the present invention, since the area corresponding to the time of the user's sleep is displayed in an arc shape or a band shape, the user can intuitively recognize the time when apnea or hypopnea occurs during the user's sleep.

[0171] According to the present invention, in order to display the area corresponding to the time when apnea and / or hypopnea occurs in different display modes according to the length of time during which apnea and / or hypopnea continues, and / or the number of times per unit time of apnea and / or hypopnea or the total or average length of the continuous time per unit time, the visibility of the analysis result regarding the user's breathing state during sleep can be enhanced.

[0172] According to the present invention, further, in order to display information indicating the total, average, or longest length of the time during which apnea and / or hypopnea continues, the total number of times of apnea and / or hypopnea, and / or the number of times per unit time of apnea and / or hypopnea or the total or average length of the continuous time per unit time, the user can confirm the analysis result regarding the occurrence of apnea or the like during the user's sleep.

[0173] According to the present invention, based on the sound information during one sleep of the user, a plurality of sound files having different times when the sound information is acquired are generated, and the generated sound files are transmitted to a second computer device. Therefore, compared with the case of transmitting the sound information during one sleep of the user as one sound file, the load when transmitting the sound information during one sleep of the user to a second computer device (server device) can be reduced. Further, in the second computer device, the sound information of the sound file can be used to specify whether apnea or hypopnea occurs during the user's sleep.

[0174] According to the present invention, in order to transmit the generated sound file in association with information capable of specifying the order in which the sound information is acquired, the sound information during one sleep of the user can be specified in the order of the time series in which it is acquired. Further, according to the present invention, when a predetermined communication line is detected, when the time of the generated sound file is less than a predetermined time, or when the data amount of the sound file is less than a predetermined amount, the generated sound file is transmitted, so that it is possible to prevent the data capacity borne by the user from becoming excessive when transmitting the sound file.

[0175] Further, according to the present invention, when there is an audio file that has not been transmitted to the second computer device, in response to detecting a predetermined communication line, notification information indicating the existence of the untransmitted audio file is notified, so that the user can be informed that there is an untransmitted audio file. Thereby, it is possible to prevent the state where the audio file remains untransmitted. Further, according to the present invention, in order to generate a plurality of audio files according to the time of the audio file or the data amount of the audio file, it is possible to prevent transmission failure due to the data amount of the audio file, and the audio information during a user's one sleep can be transmitted to the second computer device.

[0176] According to the present invention, in order to transmit the generated audio file in association with information capable of specifying the order in which the audio information was acquired, the audio information during a user's one sleep can be specified in chronological order. Further, according to the present invention, in parallel with the generation and / or transmission of one audio file, in order to acquire audio information for generating the audio file generated after the one audio file, the time for executing the generation and transmission of the audio file can be shortened.

[0177] According to the present invention, in order to generate audio files of different times or data amounts according to the computer device that executes the generation of the audio file, the load on the computer device is small when generating and transmitting the audio file.

Explanation of Signs

[0178] 1 User terminal 2 Server device 3 Communication network 10 System 11 Control unit 12 RAM 13 Storage unit 14 Acquisition unit 15 Input unit 16 Display unit 17 Communication interface 21 Control unit 22 RAM 23 Storage unit 24 Communication interface 100 Display screen 101 Analysis result 102 Region corresponding to the user's waking time 103 Region corresponding to the user's sleeping time Region corresponding to the occurrence of apnea or hypopnea 105 Scale 106 Explanation 110 Analysis information 111 Explanation 120 Icon

Claims

1. A computer device, An acquisition means for acquiring sound information of a user while sleeping; A generating means for generating a plurality of sound files based on the acquired sound information; a transmitting means for transmitting the generated sound file to a second computer device in association with information capable of identifying the order in which the sound information was acquired; Function as a A program in which a plurality of generated sound files each contain sound information captured at a different time.

2. The program according to claim 1, wherein the transmitting means transmits the generated sound file when a specified communication line is detected, when the duration of the generated sound file is less than a specified time, or when the data amount of the sound file is less than a specified amount.

3. a notification means for notifying the second computer device of notification information indicating the existence of an unsent sound file in response to detection of a predetermined communication line when the second computer device has an unsent sound file; The program according to claim 1 or 2,

4. 3. The program according to claim 1, wherein the generating means generates a plurality of sound files according to a duration of the sound file or a data amount of the sound file.

5. 3. The program of claim 1, wherein the sound information is used to identify occurrences of apnea or hypopnea during the user's sleep.

6. The program according to claim 1 or 2, wherein the acquisition means acquires sound information for generating sound files to be generated after the one sound file in parallel with the generation of the one sound file by the generation means and / or the transmission of the one sound file by the transmission means.

7. 3. The program according to claim 1, wherein the generating means generates sound files of different durations or amounts of data depending on a computer device that executes the generating means.

8. An acquisition means for acquiring sound information of a user while sleeping; A generating means for generating a plurality of sound files based on the acquired sound information; a transmitting means for transmitting the generated sound file to a second computer device in association with information capable of identifying the order in which the sound information was acquired; Equipped with A computer device in which a plurality of generated sound files each contain sound information captured at a different time.

9. A method executed on at least one computing device, comprising: An acquisition step of acquiring sound information of a user while sleeping; A generation step of generating a plurality of sound files based on the acquired sound information; a transmitting step of associating the order in which the sound information was acquired with identifiable information and transmitting the generated sound file to a second computer device; having A method in which a plurality of generated sound files each contain sound information captured at a different time.

10. A system comprising a first computing device and a second computing device, a first computing device, An acquisition means for acquiring sound information of a user while sleeping; A generating means for generating a plurality of sound files based on the acquired sound information; a transmitting means for transmitting the generated sound file to a second computer device in association with information capable of identifying the order in which the sound information was acquired; Equipped with Each of the generated sound files includes sound information acquired at a different time, a second computing device, an order specifying means for specifying the order of the sound files based on information capable of specifying the order in which the sound information was acquired; A system comprising:

11. A display control means for controlling display of analysis results of one sleep period of the user in chronological order based on the order of the sound files specified by the order specifying means. The system of claim 10 comprising:

12. 1. A method implemented in a system including a first computing device and a second computing device, comprising: a first computing device, An acquisition step of acquiring sound information of a user while sleeping; A generation step of generating a plurality of sound files based on the acquired sound information; a transmitting step of associating the order in which the sound information was acquired with identifiable information and transmitting the generated sound file to a second computer device; Equipped with Each of the generated sound files includes sound information acquired at a different time, a second computing device, an order specifying step of specifying the order of the sound files based on information capable of specifying the order of acquiring the sound information; The method comprising:

Images