Headphone-based lung function detection method, system, device and readable storage medium

Abstract

The present invention discloses a lung function detection method, system, device and computer-readable storage medium based on earphones. The present invention collects the blowing and inhaling sounds of the user through the microphone on the earphones worn by the user, so that the user does not need to prepare expensive detection equipment or go to a hospital for detection, which increases the convenience of detection implementation and reduces the cost required for the detection equipment; by performing signal processing such as spectrum conversion and signal feature analysis on the collected airflow sound, and then using a deep learning algorithm, the spectrum of the blowing and inhalation stages is converted into a final flow rate-volume curve, thereby achieving the goal of obtaining a flow rate-volume curve by performing lung function detection based on ordinary commercial earphones while maintaining low cost, greatly improving the lung function characterization capability of this solution.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to an earphone-based lung function detection method, system, device, and computer-readable storage medium. Background Art

[0002] Chronic lung function diseases (such as asthma, chronic obstructive pulmonary disease) are a type of disease that endangers human health. This type of disease cannot be completely cured at present, and existing medical intervention methods can only be used to control the development of the disease. Therefore, in order to timely detect chronic lung function diseases and timely monitor the health of patients who are already ill, medical literature recommends that patients undergo daily lung function tests. However, there are two types of existing lung function testers. The first type is traditional medical lung function test equipment, which is large and complicated to use, and is only equipped in hospitals. Therefore, it is not suitable for patients to use for daily lung function tests. The second type is home lung function test equipment, which is small and light, suitable for patients to perform lung function tests at home. However, ordinary home lung function meters are very expensive, which is a heavy economic burden for ordinary families.

[0003] At present, the academic community has proposed a series of software algorithms based on smartphones, which can be used to detect lung function. The general principle is: when the user is doing a lung function test, he needs to inhale and blow hard. The algorithm uses the microphone on the smartphone to record the airflow sound generated when the user blows and inhales hard, and analyzes these airflow sounds to obtain the user's lung function status. Although these designs significantly reduce the cost of use, their functions are greatly reduced. The above smartphone-based algorithms can only obtain a small number of lung function indicators, but cannot obtain the flow rate-volume curve that can be obtained by traditional medical spirometers and home spirometers (this curve describes the relationship between the change in gas flow rate and volume when the lungs exchange gases with the outside world when the user blows and inhales. Different lung diseases have their own specific flow rate-volume curve shapes), so the ability to characterize the user's lung function status is very limited. Summary of the invention

[0004] The main purpose of the present invention is to propose a headphone-based lung function detection method, system, device and computer-readable storage medium, aiming to solve the technical problem of how to achieve the goal of enabling a lung function detection algorithm based on a smartphone to obtain a flow rate-volume curve while maintaining low cost.

[0005] To achieve the above object, the present invention provides a lung function detection method based on earphones, the method is applied to a terminal device connected to an earphone, the earphone is equipped with a microphone, and the method comprises:

[0006] Acquire the airflow sound collected by the microphone, and generate a target frequency spectrum according to the airflow sound, wherein the airflow sound is generated by a target user wearing the headset blowing and inhaling;

[0007] Analyze the signal characteristics of the target spectrum graph, and divide the spectrum of the blowing phase and the spectrum of the inhalation phase in the target spectrum graph based on the signal characteristics;

[0008] Using a preset deep learning algorithm, the blowing phase spectrum and the inhalation phase spectrum are combined and converted into a flow rate-volume curve that characterizes the lung function of the target user.

[0009] Optionally, the step of acquiring the airflow sound collected by the microphone and generating a target spectrum diagram according to the airflow sound includes:

[0010] Performing a short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram;

[0011] The airflow sound spectrum graph is passed through a preset Mel filter bank to obtain a spectrum graph under the Mel scale as the target spectrum graph.

[0012] Optionally, the step of performing short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram comprises:

[0013] Performing a short-time Fourier transform on the airflow sound to obtain an initial spectrum diagram;

[0014] A portion of the initial spectrum graph whose frequency is within a preset frequency range is retained as the airflow sound spectrum graph.

[0015] Optionally, the step of using a preset deep learning algorithm to combine the blowing phase spectrum and the inhalation phase spectrum into a flow rate-volume curve characterizing the lung function of the target user includes:

[0016] Inputting the blowing phase spectrum and the inhalation phase spectrum into a preset convolutional neural network to obtain target convolution data;

[0017] Input the target convolution data into a preset recurrent neural network and a fully connected layer to obtain target output data;

[0018] According to the target output data, the gas flow rate signal corresponding to the frequency spectrum of the blowing phase and the matching of the gas flow rate signal of the inhalation phase is obtained, and the gas flow rate signal is integrated to obtain a gas volume signal;

[0019] The flow rate-volume curve is obtained by combining the gas flow rate signal and the gas volume signal.

[0020] Optionally, the step of analyzing the signal features in the target spectrum graph and dividing the spectrum in the target spectrum graph into the blowing phase spectrum and the inhalation phase spectrum based on the signal features comprises:

[0021] Identify the energy peak point in the target spectrum graph, and determine the blowing start time point on the horizontal axis of the target spectrum graph based on the energy peak point;

[0022] According to the preset standard duration of blowing and the blowing start time point, the blowing end time point on the horizontal axis is determined, so that the partial spectrum between the blowing start time point and the blowing end time point in the target spectrum graph is used as the blowing stage spectrum;

[0023] The target spectrum is obtained by removing the spectrum of the blowing stage from the target spectrum as a residual spectrum, and the residual spectrum is normalized to obtain a normalized residual spectrum.

[0024] A preset clustering algorithm is used to determine the portion of the normalized residual spectrum that belongs to the inhalation phase as the inhalation phase spectrum.

[0025] Optionally, the step of using a preset clustering algorithm to determine the portion of the normalized residual spectrum belonging to the inhalation phase as the inhalation phase spectrum includes:

[0026] Clustering the frequency distribution at all moments in the normalized residual spectrum into several types of signals using the clustering algorithm;

[0027] Calculate the total energy corresponding to each type of signal and determine the type of signal with the highest total energy;

[0028] The time segment corresponding to the signal of the highest total energy is used as the inhalation time segment, so as to filter out the spectrum in the inhalation time segment from the remaining spectrum diagram as the inhalation stage spectrum.

[0029] Optionally, after the step of using a preset deep learning algorithm to combine the insufflation phase spectrum and the insufflation phase spectrum into a flow rate-volume curve characterizing the lung function of the target user, the method further includes:

[0030] Analyzing the flow rate-volume curve to evaluate the breathing action quality of the target user;

[0031] When the breathing action quality does not meet the preset quality standard, a prompt message indicating that the action quality does not meet the standard is generated and output.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a lung function detection system based on earphones, wherein the system is provided in a terminal device connected with earphones, wherein the earphones are equipped with a microphone, and the system comprises:

[0033] a target spectrum generation module, used to obtain the airflow sound collected by the microphone and generate a target spectrum graph according to the airflow sound, wherein the airflow sound is generated by the target user wearing the headset blowing and inhaling;

[0034] A phase spectrum division module, used for analyzing the signal characteristics in the target spectrum graph, and dividing the spectrum in the blowing phase and the spectrum in the inhalation phase in the target spectrum graph based on the signal characteristics;

[0035] A target curve conversion module is used to use a preset deep learning algorithm to convert the insufflation phase spectrum and the insufflation phase spectrum into a flow rate-volume curve representing the lung function of the target user.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides an earphone-based pulmonary function detection device, which includes: a memory, a processor, and an earphone-based pulmonary function detection program stored in the memory and executable on the processor, wherein the earphone-based pulmonary function detection program, when executed by the processor, implements the steps of the earphone-based pulmonary function detection method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a headphone-based pulmonary function detection program is stored. When the headphone-based pulmonary function detection program is executed by a processor, the steps of the headphone-based pulmonary function detection method as described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer program product, including a computer program, which implements the steps of the above-mentioned earphone-based lung function detection method when executed by a processor.

[0039] The present invention collects the user's blowing and inhaling sounds through the microphone on the earphones worn by the user, so that the user does not need to prepare expensive detection equipment or go to the hospital for detection, which increases the convenience of detection implementation and reduces the cost of detection equipment; by performing signal processing such as spectrum conversion and signal feature analysis on the collected airflow sound, and then using a deep learning algorithm, the spectrum of the blowing and inhaling stages is converted into a final flow rate-volume curve, thereby achieving the goal of obtaining a flow rate-volume curve by performing lung function testing based on ordinary commercial earphones while maintaining low costs, greatly improving the lung function characterization capability of this solution, and solving the technical problem of how to achieve the goal of obtaining a flow rate-volume curve by a lung function detection algorithm based on a smartphone while maintaining low costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0041] Figure 2 It is a flowchart of a first embodiment of a lung function detection method based on earphones of the present invention;

[0042] Figure 3 This is a schematic diagram of the hardware composition of a specific embodiment of the first embodiment of the earphone-based lung function detection method of the present invention;

[0043] Figure 4 A flow velocity-volume curve comparison diagram of different lung diseases in a specific embodiment of the first embodiment of the earphone-based lung function detection method of the present invention;

[0044] Figure 5 It is a schematic diagram of the frequency response of the Mel filter in the second embodiment of the lung function detection method based on earphones of the present invention;

[0045] Figure 6 This is a schematic diagram of a deep learning network model of a specific embodiment of the second embodiment of the earphone-based lung function detection method of the present invention;

[0046] Figure 7 It is a schematic diagram of the residual spectrum of a specific embodiment of the third embodiment of the earphone-based lung function detection method of the present invention;

[0047] Figure 8 It is a schematic diagram of a normalized residual spectrum of a specific embodiment of the third embodiment of the earphone-based lung function detection method of the present invention;

[0048] Fig. 9 This is a schematic diagram of clustering results of a specific embodiment of the third embodiment of the earphone-based lung function detection method of the present invention;

[0049] Fig.10It is a comparison diagram of flow rate-volume curves generated by a non-standard lung function detection action in a specific embodiment of the third embodiment of the earphone-based lung function detection method of the present invention;

[0050] Fig.11 A software algorithm flow chart of a specific embodiment of a lung function detection method based on earphones of the present invention;

[0051] Fig.12 It is a schematic diagram of the functional modules of the earphone-based lung function detection system of the present invention.

[0052] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0054] At present, the academic community has proposed a series of software algorithms based on smartphones, which can be used to detect lung function. The general principle is: when the user is doing a lung function test, he needs to inhale and blow hard. The algorithm uses the microphone on the smartphone to record the airflow sound generated when the user blows and inhales hard, and analyzes these airflow sounds to obtain the user's lung function status. Although these designs significantly reduce the cost of use, their functions are greatly reduced. The above smartphone-based algorithms can only obtain a small number of lung function indicators, but cannot obtain the flow rate-volume curve that can be obtained by traditional medical spirometers and home spirometers (this curve describes the relationship between the change in gas flow rate and volume when the lungs exchange gases with the outside world when the user blows and inhales. Different lung diseases have their own specific flow rate-volume curve shapes), so the ability to characterize the user's lung function status is very limited.

[0055] To solve the above problems, the present invention provides a lung function detection method based on headphones, that is, the user's blowing and inhaling sounds are collected through the microphone on the headphones worn by the user, so that the user does not need to prepare expensive detection equipment or go to the hospital for detection, which increases the convenience of detection implementation and reduces the cost of detection equipment; by performing signal processing such as spectrum conversion and signal feature analysis on the collected airflow sound, and then using a deep learning algorithm, the spectrum of the blowing and inhalation stages is converted into a final flow rate-volume curve, thereby achieving the premise of maintaining low cost, based on ordinary commercial headphones to perform lung function detection can also obtain a flow rate-volume curve, greatly improving the lung function characterization capability of this solution, and solving the technical problem of how to achieve the premise of maintaining low cost, so that the lung function detection algorithm based on a smartphone can also obtain a flow rate-volume curve.

[0056] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0057] like Figure 1 As shown, the earphone-based pulmonary function detection system may include: a processor 1001, such as a CPU, a user interface 1003, a network interface 1004, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation of the device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0059] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a headphone-based lung function detection program.

[0060] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the earphone-based lung function detection program stored in the memory 1005 and perform the following operations:

[0061] Acquire the airflow sound collected by the microphone, and generate a target frequency spectrum according to the airflow sound, wherein the airflow sound is generated by a target user wearing the headset blowing and inhaling;

[0062] Analyze the signal characteristics of the target spectrum graph, and divide the spectrum of the blowing phase and the spectrum of the inhalation phase in the target spectrum graph based on the signal characteristics;

[0063] Using a preset deep learning algorithm, the blowing phase spectrum and the inhalation phase spectrum are combined and converted into a flow rate-volume curve that characterizes the lung function of the target user.

[0064] Furthermore, the step of acquiring the airflow sound collected by the microphone and generating a target spectrum diagram according to the airflow sound includes:

[0065] Performing a short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram;

[0066] The airflow sound spectrum graph is passed through a preset Mel filter bank to obtain a spectrum graph under the Mel scale as the target spectrum graph.

[0067] Furthermore, the step of performing short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram comprises:

[0068] Performing a short-time Fourier transform on the airflow sound to obtain an initial spectrum diagram;

[0069] A portion of the initial spectrum graph whose frequency is within a preset frequency range is retained as the airflow sound spectrum graph.

[0070] Furthermore, the step of using a preset deep learning algorithm to combine the blowing phase spectrum and the inhalation phase spectrum into a flow rate-volume curve characterizing the lung function of the target user includes:

[0071] Inputting the blowing phase spectrum and the inhalation phase spectrum into a preset convolutional neural network to obtain target convolution data;

[0072] Input the target convolution data into a preset recurrent neural network and a fully connected layer to obtain target output data;

[0073] According to the target output data, the gas flow rate signal corresponding to the frequency spectrum of the blowing phase and the matching of the gas flow rate signal of the inhalation phase is obtained, and the gas flow rate signal is integrated to obtain a gas volume signal;

[0074] The flow rate-volume curve is obtained by combining the gas flow rate signal and the gas volume signal.

[0075] Furthermore, the step of analyzing the signal features in the target spectrum graph and dividing the spectrum of the blowing phase and the spectrum of the inhalation phase in the target spectrum graph based on the signal features includes:

[0076] Identify the energy peak point in the target spectrum graph, and determine the blowing start time point on the horizontal axis of the target spectrum graph based on the energy peak point;

[0077] According to the preset standard duration of blowing and the blowing start time point, the blowing end time point on the horizontal axis is determined, so that the partial spectrum between the blowing start time point and the blowing end time point in the target spectrum graph is used as the blowing stage spectrum;

[0078] The target spectrum is obtained by removing the spectrum of the blowing stage from the target spectrum as a residual spectrum, and the residual spectrum is normalized to obtain a normalized residual spectrum.

[0079] A preset clustering algorithm is used to determine the portion of the normalized residual spectrum that belongs to the inhalation phase as the inhalation phase spectrum.

[0080] Furthermore, the step of using a preset clustering algorithm to determine the portion of the normalized residual spectrum belonging to the inhalation phase as the inhalation phase spectrum includes:

[0081] Clustering the frequency distribution at all moments in the normalized residual spectrum into several types of signals using the clustering algorithm;

[0082] Calculate the total energy corresponding to each type of signal and determine the type of signal with the highest total energy;

[0083] The time segment corresponding to the signal of the highest total energy is used as the inhalation time segment, so as to filter out the spectrum in the inhalation time segment from the remaining spectrum diagram as the inhalation stage spectrum.

[0084] Further, after the step of using a preset deep learning algorithm to combine the blowing phase spectrum and the inhalation phase spectrum into a flow rate-volume curve characterizing the lung function of the target user, the processor 1001 can be used to call the earphone-based lung function detection program stored in the memory 1005, and perform the following operations:

[0085] Analyzing the flow rate-volume curve to evaluate the breathing action quality of the target user;

[0086] When the breathing action quality does not meet the preset quality standard, a prompt message indicating that the action quality does not meet the standard is generated and output.

[0087] Based on the above hardware structure, an embodiment of the lung function detection method based on earphones of the present invention is proposed.

[0088] Reference Figure 2 , Figure 2 The flowchart of the first embodiment of the earphone-based lung function detection method of the present invention is shown in FIG. The method is applied to a terminal device connected to an earphone, and the earphone is equipped with a microphone. The method includes:

[0089] Step S10, acquiring the airflow sound collected by the microphone, and generating a target frequency spectrum according to the airflow sound, wherein the airflow sound is generated by the target user wearing the headset blowing and inhaling;

[0090] In this embodiment, the target spectrum graph refers to the spectrum graph corresponding to the airflow sound signal collected by the above-mentioned microphone. It should be noted that in order to ensure the accuracy of pulmonary function testing, when the user uses headphones to perform the inhalation and blowing actions required for the test, the following method can be used: the user puts on headphones, pinches the nose with his hand, and blows a standard pulmonary function test to an ordinary disposable mouthpiece (usually made of plastic and cheap). Among them, the standard pulmonary function test actions are: the first step is to take a deep breath until the lungs are full of gas; the second step is to blow with the greatest force to expel the gas in the lungs; the third step is to inhale with the greatest force so that the lungs are filled with gas again. When the user performs the above actions, the microphone in the headphones worn by the search can record the airflow sound. The headphones can use commercial headphones with microphones commonly found on the market.

[0091] Specifically, Figure 3 As shown. The terminal device connected with the earphone may include a microphone working circuit, a signal collection module and a computer connected to the signal collection module. The microphone working circuit is used to transmit the collected airflow sound signal to the signal collection module. The signal collection module includes ReSpeaker Voice AccessoryHAT (a microphone expansion board) produced by Seeed (Silicon Delivery) for analog-to-digital conversion, and a Raspberry Pi 3 Model B+ (a Raspberry Pi controller) for data collection and storage. The signal processed by the signal collection module can be input into the computer for signal analysis to obtain the above target spectrum diagram.

[0092] Step S20, analyzing the signal characteristics in the target spectrum diagram, and dividing the target spectrum diagram into an insufflation phase spectrum and an insufflation phase spectrum based on the signal characteristics;

[0093] In this embodiment, the signal features may specifically include slope changes in the spectrum graph, signal extreme values, etc. The blowing phase spectrum refers to the spectrum corresponding to the airflow sound generated by the blowing action of the target user when using the earphone to perform the lung function test; similarly, the inhalation phase spectrum refers to the spectrum corresponding to the airflow sound generated by the inhalation action of the target user when using the earphone to perform the lung function test.

[0094] Since the blowing stage and the inhalation stage each have different characteristics, these characteristics can be reflected in the spectrum diagram with some specific signal features. For example, for the blowing stage, the airflow sound energy of the blowing will be relatively large compared to the inhalation, and the energy will gradually increase over time; for the inhalation stage, the airflow sound energy is relatively small compared to the blowing. Therefore, the computer can divide the blowing stage spectrum belonging to the blowing stage and the inhalation stage spectrum belonging to the inhalation stage according to the representation of these characteristics on the spectrum diagram. It should be noted that in addition to the blowing stage spectrum and the inhalation stage spectrum, the target spectrum diagram may also contain environmental noise, so it needs to be removed during the extraction process.

[0095] Step S30, using a preset deep learning algorithm, the blowing phase spectrum and the inhalation phase spectrum are combined and converted into a flow rate-volume curve representing the lung function of the target user.

[0096] In this embodiment, the deep learning algorithm can be specifically constructed by a number of convolutional neural networks and a number of recurrent neural networks. The velocity-volume curve is a curve with the gas volume as the horizontal axis and the gas velocity as the vertical axis. Different lung diseases have their own specific velocity-volume curve shapes, such as Figure 4 As shown. The figure shows three flow rate-volume curves. From left to right, the first one is the flow rate-volume curve corresponding to normal lungs, the second one is the flow rate-volume curve corresponding to obstructive lung disease, and the third one is the flow rate-volume curve corresponding to restrictive lung disease. Specifically, the computer can process the insufflation phase spectrum and the inhalation phase spectrum through a preset deep learning algorithm, and after processing, the gas volume and gas flow rate corresponding to the insufflation phase and the inhalation phase can be obtained, and then the flow rate-volume curve can be drawn based on this.

[0097] After obtaining the flow rate-volume curve, common lung function indicators can be obtained by analyzing this curve.

[0098] The present embodiment provides a headphone-based lung function detection method, which collects the user's blowing and inhaling sounds through a microphone on the headphone worn by the user, so that the user does not need to prepare expensive detection equipment or go to a hospital for detection, which increases the convenience of detection implementation and reduces the cost of detection equipment; by performing signal processing such as spectrum conversion and signal feature analysis on the collected airflow sound, and then using a deep learning algorithm to convert the spectrum of the blowing and inhaling stages into a final flow rate-volume curve, it is achieved that under the premise of maintaining low cost, the flow rate-volume curve can be obtained by performing lung function detection based on ordinary commercial headphones, which greatly improves the lung function characterization capability of this solution and solves the technical problem of how to achieve the flow rate-volume curve based on the lung function detection algorithm based on a smartphone under the premise of maintaining low cost.

[0099] Furthermore, based on the above Figure 2 The first embodiment shown in the figure proposes a second embodiment of the earphone-based lung function detection method of the present invention. In this embodiment, step S10 includes:

[0100] Step S11, performing short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram;

[0101] Step S12, passing the airflow sound spectrum through a preset Mel filter bank to obtain a spectrum under the Mel scale as the target spectrum.

[0102] In this embodiment, the computer performs preliminary signal processing on the original airflow sound signal input by the signal collection module to obtain the characteristics of the signal in the frequency domain. Specifically, a short-time Fourier transform (STFT, short-timeFourier transform, or short-term Fourier transform) is first performed on it to obtain the above-mentioned airflow sound spectrum diagram based on the processing result; then the airflow sound spectrum diagram is passed through a preset Mel filter group, which includes multiple Mel filters, and the airflow sound spectrum diagram is passed through multiple Mel filters to obtain a spectrum diagram under the Mel scale (Mel frequency diagram).

[0103] Specifically, a Mel filter bank containing 100 filters is pre-designed, and the center frequency M(f) of each filter is calculated by the following formula:

[0104] M(f)=1125·log(1+f / 700).

[0105] Wherein, f is 100 frequency points equally spaced from 50 Hz to 1500 Hz.

[0106] The frequency response of each filter is calculated by the following formula (m represents the filter number, ranging from 1 to 100; k represents the kth frequency point on the frequency axis):

[0107]

[0108] The frequency responses of all Mel filters in the Mel filter bank are as follows: Figure 5 As shown, the horizontal axis represents frequency and the vertical axis represents amplitude. The Mel filter bank can be multiplied to obtain the Mel spectrum, that is:

[0109] M=H T S

[0110] Among them, M represents the Mel spectrum, H represents the Mel filter bank, and S represents the airflow sound spectrum.

[0111] The resulting Mel-spectrogram M retains the characteristics of the original spectrogram and has only 100 dimensions, which effectively reduces the signal dimension, facilitates subsequent processing, and highlights the characteristics of low-frequency signals (by observation, the low-frequency features on the spectrogram are far more complex than the high-frequency features).

[0112] Furthermore, step S11 includes:

[0113] Step S111, performing short-time Fourier transform on the airflow sound to obtain an initial spectrum diagram;

[0114] Step S112, retaining a portion of the initial spectrum graph whose frequencies are within a preset frequency range as the airflow sound spectrum graph.

[0115] In this embodiment, firstly, the collected original audio signal (i.e., the above-mentioned airflow sound) is subjected to a short-time Fourier transform (STFT), and the specific parameters are: sampling frequency: 48kHz, slice length: 50ms, number of fast Fourier transform (FFT) points: 2400, and overlapping ratio of adjacent slices: 75%. Since it has been found through experiments that the airflow sound generated by human breathing is roughly between 500Hz and 1500Hz, the part of 500-1500Hz is retained on the basis of the initial spectrum obtained after STFT (the frequency range can also be flexibly set according to actual needs). After this step, the frequency points of the obtained spectrum are 724 points.

[0116] After step S112, the spectrum graph is passed through a Mel filter bank, which has 100 filters, so the number of frequency points after passing through the Mel filter bank is 100. The Mel spectrum is obtained through the Mel filter bank and then input into the neural network.

[0117] Further, step S30 includes:

[0118] Step S31, inputting the blowing phase spectrum and the inhalation phase spectrum into a preset convolutional neural network to obtain target convolution data;

[0119] Step S32, inputting the target convolution data into a preset recurrent neural network and a fully connected layer to obtain target output data;

[0120] Step S33, obtaining the gas flow rate signal corresponding to the blowing phase spectrum and the inhalation phase matching according to the target output data, and integrating the gas flow rate signal to obtain a gas volume signal;

[0121] Step S34, combining the gas flow rate signal and the gas volume signal to obtain the flow rate-volume curve.

[0122] In this embodiment, the deep learning algorithm is designed based on convolutional neural networks, recurrent neural networks and fully connected layers.

[0123] As a specific example, two neural network models with identical structures are used in the blowing and inhalation stages. The deep learning network models used are as follows: Figure 6 As shown. The input of the model is a 100*N Mel spectrum graph (N represents the signal length). First, it passes through three convolutional neural networks (Conv), where the convolution kernel sizes of the three convolutional networks are (3, 3, 4), (3, 3, 8) and (3, 3, 16) respectively. After each convolutional layer, a LeakyReLU function (an advanced activation function) is used for activation, and a maximum pooling layer (Pool) is connected, and the kernel size of the pooling layer is (2, 2). After three convolutional layers, the dimension of the data changes to 16*25*(N / 4), and after a flattening operation, it becomes 400*(N / 4). Subsequently, the data is processed by a recurrent neural network (GRU), the hidden layer size of this recurrent neural network is 128, and the number of layers is 1. After passing through this recurrent neural network, the dimension of the data becomes 128*(N / 4). Subsequently, after two fully connected layers, the dimension of the data is processed to 1*(N / 4), where the Dropout method is used to process the data before the second fully connected layer. After the output is obtained, PCHIP interpolation (third-order Hermite interpolation) is performed to restore the output to a 1*N dimension. After the training of this neural network is completed, the output 1*N data is the gas flow rate corresponding to this segment of the signal. After obtaining the gas flow rate, the flow rate signal is integrated to obtain the gas volume signal. With the gas volume as the horizontal axis and the gas flow rate as the vertical axis, the flow rate-volume curve can be drawn.

[0124] Furthermore, based on the above Figure 2 The first embodiment shown in the figure proposes a third embodiment of the earphone-based lung function detection method of the present invention. In this embodiment, step S20 includes:

[0125] Step S21, identifying the energy peak point in the target spectrum graph, and determining the blowing start time point on the horizontal axis of the target spectrum graph based on the energy peak point;

[0126] Step S22, determining the blowing end time point on the horizontal axis according to the preset blowing standard duration and the blowing start time point, so as to use the partial spectrum between the blowing start time point and the blowing end time point in the target spectrum graph as the blowing stage spectrum;

[0127] Step S23, taking the part of the target spectrum graph excluding the spectrum of the blowing stage as the residual spectrum, and performing a normalization operation on the residual spectrum to obtain a normalized residual spectrum;

[0128] Step S24: using a preset clustering algorithm to determine the portion of the normalized residual spectrum that belongs to the inhalation phase as the inhalation phase spectrum.

[0129] In this embodiment, the spectrum of the blowing stage is extracted. Because the airflow acoustic energy of the blowing is relatively large, the Mel spectrum of the entire signal is accumulated along the frequency dimension to obtain a signal of the energy change of this signal over time. Taking the peak point of the energy signal as the starting point, look forward for the time point when the slope is less than the preset threshold, and this time point is recorded as the starting point of the blowing stage. Because the blowing stage requires the user to maintain the blowing posture for six seconds, the end point of the blowing stage is the sixth second after the starting point.

[0130] For spectrum extraction in the inhalation phase. Because the airflow sound of inhalation is very small, it is very easy to be mixed in the environmental noise and difficult to extract. Therefore, it is necessary to distinguish the inhalation signal from the environmental noise based on the frequency domain characteristics of the inhalation signal. First, remove the blowing signal extracted in the previous step from the entire signal. In the remaining Mel spectrum signal, the energy distribution of the frequency at each moment is between 0 and 1 through normalization operation. Then, the k-means clustering algorithm can be used (other clustering algorithms can also be selected according to actual conditions) to determine the spectrum of the inhalation phase.

[0131] Further, step S24 includes:

[0132] Step S241, clustering the frequency distribution at all moments in the normalized residual spectrum into several types of signals using the clustering algorithm;

[0133] Step S242, calculating the total energy corresponding to each type of signal, and determining the type of signal with the highest total energy;

[0134] Step S243 , taking the time segment corresponding to the type of signal with the highest total energy as the inhalation time segment, and filtering out the spectrum in the inhalation time segment from the remaining spectrum graph as the inhalation stage spectrum.

[0135] In this embodiment, the frequency distribution of all moments is clustered into three categories. Then, the total energy of the above three types of signals is calculated on the original Mel spectrum, and the time interval of the type of signal with the highest total energy is the time interval of the inhalation signal. Figure 7-9 shown. Figure 7 represents the remaining spectrum mentioned above, Figure 8 represents the normalized residual spectrum after normalization, Fig. 9 It represents the result of K-means clustering algorithm, where the dotted line represents the class with the highest energy, which is the time period of the inhalation signal.

[0136] Furthermore, after step S30, the method further includes:

[0137] Analyzing the flow rate-volume curve to evaluate the breathing action quality of the target user;

[0138] When the breathing action quality does not meet the preset quality standard, a prompt message indicating that the action quality does not meet the standard is generated and output.

[0139] In this embodiment, the flow rate volume curve can not only characterize different lung disease conditions, but also be used to evaluate the quality of the user's actions when performing lung function testing. Fig.10 The figure shows the flow rate-volume curves generated by three non-standard lung function tests. From left to right, the first one is the flow rate-volume curve corresponding to the action of not blowing hard enough, the second one is the flow rate-volume curve corresponding to the action of hesitating when blowing, and the third one is the flow rate-volume curve corresponding to the action of insufficient inhalation.

[0140] Because the flow rate-volume curve can be used to evaluate the quality of the user's movements, and if the quality does not meet the standard, it will affect the determination of lung function. Therefore, after obtaining the flow rate-volume curve, it can be checked whether it meets the preset quality standard (for example, the standard flow rate-volume curve is compared with the current curve, and if the key features differ too much, it is determined to be substandard). If it does not meet the standard, a prompt message can be output to prompt the user, and the user can choose to retest. The prompt message can also include the specific reason why the action does not meet the standard to clearly guide the user to correct the action.

[0141] As a specific embodiment, the software algorithm of this embodiment is divided into three modules: an audio feature extraction module, an inhalation / inhalation phase positioning module, and a flow rate-volume curve prediction module based on deep learning. The flow chart is as follows: Fig.11 shown.

[0142] Audio feature extraction module. The purpose of this module is to perform preliminary processing on the collected original audio signal to obtain the signal characteristics in the frequency domain. The method adopted is short-time Fourier transform and Mel filter bank filtering. The specific processing method has been described in detail above.

[0143] Blowing / inhaling phase positioning module. The main purpose of this module is to extract the blowing sound signal and the inhaling sound signal from the entire audio signal. First, the blowing phase positioning is performed, and then the inhaling phase positioning is performed. The specific positioning method has been described in detail above.

[0144] Flow rate-volume curve prediction module based on deep learning. The purpose of this module is to convert the features of the Mel spectrum map into gas flow rate and gas volume through deep learning. The specific processing method has been described in detail above and will not be repeated here.

[0145] like Fig.12 As shown, the present invention also provides a lung function detection system based on earphones, the system is provided in a terminal device connected to the earphones, the earphones are equipped with a microphone, and the system comprises:

[0146] a target spectrum generating module 10, configured to obtain the airflow sound collected by the microphone and generate a target spectrum diagram according to the airflow sound, wherein the airflow sound is generated by the target user wearing the headset blowing and inhaling;

[0147] A phase spectrum division module 20 is used to analyze the signal characteristics of the target spectrum graph, and divide the target spectrum graph into an inhalation phase spectrum and an inspiration phase spectrum based on the signal characteristics;

[0148] The target curve conversion module 30 is used to use a preset deep learning algorithm to combine the blowing phase spectrum and the inhalation phase spectrum to convert them into a flow rate-volume curve that represents the lung function of the target user.

[0149] Optionally, the target spectrum generating module 10 includes:

[0150] An airflow spectrum generating unit, used for performing a short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram;

[0151] The target spectrum generating unit is used to pass the airflow sound spectrum graph through a preset Mel filter bank to obtain a spectrum graph under the Mel scale as the target spectrum graph.

[0152] Optionally, the airflow spectrum generating unit is further used for:

[0153] Performing a short-time Fourier transform on the airflow sound to obtain an initial spectrum diagram;

[0154] A portion of the initial spectrum graph whose frequency is within a preset frequency range is retained as the airflow sound spectrum graph.

[0155] Optionally, the target curve conversion module 30 includes:

[0156] A convolution data acquisition unit, used for inputting the blowing phase spectrum and the inhalation phase spectrum into a preset convolution neural network to obtain target convolution data;

[0157] An output data acquisition unit, used to input the target convolution data into a preset recurrent neural network and a fully connected layer to obtain target output data;

[0158] A volume signal acquisition unit, used to obtain the gas flow rate signal corresponding to the blowing phase spectrum and the inhalation phase according to the target output data, and perform integration processing on the gas flow rate signal to obtain a gas volume signal;

[0159] A target curve generating unit is used to obtain the flow rate-volume curve by combining the gas flow rate signal and the gas volume signal.

[0160] Optionally, the stage spectrum division module 20 includes:

[0161] A blowing start determining unit, used to identify an energy peak point in the target spectrum graph, and determine a blowing start time point on the horizontal axis of the target spectrum graph based on the energy peak point;

[0162] A blowing stage determination unit, used to determine the blowing end time point on the horizontal axis according to a preset blowing standard duration and the blowing start time point, so as to use the partial spectrum between the blowing start time point and the blowing end time point in the target spectrum graph as the blowing stage spectrum;

[0163] A residual spectrum processing unit, used for taking the part of the target spectrum graph excluding the spectrum of the blowing stage as the residual spectrum, and performing a normalization operation on the residual spectrum to obtain a normalized residual spectrum;

[0164] The inhalation phase determination unit is used to determine the part of the normalized residual spectrum belonging to the inhalation phase by using a preset clustering algorithm as the inhalation phase spectrum.

[0165] Optionally, the inhalation phase determination unit is further used for:

[0166] Clustering the frequency distribution at all moments in the normalized residual spectrum into several types of signals using the clustering algorithm;

[0167] Calculate the total energy corresponding to each type of signal and determine the type of signal with the highest total energy;

[0168] The time segment corresponding to the signal of the highest total energy is used as the inhalation time segment, so as to filter out the spectrum in the inhalation time segment from the remaining spectrum diagram as the inhalation stage spectrum.

[0169] Optionally, the earphone-based pulmonary function detection system further includes:

[0170] A motion quality analysis module, used for analyzing the flow rate-volume curve to evaluate the breathing motion quality of the target user;

[0171] The prompt information output module is used to generate and output prompt information that the action quality does not meet the preset quality standard when the breathing action quality does not meet the preset quality standard.

[0172] The invention also provides a lung function detection device based on earphones.

[0173] The headphone-based pulmonary function detection device includes a processor, a memory, and a headphone-based pulmonary function detection program stored in the memory and executable on the processor, wherein when the headphone-based pulmonary function detection program is executed by the processor, the steps of the headphone-based pulmonary function detection method as described above are implemented.

[0174] Among them, the method implemented when the earphone-based pulmonary function detection program is executed can refer to the various embodiments of the earphone-based pulmonary function detection method of the present invention, and will not be repeated here.

[0175] The present invention also provides a computer-readable storage medium.

[0176] The computer-readable storage medium of the present invention stores an earphone-based pulmonary function detection program, and when the earphone-based pulmonary function detection program is executed by a processor, the steps of the earphone-based pulmonary function detection method described above are implemented.

[0177] Among them, the method implemented when the earphone-based pulmonary function detection program is executed can refer to the various embodiments of the earphone-based pulmonary function detection method of the present invention, and will not be repeated here.

[0178] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned earphone-based lung function detection method are implemented.

[0179] Among them, the method implemented when the computer program is executed can refer to the various embodiments of the earphone-based lung function detection method of the present invention, and will not be repeated here.

[0180] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0181] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0183] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting lung function based on earphones, characterized in that: The method is applied to a terminal device connected to a headset, the headset being equipped with a microphone, and the method comprises: Acquire the airflow sound collected by the microphone, and generate a target frequency spectrum according to the airflow sound, wherein the airflow sound is generated by the target user wearing the headset blowing and inhaling; Analyze the signal characteristics of the target spectrum graph, and divide the target spectrum graph into an inhalation phase spectrum and an inspiration phase spectrum based on the signal characteristics; Using a preset deep learning algorithm, the insufflation phase spectrum and the insufflation phase spectrum are combined and converted into a flow rate-volume curve representing the lung function of the target user; The step of acquiring the airflow sound collected by the microphone and generating a target spectrum diagram according to the airflow sound comprises: Performing a short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram; Passing the airflow sound spectrum through a preset Mel filter bank to obtain a spectrum under the Mel scale as the target spectrum; The step of using a preset deep learning algorithm to combine the blowing phase spectrum and the inhalation phase spectrum into a flow rate-volume curve representing the lung function of the target user includes: Inputting the blowing phase spectrum and the inhalation phase spectrum into a preset convolutional neural network to obtain target convolution data; Input the target convolution data into a preset recurrent neural network and a fully connected layer to obtain target output data; According to the target output data, the gas flow rate signal corresponding to the frequency spectrum of the blowing phase and the matching of the gas flow rate signal of the inhalation phase is obtained, and the gas flow rate signal is integrated to obtain a gas volume signal; The flow rate-volume curve is obtained by combining the gas flow rate signal and the gas volume signal.

2. The earphone-based lung function detection method according to claim 1, characterized in that: The step of performing short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram comprises: Performing a short-time Fourier transform on the airflow sound to obtain an initial spectrum diagram; A portion of the initial spectrum graph whose frequency is within a preset frequency range is retained as the airflow sound spectrum graph.

3. The earphone-based lung function detection method according to claim 1, characterized in that: The step of analyzing the signal features in the target spectrum graph and dividing the spectrum in the blowing phase and the spectrum in the inhalation phase in the target spectrum graph based on the signal features comprises: Identify the energy peak point in the target spectrum graph, and determine the blowing start time point on the horizontal axis of the target spectrum graph based on the energy peak point; According to the preset standard duration of blowing and the blowing start time point, the blowing end time point on the horizontal axis is determined, so that the partial spectrum between the blowing start time point and the blowing end time point in the target spectrum graph is used as the blowing stage spectrum; The target spectrum is obtained by removing the spectrum of the blowing stage from the target spectrum as a residual spectrum, and the residual spectrum is normalized to obtain a normalized residual spectrum. A preset clustering algorithm is used to determine the portion of the normalized residual spectrum that belongs to the inhalation phase as the inhalation phase spectrum.

4. The earphone-based lung function detection method according to claim 3, characterized in that: The step of using a preset clustering algorithm to determine the portion of the normalized residual spectrum belonging to the inhalation phase as the inhalation phase spectrum includes: Clustering the frequency distribution at all moments in the normalized residual spectrum into several types of signals using the clustering algorithm; Calculate the total energy corresponding to each type of signal and determine the type of signal with the highest total energy; The time segment corresponding to the signal of the highest total energy is used as the inhalation time segment, so as to filter out the spectrum in the inhalation time segment from the remaining spectrum diagram as the inhalation stage spectrum.

5. The earphone-based lung function detection method according to any one of claims 1 to 4, characterized in that: After the step of using a preset deep learning algorithm to combine the insufflation phase spectrum and the inhalation phase spectrum into a flow rate-volume curve representing the lung function of the target user, the method further includes: Analyzing the flow rate-volume curve to evaluate the breathing action quality of the target user; When the breathing action quality does not meet the preset quality standard, a prompt message indicating that the action quality does not meet the standard is generated and output.

6. A lung function detection system based on earphones, characterized in that: The system is provided in a terminal device connected to a headset, the headset is equipped with a microphone, and the system comprises: a target spectrum generation module, used to obtain the airflow sound collected by the microphone and generate a target spectrum graph according to the airflow sound, wherein the airflow sound is generated by the target user wearing the headset blowing and inhaling; The target spectrum generation module is further used to perform short-time Fourier transform on the airflow sound to obtain an airflow sound spectrum diagram; pass the airflow sound spectrum diagram through a preset Mel filter group to obtain a spectrum diagram under the Mel scale as the target spectrum diagram; A phase spectrum division module, used for analyzing the signal characteristics in the target spectrum graph, and dividing the spectrum in the blowing phase and the spectrum in the inhalation phase in the target spectrum graph based on the signal characteristics; A target curve conversion module, used to use a preset deep learning algorithm to convert the insufflation phase spectrum and the insufflation phase spectrum into a flow rate-volume curve representing the lung function of the target user; The target curve conversion module is also used to input the blowing phase spectrum and the inhalation phase spectrum into a preset convolutional neural network to obtain target convolution data; input the target convolution data into a preset recurrent neural network and a fully connected layer to obtain target output data; obtain the gas flow rate signal corresponding to the blowing phase spectrum and the inhalation phase according to the target output data, and integrate the gas flow rate signal to obtain a gas volume signal; and combine the gas flow rate signal and the gas volume signal to obtain the flow rate-volume curve.

7. A lung function detection device based on earphones, characterized in that: The headphone-based pulmonary function detection device comprises: a memory, a processor, and a headphone-based pulmonary function detection program stored in the memory and executable on the processor. When the headphone-based pulmonary function detection program is executed by the processor, the steps of the headphone-based pulmonary function detection method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is executed by a processor, the steps of the earphone-based lung function detection method according to any one of claims 1 to 5 are implemented.

Images