Cough sound recognition method based on learnable front end

Abstract

The invention discloses a cough sound recognition method based on a learnable front end, and belongs to the field of artificial intelligence auxiliary diagnosis. The method specifically comprises the following steps: constructing a cough sound data set and a non-cough sound data set; preprocessing the audio data, segmenting a cough segment and a non-cough segment in the audio, and removing a mute segment; designing a filter bank capable of learning parameters; designing a time domain and frequency domain enhancement module; constructing a deep learning classification model containing a learnable front end; training the deep learning model and calculating related indexes, and storing the trained model; and finally, the cough sound is detected by using the trained model. According to the method, the learnable filter bank is used for training filter parameters for cough detection events, meanwhile, the time domain and frequency domain enhancement module is used for enhancing cough event information based on time domain and frequency domain dual channels, and the learnable front end formed by the learnable filter and the time domain enhancement module can improve the feature extraction capability of the deep learning network. The cough sound detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence-assisted diagnosis, and in particular to a cough sound recognition method based on a learnable front end. Background Art

[0002] With the continuous development of industrial technology, the technology of audio classification through deep learning has become more mature, and the use of body sound to detect diseases has become a current research hotspot. Researchers have found that most patients with lung diseases are prone to coughing, so using the frequency and number of coughs to detect the incidence of lung diseases is a feasible and rapid detection method.

[0003] Currently, most common cough recognition methods use two-dimensional feature maps as model input for training. However, coughs last for a short time, and traditional feature extraction methods cannot fully reflect the information contained in coughs. Compared with the two-dimensional feature maps extracted by traditional methods, using a learnable front end to extract two-dimensional feature maps can help the model better complete the cough recognition task. Summary of the invention

[0004] In view of the problems described in the background technology, the present invention intends to combine a learnable front end and propose a cough sound recognition method based on a learnable front end to realize the detection of cough sounds and the evaluation of the efficacy of disease diagnosis.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] S1, build a dataset of cough and non-cough sounds;

[0007] S2, preprocessing the audio data, segmenting the coughing segments from the non-coughing segments in the audio, and removing the silent segments;

[0008] S3, design filter banks with learnable parameters;

[0009] S4, design time domain and frequency domain enhancement module;

[0010] S5, builds a deep learning classification model with a learnable front end;

[0011] S6, train the model and calculate relevant indicators, and save the trained model;

[0012] S7, using the trained model to detect cough sounds and evaluate the efficacy of disease diagnosis;

[0013] Further, in S1, Coswara is an open source dataset and research project related to COVID-19, and the audio categories include coughs, breath sounds, and speech. These include breath sounds (fast and slow), coughs (deep and shallow), continuously pronounced vowel phonemes, and speech audio counting at slow and fast rhythms. The entire dataset provides about 9,000 cough clips, 9,000 breath clips, and 9,000 speech clips. The Coswara dataset is collected based on real environments and contains a large amount of cough sound data. Therefore, we use the Coswara dataset as the cough sound data source for the cough detection algorithm.

[0014] At the same time, we use the ESC-50 dataset and the Urbansound8K dataset as the source of non-cough sound data for the cough detection algorithm. Among them, the ESC-50 dataset contains recording samples of various environmental sounds, divided into 50 different classes, each class has 40 samples, and a total of 2000 samples. The UrbanSound8K dataset contains common sound samples in urban environments, divided into 10 categories, with a total of 8732 samples, and the length of each sample does not exceed 4 seconds. The ESC-50 and Urbansound8K datasets provide a wealth of environmental sound samples, including animal sounds, natural sounds, human non-verbal sounds, household sounds, and external environmental sounds. At the same time, their audio samples are recorded in real environments and contain different background noises and audio characteristics. Diverse audio categories can help the model better distinguish coughs from other environmental sounds and improve the generalization ability of the model; real environmental audio helps to enhance the robustness and practicality of the model.

[0015] Furthermore, in S2, the audio data is segmented into 0.5 second audio segments, the cough audio segments are silenced, and the cough audio segments and non-cough audio segments are sorted.

[0016] Furthermore, in S3, the learnable front end is a general audio front end designed for audio event classification. It provides customized audio time-frequency representation for specific tasks, transforms the fixed parameters in the fixed parameter filter into parameters that need to be learned, and optimizes the learning through a deep learning network. Compared with the fixed time-frequency representation of the fixed parameter filter, the learnable front end shows better classification results for specific task data sets.

[0017] The learnable front end can be decomposed into three components (filtering, pooling, and dynamic compression), and each part is trained to optimize parameters.

[0018] The filtering part uses a complex-valued filter to perform convolution and square modulus operations to obtain the time-frequency representation. The filtering formula is as follows:

[0019] f n=|x*ρ n | 2 ∈R T ,n=1,……,N,

[0020] where ρ n is a complex-valued one-dimensional filter, and N is the number of filters.

[0021] The filter parameters are initialized using Gabor filters, which are generated by modulating the Gaussian kernel with a sinusoidal signal and can capture frequency and time domain information well. Compared with unconstrained filters, Gabor filters have fewer parameters, reduce the complexity of the model and improve computational efficiency.

[0022] The pooling operation reduces the size and complexity of the feature map by downsampling it, thereby reducing the amount of computation and the number of parameters in the subsequent layers, which can accelerate the training and inference process of the model and reduce the risk of overfitting. Common pooling methods include average pooling and maximum pooling. Maximum pooling and average pooling can extract the maximum or average value of the feature, making the model robust to small translations of the input data. However, compared with maximum pooling and average pooling, low-pass filtering shows better performance, especially when the signal is downsampled, it can better retain important information of the signal. The learnable front end uses Gaussian impulse response to define the frequency response of the low-pass filter. This parameterization method is simple and effective, and can better adapt to the frequency change characteristics of different channels.

[0023] The learnable front end uses dynamic compression instead of logarithmic compression of traditional features. The time-frequency features are processed by logarithmic compression to simulate the nonlinear human perception of loudness. However, the first limitation of this approach is that the same compression function is applied to all frequency channels regardless of the content of the frequency channels. The second limitation is that the nonlinear choice of the compression function is fixed. PCEN provides a learnable alternative that allows the model to learn the compression function parameters for each channel instead of adopting a fixed logarithmic function or other predefined compression function. This approach normalizes the features by considering the exponential moving average of each time step, and adds offset and exponential compression to enhance the perception of loudness while avoiding some limitations of traditional methods. Therefore, the learnable front end uses PCEN (Per-Channel Energy Normalization) to dynamically compress the time-frequency features. The PCEN formula is as follows:

[0024]

[0025] Where t=1;…;M represents the time step, and n=1;…;N represents the channel index. n , δ n, r n are learnable parameters.

[0026] The learnable front end dynamically optimizes the learnable parameters in filtering, pooling, and dynamic compression through network training.

[0027] Furthermore, in S4, most current audio classification tasks use two-dimensional feature maps as model input for training. Compared with the original two-dimensional feature map, the improved two-dimensional feature map can help the model better complete the audio classification task. The values ​​in the feature map describe the energy distribution of the audio signal in the time-frequency domain. In order to improve the two-dimensional feature map, we use the time-frequency enhancement module to enhance the features of the spectrogram in the time-frequency domain. The time-frequency enhancement module consists of several learnable enhancement blocks, which are responsible for extracting features and generating masks for enhancing the spectrogram. The mask assigns important weights to the features in the spectrogram, thereby enhancing the spectrogram. The learnable front end and the time-frequency enhancement module can be used in conjunction. When using the learnable front end as the feature extraction module of the audio, we can add the time-frequency enhancement module after the learnable front end to provide an enhanced feature map, and then input the enhanced feature map into the model for classification.

[0028] The time-frequency enhancement module consists of a time dimension branch and a frequency dimension branch, which are used to simulate the audio energy distribution and guide the model to focus on "where" (time frame) and "what" (frequency channel) respectively.

[0029] The time-frequency enhancement module uses two branches to generate a one-dimensional frequency-dimensional attention map and a one-dimensional time-dimensional attention map in parallel, and combines them through matrix multiplication to obtain the final two-dimensional attention map. The specific process is as follows Figure 3 As shown:

[0030] The input two-dimensional feature map passes through the time-frequency enhancement module, which reduces the interference of noise in both the time dimension and the frequency dimension, enhances the characteristics of the target task audio, facilitates the subsequent network to extract deep features, and improves the accuracy of cough audio classification.

[0031] Furthermore, in S5, the deep learning model training framework based on the learnable front end includes four modules, namely, an audio data preprocessing module, a learnable filter feature extraction module, a time-frequency feature enhancement module and a deep learning model. In the audio data preprocessing module, audio data fragments are obtained by performing preprocessing operations such as pre-emphasis, framing, and windowing on the audio data. The learnable filter feature extraction module dynamically adjusts parameters such as the center frequency and bandwidth by using a learnable filter to extract audio features suitable for cough detection. The time-frequency feature enhancement module enhances the extracted two-dimensional features in the time-frequency domain. The deep learning model inputs the enhanced two-dimensional feature map and is trained through the deep learning model.

[0032] Furthermore, in S6, the cough recognition model training is based on the binary classification task of cough and non-cough. The loss function of the model adopts the binary cross entropy loss function, and its formula is as follows:

[0033]

[0034] Here, y is a binary label 0 or 1, and p(y) is the probability that the output belongs to the y label.

[0035] The model minimizes the loss function and uses back propagation to dynamically adjust the parameters within the model to improve model performance.

[0036] The model evaluation indicators include accuracy, precision, recall and F1 score.

[0037] The advantages of the present invention are as follows: the present method proposes a cough recognition method based on a learnable front end, through the learnable front end composed of a learnable filter and a time-frequency enhancement module, dynamically adjusts the learnable parameters in the learnable front end during the model training process, extracts audio features suitable for cough recognition tasks from cough audio data, and improves the accuracy of cough detection. This method can be applied to subsequent real-time applications, such as health monitoring and mobile applications, and provides an effective measure to prevent related diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] As a part of the present invention, the drawings are mainly intended to facilitate the understanding of the present invention and are used for explanation, and should not constitute an improper limitation to the present invention.

[0039] Figure 1 is a flow chart of the method of the present invention.

[0040] Figure 2 It is a flow chart of the learnable parameter filter of the method described in the present invention.

[0041] Figure 3 It is a flow chart of the time-frequency enhancement module of the method described in the present invention. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, the cough sound recognition method based on a learnable front end described in the present invention comprises the following steps:

[0044] S1, build a dataset of cough and non-cough sounds;

[0045] Coswara is an open source dataset and research project related to COVID-19. The audio categories include coughs, breath sounds, and speech. These include breath sounds (fast and slow), coughs (deep and shallow), continuously pronounced vowel phonemes, and speech audio counting at slow and fast rhythms. The entire dataset provides about 9,000 cough clips, 9,000 breath clips, and 9,000 speech clips. The Coswara dataset is collected based on real environments and contains a large amount of cough sound data. Therefore, we use the Coswara dataset as the source of cough sound data for the cough detection algorithm.

[0046] At the same time, we use the ESC-50 dataset and the Urbansound8K dataset as the source of non-cough sound data for the cough detection algorithm. Among them, the ESC-50 dataset contains recording samples of various environmental sounds, divided into 50 different classes, each class has 40 samples, and a total of 2000 samples. The UrbanSound8K dataset contains common sound samples in urban environments, divided into 10 categories, with a total of 8732 samples, and the length of each sample does not exceed 4 seconds. The ESC-50 and Urbansound8K datasets provide a wealth of environmental sound samples, including animal sounds, natural sounds, human non-verbal sounds, household sounds, and external environmental sounds. At the same time, their audio samples are recorded in real environments and contain different background noises and audio characteristics. Diverse audio categories can help the model better distinguish coughs from other environmental sounds and improve the generalization ability of the model; real environmental audio helps to enhance the robustness and practicality of the model.

[0047] S2, preprocessing the audio data, segmenting the coughing segments from the non-coughing segments in the audio, and removing the silent segments;

[0048] The audio data is segmented into 0.5 second audio segments, the cough audio segments are silenced, and the cough audio segments and non-cough audio segments are sorted.

[0049] S3, design filter banks with learnable parameters;

[0050] The learnable front end is a general audio front end designed for audio event classification. It provides customized audio time-frequency representation for specific tasks, transforms the fixed parameters in the fixed parameter filter into parameters that need to be learned, and optimizes learning through a deep learning network. Compared with the fixed time-frequency representation of the fixed parameter filter, the learnable front end shows better classification results for specific task data sets.

[0051] The learnable front end can be decomposed into three components (filtering, pooling, and dynamic compression), and each part is trained to optimize parameters.

[0052] The filtering part uses a complex-valued filter to perform convolution and square modulus operations to obtain the time-frequency representation. The filtering formula is as follows:

[0053] f n =|x*ρ n | 2 ∈R T ,n=1,……,N,

[0054] where ρ n is a complex-valued one-dimensional filter, and N is the number of filters.

[0055] The filter parameters are initialized using Gabor filters, which are generated by modulating the Gaussian kernel with a sinusoidal signal and can capture frequency and time domain information well. Compared with unconstrained filters, Gabor filters have fewer parameters, reduce the complexity of the model and improve computational efficiency.

[0056] The pooling operation reduces the size and complexity of the feature map by downsampling it, thereby reducing the amount of computation and the number of parameters in the subsequent layers, which can accelerate the training and inference process of the model and reduce the risk of overfitting. Common pooling methods include average pooling and maximum pooling. Maximum pooling and average pooling can extract the maximum or average value of the feature, making the model robust to small translations of the input data. However, compared with maximum pooling and average pooling, low-pass filtering shows better performance, especially when the signal is downsampled, it can better retain important information of the signal. The learnable front end uses Gaussian impulse response to define the frequency response of the low-pass filter. This parameterization method is simple and effective, and can better adapt to the frequency change characteristics of different channels.

[0057] The learnable front end uses dynamic compression instead of logarithmic compression of traditional features. The time-frequency features are processed by logarithmic compression to simulate the nonlinear human perception of loudness. However, the first limitation of this approach is that the same compression function is applied to all frequency channels regardless of the content of the frequency channels. The second limitation is that the nonlinear choice of the compression function is fixed. PCEN provides a learnable alternative that allows the model to learn the compression function parameters for each channel instead of adopting a fixed logarithmic function or other predefined compression function. This approach normalizes the features by considering the exponential moving average of each time step, and adds offset and exponential compression to enhance the perception of loudness while avoiding some limitations of traditional methods. Therefore, the learnable front end uses PCEN (Per-Channel Energy Normalization) to dynamically compress the time-frequency features. The PCEN formula is as follows:

[0058]

[0059] Where t=1;…;M represents the time step, and n=1;…;N represents the channel index. n , δ n , r n are learnable parameters.

[0060] The learnable front end dynamically optimizes the learnable parameters in filtering, pooling, and dynamic compression through network training.

[0061] S4, design time domain and frequency domain enhancement module;

[0062] Currently, most audio classification tasks use two-dimensional feature maps as model input for training. Compared with the original two-dimensional feature map, the improved two-dimensional feature map can help the model better complete the audio classification task. The values ​​in the feature map describe the energy distribution of the audio signal in the time-frequency domain. In order to improve the two-dimensional feature map, we use the time-frequency enhancement module to enhance the features of the spectrogram in the time-frequency domain. The time-frequency enhancement module consists of several learnable enhancement blocks, which are responsible for extracting features and generating masks for enhancing the spectrogram. The mask assigns important weights to the features in the spectrogram, thereby enhancing the spectrogram. The learnable front end and the time-frequency enhancement module can be used in conjunction. When using the learnable front end as the feature extraction module of audio, we can add the time-frequency enhancement module after the learnable front end to provide an enhanced feature map, and then input the enhanced feature map into the model for classification.

[0063] The time-frequency enhancement module consists of a time dimension branch and a frequency dimension branch, which are used to simulate the audio energy distribution and guide the model to focus on "where" (time frame) and "what" (frequency channel) respectively.

[0064] The time-frequency enhancement module uses two branches to generate a one-dimensional frequency-dimensional attention map and a one-dimensional time-dimensional attention map in parallel, and combines them through matrix multiplication to obtain the final two-dimensional attention map. The specific process is as follows Figure 3 As shown:

[0065] The input two-dimensional feature map passes through the time-frequency enhancement module, which reduces the interference of noise in both the time dimension and the frequency dimension, enhances the characteristics of the target task audio, facilitates the subsequent network to extract deep features, and improves the accuracy of cough audio classification.

[0066] S5, builds a deep learning classification model with a learnable front end;

[0067] The deep learning model training framework based on the learnable front-end includes four modules, namely, the audio data preprocessing module, the learnable filter feature extraction module, the time-frequency feature enhancement module and the deep learning model. In the audio data preprocessing module, the audio data fragments are obtained by preprocessing operations such as pre-emphasis, framing and windowing. The learnable filter feature extraction module uses the learnable filter to dynamically adjust the center frequency, bandwidth and other parameters to extract audio features suitable for cough detection. The time-frequency feature enhancement module enhances the extracted two-dimensional features in the time-frequency domain. The deep learning model inputs the enhanced two-dimensional feature map and is trained through the deep learning model.

[0068] S6, train the model and calculate relevant indicators, and save the trained model;

[0069] The cough recognition model training is based on the binary classification task of cough and non-cough. The loss function of the model adopts the binary cross entropy loss function, and its formula is as follows:

[0070]

[0071] Here, y is a binary label 0 or 1, and p(y) is the probability that the output belongs to the y label.

[0072] The model minimizes the loss function and uses back propagation to dynamically adjust the parameters within the model to improve model performance.

[0073] The model evaluation indicators include accuracy, precision, recall and F1 score.

[0074] S7, use the trained model to detect cough sounds and evaluate the efficacy of disease diagnosis.

Claims

1. A cough sound recognition method based on a learnable front end, characterized in that: The method performs the following steps: S1, build a dataset of cough and non-cough sounds; S2, preprocessing the audio data, segmenting the coughing segments from the non-coughing segments in the audio, and removing the silent segments; S3, design filter banks with learnable parameters; S4, design time domain and frequency domain enhancement module; S5, builds a deep learning classification model with a learnable front end; S6, train the model and calculate relevant indicators, and save the trained model; S7, use the trained model to detect cough sounds and evaluate the efficacy of disease diagnosis.

2. The cough sound recognition method based on a learnable front end according to claim 1, characterized in that: In step S1: Coswara is an open source dataset and research project related to COVID-19. The audio categories include cough, breath, and speech. We use the Coswara dataset as the cough data source for the cough detection algorithm. At the same time, we use the ESC-50 dataset and the Urbansound8K dataset as the source of non-cough sound data for the cough detection algorithm. Among them, the ESC-50 dataset contains recording samples of various environmental sounds, divided into 50 different classes, each class has 40 samples, and a total of 2000 samples. The UrbanSound8K dataset contains common sound samples in urban environments, divided into 10 categories, with a total of 8732 samples, and the length of each sample does not exceed 4 seconds. The ESC-50 and Urbansound8K datasets provide rich environmental sound samples. At the same time, their audio samples are recorded in real environments, containing different background noises and audio characteristics. Diverse audio categories can help the model better distinguish coughs from other environmental sounds and improve the generalization ability of the model; real environmental audio helps to enhance the robustness and practicality of the model.

3. The cough sound recognition method based on a learnable front end according to claim 1, characterized in that: In step S2: The audio data is segmented into 0.5 second audio segments, the cough audio segments are silenced, and the cough audio segments and non-cough audio segments are sorted.

4. The cough sound recognition method based on a learnable front end according to claim 1, characterized in that: In step S3: The learnable front end is a general audio front end designed for audio event classification. It transforms the fixed parameters in the fixed parameter filter into parameters that need to be learned and optimizes the learning through a deep learning network. The learnable front end can be decomposed into three components (filtering, pooling, and dynamic compression), and each part is trained to optimize parameters. The filtering part uses a complex-valued filter to perform convolution and square modulus operations to obtain the time-frequency representation. The filtering formula is as follows: f n =|x*ρ n | 2 ∈R T ,n=1,……,N, where ρ n is a complex-valued one-dimensional filter, and N is the number of filters. The pooling operation reduces the size and complexity of the feature map by downsampling it, thereby reducing the amount of calculation and number of parameters in the subsequent layers, which can accelerate the training and reasoning process of the model and reduce the risk of overfitting. The learning front end uses Gaussian impulse response to define the frequency response of the low-pass filter. This parameterization method is simple and effective, and can better adapt to the frequency change characteristics of different channels. The learnable front end uses dynamic compression instead of logarithmic compression of traditional features. It allows the model to learn the compression function parameters for each channel instead of using a fixed logarithmic function or other predefined compression functions. The learnable front end uses PCEN (Per-Channel Energy Normalization) to dynamically compress time-frequency features. The PCEN formula is as follows: Where t=1;…;M represents the time step, and n=1;…;N represents the channel index. n , δ n , r n are learnable parameters. The learnable front end dynamically optimizes the learnable parameters in filtering, pooling, and dynamic compression through network training.

5. The cough sound recognition method based on a learnable front end according to claim 1, characterized in that: In step S4: The time-frequency enhancement module consists of a time dimension branch and a frequency dimension branch, which are used to simulate the audio energy distribution and guide the model to focus on "where" (time frame) and "what to do" (frequency channel). The time-frequency enhancement module uses two branches to generate a one-dimensional frequency-dimensional attention map and a one-dimensional time-dimensional attention map in parallel, and combines them through matrix multiplication to obtain the final two-dimensional attention map. The specific process is shown in Figure 3.

6. The method for cough sound recognition based on a learnable front end according to claim 1, characterized in that: In step S5: As shown in Figure 1, the deep learning model training framework based on the learnable front-end includes four modules, namely, audio data preprocessing module, learnable filter feature extraction module, time-frequency feature enhancement module and deep learning model.

7. The method for cough sound recognition based on a learnable front end according to claim 1, characterized in that: In step S6: The cough recognition model training is based on the binary classification task of cough and non-cough. The loss function of the model adopts the binary cross entropy loss function, and its formula is as follows: Here, y is a binary label 0 or 1, and p(y) is the probability that the output belongs to the y label. The model minimizes the loss function and uses back propagation to dynamically adjust the parameters within the model to improve model performance. The model evaluation indicators include accuracy, precision, recall and F1 score.

Images