A method and apparatus for detecting lung disease using cough sounds

Abstract

The present application relates to a method for detecting lung disease by cough sound, which specifically comprises: removing non-cough sound segments from original cough audio to obtain cough audio, and obtaining the start and end time of each single cough segment in the cough audio; generating a logarithmic mel-cepstrum matrix corresponding to each single cough segment according to the start and end time of each single cough segment, and calculating a position encoding matrix corresponding to each single cough segment; obtaining a feature matrix of the cough audio according to the logarithmic mel-cepstrum matrix and the position encoding matrix corresponding to each single cough segment; multiplying the normalized feature matrix of the cough audio by a proportion factor, adding the obtained product to the feature matrix of the cough audio, and inputting the result into a classification network for classification. The present application also relates to a device comprising a cough sound detection unit, an audio signal processing unit, a splicing unit, a normalization unit and a classification network unit. The method and device of the present application can improve the accuracy of the detection result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-assisted diagnosis, and in particular to a method and apparatus for detecting lung diseases using cough sounds. Background Technology

[0002] With the continuous development of industrial technology and the maturation of audio classification technology using deep learning, people are paying more attention to the practical application of related technologies, and using body sounds to detect diseases has become a current research hotspot. Coughing is one of the main symptoms after contracting lung diseases, and researchers have found that the cough sounds of patients with different lung diseases have different underlying characteristics. Therefore, using cough sounds to detect lung diseases is a feasible and rapid detection method.

[0003] Currently, there are two main methods. The first involves extracting statistical features related to cough audio at the frame or segment level and then using machine learning algorithms for classification. These features typically include root mean square energy, zero-crossing rate, spectral centroid, and their mean and variance parameters. Classification is then performed using machine learning algorithms such as logistic regression and support vector machines. The second method involves extracting time-frequency features based on short-time Fourier transform and inputting them into deep neural networks for classification. Commonly used neural networks include convolutional neural networks, long short-term memory neural networks, and Transformers. In all these studies, convolutional neural networks have achieved good classification performance, but they also have certain limitations. Summary of the Invention

[0004] In view of this, the main objective of the present invention is to provide a method and apparatus for detecting lung diseases using cough sounds, which can improve the ability of classification networks to extract differential features and location information, and make good use of the differential features of all individual cough sound segments in the cough audio for classification, thereby improving the accuracy of the detection results.

[0005] To achieve the above objectives, this application provides a method for detecting lung diseases using cough sounds, comprising:

[0006] Remove non-coughing sound segments from the original cough audio to obtain the cough audio, along with the start and end times of each individual cough segment in the cough audio;

[0007] Based on the start and end times of each individual cough segment, the log-Melp matrix corresponding to each individual cough segment is generated, and the positional encoding matrix corresponding to each individual cough segment is calculated.

[0008] The feature matrix of the cough audio is obtained based on the logarithmic Melp matrix and the positional coding matrix corresponding to each individual cough segment;

[0009] After normalizing the feature matrix of the cough audio, multiply it by a scaling factor, add the resulting product to the feature matrix of the cough audio, and input the sum into the classification network for classification.

[0010] In one possible implementation, the log-Melp matrix corresponding to the generation of a single cough fragment is:

[0011] The length of a single cough segment is the difference between its end time and start time. Starting from the start time of the single cough segment, the segment is divided into frames and windowed. The power spectrum is obtained by taking the square of the modulus after performing a short-time Fourier transform on each frame.

[0012] Multiplying the power spectrum by the characteristic dimension yields the Mel spectrum matrix, and taking the logarithm of the Mel spectrum matrix yields the log-Mel spectrum matrix.

[0013] In another possible implementation, the calculation of the positional encoding matrix corresponding to a single cough segment is as follows: The positional encoding of each feature in the log-Mel spectrum matrix corresponding to the single cough segment is calculated, as shown in the following formula:

[0014]

[0015]

[0016] Where pos represents the frame number of the feature; 2i represents the index of the feature dimension, i ranging from 0 to d / 2-1; d represents the feature dimension; PE(pos, 2i) represents the position encoding of the pos-th frame and the 2i-th dimension feature in the log-Mel spectrum matrix.

[0017] In another possible implementation, the feature matrix of the cough audio is obtained by: adding the logarithmic Melp matrix and the corresponding positional encoding matrix of each individual cough segment to output the feature matrix of each individual cough segment; and concatenating the feature matrices of each individual cough segment in the time dimension to obtain the feature matrix of the cough audio.

[0018] In another possible implementation, the feature matrix of the cough audio is obtained by concatenating the log-Melp matrices corresponding to each individual cough segment in the time dimension, and concatenating the positional encoding matrices corresponding to each individual cough segment in the time dimension; the two matrices are then added together to obtain the feature matrix of the cough audio.

[0019] On the other hand, this application also provides a device for detecting lung diseases using cough sounds, comprising: a cough sound detection unit, an audio signal processing unit, a splicing unit, a normalization unit, and a classification network unit; wherein,

[0020] The cough sound detection unit is used to remove non-cough sound segments from the original cough audio to obtain the cough audio, as well as the start and end times of each individual cough segment in the cough audio;

[0021] The audio signal processing unit is used to generate the logarithmic Melp matrix corresponding to each individual cough segment based on the start and end times of each individual cough segment; and to calculate the position encoding matrix corresponding to each individual cough segment.

[0022] The splicing unit is used to obtain the feature matrix of the cough audio based on the logarithmic Melp matrix and the positional coding matrix corresponding to each individual cough segment;

[0023] The normalization unit is used to normalize the feature matrix of the cough audio, multiply it by a scaling factor, add the product to the feature matrix of the cough audio, and input the sum into the classification network unit.

[0024] A classification network unit is used to classify the obtained data.

[0025] In one possible implementation, the audio signal processing unit, which generates the logarithmic Melp matrix corresponding to a single cough segment, is specifically used to divide the single cough segment into frames and window it, taking the difference between the end time and the start time of the single cough segment as the length of the single cough segment, taking the start time of the single cough segment as the starting point, and performing a short-time Fourier transform on each frame to obtain the power spectrum by taking the square of the modulus; and multiplying the power spectrum by the feature dimension to obtain the Mel spectrum matrix, and taking the logarithm of the Mel spectrum matrix to obtain the logarithmic Melp matrix.

[0026] In another possible implementation, the audio signal processing unit is specifically used to calculate the positional encoding matrix corresponding to a single cough segment, specifically for calculating the positional encoding of each feature in the log-Mel spectrum matrix corresponding to that single cough segment, as shown in the following formula:

[0027]

[0028]

[0029] Where pos represents the frame number of the feature; 2i represents the index of the feature dimension, i ranging from 0 to d / 2-1; d represents the feature dimension; PE(pos, 2i) represents the position encoding of the pos-th frame and the 2i-th dimension feature in the log-Mel spectrum matrix.

[0030] In another possible implementation, the splicing unit, which obtains the feature matrix of the cough audio, is specifically used to add the logarithmic Melp matrix corresponding to each individual cough segment and its corresponding positional encoding matrix to output the feature matrix of each individual cough segment; and to splice the feature matrices of each individual cough segment in the time dimension to obtain the feature matrix of the cough audio.

[0031] In another possible implementation, the feature matrix of the cough audio is obtained by the splicing unit, which splices the log-Melp matrices corresponding to each individual cough segment in the time dimension and splices the positional encoding matrices corresponding to each individual cough segment in the time dimension; the two spliced ​​matrices are then added together to obtain the feature matrix of the cough audio.

[0032] Based on the above, the method and apparatus for detecting lung diseases using cough sounds provided by the present invention have the following advantages and features:

[0033] 1. Position encoding is performed on individual cough segments, and the resulting position encoding matrix has a certain time shift invariance, thereby achieving time-domain time shift adaptation and improving the model's anti-interference ability;

[0034] 2. By encoding the position of individual cough segments, the classification network can learn the temporal position information of individual cough segments in the cough audio, and the features under the same frequency domain window still have the same position encoding, thereby enabling the classification network to have a stronger ability to distinguish differences in frequency domain features. Attached Figure Description

[0035] Figure 1 The logarithm of the cough audio containing five individual cough segments;

[0036] Figure 2 For pairs without added position codes Figure 1 The heatmap shown is obtained from the classification of cough audio.

[0037] Figure 3 To add positional encoding to the entire cough audio segment Figure 1 The heatmap shown is obtained from the classification of cough audio.

[0038] Figure 4 For the present invention to Figure 1 The heatmap shown is obtained from the classification of cough audio.

[0039] Figure 5 The logarithm of the cough audio containing two individual cough segments;

[0040] Figure 6A For pairs without added position codes Figure 5 The heatmap shown is obtained from the classification of cough audio.

[0041] Figure 6B For pairs without added location coding that experience time shifts Figure 5 The heatmap shown is obtained from the classification of cough audio.

[0042] Figure 7A To add positional encoding to the entire cough audio segment Figure 5 The heatmap shown is obtained from the classification of cough audio.

[0043] Figure 7B To add positional encoding to the entire cough audio segment, and to detect time shifts... Figure 5 The heatmap shown is obtained from the classification of cough audio.

[0044] Figure 8A For the present invention to Figure 5 The heatmap shown is obtained from the classification of cough audio.

[0045] Figure 8B For the present invention to address time shifts Figure 5 The heatmap shown is obtained from the classification of cough audio.

[0046] Figure 9 This is a schematic flowchart of a method for detecting lung diseases using cough sounds according to an embodiment of the present invention;

[0047] Figure 10 This is a schematic diagram illustrating the process of classifying cough audio containing two individual cough segments according to the present invention. Detailed Implementation

[0048] The time-frequency characteristics of cough audio recordings collected from different populations exhibit certain differences. When using these differences for classification to detect lung diseases, the network's ability to capture both these differences and location information affects the classification results. Although local filters and boundary extension methods in convolutional neural networks enable them to learn location information to some extent, their accuracy in capturing location information is not high when the data volume is small. To address this issue, existing solutions incorporate location encoding of the entire cough audio segment. However, this does not adequately solve the problem of inaccurate location information capture when the data volume is small. Consequently, the network may ignore the differences in certain individual cough sound segments and fail to effectively utilize the differences in all individual cough sound segments in the cough audio for classification, affecting the accuracy of the network's classification. This, in turn, leads to low accuracy in lung disease detection results, and even small changes to the original cough audio can cause significant deviations in the model's predictions.

[0049] This invention provides a method for detecting lung diseases using cough sounds. It involves positionally encoding each individual cough segment in a cough audio recording, and then inputting the positional encoding of each individual cough segment along with its time-frequency features into a classification network for classification. This improves the classification network's ability to extract differential features and positional information, thereby increasing the network's utilization of cough audio and improving classification accuracy. Furthermore, the positional encoding matrix obtained from individual cough segments exhibits time-shift invariance, meaning the correspondence between the time-frequency features of an individual cough segment and the positional encoding matrix does not change with the time shift of the cough audio recording. This achieves time-domain adaptiveness and enhances the model's robustness against interference.

[0050] Here, the single cough segment refers to an audio signal that carries a cough that can be distinguished by the human ear.

[0051] The above-mentioned existing lung disease detection methods and the application of the present invention have advantages over other methods. Figure 1 The experimental results of classifying cough audio as shown are as follows: Figures 2-4 As shown. Visualization using the Grad-cam method allows us to observe the contribution of each individual cough segment in the cough audio to the classification result through a heatmap; darker colors indicate greater contributions.

[0052] Figure 1 It is a logarithmic melopsal of a cough audio segment containing five individual cough fragments. Although the differential features contained in each individual cough fragment are not consistent, and their contribution is not consistent, each individual cough fragment should be utilized as much as possible during classification to obtain more accurate classification results.

[0053] Figure 2 The data shows the contribution of each of the five individual cough segments to the classification without location encoding. The heat value of the fourth cough is significantly lower than that of the other four coughs, indicating that the fourth individual cough segment was not utilized when classifying this cough audio, which may lead to some errors in the detection results.

[0054] Position encoding is performed on the entire cough audio segment to obtain Figure 3 As shown in the heatmap, the features of the fourth cough were utilized, but the contribution of the first and second coughs was significantly reduced, especially the second single cough sound segment, which contributed almost nothing to the classification.

[0055] After using the present invention to perform position encoding on the five cough segments, the following was obtained: Figure 4 The heatmap shown clearly demonstrates that each cough segment contributes significantly to the classification, indicating that by using the location encoding of individual cough segments in this invention, each individual cough segment is utilized, thus leading to more accurate classification results.

[0056] Existing lung disease detection methods and the application of this invention Figure 5 The experimental results for time-shifted cough audio are shown in comparisons 6A to 8B. Similarly, visualization is performed using the Grad-cam method, with heatmaps used to observe the contribution of each individual cough segment in the cough audio to the classification results; darker colors indicate greater contributions.

[0057] Figure 5 It is a log-Melp of a cough audio segment containing two individual cough segments C1 and C2. A robust model is expected to be developed where small changes in the cough audio do not significantly bias the model's predictions.

[0058] Figure 6A Figure 6A shows the contribution of two individual cough segments to classification without positional encoding. Figure 6B shows the contribution of two individual cough segments to classification after time shift without positional encoding. By comparison, it can be seen that the heatmap distribution of the two individual cough segments C1 and C2 in Figures 6A and 6B has changed significantly, that is, the change in cough audio causes a large deviation in the model detection results.

[0059] Figure 7A , 7B A heatmap obtained by positional encoding of the entire cough audio segment was added, and the comparison is shown below. Figure 7A , 7B The heatmap distribution of the two individual cough segments, C1 and C2, still shows significant changes, meaning that the changes in the cough audio cause a large deviation in the model's detection results.

[0060] Figure 8A , 8B The heatmaps obtained using this invention show that the distribution patterns of the two individual cough segments C1 and C2 are roughly consistent, indicating that using this invention to encode the location of individual cough segments enhances the anti-interference capability and yields more consistent detection results.

[0061] Specifically, the procedure of a method for detecting lung diseases using cough sounds according to an embodiment of the present invention is as follows: Figure 9 As shown, steps 901 to 904 are included.

[0062] Step 901: Remove non-coughing sound segments from the original cough audio to obtain the cough audio, and the start and end times of each individual cough segment in the cough audio.

[0063] Step 902: Based on the start and end times of each individual cough segment, generate the log-Melp matrix corresponding to each individual cough segment, and calculate the position encoding matrix corresponding to each individual cough segment.

[0064] Step 903: Based on the logarithmic Melp matrix and positional encoding matrix corresponding to each individual cough segment, obtain the feature matrix of the cough audio;

[0065] Step 904: Normalize the feature matrix of the cough audio, multiply it by a scaling factor, add the resulting product to the feature matrix of the cough audio, and input the sum into the classification network to classify the cough audio to identify whether it comes from a patient with a certain type of lung disease or a healthy person.

[0066] Here, step 901 can be achieved using the following method:

[0067] Step 9011: Segment the original cough audio into frames;

[0068] Step 9012: Extract 13-dimensional Mel frequency cepstral coefficients (MFCC) and their first-order and second-order difference features from each frame of audio to form a 39-dimensional feature vector.

[0069] Step 9013: Input the feature vectors of all frames into a neural network model consisting of two layers of unidirectional Long Short-Term Memory (LSTM) network and one fully connected layer. The LSM network has 128 hidden layer nodes, the fully connected layer has 128-dimensional input and 2-dimensional output, and the probability of each frame being a cough sound or not is obtained by using logsoftmax. The logsoftmax calculation method is as follows:

[0070]

[0071] Where, x i This is the output in 2D, where i = 0, 1;

[0072] Step 9014: Smooth the data through a smoothing window to obtain continuous cough frames and non-cough frames. Extract the continuous cough frames to form each cough. The start and end times are also calculated during this process.

[0073] In one possible implementation, the specific steps in step 902 for generating the log-Melp matrix corresponding to a single cough fragment are as follows:

[0074] The length of a single cough segment is the difference between its end time and start time. Starting from the start time of the single cough segment, the segment is divided into frames and windowed. The power spectrum is obtained by taking the square of the modulus after performing a short-time Fourier transform on each frame.

[0075] Multiplying the power spectrum by the characteristic dimension yields the Mel spectrum matrix, and taking the logarithm of the Mel spectrum matrix yields the log-Mel spectrum matrix.

[0076] In step 902, the positional encoding matrix corresponding to a single cough segment is calculated as follows: The positional encoding of each feature in the log-Mel spectrum matrix corresponding to the single cough segment is calculated respectively, using the following formula:

[0077]

[0078]

[0079] Where pos represents the frame number of the feature; 2i represents the index of the feature dimension, i ranging from 0 to d / 2-1; d represents the feature dimension; PE(pos, 2i) represents the position encoding of the pos-th frame and the 2i-th dimension feature in the log-Mel spectrum matrix.

[0080] Here, the frame length for framing operations typically ranges from 25 to 64 ms, and the frame shift typically ranges from 90 to 32 ms. The feature dimension typically ranges from 32 to 256.

[0081] The adjacent frames within a single cough segment exhibit certain temporal correlations. Therefore, the position code at any time interval k in the time domain can be represented by the position code at the current moment according to the following formula.

[0082] PE (pos+k，2i) =PE (pos，2i) *PE (k，2i+1) +PE (pos，2i+1) *PE (k，2i)

[0083] PE (pos+k，2i+1) =PE (pos，2i+1) *PE (k，2i+1) -PE (pos，2i) *PE (k，2i)

[0084] Thus, the positional encoding of individual cough segments provided by this invention enables the classification network to learn the temporal proximity information of individual cough segments in the cough audio.

[0085] Furthermore, the location encoding of a single cough segment provided by this invention ensures that features within the same frequency domain window still have the same location encoding even if time shift occurs, thus enabling the classification network to have a stronger ability to distinguish differences in the frequency domain.

[0086] In one possible implementation, the feature matrix of the cough audio obtained in step 903 is obtained by: adding the logarithmic Melp matrix and the corresponding positional encoding matrix of each individual cough segment to output the feature matrix of each individual cough segment; and concatenating the feature matrices of each individual cough segment in the time dimension to obtain the feature matrix of the cough audio.

[0087] In another possible implementation, the feature matrix of the cough audio obtained in step 903 is obtained by concatenating the log-Melp matrices corresponding to each individual cough segment in the time dimension, and concatenating the positional encoding matrices corresponding to each individual cough segment in the time dimension; and adding the two matrices obtained after concatenation to obtain the feature matrix of the cough audio.

[0088] In another possible implementation, the feature matrix of the cough audio is normalized in step 904 because of the diversity of the original cough audio recording devices. We use instance normalization to mitigate the impact of device differences. Since device differences are mainly reflected in the frequency domain, we calculate the mean and variance within each frequency window.

[0089]

[0090]

[0091] Where, x∈R B*F*T , indicating that the input is a tensor of an F*T time-frequency feature map containing B samples, μ bf σ bf Let f represent the mean and variance of the b-th sample within the frequency window f.

[0092] Considering that instance normalization normalizes each sample individually, potentially losing information helpful for classification, we introduce skip connections to multiply the normalized features by a scaling factor λ and add it to the original features before inputting it into the subsequent classification network. The calculation method is shown below.

[0093]

[0094] Here, ε is used to prevent small values ​​where the denominator is 0.

[0095] Here, as Figure 10 As shown, taking a cough audio clip containing two individual cough segments as an example, the invention is presented more intuitively through formulas. This invention improves the classification network's ability to extract differential features and location information, thereby increasing the utilization rate of the classification network for cough audio clips, improving classification accuracy, and achieving time-domain time-shift adaptation, thus enhancing the model's anti-interference capabilities.

[0096] Assume that after processing in step 901, the start time points of the two individual cough segments obtained are s1 and s2, and the end time points are e1 and e2, respectively.

[0097] After step 902, the two individual cough segments are framed according to the frame length window_length and frame shift hop_length respectively. The frame numbers F1 and F2 of the two individual cough segments are then:

[0098]

[0099]

[0100] When generating the log-Mel spectrum matrix, if the number of frequency windows is d, i.e. the feature dimension is d, then the sizes of the log-Mel spectrum matrices M1 and M2 corresponding to the two individual cough segments are F1*d and F2*d, respectively.

[0101] The location code for each individual cough segment is calculated using the following formula:

[0102]

[0103]

[0104] Where pos = 1, 2, ..., F represents the frame number, i = 0, 1, ... (d / 2-1), and 2i and 2i+1 represent the feature dimension numbers;

[0105] The resulting two single cough fragments correspond to positional encoding matrices PE1 and PE2, with sizes F1*d and F2*d, respectively.

[0106] In step 903, the log-Mel spectrum matrix and positional encoding matrix corresponding to each individual cough segment are added together to obtain the outputs out1 and out2:

[0107] out1 = M1 + PE1

[0108] out2 = M2 + PE2

[0109] The feature matrix of size (F1+F2)*d formed by concatenating out1 and out2 in the time dimension is: out(e1-s1)+out(e2-s2)=M(e1-s1)+PE(e1-s1)+M(e2-s2)+PE(e2-s2).

[0110] Step 904 normalizes the feature matrix out(e1-s1) + out(e2-s2) of the cough audio, multiplies it by a scaling factor, adds the resulting product to the feature matrix of the cough audio, and inputs the sum into the classification network for classification. In other words, the time-frequency features M(e1-s1) + M(e2-s2) of two individual cough segments are input together with the location encoding PE(e1-s1) + PE(e2-s2) into the classification network, improving the network's ability to extract differential features and location information from the cough audio.

[0111] Furthermore, through the above calculation method, if the original cough audio is time-shifted, PE(e1-s1) and PE(e2-s2) remain consistent with the time-shifted version. That is, the position encoding matrix corresponding to each individual cough segment will not change due to the time shift of the original cough audio. Therefore, out(e1-s1) and out(e2-s2) do not change, thus ensuring that the detection results of lung diseases will not have a large deviation.

[0112] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A device for detecting lung diseases using cough sounds, characterized in that, include: The system includes a cough sound detection unit, an audio signal processing unit, a splicing unit, a normalization unit, and a classification network unit; among which, The cough sound detection unit is used to remove non-cough sound segments from the original cough audio to obtain the cough audio, as well as the start and end times of each individual cough segment in the cough audio; The audio signal processing unit is used to generate the logarithmic Melp matrix corresponding to each individual cough segment based on the start and end times of each individual cough segment; and to calculate the position encoding matrix corresponding to each individual cough segment. The splicing unit is used to obtain the feature matrix of the cough audio based on the logarithmic Melp matrix and the positional coding matrix corresponding to each individual cough segment; The normalization unit is used to normalize the feature matrix of the cough audio, multiply it by a scaling factor, add the product to the feature matrix of the cough audio, and input the sum into the classification network unit. A classification network unit is used to classify the obtained data.

2. The apparatus according to claim 1, characterized in that, The audio signal processing unit generates a log-Melp matrix corresponding to a single cough segment. Specifically, it uses the difference between the end time and the start time of the single cough segment as the length of the single cough segment, takes the start time of the single cough segment as the starting point, performs frame windowing on the single cough segment, performs a short-time Fourier transform on each frame, and calculates the square of the modulus to obtain the power spectrum. It then multiplies the power spectrum by the feature dimension to obtain the Mel spectrum matrix, and takes the logarithm of the Mel spectrum matrix to obtain the log-Melp matrix.

3. The apparatus according to claim 2, characterized in that, The audio signal processing unit calculates the positional encoding matrix corresponding to a single cough segment. Specifically, it calculates the positional encoding of each feature in the log-Mel spectrum matrix corresponding to that single cough segment, using the following formula: Where pos represents the frame number of the feature; 2i represents the index of the feature dimension, i ranging from 0 to d / 2-1; d represents the feature dimension; PE(pos, 2i) represents the position encoding of the pos-th frame and the 2i-th dimension feature in the log-Mel spectrum matrix.

4. The apparatus according to claim 1, 2 or 3, characterized in that, The feature matrix of the cough audio is obtained by the splicing unit, which is specifically used to add the logarithmic Melp matrix and the corresponding positional encoding matrix of each individual cough segment to output the feature matrix of each individual cough segment; and to splice the feature matrices of each individual cough segment in the time dimension to obtain the feature matrix of the cough audio.

5. The apparatus according to claim 1, 2 or 3, characterized in that, The feature matrix of the cough audio is obtained by the splicing unit, which splices the logarithmic Melp matrix corresponding to each individual cough segment in the time dimension, and splices the position encoding matrix corresponding to each individual cough segment in the time dimension; the two spliced ​​matrices are added together to obtain the feature matrix of the cough audio.

Images