A method and system for classifying respiratory sounds based on mel-spectrogram

Abstract

The present application relates to the field of respiratory sound recognition, and specifically discloses a method and system for respiratory sound classification based on Mel-spectrogram, which utilizes a respiratory sound classification model with a novel network structure to improve the sensitivity and specificity of respiratory sound classification. In particular, the respiratory sound classification model with a novel network structure includes an audio spectrum graph converter network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network, wherein the multi-scale feature extraction network and the temporal feature extraction network form a feature encoding and decoding architecture to perform global and local feature reinforcement learning on the respiratory sound spectrum features output by the audio spectrum graph converter network, thereby improving the sensitivity and specificity of respiratory sound classification.

Description

Technical Field

[0001] The present application relates to the field of respiratory sound recognition, and more specifically, to a respiratory sound classification method and system based on Mel-spectrogram. Background Art

[0002] Chronic respiratory diseases are a major public health issue and a major cause of disease and death worldwide. Traditionally, doctors diagnose chronic respiratory diseases by listening to the lungs and identifying breath sounds based on experience, but the test results under this method are very inaccurate and are affected by factors such as the environment and the doctor's subjectivity.

[0003] Breath sounds can be divided into normal breath sounds and abnormal breath sounds. Common abnormal breath sounds include wheezing, dry rales, stridor, and moist rales. In terms of frequency characteristics, the frequency response range of normal breath sounds is generally 50-3000Hz, and most of the energy is concentrated in 100-1000Hz; while the frequency response range of abnormal breath sounds is generally 50-1200Hz. Stridor and coarse and moist rales are low-frequency abnormal breath sounds, and wheezing, dry rales, and fine and moist rales are high-frequency abnormal breath sounds. In terms of musical characteristics, normal breath sounds have a stable rhythm and frequency, which is consistent with the breathing process; wheezing, dry rales, and stridor are continuous breath sounds with a continuous rhythm; coarse and moist rales and fine and moist rales are discontinuous breath sounds without a complete rhythm. In terms of time distribution characteristics, normal breath sounds and abnormal breath sounds are distributed in different stages of the respiratory cycle. The respiratory cycle refers to the time from the start of one inhalation to the start of the next inhalation. A respiratory cycle includes an inhalation phase and an exhalation phase. Abnormal breath sounds may appear in the inhalation phase or exhalation phase of the respiratory cycle, or they may appear in both the inhalation phase and the exhalation phase, as a sound segment superimposed on the normal breath sounds. In common breath sound audio data sets, each breath sound audio contains at least one breath cycle, one breath cycle in normal breath sounds is a breath sound segment, and the abnormal breath sound audio contains at least one abnormal breath sound segment. In the breath sound audio containing multiple abnormal breath sound segments, there is a certain time interval between the multiple abnormal breath sound segments, showing the characteristics of discrete distribution. Taking the ICBHI breath sound database (International Conference on Biomedical and Health Informatics 2017, ICBHI2017) as an example, it includes 920 breath sounds of 126 patients, including 49 children, with an average length of 21.5s for each breath sound, and 6898 breath cycles, 1898 wheezing sound segments and 8877 wet rales segments are obtained. In the entire breath sound audio data set, abnormal breath sounds account for a smaller proportion than normal breath sounds. All of the above features increase the difficulty of abnormal breath sound classification and recognition tasks.

[0004] Mel-spectrogram is a method of converting audio signals into visual representations, and is often used to extract features of audio signals. By decomposing the audio signal into different frequency components and showing the changes of these frequency components in the entire signal, the Mel-spectrogram can capture the details of the audio signal in frequency and time. The Mel-spectrogram is based on the auditory perception characteristics of the human ear and can better simulate the way the human ear perceives sound. Therefore, after converting the breathing audio into a Mel-spectrogram, the frequency characteristics and musical characteristics in the audio signal can be more accurately captured and analyzed, thereby improving the accuracy of audio analysis. This is of great significance for the diagnosis and identification of respiratory sounds, especially in medical diagnosis, and can more accurately determine whether the respiratory sounds are normal or abnormal.

[0005] The current mainstream technology is to use convolutional neural networks to perform intelligent and automatic classification of respiratory sounds, and use sensitivity and specificity to judge the classification effect of respiratory sounds. Sensitivity refers to the ability to correctly identify abnormal respiratory sound samples, and specificity refers to the proportion of correctly identified normal respiratory sound samples in all actual normal respiratory sound samples. Since abnormal respiratory sounds vary greatly from patient to patient, abnormal respiratory sounds of the same category collected from different patients may have large differences in frequency characteristics, musical characteristics, and time distribution characteristics. Therefore, it is necessary to learn the global features of these abnormal respiratory sounds to improve the recognition ability of neural networks for abnormal respiratory sounds. It is difficult for convolutional neural networks to learn these variably global features, so the sensitivity of abnormal respiratory sounds is not high. The audio spectrogram transformer network using the transformer architecture has better learning ability for global features, but because the downsampling output of audio features will lose feature information, its sensitivity and specificity indicators are not significantly improved in the task of abnormal respiratory sound recognition and classification.

[0006] Therefore, we expect a neural network classification model that can take into account both the global and local feature information of respiratory sounds to improve the ability to recognize abnormal respiratory sounds. Summary of the invention

[0007] In order to solve the above technical problems, the present application is proposed. An embodiment of the present application provides a method and system for respiratory sound classification based on Mel-spectrogram, which utilizes a respiratory sound classification model with a novel network structure to improve the sensitivity and specificity of respiratory sound classification. In particular, the respiratory sound classification model with a novel network structure includes an audio spectrum graph converter network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network, wherein the multi-scale feature extraction network and the temporal feature extraction network form a feature encoding and decoding architecture to perform global and local feature reinforcement learning on the respiratory sound spectrum features output by the audio spectrum graph converter network, thereby improving the sensitivity and specificity of respiratory sound classification.

[0008] In order to achieve the above technical effects, the technical solution of this application is as follows:

[0009] A respiratory sound classification method based on mel-spectrogram, comprising: a training phase and an inference phase;

[0010] The training phase includes:

[0011] Acquire a respiratory sound training data set, wherein each respiratory sound training data in the respiratory sound training data set is a respiratory sound audio containing a true category label of the respiratory sound;

[0012] Extracting Mel-spectrogram features from the breathing sound audio of each breathing sound training data to obtain a set of breathing sound Mel-spectrogram graphs;

[0013] Performing data enhancement on the set of respiratory sound mel-spectrograms to obtain a set of enhanced respiratory sound mel-spectrograms;

[0014] Training a respiratory sound classification model based on the set of enhanced respiratory sound mel-spectrograms to obtain a trained respiratory sound classification model, wherein the respiratory sound classification model includes an audio spectrogram transformer network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network;

[0015] The inference stage includes:

[0016] Obtain the audio of the breathing sound to be identified;

[0017] The respiratory sound audio to be recognized is input into the trained respiratory sound classification model to obtain a respiratory sound recognition result.

[0018] In the respiratory sound classification method based on Mel-spectrogram, data enhancement is performed on the set of respiratory sound Mel-spectrograms to obtain a set of enhanced respiratory sound Mel-spectrograms, including at least one of the following steps: zero-filling the respiratory sound Mel-spectrograms; warping the respiratory sound Mel-spectrograms in the time dimension; masking the frequency channel of the respiratory sound Mel-spectrograms; and masking the time channel of the respiratory sound Mel-spectrograms.

[0019] In the respiratory sound classification method based on Mel-spectrogram, the respiratory sound classification model is trained based on the set of enhanced respiratory sound Mel-spectrograms to obtain a trained respiratory sound classification model, including: performing spectrum block processing on the enhanced respiratory sound Mel-spectrogram and inputting it into the audio spectrogram converter network to obtain a global deep semantic coding feature map of the respiratory sound Mel-spectrogram; embedding coding the true category label of the respiratory sound to obtain a true category label coding matrix of the respiratory sound; connecting the global deep semantic coding feature map of the respiratory sound Mel-spectrogram and the true category label coding matrix of the respiratory sound to obtain a feature map containing categories. The global deep semantic coding feature map of the Mel-spectrogram of respiratory sound containing label information; the global deep semantic coding feature map of the Mel-spectrogram of respiratory sound containing category label information is input into the multi-scale feature extraction network to obtain the global deep down-sampling semantic coding feature matrix of the re-encoded Mel-spectrogram of respiratory sound; the global deep down-sampling semantic coding feature matrix of the re-encoded Mel-spectrogram of respiratory sound is input into the temporal feature extraction network to obtain the global deep up-sampling semantic coding feature matrix of the Mel-spectrogram of respiratory sound; the global deep up-sampling semantic coding feature matrix of the Mel-spectrogram of respiratory sound is input into the linear output layer network to obtain the predicted category label of respiratory sound.

[0020] In the respiratory sound classification method based on Mel-spectrogram, the multi-scale feature extraction network includes a multi-head multi-scale self-attention layer, a first normalization layer, a multi-layer perceptron layer and a second normalization layer; the temporal feature extraction network includes a linear interpolation layer and a bidirectional LSTM model; the linear output layer network includes a fully connected layer and a classification layer.

[0021] In the respiratory sound classification method based on Mel-spectrogram, the enhanced respiratory sound Mel-spectrogram is subjected to spectral block processing and then input into the audio spectrogram converter network to obtain a global deep semantic coding feature map of the respiratory sound Mel-spectrogram, including: performing spectral block processing on the enhanced respiratory sound Mel-spectrogram to obtain a sequence of respiratory sound Mel-spectrogram blocks; performing linear projection and position encoding on each respiratory sound Mel-spectrogram block in the sequence of respiratory sound Mel-spectrogram blocks to obtain a sequence of respiratory sound Mel-spectrogram block embedding coding vectors; inputting the sequence of respiratory sound Mel-spectrogram block embedding coding vectors into a multi-head self-attention structure to obtain a sequence of respiratory sound Mel-spectrogram block global context semantic coding feature vectors; and performing feature reshaping on the sequence of respiratory sound Mel-spectrogram block global context semantic coding feature vectors to obtain the respiratory sound Mel-spectrogram global deep semantic coding feature map.

[0022] In the respiratory sound classification method based on Mel-spectrogram, the global deep semantic coding feature map of the respiratory sound Mel-spectrogram containing category label information is input into the multi-scale feature extraction network to obtain a re-encoded respiratory sound Mel-spectrogram global deep down-sampling semantic coding feature matrix, including: reshaping the feature shape of the global deep semantic coding feature map of the respiratory sound Mel-spectrogram containing category label information to obtain a sequence of respiratory sound Mel-spectrogram block semantic coding vectors that integrate category information; inputting the sequence of respiratory sound Mel-spectrogram block semantic coding vectors that integrate category information into the multi-head multi-scale self-attention layer to obtain a sequence of respiratory sound Mel-spectrogram block context semantic reconstruction coding feature vectors; and reshaping the respiratory sound Mel-spectrogram block context semantic reconstruction coding feature vectors. The sequence of quantities is input into the first normalization layer to obtain a sequence of normalized respiratory sound Mel-spectrogram block context semantic reconstruction coding feature vectors; the sequence of normalized respiratory sound Mel-spectrogram block context semantic reconstruction coding feature vectors is input into the multi-layer perceptron layer to obtain a sequence of respiratory sound Mel-spectrogram block context semantic deep reconstruction coding feature vectors; the sequence of respiratory sound Mel-spectrogram block context semantic deep reconstruction coding feature vectors is input into the second normalization layer to obtain a sequence of normalized respiratory sound Mel-spectrogram block context semantic deep reconstruction coding feature vectors; the sequence of normalized respiratory sound Mel-spectrogram block context semantic deep reconstruction coding feature vectors is reshaped to obtain a re-encoded respiratory sound Mel-spectrogram global deep downsampling semantic coding feature matrix.

[0023] In the respiratory sound classification method based on Mel-spectrogram, the global deep down-sampling semantic coding feature matrix of the re-encoded respiratory sound Mel-spectrogram is input into a temporal feature extraction network to obtain the global deep up-sampling semantic coding feature matrix of the respiratory sound Mel-spectrogram, including: expanding the global deep down-sampling semantic coding feature matrix of the re-encoded respiratory sound Mel-spectrogram into a sequence of down-sampling context semantic coding feature vectors of re-encoded respiratory sound Mel-spectrogram tiles; inputting the sequence of global deep down-sampling semantic coding feature vectors of the re-encoded respiratory sound Mel-spectrogram tiles into a linear interpolation layer to obtain a sequence of linear interpolation context semantic coding feature vectors of re-encoded respiratory sound Mel-spectrogram tiles; inputting the sequence of linear interpolation context semantic coding feature vectors of the re-encoded respiratory sound Mel-spectrogram tiles into a bidirectional LSTM model to obtain a sequence of local reinforcement semantic coding feature vectors of the context of the respiratory sound Mel-spectrogram; and arranging the sequence of local reinforcement semantic coding feature vectors of the context of the respiratory sound Mel-spectrogram into the global deep up-sampling semantic coding feature matrix of the respiratory sound Mel-spectrogram.

[0024] In the respiratory sound classification method based on Mel-spectrogram, the global deep upsampling semantic coding feature matrix of the respiratory sound Mel-spectrogram is input into the linear output layer network to obtain the respiratory sound prediction category label, including: expanding the global deep upsampling semantic coding feature matrix of the respiratory sound Mel-spectrogram into a global deep upsampling semantic coding feature vector of the respiratory sound Mel-spectrogram; inputting the global deep upsampling semantic coding feature vector of the respiratory sound Mel-spectrogram into the fully connected layer to obtain a dimensionally modulated respiratory sound Mel-spectrogram semantic coding vector, the length of the dimensionally modulated respiratory sound Mel-spectrogram semantic coding vector is 7; inputting the dimensionally modulated respiratory sound Mel-spectrogram semantic coding vector into a classifier based on the Softmax function to obtain the respiratory sound prediction category label.

[0025] In the respiratory sound classification method based on Mel-spectrogram, the respiratory sound classification model is trained based on the set of enhanced respiratory sound Mel-spectrograms to obtain a trained respiratory sound classification model, and also includes the steps of: calculating the cross-entropy loss function value between the true category label of the respiratory sound and the predicted category label of the respiratory sound; calculating the contrast loss function value between the sequence of embedded coding vectors of the respiratory sound Mel-spectrogram blocks and the sequence of global context semantic coding feature vectors of the respiratory sound Mel-spectrogram blocks; and training the respiratory sound classification model with the sum of the contrast loss function value and the cross-entropy loss function value as the loss function value to obtain the trained respiratory sound classification model.

[0026] A respiratory sound classification system based on mel-spectrogram, which can implement the respiratory sound classification method based on mel-spectrogram, comprises:

[0027] Wherein, the training module includes:

[0028] A training data acquisition unit, used to acquire a respiratory sound training data set, wherein each respiratory sound training data in the respiratory sound training data set is a respiratory sound audio containing a true category label of the respiratory sound;

[0029] A mel-spectrogram feature extraction unit, configured to extract mel-spectrogram features from the breathing sound audio of each breathing sound training data to obtain a set of breathing sound mel-spectrograms;

[0030] A feature enhancement unit, configured to perform data enhancement on the set of respiratory sound mel-spectrograms to obtain a set of enhanced respiratory sound mel-spectrograms;

[0031] A training unit, used for training the respiratory sound classification model based on the set of enhanced respiratory sound mel-spectrograms to obtain a trained respiratory sound classification model;

[0032] Wherein, the inference module includes:

[0033] A breathing sound acquisition unit to be identified, used to acquire the audio of the breathing sound to be identified;

[0034] The recognition unit is used to input the respiratory sound audio to be recognized into the trained respiratory sound classification model to obtain a respiratory sound recognition result.

[0035] Compared with the prior art, the respiratory sound classification method and system based on Mel-spectrogram provided by the present application utilizes a respiratory sound classification model with a novel network structure to improve the sensitivity and specificity of respiratory sound classification. In particular, the respiratory sound classification model with a novel network structure includes an audio spectrum graph converter network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network, wherein the multi-scale feature extraction network and the temporal feature extraction network form a feature encoding and decoding architecture to perform global and local feature reinforcement learning on the respiratory sound spectrum features output by the audio spectrum graph converter network, thereby improving the sensitivity and specificity of respiratory sound classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0037] Figure 1 is a flow chart of a method for classifying respiratory sounds based on a mel-spectrogram according to an embodiment of the present application;

[0038] Figure 2 is a flow chart of the training phase of the respiratory sound classification method based on the mel-spectrogram according to an embodiment of the present application;

[0039] Figure 3 is a general structural framework diagram of the respiratory sound classification model of the embodiment of the present application;

[0040] Figure 4 is a specific structural block diagram of a multi-scale feature extraction network in an embodiment of the present application;

[0041] Figure 5 It is a specific structural block diagram of the temporal feature extraction network of an embodiment of the present application;

[0042] Figure 6 It is a block diagram of a respiratory sound classification system based on a mel-spectrogram according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0044] Example 1

[0045] In one embodiment of the present application, the respiratory sound classification method based on Mel-spectrogram utilizes a respiratory sound classification model with a novel network structure to improve the sensitivity and specificity of respiratory sound classification. In particular, the respiratory sound classification model with a novel network structure includes an audio spectrogram converter network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network, wherein the multi-scale feature extraction network and the temporal feature extraction network form a feature encoding and decoding architecture to perform global and local feature reinforcement learning on the respiratory sound spectrum features output by the audio spectrogram converter network, thereby improving the sensitivity and specificity of respiratory sound classification.

[0046] Specifically, Figure 1 As shown, in the technical solution of the present application, the respiratory sound classification method based on the Mel-spectrogram includes a training phase and an inference phase, wherein the purpose of the training phase is to use the training data to train the respiratory sound classification model with the novel network structure, so that in the inference phase, the trained respiratory sound classification model can be used to classify the respiratory sound audio to be identified to obtain the respiratory sound recognition result.

[0047] The specific process is as follows Figure 1 and Figure 2 As shown: the training stage includes: S110: obtaining a respiratory sound training data set, wherein each respiratory sound training data in the respiratory sound training data set is a respiratory sound audio containing a true category label of the respiratory sound. S120: extracting Mel spectrum features from the respiratory sound audio of each respiratory sound training data to obtain a set of respiratory sound Mel spectrum graphs; S130: performing data enhancement on the set of respiratory sound Mel spectrum graphs to obtain a set of enhanced respiratory sound Mel spectrum graphs; S140: training a respiratory sound classification model based on the set of enhanced respiratory sound Mel spectrum graphs to obtain a trained respiratory sound classification model, wherein the respiratory sound classification model includes an audio spectrum graph transformer network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network.

[0048] In the step S110, a respiratory sound training data set is obtained, and each respiratory sound training data in the respiratory sound training data set is a respiratory sound audio containing a true category label of the respiratory sound. In a specific example, the respiratory sound SPRSound public data set is used as the respiratory sound training data set, wherein the respiratory sound SPRSound public data set includes seven types of respiratory sounds: normal respiratory sounds, wheezing, dry rales, stridor, coarse wet rales, fine wet rales, and wheezing mixed with wet rales. That is, in this specific example, the respiratory sound true category labels include normal respiratory sounds, wheezing, dry rales, stridor, coarse wet rales, fine wet rales, and wheezing mixed with wet rales. More specifically, the respiratory sound training data set contains 9089 respiratory events (that is, 9089 respiratory sound training samples), including 6887 normal respiratory sound events, 53 dry rales events, 865 wheezing events, 17 stridor events, 66 coarse wet rales events, 1167 fine wet rales events, and 34 wheezing mixed wet rales events from 292 pediatric patients under the age of 16, and 2683 labeled respiratory sound records. The recordings were collected using heterogeneous devices, and each segment of annotated respiratory sound audio was 6s to 15s in length.

[0049] In step S120, mel-spectrogram features are extracted from the breathing sound audio of each breathing sound training data to obtain a set of breathing sound mel-spectrograms. In a specific example, step S120 first includes preprocessing the breathing sound training data set, and the preprocessing process includes resampling and filtering.

[0050] Specifically, the preprocessing process includes uniformly resampling each respiratory sound training data in the respiratory sound training data set to obtain a resampled respiratory sound training data set, and filtering each training sample data in the resampled respiratory sound training data set to obtain a preprocessed respiratory sound training data set.

[0051] In a specific example, each respiratory sound training data in the respiratory sound training data set is resampled at 16KHz, and the passband frequency is set to 40-850Hz, and then the resampled audio signal is filtered using a fourth-order Butterworth bandpass filter.

[0052] Furthermore, the step S120 further includes performing short-time Fourier transform on each respiratory sound training data in the preprocessed respiratory sound training data set, and then performing Mel spectrum operation to convert the respiratory sound audio into a respiratory audio spectrogram to obtain a set of respiratory sound Mel spectrum graphs.

[0053] In a specific example, the process is: Considered to be of length The pixel values ​​of the short-time Fourier transform (STFT) spectrum graph of the filtered audio signal are the center frequency and time frame, then the spectrum The calculation is as follows:

[0054]

[0055] in, Represents the Hamming window function.

[0056] To generate the Mel-spectrogram, a Mel filter bank is applied, which uses frequency distortion to simulate the human auditory system. and f The corresponding relationship is:

[0057]

[0058] The Mel spectrum can be expressed as:

[0059]

[0060] In the step S130, data enhancement is performed on the set of the respiratory sound Mel-spectrograms to obtain a set of enhanced respiratory sound Mel-spectrograms. In the technical solution of the present application, data enhancement is performed on the set of the respiratory sound Mel-spectrograms to obtain a set of enhanced respiratory sound Mel-spectrograms, including at least one of the following steps: zero padding the respiratory sound Mel-spectrograms; time dimension warping the respiratory sound Mel-spectrograms; frequency channel masking the respiratory sound Mel-spectrograms;

[0061] And, the time channel masking is performed on the respiratory sound mel-spectrogram.

[0062] Specifically, zero padding of the respiratory sound Mel spectrogram is performed to unify the scale, that is, to adjust the size of each respiratory sound Mel spectrogram in the set of respiratory sound Mel spectrograms so that they have the same size and are suitable for subsequent model input. The time dimension distortion is to randomly translate the spectrogram in a time interval to the left or right along the time dimension on the time dimension of the Mel spectrogram. The present invention determines the number of randomly translated time intervals according to the total number of respiratory events in the spectrogram, thereby increasing the richness of information in the time dimension of the respiratory spectrogram. The frequency channel masking is to randomly mask the frequency interval on the frequency channel of the Mel spectrogram. The present invention only masks the frequency range of abnormal respiratory sounds according to the abnormal respiratory sound frequency characteristics, thereby enhancing the model's learning ability for abnormal respiratory sound characteristics. The time channel masking is to randomly mask the time interval on the time channel of the Mel spectrogram. Since random time channel masking may cause loss of respiratory event information, the present invention designs a random intensity masking method to mask the time interval. For the masked interval, the mean distribution of (0,1) is used to determine the masking intensity, thereby enhancing the model's learning ability for respiratory sound characteristics.

[0063] In step S140, a respiratory sound classification model is trained based on the set of enhanced respiratory sound mel-spectrograms to obtain a trained respiratory sound classification model, wherein the respiratory sound classification model includes an audio spectrogram transformer network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network. Figure 3 As shown, step S140 includes the following steps: S141, performing spectrum block processing on the enhanced respiratory sound Mel-spectrogram and inputting it into the audio spectrum converter network to obtain a global deep semantic coding feature map of the respiratory sound Mel-spectrogram; S142, embedding coding the true category label of the respiratory sound to obtain a true category label coding matrix of the respiratory sound; S143, connecting the global deep semantic coding feature map of the respiratory sound Mel-spectrogram and the true category label coding matrix of the respiratory sound to obtain a global deep semantic coding feature map of the respiratory sound Mel-spectrogram containing category label information; S1 44, input the global deep semantic coding feature map of the respiratory sound Mel-spectrogram containing category label information into the multi-scale feature extraction network to obtain the global deep down-sampling semantic coding feature matrix of the re-encoded respiratory sound Mel-spectrogram; S145, input the global deep down-sampling semantic coding feature matrix of the re-encoded respiratory sound Mel-spectrogram into the temporal feature extraction network to obtain the global deep up-sampling semantic coding feature matrix of the respiratory sound Mel-spectrogram; S146, input the global deep up-sampling semantic coding feature matrix of the respiratory sound Mel-spectrogram into the linear output layer network to obtain the predicted category label of the respiratory sound.

[0064] Specifically, in the data flow of step S141, the enhanced respiratory sound Mel-spectrogram is first subjected to a spectrum block processing to obtain a sequence of respiratory sound Mel-spectrogram blocks. For example, in a specific example, the enhanced respiratory sound Mel-spectrogram is divided into patch features of size 16×16 and step size 10×10; then, each respiratory sound Mel-spectrogram block in the sequence of respiratory sound Mel-spectrogram blocks is subjected to linear projection and position encoding to obtain a sequence of respiratory sound Mel-spectrogram block embedding coding vectors. Here, the linear projection is to vectorize each respiratory sound Mel-spectrogram block in the sequence of respiratory sound Mel-spectrogram blocks so that it is suitable for computer processing, and the position encoding is performed to retain the timing information between each respiratory sound Mel-spectrogram block. Next, the sequence of embedding coding vectors of the breathing sound mel-spectrogram tiles is input into the multi-head self-attention structure of the audio spectrogram transformer network to obtain a sequence of global context semantic coding feature vectors of the breathing sound mel-spectrogram tiles. Here, the audio spectrogram transformer network is an existing network structure, and the principle of its basic architecture is the Transformer structure, so in the technical solution of this application, its network structure is not repeated. Finally, the sequence of global context semantic coding feature vectors of the breathing sound mel-spectrogram tiles is feature reshaped to obtain the global deep semantic coding feature map of the breathing sound mel-spectrogram.

[0065] It should be understood that the multi-head self-attention structure of the audio spectrogram converter network can utilize the multi-head attention mechanism to perform global contextual semantic association encoding on each breathing sound Mel-spectrogram block embedding coding vector in the sequence of the breathing sound Mel-spectrogram block embedding coding vector to capture the global deep feature information of the enhanced breathing sound Mel-spectrogram, and downsample the global deep feature information. That is, in the technical solution of the present application, the output of the audio spectrogram converter network is the deep feature of the enhanced breathing audio spectrogram after downsampling.

[0066] In a specific example, the audio spectrogram transformer network input is an enhanced breathing audio spectrogram of size 16×16 and step size 10×10 , the embedding layer in the audio spectrogram transformer network with 768 dimensions obtains the input features with 16×16×256×1024 dimensions. The audio spectrogram transformer network learns the global deep feature information of the input features and downsamples the output deep features , the breathing sound mel spectrum map global deep semantic coding feature map .

[0067] It should be understood that although the audio spectrogram converter network can learn global semantic features of the enhanced breathing audio spectrogram, its ability to learn local semantic features is relatively weak. In order to make up for this deficiency, the present invention further supplements the audio spectrogram converter network with a multi-scale feature extraction network and a temporal feature extraction network. Figure 4 As shown, the multi-scale feature extraction network includes a multi-head multi-scale self-attention layer, a first normalization layer, a multi-layer perceptron layer and a second normalization layer. Figure 5 As shown, the temporal feature extraction network includes a linear interpolation layer and a bidirectional LSTM model, and the multi-scale feature extraction network and the temporal feature extraction network are connected in sequence to form an encoder-decoder structure, and the output of the audio spectrogram transformer network is concatenated with the classification label as a new feature for re-encoding and up-sampling to strengthen the learning of the global and local features of the audio spectrogram.

[0068] Specifically, in the data flow of step S142, in order to further combine the known category information and help the model converge faster during the training process, thereby improving the classification accuracy, the present invention further embeds the true category label of the respiratory sound, converts the category label into a numerical coding feature representation, and obtains a true category label coding matrix of the respiratory sound with a dimension of T×C, so as to achieve dimensional matching of the data and facilitate the integration of category information.

[0069] Specifically, in the data flow of step S143, the global deep semantic coding feature map of the respiratory sound Mel spectrum map and the real category label coding matrix of the respiratory sound are connected to realize the integration of category information. In a specific example, the global deep semantic coding feature map of the respiratory sound Mel spectrum map , and concatenated with the breathing sound true category label encoding matrix of dimension T×C to generate a global deep semantic encoding feature map containing category label information .

[0070] Specifically, in the data flow of step S144, the feature shape of the global deep semantic coding feature map of the respiratory sound Mel-spectrogram containing the category label information is first reshaped to obtain a sequence of semantic coding vectors of the respiratory sound Mel-spectrogram block with fused category information, so as to capture the semantic associations between different blocks. Then, the sequence of semantic coding vectors of the respiratory sound Mel-spectrogram block with fused category information is input into the multi-head multi-scale self-attention layer to obtain a sequence of contextual semantic reconstruction coding feature vectors of the respiratory sound Mel-spectrogram block. Here, the multi-head multi-scale self-attention layer is based on the multi-head multi-scale self-attention mechanism, which can capture the contextual semantic associations in the sequence of semantic coding vectors of the respiratory sound Mel-spectrogram block with fused category information at different scales, learn the deep feature information of the respiratory audio spectrogram, thereby enhancing the sensitivity to the detailed features of the respiratory sound, while maintaining the effective modeling of long-distance dependencies. Then, the sequence of contextual semantic reconstruction coding feature vectors of the respiratory sound Mel-spectrogram block is input into the first normalization layer to obtain a sequence of normalized contextual semantic reconstruction coding feature vectors of the respiratory sound Mel-spectrogram block. That is, the learning process of the multi-head multi-scale self-attention layer is stabilized by normalization processing, the network convergence is accelerated, and the risk of gradient disappearance or explosion is reduced. Secondly, the sequence of the normalized breathing sound Mel spectrum graph block context semantic reconstruction encoding feature vector is input into the multi-layer perceptron layer to obtain the sequence of the breathing sound Mel spectrum graph block context semantic deep reconstruction encoding feature vector. Here, the multi-layer perceptron layer is used to further learn the deep features of the breathing audio spectrogram, and to extract and learn the nonlinear relationship in the sequence of the breathing sound Mel spectrum graph block context semantic deep reconstruction encoding feature vector by increasing the depth and complexity of the network, so as to learn more abstract and deeper feature representations. Then, the sequence of the breathing sound Mel spectrum graph block context semantic deep reconstruction encoding feature vector is input into the second normalization layer to obtain the sequence of the normalized breathing sound Mel spectrum graph block context semantic deep reconstruction encoding feature vector. Similarly, the second normalization layer is used to further stabilize the learning process of the multi-layer perceptron module, accelerate network convergence, and ensure the robustness of the model when processing complex data. Finally, the sequence of the normalized breathing sound Mel spectrogram block context semantic deep reconstruction encoding feature vectors is reshaped to obtain the re-encoded breathing sound Mel spectrogram global deep down-sampled semantic encoding feature matrix. That is, by down-sampling the learned breathing audio spectrogram deep feature information, the amount of calculation and storage requirements are reduced, while focusing on key information, and the output is the deep feature of the re-encoded enhanced breathing audio spectrogram. , that is, the global deep downsampling semantic coding feature matrix of the re-encoded respiratory sound Mel-spectrogram .

[0071] Specifically, in the data flow of step S145, first, the global deep down-sampling semantic coding feature matrix of the re-encoded respiratory sound mel-spectrogram is expanded into a sequence of down-sampling context semantic coding feature vectors of the re-encoded respiratory sound mel-spectrogram block, so as to adapt it to the input format of the bidirectional LSTM model. Then, the sequence of global deep down-sampling semantic coding feature vectors of the re-encoded respiratory sound mel-spectrogram is input into the linear interpolation layer to obtain a sequence of linear interpolation context semantic coding feature vectors of the re-encoded respiratory sound mel-spectrogram block. That is, the sequence of global deep down-sampling semantic coding feature vectors of the re-encoded respiratory sound mel-spectrogram is up-sampled by linear interpolation to enrich its deep features and restore the spatial information lost in the above-mentioned down-sampling process. Then, the sequence of linear interpolation context semantic coding feature vectors of the re-encoded respiratory sound mel-spectrogram block is input into the bidirectional LSTM model to obtain a sequence of local enhanced context semantic coding feature vectors of the respiratory sound mel-spectrogram. Here, the bidirectional LSTM model captures the forward and backward dependencies of the sequence of linear interpolation contextual semantic coding feature vectors of the re-encoded respiratory sound Mel spectrogram blocks through the combination of its forward and backward hidden states, and can effectively combine the forward and backward contextual information in the sequence to enhance the learning of local features of each respiratory sound Mel spectrogram, thereby better understanding the temporal changes and local details of respiratory sounds and improving the model's learning ability for local features. Finally, the sequence of the respiratory sound Mel spectrogram contextual local reinforcement semantic coding feature vectors is arranged into the respiratory sound Mel spectrogram global deep upsampling semantic coding feature matrix.

[0072] In a specific example, the input of the temporal feature extraction network is the global deep down-sampled semantic coding feature matrix of the re-encoded respiratory sound Mel spectrogram After the linear interpolation operation with an upsampling coefficient of m, the bidirectional LSTM model is entered to perform contextual learning of deep feature information, and finally the deep features of the upsampled enhanced breathing audio spectrogram are output through the arrangement and integration of local features. , that is, the global deep upsampling semantic encoding feature matrix of the respiratory sound Mel-spectrogram .

[0073] It should be understood that the multi-scale feature extraction network learns the contextual depth features of different scales of the respiratory sound Mel spectrum graph through a multi-head multi-scale self-attention layer, which enhances the model's ability to understand global and local features; and the temporal feature extraction network captures the temporal dependency in the feature sequence of the respiratory sound Mel spectrum graph block from the front and back directions through a bidirectional LSTM model, thereby enhancing the learning of the local contextual information of the deep features of the respiratory sound Mel spectrum graph, and effectively improving the quality of feature representation. Then, the learned global deep upsampling semantic encoding feature matrix LO of the respiratory sound Mel spectrum graph can be input into the linear output layer network, and the respiratory sound classification is performed through the fully connected layer and classification layer of the linear output layer network.

[0074] Specifically, in the data flow of step S146, first, the global deep upsampling semantic coding feature matrix of the respiratory sound Mel-spectrogram is expanded into a global deep upsampling semantic coding feature vector of the respiratory sound Mel-spectrogram to adapt it to the input format of the fully connected layer. Then, the global deep upsampling semantic coding feature vector of the respiratory sound Mel-spectrogram is input into the fully connected layer to obtain a dimensionally modulated semantic coding vector of the respiratory sound Mel-spectrogram, and the length of the dimensionally modulated semantic coding vector of the respiratory sound Mel-spectrogram is 7. Here, the fully connected layer maps the global deep upsampling semantic coding feature vector of the respiratory sound Mel-spectrogram to a vector of fixed dimension through linear transformation and activation function. It should be understood that in the technical solution of the present application, the number of categories of respiratory sounds is 7, so the output dimension of the fully connected layer is set to 7 to match the requirements of the classification task. Next, the dimensionally modulated semantic coding vector of the respiratory sound Mel-spectrogram is input into a classifier based on the Softmax function to obtain the predicted category label of the respiratory sound. Here, the classification layer uses a softmax function to convert the dimensionally modulated respiratory sound Mel-spectrogram semantic encoding vector output by the fully connected layer into a probability distribution vector, wherein each element in the probability distribution vector corresponds to a specific respiratory sound category, indicating the probability value that the model predicts that the input respiratory sound sample belongs to this category, and finally selects the category with the highest probability as the output result, thereby obtaining the respiratory sound prediction category label prediction result.

[0075] It should be understood that in the step S140, a training step for the respiratory sound classification model is also included to minimize the difference between the predicted category label and the true category label. Specifically, the training step includes: calculating the cross entropy loss function value between the true category label of the respiratory sound and the predicted category label of the respiratory sound; calculating the contrast loss function value between the sequence of the respiratory sound mel-spectrogram block embedding coding vector and the sequence of the respiratory sound mel-spectrogram block global context semantic coding feature vector; using the sum of the contrast loss function value and the cross entropy loss function value as the loss function value to train the respiratory sound classification model to obtain the trained respiratory sound classification model.

[0076] In a specific example, during the training process of the respiratory sound classification model, assuming that x and y are the input and output of the respiratory sound classification model, respectively, the respiratory sound classification model converts the input enhanced respiratory sound Mel-spectrogram into a series of patches: ,in is the neural network layer coefficient, n is the total number of patches, is the category label patch, is the category label function, then the cross entropy loss function The calculation formula is:

[0077]

[0078] Assumptions and refers to the patches in the first and last layers of the network in the respiratory sound classification model, i.e., the sequence of embedded coding vectors of the respiratory sound Mel-spectrogram patches and the sequence of global context semantic coding feature vectors of the respiratory sound Mel-spectrogram patches. The contrast loss function of the last layer of network patches in the respiratory sound classification model and the patches in other layers is The calculation formula is:

[0079]

[0080] Accordingly, the global loss function of the respiratory sound classification model is The calculation formula is:

[0081]

[0082] During the training process of the respiratory sound classification model, the Adam optimizer was used, the learning rate was set to 0.0001, and the model was trained by reverse gradient propagation. The training ended when the training rounds reached 50 times or the classification accuracy was improved by less than 0.0001.

[0083] After the training of the respiratory sound classification model is completed, the trained respiratory sound classification model can be used to classify new respiratory sound samples. Figure 1 As shown, the inference stage includes: S150: obtaining the audio of the breathing sound to be identified; S160: inputting the audio of the breathing sound to be identified into the trained breathing sound classification model to obtain the breathing sound recognition result. Here, the trained breathing sound classification model is based on the deep learning network structure obtained by previous training, and can automatically extract the features of the audio of the breathing sound to be identified, and map it to the corresponding breathing sound category to determine the breathing sound recognition result of the audio sample.

[0084] In summary, a respiratory sound classification method based on Mel-spectrogram is explained, which utilizes a respiratory sound classification model with a novel network structure to improve the sensitivity and specificity of respiratory sound classification. In particular, the respiratory sound classification model with a novel network structure includes an audio spectrogram converter network, a multi-scale feature extraction network, a temporal feature extraction network, and a linear output layer network, wherein the multi-scale feature extraction network and the temporal feature extraction network form a feature encoding and decoding architecture to perform global and local feature reinforcement learning on the respiratory sound spectrum features output by the audio spectrogram converter network, thereby improving the sensitivity and specificity of respiratory sound classification.

[0085] On the other hand, the present application also provides a respiratory sound classification system based on Mel-spectrogram, which can implement the respiratory sound classification method based on Mel-spectrogram, such as Figure 6 As shown, the respiratory sound classification system 100 based on mel-spectrogram includes: a training module 110, including: a training data acquisition unit 111, used to acquire a respiratory sound training data set, wherein each respiratory sound training data in the respiratory sound training data set is a respiratory sound audio containing a true category label of the respiratory sound; a mel-spectrogram feature extraction unit 112, used to extract mel-spectrogram features from the respiratory sound audio of each respiratory sound training data to obtain a set of respiratory sound mel-spectrograms; a feature enhancement unit 113, used to perform data enhancement on the set of respiratory sound mel-spectrograms to obtain a set of enhanced respiratory sound mel-spectrograms; a training unit 114, used to train the respiratory sound classification model based on the set of enhanced respiratory sound mel-spectrograms to obtain a trained respiratory sound classification model; an inference module 120, including: a respiratory sound acquisition unit 121 to be identified, used to acquire the respiratory sound audio to be identified; and a recognition unit 122, used to input the respiratory sound audio to be identified into the trained respiratory sound classification model to obtain a respiratory sound recognition result.

[0086] Here, those skilled in the art can understand that the specific operations of each module in the above-mentioned respiratory sound classification system based on Mel-spectrogram have been described in the above reference. Figures 1 to 5It has been described in detail in the description of the mel-spectrogram-based respiratory sound classification method, and therefore, its repeated description will be omitted.

[0087] Example 2

[0088] In another embodiment of the present application, considering that after the real category label of the respiratory sound is embedded in the code and connected and fused with the learned deep features of the respiratory sound Mel-spectrogram (i.e., the global deep semantic coding feature map of the respiratory sound Mel-spectrogram), this direct splicing method may lead to the dilution of feature information. In particular, when the deep feature dimension of the respiratory sound Mel-spectrogram is high, the category label feature accounts for a very small proportion of the entire spliced ​​feature, which may cause the model to be unable to fully learn the supervisory information provided by the category label, thereby affecting the classification performance. Based on this, the present application proposes an optimized feature fusion method.

[0089] Specifically, first, the true category label of the respiratory sound is embedded and encoded to obtain a true category label encoding vector of the respiratory sound. Then, the global deep semantic encoding feature map of the respiratory sound mel spectrum map and the true category label encoding vector of the respiratory sound are cross-modally jointly encoded to obtain a global deep semantic encoding feature map of the respiratory sound mel spectrum map containing category label information.

[0090] Specifically, the global deep semantic coding feature map of the respiratory sound Mel spectrum map and the real category label coding vector of the respiratory sound are cross-modally jointly encoded to obtain a global deep semantic coding feature map of the respiratory sound Mel spectrum map containing category label information, including: performing local feature decomposition along the channel dimension on the global deep semantic coding feature map of the respiratory sound Mel spectrum map to obtain a set of global deep local feature matrices of the respiratory sound Mel spectrum map, which is expressed as follows:

[0091]

[0092] in, represents the global deep semantic coding feature map of the respiratory sound mel-spectrogram, represents the eigendecomposition, , , and Respectively represent the first, second, and and A global deep local feature matrix of the respiratory sound Mel-spectrogram, that is, each feature matrix of the global deep semantic coding feature map of the respiratory sound Mel-spectrogram along the channel dimension, is the number of channels of the global deep semantic coding feature map of the respiratory sound mel spectrum map. Here, the local feature decomposition along the channel dimension is performed on the global deep semantic coding feature map of the respiratory sound mel spectrum map to enhance the feature resolution and improve the cross-modal interaction capability of the detail features.

[0093] Next, the breathing sound true category label encoding vector is used as the query vector, and each breathing sound Mel spectrum global depth local feature matrix in the set of the breathing sound Mel spectrum global depth local feature matrix is ​​used as the key matrix, and the query vector and the key matrix are input into the prompt learning network to obtain the set of true category label-breathing sound Mel spectrum global depth local query prompt semantic encoding vectors, which is expressed as follows:

[0094]

[0095]

[0096] in, represents the true category label encoding vector of the respiratory sound, , 、 and Respectively represent the The first, second, and third and row vectors, namely the key vectors, is the number of rows of the global depth local feature matrix of the respiratory sound Mel-spectrogram, Indicates prompt learning network, represents vector multiplication, represents the transpose of a vector, represents the magnitude of a vector, and Respectively represent and True category labels - breathing sound Mel spectrum global depth local semantic correlation coefficient, represents the exponential function with base e, Indicates Normalized true category label-breathing sound Mel spectrum global depth local semantic correlation coefficient, It means point multiplication by position. Indicates The global deep local query hint semantic encoding vector of each true category label-breathing sound mel spectrogram. Here, the hint learning network is based on the attention mechanism, and the attention weight is calculated through a fine-grained semantic relevance measurement, and the feature part associated with the semantics of the true category label of the breathing sound is selected from the global deep local feature matrix of each breathing sound mel spectrogram, thereby enhancing the understanding of the correlation and interaction relationship between modalities.

[0097] Then, each true category label-breathing sound Mel spectrum global depth local query prompt semantic coding vector in the set of the true category label coding vector of the breathing sound and the true category label-breathing sound Mel spectrum global depth local query prompt semantic coding vector is input into the cross-modal mask weaving network based on the prompt information to obtain a set of global depth local feature mask weight matrices of the breathing sound Mel spectrum based on the prompt information, which is expressed as follows:

[0098]

[0099] in, The feature scale of the semantic encoding vector for the global depth local query prompt of the true category label-breathing sound mel-spectrogram, represents the normalized exponential function, Indicates Here, the cross-modal mask weaving network based on prompt information uses a self-attention mechanism to highlight the important interaction parts between modalities through further prompt learning, while suppressing irrelevant information and reducing the influence of noise, thereby creating a mask that emphasizes the most relevant part of the modal interaction, that is, the global deep local feature mask weight matrix of the respiratory sound mel spectrogram based on prompt information.

[0100] Furthermore, the point-by-point multiplication between the set of global depth local feature mask weight matrices of the respiratory sound Mel spectrum graph based on the prompt information and the set of global depth local feature matrices of the respiratory sound Mel spectrum graph is calculated to obtain a set of true category label-global depth cross-modal local granularity significant interaction matrices of the respiratory sound Mel spectrum graph; the set of true category label-global depth cross-modal local granularity significant interaction matrices of the respiratory sound Mel spectrum graph is subjected to feature cascade aggregation along the channel dimension to obtain the global depth semantic coding feature map of the respiratory sound Mel spectrum graph containing category label information, which is expressed as follows:

[0101]

[0102] in, , , and Respectively represent the first, second, and and A global deep local feature mask weight matrix of the respiratory sound Mel-spectrogram based on the prompt information, Indicates cascade, The global deep semantic coding feature map of the respiratory sound mel-spectrogram containing the category label information is represented. Here, each global deep local feature mask weight matrix of the respiratory sound mel-spectrogram based on the prompt information is used as the weight, and each global deep local feature matrix of the respiratory sound mel-spectrogram is weighted modulated to emphasize the feature part of the deep feature of the respiratory sound mel-spectrogram that is significantly associated with the category label, thereby achieving guided fusion of cross-modal information. Finally, each weighted modulated global deep local feature matrix of the respiratory sound mel-spectrogram is subjected to information aggregation along the channel dimension to restore the original feature map structure and generate a global deep semantic coding feature map of the respiratory sound mel-spectrogram containing the category label information.

[0103] Through this continuous prompt-assisted learning process, the attention allocation to the deep features of the respiratory sound Mel-spectrogram can be dynamically adjusted according to the true category label information, thereby significantly improving the model's recognition accuracy and classification performance of respiratory sound features.

[0104] Comparative Example 1

[0105] The technical method of this application has implemented the designed architecture using Python, and all experiments were run on a desktop computer equipped with an Intel Core i7-12700kf central processor and an NVIDIA Geforce GTX 4090 graphics card. The same data is used for training and classification, and comparative experiments are conducted with existing technologies. Existing technology 1 (Li, Jun, et al. Improving the resnet-based respiratory sound classification systems with focal loss. IEEE, 2022) uses a CNN network structure classification model, existing technology 2 (Chen, Zizhao, et al. Classify respiratory abnormality in lung sounds using stft and a fine-tuned resnet18 network. IEEE, 2022) uses a CNN+RestNet18 network structure, existing technology 3 (Zhang, Lin, et al. A feature polymerized based two-level ensemble model for respiratory sound classification. IEEE, 2022) uses an ML network structure, existing technology 4 (Ma, Wei-Bang, et al. An effective lung sound classification system for respiratory disease diagnosis using densenet cnn model with sound pre-processing engine. IEEE, 2022) uses a DenseNet169 network structure, and existing technology 5 (TaghiBeyglou, Behrad, et al. TRespNET: A dual-route exploratory CNN model for pediatric adventitious respiratory sound identification. 2024) uses ResNet50+attn network structure. Prior arts 1 to 5 belong to convolutional neural network methods.Prior art six (Gong, Yuan, YA Chung, and J. Glass. AST: Audio Spectrogram Transformer. 2021) is the original AST network and adopts the Transformer network structure. The method of this application achieves an improvement on prior art six by adding an MFE network and an LLD network on the basis of the AST network.

[0106] Since the respiratory sound dataset is an unbalanced dataset, the prediction accuracy cannot well evaluate the performance of the model. Therefore, two evaluation indicators, sensitivity and specificity (i.e., the ability to correctly identify abnormal respiratory sound samples and the ability to correctly identify normal respiratory sound samples) are used to evaluate the performance of the respiratory sound classification model of this application and the prior art. Sensitivity indicates the recognition accuracy of abnormal respiratory sound samples, and specificity indicates the recognition accuracy of normal respiratory sound samples. The higher the sensitivity and specificity, the stronger the classification model's ability to recognize abnormal / normal respiratory sounds, and the better the classification model performance. The sensitivity calculation formula is:

[0107]

[0108] The specificity calculation formula is:

[0109]

[0110] The experimental results are shown in Table 1:

[0111]

[0112] Among them, sensitivity represents the ability to correctly identify abnormal breathing sound samples, and specificity represents the ability to correctly identify normal breathing sound samples. As can be seen from Table 1, the classification sensitivity of the present application is significantly higher than the classification results of traditional convolutional neural networks, exceeding 90%; the classification specificity is as high as 98.8%, which is higher than the optimal classification specificity of traditional convolutional neural networks. The comparison results of the present application with the prior art six show that the classification sensitivity of the breathing sound classification model described in the present application is improved by 10%, and the classification specificity is also higher than the classification results of the audio spectrum converter network. Therefore, the improvement of the present application has significant beneficial effects.

[0113] In summary, the respiratory sound classification model obtained using the method of the present application has excellent respiratory sound classification performance, can be applied to the lung sound auxiliary diagnosis system, and improve the diagnostic accuracy of the lung sound auxiliary diagnosis system. Therefore, the present application has significant application value.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A method for classifying respiratory sounds based on mel-spectrogram, characterized in that: include: Training phase and inference phase; The training phase includes: Acquire a respiratory sound training data set, wherein each respiratory sound training data in the respiratory sound training data set is a respiratory sound audio containing a true category label of the respiratory sound; Extracting Mel-spectrogram features from the breathing sound audio of each breathing sound training data to obtain a set of breathing sound Mel-spectrogram graphs; Performing data enhancement on the set of respiratory sound mel-spectrograms to obtain a set of enhanced respiratory sound mel-spectrograms; The respiratory sound classification model is trained based on the set of enhanced respiratory sound mel-spectrograms to obtain a trained respiratory sound classification model, including: The enhanced respiratory sound Mel-spectrogram is subjected to a spectrum block processing and then input into the audio spectrum converter network to obtain a global deep semantic coding feature map of the respiratory sound Mel-spectrogram; Embedding and encoding the true category labels of the respiratory sounds to obtain a true category label encoding matrix of the respiratory sounds; Connecting the global deep semantic coding feature map of the respiratory sound Mel-spectrogram and the real category label coding matrix of the respiratory sound to obtain the global deep semantic coding feature map of the respiratory sound Mel-spectrogram containing category label information; Inputting the global deep semantic coding feature map of the respiratory sound Mel spectrum graph containing category label information into a multi-scale feature extraction network to obtain a re-encoded global deep down-sampled semantic coding feature matrix of the respiratory sound Mel spectrum graph; Inputting the re-encoded respiratory sound Mel-spectrogram global depth down-sampling semantic coding feature matrix into a temporal feature extraction network to obtain a respiratory sound Mel-spectrogram global depth up-sampling semantic coding feature matrix; Inputting the global deep upsampling semantic encoding feature matrix of the respiratory sound Mel-spectrogram into a linear output layer network to obtain a respiratory sound prediction category label; The respiratory sound classification model includes an audio spectrogram transformer network, a multi-scale feature extraction network, a temporal feature extraction network and a linear output layer network; The inference stage includes: Obtain the audio of the breathing sound to be identified; The respiratory sound audio to be recognized is input into the trained respiratory sound classification model to obtain a respiratory sound recognition result.

2. The method for classifying respiratory sounds based on Mel-spectrogram according to claim 1, characterized in that: Performing data enhancement on the set of respiratory sound mel-spectrograms to obtain a set of enhanced respiratory sound mel-spectrograms comprises at least one of the following steps: Zero padding is performed on the respiratory sound mel-spectrogram; Performing time dimension distortion on the respiratory sound mel-spectrogram; Performing frequency channel masking on the respiratory sound mel-spectrogram; The respiratory sound mel-spectrogram is subjected to time channel masking.

3. The method for classifying respiratory sounds based on Mel-spectrogram according to claim 1, characterized in that: The multi-scale feature extraction network includes a multi-head multi-scale self-attention layer, a first normalization layer, a multi-layer perceptron layer and a second normalization layer; the temporal feature extraction network includes a linear interpolation layer and a bidirectional LSTM model; the linear output layer network includes a fully connected layer and a classification layer.

4. The method for classifying respiratory sounds based on Mel-spectrogram according to claim 3, characterized in that: The enhanced breathing sound Mel-spectrogram is subjected to a spectrum block processing and then input into the audio spectrum converter network to obtain a global deep semantic coding feature map of the breathing sound Mel-spectrogram, including: Performing a spectrum block processing on the enhanced respiratory sound mel-spectrogram to obtain a sequence of respiratory sound mel-spectrogram blocks; Performing linear projection and position encoding on each breathing sound mel-spectrogram block in the sequence of breathing sound mel-spectrogram blocks to obtain a sequence of breathing sound mel-spectrogram block embedding coding vectors; Inputting the sequence of embedding encoding vectors of the breathing sound mel-spectrogram tiles into a multi-head self-attention structure to obtain a sequence of global context semantic encoding feature vectors of the breathing sound mel-spectrogram tiles; The sequence of global context semantic coding feature vectors of the respiratory sound mel-spectrogram block is feature reshaped to obtain the global deep semantic coding feature map of the respiratory sound mel-spectrogram.

5. The method for classifying respiratory sounds based on Mel-spectrogram according to claim 3, characterized in that: Inputting the global deep semantic coding feature map of the respiratory sound Mel spectrum graph containing category label information into the multi-scale feature extraction network to obtain a re-encoded global deep down-sampled semantic coding feature matrix of the respiratory sound Mel spectrum graph, including: Reshaping the feature shape of the global deep semantic coding feature map of the respiratory sound Mel-spectrogram containing the category label information to obtain a sequence of semantic coding vectors of the respiratory sound Mel-spectrogram block that integrates the category information; Inputting the sequence of semantic encoding vectors of the breathing sound Mel-spectrogram block fused with category information into the multi-head multi-scale self-attention layer to obtain a sequence of contextual semantic reconstruction encoding feature vectors of the breathing sound Mel-spectrogram block; Inputting the sequence of the respiratory sound Mel-spectrogram block context semantic reconstruction encoding feature vectors into the first normalization layer to obtain a sequence of normalized respiratory sound Mel-spectrogram block context semantic reconstruction encoding feature vectors; Inputting the sequence of normalized respiratory sound Mel-spectrogram block context semantic reconstruction encoding feature vectors into the multi-layer perceptron layer to obtain a sequence of respiratory sound Mel-spectrogram block context semantic deep reconstruction encoding feature vectors; Inputting the sequence of the respiratory sound Mel-spectrogram block context semantic deep reconstruction encoding feature vectors into the second normalization layer to obtain a sequence of normalized respiratory sound Mel-spectrogram block context semantic deep reconstruction encoding feature vectors; The sequence of the normalized respiratory sound Mel-spectrogram block context semantic deep reconstruction encoding feature vectors is reshaped to obtain a global deep down-sampling semantic encoding feature matrix of the re-encoded respiratory sound Mel-spectrogram.

6. The method for classifying respiratory sounds based on Mel-spectrogram according to claim 3, characterized in that: The re-encoded respiratory sound Mel-spectrogram global depth down-sampling semantic coding feature matrix is ​​input into a temporal feature extraction network to obtain a respiratory sound Mel-spectrogram global depth up-sampling semantic coding feature matrix, including: Expanding the global deep down-sampling semantic coding feature matrix of the re-encoded respiratory sound Mel-spectrogram into a sequence of down-sampling contextual semantic coding feature vectors of the re-encoded respiratory sound Mel-spectrogram block; Inputting the sequence of global depth down-sampling semantic coding feature vectors of the re-encoded respiratory sound Mel-spectrogram into a linear interpolation layer to obtain a sequence of linear interpolation context semantic coding feature vectors of the re-encoded respiratory sound Mel-spectrogram block; Inputting the sequence of linear interpolation context semantic encoding feature vectors of the re-encoded respiratory sound Mel-spectrogram block into a bidirectional LSTM model to obtain a sequence of local enhanced context semantic encoding feature vectors of the respiratory sound Mel-spectrogram; The sequence of the contextual local enhanced semantic coding feature vectors of the respiratory sound Mel-spectrogram is arranged into a global deep up-sampling semantic coding feature matrix of the respiratory sound Mel-spectrogram.

7. The method for classifying respiratory sounds based on Mel-spectrogram according to claim 3, characterized in that: Inputting the global deep upsampling semantic coding feature matrix of the respiratory sound Mel-spectrogram into the linear output layer network to obtain the respiratory sound prediction category label, including: Expanding the global deep upsampling semantic coding feature matrix of the respiratory sound Mel-spectrogram into a global deep upsampling semantic coding feature vector of the respiratory sound Mel-spectrogram; Inputting the global deep up-sampling semantic coding feature vector of the respiratory sound Mel-spectrogram into the fully connected layer to obtain a dimensionally modulated respiratory sound Mel-spectrogram semantic coding vector, wherein the length of the dimensionally modulated respiratory sound Mel-spectrogram semantic coding vector is 7; The dimensionally modulated respiratory sound mel-spectrogram semantic encoding vector is input into a classification layer based on a Softmax function to obtain the respiratory sound prediction category label.

8. The method for classifying respiratory sounds based on Mel-spectrogram according to claim 4, characterized in that: The respiratory sound classification model is trained based on the set of enhanced respiratory sound mel-spectrograms to obtain a trained respiratory sound classification model, further comprising the steps of: Calculating a cross entropy loss function value between the true category label of the respiratory sound and the predicted category label of the respiratory sound; Calculate the contrast loss function value between the sequence of embedded coding vectors of the respiratory sound mel-spectrogram block and the sequence of global context semantic coding feature vectors of the respiratory sound mel-spectrogram block; The respiratory sound classification model is trained using the sum of the contrast loss function value and the cross entropy loss function value as the loss function value to obtain the trained respiratory sound classification model.

9. A respiratory sound classification system based on Mel-spectrogram, which can implement the respiratory sound classification method based on Mel-spectrogram as described in any one of claims 1 to 8, characterized in that: The respiratory sound classification system based on Mel-spectrogram includes: a training module and an inference module; Wherein, the training module includes: A training data acquisition unit, used to acquire a respiratory sound training data set, wherein each respiratory sound training data in the respiratory sound training data set is a respiratory sound audio containing a true category label of the respiratory sound; A mel-spectrogram feature extraction unit, configured to extract mel-spectrogram features from the breathing sound audio of each breathing sound training data to obtain a set of breathing sound mel-spectrograms; A feature enhancement unit, configured to perform data enhancement on the set of respiratory sound mel-spectrograms to obtain a set of enhanced respiratory sound mel-spectrograms; A training unit, used for training the respiratory sound classification model based on the set of enhanced respiratory sound mel-spectrograms to obtain a trained respiratory sound classification model; Wherein, the inference module comprises: A breathing sound acquisition unit to be identified, used to acquire the audio of the breathing sound to be identified; The recognition unit is used to input the respiratory sound audio to be recognized into the trained respiratory sound classification model to obtain a respiratory sound recognition result.

Images