Breathing sound recognition method based on event level detection technology

Abstract

The invention provides a breath sound recognition method based on an event level detection technology, and belongs to the technical field of audio signal recognition. In order to solve the problem that an existing method is inaccurate in breath sound event detection, the breath sound recognition method based on the event-level detection technology, provided by the invention, constructs a breath sound recognition model based on a hierarchical token semantic audio Transform to detect an abnormal breath sound event, so that the detection accuracy and speed of the breath sound event are improved, and the detection efficiency of the breath sound event is improved. Therefore, the diagnosis efficiency of clinical respiratory diseases is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signal recognition, and in particular to a breathing sound recognition method based on event-level detection technology. Background Art

[0002] Respiratory diseases are highly prevalent and cause high mortality. Early detection, diagnosis, and treatment have positive clinical significance. Breathing sounds are noises produced during breathing, which reveal the normal or abnormal state of the respiratory system. Common abnormal breathing sounds include crackles, wheezes, dry rales, and wheezes from the lungs. Identifying abnormal breathing sounds plays a vital role in clinical medicine. Traditional methods of identifying breathing sounds mainly rely on handheld stethoscopes, which are affected by the doctor's diagnostic experience and subjective judgment, and cannot be quantitatively analyzed. In recent years, researchers have used computer algorithms to process and analyze breathing sounds, thereby realizing automated and quantitative identification of breathing sounds.

[0003] At present, respiratory sound recognition mainly adopts frame-based event detection methods, which divide each audio frame into an event category and then aggregate continuous frame-level predictions to identify the boundaries of sound events. In the frame-level method, the input audio signal is divided into segments of fixed length, and the sound events in each segment are further classified independently. This segmentation and classification strategy lacks the time series information of events and cannot effectively identify events that occur continuously. Sound Event Detection (SED) technology refers to the detection and classification of specific sound events in a given audio signal, and determining the start and end time of these events. This technology can describe the timing of events and is more effective in detecting events that occur continuously.

[0004] Applying sound event detection technology to respiratory sound recognition helps to further improve the diagnostic efficiency of respiratory diseases, but this method is still under research. Therefore, the present invention provides a solution to this problem. Summary of the invention

[0005] The purpose of the present invention is to provide a method for identifying respiratory sounds based on event-level detection technology, which can solve the problem that the existing methods are inaccurate in detecting respiratory sound events.

[0006] The present invention provides a method for identifying respiratory sounds based on event-level detection technology, comprising the following contents: Acquire and preprocess respiratory sound data to construct a training set; We build an initial model based on the Hierarchical Token Semantic Audio Transformer, add position encoding to each layer of the Transformer encoder input, and introduce a masked multi-head self-attention mechanism to each layer of the Transformer decoder. Train the initial model based on the training set to obtain a respiratory sound recognition model; The respiratory sound is recognized based on the respiratory sound recognition model to obtain the recognition result.

[0007] The present invention provides a respiratory sound recognition method based on event-level detection technology, which builds a respiratory sound recognition model based on a hierarchical token semantic audio Transformer to detect abnormal respiratory sound events, thereby improving the detection accuracy and speed of respiratory sound events, thereby improving the diagnostic efficiency of clinical respiratory diseases.

[0008] Optionally, the respiratory sound data includes continuous even pronunciations and discontinuous even pronunciations.

[0009] Optionally, when preprocessing the respiratory sound data, the audio signals in the dataset are resampled to 16 kHz and converted into a Mel-spectrogram.

[0010] Optionally, when preprocessing the respiratory sound data, the time and frequency masking technology is used to mask the selected area of ​​the Mel-spectrogram and then perform frequency band shifting.

[0011] Optionally, the initial model consists of a hierarchical token semantic audio Transformer, a Transformer encoder, a Transformer decoder, and a feed-forward neural network.

[0012] Optionally, when the initial model is trained based on the training set to obtain the respiratory sound recognition model, the features of the Mel-spectrogram are extracted through a hierarchical token semantic audio Transformer, and the one-dimensional position encoding is combined with the extracted features to further train the Transformer encoder, and then the event representation is generated through the Transformer decoder, and finally the generated events are converted into event detection results using a feedforward neural network, and the event-level loss is calculated using the Hungarian algorithm.

[0013] Optionally, when the event-level loss is calculated using the Hungarian algorithm, the loss function includes a position loss function and a classification loss function.

[0014] Optionally, when the respiratory sound recognition model is obtained by training the initial model based on the training set, the training set is divided into training data and verification data, the initial model is trained based on the training data, and the performance of the trained initial model is evaluated based on the verification data.

[0015] Optionally, a performance evaluation is performed on the trained initial model based on the validation data, and the evaluation indicators include positive predictive value, sensitivity, and harmonic mean of positive predictive value and sensitivity.

[0016] Optionally, when the respiratory sound is recognized based on the respiratory sound recognition model to obtain a recognition result, the recognition result is an event type and an event position represented by the respiratory sound. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Diagram of the encoding process for the Mel-spectrogram; Figure 2 It is the structural composition diagram of REDT model; Figure 3 Figure 1. Transformer encoder and decoder structure composition diagram; Figure 4 This is a diagram of the REDT model training process; Figure 5 This is the REDT model recognition result diagram. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0019] The embodiment of the present invention provides a method for identifying respiratory sounds based on event-level detection technology, including the following technical solutions: S1, obtain and preprocess respiratory sound data to construct a training set; S2. Build an initial model based on Hierarchical Token Semantic Audio Transformer (HTS-AT), add position encoding to each layer of the Transformer Encoder input, and introduce a masked multi-head self-attention mechanism to each layer of the Transformer Decoder. S3, training the initial model based on the training set to obtain a respiratory sound recognition model; S4. Recognize the respiratory sound based on the respiratory sound recognition model to obtain a recognition result.

[0020] In fact, when executing S1, the respiratory sound data obtained comes from the HF_lung_V1 public dataset, which was collected from 279 patients and includes 9765 records, each of which lasts for 15 seconds. This is the largest public lung sound record dataset to date. The dataset includes 34095 inhalations and 18349 exhalations, 13883 continuous adventitious sounds (CAS) and 15606 discontinuous adventitious sounds (DAS); continuous adventitious sounds include 8457 wheezing events, 686 wheezing events and 4740 dry rales events, and discontinuous adventitious sounds are all crackle events. According to the segmentation provided by the dataset author, the 9765 records are divided into 7809 training data and 1956 test data.

[0021] The respiratory sound data used for training and testing were preprocessed by resampling the audio signals in the data set to 16kHz. The parameters were set as follows: nfft was 1024, window size was 1024, hop size was 323, and Mel filter size was 64. Figure 1 As shown, the data is converted into a Mel-spectrogram, and the time and frequency masking techniques are used to mask the selected area of ​​each spectrum graph and then the frequency band is shifted to achieve data enhancement.

[0022] In fact, when performing S2, in some respiratory sound event detection tasks, each sound event category requires multiple separate models for detection. Therefore, in the HF_lung_V1 dataset, 4 models are required for inspiration, exhalation, CAS, and DAS respectively, which is called a single-class multi-model method. This method will undoubtedly increase the demand for computing resources and storage space, and it is difficult to apply in clinical practice. In addition, respiratory sound event detection is a typical polyphonic SED task, which requires the simultaneous identification of multiple overlapping respiratory sound events. To this end, we propose a multi-class single model REDT (Respiratory Sound Event Detection Transformer), which uses only one model to detect all respiratory sound events. Figure 2 As shown in the figure, the REDT model consists of HTS-AT, Transformer Encoder, Transformer Decoder and Feedforward Neural Network (FFN).

[0023] S21, HTS-AT is used to extract the features of Mel-spectrogram. HTS-AT is based on the Transformer deep learning model architecture. The core of the Transformer architecture is the self-attention mechanism, which enables the model to handle long-distance dependency problems in sequence data and process all elements in the sequence in parallel, which is difficult to achieve in traditional recurrent neural networks and convolutional neural networks. In audio processing tasks such as respiratory sound event detection, using Transformer as the feature extraction part can use its powerful sequence modeling capabilities to extract key features in audio signals, thereby improving the accuracy and robustness of event detection. Compared with convolutional neural networks, Transformer is more flexible in processing sequence data and can better capture the timing information and long-distance dependencies in audio signals.

[0024] S22, Transformer encoder is used to extract position information. Figure 3 As shown in Figure 1, the Transformer encoder is composed of multiple identical Encoder Layers stacked together, each of which contains a multi-head self-attention mechanism, position encoding, two position-independent layer normalization layers, two residual connections, and a feedforward neural network. In stark contrast to the standard Transformer, where the position encoding is only incorporated into the initial input of the Transformer, we add position encoding to the input of each layer to improve its localization capability.

[0025] S23, Transformer decoder is used to generate event representation. Figure 3 As shown in the figure, the input of the Transformer decoder is first converted into a vector through an embedding layer, and a positional encoding is added to capture the position information of the word. Next, the input data is processed by N decoder layers, each of which contains masked multi-head self-attention, which ensures that information about future positions is not leaked when generating sequences; the decoder pays attention to the output of the encoder and makes decoding predictions; and a feedforward neural network is used to perform further nonlinear transformations on the output of the attention layer. Finally, the output layer is a linear layer followed by a softmax function, which generates a probability distribution for each timestamp for the respiratory sound event.

[0026] S24, FFN is a basic neural network structure, which consists of multiple layers, each layer contains multiple neurons. In FFN, information flows from the input layer to the hidden layer, and then from the hidden layer to the output layer. This process is unidirectional and there is no feedback connection or loop. Two FFNs are used in this model, one for multi-label classification and one for timestamp prediction.

[0027] In fact, when executing S3, the initial model is trained based on the training data, such as Figure 4 As shown in the figure, the features of the Mel-spectrogram are first extracted through the hierarchical token semantic audio Transformer, and the one-dimensional positional encoding is combined with the extracted features for further training through the Transformer encoder, and then the Transformer decoder is used to generate events. Finally, FFN is used to convert the generated events into event detection results, and the Hungarian algorithm is used to calculate the event-level loss.

[0028] S31. Feature extraction

[0029] To improve computational efficiency, HTS-AT is used to split the mel-spectrogram into smaller partitions, and then a sliding window attention mechanism is applied to extract features, which reduces training time and computational resources while maintaining high performance and lightweight.

[0030] 1) In the Swim Transformer, the Patch Partition module divides the input RGB image of size H×W×3 into non-overlapping patches of the same size, resulting in a patch of dimension N × (P 2 × 3). Each P 2 × 3 patches are treated as a patch token and are divided into N in total, which is the length of the valid input sequence of the Transformer. The patch partitioning method used in the Swim Transformer is designed for RGB images and is not suitable for Mel spectrograms of audio, where the time and frequency axes are not uniform with the horizontal and vertical axes in the image. Time → frequency → window is used for patch embedding of Mel spectrograms.

[0031] The audio Mel spectrogram is segmented into different patch tokens by Patch-Embed CNN with kernel size (P × P), and then input into Transformer in sequence. Unlike images, the width and height of the audio Mel spectrogram represent different information, which are the time and frequency axes respectively. In general, the duration is significantly longer than the frequency span. In order to better grasp the relationship between frequencies in the same time range, we first divide the Mel spectrogram into patch windows, such as w1, w2, ..., wn. After that, we segment the patches into each window. The token sequence is arranged in the order of time, frequency, and finally window. Patches with different frequencies in the same time frame will be adjacent in the input sequence.

[0032] 2) Patch Token maps features to an arbitrary dimension (set to D), and since the number of patches is A matrix of dimension N × D is obtained. For groups 2, 3, and 4, a patch merging layer is used to reduce the sequence size. This merging operation is performed by fusing four adjacent patches into one, increasing the number of channels by four times (4D). A linear layer is then added to fix the 4D dimension to a 2D dimension. After the four network groups, the shape of the patch markers changes from Reduced by 8 times ,Therefore, the GPU memory consumption decreases exponentially after each group.

[0033] For each Swin Transformer block in the group, a windowed attention mechanism is adopted to reduce the computational complexity. First, the attention window is divided into aw1, aw2, …, awk, and each window contains M × M patches. Then we only compute the attention matrix within the attention window. Therefore, we have k windowed attention matrices instead of the overall global attention matrix. For an audio patch token of size f × t and an initial latent dimension D, the computational complexity of these two mechanisms in a single Transformer block is as follows: ; ; The window attention reduces the second complexity term by ( ) times. For audio patch tokens arranged in a time-frequency-window sequence, each window attention module computes the relationship within a specific range of consecutive frequency bins and time frames. As the network goes deeper, the Patch-Merge layer merges adjacent windows, enabling the computation of attention relationships in a larger space.

[0034] S32. Generate sound events

[0035] In order to locate the class on the time axis, the one-dimensional position encoding is combined with the extracted features and further trained by the Transformer encoder, which can better pay attention to the temporal information. Finally, the predicted timestamp is implemented by the Transformer decoder for position and class realization.

[0036] 1) Integrating one-dimensional positional encoding into our model, this encoding mechanism enables the network to gain insight into the temporal positioning of individual sound elements, thus preventing important information from being missed. Essentially, it enhances the model’s temporal perception and ensures a comprehensive analysis of the audio data. The corresponding formula can be expressed as: ; ; Where t and f are the time and frequency in the Mel-spectrogram, i is the dimension, and d is the number of Transformer attention units. Using the above formula, we can derive the same as Positional encodings of the same shape .in It is the new feature map extracted by HTS-AT, T represents the dimension of the time axis, F represents the dimension of the frequency axis, and d represents the number of channels.

[0037] 2) and P are flattened on the time and frequency axes to obtain d×TF feature maps and position encodings, which are then input into the Transformer encoder. Transformer treats sound event detection as an integrated prediction problem and assumes that each event is independent. Therefore, the standard autoregressive decoding mechanism adopted in machine translation tasks is discarded. Instead, the decoder takes N learned embeddings (called event queries) as input and outputs N event representations in parallel, where N is a hyperparameter larger than the typical number of events in an audio clip. Finally, the predicted FFN is used to convert the events from the decoder into event detection results, such as

[0038] S33. Calculate event losses

[0039] To calculate the event-level loss, a match between the target event and the predicted event is required, which can be obtained through the Hungarian algorithm, and mismatched predictions will be marked as "null". The loss function consists of a location loss and a classification loss: ; The position loss is for predictions that do not match the “null” event ( 0), which is a linear combination of the L1 norm and IOU loss between the target and predicted position vectors:

[0040] ; in , ∈ R is a hyperparameter, is the assignment given by the matching process and N is the number of predictions. The classification loss is the cross entropy between the labels and the predictions: ; S34. Evaluate training results

[0041] When evaluating the performance of the trained initial model based on the validation data, the Jaccard similarity is used to evaluate the network training effect. By comparing the prediction results of each fragment with its corresponding actual ground situation, the Jaccard similarity between the actual ground situation and the predicted event is calculated. Based on this similarity, we make the following classifications: if the Jaccard similarity is greater than 0.5, the prediction is considered to be a true positive (True Positives, TP); if the similarity is between 0 and 0.5, the prediction is considered to be a false negative (False Negative, FN); if the similarity is zero, the prediction is considered to be a false positive (False Positives, FP). It should be noted that in this event detection task, we cannot define a true negative (True Negative, TN). To evaluate the performance of our model, we use the Positive Predictive Value (TPV), Sensitivity (Se), PPv and the harmonic mean of Se (F1 Score).

[0042] 1. Positive predictive value

[0043] The positive predictive value evaluates the accuracy of the model in detecting breath sound events, that is, how many of the events detected by the model are correct, and the calculation formula is: ; Among them, TP represents the number of samples correctly predicted as positive, and FP represents the number of samples incorrectly predicted as positive.

[0044] 2. Sensitivity

[0045] Sensitivity is the ability of the model to detect actual respiratory sound events, that is, how many of the actual events are correctly detected. The calculation formula is as follows: ; Among them, TP represents the number of samples correctly predicted as positive, and FN represents the number of samples incorrectly predicted as negative.

[0046] 3. F1 Score

[0047] It is a balanced performance indicator that takes into account the accuracy of the model. The calculation formula is as follows: ;

[0048] These indicators are crucial for evaluating and optimizing the respiratory sound event detection model. By optimizing these indicators, the clinical application value of the model can be improved and reliable and effective diagnostic support can be provided in actual use.

[0049] S35. Comparative experiment on respiratory sound recognition model

[0050] There are different methods to detect respiratory sound events, including single-class multi-model method and multi-class single-model method. Multi-class single-model event detection is closer to clinical medical applications and is relatively easy to deploy. However, compared with single-class multi-model event detection, the single-class multi-model method has a higher score. We conducted two different sets of comparative experiments, namely single-class multi-model experiments and multi-class single-model experiments, to demonstrate the superiority and practicality of our model.

[0051] 1) The lung SED index (JIE_F1) and the Tollar-based event index (TBE_F1) were used to compare and evaluate the detection results of the three multi-class single models CRNN, TCN and REDT on the HF_lung_V1 dataset. The results are shown in Table 1. In all evaluation indicators, the REDT model is higher than the CRNN and TCN models. For the JIE_F1 index, compared with the baseline of the best CRNN model, the REDT model scores in all event detections are improved by more than 40%, exceeding the average level.

[0052] Table 1. Comparison results of different multi-class single models

[0053] , 2) The lung SED index (JIE_F1) was used to compare and evaluate the detection results of five single-class multi-models, LSTM, BiGRU, CNN-GRU, CNN-BiGRU, CNN-BiGRU and multi-branch TCN, with the REDT model on the HF_lung_V1 dataset. The results are shown in Table 2 below. The JIE_F1 score of the REDT model in each event category far exceeds that of the first four single-class multi-models. Compared with the most advanced multi-branch TCN model, the inhalation score of the REDT model is only 2.7% lower, and the scores of other event categories are much higher. In summary, the results show that the REDT multi-class single model based on event-level detection performs better than other single-class multi-models in respiratory sound event detection, reaching the state-of-the-art level.

[0054] Table 2. Comparison results of single-class multi-model and multi-class single model

[0055] , In fact, when executing S4, Figure 5As shown, when the respiratory sounds are recognized based on the respiratory sound recognition model to obtain the recognition result, the recognition result is the event type and event position represented by the respiratory sounds. Among them, segment C on the vertical axis represents continuous occasional pronunciation events, segment D represents discontinuous occasional pronunciation events, segment E represents inhalation events, and segment I represents exhalation events. The estimated label bar is the recognition result of the REDT model, and the reference label bar is the comprehensive judgment result of multiple senior doctors. As can be seen from the figure, the two are very close, indicating that REDT has a high recognition accuracy for respiratory sounds.

[0056] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A method for identifying respiratory sounds based on event-level detection technology, characterized in that: Acquire and preprocess respiratory sound data to construct a training set; We build an initial model based on the Hierarchical Token Semantic Audio Transformer, add position encoding to each layer of the Transformer encoder input, and introduce a masked multi-head self-attention mechanism to each layer of the Transformer decoder. Train the initial model based on the training set to obtain a respiratory sound recognition model; The respiratory sound is recognized based on the respiratory sound recognition model to obtain the recognition result.

2. The method for identifying respiratory sounds based on event-level detection technology according to claim 1, characterized in that: The respiratory sound data includes continuous even pronunciation and discontinuous even pronunciation.

3. The method for identifying respiratory sounds based on event-level detection technology according to claim 1, characterized in that: When preprocessing the respiratory sound data, the audio signals in the dataset were resampled to 16kHz and converted into Mel spectrograms.

4. The method for identifying respiratory sounds based on event-level detection technology according to claim 3, characterized in that: When preprocessing the respiratory sound data, the time and frequency masking techniques are used to mask the selected area of ​​the Mel-spectrogram and then perform frequency band shifting.

5. The method for identifying respiratory sounds based on event-level detection technology according to claim 1, characterized in that: The initial model consists of a hierarchical token semantic audio Transformer, a Transformer encoder, a Transformer decoder, and a feed-forward neural network.

6. The method for identifying respiratory sounds based on event-level detection technology according to claim 5, characterized in that: When the initial model is trained based on the training set to obtain the respiratory sound recognition model, the features of the Mel-spectrogram are extracted through the hierarchical token semantic audio Transformer, and the one-dimensional position encoding is combined with the extracted features for further training through the Transformer encoder, and then the event representation is generated through the Transformer decoder. Finally, the generated events are converted into event detection results using a feedforward neural network, and the event-level loss is calculated using the Hungarian algorithm.

7. The method for identifying respiratory sounds based on event-level detection technology according to claim 6, characterized in that: When the Hungarian algorithm is used to calculate event-level loss, the loss function includes the position loss function and the classification loss function.

8. The method for identifying respiratory sounds based on event-level detection technology according to claim 1, characterized in that: When the initial model is trained based on the training set to obtain the respiratory sound recognition model, the training set is divided into training data and verification data, the initial model is trained based on the training data, and the performance of the trained initial model is evaluated based on the verification data.

9. The method for identifying respiratory sounds based on event-level detection technology according to claim 8, characterized in that: The performance of the trained initial model is evaluated based on the validation data, and the evaluation indicators include positive predictive value, sensitivity, and the harmonic mean of positive predictive value and sensitivity.

10. The method for identifying respiratory sounds based on event-level detection technology according to claim 1, characterized in that: When the respiratory sound is recognized based on the respiratory sound recognition model to obtain a recognition result, the recognition result is the event type and event position represented by the respiratory sound.

Images