Breathing sound electronic auscultation signal identification method based on multiple modes and deep learning

Abstract

The invention discloses a breath sound electronic auscultation signal identification method based on multi-mode and deep learning, and relates to the technical field of deep learning and multi-mode learning. Firstly, audio data related to breathing sound classification and text description corresponding to the audio data are obtained; preprocessing the audio data, including de-noising and feature extraction; thirdly, matching an audio sample with the generated text description through a comparative learning method so as to optimize the representation capability of the model; evaluating and verifying the trained model, and performing label prediction on an audio sample by using a classification head; and finally, combining the metadata with the audio features by adopting a multi-modal fusion method, and carrying out classification prediction through a reasoning stage. On the basis of focusing on the classification accuracy of the model, the multi-modal input effect of the model is focused at the same time, the robustness of the model is improved by integrating the metadata, and whether the model can still keep good classification performance or not under the condition that part of the metadata is missing is verified. In addition, the method also reveals the importance of different metadata on the model prediction result and the interaction relationship of the different metadata. In conclusion, the invention can provide reliable support for clinical application.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and multimodal learning, and in particular to a method for recognizing respiratory sound electronic auscultation signals based on multimodality and deep learning. Background Art

[0002] In recent years, respiratory diseases have gradually become one of the leading causes of death worldwide, and lung-related diseases are the most common cause of death. Early diagnosis is crucial to limit the spread of respiratory diseases and reduce their adverse effects on life expectancy and quality of life. Lung auscultation is one of the effective methods for diagnosing lung diseases, and auscultation with a stethoscope is also a standard method for screening and diagnosing lung diseases.

[0003] At present, most medical places mainly use manual auscultation as the main means of auscultation. However, the frequency range of breath sounds is about 100Hz to 2000Hz, but the human ear is only sensitive enough to the frequency range of 1000Hz to 2000Hz. Therefore, manual auscultation often easily loses important information about the low-frequency part of breath sounds. In addition, diagnosis based on stethoscopes has some disadvantages: it requires well-trained medical professionals to interpret the auscultation signals, and its subjectivity in interpretation is also prone to differences. At the same time, the auscultation results are also affected by many factors such as the doctor's medical experience, hearing condition, and external environment.

[0004] Nowadays, different types of electronic stethoscopes are gradually replacing traditional stethoscopes on the market. Electronic stethoscopes can convert sound into electronic signals and further amplify them to achieve the best auscultation effect. These signals can be further processed and digitized, and then transmitted to a computer for recording and analysis. Therefore, the emergence of electronic stethoscopes provides an opportunity for change in the patient healthcare system and lays the foundation for solving the above problems.

[0005] At present, artificial intelligence technology is developing rapidly. In the field of biomedicine, the use of intelligent algorithms to assist in the analysis and evaluation of medical signals has gradually become the mainstream. The Clap (Contrastive Language-Audio Pretraining) model is a multimodal contrastive learning model that aims to achieve semantic alignment of audio and text through pretraining. The model contains an audio encoder and a text encoder, which are used to extract features of audio and text respectively and generate high-dimensional embedding vectors. During the training process, Clap uses a contrastive learning strategy to maximize the similarity of audio-text positive and negative sample pairs, so that semantically matched audio and text embeddings are closer in the shared vector space, while mismatched embeddings are farther away. The model usually uses InfoNCE loss to optimize the alignment effect between audio and text, so that Clap has strong multimodal alignment capabilities in tasks such as audio-text retrieval and audio classification. The AST model uses the structural advantages of the Transformer, especially the self-attention mechanism and the multi-head attention mechanism, to effectively process complex temporal and spectral information in audio signals. By transforming and encoding in the time-frequency domain, AST can capture long-term dependencies in audio signals and has good generalization capabilities.

[0006] The present invention proposes a ClapAST model, and uses AST as the audio channel model of the Clap architecture. The text channel of the Clap model usually adopts the BERT model, which, like the AST model, is a variant of the Transformer. The choice of this architecture can utilize the characteristics of the Transformer to extract high-quality feature representations on text and audio respectively. Due to the structural consistency of the BERT and AST models, the text and audio features will be more similar in the representation space, which is conducive to the alignment and fusion of cross-modal features. This similar feature representation method not only improves the understanding ability of the multimodal model, but also improves the performance of the model in multimodal tasks. Summary of the invention

[0007] Purpose of the invention: The purpose of this patent is to achieve more accurate recognition of electronic auscultation signals of respiratory sounds by combining audio signals and text descriptions through a multimodal and deep learning-based method. Unlike previous studies that mostly used a single audio channel, the present invention innovatively integrates multimodal information, thereby improving the extraction and recognition accuracy of respiratory sound features. This method effectively reduces the information loss and model deviation caused by a single channel, and further optimizes the recognition performance. This innovation will provide an important impetus for the development of electronic stethoscope technology and propose novel solutions for biomedical signal analysis and diagnosis.

[0008] Technical solution: The present invention is a method for electronic auscultation signal recognition of respiratory sounds based on multimodality and deep learning. By improving the original Clap model in terms of network structure, parameter setting and algorithm, it is suitable for electronic auscultation signal recognition of respiratory sounds with text data description. It is characterized by mainly including the following key steps: text and audio data collection, data preprocessing and data enhancement, construction of text-audio data pairs, construction of feature extraction model, text-audio feature splicing, respiratory sound recognition model training, model evaluation and tuning. The specific construction steps are:

[0009] Step 1, text and audio data collection, that is, under the accompaniment and guidance of professional medical staff, the electronic signal data of respiratory sounds of the experimenters are collected at a specific collection position through the electronic stethoscope equipment equipped by the researchers, and various types of text description data are obtained accordingly.

[0010] Step 2: Preprocess the text and audio data. The key steps are as follows:

[0011] 2.1 All breathing records are resampled to a uniform frequency, and the audio data is segmented according to the breathing cycle. For data with a length of less than 4 seconds, a gradual in-and-out method is used for replication and enhancement, and data with a length of more than 8 seconds is truncated.

[0012] 2.2 Use short-time Fourier transform and Mel spectrum conversion to convert the audio waveform into a logarithmic Mel filter bank (Fbank) feature sequence, and apply standard normalization on the spectrum graph; the human ear's perception of pitch is nonlinear. In order to characterize the human ear's linear perception of pitch, the Mel scale is used to map Hertz (Hz) to Mel (mel). The specific preprocessing process is as follows:

[0013] 2.2.1 The audio signal is converted into a spectrogram by short-time Fourier transform (STFT). The STFT calculation is performed on the time signal x(t) to obtain the spectrum X(f):

[0014]

[0015] Where x[n] is the original audio signal, w[n-1] is a window function, f is the frequency, and t is the time window position.

[0016] 2.2.2 Square the amplitude of the STFT to obtain the power spectrum |X(f,t)| 2 :

[0017] P(f,t)=|X(f,t)| 2

[0018] 2.2.3 Use a set of bandpass filters to convert the frequency axis to the Mel scale. The relationship between the Mel scale and Hertz is:

[0019]

[0020] A Mel filter bank is applied to the spectrum, where each filter weights the energy of the power spectrum over a specific frequency range.

[0021] 2.2.4 After weighting the Mel filter bank, the output value of each filter is obtained, that is, the Mel spectrum:

[0022]

[0023] Among them, H m (f) represents the frequency response of the mth Mel filter, and S(m,t) is the output of the Mel spectrum.

[0024] 2.2.5 Finally, take the logarithm of the Mel spectrum to get the logarithmic Mel spectrum:

[0025] log-mel(m,t)=log(S(m,t)+∈)

[0026] where ∈ is a small constant that prevents taking logarithms about zero.

[0027] 2.3 Standardize and unify the formats of text description data of different categories. Descriptive text is mainly divided into four types: age, gender, collection location, and collection equipment. Each type is divided into 2-6 categories, such as gender is divided into male and female. The types and number of categories can be appropriately adjusted and supplemented according to the text data situation.

[0028] 2.4 Match the preprocessed text description data and audio spectrum data to construct a text-audio dataset.

[0029] Step 3: Based on the Clap model infrastructure, the feature extraction encoders for audio and text channels are improved to achieve feature extraction of text and audio input data. The key steps are as follows:

[0030] 3.1 For the audio channel, the Mel spectrum is used as input, and the AST model is used to replace the traditional CNN model as the architecture of the Audio Encoder. The audio features are mapped to an embedding vector of fixed dimension through a fully connected layer so as to perform comparative learning with the output of the text channel in the same embedding space.

[0031] 3.2 For the text channel, the standardized text metadata is used as input, and the BERT model is used as the architecture of TextEncoder. The output of Transformer is pooled to generate a fixed-length text vector representation. The pooled text vector is mapped to the same vector space as the audio embedding through a fully connected layer to ensure the embedding alignment between different modalities.

[0032] 3.3 The audio and text data are processed by the audio encoder and text encoder respectively. After the embedded representations of the audio and text are obtained, the similarity matrix and symmetric cross entropy loss are calculated to ensure that the similarity between different modalities is effectively learned and optimized. In contrastive learning training, the processed audio is represented as X a , whose dimension is X a ∈R F×T , where F represents the number of spectral components and T represents the number of time components; text data is represented by X t Representation; In each batch size N, the audio-text pair is represented as {X a , X t} i The specific steps are as follows:

[0033] 3.3.1 Using f a (.) indicates the audio encoder, with f t (.) represents the text encoder, for the same batch of N:

[0034]

[0035]

[0036] here is the representation of audio, with dimension V; It is the representation of text, with dimension U.

[0037] 3.3.2 To map the representations of audio and text into a common multimodal space, we apply a learnable linear projection:

[0038] E a =L a (X a )

[0039] E t =L t (X t )

[0040] Among them, E a ∈R N×d and E t ∈R N×d is the mapped audio and text representation, L a and L t are the linear transformations of audio and text respectively.

[0041] 3.3.3 Getting audio and text embedded in E a and E t Then, calculate their similarity:

[0042] C = τ(Et ·E a T )

[0043] Among them, τ is the temperature parameter, which is used to adjust the range of the output. The generated similarity matrix C∈R N×N , the N elements on the diagonal represent the correct audio-text pairings, and the N elements off the diagonal represent the correct audio-text pairings. 2 -N elements represent unmatched pairs.

[0044] 3.3.4 The loss function is defined as:

[0045] L=λ·l text (C)+(1-λ)·l audio (C)

[0046] Among them l k represents the loss calculated along the text and audio axes respectively, and its form is:

[0047]

[0048] 3.4 The feature representations of the text channel and the audio channel output are concatenated to obtain a multimodal combined feature representation. A linear layer is added as a classifier after the improved Clap model, followed by a Softmax activation function, and the cross entropy loss is used as the loss function.

[0049] Step 4, ClapAST model training, maximizes the semantic alignment between audio and text modalities, enabling the model to bring related audio and text closer and push irrelevant ones away in the shared embedding space.

[0050] The key steps are as follows:

[0051] 4.1 Divide the text-audio dataset into training set, validation set and test set according to a certain ratio. The training set is used for model training, the validation set is used to evaluate the training effect of the model to prevent overfitting, and the test set is used to detect the migration and generalization capabilities of the model.

[0052] 4.2 Initialize model parameters, input training set and validation set to train and optimize model parameters.

[0053] Step 5, testing of the respiratory sound recognition model, inputs the pre-divided test set into the trained respiratory sound recognition model, comprehensively examines the performance of the model on unseen data, and focuses on verifying its generalization ability. In the test experiment, by comparing the recognition results of the respiratory sound features in the test set, observe whether the model can achieve good recognition effects on different respiratory sound data sets. This process evaluates the versatility of the model through multi-dataset experiments, examines its adaptability under various background conditions, verifies the model's response to various changes in respiratory sound features, and provides a basis for the feasibility of the model in different clinical application scenarios.

[0054] Beneficial Effects

[0055] For the recognition of electronic lung sound auscultation audio signals, the present invention proposes a method for electronic respiratory sound auscultation signal recognition based on multimodality and deep learning. Compared with previous studies that mainly rely on a single audio channel, this method, based on multimodal data fusion, combines audio signals and text descriptions, so that the model can obtain a richer expression of disease characteristics, thereby reducing the single-channel model's dependence on specific modal information, improving the consistency of diagnostic results, and significantly improving the model's comprehensive perception and recognition capabilities of disease characteristics. In addition, this method not only improves the accuracy of lung disease diagnosis, but also helps to enhance the application value of electronic stethoscope equipment, providing strong technical support for the early detection and prevention of lung diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of a method for identifying respiratory sound electronic auscultation signals based on multimodality and deep learning proposed by the present invention

[0057] Figure 2 The overall structure of the ClapAST model, where the audio channel uses the AST model as the encoder

[0058] Figure 3 Overall structure diagram of the audio channel AST model

[0059] Figure 4 Experimental results based on ICBHI2017 dataset training

[0061] Figure 5 Training set / test set division and distribution of ICBHI2017 dataset

[0062] Figure 6 Standard description of types and categories of text metadata DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described in detail below so that those skilled in the art can understand the present invention. However, the protection scope of the present invention is not limited to the scope of the embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

[0064] Taking the audio and text data analysis of the electronic signal of lung sound auscultation of ICBHI2017 data set as an example, the method is further described in detail in conjunction with the accompanying drawings. The data set contains 6898 respiratory cycles, lasting about 5.5 hours, and each respiratory cycle is annotated as one of four categories: normal, moist rales, wheezes, and both (moist rales and wheezes). At the same time, for most respiratory records, there are corresponding collection objects, equipment and other text information records. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0065] Step 1: text and audio data collection, that is, under the accompaniment and guidance of professional medical staff, the electronic signal data of respiratory sounds of the experimenters are collected at a specific collection position through the electronic stethoscope equipment equipped by the researchers, and various text description data are obtained.

[0066] Step 2: Preprocess the text and audio data. The key steps are as follows:

[0067] In the first step, all breathing records are resampled to 16 Hz, and the audio data is segmented according to the breathing cycle. The shorter data is replicated and enhanced using a gradual in-and-out method, and the longer data is truncated.

[0068] In the second step, the audio waveform is converted into a 128-dimensional logarithmic Mel filter bank (Fbank) feature sequence using a window and overlap size of 25ms and 10ms, and standard normalization is applied to the spectrogram. The human ear's perception of pitch is nonlinear. In order to characterize the human ear's linear perception of pitch, we need the Mel scale to map Hertz (Hz) to Mel (mel). The specific preprocessing process is as follows:

[0069] First, the audio signal is converted into a spectrogram by short-time Fourier transform (STFT), and the STFT calculation is performed on the time signal x(t) to obtain the spectrum X(f); then the amplitude of the STFT is squared to obtain the power spectrum |X(f,t)| 2 ; Next, a set of bandpass filters is used to convert the frequency axis to the Mel scale; the output value of each filter is obtained by weighting the Mel filter bank, that is, the Mel spectrum; finally, the logarithm of the Mel spectrum is taken to obtain the logarithmic Mel spectrum.

[0070] The third step is to determine the classification category of the data and standardize the format of the text description data of different categories. There are four categories of breath sounds: normal, moist rales, wheezes, and both (moist rales and wheezes). The descriptive text is divided into four types (age, gender, collection location, and collection device). Each type is divided into 2-6 categories, specifically gender (male, female), age (adult, minor), collection location (trachea, L / R front, L / R back, L / R outside), and collection device (Meditron, LittC2SE, Litt3200, AKGC417L).

[0071] The fourth step is to generate a unified text description for the audio according to the above classification standards. For example, if the metadata is "adult", "male", "L front", and "Meditron", the generated text is "This record was recorded from the left anterior chest of an adult male patient using a Meditron stethoscope". The preprocessed text description data and audio spectrum data are matched to construct a text-audio dataset.

[0072] Step 3: Based on the Clap model infrastructure, the feature extraction encoders for audio and text channels are improved, and finally feature extraction of text and audio input data is realized. The key steps are as follows:

[0073] In the first step, for the audio channel, Mel spectrum is used as input, and the AST (Audio Spectrum Transformer) model is used to replace the traditional CNN model as the architecture of the Audio Encoder. The audio features are mapped to an embedding vector of fixed dimension through a fully connected layer so that comparative learning can be performed with the output of the text channel in the same embedding space.

[0074] In the second step, for the text channel, the standardized text metadata is used as input, the BERT model is used as the architecture of TextEncoder, and the output of Transformer is pooled to generate a fixed-length text vector representation. The pooled text vector is mapped to the same vector space as the audio embedding through a fully connected layer to ensure the embedding alignment between different modalities.

[0075] In the third step, the audio and text data are processed by the audio encoder and text encoder respectively. After the embedded representation of the audio and text is obtained, the similarity matrix and symmetric cross entropy loss are calculated to ensure that the similarity between different modalities is effectively learned and optimized. In contrastive learning training, the processed audio is represented as X a , whose dimension is X a ∈R F×T, where F represents the number of spectral components and T represents the number of time components. t In each batch size of 8, the audio-text pair is represented as {X a , X t} i The specific steps are as follows:

[0076] 3.3.1 Using f a (.) indicates the audio encoder, with f t (.) represents the text encoder, which represents a batch of data:

[0077] 3.3.2 Mapping the representations of audio and text into a common multimodal space, applying a learnable linear projection: E a =L a (Xx), E t =L t (X t )

[0078] 3.3.3 Getting audio and text embedded in E a and E t Then, calculate their similarity: C = τ(E t ·E a T )

[0079] 3.3.4λ is set to 0.5, and the loss function is defined as: L = 0.5 (l text (C)+l audio (C)

[0080] In the fourth step, the feature representations of the text channel and the audio channel output are concatenated to obtain a multimodal combined feature representation. A linear layer is added after the ClapAST model as a classifier, followed by a Softmax activation function, and the cross entropy loss is used as the loss function.

[0081] Step 4, according to the method for electronic auscultation signal recognition of respiratory sounds based on multimodality and deep learning in claim 1, characterized in that the method is based on ClapAST model training, and the ClapAST architecture model is mainly developed around contrastive learning, and by maximizing the semantic alignment between audio and text modalities, the model can bring related audio and text closer and push irrelevant ones away in the shared embedding space. The key steps are as follows:

[0082] In the first step, the text-audio dataset was divided into training and test sets in a ratio of 60% / 40%. Finally, the training set included a total of 539 recordings collected from 79 patients, including 1215 wet rales, 501 wheezes, 363 wet rales and wheezes, and 2063 normal breathing cycles. The test set included 381 records from 49 patients, with a total of 649 wet rales, 385 wheezes, 143 wet rales and wheezes, and 1579 normal breathing cycles.

[0083] The second step is to initialize the model parameters, then input the training set into the model, perform iterative training, and continuously adjust and update the model parameters through the optimization algorithm to gradually optimize them. The specific steps are as follows:

[0084] 4.2.1 The maximum length of the text description is limited to 64 tokens to avoid text truncation and ensure the integrity of the model input.

[0085] 4.2.2 During the fine-tuning process, the Adam optimizer is used, the initial learning rate is 5e-5, and the cosine annealing schedule is used to adjust the learning rate.

[0086] 4.2.3 The training cycle was set to 200, the training batch size was 8, and experiments with 5 different random seeds were conducted to reduce the result fluctuations caused by random initialization.

[0087] Step 5, testing of the respiratory sound recognition model, input the pre-divided test set into the trained respiratory sound recognition model, comprehensively examine the performance of the model on unseen data, and focus on verifying its generalization ability. The test set includes 381 records from 49 patients, a total of 649 wet rales, 385 wheezes, 143 wet rales and wheezes, and 1579 normal breathing cycles.

[0088] In the test experiment, by comparing the recognition results of the respiratory sound features in the test set, we observed whether the model can achieve good recognition effects on different respiratory sound datasets. Finally, the experimental results on the ICBHI2017 dataset showed that the method proposed in this paper achieved an accuracy of 87.53%, which is 2.41% higher than the previous SOTA method, indicating that the model has made significant progress in the task of respiratory sound signal recognition. In particular, the enhancement of minority categories is very significant, especially in samples located on the right chest, where the score increased by 6.23%. This result shows that combining text data not only improves the overall performance, but also effectively improves the performance of the model on categories with less data, reflecting the value of opening up text channels in identifying minority samples.

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and variations can be made without departing from the technical principles of the present invention. For example, in the improved method, the selection of the feature network is not limited to the mentioned network, and other networks suitable for the required processing of audio data can be selected according to the method proposed by the present invention; the improved method is not limited to the model itself, and other similar models are also applicable, such as Whisper, Wav2CLIP, AudioCLIP, etc.; the model can not only process lung sound electronic auscultation signal data, but also be applied to other similar types of medical audio signal data; these improvements and variations should also be regarded as the protection scope of the present invention.

Claims

1. A method for electronic auscultation signal recognition of respiratory sounds based on multimodality and deep learning. The original Clap model is improved in terms of network structure, parameter setting and calculation process to make it suitable for electronic auscultation signal recognition of respiratory sounds with text data description. The new model is named ClapAST. It is characterized by: It mainly includes the following key steps: text and audio data collection, data preprocessing and data enhancement, construction of text-audio data pairs, construction of feature extraction models, text-audio feature splicing, respiratory sound recognition model training, model evaluation and tuning.

2. According to claim 1, a method for identifying respiratory sound electronic auscultation signals based on multimodality and deep learning, characterized in that: Text and audio data collection, that is, under the accompaniment and guidance of professional medical staff, the electronic signal data of respiratory sounds of the experimenters are collected at a specific collection position through the electronic stethoscope equipment equipped by the researchers, and the corresponding various text metadata are obtained.

3. According to claim 1, a method for electronic auscultation signal recognition of respiratory sounds based on multimodality and deep learning is characterized in that data preprocessing is performed on text and audio data, and the key steps are as follows: 3.1 Resample all breathing records to a uniform frequency, segment the audio data according to the breathing cycle, use a gradual in-out method to copy and enhance shorter data, and truncate longer data. 3.2 Use short-time Fourier transform and Mel spectrum conversion to convert the audio waveform into a logarithmic Mel filter bank (Fbank) feature sequence, and apply standard normalization on the spectrogram. 3.3 Standardize and unify the formats of text metadata of different categories, and generate text description data with a unified format. 3.4 Match the preprocessed text description data and audio spectrum data to construct a text-audio dataset.

4. The method for identifying respiratory sound electronic auscultation signals based on multimodality and deep learning according to claim 1, characterized in that: Based on the Clap model infrastructure, the feature extraction encoders for audio and text channels are improved, and finally feature extraction of text and audio input data is realized. The key steps are as follows: 4.1 For the audio channel, Mel spectrum is used as input, and the AST (Audio Spectrum Transformer) model is used to replace the traditional CNN model as the architecture of AudioEncoder. The audio features are mapped to an embedding vector of fixed dimension through a fully connected layer so as to perform comparative learning with the output of the text channel in the same embedding space. 4.2 For the text channel, the standardized text metadata is used as input, and the BERT model is used as the architecture of TextEncoder. The output of Transformer is pooled to generate a fixed-length text vector representation. The pooled text vector is mapped to the same vector space as the audio embedding through a fully connected layer to ensure the embedding alignment between different modalities. 4.3 The audio and text data are processed by the audio encoder and text encoder respectively. After obtaining the embedded representation of audio and text, the similarity matrix and symmetric cross entropy loss are calculated to effectively learn and optimize the similarity between different modalities. 4.4 The feature representations of the text channel and the audio channel output are concatenated to obtain a multimodal combined feature representation. A linear layer is added as a classifier after the improved Clap model, followed by a Softmax activation function, and the cross entropy loss is used as the loss function.

5. The method for identifying respiratory sound electronic auscultation signals based on multimodality and deep learning according to claim 1, characterized in that: Based on ClapAST model training, the ClapAST model mainly revolves around contrastive learning. By maximizing the semantic alignment between audio and text modalities, the model can bring related audio and text closer and push irrelevant ones away in the shared embedding space. The key steps are as follows: 5.1 Divide the text-audio dataset into training set, validation set and test set according to a certain ratio. The training set is used for model training, the validation set is used to evaluate the training effect of the model to prevent overfitting, and the test set is used to detect the generalization ability of the model. 5.2 Initialize model parameters, input training set and validation set to train and optimize model parameters.

Images