Cardiopulmonary disease AI diagnosis method and system based on sound stethoscope

Abstract

The invention relates to a cardiopulmonary disease AI diagnosis method and system based on a sound stethoscope. An existing digital stethoscope has a great improvement space in product sound definition and AI algorithm accuracy and precision. The heart and lung sound is collected through the collection module, the heart and lung sound is converted into a digital form through the analog-to-digital converter and stored, the stored heart and lung sound data is subjected to traditional time-frequency noise reduction processing and deep learning algorithm noise reduction processing, and after a noise-reduced stethoscope sound signal is obtained, the noise-reduced audio signal is subjected to time-domain data enhancement; after the time domain enhancement data is obtained, converting the time domain enhancement data into Mel map image data, and performing spectrum enhancement by using a SpecAugent module; and finally, AI algorithm model learning of the sound signals is carried out, the AI algorithm model learning comprises construction of a heart sound and lung sound data set, a heart sound algorithm model and a lung sound algorithm model, and heart sound and lung sound data collected by the stethoscope are output for diagnosis classification through a classification algorithm of enhanced image data. According to the method, the clinician level and even higher algorithm accuracy can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of electronic stethoscopes and processing technology for collecting cardiopulmonary sound data, in particular to an AI diagnosis method and system for cardiopulmonary diseases based on a sound stethoscope. Background Art

[0002] Cardiovascular disease is recognized as the leading cause of death worldwide, with approximately one-third of global deaths being cardiovascular related. In addition to clinical manifestations, specific clinical findings can aid in the treatment and prevention of cardiovascular disease, which may initially present as pulmonary lesions, and therefore, accurate cardiopulmonary auscultation is essential to establish an accurate diagnosis. The stethoscope is one of the most powerful tools available to physicians. First invented in 1818 by French physician René Laennec, the stethoscope is an acoustic device that transmits sound from a chest piece to the ear of the stethoscope through a hollow tube filled with air.

[0003] Lung sounds can be used as an indicator to judge the severity of COPD at the first visit (see the paper "Lung sounds can be used as an indicator to judge the severity of COPD at the first visit"). Chronic obstructive pulmonary disease (COPD) seriously endangers human health. Recent epidemiological surveys show that the total number of patients in China is about 99.9 million, and the number of new cases has reached 717,345. At the same time, more than 63% of these patients have been diagnosed with irreversible disability due to long-term damage to their respiratory function, which has constituted a major disease burden in my country. Early screening and early diagnosis have obvious therapeutic value. In addition, lung auscultation is non-invasive, has no radiation, and the patient basically does not feel uncomfortable. It is an important part of the physical examination of respiratory diseases. The clinical significance of the COPD electronic stethoscope in primary clinics is very important (in fact, our data also has the largest number of COPD cases, with a data proportion of more than 80%).

[0004] Traditional cardiopulmonary sound data noise reduction algorithms can achieve a signal-to-noise ratio gain of 5-10dB in simple noise environments, such as environments that only include fixed noise and burst noise. However, they do not perform particularly well in more complex environments, including those with multiple noise sources such as mixed human voices and outdoor motor vehicles. The use of stethoscopes in real clinics will affect the collection of cardiopulmonary sound data, directly resulting in doctors being unable to give diagnostic results, which will greatly reduce the clinic doctors' experience of using stethoscope products.

[0005] The emergence of digital stethoscopes in recent years converts sound into electronic signals, which can be further amplified for optimal listening. These electronic signals can be further processed and processed by digital algorithms. The diagnostic capabilities provided by digital stethoscopes to doctors can help change patient care management. In addition to providing higher accuracy and audibility advantages than traditional stethoscopes, digital stethoscopes are also being studied in the detection of AI algorithms for various diseases such as heart and lung sounds. However, existing digital stethoscopes still have a lot of room for improvement in terms of product sound clarity and AI algorithm accuracy. The main goal of this invention is to create high-quality, pure heart and lung sound quality for commercial-grade products and algorithm accuracy that reaches the level of clinicians or even higher. Summary of the invention

[0006] In response to the above-mentioned problems, the present invention provides an AI diagnosis method and system for cardiopulmonary diseases based on a sound stethoscope, which improves the accuracy of auscultation and can reach the level of clinicians or even higher algorithm accuracy.

[0007] To this end, the present invention adopts the following technical scheme: an AI diagnostic method for cardiopulmonary diseases based on a sound stethoscope, wherein cardiopulmonary sounds are collected by an acquisition module, converted into digital form by an analog-to-digital converter and stored, and the method is characterized in that the stored cardiopulmonary sound data are subjected to traditional time-frequency noise reduction processing and deep learning algorithm noise reduction processing, and after obtaining the noise-reduced stethoscope sound signal, the noise-reduced audio signal is subjected to time-domain data enhancement, and after obtaining the time-domain enhanced data, it is converted into Mel-atlas image data and spectral enhancement is performed using the SpecAugment module;

[0008] Finally, the AI ​​algorithm model of sound signals is learned. The first step is to build heart sound and lung sound datasets, including self-owned and cooperative datasets, open source heart sound datasets and open source lung sound datasets. The heart sound algorithm model has two categories: normal and abnormal (four abnormalities are classified into one category), and the lung sound algorithm model has eight categories: normal and the remaining seven disease abnormalities.

[0009] Among them, the self-owned and cooperative data sets were collected through cooperation with professional doctors, tertiary hospitals and research institutes. A total of 15 patients with heart diseases (more than 1,200 heart sound data), with annotation standards referring to the diagnosis and treatment guidelines for heart valve diseases and open source data annotation categories; 17 patients with lung diseases (more than 1,500 lung sound data), with annotation standards referring to the clinical diagnosis and treatment guidelines for lung diseases at the grassroots level in China and open source data annotation categories;

[0010] The data collected by the heart sound and lung sound datasets are denoised and enhanced to obtain the final training Mel-atlas image dataset, which is then divided according to a certain ratio to obtain the training set, test set, and validation set for machine learning. Finally, the heart sound and lung sound data collected by the stethoscope are output for diagnostic classification through the classification algorithm of the enhanced image data.

[0011] Preferably, the classification algorithm for the enhanced image data is an improved VIT algorithm model framework CNN-Vit, which combines the advantages of CNN and ViT feature extraction, including encoder, bottleneck, decoder and skipconnection. The algorithm process starts with two 3×3 convolutional layers in the encoder, aiming to extract local features of the image, followed by a standard encoder and a maximum pooling layer based on convolution. Through a symmetrical design, the features from the encoder contribute to the upsampling process in the decoder each time. In the decoder. The 2×2 deconvolution layer halves the feature channels and merges these channels with the corresponding encoder features through skipconnection, thereby retaining key information. The encoder and decoder are connected to two core Vit units through a bottleneck before.

[0012] Preferably, the SpecAugment module mainly processes as follows: first, the frequency data of the sound signal is converted into a Mel-scale spectrum, n_fft=512 (discrete Fourier transform window length), hop_length=512 (window movement length, windows do not overlap), fmax=8000 (maximum frequency 8000Hz), the average sound signal length is 10.23s, the time step number=10.23x16000 (sampling rate) / 512=320, and the logarithm of the Mel spectrum is calculated to calculate the Mel frequency cepstral coefficient MFCC feature, the highest is 80dB; then proceed Frequency masking is performed to mask the frequency channels [f0, f0+f), where f is selected from a uniform distribution from 0 to a frequency masking parameter F=30, and f0 is selected from (0, ν-f), where ν=128 is the number of Mel filter channels; finally, time masking is performed to mask t consecutive time steps [t0, t0+t), where t is selected from a uniform distribution from 0 to a time masking parameter T=30, and t0 is selected from [0, τ-t), where τ=320 is the number of time steps; finally, the logarithmic Mel maps of the frequency-masked and time-masked data are converted into image data as training, testing, and validation data for machine learning.

[0013] Preferably, the deep learning algorithm noise reduction processing process includes the construction of a clean heart and lung sound data set and a noise data set: the clean heart and lung sound data set includes self-collected data, an open source heart sound data set and an open source lung sound data set, the self-collected data includes more than 10,000 heart and lung sound data (each no more than 30 seconds) collected by a stethoscope in a night environment (actual monitoring clinic environment volume below 20dB) and in a daytime environment (actual monitoring clinic environment volume below 40dB) The heart and lung sound data (each no more than 30 seconds), the total time exceeds 100 hours, and the actual collection data comes from various groups of people;

[0014] The noise dataset consists of two parts. One part is the real clinic environment noise collected by the stethoscope in the daytime noise environment and the environmental sound data collected by the stethoscope in the night environment, with a total time of more than 50 hours. The other part is to select multiple different categories of audio from the open source datasets Audioset and Freesound and play them to the stethoscope through the speaker, with a total time of more than 20 hours.

[0015] The noisy heart and lung sound data is obtained by randomly synthesizing the clean heart and lung sound data and the noise data set. The obtained data set is subjected to denoising and noise removal processing. The steps are as follows:

[0016] (1) Divide the synthesized noisy heart and lung sound data and the clean heart and lung sound data into training, testing and validation ratios;

[0017] (2) Perform short-time Fourier transform (STFT) on each data with a short-time window of 20 ms, a minimum frequency of 50 Hz, and an overlap of 50% to obtain the frequency domain data of the data (including the real and imaginary data of the frequency domain graph);

[0018] (3) Extract the frequency domain data through the ERB equivalent rectangular bandwidth feature extractor and the complex feature extractor;

[0019] (4) The data extracted in step 3 are temporally encoded by the ERB self-attention encoder and the imaginary part self-attention decoder respectively. The input of the encoder is the ERB feature + temporal feature (imaginary feature + temporal feature). The temporal feature is temporally encoded according to the order of the short time frame window.

[0020] (5) Perform cross attention, which is to associate the ERB encoder and the imaginary decoder to learn the parameter matrix of the temporal feature correlation between the real and imaginary features of the spectrum;

[0021] (6) synthesizing and filtering the ERB coded and imaginary coded data passed through the decoder, and splicing them into frequency domain data of the output data;

[0022] (7) Convert the signal represented in the frequency domain back to the time domain to obtain denoised audio data.

[0023] Preferably, the self-attention function of the self-attention encoder in step (4) is an embedded query of ERB features + temporal features (imaginary features + temporal features) and a set of key-value pairs mapped to the output, where Q, K, V (query, key, value) are all self-embedded vector matrices, d is the dimension size of the embedded vector, and the self-attention calculation formula is as follows:

[0024]

[0025] Preferably, in step (1), the synthesized noisy heart and lung sound data and the clean heart and lung sound data are divided into training, testing and validation according to a total ratio of 70% / 15% / 15%.

[0026] As a preference, the data of the heart and lung sounds were collected from 7 persons, and the age structure distribution was as follows: 1 person under 20 years old, 2 persons between 20 and 40 years old, and 4 persons over 40 years old.

[0027] Preferably, the conventional time-frequency noise reduction processing method is as follows:

[0028] (a) The heart and lung sound data collected by the stethoscope are divided into frames with 160 sampling points per frame (the maximum sampling rate is 44100 Hz), and then the energy sum (sum of square amplitude) of each frame is calculated, and a discrete Fourier transform (FFT) is performed, including the function values ​​of 256 points, and the energy spectrum, amplitude spectrum, logarithmic energy spectrum and logarithmic amplitude spectrum are calculated at the same time, and the noise spectrum is estimated based on these data;

[0029] (b) The a priori and a posteriori SNR of each frame. The so-called a priori SNR is the ratio of the power spectrum of the pure signal to the power spectrum of the noise (ideal value), while the a posteriori SNR is the ratio of the power spectrum of the noisy signal to the power spectrum of the noise. The a priori SNR (estimated value) and the instantaneous SNR calculated in the previous frame are used as the result of the a priori SNR. The calculated SNR is used to update the three features of the LRT (Likelihood Rate Test) mean feature, spectrum flatness and spectrum difference.

[0030] (c) Calculate the probability of speech and noise, and update the noise spectrum;

[0031] (d) Finally, the Wiener filter is calculated to obtain the real and imaginary correction coefficients (frequency domain) of the current frame through the calculated signal-to-noise probability, and the best noise reduction audio data is obtained through IFFT (inverse Fourier transform) and frame synthesis operations.

[0032] The present invention also discloses an AI diagnosis system for cardiopulmonary diseases based on a sound stethoscope, comprising a cardiopulmonary stethoscope sound signal acquisition module, a sound signal processing and noise reduction module, a sound signal Mel-spectrogram enhancement module and a sound signal AI algorithm model learning module. The sound signal AI algorithm model learning module outputs a diagnostic classification of the collected heart sound and lung sound data according to the above method.

[0033] The present invention enhances the cardiopulmonary sound data obtained by denoising and converts it into Mel-atlas image data, and then uses a self-constructed classification algorithm for the enhanced image data to perform accurate diagnosis and classification, thereby achieving the diagnostic accuracy level of clinicians or even higher.

[0034] The present invention further reduces noise and removes noise from sound signals by adopting algorithms of traditional and deep learning models. Compared with noise data and traditional methods, the signal-to-noise ratio is significantly improved, and it can extract cardiopulmonary sounds in more complex external environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the CrossFilterSound algorithm framework.

[0036] Figure 2 Schematic diagram of the self-attention mechanism and encoder.

[0037] Figure 3 Schematic diagram of cross attention and ERB (imaginary) encoder.

[0038] Figure 4 Convert unenhanced audio data to spectrogram data.

[0039] Figure 5 Convert the enhanced audio data to spectrogram data.

[0040] Figure 6 To enhance the classification algorithm of image data.

[0041] Figure 7 The difference between multi-head attention and grouped attention. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with specific embodiments.

[0043] The present invention discloses an AI diagnosis system for cardiopulmonary diseases based on a sound stethoscope, including a cardiopulmonary stethoscope sound signal acquisition module, a sound signal processing and noise reduction module, a sound signal Mel-spectrogram enhancement module, and a sound signal AI algorithm model learning module. The main components and algorithm principles of each module are as follows:

[0044] 1. Cardiopulmonary stethoscope sound signal acquisition module:

[0045] The sound signal acquisition module is designed to optimize the sensor characteristics for collecting heart, lung and chest sound signals, and is mainly composed of two parts: a transmitter and a receiver.

[0046] The transmitter consists of the following components: (i) front-end circuit and sensor module: preamplifier, filter and power amplifier with variable gain (ii) microcontroller and wifi module (iii) PC ​​connection module.

[0047] 1) The front-end circuit is mainly for signal acquisition and preprocessing: the sensor system includes a microphone (containing two diaphragms) and a piezoelectric sensor. The stethoscope diaphragm detects the sound signal and then transmits it to another diaphragm in the microphone and then to the piezoelectric sensor. The two diaphragms separated by the airway may cause excessive environmental noise signals and inaccurate electrical signal transmission.

[0048] In a piezoelectric transducer, the sound collected by the stethoscope diaphragm causes the crystal material to distort, which in turn generates an electrical signal. The distortion of the crystal generates an electrical signal that may not truly capture the original sound determined by the diaphragm. The diaphragm of the stethoscope is suspended in a nominal capacitance field, where the capacitance changes according to the sound pressure formed by the heart sounds. When the piezoelectric material is subjected to force, an electric charge is generated on the surface. This charge is amplified and the impedance is transformed by the charge amplifier and measurement circuit to become an electrical output proportional to the external force. Piezoelectric sensors are used to measure non-electrical physical quantities that are converted into electricity by force and energy. Its advantages are wide bandwidth, high sensitivity, high signal-to-noise ratio, simple structure, reliable operation and light weight. The disadvantage is that the DC response of the output is poor, and a high input impedance circuit or charge amplifier is required to overcome this defect. The results of the sensor are sent to the signal preprocessing circuit for

[0049] The signal preprocessing circuit consists of three parts, namely the primary amplification circuit, the filtering circuit and the secondary amplification circuit. The role of the signal preprocessing circuit is to adjust the signal from the sensor through a series of amplification and filtering so that it meets the subsequent A / D (analog signal / digital signal) sampling requirements and improves the signal-to-noise ratio. The circuit is mainly designed based on the operational amplifier LM741. The preamplifier is used to increase the low signal of the condenser microphone to the line level for further amplification, and the operational amplifier LM741 is used to design the preamplifier. Its gain is 20, which is calculated by the feedback resistor value. The output of the preamplifier is fed to an active low-pass filter with a cutoff frequency of 100Hz and 1000Hz to pass heart sounds and lung breathing sounds and reduce background sounds. The frequency is selected by selecting the capacitor value. The gain of the filter is 1.6. The output signal of the filter is processed by a power amplifier to provide the power required to drive the headphones for further amplification. The LM386 circuit is an audio amplifier designed mainly for low-voltage consumer applications, which can provide voltage and current gain for the signal. Therefore, a power amplifier with variable gain is designed with the help of the operational amplifier LM386. By varying the input provided to the amplifier through a potentiometer, the gain can be varied.

[0050] 2) Microcontroller and wifi module:

[0051] The output of the signal preprocessing circuit is converted into digital form through ADC MCP4822 (Analog to Digital Converter). The built-in successive approximation 12-bit ADC of the PIC18f2423 microcontroller is used. Some parameters of PIC18f2423 are as follows: 0-40MHz operating frequency; 16KB flash program memory; 768 bytes of data memory; 12-bit ADC (10 input channels); SPI and USART serial communication. The wireless transmission module ESP32-S2 has an operating frequency of 2.4GHz and a data rate of 150Mbps.

[0052] 3) PC connection module: The analog signal is provided to the PC through the auxiliary input pin for storage. The audio signal is stored in the form of a .wav file.

[0053] 2. Sound signal processing, noise reduction and noise removal algorithm module:

[0054] Considering the specific situation of using a stethoscope, it is inevitable that interference noises from the external environment and the patient talking loudly will be collected and interfere with the cardiopulmonary sound signals. The present invention will use traditional and deep learning model algorithms to further reduce noise and remove noise from the sound signals.

[0055] It consists of two modules:

[0056] 1) Frequency domain Wiener filter module (traditional time-frequency noise reduction processing):

[0057] 1. Divide the WAV file collected by the stethoscope into 160 sampling points per frame (the maximum sampling rate is 44100Hz), and then calculate the energy and (amplitude square sum) of each frame. Perform a discrete Fourier transform (FFT), including the function value of 256 points, and calculate the energy spectrum, amplitude spectrum, logarithmic energy spectrum and logarithmic amplitude spectrum at the same time, and estimate the noise spectrum based on these data.

[0058] 2. Next, the a priori and a posteriori SNR of each frame are calculated. The so-called a priori SNR is the ratio of the power spectrum of the pure signal to the power spectrum of the noise (ideal value), and the a posteriori SNR is the ratio of the power spectrum of the noisy signal to the power spectrum of the noise. The result of smoothing the a priori SNR (estimated value) and the instantaneous SNR calculated in the previous frame is used as the result of the a priori SNR, and the calculated SNR is used to update the three features of the LRT (Likelihood Rate Test) mean feature, spectrum flatness and spectrum difference.

[0059] 3. Calculate the probability of speech and noise, and update the noise spectrum.

[0060] 4. Finally, the Wiener filter is calculated to obtain the real and imaginary correction coefficients (frequency domain) of the current frame through the calculated signal-to-noise probability, and the best noise reduction sound data is obtained through IFFT (inverse Fourier transform) and frame synthesis operations.

[0061] 2) CrossFilterSound deep learning algorithm noise reduction module: Traditional algorithms can achieve a signal-to-noise ratio gain of 5-10dB for simple noise environments, such as environments that only include fixed noise and burst noise. However, the effect is not particularly improved in more complex environments, including complex environments with multiple noise sources such as mixed human voices and outdoor motor vehicles. The use of stethoscopes in real clinics will affect the collection of cardiopulmonary sound data, which will directly lead to the inability of doctors to give diagnosis results, thereby greatly reducing the clinic doctors' experience of using stethoscope products.

[0062] (I) Construction of a deep learning dataset for cardiopulmonary sound denoising: The denoising open source datasets used by existing machine learning algorithms are mainly used to separate human voices from mixed sounds of various noises. They are not suitable for our product data situation: the mixture of cardiopulmonary sounds and various human voices and environmental noises. Directly using the existing trained model does not have a very good effect. Therefore, we must build a denoising dataset for our algorithm to train the algorithm model, which mainly includes clean cardiopulmonary sound data and noisy cardiopulmonary sound data.

[0063] The clean heart and lung sound datasets include the following three:

[0064] (1) More than 12,000 pieces of cardiopulmonary sound data (each no longer than 30 seconds) were collected by stethoscope at night (actual monitoring clinic ambient volume below 20 dB) and during the day (actual monitoring clinic ambient volume below 40 dB) with a total time of more than 100 hours. The actual data collection sources were 7 people (1 person under 20 years old, 2 people aged 20-40 years old, and 4 people aged 40 years old or older. The cardiopulmonary disease population was mainly concentrated in the 40-70 age group. When collecting data, more consideration was given to the age distribution of the cardiopulmonary disease population). The clean data with noise reduction does not need to be labeled by professional doctors. It only needs to monitor the ambient dB, which is relatively easy to collect.

[0065] (2) Open Source Heart Sound Dataset (PhysioNet Challenge2016): It includes a total of 3153 heart sound files with a duration of 10-60 seconds, mainly recording the heart sounds and ECG information of four locations: the aorta, pulmonary artery, tricuspid valve, and mitral valve.

[0066] (3) Open Source Respiratory Sound Database: It includes 920 heart and lung sound data from 126 patients, with a duration of 10-90 seconds. It mainly records the sound information of eight chest and lung positions: trachea, left front, right front, left rear, right rear, left side, and right side.

[0067] The noisy heart and lung sound dataset is mainly obtained by randomly synthesizing the following noisy dataset using the clean heart and lung sound dataset:

[0068] (1) The stethoscope collected real clinic environment noise (including various mixed human voices, external environmental noise, etc.) in daytime noise environment (actual monitoring clinic environment volume is above 55dB) and in night environment (actual monitoring clinic environment volume is above 45dB, and the speaker randomly plays the daytime recorded noise), with a total time of more than 50 hours;

[0069] (2) We selected audio from ten sound categories, including breathing sounds, coughing, and vehicle sounds, from the open source datasets Audioset and Freesound, and played them through speakers (to simulate a wider range of external environments) to be recorded by a stethoscope, with a total recording time of more than 20 hours.

[0070] The algorithm flow chart is as follows Figure 1 As shown, the main processing process is as follows:

[0071] 1. Divide the total number of synthesized noisy heart and lung sound data and pure heart and lung sound data (more than 15,000) into training, testing and validation ratios of 70% / 15% / 15%.

[0072] 2. Perform short-time Fourier transform (STFT) on each data with a short-time window of 20ms, a minimum frequency of 50Hz, and an overlap of 50% to obtain the frequency domain data of the data (including the real and imaginary data of the frequency domain graph).

[0073] 3. Feature extraction of frequency domain data includes two parts:

[0074] 1) ERB Equivalent Rectangular Bandwidth feature extractor: ERB simulates the selective attention of the human ear to sound frequencies by dividing the frequency range into equivalent rectangular bandwidths. This division takes into account the different perception abilities of the human ear to high-frequency and low-frequency sounds, resulting in different frequency resolutions in the high-frequency and low-frequency bands. The frequency bins of the spectrogram are logarithmically compressed into 32 ERB bands, and only the real-valued part of the STFT transformed data is processed;

[0075] 2) Complex Feature Extractor: The imaginary part of the STFT data is used as the input of the imaginary encoder.

[0076] 4. ERB and imaginary self-attention encoder (main structure as Figure 2 shown):

[0077] The input of the encoder is ERB feature + time series feature (imaginary feature + time series feature), and the time series feature is encoded in the order of the short time frame window.

[0078] The self-attention function maps the embedded query of ERB features + temporal features (imaginary features + temporal features) and a set of key-value pairs to the output, where Q, K, V (query, key, value) are all self-embedded vector matrices. The self-attention calculation formula is as follows:

[0079]

[0080] Where × is the dot multiplication operator, + is the addition operator, ÷ is the division operator, concat is the concatenation operator, d is the embedding vector dimension, MLP (Multilayer Perceptron) multilayer perceptron, softmax and RMSNorm normalization formulas are as follows:

[0081]

[0082] 5. Cross-attention and ERB (imaginary number) encoder (main structure such as Figure 3 shown):

[0083] Cross attention is mainly to associate the ERB encoder and the imaginary decoder to learn the parameter matrix of the temporal feature correlation of the real and imaginary features of the spectrum.

[0084] Its input is the Q matrix of the ERB encoder and the K, V matrices of the imaginary decoder (the symmetric ones are the imaginary encoder and the ERB decoder, the inputs are the Q matrix of the imaginary encoder and the K, V matrices of the ERB decoder, Q, K, V are defined as above).

[0085] where proj ERB and proj C is the dimension of the mapping function used to align the ERB encoding and the imaginary encoding. The cross attention calculation formula is as follows:

[0086]

[0087] 6.ERB gain synthesis and imaginary part filtering:

[0088] The ERB code and the imaginary code are synthesized and filtered through the decoder data and concatenated into the frequency domain data of the output data.

[0089] 7. Inverse short-time discrete Fourier transform ISTFT:

[0090] The signal represented in the frequency domain is converted back to the time domain to obtain denoised audio data.

[0091] (III) Evaluation of noise reduction effect of CrossFilterSound algorithm:

[0092] We use the indicator: maximizing the scale-invariant signal-to-distortion ratio (SI-SDR, similar to SNR) to evaluate the quality of our deep denoising model. On average, the 15% validation set data can be improved by 15.34dB, which is a significant improvement compared to noisy data and traditional methods.

[0093]

[0094] 3. Mel spectrogram enhancement module of sound signal:

[0095] After obtaining the denoised stethoscope sound signal, in order to better learn and train the classification algorithm, the denoised audio signal is enhanced in the time domain, and the data is converted into the frequency domain for spectrum enhancement to obtain its Mel spectrum image as the input of the machine learning classification algorithm. The data enhancement mainly includes the following modules:

[0096] 1. Sound signal time domain enhancement algorithm module:

[0097] Pitch_shift module: mainly adjusts the pitch of the audio signal without changing its speed. The frequency determines the pitch. The higher the frequency, the higher the pitch. Shift the pitch of the waveform by n_steps = 6 steps. The parameter bins_per_octave is set to 12 (12 steps per semitone). n_steps = 6 steps means the pitch is shifted up by 3 tones.

[0098] time_stretch module: Change the speed of the sound without affecting the pitch. Set rate = 0.8 and the random sampling rate to 10%. Randomly sample the audio signal and reduce the sound speed to 80% of the original.

[0099] 2. After obtaining the time domain enhanced data, it is converted into Mel-level atlas image data and spectral enhancement is performed using the SpecAugment module.

[0100] The SpecAugment module mainly processes the following:

[0101] First, convert the frequency data of the sound signal into a Mel-scale spectrum, n_fft = 512 (discrete Fourier transform window length), hop_length = 512 (window movement length, windows do not overlap), fmax = 8000 (maximum frequency 8000Hz), the average sound signal length is 10.23s, and the time step number =

[0102] 10.23x16000 (sampling rate) / 512=320, and calculate the logarithm of the Mel spectrum to calculate the Mel frequency cepstral coefficient MFCC features, up to 80dB. Then perform frequency masking, mask the frequency channels [f0,f0+f), where f is selected from a uniform distribution from 0 to the frequency masking parameter F=30, and f0 is selected from (0,ν-f), where ν=128 is the number of Mel filter channels. Finally, perform time masking, mask t consecutive time steps [t0,t0+t), where t is selected from a uniform distribution from 0 to the time masking parameter T=30, and t0 is selected from [0,τ-t), where τ=320 is the number of time steps.

[0103] Finally, the logarithmic Mel-scale images of the frequency-masked and time-masked data are converted into image data as training, testing, and verification data for machine learning. The image effect is as follows: Figure 4 and Figure 5 shown.

[0104] The main goal of SpecAugment is to solve the problem of overfitting in machine learning algorithms, thereby improving the generalization ability of the model and significantly improving the performance of the sound algorithm.

[0105] 4. AI algorithm model learning module for sound signals.

[0106] 1. Construction of heart sound and lung sound dataset:

[0107] High-quality heart disease audio and lung disease audio data annotated by professional doctors are the core elements of machine learning algorithms. For this purpose, we construct professionally annotated heart and lung sound data as follows:

[0108] (I) Proprietary and cooperative data sets: Through cooperation with professional doctors, tertiary hospitals and research institutes, a total of 15 patients with heart diseases (more than 1,200 heart sound data) were collected, and the annotation standards were based on the diagnosis and treatment guidelines for heart valve disease and the open source data annotation categories; 17 patients with lung diseases (more than 1,500 lung sound data), and the annotation standards were based on the clinical diagnosis and treatment guidelines for lung diseases at the grassroots level in China and the open source data annotation categories.

[0109] (ii) Open source heart sound dataset (PhysioNet Challenge2016) 3153 records.

[0110] (III) Open source Respiratory Sound Database: 920 records.

[0111] There are two categories of heart sound algorithm models: normal and abnormal (four abnormalities are classified into one category), and eight categories of lung sound algorithm models: normal and the remaining seven disease abnormalities. The data set is passed through the denoising module (module 2 above) and the enhancement module (module 3 above) to obtain the final training Mel-atlas image data set (more than 4,400 heart sounds and more than 2,500 lung sounds), which is divided according to the ratio of 70% / 15% / 15% to obtain the training set, test set and validation set for machine learning.

[0112] 2. CNN-transformer (CNN-Vit) classification algorithm based on enhanced image data: Vit (Vision in Transformer) technology extends the Transformer in the NLP field to the CV field and has achieved success in multiple visual tasks. Its advantages are obvious compared to traditional image convolution methods: global modeling, parallel processing, strong scalability, and unified architecture, which are all very good features for future algorithm optimization and iteration. However, it also has obvious shortcomings: large amounts of data and computing power requirements, lack of local feature extraction (global information extraction is better), low initial performance of transfer learning, higher computational complexity and inference delay.

[0113] To this end, we proposed an improved VIT algorithm model framework CNN-Vit, which is based on heart sounds and lung sounds to model (the basic network frameworks of the two models are roughly the same, and the network and training parameters are optimized for their respective tasks). Figure 6 As shown in Figure 2, the framework combines the advantages of CNN and ViT feature extraction and consists of computing units such as encoder, bottleneck, decoder, and skip connection.

[0114] The algorithm process starts with two 3×3 convolutional layers in the encoder, which aims to extract local features of the image, followed by a standard convolution-based encoder and a maximum pooling layer. Through a symmetrical design, the features from the encoder contribute to the upsampling process in the decoder each time. In the decoder, the 2×2 deconvolution layer halves the feature channels and merges these channels with the corresponding encoder features through skip connection, thereby retaining key information.

[0115] The encoder and decoder previously connected two core Vit units through a bottleneck. We introduced a bottleneck with two transformer blocks to capture local details and global features through the effective fusion of CNN and Vison Transformer.

[0116] In each Vison Transformer unit, in order to better accelerate the inference performance and training efficiency of the model, we have also improved and optimized the architecture of the traditional VIT. Our layer normalization function is improved to RMSNorm (the formula is shown above), and the multi-head self-attention (MSA) mechanism is improved to group query attention (Group-queryAttention, the difference is as follows Figure 7 As shown, GQA divides the query head into G groups, each group shares a key and value), two layers of MLP (multi-layer perceptron) with SwiGLU as the activation function, where the SwiGLU function is as follows:

[0117]

[0118] in is element-by-element multiplication, W, V, b, c, β are learnable parameters.

[0119] 3. Final algorithm accuracy evaluation:

[0120] Our current accuracy indicators for our own and open source datasets are shown in the following table:

[0121]

[0122] From the above table, it can be seen that the proposed method is 86.8% better than the open-source heart sound method (see the paper "Classification of Heart Sounds Using Convolutional Neural Network") and the open-source lung sound method (see the paper

[0123] The accuracy of "Respiratory diseases recognition through respiratory sound with the help of deep neural network" has been greatly improved to about 95.67%. From the perspective of the value of products in actual medical scenarios, the accuracy index of general stethoscope doctors is much higher than that of general stethoscope doctors at the existing data level, indicating that the machine learning detection algorithm of biological signals has great potential and significance in primary medical care.

[0124] In practice, the four most common locations for listening to heart sounds are named after the position of the valve that is most easily heard:

[0125] Arterial area - centered on the second right intercostal space, pulmonary artery area - the second intercostal space along the left sternal edge, tricuspid valve area - the fourth intercostal space along the left sternal edge, mitral valve area - located at the apex, the fifth intercostal space on the clavicular midline. Heart sound data from four locations are collected, and then the heart sound model is used to detect whether the heart sound is abnormal. Lung sounds include eight locations: trachea, left front, right front, left rear, right rear, left side, and right side. The probability of lung disease is detected based on the lung sound model.

[0126] Report of medical interpretation of auscultation findings:

[0127] Pathological murmurs of heart sound signals usually occur during the systolic period of the heart, that is, the main pathological information is contained in the time segment from the first heart sound (S1) to the second heart sound (S2). S1 is mainly caused by the sudden closure of the mitral valve and tricuspid valve at the beginning of ventricular contraction, and the sudden tension of the valve leaflets causing vibration. The second heart sound. S2 is mainly caused by the valve vibration caused by the sudden closure of the aortic valve and pulmonary valve at the beginning of ventricular diastole. The algorithm detects abnormal heart sounds in four parts to detect the possibility of abnormalities in the first and second heart sounds, and then gives appropriate pathological explanations and diagnosis and treatment recommendations in accordance with the guidelines for the diagnosis and treatment of heart valve diseases.

[0128] The lung sound classification algorithm currently supports 8 categories: chronic obstructive pulmonary disease, pneumonia, upper respiratory tract infection, bronchiectasis, bronchiolitis, lower respiratory tract infection, asthma, and health.

[0129] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An AI diagnostic method for cardiopulmonary diseases based on a sound stethoscope, which collects cardiopulmonary sounds through an acquisition module, converts them into digital form through an analog-to-digital converter and stores them, is characterized in that The stored heart and lung sound data is subjected to traditional time-frequency denoising and deep learning algorithm denoising to obtain the denoised stethoscope sound signal. The denoised audio signal is then enhanced in the time domain. The obtained time domain enhanced data is converted into Mel-atlas image data and spectrally enhanced using the SpecAugment module. Finally, the AI ​​algorithm model of sound signals is learned. The first step is to build heart sound and lung sound datasets, including self-owned and cooperative datasets, open source heart sound datasets and open source lung sound datasets. The heart sound algorithm model has two categories: normal and abnormal (four abnormalities are classified into one category), and the lung sound algorithm model has eight categories: normal and the remaining seven disease abnormalities. The self-owned and cooperative data sets are collected through cooperation with professional doctors, tertiary hospitals and research institutes, and the data of dozens of patients with heart diseases and lung diseases are professionally classified and labeled; The data collected by the heart sound and lung sound datasets are denoised and enhanced to obtain the final training Mel-atlas image dataset, which is then divided according to a certain ratio to obtain the training set, test set, and validation set for machine learning. Finally, the heart sound and lung sound data collected by the stethoscope are output for diagnostic classification through the classification algorithm of the enhanced image data.

2. The AI ​​diagnostic method for cardiopulmonary diseases based on sound stethoscope according to claim 1, characterized in that The classification algorithm for enhanced image data is an improved VIT algorithm model framework CNN-Vit. The framework mixes the advantages of CNN and ViT feature extraction, including encoder, bottleneck, decoder and skip connection. The algorithm process starts with two 3×3 convolutional layers in the encoder, aiming to extract local features of the image, followed by a standard convolution-based encoder and maximum pooling layer. Through the symmetrical design, the features from the encoder contribute to the upsampling process in the decoder each time; in the decoder, the 2×2 deconvolution layer halves the feature channels and merges these channels with the corresponding encoder features through skip connection, thereby retaining key information. The encoder and decoder are connected to two core Vit units through a bottleneck.

3. The AI ​​diagnostic method for cardiopulmonary diseases based on sound stethoscope according to claim 1, characterized in that The SpecAugment module mainly processes as follows: first, the frequency data of the sound signal is converted into a Mel-scale spectrum, n_fft = 512 (discrete Fourier transform window length), hop_length = 512 (window movement length, windows do not overlap), fmax = 8000 (maximum frequency 8000Hz), the average sound signal length is 10.23s, the number of time steps = 10.23x16000 (sampling rate) / 512 = 320, and the logarithm of the Mel spectrum is calculated to calculate the Mel frequency cepstral coefficient MFCC feature, the highest is 80dB; then the frequency Masking, masking the frequency channels [f0,f0+f), where f is selected from a uniform distribution from 0 to a frequency masking parameter F=30, and f0 is selected from (0,ν-f), where ν=128 is the number of Mel filter channels; finally, temporal masking, masking t consecutive time steps [t0,t0+t), where t is selected from a uniform distribution from 0 to a time masking parameter T=30, and t0 is selected from [0,τ-t), where τ=320 is the number of time steps; finally, the logarithmic Mel maps of the frequency-masked and time-masked data are converted into image data as training, testing, and validation data for machine learning.

4. The AI ​​diagnostic method for cardiopulmonary diseases based on sound stethoscope according to claim 1, characterized in that The deep learning algorithm denoising process includes the construction of a clean heart and lung sound dataset and a noisy dataset: The clean heart and lung sound dataset includes self-collected data, open-source heart sound dataset and open-source lung sound dataset. The self-collected data includes more than 10,000 heart and lung sound data collected by stethoscope in night environment (actual monitoring clinic environment volume is below 20dB) (each no more than 30s) and in daytime environment (actual monitoring clinic environment volume is below 40dB) (each no more than 30s), with a total time of more than 100 hours. The actual data collection comes from various groups of people; The noise dataset consists of two parts. One part is the real clinic environment noise collected by the stethoscope in the daytime noise environment and the environmental sound data collected by the stethoscope in the night environment, with a total time of more than 50 hours. The other part is to select multiple different categories of audio from the open source datasets Audioset and Freesound and play them to the stethoscope through the speaker, with a total time of more than 20 hours. The noisy heart and lung sound data is obtained by randomly synthesizing the clean heart and lung sound data and the noise data set. The obtained data set is subjected to denoising and noise removal processing. The steps are as follows: (1) Divide the synthesized noisy heart and lung sound data and the clean heart and lung sound data into training, testing and validation ratios; (2) Perform short-time Fourier transform (STFT) on each data with a short-time window of 20 ms, a minimum frequency of 50 Hz, and an overlap of 50% to obtain the frequency domain data of the data (including the real and imaginary data of the frequency domain graph); (3) Extract the frequency domain data through the ERB equivalent rectangular bandwidth feature extractor and the complex feature extractor; (4) The data extracted in step 3 are temporally encoded by the ERB self-attention encoder and the imaginary part self-attention decoder respectively. The input of the encoder is the ERB feature + temporal feature (imaginary feature + temporal feature). The temporal feature is temporally encoded according to the order of the short time frame window. (5) Perform cross attention, which is to associate the ERB encoder and the imaginary decoder to learn the parameter matrix of the temporal feature correlation between the real and imaginary features of the spectrum; (6) synthesizing and filtering the ERB coded and imaginary coded data passed through the decoder, and splicing them into frequency domain data of the output data; (7) Convert the signal represented in the frequency domain back to the time domain to obtain denoised audio data.

5. The AI ​​diagnostic method for cardiopulmonary diseases based on sound stethoscope according to claim 4, characterized in that The self-attention function of the self-attention encoder in step (4) is the embedding query of ERB features + temporal features (imaginary features + temporal features) and a set of key-value pairs mapped to the output, where Q, K, V (query, key, value) are all self-embedded vector matrices, d is the dimension size of the embedded vector, and the self-attention calculation formula is as follows:

6. The AI ​​diagnostic method for cardiopulmonary diseases based on sound stethoscope according to claim 4, characterized in that In the step (1), the synthesized noisy heart and lung sound data and the clean heart and lung sound data are divided into training, testing and validation according to a total ratio of 70% / 15% / 15%.

7. The AI ​​diagnostic method for cardiopulmonary diseases based on sound stethoscope according to claim 4, characterized in that The data of heart and lung sounds were collected from 7 people, and the age structure was as follows: 1 person under 20 years old, 2 people aged 20-40 years old, and 4 people over 40 years old.

8. The AI ​​diagnostic method for cardiopulmonary diseases based on sound stethoscope according to claim 1 or 4, characterized in that The traditional time-frequency noise reduction processing method is as follows: (a) The heart and lung sound data collected by the stethoscope are divided into frames with 160 sampling points per frame (the maximum sampling rate is 44100 Hz), and then the energy sum (sum of square amplitude) of each frame is calculated, and a discrete Fourier transform (FFT) is performed, including the function values ​​of 256 points, and the energy spectrum, amplitude spectrum, logarithmic energy spectrum and logarithmic amplitude spectrum are calculated at the same time, and the noise spectrum is estimated based on these data; (b) The a priori and a posteriori SNR of each frame. The so-called a priori SNR is the ratio of the power spectrum of the pure signal to the power spectrum of the noise (ideal value), while the a posteriori SNR is the ratio of the power spectrum of the noisy signal to the power spectrum of the noise. The a priori SNR (estimated value) and the instantaneous SNR calculated in the previous frame are used as the result of the a priori SNR. The calculated SNR is used to update the three features of the LRT (Likelihood Rate Test) mean feature, spectrum flatness and spectrum difference. (c) Calculate the probability of speech and noise, and update the noise spectrum; (d) Finally, the Wiener filter is calculated to obtain the real and imaginary correction coefficients (frequency domain) of the current frame through the calculated signal-to-noise probability, and the best noise reduction audio data is obtained through IFFT (inverse Fourier transform) and frame synthesis operations.

9. An AI diagnostic system for cardiopulmonary diseases based on a sound stethoscope, characterized in that It includes a cardiopulmonary stethoscope sound signal acquisition module, a sound signal processing and noise reduction module, a sound signal Mel spectrogram enhancement module and a sound signal AI algorithm model learning module, wherein the sound signal AI algorithm model learning module The method according to claim 1 outputs diagnostic classification of heart sound and lung sound data collected by a stethoscope.

Images