A heart sound detection method and system based on deep learning and heterogeneous integration strategy

Abstract

The present invention relates to a heart sound detection method and system based on deep learning and heterogeneous integration strategy, and specifically to the field of medical health platform, and specifically includes the following steps: step S00: performing wavelet threshold denoising on impure heart sound signals; step S10: extracting the high-order Shannon envelope of the heart sound signal after denoising, and applying a double threshold search time gate algorithm based on a greedy algorithm and adaptive threshold optimization to perform heart sound segmentation to form heart sound segments; step S20: using a preset function to extract multi-domain features such as time-frequency domain features, MFCC features, Mel spectrum feature maps, and second-order spectrum feature maps from the heart sound segments and the complete heart sound signals; step S30: inputting the multi-domain features into a preset deep learning heterogeneous integration model for calculation; step S40: outputting the detection results. The present application can improve the accuracy of heart sound detection.

Description

Technical Field

[0001] The present invention relates to the field of medical health platforms, and in particular to a heart sound detection method and system based on deep learning and heterogeneous integration strategy. Background Art

[0002] Statistics from the World Health Organization show that the mortality rate of cardiovascular diseases has been ranked first among all diseases in recent years. The number of deaths caused by cardiovascular diseases accounts for more than 30% of the total deaths worldwide. Cardiovascular diseases are becoming the number one factor endangering human health.

[0003] Currently, doctors can diagnose the heart condition of the subject with the help of X-rays, magnetic resonance images, electrocardiograms, heart sounds, etc. Among them, heart sounds (Phonocardiogram, PCG) play an important role in early screening and daily detection of diseases due to its fast and low-cost characteristics.

[0004] Traditional heart sound diagnosis still mainly relies on on-site auscultation by doctors. Since the final diagnosis depends on the judgment of medical staff, there is a certain degree of subjectivity. Different doctors have different knowledge levels, auscultation levels, and medical experience, which may lead to different results for the same patient diagnosed by different doctors. Therefore, relying solely on doctors' diagnosis has great limitations. In addition, heart sound signals are easily affected by factors such as clinical environmental noise, body noise, channel differences, and heart rate fluctuations, which makes the heart sound analysis process very difficult. Summary of the invention

[0005] In order to overcome the defects of the prior art, the present invention proposes a heart sound detection method and system based on deep learning and heterogeneous integration strategy. On the one hand, a heart sound signal recognition model with excellent completeness and wide practicability is established to optimize the preprocessing and feature extraction process while ensuring accuracy; on the other hand, a heart sound detection platform R&D architecture design and data visualization ideas are provided.

[0006] To achieve this object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a heart sound detection method based on deep learning and heterogeneous integration strategy, the method comprising: step S00: performing wavelet threshold denoising on a non-pure heart sound signal; step S10: extracting a high-order Shannon envelope of the heart sound signal after denoising, and applying a double threshold search time gate algorithm based on a greedy algorithm and adaptive threshold optimization to perform heart sound segmentation to form heart sound segments; step S20: using a preset function to extract multi-domain features of time-frequency domain features, MFCC features, Mel spectrum feature maps, and second-order spectrum feature maps for the heart sound segments and the complete heart sound signals respectively; step S30: inputting the multi-domain features into a preset deep learning heterogeneous integration model for calculation; step S40: outputting the detection results.

[0008] In a second aspect, the present invention provides a heart sound detection system based on deep learning and heterogeneous integration strategy, the system comprising: a heart sound uploading module, used to obtain the heart sound signal uploaded by the user and store it on a server; a heart sound detection module, used to perform heart sound segmentation, multi-domain feature extraction, and then apply the preset multi-domain features to input the preset deep learning heterogeneous integration model for calculation, output the detection results and store the detection results on the server; a visualization display module, used to visualize the detection results using the Apexcharts third-party library.

[0009] The beneficial effects of the present invention are:

[0010] A heart sound detection method based on deep learning and heterogeneous integration strategy provided by the present invention realizes the phased and progressive optimization of the heart sound detection task. The specific steps include: first, based on the paper research and a large number of comparative experiments, a threshold denoising algorithm based on the coif5 wavelet function is selected to perform heart sound denoising processing; then, the time gate search step in the traditional envelope-based heart sound segmentation algorithm is optimized by using a greedy algorithm, which reduces the missed detection rate of the time gate and improves the segmentation accuracy; in the feature processing step, various time-frequency domain features such as heart sound interval ratio class, MFCC features, Mel spectrum feature maps, second-order sound spectrum feature maps and other multi-domain features are extracted to quantify the features of heart sound signals; then, a heart sound classification model based on deep learning and heterogeneous integration strategy is proposed to improve the accuracy of heart sound detection; finally, an architecture design and a heart sound data visualization method for the research and development of a heart sound detection platform are provided, which provides ideas for the development of a medical and health platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of a heart sound detection method based on deep learning and heterogeneous integration strategy provided in a specific embodiment of the present invention;

[0012] Figure 2 is a schematic diagram of a heart sound segmentation process based on a high-order Shannon envelope in a specific implementation manner of the present invention;

[0013] Figure 3 It is a schematic diagram of interval division in a dual threshold search time gate algorithm based on a greedy algorithm and adaptive threshold optimization provided in a specific embodiment of the present invention;

[0014] Figure 4 It is an MFCC feature curve diagram of a heart sound signal classification method based on deep learning and heterogeneous integration strategy provided in a specific embodiment of the present invention;

[0015] Figure 5 It is a Mel-spectrogram feature map of a heart sound classification method based on deep learning and heterogeneous integration strategy provided in a specific implementation method of the present invention;

[0016] Figure 6 It is a second-order spectral feature map of a heart sound classification method based on deep learning and heterogeneous integration strategy provided in a specific implementation method of the present invention;

[0017] Figure 7 It is a CNN-LSTM network architecture for identifying MFCC features in a heart sound classification method based on deep learning and heterogeneous integration strategy provided in a specific implementation method of the present invention;

[0018] Figure 8 It is a heart sound classification system architecture design diagram based on deep learning and heterogeneous integration strategy provided in the specific implementation method of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in this application will be described clearly and completely below in conjunction with the drawings in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. The components of this application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the application claimed for protection, but only represents the selected embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0020] like Figure 1As shown, the embodiment of the present application provides a heart sound classification method based on deep learning and heterogeneous integration strategy, including the following steps: step S00: performing wavelet threshold denoising on the impure heart sound signal; step S10: extracting the high-order Shannon envelope of the heart sound signal after low-noise processing, and applying the double threshold search time gate algorithm based on the greedy algorithm and adaptive threshold optimization to segment the heart sound, and segmenting the complete heart sound signal into heart sound segments such as S1 interval, systolic period, S2 interval, diastolic period, etc.; step S20: using a preset function to extract multi-domain features of time-frequency domain features, MFCC features, Mel spectrum feature maps, and second-order spectrum feature maps from the heart sound segments and the complete heart sound signal; step S30: inputting the multi-domain features into a preset deep learning heterogeneous integration model for calculation. The heterogeneous integration model is composed of a machine learning branch and a deep learning branch, and the machine learning branch has a unique decision classifier, and its detection result is recorded as y1. The deep learning branch uses three deep neural network models to classify different heart sound features, and finally votes on the classification results of the three models, and the detection result with more votes is recorded as y2. If y1 is abnormal, the detection result is abnormal; otherwise, the detection result is y2; Step S40: output detection information.

[0021] Preferably, in step S00, after literature research and simulation comparative experiments, the coif5 wavelet function with the best performance is selected to apply the threshold noise reduction method to pre-process the heart sound signal, so as to filter out the miscellaneous noise in the heart sound audio collection process to the greatest extent.

[0022] Preferably, in step S10, greedy and adaptive thresholds are used to optimize the time gate search part in the traditional envelope-based heart sound segmentation algorithm.

[0023] Preferably, in step S20, in order to quantify the heart sound features more meticulously, multi-domain features are extracted from multiple directions and angles in the embodiment of the present application. First, 124 time-frequency domain features of the heart sound segment are extracted, of which the time domain signal takes the mean and standard deviation of a series of indicators, totaling 36 features, mainly including the following indicators: cardiac cycle, S1 interval, systolic period, S2 interval, diastolic period, systolic period ratio, systolic period to diastolic period ratio. The ratio of the average absolute amplitude of the systolic period of each heartbeat to the average absolute amplitude of the S1 interval, the ratio of the average absolute amplitude of the diastolic period to the S2 interval of each heartbeat, the skewness of the amplitude of the S1 interval of each heartbeat, the skewness of the amplitude of the S2 interval of each heartbeat, the skewness of the amplitude of the systolic period of each heartbeat, the skewness of the amplitude of the diastolic period of each heartbeat, the kurtosis of the amplitude of the S1 interval in each heartbeat, the kurtosis of the amplitude of the S2 interval of each heartbeat, the kurtosis of the amplitude of the systolic period of each heartbeat, and the kurtosis of the amplitude of the diastolic period of each heartbeat. 88 frequency domain features are extracted, including: the mean of the median power in 9 frequency bands (25-45, 45-65, 65-85, 85-105, 105-125, 125-150, 150-200, 200-300, 300-400Hz) of the S1 interval, systole, S2 interval, and diastole in all cycles in the sample (9×4=36 features). 13-dimensional mfcc features of all cycles in the sample (4×13=52 features). The above 124 multi-domain features are used as input features of the machine learning branch. Secondly, the MFCC feature sequence and spectral domain image features such as Mel-spectrogram and second-order spectrogram of the complete heart sound signal are extracted and used as input features of the deep learning branch.

[0024] Preferably, in step S30, XGBoost is selected as the only classifier of the machine learning branch to identify 124-dimensional time-frequency domain features; in the deep learning branch, CNN-LSTM is selected as the MFCC feature recognition network, AOCT network is used as the recognition network of the second-order spectral feature map, and Densenet121 is used as the recognition network of the Mel-spectrogram feature map.

[0025] like Figure 2 As shown, the heart sound segmentation process based on the high-order Shannon envelope provided in the embodiment of the present application includes the following specific steps:

[0026] Step S101: extracting a high-order Shannon envelope from the heart sound signal after noise reduction processing.

[0027] The envelope curve is extracted by calculating the normalized average high-order Shannon entropy. Each segment is composed of 32 points, with 16 points overlapping. The average high-order Shannon entropy of each segment is calculated. The average high-order Shannon entropy of the k-th segment signal is defined as:

[0028]

[0029] Where: x is the normalized signal amplitude; N is the number of sampling points of each signal segment; M is the number of overlapping sampling points; n is the order of high-order Shannon entropy.

[0030] The relationship between segment k and time t is as follows:

[0031]

[0032] Where: f s is the sampling rate.

[0033] In order to make the packet correspond to the signal on the time axis, the sequence {E hs (k)} becomes the sequence {E hs (t)}. Then, the normalized average high-order Shannon entropy of the entire heart sound signal is calculated as:

[0034]

[0035] Where: M(E hs (T)) is {E hs (t)}; S(E hs (t)) is the standard deviation. The normalized average high-order Shannon entropy is used as the envelope of the heart sound signal.

[0036] Step S102: performing lowess smoothing on the heart sound envelope.

[0037] Step S103 : searching for an envelope time gate using a greedy algorithm and a dual threshold search time gate algorithm optimized by an adaptive threshold.

[0038] Time gate search refers to filtering out the extreme peaks in the smoothed envelope curve and converting them into corresponding time gates. The amplitudes of the S1 and S2 intervals in the heart sound signal are large, and the calculated Shannon entropy values ​​are high, so they correspond to the extreme peaks in the envelope curve. After determining the time gate, the interval is determined based on the distance between the time gates according to the rule that the systolic period is greater than the diastolic period, and the complete audio can be divided into several heart sound segments.

[0039] Since the traditional dual threshold needs to set the upper and lower thresholds before searching, and the extreme peaks during S1 and S2 of different audio signals are quite different, an adaptive threshold is adopted in the embodiment of the present application to determine the search starting point.

[0040] Each time the missing interval is traversed (the entire interval at the beginning), all extreme values ​​are stored and sorted in ascending order. The upper 10% quantile in the sequence is used as the upper threshold, and the lower 5% quantile is used as the lower threshold. The extreme value points greater than the upper threshold are screened out for left and right searches. Each time for the extreme value point x that meets the requirements i Search within the window width, distance x i The nearest point that is less than the lower threshold is the boundary point of the interval.

[0041] The algorithm process includes:

[0042] Step S1: Enter the current missing interval and sort all extreme values ​​extrem[] in the interval in ascending order;

[0043] Step S2: Set the upper threshold to extreme [floor (len (extrem) * 0.1)], and the lower threshold to extreme [floor (len (extrem) * 0.95)];

[0044] Step S3: enumerate extreme value points whose extreme values ​​are greater than the upper threshold, and search left and right within the window width;

[0045] Step S4: Determine whether the currently searched extreme value is less than the lower threshold value. If so, determine it as a boundary, otherwise proceed to step S3.

[0046] The search strategy within the missing interval is determined. How to determine the missing interval is another important problem to be solved by the embodiments of the present application.

[0047] Taking into account the regularity of the cardiac cycle, each complete cardiac cycle consists of the S1 interval, the systolic period, the S2 interval, and the diastolic period, wherein the systolic period is shorter than the diastolic period. Analysis shows that whether the S1 time gate or the S2 time gate is missing, the missing interval is composed of a systolic period, a diastolic period, and a missing time gate. Therefore, the embodiment of the present application calculates the intervals between the searched time gates and sorts them in ascending order. If there are still missing time gates when traversing to a certain state, the current interval state can be divided into three intervals, interval 1 is recorded as D1, interval 2 is recorded as D2, and interval 3 is recorded as D3. Assume that the systolic interval index set is Systole, the diastolic index set is Distole, and the missing interval index set is Miss, D1 = {dist i |i∈Systole}D2={dist i |i∈Distole}D3={dist i |i∈Miss}. The problem will be transformed into a double pointer to determine the partition interval, such as Figure 3 As shown. It is necessary to minimize the variance of the sets D1, D2, and D3 at the same time, that is, to achieve a multi-objective planning problem with the following objective function:

[0048]

[0049] in

[0050] Make a greedy assumption: if any interval is divided into the wrong set x, the variance of x will increase significantly, and y = Z1 + Z2 + Z3 will increase, and every time an incorrectly divided interval is corrected, y will decrease. When all intervals are divided correctly, y takes the minimum value, and D3 after pointer 2 is the missing interval.

[0051] According to the idea, a complex and difficult to calculate multi-objective planning problem is transformed into a simple single-objective planning problem. The bottleneck of the algorithm lies in enumerating double pointers. The estimated algorithm complexity is O(n 2 ), and experiments show that n<500, so the actual execution speed is very fast.

[0052] Step S104: heart sound status mark.

[0053] After determining all time gates, calculate the intervals between all time gates, take the maximum interval as the diastole, its left adjacent gate as S2, and its right adjacent gate as S1, iterate and mark left and right respectively to make a heart sound state ladder curve. The heart sound state ladder curve is divided into 4 levels, recorded as 1, 2, 3, 4, which are represented from front to back as S1 interval, systole, S2 interval, and diastole.

[0054] Figure 4 An MFCC feature curve diagram of a heart sound classification method based on deep learning and heterogeneous integration strategy provided in an embodiment of the present application; Mel-frequency cepstral coefficient (MFCC) is a group of cepstral parameter sequences extracted from the energy term of a speech signal in the Mel-frequency domain. Because of its ability to describe harmonic and sideband characteristics in the signal spectrum domain, it can better characterize the nonlinear characteristics of speech signals and is widely used in speech recognition, speaker recognition, emotion recognition and other fields. The specific extraction process includes: pre-emphasis, which is used to compensate for the high-frequency components lost in the signal; frame windowing, which uses the sliding window technology to divide the non-stationary heart sound signal of the entire time series into a signal frame sequence with short-term stability. Since adjacent frame sequences usually overlap, Hamming window processing is usually used to reduce their influence; the time domain signal is obtained by discrete Fourier transform to obtain the frequency domain signal frame sequence X(k); Mel frequency domain filter bank filtering is used to first calculate the power spectrum P(k) by taking the L2 norm of the frequency domain frame sequence X(k), and then the power spectrum is filtered by the Mel filter. The product of the power spectrum P(k) and the filter is calculated at each frequency point to obtain the Mel frequency spectrum coefficient (MFSC); the Mel frequency spectrum coefficient is logarithmically operated to obtain the logarithmic energy spectrum S(m) of the frequency domain signal frame; the discrete cosine transform is applied to the Mel filter to select the largest acceleration coefficient, and the logarithmic spectrum relationship is separated from the filter to obtain the MFCC.

[0055] Figure 5The mel-spectrogram feature graph of a heart sound classification method based on deep learning and heterogeneous integration strategy provided in an embodiment of the present application has the following extraction process: first, the MFSC features are extracted using the above steps, and then the mel-spectrogram is drawn using specshow in the librosa.display module.

[0056] Figure 6 A second-order spectral feature graph of a heart sound classification method based on deep learning and heterogeneous integration strategy provided in an embodiment of the present application. In the heart sound signal analysis scenario, the second-order spectrum analysis can effectively eliminate white noise (Gaussian noise) by suppressing the phase relationship in the signal, while effectively detecting the phase coupling and nonlinear relationship in the heart sound signal, and quantizing it as a second-order spectral coefficient as a representative feature for distinguishing positive and abnormal heart sound signals. The extraction process includes: pre-emphasis; frame division and windowing; calculating the third-order cumulant of the heart sound signal to obtain a third-order cumulative signal frame sequence; performing a second-order Fourier transform on the third-order cumulant; then splicing the second-order spectral coefficient vector in frame order to obtain a second-order spectral matrix, and drawing a heat map to obtain a second-order spectral feature graph.

[0057] Figure 7 A CNN-LSTM network architecture for identifying MFCC features in a heart sound classification method based on deep learning and heterogeneous integration strategy provided in an embodiment of the present application. This model combines the advantages of CNN's sensitivity to spatial features and LSTM's sensitivity to temporal features. In this architecture, a convolution module is first used for feature extraction, and then LSTM is used to encode high-dimensional abstract features, which are decoded using a linear layer, and finally mapped through a SoftMax layer to obtain the classification result. The number of layers of the convolutional pooling structure, the convolution and built-in parameters, and the built-in parameters of the LSTM encoding layer are all hyperparameters, which can be determined through experimental tuning.

[0058] Figure 8A heart sound classification system architecture design diagram based on deep learning and heterogeneous integration strategy provided for the embodiment of the present application. Based on the design of the front-end and back-end separation idea, the system is divided into multiple levels, mainly including: application layer, business layer, computing and implementation layer, data center layer and access layer. The layered implementation strategy enables the entire system to have the characteristics of high availability and high scalability. Among them, the implementation layer defines the specific functions that are relevant to the user and can be operated. The business layer classifies the relevant functions and defines the method interface under each category. The computing and implementation layer is a specific implementation of the interface defined in the business layer, and different technical frameworks are used for implementation according to different requirements. The back-end calculation uses the Django framework, the front-end rendering tool uses Vue and Apexcharts, and the artificial intelligence calculation is implemented using Pytorch. The computing center layer implements the preliminary processing of heart sounds, such as loading, denoising, and extracting envelopes of heart sound data. The access layer is a definition of persistent data. For file-type data, it will be stored in the file service directory of Django, and for structured data, MySQL will be used for persistent storage.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A heart sound detection method based on deep learning and heterogeneous integration strategy, characterized in that: The method comprises: Step S00: performing wavelet threshold denoising processing on the impure heart sound signal; Step S10: extracting the high-order Shannon envelope of the heart sound signal after noise reduction processing, and applying a double threshold search time gate algorithm based on a greedy algorithm and adaptive threshold optimization to segment the heart sound to form heart sound segments, specifically including: Step S101, extracting a high-order Shannon envelope from the heart sound signal after noise reduction processing; Step S102, performing lowess smoothing on the heart sound envelope; Step S103, using a greedy algorithm and a dual threshold search time gate algorithm optimized by an adaptive threshold to search for an envelope time gate; wherein the time gate search refers to screening out the extreme peaks in the smoothed envelope curve and converting them into corresponding time gates. After determining the time gate, the interval is determined according to the distance between the time gates in combination with the rule that the systolic period is greater than the diastolic period, and the complete audio is divided into several heart sound segments; an adaptive threshold is used to determine the search starting point. The algorithm flow includes: Step S1: Entering the current missing interval, all extreme values ​​extrem[ ] Sort in ascending order; Step S2: Set the upper threshold to extreme [floor (len (extrem) * 0.1)], and the lower threshold to extreme [floor (len (extrem) * 0.95)]; Step S3: Enumerate the extreme points whose extreme values ​​are greater than the upper threshold, and search left and right within the window width; Step S4: Determine whether the extreme value currently searched is less than the lower threshold. If it is less than, it is determined as the boundary, otherwise enter step S3; The missing intervals are composed of a systolic period, a diastolic period, and a missing time gate. If there is still a missing time gate when traversing to a certain state, the current interval state is divided into three intervals, and interval 1 is recorded as , interval 2 is recorded as , interval 3 is recorded as , , the problem will be transformed into determining the double pointer of the partition interval, which needs to be minimized at the same time , , The variance of the set is the multi-objective programming problem that realizes the following objective function: ； in, ; Make a greedy assumption: if any interval is classified into the wrong set x, the variance of x will increase significantly, and will increase, and every time an incorrectly divided interval is corrected, y will decrease; when all intervals are divided correctly, y takes the minimum value, and the interval after pointer 2 is the missing interval; Step S104: heart sound status mark; Step S20: using a preset function to extract time-frequency domain features from the heart sound segment and to extract multi-domain features of MFCC features, Mel-spectrum feature maps, and second-order spectrum feature maps from the complete heart sound signal; Step S30: inputting the multi-domain features into a preset deep learning heterogeneous integration model for calculation; Step S40: Output the detection result.

2. The method according to claim 1, characterized in that The heart sound segments include Interphase, systolic period, Interphase, diastole.

3. The method according to claim 1, characterized in that The deep learning heterogeneous integration model consists of a machine learning branch and a deep learning branch; the machine learning branch uses XGBoost as the only decision classifier; the deep learning branch uses the CNN-LSTM model, the Densenet121 model and the AOCT model.

4. The method according to claim 3, characterized in that The output detection result specifically includes: The prediction result of XGBoost is recorded as ; The classification results of the CNN-LSTM model, the Densenet121 model and the AOCT model are voted and the result with more votes is recorded as ; like is abnormal, the output test result is abnormal; otherwise, the output test result is .

5. The method according to claim 1, characterized in that After outputting the detection result, the method further includes: The detection results are visualized on the front-end page using the Apexcharts third-party library.

6. A heart sound detection system based on deep learning and heterogeneous integration strategy using the heart sound detection method based on deep learning and heterogeneous integration strategy according to any one of claims 1 to 5, characterized in that: The system comprises: Heart sound upload module, used to obtain the heart sound signals uploaded by users and store them on the server; A heart sound detection module, used for performing heart sound segmentation and multi-domain feature extraction, then applying preset multi-domain features to input a preset deep learning heterogeneous integration model for calculation, outputting detection results and storing the detection results in the server; The visualization module is used to visualize the detection results using the Apexcharts third-party library.

Images