Two-channel heart and lung sound intelligent analysis method based on time-frequency feature fusion

Abstract

The invention provides a two-channel heart and lung sound intelligent analysis method based on time-frequency feature fusion. The method is used for solving the problem that in the prior art, the heart and lung sound signal analysis recognition capacity is insufficient. According to the method, the time-frequency features of the heart and lung sound signals of the patient are fused, and a plurality of feature extraction methods are combined, so that the accuracy of an analysis result is remarkably improved. The method is characterized by comprising the following steps of S1, performing data analysis on heart and lung sound signals through two feature extraction methods to obtain two feature images with time domain features and frequency domain features of heart and lung sound; s2, transmitting the two feature images with the time domain features and the frequency domain features of the heart and lung sounds to a dual-channel heart and lung sound intelligent analysis model; and S3, the two-channel cardiopulmonary sound intelligent analysis model performs dynamic feature fusion based on the two feature images with the cardiopulmonary sound time domain features and the cardiopulmonary sound frequency domain features, and obtains a cardiopulmonary sound analysis result based on a fusion result.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and medicine, and relates to a method for classifying heart sounds and lung sounds, specifically to a dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion. Background Art

[0002] Respiratory diseases and cardiovascular diseases are among the leading causes of death globally. Early diagnosis and timely treatment are of great significance for preventing the deterioration of the condition and reducing the mortality rate. As an important biomarker for cardiovascular and respiratory diseases, cardiopulmonary sound signals have been proven to be able to effectively reflect early pathological information, providing a reliable basis for the early detection of latent cardiovascular diseases. The current mainstream research data sources include: the PhysioNet / CinC Challenge heart sound dataset in 2016 (including two categories of normal / abnormal) and the ICBHI lung sound dataset (subdivided into four categories: normal, wheezing type, crackling type, and mixed type), providing a benchmark support for the development of intelligent diagnosis algorithms.

[0003] Manual auscultation highly depends on physicians with rich clinical experience. However, the resources of medical staff are limited and far from meeting the growing medical needs. To solve this problem, researchers have begun to explore new ways to integrate intelligent analysis systems and artificial intelligence to break through traditional diagnosis in order to achieve the automated diagnosis of cardiopulmonary diseases.

[0004] Currently, although many existing intelligent analysis methods can alleviate the dependence on manual work, there are still significant technical defects: 1) At the feature processing level, most methods simply jointly model time-domain and frequency-domain features, resulting in the loss of specific information between multi-modal features; 2) At the feature extraction level, over-reliance on a single technical route (such as only using STFT or MFCC) makes it difficult to comprehensively represent the time-frequency coupling characteristics of cardiopulmonary sound signals; 3) The existing models have insufficient recognition ability for cardiopulmonary co-lesions, seriously restricting the clinical diagnosis accuracy and generalization performance. Summary of the Invention

[0005] The present invention is made to solve the above problems, and aims to provide a dual-channel intelligent analysis method for cardiopulmonary sounds that can fuse the time-frequency features of patients' cardiopulmonary sound signals and integrate multiple feature extraction methods to obtain more accurate analysis results. The present invention adopts the following technical solutions:

[0006] The present invention provides a dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion, which has the following technical features. The method includes the following steps: Step S1, perform data analysis on the cardiopulmonary sound signal through two feature extraction methods respectively to obtain images with time-domain features and frequency-domain features of cardiopulmonary sounds; Step S2, transmit the two feature images with time-domain features and frequency-domain features of cardiopulmonary sounds to the dual-channel intelligent analysis model for cardiopulmonary sounds; Step S3, the dual-channel intelligent analysis model for cardiopulmonary sounds performs dynamic fusion of features based on the two feature images with time-domain features and frequency-domain features of cardiopulmonary sounds, and obtains the cardiopulmonary sound analysis result based on the fusion result. Among them, the dual-channel intelligent analysis model for cardiopulmonary sounds includes: an adaptive time-frequency fusion module, which is used to perform time-domain and frequency-domain feature fusion on the two feature images with time-domain features and frequency-domain features of cardiopulmonary sounds to obtain time-frequency features; a time-frequency fusion lightweight convolution module, which is used to extract the primary representation of the time-frequency features and optimize the feature weights through a channel attention mechanism; and a time-frequency fusion cardiopulmonary sound Transformer module, which is used to enhance the global feature modeling ability through a multi-axis sparse self-attention mechanism and output the cardiopulmonary sound analysis result.

[0007] The dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion provided by the present invention may also have the following technical features. Among them, the cardiopulmonary sound signal includes a heart sound signal and a lung sound signal, and Step S1 includes the following sub-steps: Step S1-1, perform noise reduction processing on the heart sound signal and the lung sound signal respectively; Step S1-2, perform two feature extractions of short-time Fourier transform and Mel frequency cepstral coefficients on the denoised heart sound signal and the denoised lung sound signal respectively to obtain two feature maps of the heart sound signal and two feature maps of the lung sound signal respectively; Step S1-3, perform data augmentation on each of the feature maps respectively to form a time-domain feature image and a frequency-domain feature image of cardiopulmonary sounds.

[0008] The dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion provided by the present invention may also have the following technical features. Among them, in Step S1-1, a fifth-order Butterworth band-pass filter is used to filter the heart sound signal and the lung sound signal respectively. For the heart sound signal, the cut-off frequencies of the Butterworth band-pass filter are set to 20 Hz and 800 Hz. For the lung sound signal, the cut-off frequencies of the Butterworth band-pass filter are set to 200 Hz and 1300 Hz.

[0009] The dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion provided by the present invention may also have such technical features, wherein, in step S1-3, the data enhancement includes randomly oversampling the feature map, randomly cropping the feature map within each batch, and linearly combining the feature maps of different categories through Mixup data enhancement technology to generate new synthetic samples.

[0010] The dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion provided by the present invention may also have such technical features, wherein the adaptive time-frequency fusion module includes two independent channel branches, a time-frequency fusion attention block, an average pooling layer, a maximum pooling layer and an activation layer, and the channel branch includes a time domain feature extraction network and a frequency domain feature extraction network.

[0011] The dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion provided by the present invention may also have such a technical feature, wherein the adaptive time-frequency fusion module processes the two feature images having the time domain features and frequency domain features of the cardiopulmonary sound through the two channel branches respectively, which is expressed as:

[0012] x t =Conv T (BN-Act(Conv T (x)))

[0013] x f =Conv F (BN-Act(Conv F (x)))

[0014] Where, Conv T and Conv F Respectively represent convolution kernels with larger spans in the time and frequency axes. Next, the adaptive time-frequency fusion module uses a dual attention mechanism of channel and space to fuse the time features and the frequency features. In each channel branch, channel information is aggregated by global pooling across the channel dimension, which is expressed as:

[0015] S c =Concat(Avg(x t ),Max(x t ),Avg(x f ),Max(x f ))

[0016] Where S c represents the aggregated channel features, x t and x frespectively represent the time and frequency characteristics of cardiopulmonary sounds, and Avg(·) and Max(·) respectively represent global average pooling and global max pooling across the channel dimension. The aggregated channel features are passed to two one-dimensional convolutional layers to determine the channel weights of the two channel branches, which are expressed as:

[0017] W c1 ,W c2 =Conv1(S c ),Conv2(S c )

[0018] In the formula, W c1 and W c2 respectively represent the channel weights of the time-frequency characteristics of cardiopulmonary sounds, and Conv1(·) and Conv2(·) represent one-dimensional convolutional operations. Subsequently, the Softmax function is used to normalize the channel weights of the time-frequency characteristics, which is expressed as:

[0019]

[0020] In the formula, W c ′ 1 and W c ′ 2 respectively represent the output time-frequency channel weights.

[0021] The dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion provided by the present invention may further have the following technical feature: the time-frequency fusion lightweight convolution module includes a time-domain convolution block, a frequency-domain convolution block, a pointwise convolution block, a squeeze-and-excitation block, and a downsampling layer.

[0022] The dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion provided by the present invention may further have the following technical feature: the time-frequency fusion lightweight convolution module decomposes the input features into time-domain features and frequency features, fuses the time-domain features and the frequency features through a time-frequency fusion mechanism, then enhances the weights of important features through the squeeze-and-excitation module, adjusts the channel dimension through pointwise convolution, and finally performs a residual connection between the adjusted features and the downsampled features to form output features, which is expressed as:

[0023] x=BN-Act(Conv 1×1 (BN(X)))

[0024] x=TFFM(x)

[0025] x=SE(BN-Act(x))

[0026] x=Conv 1×1 (x)

[0027]

[0028] wherein, BN represents batch normalization, Act represents the activation function, and Conv 1×1 represents pointwise convolution.

[0029] The dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion provided by the present invention may further have the following technical feature: wherein, the time-frequency fusion cardiopulmonary sound Transformer module includes a window self-attention block, a feed-forward neural network block, and a grid self-attention block.

[0030] The training of the dual-channel intelligent analysis model for cardiopulmonary sounds provided by the present invention may further have the following technical feature: wherein, the training of the dual-channel intelligent analysis model includes the following steps: Step T1, constructing an initial dual-channel cardiopulmonary sound analysis model, which includes the adaptive time-frequency fusion module, the time-frequency fusion lightweight convolution module, and the time-frequency fusion cardiopulmonary sound Transformer module; Step T2, sequentially inputting a training set containing multiple groups of cardiopulmonary sound data into the initial dual-channel cardiopulmonary sound analysis model for model training and performing iteration; Step T3, respectively calculating the loss error using the model parameters of the last layer and backpropagating the calculated loss error to update the model parameters; Step T4, repeating Step T2 to Step T3 until a predetermined training completion condition is reached to obtain the dual-channel intelligent analysis model for cardiopulmonary sounds.

[0031] Functions and effects of the invention

[0032] According to the dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion provided by the present invention, the time-domain feature image and the frequency-domain feature image of the cardiopulmonary sound signal are first extracted respectively, and then the time-domain feature image and the frequency-domain feature image are input into the dual-channel intelligent analysis model for cardiopulmonary sounds, which not only realizes the automatic recognition and classification of the cardiopulmonary sound signal, but also better fuses the complementary information of the time-frequency features through the dual-channel intelligent analysis model for cardiopulmonary sounds, thereby realizing more accurate and effective recognition and classification of cardiopulmonary sounds, improving the recognition ability of cardiopulmonary co-lesions, and making this method have high practical application value in clinical applications. Brief description of the drawings

[0033] Figure 1 is a flowchart of the dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion in an embodiment of the present invention;

[0034] Figure 2 is a comparison schematic diagram of the original sound wave diagram and the denoised sound wave diagram of the cardiopulmonary sound signal in an embodiment of the present invention;

[0035] Figure 3It is a schematic diagram of the extraction of cardiorespiratory sound STFT features in an embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of the extraction of cardiorespiratory sound MFCC features in an embodiment of the present invention;

[0037] Figure 5 It is a schematic diagram of the structure of the adaptive time-frequency fusion module in an embodiment of the present invention;

[0038] Figure 6 It is a schematic diagram of the structure of the time-frequency fusion lightweight convolution module in an embodiment of the present invention;

[0039] Figure 7 It is a schematic diagram of the structure of the time-frequency fusion cardiorespiratory sound Transformer module in an embodiment of the present invention;

[0040] Figure 8 It is a schematic diagram of the sparse self-attention mechanism based on multi-axis decomposition in an embodiment of the present invention.

[0041] Figure 9 It is a flowchart of the training process of the dual-channel cardiorespiratory sound intelligent analysis model based on time-frequency feature fusion in an embodiment of the present invention. Specific implementation manner

[0042] In order to make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the dual-channel cardiorespiratory sound intelligent analysis method based on time-frequency feature fusion of the present invention will be specifically described below in conjunction with embodiments and drawings.

[0043] <Example>

[0044] The dual-channel cardiorespiratory sound intelligent analysis method based on time-frequency feature fusion in this embodiment is implemented based on a computer, and the computer contains at least one graphics card for GPU acceleration. The dual-channel cardiorespiratory sound intelligent analysis model and the cardiorespiratory sound analysis process used in the dual-channel cardiorespiratory sound intelligent analysis method in this embodiment are stored in the computer in the form of executable code.

[0045] The heart sound dataset used in this embodiment comes from The 2016 PhysioNet / CinC Challenge. The data provided by this dataset consists of two parts, a training set and a test set, and includes two categories: normal heart sounds and abnormal heart sounds. Among them, there are 2,575 records of normal heart sounds and 665 abnormal heart sounds, for a total of 3,240 heart sound records. The lung sound dataset used is the ICBHI dataset used in a scientific challenge on the automatic detection of pulmonary crackles and wheezes hosted by the International Conference on Biomedical and Health Informatics in 2017. This dataset contains a total of 6,898 respiratory cycles, including 3,642 normal lung sounds, 1,864 crackles, 886 wheezes, and 506 that contain both crackles and wheezes.

[0046] Figure 1 is the flowchart of the dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion in this embodiment.

[0047] As Figure 1 shown, the dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion in this embodiment includes the following steps:

[0048] Step S1, perform data analysis on the cardiopulmonary sound signal of the patient through a time-domain analysis method and a frequency-spectrum analysis method respectively to obtain the time-domain feature image and frequency-domain feature image of the cardiopulmonary sound.

[0049] Step S2, transfer the time-domain feature image and frequency-domain feature image to the dual-channel cardiopulmonary sound intelligent analysis model.

[0050] Step S3, the dual-channel cardiopulmonary sound intelligent analysis model performs dynamic fusion of features based on the time-domain feature image and frequency-domain feature image, and obtains the cardiopulmonary sound analysis result based on the fusion result.

[0051] The above steps will be described in detail below.

[0052] Step S1, perform data analysis on the cardiopulmonary sound signal through two feature extraction methods respectively to obtain two feature images with the time-domain features and frequency-domain features of the cardiopulmonary sound.

[0053] Among them, the cardiopulmonary sound signal includes a heart sound signal and a lung sound signal.

[0054] Since, during the signal acquisition process, it will be affected by various interference factors. In addition, due to the class imbalance problem in the cardiopulmonary sound dataset, and data imbalance will cause the model to be more inclined to learn the category with a larger number of samples during the training process, thereby weakening the recognition ability of the minority class and reducing the overall performance. Therefore, in order to reduce interference factors and improve the model recognition accuracy, it is necessary to preprocess the collected cardiopulmonary sound signal first.

[0055] Step S1 specifically includes the following sub-steps:

[0056] Step S1-1, perform noise reduction processing on the heart sound signal and the lung sound signal respectively.

[0057] In this embodiment, a fifth-order Butterworth band-pass filter is used to filter the cardiopulmonary sound signal. Since the frequency characteristics of heart sounds and lung sounds are slightly different, the heart sounds and lung sounds are analyzed separately.

[0058] Among lung sounds, wheezing is a high-frequency continuous sound caused by airway obstruction, and its frequency is about 400 Hz. Snoring is a low-frequency continuous sound caused by excessive respiratory secretions, and its frequency is about 200 Hz. Stridor is a high-frequency continuous sound caused by upper airway obstruction, and the highest frequency of its inspiratory stridor can reach 1300 Hz. The frequencies of other lung sound categories are higher than that of snoring but lower than the frequency of inspiratory stridor. In order to minimize the influence of noise on the lung sound recognition process, the cut-off frequencies of the Butterworth band-pass filter are set to 200 Hz and 1300 Hz to identify lung diseases through sound wave frequencies.

[0059] For heart sounds, the heart sound frequencies of normal people are roughly 20 - 800 Hz, among which the frequency range of the first heart sound is 20 - 140 Hz, and the second heart sound is 20 - 400 Hz. When the heart is abnormal, individual heart sound samples may have high-frequency components of 600 - 800 Hz. Therefore, the cut-off frequencies of the Butterworth band-pass filter are set to 20 Hz and 800 Hz.

[0060] Figure 2 It is a comparison schematic diagram of the original sound wave diagram and the sound wave diagram after noise reduction of the cardiopulmonary sound signal in this embodiment.

[0061] As Figure 2 shown, after noise reduction processing, the signals corresponding to noises such as crackles and wheezes are significantly attenuated, which can reduce their influence on the cardiopulmonary sound recognition process.

[0062] Step S1-2, perform two feature extractions of short-time Fourier transform (STFT) and Mel-frequency cepstral coefficients (MFCC) on the noise-reduced heart sound signal and lung sound signal respectively, and obtain two feature maps of the heart sound signal and two feature maps of the lung sound signal respectively.

[0063] Figure 3 It is a schematic diagram of STFT feature extraction of cardiopulmonary sounds in this embodiment, Figure 4 It is a schematic diagram of MFCC feature extraction of cardiopulmonary sounds in this embodiment, which respectively shows the schematics of the feature maps obtained by the two feature extraction methods.

[0064] Step S1-3, perform data augmentation on various feature maps respectively to form the time-domain feature image and frequency-domain feature image of cardiorespiratory sounds.

[0065] In this embodiment, random oversampling is performed on the feature maps to ensure that the number of samples of each category in each batch is approximately equal. This method can effectively balance the sample distribution and prevent the model from overly relying on the features of a certain category. At the same time, to improve the robustness of the model in the case of local information loss, random cropping is performed on the feature maps within each batch, reducing 10 frequency bands respectively in both the time and frequency dimensions. By simulating the situation of partial information loss, this strategy prompts the model to learn more generalizable feature expressions. In addition, the Mixup data augmentation technique is adopted, and new synthetic samples are generated by linearly combining feature maps of different categories. Specifically, for two randomly selected feature maps, they are mixed in a certain proportion to generate new training samples. This mixing strategy significantly increases the diversity of the training data and expands the Fisher discriminant criterion, that is, the ratio of the between-class distance to the within-class variance in the feature space, thereby improving the model's performance in class discrimination.

[0066] Step S2, transfer the above two feature images with cardiorespiratory sound time-domain features and frequency-domain features to the dual-channel cardiorespiratory sound intelligent analysis model. That is, the two feature images are respectively used as the dual-channel inputs of the model.

[0067] Step S3, the dual-channel cardiorespiratory sound intelligent analysis model performs dynamic fusion of features based on the above two feature images with cardiorespiratory sound time-domain features and frequency-domain features, and obtains the cardiorespiratory sound analysis result based on the fusion result.

[0068] The dual-channel cardiorespiratory sound intelligent analysis model based on time-frequency feature fusion is mainly built using the existing deep learning framework pytorch. This dual-channel cardiorespiratory sound intelligent analysis model includes three modules, namely the adaptive time-frequency fusion module, the time-frequency fusion lightweight convolution module, and the time-frequency fusion cardiorespiratory sound Transformer module.

[0069] Figure 5 It is the structural schematic diagram of the adaptive time-frequency fusion module in this embodiment.

[0070] As Figure 5 shown, the adaptive time-frequency fusion module (TFFM) includes two independent processing branches, namely the time-domain feature extraction network (in the figure, the STFT stream TFF-HAT Network, also called Conv T block), the frequency-domain feature extraction network (in the figure, the MFCCs stream TFF-HAT Network, also called Conv FBlock), Time-Frequency Fusion Attention block, Average Pooling layer, Max Pooling layer, and Activation layer (Softmax). This module extracts time features and frequency features through corresponding independent processing branches respectively, and uses a dual attention mechanism strategy to dynamically identify and weight important parts of time-frequency features. This module not only focuses on the separate extraction of time-domain features and frequency-domain features, but also emphasizes the interaction and importance evaluation between time-frequency features.

[0071] Specifically, the adaptive time-frequency fusion module overcomes the fixed fusion coefficient α in traditional methods and better utilizes the complementarity of heart and lung sounds in time and frequency after STFT and MFCC feature extraction. First, for the input feature map x, it is processed through two independent processing branches respectively, using convolutional kernels with larger spans on the time axis and frequency axis. This design aims to decouple time and frequency information, so as to more effectively extract time-frequency related features. The specific processing process can be expressed as:

[0072] x t = Conv T (BN - Act(Conv T (x)))

[0073] x f = Conv F (BN - Act(Conv F (x)))

[0074] In the formula, Conv T and Conv F represent convolutional kernels with larger spans on the time axis and frequency axis respectively. Through this design, time and frequency features are independently processed and their respective time-frequency information is captured.

[0075] Then, a dual channel and spatial attention mechanism is used to effectively fuse the heart and lung sound time feature x t and the frequency feature x f , and two kinds of attention are used to determine the important parts in time-frequency features. In the channel branch, the input heart and lung sound time-frequency features aggregate channel information through global pooling across the channel dimension. The aggregation process can be expressed as:

[0076] S c = Concat(Avg(x t ), Max(x t ), Avg(x f ), Max(x f ))

[0077] In the formula, S c represents the aggregated channel features, and x t and x f represent the time feature and frequency feature of the cardiopulmonary sound respectively. Avg(·) and Max(·) represent global average pooling and global max pooling across the channel dimension respectively.

[0078] The aggregated spatial features are passed to two one-dimensional convolutional layers, which are similar to the lightweight channel attention (ECA) module, to determine the channel weights of the input cardiopulmonary sound time-frequency features. The two channel weights can be expressed as:

[0079] W c1 , W c2 = Conv1(S c ), Conv2(S c )

[0080] In the formula, W c1 and W c2 represent the channel weights of the cardiopulmonary sound time-frequency features respectively, and Conv1(·) and Conv2(·) represent one-dimensional convolutional operations.

[0081] Subsequently, the Softmax function is used to normalize the time-frequency channel weights so that their sum is 1. This process means that by comparing the weights of the cardiopulmonary sound time-frequency features, the most important feature part in the channel dimension can be determined. The calculation process of the Softmax function is shown in the following formula:

[0082]

[0083] In the formula, W c ′ 1 and W c ′ 2 represent the output time-frequency channel weights respectively. The cardiopulmonary sound time-frequency spatial weights W s ′ 1 and W s ′ 2 are calculated in a similar way in the spatial branch to determine the importance of the cardiopulmonary sound time-frequency features in the spatial dimension.

[0084] Finally, the cardiopulmonary sound time-frequency channel weights and time-frequency spatial weights are integrated to obtain the overall time-frequency weights, so as to effectively identify the most critical part of the cardiopulmonary sound time-frequency features. Specifically, the cardiopulmonary sound time-frequency weights are multiplied by the cardiopulmonary sound time-frequency features and summed up to effectively fuse the cardiopulmonary sound time-frequency features. The output of the adaptive time-frequency fusion module is as follows:

[0085] Output = (W c ′ 1 + W s′ 1)*x t +(W c ′ 2+W s ′ 2)*x f

[0086] In the formula, Output represents the fused feature. Since the sum of the time-frequency weights of the cardiorespiratory sound is equal to 1, the useful part between the time-frequency features is retained, while the useless part is discarded, thus realizing effective feature fusion.

[0087] Figure 6 It is a schematic structural diagram of the time-frequency fusion lightweight convolution module in this embodiment.

[0088] As Figure 6 shown, the time-frequency fusion lightweight convolution module includes a time-domain convolution block (ConvDW T block), a frequency-domain convolution block (ConvDW F block), a pointwise convolution block (Conv 1x1 block), a squeeze-and-excitation block (SE block), and a downsampling layer (Drop Path layer).

[0089] The time-frequency fusion lightweight convolution module is used to extract the primary representations of time-domain features and frequency features, and optimize the feature weights through the channel attention mechanism. This module makes full use of the complementary information of STFT features in time and frequency, proposes a time-frequency feature fusion mechanism based on the dual channel and spatial attention mechanisms, and integrates it into the entire process of feature extraction. This mechanism effectively adjusts the fusion method of the two by dynamically learning the relationship between time and frequency features, enabling the fused features to comprehensively reflect the time and frequency characteristics of the cardiorespiratory sound signal.

[0090] Specifically, first, given the input feature x ∈ R B×C×T×F , where B represents the batch size, C represents the number of channels, and T and F represent the lengths of the time and frequency dimensions respectively, the input feature is downsampled or directly retained to meet the requirements of different resolutions, as shown in the following formula:

[0091]

[0092] Next, use pointwise convolution (1×1) to extract preliminary features, and combine batch normalization (BN) and activation function (Act) to enhance the network's non-linear modeling ability. On this basis, dynamically fuse time and frequency features. Specifically, first decompose the input feature x into the time-domain feature x T ∈ R B×C×T×1 and the frequency-domain feature x F ∈ R B×C×1×F. Through the time-frequency fusion mechanism, the time and frequency features are effectively fused. Subsequently, a Squeeze-and-Excitation (SE) module is applied to enhance the weights of important features, and the channel dimension is adjusted through pointwise convolution. Finally, the adjusted features are residually connected with the downsampled features

[0093] x = BN-Act(Conv 1×1 (BN(X)))

[0094] x = TFFM(x)

[0095] x = SE(BN-Act(x))

[0096] x = Conv 1×1 (x)

[0097]

[0098] Through this time-frequency feature fusion method based on the dual attention mechanism, the deep interaction between time and frequency features is achieved, enabling the network to better capture the global and local characteristics of audio signals, thus demonstrating stronger performance and robustness in diverse audio tasks. This design logically fully combines the idea of time-frequency feature fusion and effectively improves the network's ability to model complex audio signals.

[0099] Figure 7 is the structural schematic diagram of the time-frequency fusion cardiorespiratory sound Transformer module in this embodiment, Figure 8 is the schematic diagram of the sparse self-attention mechanism based on multi-axis decomposition in this embodiment.

[0100] As Figure 7 shown, the time-frequency fusion cardiorespiratory sound Transformer module includes a Window Attention block, a Feed-Forward Neural Network block (FNN block), and a Grid Attention block.

[0101] The time-frequency fusion cardiorespiratory sound Transformer module significantly improves the feature modeling ability through attention enhancement and feature enhancement on the frequency and time axes. Among them, in order to further optimize the interaction between local and global contexts, a sparse self-attention mechanism based on multi-axis decomposition is introduced, as Figure 8 shown. This mechanism combines window self-attention and grid self-attention, effectively enhancing the global feature perception ability through multi-scale interaction while retaining the time-frequency features constructed by the time-frequency fusion lightweight convolution module.

[0102] The self-attention mechanism has shown significant advantages in global feature modeling. However, the computational complexity of directly applying the attention operation in the global space is relatively high, which limits the processing ability of large-scale data. Therefore, in this embodiment, a sparse self-attention mechanism based on multi-axis decomposition is proposed, which constitutes a time-frequency fusion cardiorespiratory sound Transformer module. By combining local and global interaction mechanisms, the computational complexity is reduced while retaining the modeling ability of global information. The core of this mechanism consists of three parts: a time-frequency fusion lightweight convolution module, window self-attention, and grid self-attention.

[0103] Figure 9 It is a flowchart of the training process of the dual-channel cardiorespiratory sound intelligent analysis model based on time-frequency feature fusion in this embodiment.

[0104] As Figure 9 shown, the above-mentioned dual-channel cardiorespiratory sound intelligent analysis model based on time-frequency feature fusion is trained through the following model training process steps:

[0105] Step T1, construct an initial dual-channel cardiorespiratory sound analysis model.

[0106] That is, construct an initial model including the above-mentioned adaptive time-frequency fusion module, time-frequency fusion lightweight convolution module, and time-frequency fusion cardiorespiratory sound Transformer module.

[0107] Among them, when constructing the initial model, the parameters of the image feature extraction module in the model are pre-trained, and the initialization of the parameters of each module in the model is random. For the parameters of the feature extraction network, the pre-trained parameters trained by MaxViT on images-1k are first loaded to accelerate the convergence speed of the model.

[0108] Step T2, input the training set containing multiple groups of cardiorespiratory sound data into the initial dual-channel cardiorespiratory sound analysis model in sequence for model training, and perform iteration.

[0109] In this embodiment, after the audio in the cardiorespiratory sound dataset is preprocessed and data-augmented through step S1, 80% of the dataset is taken as the training set. When inputting the training set into the initial cardiorespiratory sound analysis model, 32 cardiorespiratory sound audio are taken as a batch, and a total of 200 iterations are performed, that is, the value of epoch is taken as 200.

[0110] To intuitively display the configuration information of each module, the adaptive time-frequency fusion module combines the idea of time-frequency feature fusion, and two convolutions (Conv F)Operation: The size of the first convolutional kernel is 3x6, and the downsampling stride is 2; the size of the second convolutional kernel is 3x6, without downsampling, the stride is 1, and the padding mode is Same. The final number of output channels is 128. The extraction of time features also uses two convolutions (Conv T )Operation: The size of the first convolutional kernel is 6x3, and the stride is 2; the size of the second convolutional kernel is 6x3, and the stride is 1, and the padding mode is Same. Subsequently, a cross-attention mechanism is used for feature fusion. First, the time and frequency features are concatenated in the channel dimension, and then dimensionality reduction is performed through a 1× convolution. The final number of output channels is 128. The compared Stem module uses two consecutive 3×3 convolutions for feature extraction. Among them, the first convolution performs downsampling with a stride of 2; the second convolution does not perform downsampling, the stride is 1, and the padding mode is same. The final number of output channels is also 128. It can be seen that the adaptive time-frequency fusion module optimizes the feature extraction and fusion capabilities while maintaining the same output channel dimension as the Stem module by introducing time-frequency decoupling and cross-attention mechanisms. For the comparison between the time-frequency fusion lightweight convolutional module and MBConv, the time-frequency decoupling module CATED is replaced with the original depth convolution, and the size of the convolutional kernel is 3×3.

[0111] Step T3, use the model parameters of the last layer to calculate the loss error respectively and backpropagate the calculated loss error to update the model parameters.

[0112] In step T2, after a training image passes through the entire model, the loss error will be calculated in step T3. In this embodiment, the selected loss function is cross-entropy. After calculating the loss error with the result obtained by the model, backpropagation is performed to update the parameters of the model. In addition, this embodiment also uses L2 regularization in the loss function to prevent the model from overfitting, and the set value of the L2 regularization parameter is 1e-5.

[0113] Step T4, repeat steps T2 to T3 until the training completion condition is reached to obtain a dual-channel intelligent cardiopulmonary sound analysis model.

[0114] In this embodiment, the training completion condition is a conventional condition, that is, the training is completed after the model converges.

[0115] In this embodiment, the remaining 20% of the cardiopulmonary sound dataset is used as the test set of the dual-channel intelligent cardiopulmonary sound analysis model. The preprocessing steps of the test set data are the same as those of the training set, but do not include data augmentation operations. After inputting the test set into the model, for heart sound data, the model outputs a score vector with a dimension of 2; for lung sound data, the model outputs a score vector with a dimension of 4. By obtaining the position corresponding to the maximum probability value in the score vector, the category of the cardiopulmonary sound is determined.

[0116] To evaluate the performance of the model, in this embodiment, accuracy (Acc), sensitivity (Sen), precision (Pre), and F1-score are used as evaluation metrics. The experimental results show that on the test set, the classification performance of the model for heart sounds reaches Acc of 97.07%, Sen of 96.91%, Pre of 97.21%, and F1-score of 97.06%; the classification performance for lung sounds reaches Acc of 68.62%, Sen of 64.58%, Pre of 77.61%, and F1-score of 68.76%.

[0117] Functions and effects of the embodiment

[0118] According to the dual-channel cardiopulmonary sound intelligent analysis method based on time-frequency feature fusion provided in this embodiment, the time-domain feature image and frequency-domain feature image of the cardiopulmonary sound signal are extracted respectively first, and then the time-domain feature image and frequency-domain feature image are input into the dual-channel cardiopulmonary sound intelligent analysis model, which not only realizes the automatic recognition and classification of the cardiopulmonary sound signal, but also better fuses the complementary information of the time-frequency features through the dual-channel cardiopulmonary sound intelligent analysis model, thereby realizing more accurate and effective cardiopulmonary sound recognition and classification, improving the recognition ability of cardiopulmonary collaborative lesions, and making this method have high practical application value in clinical applications.

[0119] In the embodiment, through preprocessing operations such as audio noise reduction, short-time Fourier transform (STFT), Mel-frequency cepstral coefficient (MFCC) feature extraction, and data augmentation on the cardiopulmonary sound signals of patients, the interference of non-cardiopulmonary sound signals is effectively reduced, thereby reducing the risk of analysis errors caused by audio quality problems.

[0120] Furthermore, the dual-channel cardiopulmonary sound intelligent analysis model includes the following core modules: the adaptive time-frequency fusion module is used to dynamically fuse the time-domain and frequency-domain feature maps of cardiopulmonary sounds; the time-frequency fusion lightweight convolution module is used to extract the primary representation of time-frequency features and optimize the feature weights through the channel attention mechanism; the time-frequency fusion cardiopulmonary sound Transformer module enhances the global feature modeling ability through the multi-axis sparse self-attention mechanism, and finally outputs a high-precision cardiopulmonary sound analysis result.

[0121] Furthermore, the dual-channel cardiopulmonary sound intelligent analysis model enables the model to learn time-domain and frequency-domain features simultaneously through the dual-channel structure combining STFT and MFCC features, significantly improving the feature expression ability. Since STFT and MFCC have complementary advantages in audio analysis tasks, the dual-channel analysis model ensures the similarity and consistency of the two features through a customized loss function, thereby better fusing the complementary information of time-frequency features.

[0122] Furthermore, in the adaptive time-frequency fusion module, the time and frequency features are processed independently to capture their respective time-frequency information. Through the corresponding weight setting, the useful parts between the time-frequency features are retained, while the useless parts are discarded, thus achieving effective feature fusion.

[0123] Furthermore, in the time-frequency fusion lightweight convolution module, due to the adoption of the time-frequency feature fusion mechanism based on the channel and spatial dual attention mechanisms and its integration into the whole process of feature extraction, the relationship between the time and frequency features can be dynamically learned, the fusion method of the two can be effectively adjusted, and the fused features can comprehensively reflect the time and frequency characteristics of the cardiopulmonary sound signal. Moreover, through the time-frequency feature fusion method based on the dual attention mechanism, the deep interaction between the time and frequency features is realized, enabling the network to better capture the global and local characteristics of the audio signal, and thus showing stronger performance and robustness in diverse audio tasks.

[0124] Furthermore, in the time-frequency fusion cardiopulmonary sound Transformer module, through the sparse self-attention method based on multi-axis decomposition, combined with the local and global interaction mechanisms, the computational complexity is reduced while the modeling ability of the global information is retained.

[0125] In addition, to verify the effectiveness of the model, the rationality of the dual-channel structure is verified through a series of experiments in the embodiments. The experimental results show that the proposed dual-channel intelligent cardiopulmonary sound analysis algorithm has achieved the latest SOTA (State-of-the-Art) effect in the cardiopulmonary sound analysis task, fully demonstrating the effectiveness and superiority of the dual-channel structure in intelligent cardiopulmonary sound analysis.

[0126] The above embodiments are only used to illustrate the specific implementation manners of the present invention, and the present invention is not limited to the description scope of the above embodiments. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A dual-channel intelligent analysis method for cardiopulmonary sounds based on time-frequency feature fusion, characterized in that It includes the following steps: Step S1, perform data analysis on the cardio-pulmonary sound signal through two feature extraction methods respectively to obtain two feature images with time-domain features and frequency-domain features of the cardio-pulmonary sound; Step S2, transfer the two feature images with time-domain features and frequency-domain features of the cardio-pulmonary sound to the dual-channel cardio-pulmonary sound intelligent analysis model; Step S3, the dual-channel cardio-pulmonary sound intelligent analysis model performs dynamic fusion of features based on the two feature images with time-domain features and frequency-domain features of the cardio-pulmonary sound, and obtains the cardio-pulmonary sound analysis result based on the fusion result, wherein, the dual-channel cardio-pulmonary sound intelligent analysis model includes: An adaptive time-frequency fusion module for performing time-domain and frequency-domain feature fusion on the two feature images with time-domain features and frequency-domain features of the cardio-pulmonary sound to obtain time-frequency features; A time-frequency fusion lightweight convolution module for extracting the primary representation of the time-frequency features and optimizing the feature weights through a channel attention mechanism; and A time-frequency fusion cardio-pulmonary sound Transformer module for enhancing the global feature modeling ability through a multi-axis sparse self-attention mechanism and outputting the cardio-pulmonary sound analysis result.

2. The dual-channel cardio-pulmonary sound intelligent analysis method based on time-frequency feature fusion according to claim 1, characterized in that: Among them, The cardio-pulmonary sound signal includes a heart sound signal and a lung sound signal, Step S1 includes the following sub-steps: Step S1-1, perform noise reduction processing on the heart sound signal and the lung sound signal respectively; Step S1-2, perform two feature extractions of short-time Fourier transform and Mel frequency cepstral coefficients on the denoised heart sound signal and the denoised lung sound signal respectively to obtain two feature maps of the heart sound signal and two feature maps of the lung sound signal; Step S1-3, perform data augmentation on each of the feature maps respectively, so as to form a time-domain signal feature image and a spectral feature image of the cardio-pulmonary sound.

3. The dual-channel cardio-pulmonary sound intelligent analysis method based on time-frequency feature fusion according to claim 2, characterized in that: Among them, In step S1-1, a fifth-order Butterworth band-pass filter is used to filter the heart sound signal and the lung sound signal respectively, For the heart sound signal, the cut-off frequencies of the Butterworth band-pass filter are set to 20 Hz and 800 Hz, For the lung sound signal, the cut-off frequencies of the Butterworth band-pass filter are set to 200 Hz and 1300 Hz.

4. The dual-channel cardio-pulmonary sound intelligent analysis method based on time-frequency feature fusion according to claim 2, characterized in that: Among them, In step S1-3, the data augmentation includes randomly oversampling the feature maps, randomly cropping the feature maps within each batch, and linearly combining the feature maps of different classes through the Mixup data augmentation technique to generate new synthetic samples.

5. The dual-channel cardio-pulmonary sound intelligent analysis method based on time-frequency feature fusion according to claim 1, characterized in that: Among them, The adaptive time-frequency fusion module includes two independent channel branches, a time-frequency fusion attention block, an average pooling layer, a maximum pooling layer, and an activation layer, The channel branch includes a time-domain feature extraction network and a frequency-domain feature extraction network.

6. The intelligent dual-channel cardiopulmonary sound analysis method based on time-frequency feature fusion according to claim 5, wherein: Among them, The adaptive time-frequency fusion module processes the two feature images with time-domain and frequency-domain features of cardiopulmonary sounds through the two channel branches respectively, expressed as: x t = Conv T (BN-Act(Conv T (x))) x f = Conv F (BN-Act(Conv F (x))) where Conv T and Conv F respectively represent convolution kernels with larger spans of the convolution operation on the time axis and the frequency axis. Next, the adaptive time-frequency fusion module uses a dual-channel and spatial attention mechanism to fuse the time features and the frequency features. In each channel branch, the channel information is aggregated through global pooling across the channel dimension, expressed as: S c = Concat(Avg(x t ), Max(x t ), Avg(x f ), Max(x f )) where, S c represents the aggregated channel features, x t and x f respectively represent the time features and frequency features of the cardiopulmonary sound, Avg(·) and Max(·) respectively represent global average pooling and global max pooling across the channel dimension The aggregated channel features are passed to two one-dimensional convolutional layers to determine the channel weights of the two channel branches, and the channel weights are expressed as: W c1 ,W c2 =Conv1(S c ),Conv2(S c ) where W c1 and W c2 respectively represent the channel weights of the cardiorespiratory sound time-frequency features, and Conv1(·) and Conv2(·) represent one-dimensional convolution operations Subsequently, the Softmax function is used to normalize the channel weights of the time-frequency features, expressed as: Where, W c ′ 1 and W c ′ 2 respectively represent the output time-frequency channel weights.

7. The intelligent dual-channel cardiopulmonary sound analysis method based on time-frequency feature fusion according to claim 6, wherein: Among them, The time-frequency fusion lightweight convolution module includes a time-domain convolution block, a frequency-domain convolution block, a pointwise convolution block, a squeeze-and-excitation block, and a downsampling layer.

8. The intelligent dual-channel cardiopulmonary sound analysis method based on time-frequency feature fusion according to claim 7, wherein: Among them, [[ID=h]]The time-frequency fusion lightweight convolution module decomposes the input features into time-domain features and frequency-domain features, fuses the time-domain features and the frequency-domain features through a time-frequency fusion mechanism, then enhances the weights of important features through the squeeze-and-excitation module, adjusts the channel dimension through pointwise convolution, and finally performs a residual connection between the adjusted features and the downsampled features to form output features, expressed as: x = BN-Act(Conv 1×1 (BN(X))) x = TFFM(x) x = SE(BN-Act(x)) x = Conv 1×1 (x) Wherein, BN represents batch normalization, Act represents an activation function, and Conv 1×1 represents pointwise convolution.

9. The intelligent dual-channel cardiopulmonary sound analysis method based on time-frequency feature fusion according to claim 1, wherein: Among them, The time-frequency fusion cardiopulmonary sound Transformer module includes a window self-attention block, a feed-forward neural network block, and a grid self-attention block.

10. The dual-channel cardiorespiratory sound intelligent analysis method based on time-frequency feature fusion according to claim 1, It is characterized in that: Among them, the training of the intelligent dual-channel cardiopulmonary sound analysis model includes the following steps: Step T1, construct an initial dual-channel cardiopulmonary sound analysis model, which includes the adaptive time-frequency fusion module, the time-frequency fusion lightweight convolution module, and the time-frequency fusion cardiopulmonary sound Transformer module; Step T2, input a training set containing multiple groups of cardiopulmonary sound data into the initial dual-channel cardiopulmonary sound analysis model in sequence for model training and perform iteration; Step T3, calculate the loss error using the model parameters of the last layer respectively and backpropagate the calculated loss error to update the model parameters; Step T4, repeat Step T2 to Step T3 until a predetermined training completion condition is reached to obtain the intelligent dual-channel cardiopulmonary sound analysis model.

Images