An end-to-end heart sound segmentation method based on Residual Bi-LSTM network

Abstract

The present invention relates to the field of heart sound segmentation in deep learning, and in particular proposes a residual long-short term memory neural network (Residual Bidirectional Long-Short Term Memory Neural Networks) technology. The end-to-end heart sound segmentation method based on the Residual Bi-LSTM network described in the present invention mainly includes the following steps: Step 1: synchronously collecting heart sound and electrocardiogram (ECG) signals of healthy and diseased volunteers; Step 2: dividing the heart sound signals collected in Step 1 into several segments according to a certain length L; Step 3: annotating the collected heart sound signals according to the gold standard corresponding to the heart sound and ECG, and organizing them into a data set; Step 4: using the data obtained in Step 2 to train the Residual Bidirectional LSTM initial model; Step 5: using the above-mentioned model trained with the data set to perform forward propagation calculations on new heart sound signals not in the data set, and finally achieving the task of segmenting the heart sound signals.

Description

Technical Field

[0001] The present invention relates to the field of heart sound segmentation in deep learning, and in particular proposes a residual bidirectional long-short term memory neural network (LSTM) technology for use in the field of heart sound segmentation. Background Art

[0002] By definition, a phonocardiogram (PCG) is a graphical representation of heart sound vibrations converted into a time-series recording of vibration waves. While serving as a crucial basis for early diagnosis of heart disease, it also plays a crucial role in the final diagnosis. A phonocardiogram probe records heart sounds and heart murmurs for clinical analysis, aiding in diagnosing the causes of heart disease and understanding the mechanisms of heart murmur generation. During auscultation, heart sounds are characterized by high timeliness and regularity, making in-depth analysis of heart sound signals crucial. Normal heart sound signals are primarily divided into the first heart sound (S1), systole, second heart sound (S2), and diastole. Systole refers to the time interval between the start of S1 and the start of S2, while diastole refers to the time interval between the start of S2 and the start of S1 in the next cardiac cycle. These four components constitute a complete cardiac cycle, and the time-frequency characteristics of each component often correspond to different types of heart disease. For example, the first heart sound is a mechanical wave caused by the sudden closure of the atrioventricular valve and the subsequent ejection of blood into the aorta. If this heart sound splits, it is often considered to be due to pulmonary hypertension and delayed tricuspid valve closure. For another example, pulmonary hypertension often manifests as an exaggerated second heart sound, so the specific manifestation of the second heart sound is more closely monitored during analysis. Furthermore, the manifestations of different components of heart sound in different auscultation areas can vary significantly. Different categories of heart sounds are expressed differently in their different components. The significance of accurate segmentation is to conduct a more in-depth analysis of heart sounds in the time domain, frequency domain and medical field in order to extract better features that are conducive to distinguishing each category, thereby achieving higher accuracy and reliability. In view of this, accurate segmentation of heart sounds becomes crucial.

[0003] Chinese patent application No. 202110121993.8 discloses a traffic status prediction method based on the ResLS-C deep learning combination. The steps of this method are as follows: Step 1: Construct a multi-layer convolutional neural network with a residual structure to achieve the effect of extracting spatial features, Step 2: Construct an LSTM network, and use the output of the network in step 1 as input to extract the temporal features of the relevant time, Step 3: Construct a deconvolution layer, and the depooling layer performs a dimensionality reduction operation on the high-dimensional features and restores them to the original space to make a prediction of the traffic status. This algorithm mainly uses the residual network to perform feature mapping on the original information, which is similar to the role of the Embedding layer in text processing, mapping the input into a feature vector; the structure is reflected in that the residual convolutional neural network is used to perform feature mapping on the input of each time step, and then, after obtaining the mapping result, it is put into the LSTM-Cell for learning. Its essence is: using the residual network to perform feature mapping on the input sequence to achieve the purpose of improving the input of the LSTM-Cell. In terms of method, the patent's method of using residual convolutional networks (Residual Convolutional Networks) to perform feature mapping before the sequence is input into the long short-term memory neural network cell (LSTM Cell) is different from the method proposed in this patent of combining the identity mapping in the Residual Networks with the Bi-LSTM.

[0004] Chinese patent application No. 202110850022.7 discloses a modeling method and transfer learning method of a deep residual LSTM network and a thermal error prediction model. The steps of the network are as follows: Step 1: The input sequence is convolved and pooled to extract the features of the input data. Step 2: The output in Step 1 passes through a reshaping layer to obtain a sequence output after mapping by the previous convolution layer. Step 3: Pass through the LSTM layer to learn the contextual relationship between sequences. After obtaining the result, it is standardized through the BN (Batch Normalization) layer, and then connected to a deep residual network for feature learning. The essence of its network structure is to achieve the best result by splicing multiple network structures so that multiple blocks perform their respective functions. Summary of the Invention

[0005] The purpose of the present invention is to improve the network based on the residual identity mapping technology in the residual network and the advantages of the Bi-LSTM network for sequence data processing, and to provide an end-to-end heart sound segmentation method based on the Residual Bi-LSTM network.

[0006] The present invention describes an end-to-end heart sound segmentation method based on a Residual Bi-LSTM network. This method, which eliminates the complex feature engineering typically performed on audio signals and directly generates segmentation results from heart sound input, provides a more convenient method for analyzing various heart diseases, including obtaining the first heart sound (S1), systole, second heart sound (S2), and diastole of the cardiac cycle. Based on this application, the structure of the Residual Bi-LSTM network and its potential applications are summarized.

[0007] The first basis is the identity mapping effect of the residual structure proposed by Kaiming He in the Deep Residual Convolutional Neural Network in 2017, which addresses the problem of network degradation caused by increasing network depth. The basic idea is to introduce a shortcut branch so that the network can reach a saturation state in feature learning during the continuous stacking process (that is, when the network features of the current layer have achieved relatively good performance), the subsequent layers can perform forward propagation calculations with slight adjustments to the features, or even no adjustments at all.

[0008] The second basis is that the LSTM network has the function of mapping the contextual relationship between sequences into a multi-dimensional feature vector. The superposition of multiple layers of LSTM can better learn the relationship between contexts. Combined with the idea of ​​bidirectional propagation calculation, the forward context features and directional context features of the sequence can be learned into a multi-dimensional feature vector. The multi-layer Bi-LSTM network can better learn the contextual features of the input sequence into the feature vector (hierarchically). For example, when we try to identify the semantics of a sentence, we usually first grasp its subject-verb-object structure, and then understand its structural features such as adverbials, complements, attributives and predicatives in a hierarchical manner. In other words, in a deep LSTM, learning the contextual relationship between sequences layer by layer can achieve better results.

[0009] The end-to-end heart sound segmentation method based on the Residual Bi-LSTM network described in the present invention mainly includes the following steps:

[0010] Step 1: Synchronously collect heart sounds and ECG signals from healthy and diseased volunteers, ensuring that each acquisition covers multiple cardiac cycles.

[0011] Step 2: Divide the heart sound signal collected in Step 1 into several segments of a certain length L. The fixed length should be guaranteed to contain at least one cardiac cycle.

[0012] Step 3: Label the collected heart sound signals according to the gold standard corresponding to heart sound and electrocardiogram, and organize them into a data set;

[0013] Step 4: Use the data obtained in Step 2 to train the Residual Bidirectional LSTM initial model;

[0014] Step 5: Use the Residual Bidirectional LSTM network model trained with the dataset to perform forward propagation calculations on new heart sound signals that are not in the dataset, and finally achieve the task of segmenting the heart sound signals.

[0015] The heart sound signal and electrocardiogram signal described in Step 1 are PCG and ECG respectively. The sampling frequency of the heart sound signal is determined by the acquisition device and can be adjusted using downsampling technology according to the computing capacity of the computer during processing.

[0016] When segmenting the heart sound signal in Step 2, the following two factors must be considered: First, the length L of the segmented heart sound signal may be too long, causing the computer to be unable to calculate due to insufficient memory. Second, segments that are less than the fixed length L can be discarded by setting a threshold or padded with zeros to the fixed length L. A tensor is set to record the real length Real-L of each heart sound sequence. Assuming the number of heart sound sequences in each batch is Batch size, the shape of Real-L is Batch size * 1, and each element represents the real length of the corresponding heart sound sequence.

[0017] Step 3: The gold standard annotation of the central tone and the preparation of the dataset are mainly divided into the following two steps:

[0018] Step 301: The gold standard mentioned in the preceding text refers to the electrocardiogram (ECG) with the R peak corresponding to the first heart sound S1, followed by the systole, the T wave corresponding to the second heart sound S2, and finally the diastole.

[0019] Step 302: Create a dataset with the labeled data and split it into a validation set and a training set in a ratio of 1:4.

[0020] The network training in Step 4 is mainly divided into two processes: forward propagation calculation and backward propagation calculation. The calculation process is shown in the following steps:

[0021] Step 401: After the original heart sound signal is segmented into segments of fixed length L, the embedding layer maps the data at each time point from one dimension to a higher dimension to better represent the relationship between each time node.

[0022] Step 402: Input the high-dimensional data obtained in Step 401 into the Residual Bi-LSTMBlock for calculation. The calculation process is as follows:

[0023] f t =σ(W f ·[h t-1 , x t ]+b f )#(1)

[0024] i t =σ(W i ·[h t-1 , x t ]+b i )#(2)

[0025]

[0026]

[0027] o t =σ(W o ·[h t-1 , x t ]+b o )#(5)

[0028] h t =o t *tanh(C t )#(6)

[0029] In the formula, all W and b are trainable model weight parameters, the subscript t represents the time node, σ represents the sigmoid function, f represents the proportion of the information of the previous time node that needs to be forgotten at the current time node, and i represents the memory rate of the information of the current time node. represents the information mapping result of the current time node, C represents the feature mapping result of the selected memory information of the current time node, o represents the information memory rate of the current time node, and h represents the hidden state calculation result of the current time node. Typically, it is first mapped to a high-dimensional space and then mapped to the final state value through the fully connected layer. In the formula, "·" represents matrix multiplication, and "*" represents element-by-element multiplication of matrix elements.

[0030] A heart sound sequence undergoes the aforementioned forward and reverse calculation processes to complete a Bi-LSTM layer operation. Given the tensor Real-L, the Bi-LSTM calculation is performed only up to the real time length; the padding mentioned in Step 2 is not calculated. Finally, the hidden state values ​​of all time nodes are returned at the end of the calculation, yielding the result of a Bi-LSTM layer.

[0031] Step 403: The result of Step 402 is then passed through two layers of Bi-LSTM to obtain the result f(x). If the Residual Bi-LSTM Block is the first Block, then x refers to the result of the input heart sound sequence after word embedding (Embedding layer) mapping. If not, then x refers to the output of the previous Residual Bi-LSTM Block, and f refers to the mapping formed by the three layers of Bi-LSTM.

[0032] Step 404: Save the calculation result of the first layer Bi-LSTM as s(x), where s is the mapping constructed by the first layer Bi-LSTM;

[0033] Step 405: Add the result f(x) obtained in Step 403 and the result s(x) obtained in Step 404 to obtain the output result h(x) of a Residual Bi-LSTM Block. Here, h refers to a complete Residual Bi-LSTM Block mapping:

[0034] h(x)=f(x)+s(x)#(7)

[0035] Step 406: Multiple Residual Bi-LSTM blocks with different structures are superimposed on each other to obtain a structure that is mapped by a fully connected layer and then passes through SoftMax.

[0036] Step 407: The result of Step 406 is passed through the loss function to calculate the loss, and then the parameters in the model are optimized and updated according to the gradient of the loss function for all trainable parameters in the model, that is, backpropagation calculation, thereby completing the training process.

[0037] The idea in Step 5 is to call the network trained in Step 4 to perform the task of heart sound segmentation. That is, a heart sound signal and its real sequence length Real-L are directly input into the trained model. After the network performs forward propagation calculations, the state of each non-filled time node of the input heart sound is obtained.

[0038] Existing Bi-LSTM networks complete common sequence processing tasks by stacking layers, and this stacking method easily causes network degradation. The Residual Bi-LSTM network proposed in the present invention establishes shortcut branches between layers through a residual structure, which can adjust the output results of the intermediate layers during the network forward propagation calculation process, so that the output of the Embedding layer or the previous Residual Bi-LSTM Block only fine-tunes the features of the previous layer or even does not adjust them after the loss is reduced to a certain level. That is, the data is mainly forward propagated through shortcut branches. In this way, deep stacking of Bi-LSTM can be achieved, thereby completing complex sequence processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to provide further understanding and reference for the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention but do not constitute a limitation of the present invention.

[0040] Figure 1 This is one of the schematic diagrams for performing signal segmentation in Step 2 of the present invention.

[0041] Figure 2 This is one of the schematic diagrams of the heart sound segmentation gold standard mentioned in Step 3 of the present invention.

[0042] Figure 3 This is one of the experimental images in Step 3 of the present invention for labeling heart sounds using the heart sound and electrocardiogram gold standard method.

[0043] Figure 4 This is one of the internal structure diagrams of the LSTM described in Step 4 of the present invention.

[0044] Figure 5 It is a structure of the Residual Bi-LSTM Block mentioned in S402 in Step 4 of the present invention.

[0045] Figure 6 This is another structure of the Residual Bi-LSTM Block mentioned in Step 402 of Step 4 of the present invention.

[0046] Figure 7 This is one of the structural diagrams of the Residual Bi-LSTM neural network proposed in this invention. DETAILED DESCRIPTION

[0047] This embodiment provides an end-to-end heart sound segmentation method based on a Residual Bi-LSTM network. In addition to heart sound segmentation, this method can also be applied to named entity recognition in natural language processing. Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are intended only to illustrate and explain the present invention and are not intended to limit the present invention.

[0048] Example 1:

[0049] like Figure 6 As shown, the present invention provides an end-to-end heart sound segmentation method based on the Residual Bi-LSTM network, and its specific implementation steps are as follows:

[0050] Step 1: Build a database for network training by synchronously collecting heart sounds (PCG) and electrocardiogram (ECG);

[0051] Step 2: Label the collected data according to the gold standard, that is, label each time node with a label value of 0, 1, 2, and 3, corresponding to S1, systole, S2, and diastole, respectively, to obtain a labeled database;

[0052] Step 3: To meet computer computing capacity requirements, in this embodiment, the sampling frequency of the heart sound signal is downsampled to 5000 Hz. To ensure that each segment contains at least one cardiac cycle, the length of each signal is controlled to 10,000 time nodes, that is, a time length of 2 seconds. In other words, the shape of the input signal after segmentation and padding is 1×10,000. A tensor Real-L is set to record the real sequence length of the heart sound sequence (see Steps 2 and 4 of the Summary of the Invention for related descriptions).

[0053] Step 4: Import the segmented labeled data into the network for training. During the training process, each piece of data passes through each layer of the network, assuming that there is only one Figure 4 The Residual Bi-LSTM Block structure is shown in the figure, and the parameter dimensions (also called number units) of each Bi-LSTM layer of the Block are 16, 32, and 16 respectively. Figure 6 In the case shown, the shape of the heart sound data changes when it is calculated in each layer of the network as shown in Table 1:

[0054] Table 1. Shape changes of data calculated in the network (Structure 1)

[0055]

[0056] If the Residual Bi-LSTM Block structure is Figure 5 When the structure shown in the figure is used, and the parameter dimensions (also called number units) of the Bi-LSTM in each layer of the Block are 16, 32, 64, and 64 for the shortcut branch, the shape of the heart sound data changes as shown in Table 2:

[0057] Table 2. Shape changes of data calculated in the network (Structure 2)

[0058] The output of the output layer corresponds to the state of each time node, that is, 0, 1, 2 or 3.

[0059] The refinement steps involved in Step 4 mainly include the following steps:

[0060] Step 401: The original heart sound signal input is 1×10000 data. After passing through the Embedding layer, each time is mapped from one dimension to a higher dimension to better represent the relationship between each time node.

[0061] Step 402: Input the high-dimensional data obtained in Step 401 into the Residual Bi-LSTMBlock for calculation. The calculation process is as follows:

[0062] f t =σ(W f ·[h t-1 , x t ]+b f )#(1)

[0063] i t =σ(W i ·[h t-1 , x t ]+b i )#(2)

[0064]

[0065]

[0066] o t =σ(W o ·[h t-1 , x t ]+b o )#(5)

[0067] h t =o t *tanh(C t )#(6)

[0068] After a heart sound sequence is mapped by word embedding (Embedding layer), it undergoes the forward calculation process and the reverse calculation process described above to complete a Bi-LSTM layer operation. Finally, at the end of the operation, the hidden state values ​​of all time nodes are returned to obtain the operation result of a Bi-LSTM layer.

[0069] Step 403: The result of Step 402 is then passed through two layers of Bi-LSTM to obtain the result f(x). If the Residual Bi-LSTM Block is the first block, then x refers to the input heart sound sequence. If not, then x refers to the output of the previous block, and f refers to the mapping formed by the three layers of Bi-LSTM.

[0070] Step 404: Save the calculation result of the first layer Bi-LSTM as s(x), where s is the mapping constructed by the first layer Bi-LSTM;

[0071] Step 405: Add the result f(x) obtained in Step 403 and the result s(x) obtained in Step 404 to obtain the output result h(x) of the Residual Bi-LSTM Block. Here, h refers to a complete Residual Bi-LSTM Block mapping:

[0072] h(x)=f(x)+s(x)#(7)

[0073] Step 406: Multiple Residual Bi-LSTM blocks with different structures are superimposed on each other to obtain a structure that is mapped by a fully connected layer and then passes through SoftMax.

[0074] Step 407: Calculate the loss using the loss function for the result of Step 406, and then optimize and update the parameters in the model based on the gradient of the loss function for all trainable parameters in the model, i.e., backpropagation calculation, to complete the training process.

[0075] Step 5: After 50 epochs of training, plot the relationship between the loss and accuracy of the model training and test sets. Determine the approximate number of epochs in which the data converges to the optimal value, and then save the weight parameters of the optimal model.

[0076] Step 6: In the prediction phase, the model weight parameters saved in the previous training phase are directly called to perform forward propagation calculations. That is, the prediction process no longer performs backpropagation calculations to update parameters, and directly performs forward propagation calculations to give prediction results, thus completing the task of heart sound segmentation.

Claims

1. An end-to-end heart sound segmentation method based on Residual Bi-LSTM network, characterized by The following steps are involved: Step 1: Synchronously collect heart sounds and ECG signals from healthy and diseased volunteers, ensuring that each acquisition duration covers several cardiac cycles. Step 2: Divide the heart sound signal collected in Step 1 into several segments of a certain length L. The certain length L should be guaranteed to contain at least one cardiac cycle. Segments shorter than the length L are padded or discarded by setting a threshold. Step 3: Label the collected heart sound signals according to the gold standard corresponding to heart sound and ECG, and organize them into a data set; Step 4: Use the data obtained in Step 3 to train the Residual Bidirectional LSTM initial model; Step 5: Use the Residual Bidirectional LSTM network model trained on the dataset to perform forward propagation calculations on new heart sound signals that are not in the dataset, and finally achieve the task of segmenting the heart sound signals; The network training in Step 4 is mainly divided into two processes: forward propagation calculation and backward propagation calculation. The calculation process is shown in the following steps: Step 401: After the original heart sound signal is segmented into segments of a certain length L, the embedding layer maps the data at each time point from one dimension to a higher dimension to better represent the relationship between each time node. Step 402: Input the high-dimensional data obtained in Step 401 into the first Bi-LSTM layer of the Residual Bi-LSTM Block for calculation; Step 403: The result of Step 402 is then passed through two layers of Bi-LSTM to obtain the result f(x). If the Residual Bi-LSTM Block is the first Block, then x refers to the result of the input heart sound sequence mapped by the Embedding layer. If not, then x refers to the output of the previous Residual Bi-LSTM Block, and f refers to the mapping formed by the three layers of Bi-LSTM. Step 404: Save the calculation result of the first layer Bi-LSTM as s(x), where s is the mapping constructed by the first layer Bi-LSTM; Step 405: Add the result f(x) obtained in Step 403 and the result s(x) obtained in Step 404 to obtain the output result h(x) of a Residual Bi-LSTM Block. Here, h refers to a complete Residual Bi-LSTM Block mapping: h(x)=f(x)+s(x) Step 406: Multiple Residual Bi-LSTM blocks with different structures are superimposed on each other to obtain a structure that is mapped by a fully connected layer and then passes through SoftMax. Step 407: The result of Step 406 is passed through the loss function to calculate the loss, and then the parameters in the model are optimized and updated according to the gradient of the loss function for all trainable parameters in the model, that is, backpropagation calculation, thereby completing the training process.

Images