Segmental ventricular wall motion analysis method and system based on left ventricular opacity

Abstract

The invention belongs to the technical field of artificial intelligence heart movement analysis, and relates to a segmental ventricular wall movement analysis method and system based on left ventricular opacity. The method comprises the following steps: segmenting a left ventricle contour in a four-cavity section, a two-cavity section and a three-cavity section of a heart apex of an LVO video sequence by using a segmentation network; dividing the left ventricular wall into a plurality of segments, and extracting node features to obtain a sample set; constructing three same space-time diagram convolutional networks; and respectively inputting the node features into classifiers of the three space-time diagram convolutional networks, and respectively detecting segmental ventricular wall motion states related to the three coronary artery branches. The left ventricle contour is segmented by using the segmentation network, and the left ventricle wall is divided into a plurality of segments, so that the sample size of model training is greatly reduced; fusion modeling is carried out on multiple sections by utilizing GCN, a three-dimensional myocardial topological structure is combined, and a corresponding relation between each coronary artery branch and a myocardial segment is considered, so that classification of ventricular wall movement is more detailed.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence cardiac motion analysis, and in particular, relates to a segmental wall motion analysis method and system based on left ventricular opacity. Background Art

[0002] Coronary heart disease refers to a heart disease caused by myocardial ischemia and hypoxia due to coronary atherosclerosis, which seriously affects human health. The branches of the coronary arteries supply blood to different areas of the myocardium, ensuring the myocardial motor function. If some branches are narrowed or blocked, the myocardial area they control will be ischemic or even have reduced mobility, that is, segmental wall motion abnormalities. In clinical practice, the assessment of left ventricular motor function, especially segmental wall motion, has become increasingly important in determining the severity and prognosis of coronary heart disease.

[0003] Echocardiography has high temporal and spatial resolution and is an ideal noninvasive method for evaluating ventricular wall motion. Left ventricular opacification (LVO) using intravenous ultrasound contrast agents can more clearly display the cardiac structure and further improve the accuracy of wall motion analysis. However, traditional wall motion analysis is based on manual observation and measurement of endocardial excursion and myocardial thickening, relying on the subjective experience of ultrasound physicians and having low reproducibility.

[0004] Currently, most deep learning methods for automatically analyzing myocardial function are based on quantitative measurements, usually measuring indicators such as ejection fraction and strain based on automatic segmentation of the left ventricle; existing convolutional neural networks are used to directly classify segmental wall motion qualitatively, but no left ventricular segmentation is performed; automatically measured myocardial function-related indicators can only reflect the wall motion to a certain extent and are used for auxiliary diagnosis, and are highly dependent on the accuracy of left ventricular segmentation. Due to the poor quality and strong interference of ultrasound images, the performance of existing deep learning image segmentation methods in echocardiography, especially contrast ultrasound, is unstable; for measurement methods that do not rely on left ventricular segmentation, there is a large amount of redundant information outside the left ventricular area in ultrasound images. If the region of interest is not extracted (i.e., left ventricular image segmentation), more data is needed to accurately train the network. However, large-scale and comprehensive samples are difficult to obtain in clinical practice. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a segmental wall motion analysis method and system based on left ventricular opacity.

[0006] In a first aspect, the present invention provides a segmental wall motion analysis method based on left ventricular opacification, comprising: Introduce dilated convolution into residual convolution and build a segmentation network based on U-Net and Transformer; The left ventricular contour was segmented using a segmentation network in the apical four-chamber, two-chamber, and three-chamber views of the LVO video sequence. According to the segmented left ventricular contour, the left ventricular wall is divided into several segments, the center of each segment is taken as a node, and the two-dimensional coordinates are extracted as node features to obtain a sample set; Construct three identical spatiotemporal graph convolutional networks; the spatiotemporal graph convolutional network is composed of multi-faceted spatial convolution and multi-scale temporal convolution alternately, and finally classified by the classifier; the multi-faceted spatial convolution includes several multi-head self-attention graph convolution modules; The node features of each node in the sample set are input into the classifiers of three spatiotemporal graph convolutional networks respectively to detect the segmental wall motion states related to the left anterior descending artery, the left circumflex artery and the right coronary artery, and the trained spatiotemporal graph convolutional network is obtained; The detected segmental wall motion status was analyzed.

[0007] In a second aspect, the present invention provides a left ventricular segmental wall motion analysis system, comprising a first construction unit, a segmentation unit, a sample extraction unit, a second construction unit, a training unit and a detection unit; The first construction unit is used to introduce the hole convolution into the residual convolution and build a segmentation network based on U-Net and Transformer; A segmentation unit, for segmenting the left ventricular contour in the apical four-chamber view, the two-chamber view, and the three-chamber view of the LVO video sequence using a segmentation network; A sample extraction unit is used to divide the left ventricular wall into several segments, take the center of each segment as a node, extract the two-dimensional coordinates as node features, and obtain a sample set; The second construction unit is used to construct three identical spatiotemporal graph convolutional networks; the spatiotemporal graph convolutional network is composed of multi-faceted spatial convolution and multi-scale temporal convolution alternately, and finally classified by a classifier; the multi-faceted spatial convolution includes several multi-head self-attention graph convolution modules; A training unit, used for inputting node features of each node in the sample set into the classifiers of three spatiotemporal graph convolutional networks respectively, detecting the segmental wall motion states related to the left anterior descending artery, the left circumflex artery and the right coronary artery respectively, and obtaining a trained spatiotemporal graph convolutional network; Based on the above technical solution, the present invention can also be improved as follows.

[0008] Furthermore, the segmentation network includes a residual dilated convolution, a Transformer encoder, a Transformer decoder, a 2x2 max pooling layer, a 3x3 convolution layer, an activation layer, and a 1x1 convolution layer, which are connected in sequence.

[0009] Further, let Q be the query value, K be the key value, and d be the intermediate dimension. is the total number of expansion heads of the multi-head self-attention graph convolution module, i is the number of expansion heads of the multi-head self-attention graph convolution module, A is the myocardial topology of the multi-head self-attention graph convolution module, is a hyperparameter, and the attention matrix of the i-th expansion head is , is the query value of the number of extension headers of the ith is the key value of the i-th extension header, T is the number of frames in a video sequence, Represents the activation function. In the multi-head self-attention graph convolution module, Q and K are obtained by convolution and average pooling of T, then: .

[0010] Furthermore, let the node feature input of each node in the sample set be , the Value of the i-th extended head obtained by convolution is , is the total number of extended heads of the multi-head self-attention graph convolution module, is the first layer of the multi-head self-attention graph convolution module The transposed matrix of the attention matrix of the extended head, the output of the multi-head self-attention graph convolution module is ,but: .

[0011] Furthermore, the first-layer multi-head self-attention graph convolution module in the multi-section spatial convolution uses the two-dimensional left ventricular structure topology of each section, and the second-layer multi-head self-attention graph convolution module uses the three-dimensional global topology of the left ventricle; the correspondence between the three-dimensional global topological structure of the left ventricle and the 18 segments of the left ventricular wall under the short-axis section is established.

[0012] Furthermore, the multi-scale temporal convolution contains four channels and a ReLU activation layer; the first channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 1, the second channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 2, the third channel includes a 1x1 convolution and a 4x1 average pooling layer, and the fourth channel includes a 1x1 convolution; the output ends of the four channels are respectively connected to the ReLU activation layer.

[0013] The beneficial effects of the present invention are as follows: after refining the left ventricular wall into multiple segments, the present invention uses spatiotemporal graph convolution to model and classify. Different from the method of directly classifying in a single slice without using image segmentation, the present invention uses a segmentation network to segment the left ventricular contour, divides the left ventricular wall into several segments, first extracts and refines the region of interest, and greatly reduces the sample size of model training; GCN is used to fuse multiple slices for modeling, fully considering the correspondence between each coronary artery branch and myocardial segment, making the classification of ventricular wall motion more detailed, which is conducive to targeted adjustment of the myocardial topological structure and improving the sensitivity of apical segment detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of the segmental wall motion analysis method based on left ventricular opacification provided in Example 1 of the present invention; Figure 2 It is a flow chart of the segmental wall motion analysis method based on left ventricular opacification of the present invention; Figure 3 The schematic diagram of the multi-slice spatial convolution and MSAGC module; Figure 4 It is a schematic diagram of the distribution and global topology of the 18 segments of the left ventricular wall; Figure 5 This is the principle diagram of multi-scale temporal convolution; Figure 6 This is a schematic diagram of the left ventricular segmental wall motion analysis system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0016] Example 1 As an example, Figure 1 As shown, in order to solve the above technical problems, this embodiment provides a segmental wall motion analysis method based on left ventricular opacity, including: Introduce dilated convolution into residual convolution and build a segmentation network based on U-Net and Transformer; The left ventricular contour was segmented using a segmentation network in the apical four-chamber (A4C), two-chamber (A2C) and three-chamber (A3C) views of the LVO video sequence. According to the segmented left ventricular contour, the left ventricular wall is divided into several segments, the center of each segment is taken as a node, and the two-dimensional coordinates are extracted as node features to obtain a sample set; Construct three identical spatiotemporal graph convolutional networks; the spatiotemporal graph convolutional network is composed of multi-faceted spatial convolution and multi-scale temporal convolution alternately, and finally classified by the classifier; the multi-faceted spatial convolution includes several multi-head self-attention graph convolution modules; The node features of each node in the sample set are input into the classifiers of three spatiotemporal graph convolutional networks respectively to detect the segmental wall motion states related to the left anterior descending artery, the left circumflex artery and the right coronary artery, and the trained spatiotemporal graph convolutional network is obtained; The detected segmental wall motion status was analyzed.

[0017] In actual application, as shown in the attached Figure 2 As shown in the figure, the left ventricular contour is segmented in the apical four-chamber view, two-chamber view, and three-chamber view of the LVO video sequence; the left ventricular wall is divided into 18 segments (6 segments in each view), and the center of each segment is taken as a node; these nodes are input into three spatial temporal graph convolution network (ST-GCN) classifiers to detect segmental wall motion abnormalities related to the left anterior descending artery (LAD), left circumflex artery (LCX), and right coronary artery (RCA), respectively. Segmental wall motion states include normal state and abnormal state, and the weakened amplitude of segmental wall motion is an abnormal state. An improved image segmentation network based on U-Net and Transformer is used to segment the left ventricular contour in low-quality and strongly interfering contrast echocardiograms such as LVO.

[0018] The present invention introduces dilated convolution into residual convolution, which improves the ability to extract edge features in complex textures of images; uses an improved multi-head self-attention (MSA) mechanism to reduce time complexity and change the position encoding method, so that Transformer can be applied to shallower layers of the U-Net network, further improving the global consistency of segmentation.

[0019] The segmented left ventricular wall is divided into 18 segments in three sections (6 segments in each section), the center of each segment is taken as a node, and its two-dimensional coordinates are obtained as node features. Let T be the number of frames in a video sequence, N be the number of nodes in each frame, that is, 18, and C be the number of node feature channels, that is, 2, to obtain the sample .

[0020] The three ST-GCNs have exactly the same structure, but they are trained using different case samples. The training set used by the RCA classifier includes samples and abnormal samples of the RCA segment; the training set used by the LAD classifier includes samples and abnormal samples of the LAD segment; the training set used by the LCX classifier includes samples and abnormal samples of the LCX segment. Abnormal samples are those with weakened motion amplitude in some myocardial segments. Each ST-GCN classifier performs a binary classification function to diagnose whether the three coronary artery branches are normal or not. For example, each ST-GCN is composed of 10 multi-section spatial convolutions and 10 multi-scale temporal convolutions alternately, and finally a Softmax classifier is used for classification.

[0021] Multi-plane spatial convolution: as shown in the attached Figure 3 As shown, Figure 3 (a) is a multi-head self-attention graph convolution module (MSAGC), Q is the query value, K is the key value, and V is the Value value of the extended head. The values ​​of Q, K, and V represent the Query value, Key value, and Value value of the Transform model respectively. is the query value of the first extended header, It is the key value of the first extension header, represented by X:[T,N,h]. In the figure (b), it is a multi-faceted spatial convolution, which consists of several MSAGCs, each of which uses a different topological structure. X:[T,18,h] represents the input node feature, X:[T,6,h] is split from X:[T,18,h], and X:[T,18,C'] represents the output node feature.

[0022] Optionally, the segmentation network includes a residual dilated convolution, a Transformer encoder, a Transformer decoder, a 2x2 maximum pooling layer, a 3x3 convolution layer, an activation layer, and a 1x1 convolution layer, which are connected in sequence.

[0023] Optionally, let Q be the query value, K be the key value, and d be the intermediate dimension. is the total number of expansion heads of the multi-head self-attention graph convolution module, i is the number of expansion heads of the multi-head self-attention graph convolution module, A is the myocardial topology of the multi-head self-attention graph convolution module, is a hyperparameter, and the attention matrix of the i-th expansion head is , is the query value of the number of extension headers of the ith is the key value of the i-th extension header, T is the number of frames in a video sequence, Represents the activation function. In the multi-head self-attention graph convolution module, Q and K are obtained by convolution and average pooling of T, then: .

[0024] Optionally, let the node feature input of each node in the sample set be , the Value of the i-th extended head obtained by convolution is , is the total number of extended heads of the multi-head self-attention graph convolution module, is the first layer of the multi-head self-attention graph convolution module The transposed matrix of the attention matrix of the extended head, the output of the multi-head self-attention graph convolution module is ,but: .

[0025] Optionally, the first-layer multi-head self-attention graph convolution module in the multi-plane spatial convolution uses the two-dimensional left ventricular structure topology of each plane, and the second-layer multi-head self-attention graph convolution module uses the three-dimensional global topology of the left ventricle; a correspondence between the three-dimensional global topological structure of the left ventricle and the 18 segments of the left ventricular wall under the short-axis plane is established.

[0026] As attached Figure 3 As shown in (b), the first layer of MSAGC module in the multi-section spatial convolution uses the two-dimensional left ventricular structural topology of each section, and the second layer of MSAGC module uses the three-dimensional global topology of the left ventricle. The corresponding relationship between the structure and the 18 segments of the left ventricular wall under the short axis section is shown in the attached figure. Figure 4 Attached Figure 4 (a) is a schematic diagram of the distribution of the 18 segments of the left ventricle. Figure 4(b) is the global topology of 18 segments. The 16-node topology of the apical region forms a complete graph, because the apical region is mainly dominated by the LAD and the segments are highly correlated with each other. In addition, the motion abnormality of the apical segment is smaller than that of other parts. Using a complete graph can enhance the information propagation between nodes and increase the sensitivity of the model to the motion abnormality of the apical segment.

[0027] Optionally, the multi-scale temporal convolution includes four channels and a ReLU activation layer; the first channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 1, the second channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 2, the third channel includes a 1x1 convolution and a 4x1 average pooling layer, and the fourth channel includes a 1x1 convolution; the output ends of the four channels are respectively connected to the ReLU activation layer.

[0028] As attached Figure 5 As shown in the figure, there are 4 channels in the multi-scale temporal convolution, which extract features of different scales in the time dimension T. The maximum receptive field is 9, which is close to half a cardiac cycle. The large-scale receptive field and average pooling operation effectively reduce the error caused by inaccurate segmentation of a certain frame during the image segmentation stage.

[0029] After refining the left ventricular wall into 18 segments, the present invention uses spatiotemporal graph convolution to model and classify. Different from the method of directly classifying in a single slice without using image segmentation, the present invention uses a segmentation network to segment the left ventricular contour and divides the left ventricular wall into several segments. The region of interest is first extracted and refined, which greatly reduces the sample size of model training; GCN is used to fuse multiple slices for modeling, fully considering the correspondence between each coronary artery branch and myocardial segment, making the classification of ventricular wall motion more detailed, which is conducive to targeted adjustment of the myocardial topological structure and improving the sensitivity of apical segment detection.

[0030] Example 2 Based on the same principle as the method shown in Example 1 of the present invention, as shown in the attached Figure 6 As shown, an embodiment of the present invention also provides a left ventricular segmental wall motion analysis system, including a first construction unit, a segmentation unit, a sample extraction unit, a second construction unit, a training unit and a detection unit; The first construction unit is used to introduce the hole convolution into the residual convolution and build a segmentation network based on U-Net and Transformer; A segmentation unit, for segmenting the left ventricular contour in the apical four-chamber view, the two-chamber view, and the three-chamber view of the LVO video sequence using a segmentation network; A sample extraction unit is used to divide the left ventricular wall into several segments, take the center of each segment as a node, extract the two-dimensional coordinates as node features, and obtain a sample set; The second construction unit is used to construct three identical spatiotemporal graph convolutional networks; the spatiotemporal graph convolutional network is composed of multi-faceted spatial convolution and multi-scale temporal convolution alternately, and finally classified by a classifier; the multi-faceted spatial convolution includes several multi-head self-attention graph convolution modules; A training unit, used for inputting node features of each node in the sample set into the classifiers of three spatiotemporal graph convolutional networks respectively, detecting the segmental wall motion states related to the left anterior descending artery, the left circumflex artery and the right coronary artery respectively, and obtaining a trained spatiotemporal graph convolutional network; Optionally, the segmentation network includes a residual dilated convolution, a Transformer encoder, a Transformer decoder, a 2x2 maximum pooling layer, a 3x3 convolution layer, an activation layer, and a 1x1 convolution layer, which are connected in sequence.

[0031] Optionally, let Q be the query value, K be the key value, and d be the intermediate dimension. is the total number of expansion heads of the multi-head self-attention graph convolution module, i is the number of expansion heads of the multi-head self-attention graph convolution module, A is the myocardial topology of the multi-head self-attention graph convolution module, is a hyperparameter, and the attention matrix of the i-th expansion head is , is the query value of the number of extension headers of the ith is the key value of the i-th extension header, T is the number of frames in a video sequence, Represents the activation function. In the multi-head self-attention graph convolution module, Q and K are obtained by convolution and average pooling of T, then: .

[0032] Optionally, let the node feature input of each node in the sample set be , the Value of the i-th extended head obtained by convolution is , is the total number of extended heads of the multi-head self-attention graph convolution module, is the first layer of the multi-head self-attention graph convolution module The transposed matrix of the attention matrix of the extended head, the output of the multi-head self-attention graph convolution module is ,but: .

[0033] Optionally, the first-layer multi-head self-attention graph convolution module in the multi-plane spatial convolution uses the two-dimensional left ventricular structure topology of each plane, and the second-layer multi-head self-attention graph convolution module uses the three-dimensional global topology of the left ventricle; a correspondence between the three-dimensional global topological structure of the left ventricle and the 18 segments of the left ventricular wall under the short-axis plane is established.

[0034] Optionally, the multi-scale temporal convolution includes four channels and a ReLU activation layer; the first channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 1, the second channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 2, the third channel includes a 1x1 convolution and a 4x1 average pooling layer, and the fourth channel includes a 1x1 convolution; the output ends of the four channels are respectively connected to the ReLU activation layer.

[0035] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A segmental wall motion analysis method based on left ventricular opacification, characterized in that: include: Introduce dilated convolution into residual convolution and build a segmentation network based on U-Net and Transformer; The left ventricular contour was segmented using a segmentation network in the apical four-chamber, two-chamber, and three-chamber views of the LVO video sequence. According to the segmented left ventricular contour, the left ventricular wall is divided into several segments, the center of each segment is taken as a node, and the two-dimensional coordinates are extracted as node features to obtain a sample set; Construct three identical spatiotemporal graph convolutional networks; the spatiotemporal graph convolutional network is composed of multi-faceted spatial convolution and multi-scale temporal convolution alternately, and finally classified by the classifier; the multi-faceted spatial convolution includes several multi-head self-attention graph convolution modules; The node features of each node in the sample set are input into the classifiers of three spatiotemporal graph convolutional networks respectively to detect the segmental wall motion states related to the left anterior descending artery, the left circumflex artery and the right coronary artery, and the trained spatiotemporal graph convolutional network is obtained; The detected segmental wall motion status was analyzed.

2. The segmental wall motion analysis method based on left ventricular opacification according to claim 1, characterized in that: The segmentation network consists of a residual dilated convolution, a Transformer encoder, a Transformer decoder, a 2x2 max pooling layer, a 3x3 convolution layer, an activation layer, and a 1x1 convolution layer connected in sequence.

3. The segmental wall motion analysis method based on left ventricular opacification according to claim 1, characterized in that: Let Q be the query value, K be the key value, and d be the intermediate dimension. is the total number of expansion heads of the multi-head self-attention graph convolution module, i is the number of expansion heads of the multi-head self-attention graph convolution module, A is the myocardial topology of the multi-head self-attention graph convolution module, is a hyperparameter, and the attention matrix of the i-th expansion head is , is the query value of the number of extension headers of the ith is the key value of the i-th extension header, T is the number of frames in a video sequence, Represents the activation function, through convolution and average pooling of T, obtain Q and K, then: 。 4. The segmental wall motion analysis method based on left ventricular opacification according to claim 1, characterized in that: Assume that the node feature input of each node in the sample set is , the Value of the i-th extended head obtained by convolution is , is the total number of extended heads of the multi-head self-attention graph convolution module, is the first layer of the multi-head self-attention graph convolution module The transposed matrix of the attention matrix of the extended head, the output of the multi-head self-attention graph convolution module is ,but: 。 5. The segmental wall motion analysis method based on left ventricular opacification according to claim 1, characterized in that: The first-layer multi-head self-attention graph convolution module in the multi-section spatial convolution uses the two-dimensional left ventricular structure topology of each section, and the second-layer multi-head self-attention graph convolution module uses the three-dimensional global topology of the left ventricle; the correspondence between the three-dimensional global topology of the left ventricle and the 18 segments of the left ventricular wall under the short-axis section is established.

6. The segmental wall motion analysis method based on left ventricular opacification according to claim 1, characterized in that: The multi-scale temporal convolution contains four channels and a ReLU activation layer; the first channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 1, the second channel includes a 1x1 convolution and a 5x1 convolution with a dilation rate of 2, the third channel includes a 1x1 convolution and a 4x1 average pooling layer, and the fourth channel includes a 1x1 convolution; the output ends of the four channels are connected to the ReLU activation layer respectively.

7. Left ventricular segmental wall motion analysis system, characterized in that: It includes a first construction unit, a segmentation unit, a sample extraction unit, a second construction unit, a training unit and a detection unit; The first construction unit is used to introduce the hole convolution into the residual convolution and build a segmentation network based on U-Net and Transformer; A segmentation unit, for segmenting the left ventricular contour in the apical four-chamber view, the two-chamber view, and the three-chamber view of the LVO video sequence using a segmentation network; A sample extraction unit is used to divide the left ventricular wall into several segments, take the center of each segment as a node, extract the two-dimensional coordinates as node features, and obtain a sample set; The second construction unit is used to construct three identical spatiotemporal graph convolutional networks; the spatiotemporal graph convolutional network is composed of multi-faceted spatial convolution and multi-scale temporal convolution alternately, and finally classified by a classifier; the multi-faceted spatial convolution includes several multi-head self-attention graph convolution modules; A training unit, used for inputting node features of each node in the sample set into the classifiers of three spatiotemporal graph convolutional networks respectively, detecting the segmental wall motion states related to the left anterior descending artery, the left circumflex artery and the right coronary artery respectively, and obtaining a trained spatiotemporal graph convolutional network; The detection unit is used to analyze the segmental ventricular wall motion state obtained by detection.

Images