Coronary branch recognition method and device of XRA image

Abstract

The method and device for identifying coronary branch of XRA image can quickly and accurately realize the branch identification of coronary XRA image. It comprises: (1) fast adaptive parameter pre-segmentation of coronary artery; (2) fast middle line marching optimization method for drawing coronary segment path; (3) construction of multi-modal coronary topological graph by traversing and eliminating method; (4) feature extraction of internal orientation vector field of coronary segment; (5) angle guidance and scale fusion of three-dimensional coronary features; (6) fusion of three types of coronary features by joint graph attention network structure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual image processing, and in particular to a coronary branch recognition method of XRA image and a coronary branch recognition device of XRA image. BACKGROUND

[0002] Coronary artery disease (CAD) caused by coronary artery stenosis or obstruction is one of the most common cardiovascular diseases. Different coronary branches with lesions will have different effects on the disease, so in clinical practice, doctors need to distinguish the lesions of different coronary branches to diagnose the disease. However, the structure of coronary branches is complex and difficult to automatically identify, which brings difficulties to CAD diagnosis and subsequent treatment.

[0003] In the prior art, the traditional method generally constructs a coronary vessel model, determines the main branches of the vessel tree through registration, and then distinguishes the categories of other branches. However, due to the complex topology of coronary arteries and large individual differences, it is difficult to construct a universal vessel model using this method. At the same time, the traditional algorithm has poor robustness and relies on manual experience, and the operation is relatively cumbersome. In order to improve this problem, the pixel-level semantic segmentation method judges the category of each pixel of the coronary image through data-driven method. However, due to the different imaging conditions, the grayscale of the coronary image is not uniform, the contrast is low, and there are artifacts, which leads to poor expression ability of the grayscale feature. At the same time, in terms of imaging data, the XRA (X Ray Angiograms) image has the problems of overlapping of coronary branches and lack of spatial information, which leads to misidentification of branches. These problems bring great challenges to the recognition of coronary branches. SUMMARY

[0004] In order to overcome the defects of the prior art, the technical problem to be solved by the present application is to provide a coronary branch recognition method of XRA image, which can quickly and accurately realize the branch recognition of coronary XRA image.

[0005] The technical scheme of the present application is as follows: the coronary branch recognition method of XRA image comprises the following steps:

[0006] (1) Coronary adaptive parameter fast pre-segmentation, using an adaptive framework nnUnet based on 2D U-Net, using depth separable convolution instead of traditional convolution, reducing the parameter amount and speeding up the inference speed without changing the segmentation accuracy. In the preprocessing aspect, in order to reduce the calculation amount, the framework uses cropping preprocessing for zero value area;

[0007] (2) Fast midline marching optimization method is used to draw the coronary segment path, and a semi-automatic method based on marching optimization is used to extract the coronary centerline;

[0008] (3) The multi-modal coronary topology graph is constructed by the iterative elimination method. The iterative elimination method is used as the core to complete automatic identification of coronary center line endpoints, identification of branch boundaries, and resampling of center line points to make the number consistent. The coronary topology graph is constructed. The continuous coronary center line points are cut off into identifiable branches to obtain the required nodes and edges in the topology graph.

[0009] (4) The internal orientation vector field feature of the coronary segment is extracted. The internal orientation vector field is defined to simulate the dependence of different coronary branches on other internal branches. The relative position and growth direction of each coronary branch are calculated to evaluate the relationship between different coronary branches. After the relative distance, direction and growth direction features of all coronary branches are calculated, the features are concatenated to construct the internal orientation vector field of the coronary, and the node features of the coronary topology graph are provided.

[0010] (5) The angle guide and scale fusion of the three-dimensional coronary features are performed. The XRA image is divided into four views to observe different branches. The 2D features are guided by the 3D coronary branch recognition result. The angle guide feature is extracted. Then, the 3D and 2D features are fused to obtain the fusion feature, which fully supplements the missing spatial information.

[0011] (6) The three types of coronary features are fused together by the joint graph attention network structure to improve the feature expression ability. The joint graph attention network consists of two layers of graph attention, which are called node layer and joint layer. The weight coefficients of the neighborhood nodes to the center node are automatically learned in a progressive manner. Finally, the recognition result is obtained by weighted aggregation. The node layer learns the weight of the neighborhood nodes in each group of coronary topology graph, and the joint graph node is calculated by aggregation. Each of the three joint graph nodes represents different types of coronary feature information. The joint layer extracts the relationship of the three joint graph nodes to measure the importance of the three branch information to the branch classification. Finally, the branch recognition result is obtained by weighted aggregation.

[0012] The application measures the importance of the 2D orientation vector field feature, the angle guide feature and the fusion feature to the branch recognition task in a progressive manner through a two-layer graph attention network, calculates their weight coefficients, and finally fuses the three types of feature information together through weighted aggregation. The expression ability of the 2D coronary feature is enhanced, the coronary branch recognition effect of the XRA image is improved, and the coronary branch recognition of the XRA image can be quickly and accurately realized.

[0013] An XRA image coronary branch recognition device is also provided, which comprises:

[0014] The coronary artery adaptive parameter fast pre-segmentation module is configured to use the nnUnet adaptive framework based on 2D U-Net. It adopts depthwise separable convolution instead of traditional convolution, which reduces the number of parameters and speeds up inference while maintaining the same segmentation accuracy. In terms of preprocessing, in order to reduce the amount of computation, the framework uses cropping preprocessing for zero-value regions.

[0015] The fast relay travel optimization method for drawing coronary artery segment paths is configured to use a semi-automatic method based on travel optimization to extract the coronary artery centerline.

[0016] The traversal elimination method is used to construct a multimodal coronary artery topology graph module. Its configuration is based on the traversal elimination method, which completes the automatic identification of the endpoints of the coronary artery centerline, the identification of branch boundaries, and the resampling of centerline points to make their numbers consistent, constructs the coronary artery topology graph, cuts the continuous coronary artery centerline points into identifiable branches, and obtains the required nodes and edges in the topology graph.

[0017] The coronary artery segment internal orientation vector field feature extraction module is configured to define the internal orientation vector field to simulate the dependence of different coronary artery branches on other internal branches. By calculating the relative position and growth direction of each coronary artery branch separately, the relationship between different coronary artery branches is evaluated. After calculating the relative distance, direction, and growth direction features of all coronary artery branches, these feature information are concatenated to complete the construction of the coronary artery internal orientation vector field, providing a basis for the extraction of features from the internal orientation vector field of the coronary artery segment.

[0018] The pulse topology graph provides node features;

[0019] The three-dimensional coronary artery feature angle guidance and proportional fusion module is configured to divide the XRA image into four perspectives to observe different branches. It uses the 3D coronary artery branch recognition results to guide 2D features, extracts angle guidance features, and then fuses the 3D and 2D features to obtain fused features, which fully supplements the missing spatial information.

[0020] The coronary artery feature fusion module is configured to fuse three types of coronary artery features together through a joint graph attention network structure, thereby improving feature representation capabilities. The joint graph attention network consists of two graph attention layers, called the node layer and the joint layer, which progressively and automatically learn the weight coefficients of neighboring nodes to the central node. Finally, the recognition result is obtained through weighted aggregation. The node layer learns the weights of neighboring nodes in each group of coronary artery topology graphs and aggregates and calculates joint graph nodes. Each of the three joint graph nodes represents different types of coronary artery feature information. The joint layer extracts the correlation between the three joint graph nodes, measures the importance of the three types of branch information to the branch category, and finally aggregates the branch recognition result through weighted aggregation. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for identifying coronary artery branches from XRA images according to the present invention. DETAILED DESCRIPTION

[0022] As shown in the figure, the coronary branch recognition method of such XRA image includes the following steps: Figure 1

[0023] (1) Coronary adaptive parameter fast pre-segmentation, using an adaptive framework nnUnet based on 2D U-Net, using depth separable convolution instead of traditional convolution, reducing the parameter amount and speeding up the inference speed without changing the segmentation accuracy; in the preprocessing aspect, in order to reduce the calculation amount, the framework uses cropping preprocessing for zero value area;

[0024] (2) Fast marching optimization method is used to draw the coronary segment path, and a semi-automatic method based on marching optimization is used to extract the coronary centerline;

[0025] (3) The traversal rejection method is used to construct the multi-modal coronary topological graph. Taking the traversal rejection method as the core, the automatic identification of the coronary centerline endpoint, the identification of the branch boundary and the resampling of the centerline point are completed to make the number consistent, the coronary topological graph is constructed, the continuous coronary centerline point is cut off to be identifiable branch, and the required nodes and edges in the topological graph are obtained;

[0026] (4) Coronary segment internal orientation vector field feature extraction, define internal orientation vector field to simulate the dependence of different coronary branches on other branches, by calculating the relative position and growth direction of each coronary branch, to evaluate the relationship between different coronary branches, after calculating the relative distance, direction and growth direction features of all coronary branches, cascade these feature information to complete the construction of coronary internal orientation vector field, provide node features for coronary topological graph;

[0027] (5) Angle guidance and scale fusion of three-dimensional coronary features, divide the XRA image into four views to observe different branches respectively, use 3D coronary branch recognition result to guide 2D feature,

[0028] extract angle guidance feature, then fuse 3D and 2D features to get fusion feature,

[0029] fully supplement the missing spatial information;

[0030] (6) Fuse three types of coronary features together through joint graph attention network structure to improve feature expression ability, the joint graph attention network consists of two layers of graph attention, which are called node layer and joint layer, which automatically learn the weight coefficient of the neighborhood node to the center node in a progressive manner, and finally get the recognition result through weighted aggregation, the node layer learns the weight of the neighborhood node in each group of coronary topological graph, and the joint graph node is calculated by aggregation,

[0031] ​The three joint graph nodes respectively represent different types of coronary feature information, the joint layer extracts the correlation of the three joint graph nodes, measures the importance of the three branch information to the branch category, and finally aggregates the branch recognition result by weighting.

[0032] The application measures the importance of 2D orientation vector field features, angle guide features and fusion features to branch recognition tasks in a progressive manner through a two-layer graph attention network, calculates their weight coefficients, and finally fuses the three types of feature information together through weighted aggregation, enhances the expression ability of 2D coronary features, improves the coronary branch recognition effect of XRA images, and thus can quickly and accurately realize the branch recognition of coronary XRA images.

[0033] Preferably, in step (1), in terms of network, the U-Net network structure includes an encoder, a jump connection, a decoder, the encoder is composed of a depth separable convolution layer, a leaky ReLU activation function, an instance normalization and a 2*2 maximum pooling layer, the U-Net extracts low-level and high-level semantic feature maps of the coronary image, and then fuses the semantic features of different scales; the convolution layer adopts depth separable convolution, and convolution calculation is divided into two steps, the first step is channel-by-channel convolution, the number of convolution kernels is the same as the channel number of the previous layer, the channels and the convolution kernels are one-to-one corresponding, and the second step is point-by-point convolution, the size of the convolution kernel is 1*1*M, M is the channel number of the previous layer, and the point-by-point convolution operation combines the feature maps of the first step in the depth direction to generate a new feature map.

[0034] In terms of network training, the training hyperparameters are adaptively adjusted according to the size of the coronary image, the framework nnU-Net monitors the memory occupation in real time, automatically sets the batch size and image patch size to prevent memory explosion.

[0035] Preferably, in step (1), the Batch size is adjusted prior to the Patch size; the size of the Patch ensures that at least one third of the sampled area is a coronary foreground; the sum of cross-entropy loss and Dice loss is used as the loss function; and the learning rate is adaptively adjusted according to the exponential average of the loss function.

[0036] Fast Marching and Minimum Path method can realize image linear feature analysis and can be applied to coronary structure topology graph extraction. The method uses the image feature difference between the coronary vessel and the background to construct an energy graph. In the energy graph, the energy value at the center of the coronary vessel is smaller, and the energy value gradually increases in the direction of the vessel edge and the background. In the semi-automatic method of this section, a starting point is manually set, and the minimum energy graph corresponding to the point is obtained by solving the Eikonal equation on the velocity graph using the fast marching method. Then, a termination point is set, and the starting point is searched in reverse to make the energy of the path minimum, so that the shortest path is obtained, which can be used as the center line of the coronary artery. Preferably, in step (2), the branch end points and bifurcation points of the coronary arteries of interest are manually determined as key points, and the improved marching optimization method is used to realize the shortest path search and drawing between multiple key points; the image feature difference between the coronary vessel and the background is used to construct an energy graph, in which the energy value at the center of the coronary vessel is smaller, and the energy value gradually increases in the direction of the vessel edge and the background; a starting point is manually set, and the minimum energy graph corresponding to the point is obtained by solving the Eikonal equation on the velocity graph using the fast marching method, and then a termination point is set, and the starting point is searched in reverse to make the energy of the path minimum, so that the shortest path is obtained, which is used as the center line of the coronary artery.

[0037] Preferably, in step (2), regarding the solution of the energy graph, first define the coronary image I, in which the pixel value at the center line position has a smaller value, assume that there are two points p1 and p2 on the graph, B p is the set of paths between p1 and p2, define the energy graph function E: is expressed as:

[0038] E(δ) = ∫I(δ(z))dz (2.1)

[0039] where δ is a path between two points, and z is the length parameter. The δ that minimizes the energy E(δ) is the coronary center line between the two points; when searching for the center line, the minimum energy graph Φ corresponding to p1 is solved: The value of any point on the image in the minimum energy graph is p1 to The energy value corresponding to the coronary center line:

[0040]

[0041] satisfies the Eikonal equation:

[0042]

[0043] Fast marching method is a numerical solution to Eikonal equation, which uses inverse difference method to solve stable solution of Eikonal equation, and uses first-order forward and backward difference operator instead of derivative, and the calculation of a single pixel point is:

[0044]

[0045] Where, h x and h y represent the pixel difference value in the x and y direction of the image.

[0046] Fast marching algorithm does not need additional prior knowledge and post-processing, only needs to give the starting point and the end point, but in the extraction of coronary artery topology, due to the large number of coronary artery branches, each branch needs to be manually marked with a starting point and an end point, which is too cumbersome. At the same time, the coronary artery itself has the characteristics of single tree structure and good connectivity. Therefore, this section uses relay point to optimize the fast marching algorithm, which only needs to calculate the energy map once, improves the calculation efficiency, reduces the number of manual marking, and avoids possible false shortcuts.

[0047] Since there is only one coronary tree in the coronary image, and the coronary has good connectivity, theoretically, the energy map corresponding to a starting point in the coronary can be calculated, and then an arbitrary end point can be selected, so that a shortest path can be searched backward. In order to avoid false shortcuts, relay points can be added to complete multiple backward searches, and the starting point will be searched eventually. Therefore, if the coronary image has not been calculated, the energy map corresponding to the first key point in the queue is calculated and superimposed with the coronary pre-segmentation result as the global minimum energy map Φ. The specific solving steps are shown in Table 1 pseudo code.

[0048] Table 1

[0049]

[0050]

[0051] After manually selecting the coronary endpoints and branch intersection points, these key points are stored in the queue. When solving the energy map, each time the pixel point that completes the energy calculation is taken as the initial point, and the minimum energy of the four neighboring pixel points is updated according to the energy of the point. This cycle is repeated until the minimum energy map of the first key point in the queue is obtained.

[0052] Regarding the search of the shortest path, after obtaining the minimum energy map of the first key point in the queue, the search is started from the end point along the direction with the smallest minimum energy value, and the next point in the queue is searched, which is taken as the end point, and the process is repeated. The search path is the shortest path between the first point and the last point in the queue, which is the center line of the coronary artery. The specific calculation method is shown in Table 2.

[0053] Table 2

[0054]

[0055]

[0056] In step (3), the coronary topology graph is composed of nodes, edges and node features. In the subsequent step, the importance of surrounding nodes to the center node is measured by aggregating node features by edge, and the classification of nodes is completed. The branches of the coronary artery are abstracted as nodes, and the connected branch nodes are connected by edges, and vice versa. For XRA images, according to clinical experience, this chapter shows four views of images, which are used to observe and identify specific coronary branches.

[0057] Since the manual selection of coronary centerline key points does not distinguish between end points and intersection points, and the subsequent truncation of the centerline requires end point positioning of the centerline point coordinates. Therefore, it is necessary to distinguish between end points with a degree of 1 and intersection points with a degree of 2 and 3. First, find the coronary end points in all manually extracted key points. The positioning principle of the coronary end point is that the neighborhood of the end point is roughly distributed in a specific direction. Specifically, the angle between each pair of neighborhood points and the center point is calculated by traversing the centerline points and their neighborhood centerline points, and when all the angles are less than a certain threshold, the center point is the coronary end point.

[0058] Since the fast marching algorithm is used to search for the coronary centerline, the centerline between the coronary end points is continuous, and the intersection point D acting as a relay point is also in the centerline. Therefore, after obtaining the coronary end points, first obtain the intersection points with a degree of 3, pair the end points two by two, and find the centerline points between the end points and the manually extracted intersection points. Then compare any two centerlines AB and AC with each other, if some centerline points overlap or are very close, it can be inferred that the two centerline segments in the overlapping part represent the same coronary segment, and one group can be removed. Then record the point D coordinates at the separation of the two centerline segments, and match it with all intersection points, if successful, directly store the connection relationship of the three centerline segments into the topology graph, if not successful, treat the point as an intersection point and store the connection relationship into the topology graph. Finally, consider the remaining intersection points with a degree of 2, store the coronary segments and connection relationship on both sides of these intersection points into the topology graph.

[0059] In order to extract node features later, resample the centerline of each coronary segment to make the number of points the same. The specific implementation steps of the traversal elimination method for extracting the coronary topology structure are shown in Table 3.

[0060] Table 3

[0061]

[0062] Preferably, the step (4) is represented by a vector where M represents the number of truncated coronary branches, N represents the number of centerline points of the resampled coronary branches, and d represents the coordinate dimension, which is 3 in the CTA image and 2 in the XRA image.

[0063] By calculating the relative position and growth direction of each coronary branch, the relationship between different coronary branches is evaluated.

[0064]

[0065]

[0066]

[0067] wherein, represents the relative distance feature in the internal orientation vector field, represents the relative direction feature, represents the growth direction feature, for each coronary branch m, is taken as the extracted centerline point of m, is taken as the centerline point of other coronary branches except m, where i represents the i-th point in the coronary branch centerline, j represents the coordinate dimension, and start, mid, and end represent the start point, middle point, and end point of the centerline, respectively, and | represents the splicing operation.

[0068] After calculating the relative distance, direction, and growth direction features of all coronary branches, these feature information is concatenated to complete the construction of the internal orientation vector field of the coronary, providing node features for the coronary topology graph

[0069]

[0070] wherein, represents the internal orientation vector field of the coronary, with a dimension of the sum of the three feature dimensions.

[0071] Preferably, in the step (5), 20 cases of complete coronary branches in the 3D coronary data set are selected, and the average number of directions and lengths of these 3D coronary branches is calculated to construct a 3D coronary branch average model, which maximizes the representation of the general characteristics of 3D coronary and reduces the influence of individual differences on the average model; the horizontal angle difference α and the height angle difference β of the 3D average model main branch and each 2D coronary main branch are calculated to form an angle guide feature, and the coarse guidance is completed.

[0072] Preferably, in the step (5), each view will calculate the feature respectively, and the 3D main branch vector used by each view is also different. The left coronary left anterior oblique uses the LAD branch in the 3D coronary artery, the left coronary right anterior oblique uses the LCX branch, the right coronary left anterior oblique uses the RCA branch, the right coronary right anterior oblique uses the R-PDB branch, and the main branch vector of the 2D coronary artery is determined by the first key point and the last key point extracted manually. The two main branch vectors are normalized to unit vectors at the same time, assuming that the starting points are at the origin of the same coordinate system, then the first two items of the 3D vector are taken to calculate the horizontal angle difference with the 2D vector, and the third item of the 3D vector is taken to calculate the height angle difference with the first two items, both of which are expressed in radians. The horizontal angle difference and the height angle difference are spliced to obtain the angle guide feature

[0073]

[0074]

[0075] wherein, represents a unit vector composed of the first two items of the 3D main branch vector, represents a 2D main branch unit vector, represents a unit vector of the last item of the 3D main branch vector, ‖·‖ represents the modulus calculation, cos -1 represents the inverse function of cos(x), sin -1 represents the inverse function of sin(x);

[0076] The results of the 3D coronary branch are encoded, and there are 9 types of branches in total, so the encoding range is [1, 9];

[0077] Then the 2D main branch vector is aligned with the 3D main branch vector by 0, and the angle difference θ between the two vectors is calculated. The length and radius information of the 3D main branch are projected into each 2D branch vector through the angle difference θ

[0078]

[0079] a=Rbcosθ+c (4.4)

[0080] wherein, represents a unit vector of the 3D main branch vector, θ is the angle between the 2D vector and the 3D vector, a is the fusion feature, b is the length and radius information of the main branch in the 3D orientation vector field, c is the length and radius information of each branch in the 2D orientation vector field, R is the identification result code of the 3D coronary main branch, the range is [1, 9], which represents each branch respectively, and different codes will be selected for each view.

[0081] Preferably, in the step (6), assuming that N represents all coronary branches, for each coronary branch All belong to one of the three types of topology, assuming the type is U t , t∈1,2,3, the node layer will calculate The weight of the neighborhood node t of type U The importance of the neighborhood branch pair

[0082]

[0083] Where, attention is a layer of feedforward neural network, using LeakyReLU activation function, f ni represent the node features of , f tj represent the node features of the neighborhood node t of type U t , only when the neighborhood node of belongs to type U nt , attention will be calculated, otherwise attention = 0; Then get the weight coefficient

[0084]

[0085] Each type of topology graph branch is weighted and aggregated into a joint graph node U nt using the weight coefficient, using multi-head attention mechanism, calculating the average result after multiple weighted aggregation

[0086]

[0087] Where, S represents the number of times of independent calculation of multi-head attention, represent the weight coefficient in the s-th calculation, σ represents the LeakyReLU activation function, in order to reduce the dimension of the output matrix, the average operation is used to calculate the final output vector U nt , U nj is the joint graph node of type t; In order to obtain sufficient learning ability to calculate the weight between joint graph nodes, a learnable weight matrix W is used to assist in extracting the influence weight between joint graph nodes

[0088]

[0089] Where, represent the weight between the i-th joint graph node, N ni represent the j-th neighborhood joint graph node, union represents a layer of feedforward neural network, and the activation function is LeakyReLU; ​​​

[0090] The weight coefficient is obtained by normalizing the weight of each joint graph node by a softmax function, and the finally fused coefficient is represented as Finally, the average output vector η is obtained through the multi-head attention mechanism ni

[0091]

[0092]

[0093] wherein U ni represents a neighborhood joint graph node.

[0094] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be ROM / RAM, a magnetic disc, an optical disc, a memory card, etc. Therefore, corresponding to the method of the present application, the present application also simultaneously includes an XRA image coronary branch identification device. The device includes:

[0095] The coronary adaptive parameter fast pre-segmentation module is configured to use an adaptive framework nnUnet based on 2D U-Net, and adopts a depth separable convolution instead of a traditional convolution, thereby reducing the parameter amount and speeding up the inference speed under the condition that the segmentation precision is unchanged. In the preprocessing aspect, in order to reduce the calculation amount, the framework uses a clipping preprocessing for the zero value region;

[0096] The fast midline marching optimization method draws a coronary segment path module, which is configured to extract the coronary centerline by using a semi-automatic method based on marching optimization;

[0097] The traversal rejection method constructs a multi-modal coronary topological graph module, which is configured to complete the automatic identification of the coronary centerline endpoints, the identification of the branch boundary, and the resampling of the centerline points to make the number consistent, construct the coronary topological graph, cut the continuous coronary centerline points into identifiable branches, and obtain the required nodes and edges in the topological graph;

[0098] The coronary segment internal orientation vector field feature extraction module is configured to define the internal orientation vector field to simulate the dependency of different coronary branches on other internal branches, evaluate the relationship between different coronary branches by calculating the relative position and growth direction of each coronary branch, and after calculating the relative distance, direction and growth direction features of all coronary branches, cascade these feature information to complete the construction of the coronary internal orientation vector field, and provide node features for the coronary topological graph;

[0099] The angle guidance and scale fusion module of three-dimensional coronary features is configured to divide the XRA image into four views to observe different branches respectively, guide the 2D features by using the 3D coronary branch recognition result, extract the angle guidance features, and then fuse the 3D and 2D features to obtain the fusion features, so as to fully supplement the missing spatial information.

[0100] The coronary feature fusion module is configured to fuse the three types of coronary features together by a joint graph attention network structure, improve the feature expression capability, and the joint graph attention network is composed of two layers of graph attentions, which are respectively called node layer and joint layer, and the weight coefficients of the neighborhood nodes to the center node are automatically learned in a progressive manner, finally the recognition result is obtained by weighted aggregation, the node layer learns the weight of the neighborhood nodes in each group of coronary topological graphs respectively, and the joint graph nodes are calculated by aggregation, the three joint graph nodes respectively represent different types of coronary feature information, the joint layer extracts the correlation of the three joint graph nodes, measures the importance of the three branch information to the branch category, and finally the branch recognition result is obtained by weighted aggregation.

[0101] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiment are still within the protection scope of the technical solution of the present application.

Claims

1. A method for coronary branch identification of an XRA image, characterized in that: It comprises the following steps: (1) The coronary adaptive parameter fast pre-segmentation uses an adaptive framework nnUnet based on 2D U-Net, adopts a depth separable convolution instead of a traditional convolution, reduces the parameter quantity and speeds up the inference speed without changing the segmentation accuracy; in the preprocessing aspect, in order to reduce the calculation amount, the framework uses a clipping preprocessing for the zero value area; (2) The fast marching optimization method is used to draw a coronary segment path, and a semi-automatic method based on marching optimization is used to extract a coronary centerline; (3) The traversal rejection method is used to construct a multi-modal coronary topological graph, the traversal rejection method is used as the core to complete automatic identification of coronary centerline endpoints, identification of branch boundaries and resampling of centerline points to make the number consistent, construct a coronary topological graph, cut the continuous coronary centerline points into identifiable branches, and obtain nodes and edges required in the topological graph; (4) The coronary segment internal orientation vector field feature extraction defines an internal orientation vector field to simulate the dependency of different coronary branches on other internal branches, evaluates the relationship between different coronary branches by calculating the relative position and growth direction of each coronary branch, and after calculating the relative distance, direction and growth direction features of all coronary branches, cascades the feature information to complete construction of the coronary internal orientation vector field and provide node features for the coronary topological graph; (5) The angle guidance and scale fusion of three-dimensional coronary features divide the XRA image into four views to observe different branches, use the 3D coronary branch identification result to guide the 2D feature, extract the angle guidance feature, and then fuse the 3D and 2D features to obtain the fusion feature and fully supplement the missing spatial information; (6) The three types of coronary features are fused together through a joint graph attention network structure to improve the feature expression capability, the joint graph attention network is composed of two layers of graph attention, which are respectively called node layer and joint layer, and the weight coefficients of the neighborhood nodes to the center node are automatically learned in a progressive manner, and finally the recognition result is obtained through weighted aggregation, the node layer learns the weight of the neighborhood nodes in each group of coronary topological graph, and the joint graph node is calculated through aggregation, the three joint graph nodes respectively represent different types of coronary feature information, the joint layer extracts the correlation of the three joint graph nodes, measures the importance of the three branch information to the branch category, and finally the branch recognition result is obtained through weighted aggregation; In step (2), the branch endpoints and bifurcation points of the part of the coronary of interest are manually determined as key points, and an improved marching optimization method is used to draw the shortest path search between multiple key points; the energy graph is constructed by using the image feature difference between the coronary blood vessels and the background, in the energy graph, the energy value at the center of the coronary blood vessels is small, and the energy value gradually increases in the direction of the vessel edge and the background; a starting point is manually set, the minimum energy graph corresponding to the point is obtained by using the fast marching method to solve the Eikonal equation on the velocity graph, then a termination point is set, the energy of the path is minimized by searching to the starting point in reverse, and the shortest path is obtained, which is used as the centerline of the coronary; Regarding the solution of the energy map, first define the coronary image I, in which the pixel value at the centerline position has a small value. Suppose there are two points p1 and p2 on the graph, B p is the set of paths between p1 and p2, and the energy map function E is defined as: is expressed as: wherein, δ is a path between two points, z is a length parameter, and δ with the minimum energy E(δ) is the coronary centerline between two points; in searching the centerline, the minimum energy map Φ corresponding to p1 is found: The value of any point x on the image in the minimum energy map is the energy value corresponding to the coronary centerline of x: Φ(x) satisfies the Eikonal equation: Fast marching method is a numerical solution to Eikonal equation, which uses the backward difference method to solve the stable solution of Eikonal equation, and uses the first-order forward and backward difference operator instead of the derivative. The calculation of a single pixel point is: where h x and h y represent the pixel difference size in the x, y direction of the image.

2. The method of claim 1, wherein: In step (1), In terms of network, the U-Net network structure includes an encoder, a skip connection, a decoder, the encoder is composed of a deep separable convolution layer, a leaky ReLU activation function, an instance normalization, and a 2*2 maximum pooling layer, the U-Net extracts low-level and high-level semantic feature maps of the coronary image, and then fuses semantic features of different scales; the convolution layer adopts deep separable convolution, and the convolution calculation is divided into two steps, the first step is channel-by-channel convolution, the number of convolution kernels is the same as the number of channels of the previous layer, and the channels and the convolution kernels are one-to-one corresponding, the second step is point-by-point convolution, the size of the convolution kernel is 1*1*M, M is the number of channels of the previous layer, and the point-by-point convolution operation combines the feature maps of the first step in the depth direction to generate a new feature map; In terms of network training, the training hyperparameters are adaptively adjusted according to the size of the coronary image, and the framework nnU-Net monitors the memory occupation in real time, automatically sets the batch size and image block size to prevent memory explosion.

3. The method of claim 2, wherein: In the step (1), the Batch size is adjusted prior to the Patch size; the Patch size is ensured to have at least one third of the sampled region being the coronary foreground; the sum of the cross-entropy loss and the Dice loss is used as the loss function; and the learning rate is adaptively adjusted according to the exponential average of the loss function.

4. The method of claim 3, wherein: The step (4) is expressed by a vector , wherein M represents the number of truncated coronary branch, N represents the number of center line points after resampling of the coronary branch, and d represents the coordinate dimension, which is 3 in the CTA image and 2 in the XRA image. By calculating the relative position and growth direction of each coronary branch, the relationship between different coronary branches is evaluated: wherein, represents a relative distance feature in the internal orientation vector field, represents a relative direction feature, represents a growth direction feature, for each coronary branch m, as its extracted centerline point, as the centerline point of other coronary branch except m, wherein i represents the i-th point in the coronary branch centerline, j represents the coordinate dimension, and start, mid, end represent the start point, middle point, and end point in the centerline, respectively, and | represents the splicing operation. After calculating the relative distance, direction and growth direction features of all coronary branches, these feature information is concatenated to construct the coronary internal orientation vector field, providing node features for the coronary topology graph wherein, represents the internal orientation vector field of the coronary, with three dimensions being the sum of the characteristic dimensions.

5. The method of claim 4, wherein: In the step (5), 20 cases of complete branch in the 3D coronary data set are selected, and the average number of direction and length of these 3D coronary branches is calculated to construct a 3D average model of coronary branches, which maximizes the representation of the general characteristics of 3D coronary and reduces the influence of individual differences on the average model; the horizontal angle difference α and the height angle difference β between the 3D average model main branch and each 2D coronary main branch are calculated to form an angle guide feature and complete the coarse guidance.

6. The method of claim 5, wherein: In the step (5), each view will calculate features respectively, and the 3D main branch vector used by each view is also different. The left coronary left anterior oblique uses the LAD branch in the 3D coronary, the left coronary right anterior oblique uses the LCX branch, the right coronary left anterior oblique uses the RCA branch, and the right coronary right anterior oblique uses the R-PDB branch. The main branch vector of the 2D coronary is determined by the first key point and the last key point extracted manually, and the two main branch vectors are normalized to unit vectors at the same time. It is assumed that the starting points are at the origin of the same coordinate system, then the horizontal angle difference is calculated by taking the first two items of the 3D vector and the 2D vector, and the height angle difference is calculated by taking the third item of the 3D vector and the first two items, both of which are expressed in radians. The horizontal angle difference and the height angle difference are concatenated to obtain the angle guide feature where, a unit vector representing the first two terms of the 3D principal branch vector, a unit vector representing the 2D principal branch unit vector, a unit vector representing the last term of the 3D principal branch vector, ||·|| represents the modulus calculation, cos -1 represents the inverse function of cos(x), sin -1 represents the inverse function of sin(x); The results of the 3D coronary branch are encoded, and there are 9 types of branches, so the encoding range is [1, 9]; Then the 2D main branch vector is supplemented with 0 and aligned with the 3D main branch vector, the angle difference θ between the two vectors is calculated; the length and radius information of the 3D main branch is projected into each 2D branch vector through the angle difference θ wherein, wherein, u represents the unit vector of 3D main branch vector, θ is the angle between 2D vector and 3D vector, a is the fusion feature, b is the length, radius information of the main branch in 3D orientation vector field, c is the length, radius information of each branch in 2D orientation vector field, R is the recognition result code of 3D coronary main branch, the range is [1, 9], which represents each branch respectively, and different codes will be selected for each view angle.

7. The method of claim 6, wherein: In step (6), assuming N represents all coronary artery branches, for each coronary artery branch... They all belong to one of the three types of topological graphs, let's assume this type is U. t For t∈1,2,3, the node layer will calculate In type U t Neighboring nodes weight Represents the neighborhood branch pairs of n Importance Here, attention is a one-layer feedforward neural network using the LeakyReLU activation function, f ni represent The node characteristics, f tj Representing U t Neighboring nodes of type The node characteristics, only when The neighboring nodes belong to U t Attention is only calculated when the type is specified; otherwise, attention = 0. Then, the weight coefficients are obtained through normalization. weighting and aggregating branches in each type of topology graph into joint graph nodes U by weight coefficients nt using multi-head attention mechanism, computing the average result after weighting and aggregating multiple times wherein S represents the number of times of multi-head attention independent calculation, represents the weight coefficient in the s-th calculation, and σ represents a LeakyReLU activation function. In order to reduce the dimension of the output matrix, an average operation is used to calculate the final output vector U nt , U nt is a joint graph node of type t; in order to obtain sufficient learning ability to calculate the weight between the joint graph nodes, a learnable weight matrix W is used to assist in extracting the influence weight between the joint graph nodes wherein, represents the weight between the ith joint graph node, N nj represents the jth neighborhood joint graph node, union represents a layer of feedforward neural network, and the activation function is LeakyReLU; the weight of each joint graph node is normalized by a softmax function to obtain a weight coefficient, and the final fused coefficient is represented as Finally, an average output vector η is obtained through a multi-head attention mechanism ni wherein U ni represent neighborhood joint graph nodes.

8. The apparatus of the method of identifying coronary branches of an XRA image according to claim 1, characterized in that: It comprises: A coronary adaptive parameter fast pre-segmentation module configured to use an adaptive framework nnUnet based on 2D U-Net, replace traditional convolution with depth separable convolution, reduce parameter quantity and speed up inference speed without changing segmentation accuracy; in the aspect of pretreatment, in order to reduce the amount of calculation, the framework uses cropping pretreatment for zero value area; A fast midline marching optimization method to draw coronary segment path module configured to extract coronary centerline by using a semi-automatic method based on marching optimization; A traversal rejection method to construct a multi-modal coronary topological graph module configured to complete automatic identification of coronary centerline endpoints, identification of branch boundaries and resampling of centerline points to make the number consistent, construct a coronary topological graph, cut the continuous coronary centerline points into identifiable branches, and obtain the required nodes and edges in the topological graph; a coronary segment internal orientation vector field feature extraction module configured to define an internal orientation vector field to simulate the dependence of different coronary branches on other branches, evaluate the relationship between different coronary branches by calculating the relative position and growth direction of each coronary branch, and construct the internal orientation vector field of the coronary after calculating the relative distance, direction and growth direction features of all coronary branches to provide node features for the coronary topological graph; An angle guidance and scale fusion module of three-dimensional coronary features configured to divide the XRA image into four views to observe different branches respectively, guide 2D features using 3D coronary branch identification results, extract angle guidance features, and then fuse 3D and 2D features to obtain fusion features and fully supplement the missing spatial information; A coronary feature fusion module configured to fuse three types of coronary features together through a joint graph attention network structure to improve feature expression capability, the joint graph attention network consists of two layers of graph attention, which are called node layer and joint layer, respectively, and progressively automatically learns the weight coefficients of the neighborhood nodes to the center node, and finally obtains the recognition result through weighted aggregation, the node layer learns the weight of the neighborhood nodes in each group of coronary topological graph, and the joint graph node is calculated by aggregation, three joint graph nodes respectively represent different types of coronary feature information, the joint layer extracts the relationship of the three joint graph nodes, measures the importance of three branch information to branch category, and finally aggregates the branch recognition result by weighting.

Images