Blood vessel matching method and device under different visual angles
By using vascular matching methods at different perspectives in coronary angiography images, using deep learning and weighted similarity calculations, efficient and accurate automated vascular matching is achieved, and the problem of low matching difficulty and accuracy in the existing technology is solved. Reduced time and labor costs.
Patent Information
- Application Number
- CN202411002137.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art has difficulty in matching due to complex morphology and angle differences in the automated matching of coronary angiography images. Relying on manual matching leads to high time and labor costs, and the accuracy of existing deep learning methods is low and noise is high.
The matching method of blood vessels at different perspectives is adopted, and blood vessel images are obtained at different angles, and the blood vessel segment extraction and coordinate point set determination are performed. The pre-trained vascular matching model is used for deep learning matching, combining feature determination sub-models and vascular predictor sub-models , calculate distance similarity, alignment similarity and geometric similarity, perform weighted summing to determine the similarity of blood vessel segments, and achieve automated matching.
Efficient and accurate automated blood vessel matching is achieved, reducing time and labor costs, improving matching accuracy, and improving matching accuracy through weighted similarity.
Smart Images

Figure CN119027698A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of medical image processing, and in particular to a method and device for matching blood vessels at different viewing angles. Background Art
[0002] Coronary angiography images are important diagnostic tools in cardiology. They play an indispensable role in the diagnosis, treatment planning and follow-up of cardiovascular diseases, and provide detailed and intuitive information about vascular structure and blood flow. In order to have a more comprehensive and accurate understanding of coronary vascular information, vascular matching processing of coronary angiography images is required. However, due to the complex morphology and irregularity of the coronary arteries, as well as the significant differences in images at different angles, automated matching becomes an extremely challenging task. In actual operation, coronary angiography images are often entangled and occluded with each other, which further increases the difficulty of automated processing.
[0003] At present, most products still rely on manual matching of key points to obtain vascular matching results. This method requires professionally trained physicians with rich imaging experience to operate, which is not only time-consuming and labor-intensive, but also has high requirements for physicians' operational proficiency and experience, thereby increasing the time and labor costs of diagnosis and treatment. However, with the continuous growth of medical imaging data and the increase in clinical needs, the efficiency and operability of manual matching have become insufficient. Therefore, the development of more efficient and accurate automated vascular matching technology has become an important issue that needs to be solved urgently. This will not only help improve the efficiency and accuracy of three-dimensional reconstruction of coronary angiography images, but also reduce the workload of physicians. Summary of the invention
[0004] In view of this, the purpose of the present application is to provide a method and device for matching blood vessels under different viewing angles, which can realize efficient and accurate automated blood vessel matching, thereby reducing time and labor costs.
[0005] The present application embodiment provides a method for matching blood vessels at different viewing angles, and the matching method includes:
[0006] Acquire a first blood vessel image and a second blood vessel image determined by photographing the same position of the target patient at different angles;
[0007] Performing blood vessel segment extraction processing on the first blood vessel image and the second blood vessel image respectively, determining a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched, and determining at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment;
[0008] Inputting the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model, and outputting a third coordinate point set of the predicted blood vessel segment;
[0009] For each second blood vessel segment, determining the overall similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0010] The second blood vessel segment corresponding to the maximum overall similarity is determined as the target blood vessel segment of the blood vessel segment to be matched, and matching processing is performed.
[0011] Optionally, the vascular matching model includes a feature determination sub-model and a vascular prediction sub-model; the step of inputting the first vascular image, the second vascular image, and the first coordinate point set of the vascular segment to be matched into a pre-trained vascular matching model, and outputting a third coordinate point set of the predicted vascular segment, comprises:
[0012] Inputting the first blood vessel image and the second blood vessel image into the feature determination sub-model to determine a fusion feature; wherein the fusion feature is obtained by fusing the first feature of the first blood vessel image and the second feature of the second blood vessel image;
[0013] The fusion feature and the first coordinate point set of the to-be-matched blood vessel segment are input into the blood vessel prediction sub-model for blood vessel prediction processing, and a third coordinate point set of the predicted blood vessel segment is output.
[0014] Optionally, determining the overall similarity between the second vascular segment and the predicted vascular segment based on the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment includes:
[0015] Determining distance similarity, alignment similarity, and geometric similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0016] A weighted summation process is performed according to the distance similarity, the alignment similarity and the geometric similarity to determine the overall similarity.
[0017] Optionally, the distance similarity is determined by the following steps:
[0018] For each third coordinate point in the third coordinate point set, searching for a second coordinate point closest to the third coordinate point in the second coordinate point set, and calculating the distance between the two points to obtain a first distance;
[0019] The sum of all the first distances is determined as the distance similarity.
[0020] Optionally, the alignment similarity is determined by the following steps:
[0021] constructing a distance matrix according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0022] According to the distance matrix, a cumulative distance matrix is determined using a dynamic programming method;
[0023] According to the cumulative distance matrix, a minimum alignment loss is calculated, and the minimum alignment loss is determined as the alignment similarity.
[0024] Optionally, the geometric similarity is determined by the following steps:
[0025] Determining the length similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set and the third coordinate point set;
[0026] Determining the directional similarity between the second blood vessel segment and the predicted blood vessel segment according to the first and last coordinate points of the second coordinate point set and the first and last coordinate points of the third coordinate point set;
[0027] A weighted summation process is performed according to the length similarity and the direction similarity to determine the geometric similarity.
[0028] Optionally, the feature determination submodel is determined by training a convolutional neural network, and the blood vessel prediction submodel is determined by training an attention mechanism neural network.
[0029] The embodiment of the present application further provides a matching device for blood vessels at different viewing angles, the matching device comprising:
[0030] An acquisition module, used for acquiring a first blood vessel image and a second blood vessel image determined by photographing the same position of a target patient at different angles;
[0031] an extraction module, configured to perform a blood vessel segment extraction process on the first blood vessel image and the second blood vessel image respectively, determine a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched, and determine at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment;
[0032] A prediction module, used for inputting the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model, and outputting a third coordinate point set of the predicted blood vessel segment;
[0033] A first determination module is used to determine, for each second blood vessel segment, an overall similarity between the second blood vessel segment and the predicted blood vessel segment according to a second coordinate point set of the second blood vessel segment and a third coordinate point set of the predicted blood vessel segment;
[0034] The second determination module is used to determine the second blood vessel segment corresponding to the maximum overall similarity as the target blood vessel segment of the blood vessel segment to be matched, and perform matching processing.
[0035] Optionally, the blood vessel matching model includes a feature determination sub-model and a blood vessel prediction sub-model; when the prediction module is used to input the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model and output a third coordinate point set of the predicted blood vessel segment, the prediction module is used to:
[0036] Inputting the first blood vessel image and the second blood vessel image into the feature determination sub-model to determine a fusion feature; wherein the fusion feature is obtained by fusing the first feature of the first blood vessel image and the second feature of the second blood vessel image;
[0037] The fusion feature and the first coordinate point set of the to-be-matched blood vessel segment are input into the blood vessel prediction sub-model for blood vessel prediction processing, and a third coordinate point set of the predicted blood vessel segment is output.
[0038] Optionally, when the first determining module is used to determine the overall similarity between the second vascular segment and the predicted vascular segment according to the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment, the first determining module is used to:
[0039] Determining distance similarity, alignment similarity, and geometric similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0040] A weighted summation process is performed according to the distance similarity, the alignment similarity and the geometric similarity to determine the overall similarity.
[0041] Optionally, the first determining module is further configured to determine the distance similarity by the following steps:
[0042] For each third coordinate point in the third coordinate point set, searching for a second coordinate point closest to the third coordinate point in the second coordinate point set, and calculating the distance between the two points to obtain a first distance;
[0043] The sum of all the first distances is determined as the distance similarity.
[0044] Optionally, the first determining module is further configured to determine the alignment similarity by the following steps:
[0045] constructing a distance matrix according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0046] According to the distance matrix, a cumulative distance matrix is determined using a dynamic programming method;
[0047] According to the cumulative distance matrix, a minimum alignment loss is calculated, and the minimum alignment loss is determined as the alignment similarity.
[0048] Optionally, the first determining module is further used to determine the geometric similarity through the following steps:
[0049] Determining the length similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set and the third coordinate point set;
[0050] Determining the directional similarity between the second blood vessel segment and the predicted blood vessel segment according to the first and last coordinate points of the second coordinate point set and the first and last coordinate points of the third coordinate point set;
[0051] A weighted summation process is performed according to the length similarity and the direction similarity to determine the geometric similarity.
[0052] Optionally, the feature determination submodel is determined by training a convolutional neural network, and the blood vessel prediction submodel is determined by training an attention mechanism neural network.
[0053] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the matching method as described above are performed.
[0054] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the matching method described above are executed.
[0055] The embodiment of the present application provides a method and device for matching blood vessels under different viewing angles. The matching method includes: obtaining a first blood vessel image and a second blood vessel image determined by photographing the same position of a target patient at different angles; performing blood vessel segment extraction processing on the first blood vessel image and the second blood vessel image respectively, determining a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched, and determining at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment; inputting the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model, and outputting a third coordinate point set of a predicted blood vessel segment; for each second blood vessel segment, determining the overall similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment; determining the second blood vessel segment corresponding to the maximum overall similarity as the target blood vessel segment of the blood vessel segment to be matched, and performing matching processing.
[0056] In this way, the fully automated blood vessel matching solution provided by this application can achieve efficient and accurate automated blood vessel matching from different perspectives, thereby reducing time and labor costs; and this solution adopts a "blood vessel segment to blood vessel segment" deep learning matching method, which greatly improves the matching accuracy. And three weighted similarities at different angles are designed to further improve the matching accuracy.
[0057] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0059] Figure 1 The schematic diagram of the existing technology for matching blood vessels at different viewing angles;
[0060] Figure 2 A schematic diagram of matching blood vessels at different viewing angles provided for this application;
[0061] Figure 3 A flow chart of a method for matching blood vessels at different viewing angles provided in an embodiment of the present application;
[0062] Figure 4A schematic diagram of a blood vessel segment and a process of determining a coordinate point set thereof provided in the present application;
[0063] Figure 5 A schematic diagram of a process of vascular prediction based on a vascular matching model provided in this application;
[0064] Figure 6 A schematic diagram of the structure of a blood vessel matching device at different viewing angles provided in an embodiment of the present application;
[0065] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0066] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.
[0067] Coronary angiography images are important diagnostic tools in cardiology. They play an indispensable role in the diagnosis, treatment planning and follow-up of cardiovascular diseases, and provide detailed and intuitive information about vascular structure and blood flow. In order to have a more comprehensive and accurate understanding of coronary vascular information, vascular matching processing of coronary angiography images is required. However, due to the complex morphology and irregularity of the coronary arteries, as well as the significant differences in images at different angles, automated matching becomes an extremely challenging task. In actual operation, coronary angiography images are often entangled and occluded with each other, which further increases the difficulty of automated processing.
[0068] At present, most products still rely on manual matching of key points to obtain vascular matching results. This method requires professionally trained physicians with rich imaging experience to operate, which is not only time-consuming and labor-intensive, but also requires high proficiency and experience of physicians, thereby increasing the time and labor costs of diagnosis and treatment. However, with the continuous growth of medical imaging data and the increase in clinical needs, the efficiency and operability of manual matching have become insufficient.
[0069] Although conventional learning or deep learning methods have been proposed in the prior art for vascular matching, since different coronary angiography images are non-rigid deformations and the length of blood vessels is inconsistent with the change of contrast agents, the matching methods based on conventional learning (such as registration) have low accuracy and are not suitable for clinical use. The currently proposed deep learning-based methods mainly predict matching key points and match them based on key points, but the accuracy of key point matching is also low and the noise is large.
[0070] For examples, see Figure 1 , Figure 1 The schematic diagram of the prior art for matching blood vessels at different viewing angles is shown in FIG. Figure 1 As shown, the prior art uses point-to-point prediction when searching for the same blood vessels in two viewing angles. Figure 1 As shown in the right side of the middle prediction point, it is not possible to accurately locate which blood vessel the prediction point belongs to. Therefore, the existing blood vessel matching technology has the problem of low accuracy.
[0071] Based on this, in order to achieve efficient and accurate automated blood vessel matching, this application provides a new matching solution, please refer to Figure 2 , Figure 2 A schematic diagram of matching blood vessels at different viewing angles provided in this application. Figure 2 As shown, the present application adopts an edge-to-edge prediction method (i.e., vessel-to-vessel prediction), such as Figure 2 As shown, according to the prediction of the blood vessels to be matched in the angle 1 image, Figure 1 The predicted blood vessel shown on the right can accurately locate the target blood vessel corresponding to the blood vessel to be matched from the angle 2 image based on the similarity of the curves.
[0072] In this way, through the technical solution of the present application, efficient and accurate automated blood vessel matching can be achieved, thereby also reducing time and labor costs.
[0073] In order to enable technical personnel in this field to accurately understand the implementation process of this solution, please refer to Figure 3 , Figure 3 This is a flow chart of a method for matching blood vessels at different viewing angles provided in an embodiment of the present application. Figure 3 As shown in , the matching method provided in the embodiment of the present application includes:
[0074] S301 , obtaining a first blood vessel image and a second blood vessel image determined by photographing the same position of a target patient at different angles.
[0075] S302, performing blood vessel segment extraction processing on the first blood vessel image and the second blood vessel image respectively, determining a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched, and determining at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment.
[0076] S303, inputting the first blood vessel image, the second blood vessel image, and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model, and outputting the predicted blood vessel segment of the blood vessel segment to be matched and the third coordinate point set of the predicted blood vessel segment;
[0077] S304, for each second blood vessel segment, determining the overall similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0078] S305 : Determine the second blood vessel segment corresponding to the maximum overall similarity as the target blood vessel segment of the blood vessel segment to be matched, and perform matching processing.
[0079] For step S301, for example, the first blood vessel image I r It can be a first coronary angiography image, and the second blood vessel image I t It may be a second coronary angiography image. Therefore, the corresponding same position of the target patient is the heart position.
[0080] For example, the first vascular image and the second vascular image may be images determined by photographing the same position of the target patient at different angles at the same time, or images determined by photographing the same position of the target patient at different angles within a certain period of time.
[0081] With respect to step S302, in this step, a blood vessel segment extraction process is performed on the first blood vessel image to determine a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched; a blood vessel segment extraction process is performed on the second blood vessel image to determine at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment.
[0082] Here, when the blood vessel segments to be matched and the first coordinate point sets of the blood vessel segments to be matched are provided, the method specifically includes: performing blood vessel segment extraction processing on the first blood vessel image to determine at least one first blood vessel segment in the first blood vessel image and the first coordinate point set of each first blood vessel segment; in response to a user's selection or according to a preset selection rule, selecting a blood vessel segment from all the first blood vessel segments as the blood vessel segment to be matched, and determining the first coordinate point set of the selected first blood vessel segment as the first coordinate point set of the blood vessel segment to be matched.
[0083] For further understanding of the process of determining the vessel segment and the coordinate point set, please refer to Figure 4 , Figure 4 A schematic diagram of a blood vessel segment and its coordinate point set determination process provided in this application. First, a multi-class segmentation network is trained to extract the blood vessels of interest in the coronary angiography image; Figure 4 As shown in the figure on the left, the prediction results of the multi-class segmentation network are shown in Figure 1. The main branch (green LAD) is one class, the main branches are one class (blue D1, D2...), other blood vessels are one class (red), and the background is one class. Next, the main branch and main branches of the above results are skeletonized to obtain the corresponding centerline results ( Figure 4 Next, calculate the number of neighbors of each point of the center line result. If the number of neighbors is equal to 1, it is an endpoint ( Figure 4 In the right figure), if the number of neighbors is greater than 2, it is a bifurcation point ( Figure 4 In the right figure), if the number of neighbors is equal to 2, it is the middle point. For the other endpoints except the two endpoints on the main branch, the minimum path loss is calculated with each bifurcation point, and the path with the minimum loss is selected as the branch corresponding to the endpoint. Finally, the point coordinate point set of each vascular segment of interest on the image is automatically obtained. For example, for the first vascular image I r Obtain the first coordinate point set P of each first blood vessel segment rf (≥1), P rf is the first coordinate point set of the fth first blood vessel segment. t Obtain the second coordinate point set P of each second blood vessel segment tg (≥1), P tg is the second punctuation set of the g-th second blood vessel segment.
[0084] In addition, the first coordinate point set of the blood vessel segment to be matched is P r express.
[0085] For step S303, for example, the third coordinate point set of the predicted blood vessel segment is P d It means that the goal of the vessel matching model is to learn a matching mapping: F:(P r |I r ,I t )→P d , that is, in I r ,I t Input P under the condition r In the vascular matching model, the prediction is t The corresponding blood vessel segment coordinate point set P d In short, the model inputs a certain vascular segment of a coronary angiography image at one angle and is able to predict the corresponding vascular segment of a coronary angiography image at another angle.
[0086] In this way, the morphological features of the predicted blood vessel segment can be determined according to the predicted third coordinate point set of the blood vessel segment to be matched.
[0087] Continuing with step S303, in one embodiment provided in the present application, the vascular matching model includes a feature determination sub-model and a vascular prediction sub-model; the first vascular image, the second vascular image, and the first coordinate point set of the vascular segment to be matched are input into the pre-trained vascular matching model, and the third coordinate point set of the predicted vascular segment is output, including:
[0088] S3031. Input the first blood vessel image and the second blood vessel image into the feature determination sub-model to determine a fusion feature.
[0089] S3032: Input the fusion feature and the first coordinate point set of the to-be-matched blood vessel segment into the blood vessel prediction sub-model to perform blood vessel prediction processing, and output a third coordinate point set of the predicted blood vessel segment.
[0090] Here, the feature determination sub-model is determined by training a convolutional neural network (CNN), and the blood vessel prediction sub-model is determined by training an attention mechanism neural network (Transformer).
[0091] With respect to step S1031 , the fused feature is obtained by fusing the first feature of the first vascular image and the second feature of the second vascular image.
[0092] For example, the process of determining the fusion feature may be: first, I r ,I r The input feature is encoded in the sub-model to obtain the corresponding feature map f r 、f r ; Then f r 、f r Concatenate and fuse the information of the two feature maps to obtain the fused feature F.
[0093] In order to improve processing efficiency, two CNNs may be set in the feature determination sub-model.
[0094] For step S1031, the fusion feature F and the first coordinate point set P of the blood vessel segment to be matched are combined. r Perform position encoding and input into the vascular prediction sub-model (Transformer network), and output the corresponding third coordinate point set P on the predicted vascular segment. d .
[0095] For examples, see Figure 5 , Figure 5A schematic diagram of a process for vascular prediction based on a vascular matching model provided in this application. Figure 5 As shown, the prediction process includes: first, I r ,I t Input into CNN for encoding and obtain the corresponding feature map f r 、f t Then, f r 、f t Splice and fuse the information of the two feature maps to obtain the fused feature F. Finally, F and P r Perform position encoding and input to the Transformer network, and output in P d The corresponding blood vessel segment coordinate point set P d . (P r and P d They are all two-dimensional coordinate point sets of blood vessel segments, and are arranged in order from the beginning to the end of the blood vessel).
[0096] Here, the training process of the blood vessel matching model includes the following steps:
[0097] Prepare data: The training data consists of paired angiography data from different angles. The vascular segments corresponding to the paired angiography images are labeled for network training.
[0098] Iterative optimization: The loss function may adopt common mean square error loss and cosine similarity loss, etc., and the network is optimized by continuous iteration until the loss approaches 0 and the model converges, thereby obtaining the blood vessel matching model.
[0099] For step S304, for each second vascular segment, the overall similarity between the second vascular segment and the predicted vascular segment is determined based on the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment, thereby determining the overall similarity between the predicted vascular segment and each second vascular segment.
[0100] Continuing with step S304, in an embodiment provided in the present application, the overall similarity between the second vascular segment and the predicted vascular segment is determined based on the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment;
[0101] S3041: Determine the distance similarity, alignment similarity, and geometric similarity between the second vascular segment and the predicted vascular segment according to the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment.
[0102] S3042: Perform weighted sum processing according to the distance similarity, the alignment similarity, and the geometric similarity to determine the overall similarity.
[0103] For step S3041, for example, in one implementation provided in the present application, the distance similarity is determined by the following steps:
[0104] S11. For each third coordinate point in the third coordinate point set, search the second coordinate point closest to the third coordinate point in the second coordinate point set, and calculate the distance between the two points to obtain a first distance.
[0105] S12: Determine the sum of all first distances as the distance similarity.
[0106] In order to understand the calculation process more intuitively, assume that P d =[[x0,y0],[x1,y1],…,[x n ,y n ]]=[p0,px,…,p n ](n means there are n coordinate points in the blood vessel segment to be matched), select the second blood vessel segment and take P T A blood vessel P t1 =[[a0,b0],[a1,b1],…,[a m ,b m ]]=[q0,q1,…,q m ](n represents the second blood vessel segment P t1 There are m coordinate points) for explanation.
[0107] For the contents of step S11-step S12, the calculation formula of the distance similarity is as follows:
[0108]
[0109] A is the distance similarity.
[0110] For example, in one implementation provided in the present application, the alignment similarity is determined by the following steps:
[0111] S21 . Construct a distance matrix according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment.
[0112] S22. Determine a cumulative distance matrix using a dynamic programming method according to the distance matrix.
[0113] S23. Calculate a minimum alignment loss according to the cumulative distance matrix, and determine the minimum alignment loss as the alignment similarity.
[0114] For step S21, establish a distance matrix M(i,j)=distance(p i ,q j), the distance matrix D is a matrix representing the pairwise distances between a set of objects, and the elements in the matrix represent the distances between the objects.
[0115] For example, the distance may be Euclidean distance, Manhattan distance, etc.
[0116] With respect to step S22, the dynamic programming method may specifically be a dynamic time warping method.
[0117] When the cumulative distance matrix C is determined using the dynamic programming method based on the distance matrix D, the specific steps include: first, create an initial cumulative distance matrix C of the same size as the distance matrix D; then, determine the boundary conditions, specifically, the first element of the initial cumulative distance matrix is the first element of the distance matrix. For other elements in the first row and the first column, the cumulative value is the cumulative distance of the path to the location; finally, fill the matrix and calculate the rest of the cumulative distance matrix through a recursive relationship. Each element C(i,j) is equal to the corresponding element D(i,j) of the distance matrix plus the minimum cumulative path value min(C(i-1,j),C(i,j-1),C(i-1,j-1)) to reach the location. In this way, the final cumulative distance matrix is determined.
[0118] For step S22, the step specifically includes: starting from the last element C(n,m) of the cumulative distance matrix, backtracking to find a path that represents the best alignment between the two sequences. The backtracking method is as follows: Assuming the current index is (i,j), select the point with the smallest cumulative cost from the three possible predecessor points: C(i-1,j), C(i,j-1) and C(i-1,j-1). When the index reaches (0,0), the backtracking process ends and the minimum alignment loss is obtained. The alignment similarity W is determined based on the minimum alignment loss.
[0119] For example, in one implementation provided in the present application, the geometric similarity is determined by the following steps:
[0120] S31. Determine, based on the second coordinate point set and the third coordinate point set, a length similarity between the second blood vessel segment and the predicted blood vessel segment.
[0121] S32: Determine the directional similarity between the second blood vessel segment and the predicted blood vessel segment according to the first and last coordinate points of the second coordinate point set and the first and last coordinate points of the third coordinate point set.
[0122] S33. Perform weighted sum processing according to the length similarity and the direction similarity to determine the geometric similarity.
[0123] For step S31, the calculation formula of the length similarity is:
[0124]
[0125] For step S32, the calculation formula of the geometric similarity is:
[0126]
[0127] For step S33, a weighted summation process is performed based on the length similarity and the direction similarity to determine the geometric similarity G. The weighted formula used may be:
[0128] G=αG1+βG2
[0129] α and β are weight coefficients, and their specific values can be determined adaptively according to the actual situation. For example, it is assumed that α is 1.
[0130] For step S3042, a weighted summation process is performed based on the distance similarity, the alignment similarity, and the geometric similarity to determine the overall similarity. The weighted formula used may be:
[0131] S=γA+δW+εG
[0132] γ, δ and ε are weight coefficients, and the specific values can be adaptively determined according to the actual situation. γ, δ and ε can be the same or different.
[0133] With respect to step S305, the second blood vessel segment corresponding to the maximum overall similarity is determined as the target blood vessel segment of the blood vessel segment to be matched, and matching processing is performed according to the image registration algorithm.
[0134] In this way, the fully automated blood vessel matching solution provided by this application can achieve efficient and accurate automated blood vessel matching from different perspectives, thereby reducing time and labor costs; and this solution adopts a "blood vessel segment to blood vessel segment" deep learning matching method, which greatly improves the matching accuracy. And three weighted similarities at different angles are designed to further improve the matching accuracy.
[0135] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a blood vessel matching device under different viewing angles provided in an embodiment of the present application. Figure 6 As shown, the matching device 600 includes:
[0136] An acquisition module 610 is used to acquire a first blood vessel image and a second blood vessel image determined by photographing the same position of a target patient at different angles;
[0137] An extraction module 620 is used to perform blood vessel segment extraction processing on the first blood vessel image and the second blood vessel image respectively, determine a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched, and determine at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment;
[0138] A prediction module 630, configured to input the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model, and output a third coordinate point set of the predicted blood vessel segment;
[0139] A first determination module 640 is configured to determine, for each second blood vessel segment, an overall similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0140] The second determination module 650 is used to determine the second blood vessel segment corresponding to the maximum overall similarity as the target blood vessel segment of the blood vessel segment to be matched, and perform matching processing.
[0141] Optionally, the blood vessel matching model includes a feature determination sub-model and a blood vessel prediction sub-model; when the prediction module 630 is used to input the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model and output a third coordinate point set of the predicted blood vessel segment, the prediction module 630 is used to:
[0142] Inputting the first blood vessel image and the second blood vessel image into the feature determination sub-model to determine a fusion feature; wherein the fusion feature is obtained by fusing the first feature of the first blood vessel image and the second feature of the second blood vessel image;
[0143] The fusion feature and the first coordinate point set of the to-be-matched blood vessel segment are input into the blood vessel prediction sub-model for blood vessel prediction processing, and a third coordinate point set of the predicted blood vessel segment is output.
[0144] Optionally, when the first determining module 640 is used to determine the overall similarity between the second vascular segment and the predicted vascular segment according to the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment, the first determining module 640 is used to:
[0145] Determining distance similarity, alignment similarity, and geometric similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0146] A weighted summation process is performed according to the distance similarity, the alignment similarity and the geometric similarity to determine the overall similarity.
[0147] Optionally, the first determining module 640 is further configured to determine the distance similarity by the following steps:
[0148] For each third coordinate point in the third coordinate point set, searching for a second coordinate point closest to the third coordinate point in the second coordinate point set, and calculating the distance between the two points to obtain a first distance;
[0149] The sum of all the first distances is determined as the distance similarity.
[0150] Optionally, the first determining module 640 is further configured to determine the alignment similarity by the following steps:
[0151] constructing a distance matrix according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment;
[0152] According to the distance matrix, a cumulative distance matrix is determined using a dynamic programming method;
[0153] According to the cumulative distance matrix, a minimum alignment loss is calculated, and the minimum alignment loss is determined as the alignment similarity.
[0154] Optionally, the first determining module 640 is further configured to determine the geometric similarity by the following steps:
[0155] Determining the length similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set and the third coordinate point set;
[0156] Determining the directional similarity between the second blood vessel segment and the predicted blood vessel segment according to the first and last coordinate points of the second coordinate point set and the first and last coordinate points of the third coordinate point set;
[0157] A weighted summation process is performed according to the length similarity and the direction similarity to determine the geometric similarity.
[0158] Optionally, the feature determination submodel is determined by training a convolutional neural network, and the blood vessel prediction submodel is determined by training an attention mechanism neural network.
[0159] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown in , the electronic device 700 includes a processor 710 , a memory 720 and a bus 730 .
[0160] The memory 720 stores machine-readable instructions executable by the processor 710. When the electronic device 700 is running, the processor 710 communicates with the memory 720 via the bus 730. When the machine-readable instructions are executed by the processor 710, the above-mentioned Figure 1 as well as Figure 4 , Figure 5 The specific implementation of the steps in the method embodiment shown can be found in the method embodiment, and will not be repeated here.
[0161] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 4 , Figure 5 The specific implementation of the steps in the method embodiment shown can be found in the method embodiment, and will not be repeated here.
[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0163] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0166] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0167] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A blood vessel matching method under different viewing angles. It is characterized in that The matching method comprises: Acquire a first blood vessel image and a second blood vessel image determined by photographing the same position of the target patient at different angles; Performing blood vessel segment extraction processing on the first blood vessel image and the second blood vessel image respectively, determining a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched, and determining at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment; Inputting the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model, and outputting a third coordinate point set of the predicted blood vessel segment; For each second blood vessel segment, determining the overall similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment; The second blood vessel segment corresponding to the maximum overall similarity is determined as the target blood vessel segment of the blood vessel segment to be matched, and matching processing is performed.
2. The matching method according to claim 1, It is characterized in that The blood vessel matching model includes a feature determination sub-model and a blood vessel prediction sub-model; the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched are input into the pre-trained blood vessel matching model, and the third coordinate point set of the predicted blood vessel segment is output, including: Inputting the first blood vessel image and the second blood vessel image into the feature determination sub-model to determine a fusion feature; wherein the fusion feature is obtained by fusing the first feature of the first blood vessel image and the second feature of the second blood vessel image; The fusion feature and the first coordinate point set of the to-be-matched blood vessel segment are input into the blood vessel prediction sub-model for blood vessel prediction processing, and a third coordinate point set of the predicted blood vessel segment is output.
3. The matching method according to claim 1, It is characterized in that Determining the overall similarity between the second vascular segment and the predicted vascular segment based on the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment includes: Determining distance similarity, alignment similarity, and geometric similarity between the second vascular segment and the predicted vascular segment according to the second coordinate point set of the second vascular segment and the third coordinate point set of the predicted vascular segment; A weighted summation process is performed according to the distance similarity, the alignment similarity and the geometric similarity to determine the overall similarity.
4. The matching method according to claim 3, It is characterized in that The distance similarity is determined by the following steps: For each third coordinate point in the third coordinate point set, searching for a second coordinate point closest to the third coordinate point in the second coordinate point set, and calculating the distance between the two points to obtain a first distance; The sum of all the first distances is determined as the distance similarity.
5. The matching method according to claim 3, It is characterized in that The alignment similarity is determined by the following steps: constructing a distance matrix according to the second coordinate point set of the second blood vessel segment and the third coordinate point set of the predicted blood vessel segment; According to the distance matrix, a cumulative distance matrix is determined using a dynamic programming method; According to the cumulative distance matrix, a minimum alignment loss is calculated, and the minimum alignment loss is determined as the alignment similarity.
6. The matching method according to claim 3, It is characterized in that The geometric similarity is determined by the following steps: Determining the length similarity between the second blood vessel segment and the predicted blood vessel segment according to the second coordinate point set and the third coordinate point set; Determining the directional similarity between the second blood vessel segment and the predicted blood vessel segment according to the first and last coordinate points of the second coordinate point set and the first and last coordinate points of the third coordinate point set; A weighted summation process is performed according to the length similarity and the direction similarity to determine the geometric similarity.
7. The matching method according to claim 2, It is characterized in that The feature determination sub-model is determined by training a convolutional neural network, and the blood vessel prediction sub-model is determined by training an attention mechanism neural network.
8. A matching device for blood vessels at different viewing angles. It is characterized in that The matching device comprises: An acquisition module, used for acquiring a first blood vessel image and a second blood vessel image determined by photographing the same position of a target patient at different angles; an extraction module, configured to perform a blood vessel segment extraction process on the first blood vessel image and the second blood vessel image respectively, determine a blood vessel segment to be matched in the first blood vessel image and a first coordinate point set of the blood vessel segment to be matched, and determine at least one second blood vessel segment in the second blood vessel image and a second coordinate point set of each second blood vessel segment; A prediction module, used for inputting the first blood vessel image, the second blood vessel image and the first coordinate point set of the blood vessel segment to be matched into a pre-trained blood vessel matching model, and outputting a third coordinate point set of the predicted blood vessel segment; A first determination module is used to determine, for each second blood vessel segment, an overall similarity between the second blood vessel segment and the predicted blood vessel segment according to a second coordinate point set of the second blood vessel segment and a third coordinate point set of the predicted blood vessel segment; The second determination module is used to determine the second blood vessel segment corresponding to the maximum overall similarity as the target blood vessel segment of the blood vessel segment to be matched, and perform matching processing.
9. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the matching method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the matching method according to any one of claims 1 to 7 are executed.
