Image matching method and device based on three-dimensional reconstruction, equipment and storage medium

Abstract

The invention relates to an image matching method and device based on three-dimensional reconstruction, equipment and a storage medium. The method comprises the steps of obtaining a radiography image sequence conforming to a preset body position angle, extracting an initial center line corresponding to a target blood vessel area in the radiography image sequence, obtaining angle information and space information of imaging of the radiography image sequence, conducting three-dimensional reconstruction on the initial center line according to the angle information and the space information to obtain a target center line, and obtaining the target center line according to the target center line. Acquiring a first blood vessel cross section area set along the direction of the target center line according to the target center line, acquiring an IVUS image corresponding to the angiography image sequence, acquiring a second blood vessel cross section area set of the target blood vessel region according to the IVUS image, and performing registration calculation on the first blood vessel cross section area set and the second blood vessel cross section area set to obtain a target blood vessel region; and obtaining a registration result of the contrast image sequence and the IVUS image. According to the invention, the position of the IVUS image can be matched with the exact position of the IVUS image on the blood vessel route map obtained by coronary artery angiography.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for image matching based on three-dimensional reconstruction. Background Art

[0002] Coronary angiography image sequences can reflect the two-dimensional anatomical morphology of coronary arteries, so as to determine the specific location and degree of stenosis of coronary heart disease, but it is difficult to clearly display the abnormal location in the vascular wall and cannot provide information such as the morphology and properties of plaques. Intravascular ultrasound (IVUS) and other intravascular images can clearly display the cross-section of the blood vessels and reflect the organizational structure and fine geometric anatomical morphology of the vascular wall, but intravascular images cannot well reflect the topological structure information of the blood vessels. Based on the functional complementarity of coronary angiography image sequences and intravascular images, the coronary angiography image sequences are registered and fused with intravascular images, which can not only observe the spatial position of the coronary vessels, but also obtain the quantitative data of the inner and outer membranes and plaques of the vascular segments of interest.

[0003] At present, the existing technology mainly uses the IVUS automatic retraction device to maintain the uniformity of IVUS retraction for the registration of angiography images and IVUS images, and then performs mapping and registration based on the retraction distance. This method requires the use of the IVUS automatic retraction device to maintain the uniformity of retraction for registration, and cannot achieve non-uniform manual pullback registration. In addition, the accuracy of distance mapping will be affected by the patient's heartbeat, respiratory movement and coronary angiography equipment.

[0004] Therefore, how to achieve matching of the position of the IVUS image with its exact position on the vascular roadmap obtained by angiography has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0005] In view of the above, the present application provides an image matching method, device, equipment and storage medium based on three-dimensional reconstruction, which aims to solve the above technical problems.

[0006] In a first aspect, the present application provides an image matching method based on three-dimensional reconstruction, the method comprising:

[0007] Acquire an angiography image sequence that meets a preset body position angle, and extract an initial center line corresponding to a target blood vessel region in the angiography image sequence;

[0008] Acquiring the imaging angle information and spatial information of the angiographic image sequence, and performing three-dimensional reconstruction on the initial center line to obtain the target center line according to the angle information and the spatial information;

[0009] According to the target centerline, acquiring a first blood vessel cross-sectional area set along the target centerline direction;

[0010] Acquire an IVUS image corresponding to the angiography image sequence, and acquire a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image;

[0011] A registration calculation is performed on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain a registration result of the angiography image sequence and the IVUS image.

[0012] In a second aspect, the present application provides an image matching device based on three-dimensional reconstruction, the device comprising:

[0013] Acquisition module: used to acquire angiography image sequence that meets the preset body position angle, and extract the initial center line corresponding to the target blood vessel area in the angiography image sequence;

[0014] Reconstruction module: used to obtain the angle information and spatial information of the imaging sequence of the angiography images, and to perform three-dimensional reconstruction on the initial center line to obtain the target center line according to the angle information and the spatial information;

[0015] A first calculation module: used for obtaining a first blood vessel cross-sectional area set along the target centerline according to the target centerline;

[0016] A second calculation module: used for acquiring an IVUS image corresponding to the angiography image sequence, and acquiring a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image;

[0017] Registration module: used for performing registration calculation on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain the registration result of the angiography image sequence and the IVUS image.

[0018] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0019] Memory, used to store computer programs;

[0020] The processor is used to implement the image matching method based on three-dimensional reconstruction described in any embodiment of the first aspect when executing the program stored in the memory.

[0021] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image matching method based on three-dimensional reconstruction as described in any embodiment of the first aspect is implemented.

[0022] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0023] The present application obtains angiography image sequences that meet preset body angles, extracts the initial centerline corresponding to the target blood vessel region in the angiography image sequence, obtains the angle information and spatial information of the imaging of the angiography image sequence, performs three-dimensional reconstruction of the initial centerline according to the angle information and spatial information to obtain the target centerline, obtains a first blood vessel cross-sectional area set along the target centerline direction, obtains an IVUS image corresponding to the angiography image sequence, obtains a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image, performs registration calculation on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set, and obtains the registration result of the angiography image sequence and the IVUS image. The influence of the patient's heartbeat, respiratory movement and the accuracy of the coronary angiography instrument on the registration accuracy can be reduced, and the position of the IVUS image can be matched with its exact position on the blood vessel roadmap obtained by coronary angiography. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 This is a flow chart of an embodiment of an image matching method based on three-dimensional reconstruction of the present application;

[0027] Figure 2 A schematic diagram of a first coordinate system and a second coordinate system in an embodiment of the present application;

[0028] Figure 3 This is a module diagram of a preferred embodiment of the image matching device based on three-dimensional reconstruction of the present application;

[0029] Figure 4 A schematic diagram of a preferred embodiment of the electronic device of the present application;

[0030] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0032] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0033] The present application provides an image matching method based on three-dimensional reconstruction. Figure 1 , which is a schematic diagram of a method flow of an embodiment of the image matching method based on three-dimensional reconstruction of the present application. The method can be executed by an electronic device, and the electronic device can be implemented by software and / or hardware. The image matching method based on three-dimensional reconstruction includes:

[0034] Step S10: acquiring an angiography image sequence that meets a preset body position angle, and extracting an initial center line corresponding to a target blood vessel region in the angiography image sequence;

[0035] Step S20: Acquire the angle information and spatial information of the imaging of the contrast image sequence, and perform three-dimensional reconstruction on the initial center line according to the angle information and the spatial information to obtain the target center line;

[0036] Step S30: acquiring a first blood vessel cross-sectional area set along the target centerline according to the target centerline;

[0037] Step S40: acquiring an IVUS image corresponding to the angiography image sequence, and acquiring a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image;

[0038] Step S50: performing registration calculation on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain a registration result of the angiography image sequence and the IVUS image.

[0039] In this embodiment, the angiography image sequence refers to the coronary angiography image sequence corresponding to the IVUS image. The angiography image sequence is to introduce contrast agent into the target blood vessel, eliminate the bone and soft tissue images on the angiography film through a computer, and only display the image of the blood vessel on the image film. The angiography image sequence can be imported from the medical image archiving and communication system (Picture Archiving and Communication Systems, PACS) system, or obtained from the screen of the coronary angiography instrument based on a video acquisition card, wherein the angiography image sequence is collected before the IVUS is withdrawn. The IVUS image is a cross-sectional image of the blood vessel developed by an electronic imaging system installed in the ultrasound catheter after a catheter equipped with a miniaturized ultrasonic transducer is sent into the blood vessel cavity. Therefore, the IVUS image can not only observe the morphology of the lumen, but also the structure or lesions of the vessel wall. The IVUS image is collected during the IVUS withdrawal process. Among them, the working process of IVUS is as follows:

[0040] 1. Move the tip of the guiding catheter to the coronary ostium;

[0041] 2. Move the guide wire along the guiding catheter into the coronary artery (blood vessel);

[0042] 3. The IVUS catheter is moved along the guidewire into the coronary artery (blood vessel);

[0043] 4. The ultrasonic transducer in the IVUS catheter retracts and rotates to collect IVUS images.

[0044] In actual clinical operation specifications, different coronary vessels have specific angiography position requirements, so it is necessary to select a suitable sequence from a large number of angiography image sequences for three-dimensional reconstruction. Specifically, the user (doctor) receives the instruction of the vascular type selected through the human-computer interaction interface, and obtains the angiography image sequence of the specified vascular type from a large number of angiography image sequences. The vascular types include the left anterior descending artery (LAD), the circumflex artery (LCX), and the right coronary artery (RCA). Then, the intermediate angiography image sequence that meets the body position angle is screened out from the angiography image sequence of the specified vascular type. Among them, the body position information that meets the three-dimensional reconstruction includes:

[0045] LAD first position RAO: 15°--RAO: 45°, CRA: 15°--CRA: 45°, second position RAO: 15°--LAO: 15°, CRA: 15°--CRA: 45°;

[0046] LCX first position RAO: 15°--LAO: 15°, CAU: 15°--CAU: 45°, second position RAO: 15°--LAO: 15°, CAU: 15°--CAU: 45°;

[0047] RCA first position LAO: 30°--LAO: 55°, CAU: 15°--CRA: 45°, second position LAO: 10°--LAO: 45°, CRA: 15°--CRA: 45°;

[0048] Among them, RAO stands for right anterior oblique position, CRA stands for head position, LAO stands for left anterior oblique position, and CAU stands for foot position.

[0049] After obtaining an angiography image sequence that meets the preset body position angle, an initial center line corresponding to the target blood vessel area in the angiography image sequence is extracted through a pre-trained artificial intelligence model. Specifically, the extracting the initial center line corresponding to the target blood vessel area in the angiography image sequence includes:

[0050] Detecting key frames of the angiographic image sequence using a pre-trained detection model;

[0051] Segmenting the target blood vessel region of the key frame using a pre-trained segmentation model;

[0052] performing binarization processing on the target blood vessel region to obtain a binary image of the target blood vessel;

[0053] The binary image of the target blood vessel is thinned to obtain an initial center line corresponding to the target blood vessel region.

[0054] The key frame can be an image with the largest blood vessel area in the angiography image sequence. The key frame of the angiography image sequence can be detected using a pre-trained detection model, and the target blood vessel area of ​​the key frame can be segmented using a pre-trained segmentation model. Among them, both the detection model and the segmentation model can be obtained by model training through a large amount of labeled sample data. The target blood vessel area is binarized to obtain a binary image of the target blood vessel. The non-blood vessel area in the binary image of the target blood vessel is set to 0, and the blood vessel area is set to 1. The binary image of the target blood vessel is then refined to obtain the initial center line corresponding to the target blood vessel area. For example, in the image refinement process, the binary image is smoothed using algorithms such as mean, median, and Gaussian filtering to obtain a smoothed image, and then the smoothed image is refined using a layer-by-layer peeling and refinement algorithm.

[0055] Specifically, the thinning process of the binary image of the target blood vessel to obtain the initial center line corresponding to the target blood vessel region includes:

[0056] Scanning the pixel points in the binary image one by one to determine whether there is a non-vascular area in the neighborhood of the pixel point;

[0057] If it exists, the layer number of the pixel point is set to 1; if it does not exist, the layer number of the pixel point is set to the minimum layer value of the neighborhood plus 1, until the layer numbers of all pixels in the binary image are obtained;

[0058] Then, the pixels in the binary image are scanned one by one to determine whether the current layer number of the pixel is the maximum layer number in the neighborhood;

[0059] If yes, the pixel point is retained; otherwise, the pixel point is removed to obtain the initial center line corresponding to the target blood vessel region.

[0060] Scan the pixels in the binary image point by point to determine whether there is a non-vascular area in the neighborhood of the scanned pixel point. If so, the layer number of the pixel point is set to 1; if not, the layer number of the pixel point is set to the minimum layer value of the neighborhood plus 1, until the layer number of all pixels in the binary image is obtained. The calculation formula for the grayscale value of the neighborhood of the pixel point is:

[0061]

[0062] After the number of layers of all pixel points is calculated, the pixel points in the binary image are scanned point by point to determine whether the current number of layers of the scanned pixel points is the maximum number of layers in the neighborhood. If so, it is retained, otherwise it is removed. After all pixel points are scanned, the initial center line of the target blood vessel area can be obtained.

[0063] It is also possible to determine whether a pixel point belongs to the contour line of the target blood vessel area by judging whether the number of layers is 1. After obtaining the contour of the target blood vessel area, the corresponding diameter data can be calculated point by point in combination with the initial center line. The diameter measurement method along the initial center line can be sampled by the vertical line extraction method and the disk rolling method. Taking the vertical line extraction method as an example, a vertical line is drawn along the obtained initial center line, and the distance between the intersection point of the vertical line and the two edge lines is the diameter of the blood vessel at that point. Assume that any point on the initial center line is P i (x i ,y i ), select an adjacent point P i+1 (x i+1 ,y i+1 ), let the straight line passing through the two points be y=kx+b, then its line passing through P i (x i ,y i ) is:

[0064]

[0065] Substitute all discrete points on the contour line into the vertical line equation. When the difference between the two sides of the equation is the smallest and non-adjacent, it is P. i (x i ,yi ) corresponding to the contour point C i1 (x i1 ,y i1 ) and C i2 (x i2 ,y i2 ), then P can be calculated by the following formula i (x i ,y i ) corresponds to the diameter:

[0066] d=((x i1 -x i2 ) 2 +(y i1 -y i2 ) 2 ) 1 / 2

[0067] The target vascular region can be appropriately cropped using the initial centerline, edge contour line, and diameter data of the target vascular region to facilitate subsequent matching of the IVUS pullback region.

[0068] After obtaining the initial center line corresponding to the target blood vessel region, the angle information and spatial information of the imaging of the contrast image sequence can be obtained through the Digital Imaging and Communications in Medicine (DICOM) information, and the initial center line is three-dimensionally reconstructed to obtain the target center line according to the obtained angle information and spatial information. Specifically, the three-dimensional reconstruction of the initial center line to obtain the target center line according to the angle information and the spatial information includes:

[0069] Obtaining first coordinate information of each pixel point of the initial center line in the first coordinate system and second coordinate information in the second coordinate system according to the angle information and the spatial information;

[0070] Obtaining a conversion formula between the first coordinate system and the second coordinate system;

[0071] According to the conversion formula, the first coordinate information and the second coordinate information, three-dimensional reconstruction is performed on each pixel point of the initial center line to obtain a target center line.

[0072] like Figure 2 As shown in FIG. 1 , it is a schematic diagram of the first coordinate system and the second coordinate system in the embodiment of the present application. Assume that the three-dimensional coordinates of the point P in the space in the coordinate system of the i-th projection are (x i ,y i , z i), so in the first coordinate system A, P1 (x1, y1, z1) is projected on the imaging plane as the first coordinate information P1 (u1, v1). In the second coordinate system B, P2 (x2, y2, z2) is projected on the imaging plane as the second coordinate information P2 (u2, v2). Among them, the conversion formula from coordinate system A to coordinate system B is:

[0073]

[0074]

[0075] Where R is the rotation matrix. Assuming that the origins of the two coordinate systems coincide, and the rotation transformation from coordinate system A to coordinate system B is S, then R = S -1 , t is the translation vector, which is also the coordinate of the origin of the second coordinate system B in the first coordinate system A. According to the conversion formula, the first coordinate information and the second coordinate information, each pixel point of the initial center line is three-dimensionally reconstructed to obtain the target center line.

[0076] Further, according to the conversion formula, the first coordinate information and the second coordinate information, three-dimensionally reconstructing each pixel point of the initial center line to obtain a target center line includes:

[0077] Acquire a first distance from the light source to the imaging plane when the pixel point is imaged in the first coordinate system and a second distance from the light source to the imaging plane when the pixel point is imaged in the second coordinate system;

[0078] According to the geometric relationship of perspective projection, a first relational expression between the first coordinate information and the first distance is obtained, and a second relational expression between the second coordinate information and the second distance is obtained;

[0079] Combining the conversion formula, the first relational expression and the second relational expression to obtain the three-dimensional coordinate information of the pixel point;

[0080] The target center line is obtained by performing three-dimensional reconstruction based on the three-dimensional coordinate information of each pixel point of the initial center line.

[0081] According to the geometric relationship of perspective projection, the following relationship can be obtained:

[0082]

[0083]

[0084]

[0085]

[0086] D1 and D2 refer to the distance from the light source to the imaging plane when imaging in coordinate system A and coordinate system B respectively. Combining the above formulas, we can get:

[0087]

[0088]

[0089] The above formula can be solved to get The coordinates of the point can be obtained from the relationship between coordinate system A and coordinate system B. Since this equation is an overdetermined system (AX=B), it can be solved by least squares solution:

[0090] X=(A T A) -1 ·A T ·B

[0091] Through the above calculation method, the initial center line of the target blood vessel region can be three-dimensionally reconstructed to obtain the target center line, where the target center line is the center line of the three-dimensional space, and X is a point on the target center line.

[0092] After obtaining the target centerline, a set of blood vessel cross-sectional areas along the target centerline direction is obtained according to the target centerline (recorded as the first blood vessel cross-sectional area set). If any point P on the target centerline is known, i (x i ,y i , z i ), and the projection P on different imaging planes i，1 (u i，1 , v i，1 , P i，2 (u i，2 , v i，2 ), and the diameter D corresponding to the projection point is known i，1 , D i，2 Assume that point P i If the blood vessel shape is an ellipse, the area of ​​the current node is calculated according to the formula of the ellipse area: S = D i，1 *D i,2 *π.

[0093] An IVUS image corresponding to the angiography image sequence is obtained, a blood vessel cross-sectional area set of a target blood vessel region is obtained according to the IVUS image (recorded as a second blood vessel cross-sectional area set), diameter information of the target blood vessel region in the IVUS image is calculated according to the centroid coordinates of the target blood vessel region in the IVUS image (the diameter is an average diameter), and a second blood vessel cross-sectional area set of the target blood vessel region is calculated and obtained according to the diameter information of the target blood vessel region. Specifically, obtaining the second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image includes:

[0094] Obtaining the coordinates of each discrete point in the target blood vessel region and a preset weight of each discrete point;

[0095] Calculating the centroid coordinates of the target blood vessel region according to the coordinates of the discrete points and the preset weights;

[0096] Generating straight lines along a preset angle range with the centroid coordinates as the center, and obtaining the coordinate information of the intersection set between each straight line and the target blood vessel region;

[0097] Calculating diameter information of the target blood vessel region based on the intersection point set coordinate information;

[0098] Based on the diameter information of the target blood vessel region, a second blood vessel cross-sectional area set of the target blood vessel region is acquired.

[0099] In the calculation of the diameter of the IVUS target vascular area, the centroid coordinates of the IVUS target vascular area are first calculated. The centroid calculation formula is as follows:

[0100]

[0101] Among them, X G , Y G is the centroid coordinate of the target vascular region, i is the discrete point number of the target vascular region, and w i is the weight of each discrete point, x i ,y i are the coordinates of each discrete point in the target blood vessel area.

[0102] Draw straight lines along 0-360 degrees with the centroid of the target blood vessel area as the center. Each straight line has two intersection points with the edge of the target blood vessel area. Assuming that the sampling interval of the angle is t, the coordinates of the intersection points are:

[0103] C i (x c ,y c ), D i (x d ,y d ), i=0, 1, 2,..., floor(360 / t)

[0104] The average diameter of the IVUS target vessel area is:

[0105]

[0106] Among them, floor means rounding down.

[0107] After obtaining a first blood vessel cross-sectional area set of a target blood vessel region in an angiography image sequence and a second blood vessel cross-sectional area set of a target blood vessel region in an IVUS image, a registration calculation is performed on the first blood vessel cross-sectional area and the second blood vessel cross-sectional area to obtain a registration result of the angiography image sequence and the IVUS image. Specifically, the registration calculation of the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain a registration result of the angiography image sequence and the IVUS image includes:

[0108] The first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set are registered and calculated, and the result with the minimum error is used as the registration result. For example, the first blood vessel cross-sectional area and the second blood vessel cross-sectional area are registered and calculated using the least square method.

[0109] In another embodiment, the present application can be used in conjunction with the branch information of the target for auxiliary registration. By calibrating the branch information, the registration result of the angiography image sequence and the IVUS image can be made more accurate. The registration calculation based on the branch information specifically includes:

[0110] Assume that the initial centerline length of the target blood vessel in the angiography image sequence is N, and the first blood vessel cross-sectional area along the centerline is A. x , branch position and diameter are SA i (x, y), x is the position on the center line, y is the diameter;

[0111] Assume that the number of IVUS frames is M and the cross-sectional area of ​​the second blood vessel along the retracement direction is B x , the branch position and diameter are SB i (x, y), x is the retracement distance in the retracement direction, y is the diameter;

[0112] It is assumed that the initial centerline of the target vessel in the angiography image sequence and the IVUS pullback sequence have completed distance calibration, that is, the physical distance between any two adjacent points on the initial centerline of the target vessel in the angiography image sequence is equal to the physical distance between two adjacent frames in the IVUS pullback sequence.

[0113] When the above assumptions are met, the following objective function is obtained, so that when the cross-sectional area of ​​the first blood vessel matches the cross-sectional area of ​​the second blood vessel, and the branch position and diameter on the first blood vessel match the branch position and diameter on the second blood vessel, the objective function has a minimum value, and the objective function includes:

[0114]

[0115] Among them, x1, y1 and x2, y2 represent the matching positions of the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set respectively, and i1, i2 represent the branch numbers of the first blood vessel and the second blood vessel respectively.

[0116] Reference Figure 3 , which is a schematic diagram of functional modules of the image matching device 100 based on three-dimensional reconstruction of the present application.

[0117] The image matching device 100 based on three-dimensional reconstruction described in the present application can be installed in an electronic device. According to the functions implemented, the image matching device 100 based on three-dimensional reconstruction can include an acquisition module 110, a reconstruction module 120, a first calculation module 130, a second calculation module 140 and a registration module 150. The module described in the present application can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0118] In this embodiment, the functions of each module / unit are as follows:

[0119] Acquisition module 110: used to acquire angiography image sequences that meet preset body angles, and extract an initial center line corresponding to a target blood vessel region in the angiography image sequence;

[0120] Reconstruction module 120: used to obtain the angle information and spatial information of the imaging sequence of the angiography images, and perform three-dimensional reconstruction on the initial center line to obtain the target center line according to the angle information and the spatial information;

[0121] A first calculation module 130 is used to obtain a first blood vessel cross-sectional area set along the target centerline according to the target centerline;

[0122] The second calculation module 140 is used to obtain an IVUS image corresponding to the angiography image sequence, and obtain a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image;

[0123] Registration module 150: used for performing registration calculation on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain the registration result of the angiography image sequence and the IVUS image.

[0124] In one embodiment, extracting the initial centerline corresponding to the target blood vessel region in the angiography image sequence includes:

[0125] Detecting key frames of the angiographic image sequence using a pre-trained detection model;

[0126] Segmenting the target blood vessel region of the key frame using a pre-trained segmentation model;

[0127] performing binarization processing on the target blood vessel region to obtain a binary image of the target blood vessel;

[0128] The binary image of the target blood vessel is thinned to obtain an initial center line corresponding to the target blood vessel region.

[0129] In one embodiment, the thinning process of the binary image of the target blood vessel to obtain the initial center line corresponding to the target blood vessel region includes:

[0130] Scanning the pixel points in the binary image one by one to determine whether there is a non-vascular area in the neighborhood of the pixel point;

[0131] If it exists, the layer number of the pixel point is set to 1; if it does not exist, the layer number of the pixel point is set to the minimum layer value of the neighborhood plus 1, until the layer numbers of all pixels in the binary image are obtained;

[0132] Then, the pixels in the binary image are scanned one by one to determine whether the current layer number of the pixel is the maximum layer number in the neighborhood;

[0133] If yes, the pixel point is retained; otherwise, the pixel point is removed to obtain the initial center line corresponding to the target blood vessel region.

[0134] In one embodiment, the three-dimensional reconstruction of the initial center line to obtain the target center line according to the angle information and the spatial information includes:

[0135] Obtaining first coordinate information of each pixel point of the initial center line in the first coordinate system and second coordinate information in the second coordinate system according to the angle information and the spatial information;

[0136] Obtaining a conversion formula between the first coordinate system and the second coordinate system;

[0137] According to the conversion formula, the first coordinate information and the second coordinate information, three-dimensional reconstruction is performed on each pixel point of the initial center line to obtain a target center line.

[0138] In one embodiment, performing three-dimensional reconstruction on each pixel point of the initial center line to obtain a target center line according to the conversion formula, the first coordinate information, and the second coordinate information includes:

[0139] Acquire a first distance from the light source to the imaging plane when the pixel point is imaged in the first coordinate system and a second distance from the light source to the imaging plane when the pixel point is imaged in the second coordinate system;

[0140] According to the geometric relationship of perspective projection, a first relational expression between the first coordinate information and the first distance is obtained, and a second relational expression between the second coordinate information and the second distance is obtained;

[0141] Combining the conversion formula, the first relational expression and the second relational expression to obtain the three-dimensional coordinate information of the pixel point;

[0142] The target center line is obtained by performing three-dimensional reconstruction based on the three-dimensional coordinate information of each pixel point of the initial center line.

[0143] In one embodiment, acquiring a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image includes:

[0144] Obtaining the coordinates of each discrete point in the target blood vessel region and a preset weight of each discrete point;

[0145] Calculating the centroid coordinates of the target blood vessel region according to the coordinates of the discrete points and the preset weights;

[0146] Generating straight lines along a preset angle range with the centroid coordinates as the center, and obtaining the coordinate information of the intersection set between each straight line and the target blood vessel region;

[0147] Calculating diameter information of the target blood vessel region based on the intersection point set coordinate information;

[0148] Based on the diameter information of the target blood vessel region, a second blood vessel cross-sectional area set of the target blood vessel region is acquired.

[0149] In one embodiment, the registering calculation of the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain the registration result of the angiography image sequence and the IVUS image includes:

[0150] Perform registration calculation on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set, and use the calculation result with the smallest error as the registration result.

[0151] Reference Figure 4 , which is a schematic diagram of a preferred embodiment of the electronic device 1 of the present application.

[0152] The electronic device 1 includes but is not limited to: a memory 11, a processor 12, a display 13 and a communication interface 14. The electronic device 1 can be connected to a network through the communication interface 14. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, a call network, etc.

[0153] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a hard disk or memory of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped with the electronic device 1, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 11 can also include both the internal storage unit of the electronic device 1 and its external storage device. In this embodiment, the memory 11 is generally used to store an operating system and various computer programs installed on the electronic device 1, such as the program code of the image matching program 10 based on three-dimensional reconstruction, etc. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.

[0154] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the program code of the image matching program 10 based on three-dimensional reconstruction.

[0155] The display 13 may be referred to as a display screen or a display unit. In some embodiments, the display 13 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an organic light-emitting diode (OLED) touch device, etc. The display 13 is used to display information processed in the electronic device 1 and to display a visual working interface.

[0156] The communication interface 14 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and the communication interface 14 is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0157] Figure 4 Only the electronic device 1 having the components 11 - 14 and the 3D reconstruction based image matching program 10 is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0158] In the above embodiment, the processor 12 can implement the following steps when executing the 3D reconstruction-based image matching program 10 stored in the memory 11:

[0159] Acquire an angiography image sequence that meets a preset body position angle, and extract an initial center line corresponding to a target blood vessel region in the angiography image sequence;

[0160] Acquiring the imaging angle information and spatial information of the angiographic image sequence, and performing three-dimensional reconstruction on the initial center line to obtain the target center line according to the angle information and the spatial information;

[0161] According to the target centerline, acquiring a first blood vessel cross-sectional area set along the target centerline direction;

[0162] Acquire an IVUS image corresponding to the angiography image sequence, and acquire a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image;

[0163] A registration calculation is performed on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain a registration result of the angiography image sequence and the IVUS image.

[0164] The storage device may be the memory 11 of the electronic device 1 , or may be another storage device that is communicatively connected to the electronic device 1 .

[0165] For a detailed description of the above steps, please refer to the above Figure 3 Functional module diagram of an embodiment of an image matching device 100 based on 3D reconstruction and Figure 1 Description of a flowchart of an embodiment of an image matching method based on three-dimensional reconstruction.

[0166] In addition, the embodiment of the present application also proposes a computer-readable storage medium, which can be non-volatile or volatile. The computer-readable storage medium can be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area, and the program storage area stores an image matching program 10 based on three-dimensional reconstruction. When the image matching program 10 based on three-dimensional reconstruction is executed by a processor, the following operations are implemented:

[0167] Acquire an angiography image sequence that meets a preset body position angle, and extract an initial center line corresponding to a target blood vessel region in the angiography image sequence;

[0168] Acquiring the imaging angle information and spatial information of the angiographic image sequence, and performing three-dimensional reconstruction on the initial center line to obtain the target center line according to the angle information and the spatial information;

[0169] According to the target centerline, acquiring a first blood vessel cross-sectional area set along the target centerline direction;

[0170] Acquire an IVUS image corresponding to the angiography image sequence, and acquire a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image;

[0171] A registration calculation is performed on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain a registration result of the angiography image sequence and the IVUS image.

[0172] The specific implementation of the computer-readable storage medium of the present application is substantially the same as the specific implementation of the above-mentioned image matching method based on three-dimensional reconstruction, and will not be described in detail here.

[0173] It should be noted that the serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof herein are intended to cover non-exclusive inclusion, so that the process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, device, article or method including the element.

[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware simulation platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, an electronic device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0175] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image matching method based on three-dimensional reconstruction, characterized in that: The method comprises: Acquire an angiography image sequence that meets a preset body position angle, and extract an initial center line corresponding to a target blood vessel region in the angiography image sequence; Acquiring the imaging angle information and spatial information of the angiographic image sequence, and performing three-dimensional reconstruction on the initial center line to obtain the target center line according to the angle information and the spatial information; According to the target centerline, acquiring a first blood vessel cross-sectional area set along the target centerline direction; Acquire an IVUS image corresponding to the angiography image sequence, and acquire a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image; A registration calculation is performed on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain a registration result of the angiography image sequence and the IVUS image.

2. The image matching method based on three-dimensional reconstruction according to claim 1, characterized in that: The extracting the initial center line corresponding to the target blood vessel region in the angiography image sequence includes: Detecting key frames of the angiographic image sequence using a pre-trained detection model; Segmenting the target blood vessel region of the key frame using a pre-trained segmentation model; performing binarization processing on the target blood vessel region to obtain a binary image of the target blood vessel; The binary image of the target blood vessel is thinned to obtain an initial center line corresponding to the target blood vessel region.

3. The image matching method based on three-dimensional reconstruction according to claim 2, characterized in that: The thinning process of the binary image of the target blood vessel to obtain an initial center line corresponding to the target blood vessel region includes: Scanning the pixel points in the binary image one by one to determine whether there is a non-vascular area in the neighborhood of the pixel point; If it exists, the layer number of the pixel point is set to 1; if it does not exist, the layer number of the pixel point is set to the minimum layer value of the neighborhood plus 1, until the layer numbers of all pixels in the binary image are obtained; Then, the pixel points in the binary image are scanned one by one to determine whether the current layer number of the pixel point is the maximum layer number in the neighborhood; If yes, the pixel point is retained; otherwise, the pixel point is removed to obtain the initial center line corresponding to the target blood vessel region.

4. The image matching method based on three-dimensional reconstruction according to claim 1, characterized in that: The three-dimensional reconstruction of the initial center line to obtain the target center line according to the angle information and the spatial information includes: Obtaining first coordinate information of each pixel point of the initial center line in the first coordinate system and second coordinate information in the second coordinate system according to the angle information and the spatial information; Obtaining a conversion formula between the first coordinate system and the second coordinate system; According to the conversion formula, the first coordinate information and the second coordinate information, three-dimensional reconstruction is performed on each pixel point of the initial center line to obtain a target center line.

5. The image matching method based on three-dimensional reconstruction according to claim 4, characterized in that: The three-dimensional reconstruction of each pixel point of the initial center line to obtain the target center line according to the conversion formula, the first coordinate information and the second coordinate information includes: Acquire a first distance from the light source to the imaging plane when the pixel point is imaged in the first coordinate system and a second distance from the light source to the imaging plane when the pixel point is imaged in the second coordinate system; According to the geometric relationship of perspective projection, a first relational expression between the first coordinate information and the first distance is obtained, and a second relational expression between the second coordinate information and the second distance is obtained; Combining the conversion formula, the first relational expression and the second relational expression to obtain the three-dimensional coordinate information of the pixel point; The target center line is obtained by performing three-dimensional reconstruction based on the three-dimensional coordinate information of each pixel point of the initial center line.

6. The image matching method based on three-dimensional reconstruction according to claim 1, characterized in that: The step of acquiring a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image comprises: Obtaining the coordinates of each discrete point in the target blood vessel region and a preset weight of each discrete point; Calculating the centroid coordinates of the target blood vessel region according to the coordinates of the discrete points and the preset weights; Generating straight lines along a preset angle range with the centroid coordinates as the center, and obtaining the coordinate information of the intersection set between each straight line and the target blood vessel region; Calculating diameter information of the target blood vessel region based on the intersection point set coordinate information; Based on the diameter information of the target blood vessel region, a second blood vessel cross-sectional area set of the target blood vessel region is acquired.

7. The image matching method based on three-dimensional reconstruction according to any one of claims 1 to 6, characterized in that: The registering calculation of the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain the registration result of the angiography image sequence and the IVUS image includes: Perform registration calculation on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set, and use the result with the minimum error as the registration result.

8. An image matching device based on three-dimensional reconstruction, characterized in that: The device comprises: Acquisition module: used to acquire angiography image sequence that meets the preset body position angle, and extract the initial center line corresponding to the target blood vessel area in the angiography image sequence; Reconstruction module: used to obtain the angle information and spatial information of the imaging sequence of the angiography images, and to perform three-dimensional reconstruction on the initial center line to obtain the target center line according to the angle information and the spatial information; A first calculation module: used for obtaining a first blood vessel cross-sectional area set along the target centerline according to the target centerline; A second calculation module: used for acquiring an IVUS image corresponding to the angiography image sequence, and acquiring a second blood vessel cross-sectional area set of the target blood vessel region according to the IVUS image; Registration module: used for performing registration calculation on the first blood vessel cross-sectional area set and the second blood vessel cross-sectional area set to obtain the registration result of the angiography image sequence and the IVUS image.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the image matching method based on three-dimensional reconstruction described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction-based image matching method according to any one of claims 1 to 7 is implemented.

Images