Method and system for real-time identification of interventional instrument head-end pose and angle based on x-ray fluoroscopic images

Abstract

The application provides a method and system for real-time identification of the head posture and angle of an interventional instrument based on an X-ray perspective image, the method comprising the following steps: S1, constructing a picture coordinate system of the X-ray perspective image; S2, identifying three key points on the head of the instrument in the X-ray perspective image by using a trained deep learning model, and outputting the coordinates of the three key points in the picture coordinate system; S3, converting the picture coordinate system into a posture coordinate system; S4, judging the posture of the head of the interventional instrument in the posture coordinate system; S5, judging the angle of the head in the posture coordinate system; and S6, judging the rotation of the interventional instrument in the vertical axial direction in the posture coordinate system. The application can efficiently and accurately identify the head posture and angle of the interventional instrument based on the X-ray image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for recognizing the posture and angle of the tip of interventional instruments. Background Technology

[0002] In complex medical conditions, such as coronary artery disease and cerebral aneurysms, highly accurate instrument navigation is crucial for successful treatment. This makes interpreting X-ray fluoroscopy images a challenging task. Even experienced physicians can make mistakes, especially under conditions of prolonged surgery and accumulated fatigue. Furthermore, excessive X-ray exposure can have long-term health effects on both doctors and patients. This invention provides a method for automatically identifying and determining the posture and orientation of interventional instrument tips in X-ray images, which is of significant value in improving surgical precision, reducing surgical time, and minimizing the side effects of X-ray exposure.

[0003] Patent CN112451095A discloses a posture estimation method based on a fluorescence fluoroscopic imaging device, and a system and method for three-dimensional imaging of body structures. This method first acquires three-dimensional coordinate data of points by scanning with a fluorescence fluoroscopic device. The drawback is that this method requires additional equipment such as electromagnetic sensors or complex image processing methods, which increases the space usage in the clinical operating room. Furthermore, determining the three-dimensional shape and projection angle of the catheter may require complex calculations and calibrations. If the catheter shape changes, or if bending or twisting occurs during operation, the device angle needs to be adjusted in real time, increasing the complexity of the surgery.

[0004] The invention patent with publication number CN105395252A proposes a wearable three-dimensional image navigation device for vascular interventional surgery with human-computer interaction. This device includes an electromagnetic positioning system that determines the catheter's position and orientation during the procedure through electromagnetic induction. The drawback is that this device relies on accurate spatial registration and information fusion. If any errors occur during registration, or if data loss or corruption occurs during information fusion, the orientation of the interventional instruments may not match the actual situation, severely impacting the clinical surgical outcome. Summary of the Invention

[0005] To overcome the aforementioned technical deficiencies, the present invention aims to provide a method for real-time identification of the posture and angle of the tip of an interventional instrument based on X-ray fluoroscopy images, comprising:

[0006] Step S1: Modify the image coordinate system to the lower left corner to construct the image coordinate system of the X-ray perspective image;

[0007] Step S2: Use the trained deep learning model to identify three key points on the instrument tip in the X-ray fluoroscopy image, and output the coordinates of the three key points in the image coordinate system. The first key point is the end of the interventional instrument tip, the second key point is the bending point of the interventional instrument tip, and the third key point is the intersection of the circle with the second key point as the center and the distance between the first and second key points as the radius with the interventional instrument tip.

[0008] Step S3: Parallel to the image coordinate system, and with the third key point as the origin of the coordinate system, convert the image coordinate system into the pose coordinate system;

[0009] Step S4: In the attitude coordinate system, with the third key point as the origin of the coordinate system, calculate the direction vector from the third key point to the second key point and its components on the x-axis and y-axis of the attitude coordinate system. Then, normalize the components on the x-axis and y-axis to obtain unit vectors. Finally, calculate the first angle between the direction vector and the y-axis of the attitude coordinate system. Determine the attitude of the interventional instrument head based on the range of the first angle.

[0010] Step S5: In the attitude coordinate system, take the third key point as the origin of the coordinate system, calculate the direction vector from the second key point to the first key point and its components on the x-axis and y-axis of the attitude coordinate system, and finally calculate the second angle between the direction component and the x-axis of the attitude coordinate system. Determine the angle of the head end based on the range of the second angle.

[0011] Step S6: In the attitude coordinate system, calculate the relative distance between the first key point and the third key point on the x-axis at the current moment, and then subtract it from the relative distance between the first key point and the third key point on the x-axis at the previous moment. Take the absolute value of the difference. When the absolute value changes, it is determined that the tip of the interventional instrument has rotated in the vertical axis.

[0012] It is worth noting that the interventional devices in this application include guidewires, angiography catheters, and stents, etc., which require determination of the orientation of their tips. The tip orientation of the interventional device in this application refers to determining the direction of the interventional device's advance along the vertical axis within the blood vessel lumen. The tip angle of the interventional device in this application refers to the angle of the device tip's orientation and its rotation along its own vertical axis in an X-ray image. The X-ray fluoroscopic image in this application is an image used in two-dimensional X-ray fluorescence fluoroscopy imaging equipment.

[0013] Furthermore, in step S4, the correspondence between the range of values ​​for the first included angle and the different postures of the interventional instrument tip is as follows:

[0014]

[0015] Where θ1 is the first included angle, Uy is the direction vector from the third key point to the second key point and its component on the y-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the y-axis, and Ux is the direction vector from the third key point to the second key point and its component on the x-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the x-axis.

[0016] When θ1∈(0, π / 2), it represents the first quadrant of the attitude coordinate system of the interventional device; when θ1∈(π / 2, π), it represents the second quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system of the interventional device.

[0017] Furthermore, in step S5, the correspondence between the range of values ​​for the second included angle and the angle at which the tip of the interventional device faces is as follows:

[0018]

[0019] Where θ2 is the first included angle, Vy is the component of the direction vector from the second key point to the first key point on the y-axis of the attitude coordinate system, and Vx is the component of the direction vector from the second key point to the first key point on the x-axis of the attitude coordinate system.

[0020] When θ2∈(0, π / 2), it represents the first quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(π / 2, π), it represents the second quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system where the tip of the interventional device is oriented.

[0021] Furthermore, in step S1, assuming that the coordinates of a point in the original coordinate system are (x, y) and its coordinates in the image coordinate system are (x′, y′), then:

[0022] x′=x

[0023] y′=Hy

[0024] Further, in step S3, assuming the coordinates of the first keypoint in the image coordinate system are (x1, y1), the coordinates of the second keypoint in the image coordinate system are (x2, y2), the coordinates of the third keypoint in the image coordinate system are (x3, y3), the coordinates of the first keypoint in the pose coordinate system are (x1′, y2′), and the coordinates of the second keypoint in the pose coordinate system are (x2′, y2′), then:

[0025] x1′=x2-x3

[0026] y1′=y1-y3,

[0027] x2′=x2-x3

[0028] y2′=y2-y3.

[0029] The second aspect of this application provides a system for real-time identification of the posture and angle of the tip of an interventional instrument based on X-ray fluoroscopy images, comprising:

[0030] The image coordinate system transformation module is used to modify the image coordinate system to the lower left corner and construct the image coordinate system of the X-ray perspective image.

[0031] The key point extraction module is used to identify three key points on the instrument tip in the X-ray fluoroscopic image using a trained deep learning model, and output the coordinates of the three key points in the image coordinate system. The first key point is the end of the interventional instrument tip, the second key point is the bending point of the interventional instrument tip, and the third key point is the intersection of the circle with the second key point as the center and the distance between the first and second key points as the radius with the interventional instrument tip.

[0032] The attitude coordinate system transformation module is used to transform the image coordinate system into an attitude coordinate system that is parallel to the image coordinate system and uses the third key point as the origin of the coordinate system.

[0033] The head end attitude determination module is used to calculate the direction vector from the third key point to the second key point and its components on the x-axis and y-axis of the attitude coordinate system, with the third key point as the origin of the coordinate system. Then, the components on the x-axis and y-axis are normalized to obtain unit vectors. Finally, the first angle between the direction vector and the y-axis of the attitude coordinate system is calculated, and the attitude of the interventional device head end is determined according to the range of the value of the first angle.

[0034] The head orientation determination module is used to calculate the direction vector from the second key point to the first key point and its components on the x-axis and y-axis of the attitude coordinate system, with the third key point as the origin of the coordinate system. Finally, it calculates the second angle between the direction component and the x-axis of the attitude coordinate system and determines the head orientation angle based on the range of the second angle.

[0035] The instrument axial rotation judgment module is used to calculate the relative distance between the first and third key points on the x-axis at the current moment in the attitude coordinate system, and then subtract it from the relative distance between the first and third key points on the x-axis at the previous moment. The absolute value of the difference is taken. When the absolute value changes, it is determined that the interventional instrument has rotated in the vertical axis.

[0036] Furthermore, the correspondence between the range of values ​​for the first included angle and the different postures of the interventional device tip is as follows:

[0037]

[0038] Where θ1 is the first included angle, Uy is the direction vector from the third key point to the second key point and its component on the y-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the y-axis, and Ux is the direction vector from the third key point to the second key point and its component on the x-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the x-axis.

[0039] When θ1∈(0, π / 2), it represents the first quadrant of the attitude coordinate system of the interventional device; when θ1∈(π / 2, π), it represents the second quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system of the interventional device.

[0040] Furthermore, the correspondence between the range of values ​​for the second included angle and the angle at which the tip of the interventional device faces is as follows:

[0041]

[0042] Where θ2 is the first included angle, Vy is the component of the direction vector from the second key point to the first key point on the y-axis of the attitude coordinate system, and Vx is the component of the direction vector from the second key point to the first key point on the x-axis of the attitude coordinate system.

[0043] When θ2∈(0, π / 2), it represents the first quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(π / 2, π), it represents the second quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system where the tip of the interventional device is oriented.

[0044] Furthermore, assuming that the coordinates of a point in the original coordinate system are (x, y) and its coordinates in the image coordinate system are (x′, y′), then:

[0045] x′=x

[0046] y′=Hy.

[0047] Furthermore, assuming the coordinates of the first keypoint in the image coordinate system are (x1, y1), the second keypoint in the image coordinate system are (x2, y2), the third keypoint in the image coordinate system are (x3, y3), the first keypoint in the pose coordinate system is (x1′, y2′), and the second keypoint in the pose coordinate system is (x2′, y2′), then:

[0048] x1′=x2-x3

[0049] y1′=y1-y3,

[0050] x2′=x2-x3

[0051] y2′=y2-y3.

[0052] Compared with existing technologies, the above technical solution has the following advantages:

[0053] This invention achieves accurate identification of the posture and angle of the interventional instrument tip in X-ray images, realizing accurate, efficient, and adaptive technical effects. Specifically, it includes:

[0054] 1. Posture and Angle Recognition: This invention uses a deep learning model to learn and extract complex image features from a large amount of labeled X-ray image data to accurately identify key points and their coordinates on the tip of the interventional instrument, thereby accurately identifying the posture and angle of the instrument tip. Furthermore, by setting a specific posture coordinate system and utilizing vector analysis, this invention can make more detailed and accurate judgments on posture and angle.

[0055] 2. Rapid Processing: Utilizing the processing power of deep learning models, this invention can complete the recognition and processing of X-ray images in a short time, thereby obtaining the posture and angle information of the instrument tip. This rapid processing capability is extremely useful in medical scenarios requiring quick decision-making, such as emergency surgery.

[0056] 3. Vertical Axis Rotation Recognition: This invention can identify the vertical axis rotation of interventional instruments under X-ray images by measuring the change in distance between key points along the X-axis. This recognition method can provide more comprehensive posture information and offer more accurate guidance for medical procedures.

[0057] 4. Adaptive Optimization: With the continuous accumulation of training data and ongoing model optimization, the recognition performance of this invention can be continuously improved. Even when faced with unfamiliar situations, this invention can make accurate recognitions, demonstrating strong adaptability. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the structural components of angiography catheter interventional devices;

[0059] Figure 2 This is a flowchart of a method for real-time identification of the posture and angle of interventional instrument tip based on X-ray fluoroscopy images;

[0060] Figure 3A This is a schematic diagram based on the orientation of the instrument within the blood vessel lumen and the first quadrant of its corresponding attitude coordinate system.

[0061] Figure 3B This is a schematic diagram based on the orientation of the instrument within the blood vessel lumen and the second quadrant of its corresponding attitude coordinate system.

[0062] Figure 3C This is a schematic diagram based on the orientation of the instrument within the blood vessel lumen and the corresponding third quadrant of the attitude coordinate system.

[0063] Figure 3D This is a schematic diagram based on the orientation of the instrument within the blood vessel lumen and the corresponding fourth quadrant of the attitude coordinate system.

[0064] Figure 4A This is a schematic diagram based on the orientation of the instrument tip within the blood vessel lumen and the first quadrant of its corresponding attitude coordinate system.

[0065] Figure 4B This is a schematic diagram based on the orientation of the instrument tip within the blood vessel lumen and the corresponding second quadrant of the attitude coordinate system;

[0066] Figure 4C This is a schematic diagram based on the orientation of the instrument tip within the blood vessel lumen and the corresponding third quadrant of the attitude coordinate system.

[0067] Figure 4D This is a schematic diagram based on the orientation of the instrument tip within the blood vessel lumen and the corresponding fourth quadrant of the attitude coordinate system.

[0068] Figure 5 This is a schematic diagram illustrating the principle of determining the rotation of the interventional instrument along the vertical axis based on the absolute value of the difference between the relative distances of the first and third key points on the x-axis.

[0069] Figure 6 This is a schematic diagram illustrating the dynamic loop principle of a method for real-time identification of the posture and angle of interventional instrument tip based on X-ray fluoroscopy images. Detailed Implementation

[0070] The advantages of the present invention are further illustrated below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that the following detailed description is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0071] This embodiment provides a system for real-time identification of the posture and angle of the head end of interventional instruments based on X-ray fluoroscopy images. It includes an image coordinate system transformation module, a key point extraction module, a posture coordinate system transformation module, a head end posture judgment module, a head end orientation judgment module, and an instrument axial rotation judgment module.

[0072] The image coordinate system transformation module is used to modify the image coordinate system to the lower left corner, constructing the image coordinate system for X-ray perspective images.

[0073] The key point extraction module is used to identify three key points on the instrument tip in the X-ray fluoroscopic image using a trained deep learning model, and outputs the coordinates of the three key points in the image coordinate system. The first key point is the end of the interventional instrument tip, the second key point is the bending point of the interventional instrument tip, and the third key point is the intersection of the circle with the second key point as the center and the distance between the first and second key points as the radius with the interventional instrument tip.

[0074] The pose coordinate system transformation module is used to transform the image coordinate system into a pose coordinate system that is parallel to the image coordinate system and uses the third key point as the origin of the coordinate system.

[0075] The head-end attitude determination module is used to calculate the direction vector from the third key point to the second key point and its components on the x-axis and y-axis of the attitude coordinate system, with the third key point as the origin of the coordinate system. Then, the components on the x-axis and y-axis are normalized to obtain unit vectors. Finally, the first angle between the direction vector and the y-axis of the attitude coordinate system is calculated, and the attitude of the interventional device head end is determined based on the range of the first angle.

[0076] The head orientation determination module is used to calculate the direction vector from the second key point to the first key point and its components on the x-axis and y-axis of the attitude coordinate system, with the third key point as the origin of the coordinate system. Finally, it calculates the second angle between the direction component and the x-axis of the attitude coordinate system and determines the head orientation angle based on the range of the second angle.

[0077] The instrument axial rotation judgment module is used to calculate the relative distance between the first and third key points on the x-axis at the current moment in the attitude coordinate system, and then subtract it from the relative distance between the first and third key points on the x-axis at the previous moment. The absolute value of the difference is taken. When the absolute value changes, it is determined that the interventional instrument has rotated in the vertical axis.

[0078] The orientation of the interventional device tip refers to determining the direction of the interventional device's movement along its vertical axis within the blood vessel lumen. The angle of the interventional device tip refers to the angle at which the device tip faces and its rotation along its vertical axis in an X-ray image. The interventional devices in this application include, but are not limited to, guidewires, angiography catheters, and stents, as well as other devices requiring determination of tip orientation.

[0079] like Figure 1 As shown, the interventional device of this application includes a vertical axis 10 and a head end 20 extending from the vertical axis 10 to the distal end. The head end 20 has three key points, namely a first key point 21, a second key point 22 and a third key point 23.

[0080] Since the identification methods for interventional devices described above are basically similar, the following description will primarily focus on angiography catheters. For example, such as... Figure 2 As shown, the method for real-time identification of the posture and angle of interventional instrument tip using a system based on X-ray fluoroscopy images includes the following steps:

[0081] Step S1: Modify the image coordinate system to the lower left corner to construct the image coordinate system of the X-ray perspective image.

[0082] First, the coordinate system is transformed. By default, the computer-processed image uses the top left corner as the coordinate point. To facilitate recognition, the image coordinate system is changed to the bottom left corner: Let the original coordinates be (x, y) and the image height be H.

[0083] Suppose a point has coordinates (x, y) in the original coordinate system and (x′, y′) in the image coordinate system, then:

[0084] x′=x

[0085] y′=Hy

[0086] This formula means that the new y-coordinate is the image height minus the original y-coordinate. In computer graphics, the y-axis points downwards, but in practical applications, the y-axis points upwards, requiring subtraction to flip the y-coordinate.

[0087] Step S2: Use the trained deep learning model to identify three key points on the instrument tip in the X-ray fluoroscopic image and output the coordinates of the three key points in the image coordinate system.

[0088] The first key point 21 is the end of the interventional device tip 20, the second key point 22 is the bending point of the interventional device tip 20, and the third key point 23 is the intersection of the circle drawn with the second key point 22 as the center and the distance between the first key point 21 and the second key point 22 as the radius with the interventional device tip 20.

[0089] First, a deep learning model is trained using labeled X-ray image data, including labeled instrument recognition boxes and instrument key points, to accurately identify multiple key points on the tip 20 of the interventional instrument in X-ray fluoroscopy images. Then, the input X-ray image to be identified is processed by the deep learning model, which quickly and accurately identifies multiple key points on the tip 20 of the interventional instrument and outputs the coordinates of these key points. For example, the deep learning models used in this application include the object detection key point detection network YOLO and the image segmentation neural network U-net.

[0090] Among them, such as Figure 2 As shown, there are three key points at the tip of the interventional instrument 20: key point 1, key point 2, and key point 3. Key point 1 is the tip of the instrument, i.e., the first key point 21. Key point 2 is the bending point of the instrument tip 20, i.e., the second key point 22. The distance between key point 1 and key point 2 is S. Key point 3 is the intersection of the circle drawn with key point 2 as the center and S as the radius with the instrument, i.e., the third key point 23. The coordinates of key point 1 are defined as (x1, y1); the coordinates of key point 2 are defined as (x2, y2); and the coordinates of key point 3 are defined as (x3, y3).

[0091] Step S3: Parallel to the image coordinate system, and using the third key point as the origin of the coordinate system, convert the image coordinate system into the pose coordinate system.

[0092] Determining the attitude and angle of the interventional instrument tip requires a specific attitude coordinate system. The attitude coordinate system is parallel to the image coordinate system, and the third key point 23 is used as the origin of the coordinate system to transform the image coordinate system into the attitude coordinate system.

[0093] For example, suppose the coordinates of the first keypoint 21 in the image coordinate system are (x1, y1), the coordinates of the second keypoint 22 in the image coordinate system are (x2, y2), the coordinates of the third keypoint 23 in the image coordinate system are (x3, y3), the coordinates of the first keypoint 21 in the pose coordinate system are (x1′, y2′), and the coordinates of the second keypoint 22 in the pose coordinate system are (x2′, y2′), then:

[0094] x1′=x2-x3

[0095] y1′=y1-y3,

[0096] x2′=x2-x3

[0097] y2′=y2-y3

[0098] Step S4: Determine the orientation of the interventional device tip

[0099] In the attitude coordinate system, with the third key point 23 as the origin of the coordinate system, the direction vector from the third key point 23 to the second key point 22 and its components on the x-axis and y-axis of the attitude coordinate system are calculated. Then, the components on the x-axis and y-axis are normalized to obtain unit vectors. Finally, the first angle between the direction vector and the y-axis of the attitude coordinate system is calculated. The attitude of the interventional instrument head 20 is determined based on the range of the first angle.

[0100] The direction vector from key point 3 to key point 2 is V1, which has components V1x and V1y on the x-axis and y-axis, respectively.

[0101] V1x=x2′-0=x2′

[0102] V1y=y2′-0=y2′

[0103] Normalize vectors V1x and V1y so that they only have directional attributes, resulting in unit vectors Ux and Uy.

[0104] in:

[0105]

[0106]

[0107] Finally, we obtain the angle between the direction vector U(Ux,Uy) and the y-axis of the attitude coordinate system: θ1.

[0108] like Figures 3A-3D As shown, based on the different orientations of the instruments within the blood vessel lumen, they can be categorized into different quadrants. The range of values ​​for θ1 and their corresponding orientation coordinate system quadrants can be categorized as follows:

[0109] (1) First quadrant: Ux>0, Uy>0, θ1∈(0,π / 2);

[0110] (2) Second quadrant: Ux<0, Uy>0, θ1∈(π / 2,π];

[0111] (3) Third quadrant: Ux<0, Uy<0, θ1∈(-π,-π / 2);

[0112] (4) Fourth quadrant: Ux>0, Uy<0, θ1∈(-π / 2,0).

[0113] in:

[0114]

[0115] From a programming perspective, this formula requires first determining which quadrant of the attitude coordinate system the interventional device is in. `atan2` is used to calculate the angle between two points (x, y coordinates). The `atan2` function returns the angle between the line connecting these two points to the origin and the positive x-axis, with the result ranging from -π to π.

[0116] Step S5: Determine the orientation of the interventional instrument tip.

[0117] In the attitude coordinate system, with the third key point 23 as the origin of the coordinate system, the direction vector from the second key point 22 to the first key point 21 and its components on the x-axis and y-axis of the attitude coordinate system are calculated. Finally, the second angle between the direction component and the x-axis of the attitude coordinate system is calculated, and the angle of the head end 20 is determined based on the range of the second angle.

[0118] Determining the orientation of head end 20 requires an attitude coordinate system. Take key point 3 as the origin of the attitude coordinate system. Let the orientation vector of head end 20 be V2. V2 has components V2x and V2y on the x-axis and y-axis, respectively.

[0119] V2=(V2x,V2y)=(x2-x1,y2-y1)

[0120] like Figures 4A-4D As shown, based on the different orientations of the instrument tip within the blood vessel lumen, they can be categorized into different quadrants. The range of values ​​for θ2 and their corresponding orientation coordinate system quadrants can be categorized as follows:

[0121] (1) First quadrant: V2x>0, V2y>0, θ2∈(0,π / 2);

[0122] (2) Second quadrant: V2x<0, V2y>0, θ2∈(π / 2,π];

[0123] (3) Third quadrant: V2x<0, V2y<0, θ2∈(-π,-π / 2);

[0124] (4) Fourth quadrant: V2x>0, V2y<0, θ2∈(-π / 2,0).

[0125] in:

[0126]

[0127] Step S6: Determine the rotation of the interventional instrument along its vertical axis.

[0128] In the attitude coordinate system, calculate the relative distance between the first key point 21 and the third key point 23 on the x-axis at the current moment, and then subtract it from the relative distance between the first key point 21 and the third key point 23 on the x-axis at the previous moment. Take the absolute value of the difference. When the absolute value changes, it is determined that the interventional instrument has rotated upward on the vertical axis 10.

[0129] like Figure 5 As shown, a posture coordinate system is needed to determine the rotation of the interventional instrument tip 20 along the vertical axis 10. By measuring the distance Sx' between key points 1 and 3 on the x-axis and subtracting the absolute value from the previous distance Sx, the rotation of the instrument along the vertical axis 10 in the X-ray image can be determined. The difference is defined as ΔS, which represents the distance between key points 1 and 3 on the x-axis, i.e., their horizontal distance. A change in the value of ΔS indicates that the interventional instrument has rotated along the vertical axis 10.

[0130] The formula for calculating ΔS is as follows:

[0131] Sx=x 3t1 -x 1t1 (Time t1)

[0132] Sx′=X 3t1 -x 1t2 (Time t2)

[0133] ΔS = |Sx′ - Sx|.

[0134] It is worth noting that this application does not limit the order of steps S4, S5 and S6, and the order can be arbitrarily changed, or they can be performed simultaneously.

[0135] like Figure 6 As shown, the information obtained in steps S4, S5, and S6 will be output to the X-ray image. After the information is updated, the X-ray image is input into the deep learning model again for image processing, and step S2 is repeated. This process is repeated dynamically, thereby enabling real-time identification of the posture and angle of the interventional instrument tip based on the X-ray fluoroscopic image.

[0136] To determine the vertical axis direction, the X-ray imaging camera lens is rotated on the horizontal axis according to the physician's visual angle requirements. The new angle obtained by the rotation can also be used to determine the head posture using this method, thereby obtaining the vertical axis head direction that cannot be determined in the original position.

[0137] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for real-time identification of the posture and angle of the tip of an interventional instrument based on X-ray fluoroscopy images, characterized in that, include: Step S1: Modify the image coordinate system to the lower left corner to construct the image coordinate system of the X-ray perspective image; Step S2: Use the trained deep learning model to identify three key points on the instrument tip in the X-ray fluoroscopy image, and output the coordinates of the three key points in the image coordinate system. The first key point is the end of the interventional instrument tip, the second key point is the bending point of the interventional instrument tip, and the third key point is the intersection of the circle with the second key point as the center and the distance between the first and second key points as the radius with the interventional instrument tip. Step S3: Parallel to the image coordinate system, and with the third key point as the origin of the coordinate system, convert the image coordinate system into the pose coordinate system; Step S4: In the attitude coordinate system, with the third key point as the origin of the coordinate system, calculate the direction vector from the third key point to the second key point and its components on the x-axis and y-axis of the attitude coordinate system. Then, normalize the components on the x-axis and y-axis to obtain unit vectors. Finally, calculate the first angle between the direction vector and the y-axis of the attitude coordinate system. Determine the attitude of the interventional instrument head based on the range of the first angle. Step S5: In the attitude coordinate system, take the third key point as the origin of the coordinate system, calculate the direction vector from the second key point to the first key point and its components on the x-axis and y-axis of the attitude coordinate system, and finally calculate the second angle between the direction component and the x-axis of the attitude coordinate system. Determine the angle of the head end based on the range of the second angle. Step S6: In the attitude coordinate system, calculate the relative distance between the first key point and the third key point on the x-axis at the current moment, and then subtract it from the relative distance between the first key point and the third key point on the x-axis at the previous moment. Take the absolute value of the difference. When the absolute value changes, it is determined that the interventional instrument has rotated in the vertical axis.

2. The method for real-time identification of the posture and angle of the interventional instrument tip based on X-ray fluoroscopy images as described in claim 1, characterized in that, In step S4, the correspondence between the range of values ​​for the first included angle and the different postures of the interventional instrument tip is as follows: Where θ1 is the first included angle, Uy is the direction vector from the third key point to the second key point and its component on the y-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the y-axis, and Ux is the direction vector from the third key point to the second key point and its component on the x-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the x-axis. When θ1∈(0, π / 2), it represents the first quadrant of the attitude coordinate system of the interventional device; when θ1∈(π / 2, π), it represents the second quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system of the interventional device.

3. The method for real-time identification of the posture and angle of the interventional instrument tip based on X-ray fluoroscopy images as described in claim 1, characterized in that, In step S5, the correspondence between the range of values ​​for the second included angle and the angle at which the tip of the interventional device faces is as follows: Where θ2 is the first included angle, Vy is the component of the direction vector from the second key point to the first key point on the y-axis of the attitude coordinate system, and Vx is the component of the direction vector from the second key point to the first key point on the x-axis of the attitude coordinate system. When θ2∈(0, π / 2), it represents the first quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(π / 2, π), it represents the second quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system where the tip of the interventional device is oriented.

4. The method for real-time identification of the posture and angle of the interventional instrument tip based on X-ray fluoroscopy images as described in claim 1, characterized in that, In step S1, assuming a point has coordinates (x, y) in the original coordinate system and (x', y') in the image coordinate system, then: Where H is the height of the image.

5. The method for real-time identification of the posture and angle of the interventional instrument tip based on X-ray fluoroscopy images as described in claim 1, characterized in that, In step S3, assuming the coordinates of the first keypoint in the image coordinate system are (x1, y1), the coordinates of the second keypoint in the image coordinate system are (x2, y2), the coordinates of the third keypoint in the image coordinate system are (x3, y3), the coordinates of the first keypoint in the pose coordinate system are (x1', y2'), and the coordinates of the second keypoint in the pose coordinate system are (x2', y2'), then: ， 。 6. A system for real-time identification of the posture and angle of interventional instrument tip based on X-ray fluoroscopy images, characterized in that, include: The image coordinate system transformation module is used to modify the image coordinate system to the lower left corner and construct the image coordinate system of the X-ray perspective image. The key point extraction module is used to identify three key points on the instrument tip in the X-ray fluoroscopic image using a trained deep learning model, and output the coordinates of the three key points in the image coordinate system. The first key point is the end of the interventional instrument tip, the second key point is the bending point of the interventional instrument tip, and the third key point is the intersection of the circle with the second key point as the center and the distance between the first and second key points as the radius with the interventional instrument tip. The attitude coordinate system transformation module is used to transform the image coordinate system into an attitude coordinate system that is parallel to the image coordinate system and uses the third key point as the origin of the coordinate system. The head end attitude determination module is used to calculate the direction vector from the third key point to the second key point and its components on the x-axis and y-axis of the attitude coordinate system, with the third key point as the origin of the coordinate system. Then, the components on the x-axis and y-axis are normalized to obtain unit vectors. Finally, the first angle between the direction vector and the y-axis of the attitude coordinate system is calculated, and the attitude of the interventional device head end is determined according to the range of the value of the first angle. The head orientation determination module is used to calculate the direction vector from the second key point to the first key point and its components on the x-axis and y-axis of the attitude coordinate system, with the third key point as the origin of the coordinate system. Finally, it calculates the second angle between the direction component and the x-axis of the attitude coordinate system and determines the head orientation angle based on the range of the second angle. The instrument axial rotation judgment module is used to calculate the relative distance between the first and third key points on the x-axis at the current moment in the attitude coordinate system, and then subtract it from the relative distance between the first and third key points on the x-axis at the previous moment. The absolute value of the difference is taken. When the absolute value changes, it is determined that the interventional instrument has rotated in the vertical axis.

7. The system for real-time identification of the posture and angle of interventional instrument tip based on X-ray fluoroscopy images as described in claim 6, characterized in that, The correspondence between the range of values ​​for the first included angle and the different postures of the interventional device tip is as follows: Where θ1 is the first included angle, Uy is the direction vector from the third key point to the second key point and its component on the y-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the y-axis, and Ux is the direction vector from the third key point to the second key point and its component on the x-axis of the attitude coordinate system, and then the unit vector obtained by normalizing the component on the x-axis. When θ1∈(0, π / 2), it represents the first quadrant of the attitude coordinate system of the interventional device; when θ1∈(π / 2, π), it represents the second quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system of the interventional device; when θ1∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system of the interventional device.

8. The system for real-time identification of the posture and angle of interventional instrument tip based on X-ray fluoroscopy images as described in claim 6, characterized in that, The relationship between the range of values ​​for the second included angle and the angle at which the tip of the interventional device faces is as follows: Where θ2 is the first included angle, Vy is the component of the direction vector from the second key point to the first key point on the y-axis of the attitude coordinate system, and Vx is the component of the direction vector from the second key point to the first key point on the x-axis of the attitude coordinate system. When θ2∈(0, π / 2), it represents the first quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(π / 2, π), it represents the second quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π, -π / 2), it represents the third quadrant of the attitude coordinate system where the tip of the interventional device is oriented; when θ2∈(-π / 2, 0), it represents the fourth quadrant of the attitude coordinate system where the tip of the interventional device is oriented.

9. The system for real-time identification of the posture and angle of interventional instrument tip based on X-ray fluoroscopy images as described in claim 6, characterized in that, Suppose a point has coordinates (x, y) in the original coordinate system and (x', y') in the image coordinate system, then: Where H is the height of the image.

10. The system for real-time identification of the posture and angle of interventional instrument tip based on X-ray fluoroscopy images as described in claim 6, characterized in that, Assuming the coordinates of the first keypoint in the image coordinate system are (x1, y1), the second keypoint in the image coordinate system are (x2, y2), the third keypoint in the image coordinate system are (x3, y3), the first keypoint in the pose coordinate system is (x1', y2'), and the second keypoint in the pose coordinate system is (x2', y2'), then: ， 。

Images