Image generation method and device applied to vascular interventional operation

Abstract

The invention provides an image generation method and device applied to a vascular interventional operation. The method comprises the steps of obtaining a preoperative three-dimensional image and an intraoperative two-dimensional image obtained from at least two different scanning angles; projecting the preoperative three-dimensional image according to at least two different scanning angles to obtain at least two corresponding preoperative two-dimensional projection images; registering the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image to obtain image transformation relation data; processing the preoperative two-dimensional projection image by using the image transformation relation data to obtain at least two corresponding registered two-dimensional images; fusing the registered two-dimensional image with the corresponding intra-operative two-dimensional image to obtain at least two corresponding fused two-dimensional images; determining three-dimensional position information of the vascular interventional surgical instrument in a preoperative three-dimensional image coordinate system according to the positions of the vascular interventional surgical instrument images in the at least two fused two-dimensional images; and forming a three-dimensional vascular intervention surgical instrument image in the preoperative three-dimensional image according to the three-dimensional position information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, in particular to an image generation method and device applied to vascular intervention surgery. BACKGROUND

[0002] In current vascular intervention surgery, doctors need to determine the real-time position of surgical instruments such as catheters and guide wires in the human body with the help of DSA images. DSA images belong to intraoperative images, which are two-dimensional projections of X-rays penetrating the human body. Two frames of X-ray images taken before and after the injection of contrast medium are processed through subtraction, enhancement and reimaging to obtain clear pure vascular images. DSA images are affected by blood vessel diameter and blood flow velocity. When the dose of contrast medium is low, the visualization of blood vessels is not clear and complete, and only the local image of the flow direction of the contrast point can be displayed.

[0003] CT images can provide three-dimensional spatial anatomic structure of blood vessel morphology. Compared with DSA, CTA can more clearly display blood vessel images and three-dimensional structures. However, CT images belong to preoperative images and cannot reflect the real-time position of surgical instruments such as catheters and guide wires in the human body during surgery.

[0004] Therefore, doctors can only imagine the three-dimensional spatial trend and morphology of blood vessels based on the real-time position of surgical instruments such as catheters and guide wires in DSA images, combined with the tissue structure characteristics around the corresponding field of view of CT images, and conceive the three-dimensional spatial delivery path of surgical instruments in their minds. SUMMARY

[0005] Therefore, doctors can only imagine the three-dimensional spatial trend and morphology of blood vessels based on the real-time position of surgical instruments such as catheters and guide wires in DSA images, combined with the tissue structure characteristics around the corresponding field of view of CT images, and conceive the three-dimensional spatial delivery path of surgical instruments in their minds. Therefore, doctors can only imagine the three-dimensional spatial trend and morphology of blood vessels based on the real-time position of surgical instruments such as catheters and guide wires in DSA images, combined with the tissue structure characteristics around the corresponding field of view of CT images, and conceive the three-dimensional spatial delivery path of surgical instruments in their minds.

[0006] Optionally, after forming the three-dimensional blood vessel interventional surgery instrument image, the method further comprises: obtaining position information of the blood vessel interventional surgery instrument in a positioning coordinate system; and updating a position of the three-dimensional blood vessel interventional surgery instrument image in the preoperative three-dimensional image according to a mapping relationship between the positioning coordinate system and the preoperative three-dimensional image coordinate system and using the position information.

[0007] Optionally, after forming the three-dimensional blood vessel interventional surgery instrument image, the method further comprises: obtaining rotation motion information of the blood vessel interventional surgery instrument; and setting the three-dimensional blood vessel interventional surgery instrument image to rotate according to the rotation motion information, so as to simulate the rotation motion of the blood vessel interventional surgery instrument.

[0008] Optionally, the registration of the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image further comprises: obtaining a first intraoperative two-dimensional image in a state without injection of contrast agent and a second intraoperative two-dimensional image in a state with injection of contrast agent at the same scanning angle; obtaining a two-dimensional blood vessel image according to the first intraoperative two-dimensional image and the second intraoperative two-dimensional image; removing a background in the preoperative two-dimensional projection image to retain only a blood vessel image; constructing a Gaussian difference scale space image of the preoperative two-dimensional projection image after removal of the background and the two-dimensional blood vessel image by using a preset scale parameter; respectively scanning extreme points in the Gaussian difference scale space image, and retaining extreme points with a contrast higher than a contrast threshold as blood vessel feature points; taking a feature region of a preset size as a center of each blood vessel feature point, dividing the feature region into a plurality of sub-blocks, determining a seed point based on a histogram of oriented gradients of the sub-blocks, and determining a descriptor of the corresponding blood vessel feature point by using a direction value of the seed point; determining the blood vessel feature points matched with each other by using the descriptors of the blood vessel feature points in the preoperative two-dimensional projection image and the two-dimensional blood vessel image; randomly taking a plurality of groups of matching points from the blood vessel feature points matched with each other, and calculating an affine matrix by using the descriptors and a binary graph as the image transformation relationship data.

[0009] Optionally, the registration of the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image further comprises: segmenting bone images in the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image respectively to obtain a preoperative two-dimensional bone image and an intraoperative two-dimensional bone image; constructing a Gaussian difference scale space image of the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image by using a preset scale parameter; scanning extreme points in the Gaussian difference scale space image respectively, and retaining extreme points with a contrast higher than a contrast threshold as bone feature points; taking a feature region of a preset size centered on each bone feature point, dividing the feature region into a plurality of sub-blocks, determining a seed point based on a histogram of oriented gradients of the sub-blocks, and determining a descriptor of the corresponding bone feature point by using an orientation value of the seed point; determining the bone feature points matched with each other by using the descriptors of the bone feature points in the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image; and randomly taking a plurality of matching point pairs from the bone feature points matched with each other, and calculating an affine matrix by using the descriptors and a binary graph as the image transformation relationship data.

[0010] Optionally, the registration of the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image further comprises: segmenting bone images in the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image respectively to obtain a preoperative two-dimensional bone image and an intraoperative two-dimensional bone image; constructing a Gaussian difference scale space image of the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image by using a preset scale parameter; scanning extreme points in the Gaussian difference scale space image respectively, and retaining extreme points with a contrast higher than a contrast threshold as bone feature points; taking a feature region of a preset size centered on each bone feature point, dividing the feature region into a plurality of sub-blocks, determining a seed point based on a histogram of oriented gradients of the sub-blocks, and determining a descriptor of the corresponding bone feature point by using an orientation value of the seed point; determining the bone feature points matched with each other by using the descriptors of the bone feature points in the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image; and randomly taking a plurality of matching point pairs from the bone feature points matched with each other, and calculating an affine matrix by using the descriptors and a binary graph as the image transformation relationship data.

[0011] Optionally, before projecting the preoperative three-dimensional image according to at least two different scanning angles, the method further comprises: acquiring an intraoperative scene image shot by a multi-view camera device, wherein the intraoperative scene image includes an intraoperative imaging device and a surgical bed; constructing a three-dimensional coordinate system based on a fixed axis of the surgical bed, extracting a structural feature of the intraoperative imaging device in the intraoperative scene image, and determining the scanning angles according to the positions of the structural feature in the three-dimensional coordinate system.

[0012] Optionally, the preoperative three-dimensional image is a preoperative CTA image, and the intraoperative two-dimensional image is an intraoperative DSA image.

[0013] Optionally, the blood vessel intervention surgical instrument image includes a catheter image and / or a guide wire image.

[0014] Correspondingly, the application provides an image generation device, which is characterized by comprising: a processor and a memory connected with the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the image generation method.

[0015] The application further provides a vascular interventional surgery system, comprising the image generation device and a positioning system of a vascular interventional surgery instrument; wherein the positioning system comprises a position sensor arranged on the vascular interventional surgery instrument and a position acquisition device configured to determine position information of a front end of the vascular interventional surgery instrument according to the position sensor.

[0016] According to the image generation method and device provided by the embodiment of the application, the preoperative three-dimensional image is projected to obtain at least two projection images according to the scanning angles of the at least two intraoperative two-dimensional images, all the projection images can clearly display the blood vessels and accurately reflect the projection of the three-dimensional structure on the two-dimensional plane, so that the blood vessel structure is more accurate and other tissues can be displayed; after the projection images are registered with the corresponding intraoperative two-dimensional images to obtain the registered images with high enough similarity, the registered images are fused with the intraoperative two-dimensional images, the blood vessels and the surrounding tissues can be clearly displayed while the vascular interventional surgery instrument is displayed, then the three-dimensional position of the vascular interventional surgery instrument in the preoperative three-dimensional image is calculated according to the two-dimensional positions of the vascular interventional surgery instrument in the at least two two-dimensional images, and finally the vascular interventional instrument image is generated in the preoperative three-dimensional image, which can enhance the spatial cognition of the medical staff on the tissue structure and more accurately obtain the positional relationship between the blood vessels and the interventional instrument, and the scheme can be used for interventional surgery planning and navigation to improve the efficiency of the interventional surgery. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0018] Figure 1 The flowchart of the image generation method in the embodiment of the application. DETAILED DESCRIPTION

[0019] The technical solutions of the application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are some of the embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0020] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0021] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements, it can be wireless connection, or it can be wired connection. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0022] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict.

[0023] The embodiment of the present application provides an image generation method applied to a vascular interventional surgery, which can be executed by a computer or a server and the like electronic device, as shown in the figure Figure 1 The method comprises the following operations:

[0024] S1, obtaining a preoperative three-dimensional image and intraoperative two-dimensional images obtained at at least two different scanning angles, wherein the preoperative three-dimensional image at least contains a blood vessel image, the intraoperative two-dimensional image at least contains a blood vessel interventional surgery instrument image, and the angle difference between any two scanning angles of the at least two different scanning angles is greater than an angle threshold. The preoperative three-dimensional image can be specifically an image scanned by a medical imaging device, such as a CT device, and in the case of using a contrast agent, a CT angiography (CTA) image can be scanned; or an MRA (magnetic resonance angiography) image scanned by a nuclear magnetic scanning device. The two images belong to three-dimensional body data, and can clearly show the three-dimensional blood vessel structure of the surgical object. Since the image is scanned for preoperative diagnosis or surgical planning, the blood vessel interventional surgery instrument image does not exist in the image.

[0025] Intraoperative two-dimensional images refer to images obtained using medical imaging equipment in the operating room after vascular interventional surgical instruments have entered the blood vessels of the surgical patient during the procedure. These images are typically X-ray-based scans. While they can display the image of the vascular interventional surgical instruments, they cannot clearly or stably display the image of the blood vessels. Interventional surgical instruments can be guidewires, catheters, or any suitable device.

[0026] Regarding the scanning angle, during interventional surgery, when performing X-ray scans on the surgical object, the surgeon can set the scanning angle of the medical imaging equipment according to the actual situation. In this embodiment, imaging is required at least two scanning angles to obtain at least two intraoperative images from different angles. For ease of subsequent description, two intraoperative two-dimensional images are used as an example, denoted as DSA. α and DSA β α and β represent their scanning angles, and in practical applications, the difference between α and β needs to be greater than 30° (angle threshold). It should be noted that DSA... α and DSA β The only difference is the scanning angle; the object being scanned remains the same, while other factors such as the patient's position and the actual location of the vascular interventional surgical instrument are identical.

[0027] S2, Project the preoperative three-dimensional image according to at least two different scanning angles to obtain at least two corresponding preoperative two-dimensional projection images.

[0028] If the device executing this method cannot directly read the scanning angle data of the medical imaging device due to issues such as data interface and permissions, the angle data can be obtained by manual input or with the help of a visual tracking system, or by traversing the projection angles to find the projection image with the highest similarity to the intraoperative two-dimensional image. The projection angle corresponding to this projection image can be regarded as the scanning angle of the intraoperative two-dimensional image.

[0029] Preoperative 3D images can be viewed as a 3D model. Given the projection angle, calculations can be performed on the 3D data based on that angle to obtain a 2D image with a high degree of similarity to the intraoperative 2D image. Specifically, the DRR (Digitally Reconstructed Radiograph) algorithm can be used to generate a 2D image from the 3D volume data (multiple cross-sectional image data) through mathematical simulation algorithms.

[0030] To obtain the projected image, a 3D coordinate system needs to be constructed with the bed orientation and its vertical plane. The deviation angle theta (including three directions) between the actual exposure angle of the intraoperative DSA equipment and the exposure angle of the preoperative CTA equipment is calculated. The CTA data is rotated around the center point by theta, thereby calculating the planar 2D projection of the CTA at that angle.

[0031] This step will obtain the projected images corresponding to each projection angle. Taking two projection angles as an example, they are respectively based on DSA. α The preoperative two-dimensional projection image DRR obtained by projecting the scanning angle α α and according to DSA β The preoperative two-dimensional projection image DRR obtained by projecting the scanning angle β β .

[0032] S3 involves registering the preoperative two-dimensional projection image with the corresponding intraoperative two-dimensional image to obtain image transformation relationship data. Specifically, this step refers to the registration of the DSA image. α and DRR α Image transformation relationship data T is obtained through registration. α and DSA β and DRR β Image transformation relationship data T is obtained through registration. β That is, there is image transformation relationship data T between each intraoperative two-dimensional image and the corresponding preoperative two-dimensional projection image.

[0033] Although the scanning angle of the intraoperative image was used in step S2, the preoperative projection image obtained directly is not entirely equivalent to the intraoperative image because the condition of the surgical subject differs between the intraoperative and preoperative states. In other words, DSA... α With DRR α Similarity, and DSA β With DRR β The similarity is not high enough to allow for direct fusion.

[0034] The so-called registration in this application refers to extracting corresponding feature points (or key points) from the preoperative projection image and the intraoperative image, performing registration operations using the feature information of these feature points, obtaining image transformation relationship data T, and using T to process the preoperative projection image to obtain a result that is sufficiently close to the intraoperative image. The registration process is the process of obtaining T.

[0035] There are various types of feature points, selection methods, and registration calculation methods. Feature points must be identifiable in both images; specifically, they can be bones, blood vessels, or other visualizeable human tissues or organs. It should be noted that intraoperative images can be understood as either the X-ray image image1 with contrast agent injected, or the X-ray image image2 without contrast agent injected. The former can visualize blood vessels and bones, while the latter can visualize bones. Intraoperative images can also be understood as image2 - image1 = image3, which only includes vascular images.

[0036] S4, process the preoperative two-dimensional projection image using image transformation relationship data to obtain at least two corresponding registered two-dimensional images. Specifically, use the image transformation relationship data T obtained in step S3. α For DRR α Perform line transformation processing and denote the resulting registered 2D image as DRR. α+ Using image transformation relational data T α For DRR β Perform line transformation processing and denote the resulting registered 2D image as DRR. β+ DRR β+ With DSA β With higher similarity, DRR α+ With DSA α They have a higher degree of similarity.

[0037] S5 involves fusing the registered 2D image with the corresponding intraoperative 2D image to obtain at least two fused 2D images, each including at least vascular images and images of vascular interventional surgical instruments. Specifically, this refers to fusing the DRR image with the corresponding intraoperative 2D image. α+ With DSA α To perform fusion, and to integrate DRR β+ With DSA β To perform fusion. Due to DRR α+ and DRR β+ All images are from CTA, which inevitably contains vascular images, while DSA... α and DSA β The image itself contains images of vascular interventional surgical instruments; therefore, the superimposed result of the two images simultaneously contains both of these crucial images. In a preferred embodiment, the DRR can be analyzed first. α+ With DSA α Similarity and DRR β+ With DSA β The similarity is judged. If it is higher than the threshold, fusion is performed. Otherwise, the doctor can be prompted to change the angle of scanning the intraoperative two-dimensional image and repeat steps S1-S5.

[0038] For ease of description, DRR will be used. β+ With DSA β The result of the fusion is denoted as DRR. β++ DRR α+ With DSA α The result of the fusion is denoted as DRR. α++ DRR β++ and DRR α++ Both include the same scanning objects (blood vessels and vascular interventional devices), but the scanning / imaging angles are different.

[0039] S6, based on the positions of the vascular interventional surgical instrument images in at least two fused two-dimensional images, determine the three-dimensional position information of the vascular interventional surgical instrument in the preoperative three-dimensional image coordinate system. Specifically, in DRR... β++ and DRR α++ The process involves identifying vascular interventional devices and determining their coordinates in an image. This can involve identifying the entire vascular interventional device or identifying key parts or areas of the device, such as the anterior end.

[0040] Taking the guidewire tip as an example, DRR α++ The guidewire tip coordinates identified in the image are marked as M1. α (x1, y1), these coordinates represent the straight line L1 (x = x1, y = y1) after rotation α in the 3D view of the preoperative 3D image (CTA); DRR β++ The guidewire tip coordinates identified in the image are marked as M1. β (x2, y2) represents the straight line L2 (x = x2, y = y2) after β rotation in the 3D view of the preoperative 3D image (CTA). The intersection of the two straight lines L1 and L2 corresponding to the original CTA coordinates is the position of the guidewire tip in the original CTA 3D image (x3, y3, z3). Referring to the example of the guidewire tip, the position of any point on the vascular interventional device in the original CTA 3D image can be calculated.

[0041] If more scanning angles are used to obtain more fused 2D images, other calculation methods can be used to calculate the 3D coordinates of the same point in 3D space based on the 2D coordinates of the same point in all fused 2D images.

[0042] S7. Based on the three-dimensional position information, a three-dimensional image of the vascular interventional surgical instrument is generated in the preoperative three-dimensional image. After calculating the position of any point on the vascular interventional instrument in the original CTA 3D image, an image can be generated at that position to represent the three-dimensional vascular interventional surgical instrument.

[0043] According to the image generation method provided in the embodiments of the present invention, at least two projection images are obtained by projecting a preoperative three-dimensional image according to the scanning angles of at least two intraoperative two-dimensional images. Both projection images can clearly display blood vessels and accurately reflect the projection of the three-dimensional structure on the two-dimensional plane, making the vascular structure more accurate and displaying other tissues. The projection images are registered with the corresponding intraoperative two-dimensional images to obtain a registered image with a sufficiently high similarity. The registered image is then fused with the intraoperative two-dimensional image, which can clearly display blood vessels and their surrounding tissues while displaying the vascular interventional surgical instruments. Based on the two-dimensional position of the vascular interventional surgical instruments in the at least two two-dimensional images, their three-dimensional position in the preoperative three-dimensional image is calculated. Finally, an image of the vascular interventional instruments is generated in the preoperative three-dimensional image. This result can enhance the surgeon's spatial cognition of tissue structure, intuitively see the state of the three-dimensional human body structure and interventional instruments in the three-dimensional human body structure, and thus more accurately understand the positional relationship between blood vessels and interventional instruments. Applying this solution to interventional surgery planning and navigation can improve the efficiency of interventional surgery.

[0044] In one embodiment, the movement of the interventional surgical instrument in an actual blood vessel can also be simulated by image simulation, displaying the movement trajectory of the interventional surgical instrument fed back by the positioning device onto a preoperative 3D image. In this embodiment, a positioning system for the interventional surgical instrument is required to locate and track the tip of the instrument. The positioning system includes a position sensor and a position acquisition device. The position sensor is mounted on the interventional surgical instrument, and the position acquisition device is used to determine the position information of the tip of the interventional surgical instrument based on the position sensor.

[0045] Specifically, position sensors and position acquisition devices can be devices that locate based on electromagnetic signals. The position sensor can be an electromagnetic sensor and a magnetic field generating device. The electromagnetic sensor is set at the front end of the guidewire / conduit. The electromagnetic sensor generates electromagnetic induction in the magnetic field, and the electromagnetic induction signal indicates its spatial position. The position sensor can also be a device that locates based on optical signals. The position sensor can be a positioning optical fiber and an optical signal transceiver. The front end of the optical fiber is set at the front end of the guidewire / conduit, or it can be integrated into the optical fiber. The optical signal reflected and refracted by the optical fiber indicates its spatial position.

[0046] With the help of the positioning system described above, the following operations can be performed after step S7:

[0047] S8A, obtain the position information of the tip of the vascular interventional surgical instrument in the positioning coordinate system;

[0048] S9A updates the position of the three-dimensional vascular interventional surgical instrument image in the preoperative three-dimensional image based on the mapping relationship between the positioning coordinate system and the preoperative three-dimensional image coordinate system.

[0049] Specifically, the orientation of the positioning coordinate system is corrected according to the orientation of the bed before the operation, and the mapping between the positioning coordinate system and the preoperative three-dimensional image coordinate system is established during the operation. The navigation coordinate system is the CTA coordinate system.

[0050] In step S7, given the initial position O(x0,y0,z0) of any point on the guidewire / catheter, according to the mapping relationship G between the positioning coordinate system and the preoperative three-dimensional image coordinate system, when the guidewire moves L1 along the direction theta1, the corresponding coordinate represented by the guidewire in the preoperative three-dimensional image coordinate system can be moved L2 along theta2.

[0051] S(theta2, L2) = G*S1(theta1, L1), where S represents the displacement of the guidewire / catheter image in the fused two-dimensional image in the preoperative three-dimensional image coordinate system, and S1 represents the actual displacement of the guidewire / catheter instrument in the positioning coordinate system.

[0052] More specifically, in one embodiment, the guidewire as a whole or near its front end is a hollow structure, and a slender electromagnetic sensor is disposed in the hollow structure of the guidewire. The distance between the electromagnetic sensor and the front end of the guidewire is a fixed and known parameter. Based on the position and orientation information of the electromagnetic sensor, as well as the known parameter, the position information of the front end of the guidewire can be calculated, thereby allowing the position of the guidewire to be displayed and updated at least in the preoperative three-dimensional image.

[0053] In an optional embodiment, the position of the catheter can also be displayed and updated simultaneously. In this embodiment, no electromagnetic sensor is installed in the catheter. The vascular interventional surgical instrument can calculate and report the distance difference between the tip of the guidewire and the tip of the catheter based on the amount of guidewire and catheter delivery. Based on the position information and orientation information of the guidewire tip and the distance difference, the position of the catheter tip can be calculated, and the position of the guidewire can be displayed and updated in the fused two-dimensional image.

[0054] According to the above embodiments, by using a positioning system and image simulation, the guidewire / catheter in the three-dimensional image moves accordingly to the movement of the actual instrument, reducing intraoperative X-ray scanning operations and radiation dose.

[0055] In one embodiment, the rotation of the interventional surgical instrument in the actual blood vessel can also be simulated by displaying the rotational motion of the interventional surgical instrument fed back by the positioning device onto the preoperative three-dimensional image through image simulation.

[0056] Specifically, the following operations can be performed after step S5:

[0057] S8B, acquire rotational motion information of vascular interventional surgical instruments. In some embodiments, the interventional surgical robot can control the rotation of long, flexible instruments such as guidewires and catheters within blood vessels. The rotation can be controlled by human intervention or automatically performed by directional rotation, bidirectional rotation (counterclockwise rotation, clockwise rotation), constant speed rotation, variable speed rotation, etc. Therefore, the rotational motion information that the robot can provide feedback can be rotation direction and rotation speed.

[0058] S9B, the three-dimensional vascular interventional surgical instrument image is rotated based on the rotational motion information to simulate the rotational motion of the vascular interventional surgical instrument. The instrument image formed in step S7 is a three-dimensional model. By injecting the actual rotational motion information of the vascular interventional surgical instrument into the three-dimensional model, the three-dimensional model can perform the same rotational motion, such as rotating in the same direction or at the same speed. This allows the surgeon to see the actual rotational state of the vascular interventional instrument in the human body in the three-dimensional image, thereby avoiding surgical risks and improving the safety of vascular interventional surgery.

[0059] Regarding the scanning angle in step S2 above, in cases where the scanning angle cannot be directly read from the intraoperative medical imaging equipment (DSA equipment), in an optional embodiment, the scanning angle of the equipment can be obtained in the following way:

[0060] S21, Acquire intraoperative scene images captured by multi-camera equipment, including intraoperative imaging equipment and operating table;

[0061] S22, a three-dimensional coordinate system is constructed based on the fixed axis of the operating table. The structural features of the intraoperative imaging equipment are extracted from the intraoperative scene image, and the scanning angle is determined according to the position of the structural features in the three-dimensional coordinate system.

[0062] Specifically, if it is not possible to obtain the tilt angle data of the DSA device in real time during the operation, an optical positioning device can be used for assistance. Specifically, it can be a multi-camera device. A three-dimensional coordinate system is constructed with the operating table as the fixed axis, the fixed features of the DSA device are extracted, and the position of the C-arm or the table of the DSA device is tracked to calculate the offset angle in three directions during exposure.

[0063] Regarding the image registration in step S3 above, in one embodiment, blood vessel images in the image are used as feature points for registration, and step S3 further includes the following operations:

[0064] S31A: Acquire a first intraoperative two-dimensional image at the same scanning angle, without contrast agent injection, and a second intraoperative two-dimensional image with contrast agent injection. Taking scanning angle α as an example, in this embodiment, contrast agent is needed intraoperatively to acquire DSA. α The image, specifically, is an X-ray image obtained by scanning without the use of contrast agents.α Then, maintaining the current scanning position and angle, contrast agent is injected into the blood vessel, and another X-ray image is obtained. α The only difference between the two images is image1. α The outline of the blood vessels is unclear or even invisible, image2 α The outline of the blood vessels is clearly visible.

[0065] S32A: A two-dimensional vascular image is obtained based on the first and second intraoperative two-dimensional images. The two-dimensional vascular image (image3) is obtained by subtracting the two images. α Only vascular images were present.

[0066] S33A performs registration between preoperative two-dimensional projection images and two-dimensional vascular images. Specifically, it refers to the registration of image3... α and DRR α The two images are registered, and the affine matrix T is calculated so that DRR... α Through T α It can be transformed into image3 α .

[0067] In this embodiment, selecting blood vessels as the basis for registration can improve the accuracy of the registration results. As described above, DRR... α The data comes from preoperative CTA, which contains other tissues besides blood vessels, such as bone, so DRR is also necessary. α Processing is required before calculation can be performed. Specifically, step S33A may include the following operations:

[0068] S33A1 removes the background from preoperative two-dimensional projection images, retaining only the vascular image. Due to DRR... α The vascular images in the image are clear enough, and the vascular features are sufficiently distinct, making it relatively easy to extract blood vessels (remove background) using machine vision algorithms. Additionally, considering that CT and DSA devices may have different resolutions, DRR needs to be checked before registration. α and image3 α These two images can be adjusted to the same resolution using algorithms such as linear interpolation.

[0069] S33A2 uses preset scale parameters to construct Gaussian difference scale space images of preoperative two-dimensional projection images and two-dimensional vascular images after background removal.

[0070] For ease of description, the DRR after removing the background will be used. α Noted as DRR α- (x, y), let the two-dimensional blood vessel image be denoted as image3. α The Gaussian difference-scaled spatial image of these two images (x, y) is denoted as DRR.α (x,y,σ), image3(x,y,σ), where σ is a preset scale parameter, DRR α (x,y,σ)=DRR α (x,y)*G(x,y,σ), image3 α (x,y,σ)=image3 α (x,y)*G(x,y,σ), where G(x,y,σ) is the Gaussian smoothing kernel function.

[0071] S33A3 involves scanning extreme points in the Gaussian difference-of-scale image and retaining those with contrast exceeding a contrast threshold as vascular feature points. Specifically, in DRR... α (x,y,σ) and image3 α- After scanning the extreme points in (x, y, σ) to generate a Gaussian difference-of-scale spatial image, each sampling point is scanned and compared with its surrounding 26 pixels (8 pixels in the neighborhood and 9*2 pixels in the adjacent layers above and below) to determine whether it is an extreme point. The local extreme points found in this way are the coarse vascular feature points (key points) of the image.

[0072] S33A4: A feature region of a preset size is selected centered on each blood vessel feature point. This feature region is divided into multiple sub-blocks. Seed points are determined based on the histogram of the directional gradients of the sub-blocks. The direction values ​​of the seed points are then used to determine the descriptors of the corresponding blood vessel feature points. Specifically, after selecting the coarse blood vessel feature points of the image, a difference algorithm is used to determine the position and scale of key points. Then, low-contrast extreme points are removed, and the Hessian matrix is ​​used to remove edge response interference caused by Gaussian difference operations, thereby optimizing the feature point detection results.

[0073] In addition to its coordinate values ​​(planar position and scale), the feature vector of a blood vessel feature point also needs to have its orientation value determined by the gradient direction of its neighboring pixels. For example, taking a 16×16 pixel region centered on the blood vessel feature point, dividing this region into 4×4 sub-blocks, and calculating the orientation gradient histogram of each sub-block, yields a seed point. Each feature point consists of 4×4 seed points, and each seed point is divided into 8 directions, thus forming a 4×4×8 dimensional feature vector as the descriptor of the corresponding feature point. This vector possesses rotation invariance, scale invariance, and other properties.

[0074] S33A5, using descriptors of vascular feature points in preoperative two-dimensional projection images and two-dimensional vascular images, determines mutually matching vascular feature points. Specifically, this can be based on image3. α Descriptors of feature points in DRR α-The descriptors of the feature points in the data are used to calculate the distance (such as Euclidean distance, Hamming distance, etc.), and the matching is determined based on the distance, thereby identifying several pairs of vascular feature points.

[0075] S33A6 randomly selects multiple sets of matching points from mutually matched vascular feature points, and calculates an affine matrix using descriptors and binary images as image transformation relationship data. In practical applications, at least three sets of matching points need to be randomly selected. The calculated affine matrix is ​​used to translate, rotate, and stretch the image.

[0076] Using affine matrices to apply DRR α The content is processed by performing transformations such as translation, rotation, and stretching. The result of the transformation is DRR. α+ It can be used with DSA α To integrate.

[0077] For DRR β and DSA β The same process can be followed as described above, and will not be repeated here.

[0078] This embodiment extracts local features from the two-dimensional projection image and the two-dimensional blood vessel image. Even if there are only a few blood vessel images in the image, a large number of blood vessel feature points can be extracted. The descriptors of these blood vessel feature points have rich information content, thereby enabling fast and accurate matching of blood vessel feature points in the two images. The affine matrix generated in this way is used to transform the image, which remains unchanged by image rotation, scaling, and brightness changes, and also maintains a certain degree of stability against viewpoint changes, affine transformations, and noise. Regarding the image registration in step S3 above, in one embodiment, the skeletal image in the image is used as the feature points for registration. Step S3 further includes the following operations:

[0079] S31B, segmenting the bone image from the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image, respectively, to obtain the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image. Using DRR... α and DSA α For example, both CT and X-ray scans can visualize bone, but in this embodiment, no contrast agent is needed, so only one intraoperative two-dimensional image is required. Due to DRR... α (from CTA) and DSA α The skeletal images in the image are clear enough and the skeletal features are distinct enough that it is relatively easy to extract the skeleton from the image using machine vision algorithms.

[0080] S32B performs registration between preoperative and intraoperative two-dimensional skeletal images. This embodiment selects the skeleton as the basis for registration, performing image registration without the aid of contrast agents. This reduces the adverse effects of contrast agents and high-dose radiation on patients and doctors, and improves the efficiency of image fusion.

[0081] Similar to step S33A above, step S32B may include the following operations:

[0082] S32B1 constructs Gaussian difference-of-scale spatial images of preoperative and intraoperative two-dimensional skeleton images using preset scale parameters. For ease of description, the preoperative two-dimensional skeleton image is denoted as DRR. αb (x,y), the intraoperative two-dimensional skeletal image is denoted as DSA. αb The Gaussian difference scale-space image of at least two images (x,y) is denoted as DRR. αb (x,y,σ), DSA αb (x,y,σ), where σ is a preset scaling parameter, DRR αb (x,y,σ)=DRR αb (x,y)*G(x,y,σ), DSA αb (x,y,σ)=DSA αb (x,y)*G(x,y,σ), where G(x,y,σ) is the Gaussian smoothing kernel function.

[0083] S32B2 scans extreme points in the Gaussian difference-of-scale image, retaining extreme points with contrast above a contrast threshold as skeletal feature points. Specifically, in DRR... αb (x,y,σ) and DSA αb After scanning the extreme points in (x, y, σ) to generate a Gaussian difference-of-scale image, each sampling point is scanned and compared with its surrounding 26 pixels (8 pixels in the neighborhood and 9*2 pixels in the adjacent layers above and below) to determine whether it is an extreme point. The local extreme points found in this way are the coarse skeletal feature points (keypoints) of the image.

[0084] S32B3: A feature region of a preset size is selected centered on each of the stated skeletal feature points. This feature region is divided into multiple sub-blocks. Seed points are determined based on the histogram of the directional gradients of the sub-blocks. The direction values ​​of the seed points are then used to determine the descriptors of the corresponding skeletal feature points. Specifically, after selecting the coarse skeletal feature points of the image, a difference algorithm is used to determine the position and scale of the key points. Then, low-contrast extreme points are removed, and the Hessian matrix is ​​used to remove edge response interference caused by the difference of Gaussians operation, thereby optimizing the feature point detection results.

[0085] In addition to their coordinate values ​​(planar position and scale), the feature vectors of skeletal feature points also need to have their orientation values ​​determined by the gradient directions of the neighboring pixels. A 16×16 pixel region is taken centered on the blood vessel feature point, and this region is divided into 4×4 sub-blocks. The orientation gradient histogram of each sub-block is calculated to obtain a seed point. Each feature point consists of 4×4 seed points, and each seed point is divided into 8 directions, thus forming a 4×4×8 dimensional feature vector as the descriptor of the corresponding feature point. This vector possesses rotation invariance, scale invariance, and other properties.

[0086] S32B4, using descriptors of bone feature points in preoperative and intraoperative two-dimensional bone images, determines mutually matching bone feature points. Specifically, this can be based on DRR. αb Descriptors of feature points in DSA αb The descriptors of the feature points in the data are used to calculate the distance (such as Euclidean distance, Hamming distance, etc.), and the matching is determined based on the distance, thereby identifying several pairs of vascular feature points.

[0087] S32B5: Randomly select multiple sets of matching point pairs from the mutually matched skeletal feature points, and calculate the affine matrix using descriptors and binary images as image transformation relationship data. In practical applications, at least 3 sets of matching points need to be randomly selected. The calculated affine matrix is ​​used to translate, rotate, and stretch the image.

[0088] Using affine matrices to apply DRR α The content is processed by performing transformations such as translation, rotation, and stretching. The result of the transformation is DRR. α+ It can then be integrated with DSA.

[0089] For DRR β and DSA β The same process can be followed as described above, and will not be repeated here.

[0090] This embodiment extracts local features from preoperative and intraoperative two-dimensional skeletal images. Even if there are only a few skeletal images in the images, a large number of skeletal feature points can be extracted. The descriptors of these skeletal feature points have rich information content, thereby enabling fast and accurate matching of skeletal feature points in the two images. The affine matrix generated in this way is used to transform the image, which remains unchanged by image rotation, scaling, and brightness changes, and also maintains a certain degree of stability against viewpoint changes, affine transformations, and noise.

[0091] Furthermore, in the case of adopting the above two embodiments of steps S31A-S33A or S31B-S32B, in step S5, image similarity can be calculated based on the number of group matching points, the affine matrix, and the binary image, and it can be determined whether the similarity is higher than a threshold. The binary image refers to the image similarity to the DRR (Digital Resonance Map). α+ and DRR β+ The image after binarization, and the DSA α and DSA β The image after binarization. If the similarity is higher than the threshold, the registered 2D image is fused with the intraoperative 2D image; if the similarity is lower than the threshold, the user can be prompted to confirm whether to perform image fusion, or the process can return to step S1 to obtain intraoperative 2D images from different scanning angles and repeat steps S2-S3.

[0092] Regarding the calculation of image similarity, a neural network model can be used as an example. Specifically, the number of matching points, the degree of deviation indicated by the affine matrix (translation, rotation, deformation), and the degree of overlap between the two binary images are taken as input data, and the neural network model outputs the image similarity.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a processFigure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An image generation method for use in vascular interventional surgery, characterized in that, include: Acquire a preoperative three-dimensional image and an intraoperative two-dimensional image obtained from at least two different scanning angles, wherein the preoperative three-dimensional image contains at least a vascular image, the intraoperative two-dimensional image contains at least an image of a vascular interventional surgical instrument, and the angle difference between any two of the at least two different scanning angles is greater than an angle threshold. The preoperative three-dimensional image is projected according to the at least two different scanning angles to obtain at least two corresponding preoperative two-dimensional projection images; The preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image are registered to obtain image transformation relationship data; The preoperative two-dimensional projection image is processed using the image transformation relationship data to obtain at least two corresponding registered two-dimensional images; The registered two-dimensional image is fused with the corresponding intraoperative two-dimensional image to obtain at least two fused two-dimensional images, each of which includes at least a vascular image and an image of vascular interventional surgical instruments. Based on the position of the vascular interventional surgical instrument images in the at least two fused two-dimensional images, the three-dimensional position information of the vascular interventional surgical instrument in the preoperative three-dimensional image coordinate system is determined. Based on the three-dimensional position information, a three-dimensional vascular interventional surgical instrument image is formed in the preoperative three-dimensional image.

2. The image generation method according to claim 1, characterized in that, After generating three-dimensional images of vascular interventional surgical instruments, the process also includes: Obtain the position information of the tip of the vascular interventional surgical instrument in the positioning coordinate system; Based on the mapping relationship between the positioning coordinate system and the preoperative three-dimensional image coordinate system, the position of the three-dimensional vascular interventional surgical instrument image in the preoperative three-dimensional image is updated using the position information; Or it may also include: Obtain rotational motion information of vascular interventional surgical instruments; The three-dimensional vascular interventional surgical instrument image is rotated according to the rotational motion information to simulate the rotational motion of the vascular interventional surgical instrument.

3. The image generation method according to claim 1, characterized in that, Registration of the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image further includes: Acquire the first intraoperative two-dimensional image under the same scanning angle without contrast agent injection, and the second intraoperative two-dimensional image under the same scanning angle with contrast agent injection; Two-dimensional vascular images were obtained based on the first intraoperative two-dimensional image and the second intraoperative two-dimensional image. The background is removed from the preoperative two-dimensional projection image, leaving only the vascular image; Using preset scale parameters, Gaussian difference scale space images of the preoperative two-dimensional projection image and the two-dimensional vascular image after removing the background are constructed; In the Gaussian difference scale space image, extreme points are scanned respectively, and extreme points with contrast higher than the contrast threshold are retained as vascular feature points. Each of the blood vessel feature points is centered on a feature region of a preset size. The feature region is divided into multiple sub-blocks. Seed points are determined based on the orientation gradient histogram of the sub-blocks. The orientation values ​​of the seed points are used to determine the descriptors of the corresponding blood vessel feature points. Using the descriptors of the vascular feature points in the preoperative two-dimensional projection image and the two-dimensional vascular image, the mutually matching vascular feature points are determined; Multiple sets of matching points are randomly selected from the mutually matched vascular feature points, and the affine matrix is ​​calculated using the descriptor and binary image as the image transformation relationship data.

4. The image generation method according to claim 1, characterized in that, Registration of the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image further includes: The bone image is segmented from the preoperative two-dimensional projection image and the corresponding intraoperative two-dimensional image to obtain the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image. Using preset scale parameters, Gaussian difference scale space images of the preoperative two-dimensional skeletal image and the intraoperative two-dimensional skeletal image are constructed; In the Gaussian difference scale space image, extreme points are scanned respectively, and extreme points with contrast higher than the contrast threshold are retained as skeletal feature points. Each of the aforementioned skeletal feature points is centered on a feature region of a preset size. The feature region is divided into multiple sub-blocks. Seed points are determined based on the orientation gradient histogram of the sub-blocks. The orientation values ​​of the seed points are used to determine the descriptors of the corresponding skeletal feature points. Using the descriptors of the bone feature points in the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image, mutually matching bone feature points are determined; Multiple sets of matching point pairs are randomly selected from the mutually matched skeletal feature points, and the affine matrix is ​​calculated using the descriptor and the binary image as the image transformation relationship data.

5. The image generation method according to claim 3 or 4, characterized in that, Fusing the registered two-dimensional image with the corresponding intraoperative two-dimensional image further includes: The image similarity is calculated based on the number of matching points in the group, the affine matrix, and the binary image, and it is determined whether the similarity is higher than a threshold. If the similarity is higher than a threshold, the registered two-dimensional image is fused with the corresponding intraoperative two-dimensional image.

6. The image generation method according to claim 1, characterized in that, Before projecting the preoperative three-dimensional image according to the at least two different scanning angles, the method further includes: Acquire intraoperative scene images captured by multi-view camera equipment, including intraoperative imaging equipment and operating table; A three-dimensional coordinate system is constructed based on the fixed axis of the operating table. The structural features of the intraoperative imaging equipment are extracted from the intraoperative scene image, and the scanning angle is determined based on the position of the structural features in the three-dimensional coordinate system.

7. The image generation method according to any one of claims 1-6, characterized in that, The preoperative three-dimensional image is a preoperative CTA image, and the intraoperative two-dimensional image is an intraoperative DSA image.

8. The image generation method according to any one of claims 1-6, characterized in that, The images of the vascular interventional surgical instruments include catheter images and / or guidewire images.

9. An image generation device, characterized in that, include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the processor to perform the image generation method according to any one of claims 1-8.

10. A vascular interventional surgical system, characterized in that, include: The image generation device of claim 9, and the positioning system of the vascular interventional surgical instrument; wherein the positioning system includes a position sensor and a position acquisition device, the position sensor is disposed on the vascular interventional surgical instrument, and the position acquisition device is used to determine the position information of the front end of the vascular interventional surgical instrument based on the position sensor.

Images