Dsa image and ct image registration method, system, storage medium and electronic device

Abstract

The application provides a DSA image and CT image registration method, system, storage medium and electronic equipment. The method comprises the following steps: acquiring a preoperative CT image, inputting the CT image into a first image processing model to obtain a three-dimensional rib model; acquiring an intraoperative DSA image; determining an initial value of camera pose parameters based on the DSA image and the three-dimensional rib model, the camera being used for projecting the three-dimensional rib model onto a two-dimensional plane; rendering the three-dimensional rib model based on the initial value of the camera pose parameters to obtain a two-dimensional rendering image; updating the camera pose parameters based on the DSA image and the two-dimensional rendering image to obtain a final value of the camera pose parameters; and adjusting the registration of the three-dimensional rib model and the DSA image based on the final value of the camera pose parameters and displaying in real time. Thus, high-precision image registration is achieved, which is conducive to improving the safety and success rate of surgery.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, storage medium, and electronic device for registering DSA and CT images. Background Technology

[0002] Transcatheter aortic valve implantation (TAVI) is an interventional procedure that involves inserting a catheter through the femoral artery to deliver an artificial heart valve to the aortic valve region and open it, thereby completing the implantation and restoring valve function. During the procedure, real-time DSA (Digital Subtraction Angiography) images serve as the sole source of information for surgical navigation, guiding the surgeon's movements. However, DSA images, being two-dimensional, have limitations; it is difficult for surgeons to assess the spatial relationship between the valve and surrounding tissues, necessitating frequent manipulation of the DSA device during the procedure to obtain images from different angles.

[0003] Currently, this field utilizes preoperative CT (Computed Tomography) images to register and fuse intraoperative DSA images with preoperative CT images, helping doctors to observe the surgical process more intuitively. However, registering DSA images with CT images is difficult and inefficient, making it hard to provide highly accurate surgical navigation, thus affecting the safety and success rate of the surgery. Summary of the Invention

[0004] In view of the above, this application provides a training method for an apical chamber segmentation model, an apical chamber segmentation method, an apical chamber segmentation device, a storage medium, and an electronic device to solve at least one problem existing in the background art.

[0005] In a first aspect, one embodiment of this application provides a method for registering subtraction angiography DSA images with CT images, the method comprising:

[0006] Acquire preoperative CT images and input the CT images into a first image processing model to obtain a three-dimensional rib model;

[0007] Acquire intraoperative DSA images;

[0008] The initial values ​​of the camera pose parameters are determined based on the DSA image and the three-dimensional rib model, and the camera is used to project the three-dimensional rib model onto a two-dimensional plane;

[0009] Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered to obtain a two-dimensional rendered image;

[0010] Based on the DSA image and the 2D rendered image, the camera pose parameters are updated to obtain the final values ​​of the camera pose parameters;

[0011] Based on the final values ​​of the camera pose parameters, the three-dimensional rib model is adjusted to be registered with the DSA image and displayed in real time.

[0012] In conjunction with the first aspect of this application, in an optional embodiment, rendering the three-dimensional rib model based on the initial values ​​of the camera pose parameters to obtain a two-dimensional rendered image; and updating the camera pose parameters based on the DSA image and the two-dimensional rendered image to obtain the final values ​​of the camera pose parameters, includes:

[0013] Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered in a differentiable manner to obtain the two-dimensional rendered image.

[0014] Based on the DSA image and the two-dimensional rendered image, calculate the mean square error between pixels;

[0015] Using the mean squared error as the loss function, the gradient of the loss function with respect to the camera pose parameters is calculated using the backpropagation algorithm;

[0016] The camera pose parameters are updated based on the gradient.

[0017] Repeat the above steps for n iterations until the number of iterations reaches a threshold or the mean square error reaches a threshold, to obtain the final value of the camera pose parameters; where n≥1 and n is an integer.

[0018] In conjunction with the first aspect of this application, in an optional embodiment, after acquiring the intraoperative DSA image, the method further includes: binarizing the DSA image to form a binarized image.

[0019] In conjunction with the first aspect of this application, in an optional embodiment, rendering the three-dimensional rib model based on the initial values ​​of the camera pose parameters to obtain a two-dimensional rendered image includes:

[0020] Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered and binarized to obtain the two-dimensional rendered image, which is a binarized image.

[0021] In conjunction with the first aspect of this application, in an optional embodiment, determining the initial values ​​of camera pose parameters based on the DSA image and the three-dimensional rib model includes: a user manually adjusting the registration of the three-dimensional rib model with the DSA image through a visualization window to determine the initial values ​​of the camera pose parameters.

[0022] In conjunction with the first aspect of this application, in an optional embodiment, adjusting the registration of the three-dimensional rib model with the DSA image includes:

[0023] The pixel values ​​of the three-dimensional rib model and the pixel values ​​of the DSA image are weighted so that the pixel values ​​of the overlapping parts of the three-dimensional rib model and the DSA image are less than or equal to 255.

[0024] Secondly, embodiments of this application provide a DSA image and CT image registration system, characterized in that the system comprises:

[0025] The image acquisition module is used to acquire preoperative CT images and intraoperative DSA images;

[0026] The first image processing module is used to process the CT image to obtain a three-dimensional rib model;

[0027] A camera pose parameter determination module is used to determine the initial values ​​of camera pose parameters based on the DSA image and the three-dimensional rib model, wherein the camera is used to project the three-dimensional rib model onto a two-dimensional plane;

[0028] The rendering module is used to render the three-dimensional rib model based on the initial values ​​of the camera pose parameters to obtain a two-dimensional rendered image.

[0029] The camera pose parameter update module is used to update the camera pose parameters based on the DSA image and the two-dimensional rendered image to obtain the final value of the camera pose parameters.

[0030] An image registration module is used to adjust the registration of the three-dimensional rib model with the DSA image based on the final value of the camera pose parameters;

[0031] The interactive module is used to perform real-time visualization of the registration between the three-dimensional rib model and the DSA image.

[0032] In conjunction with the second aspect of this application, in an optional embodiment, the rendering module and the camera pose parameter update module are specifically used for:

[0033] Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered in a differentiable manner to obtain the two-dimensional rendered image.

[0034] Based on the DSA image and the two-dimensional rendered image, calculate the mean square error between pixels;

[0035] Using the mean squared error as the loss function, the gradient of the loss function with respect to the camera pose parameters is calculated using the backpropagation algorithm;

[0036] The camera pose parameters are updated based on the gradient.

[0037] Repeat the above steps for n iterations until the number of iterations reaches a threshold or the mean square error reaches a threshold, to obtain the final value of the camera pose parameters; where n≥1 and n is an integer.

[0038] In conjunction with the second aspect of this application, in an optional embodiment, the system further includes a second image processing module for binarizing the DSA image to form a binarized image.

[0039] In conjunction with a second aspect of this application, in an optional embodiment, the rendering module is specifically used for:

[0040] Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered and binarized to obtain the two-dimensional rendered image, which is a binarized image.

[0041] In conjunction with the second aspect of this application, in an optional embodiment, the interactive module is further configured to allow a user to manually adjust the registration of the three-dimensional rib model with the DSA image through a visual window;

[0042] The camera pose parameter determination module is specifically used to: determine the initial value of the camera pose parameter when the user manually adjusts the registration of the three-dimensional rib model with the DSA image.

[0043] In conjunction with the second aspect of this application, in an optional embodiment, the interaction module is further configured to perform weighted processing on the pixel values ​​of the three-dimensional rib model and the pixel values ​​of the DSA image, so that the pixel values ​​of the overlapping portion of the three-dimensional rib model and the DSA image being registered are less than or equal to 255.

[0044] Thirdly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method described in any of the first aspects above.

[0045] Fourthly, embodiments of this application provide an electronic device, the electronic device comprising:

[0046] processor;

[0047] Memory used to store executable instructions for a computer;

[0048] The processor is configured to execute the computer-executable instructions to implement the method described in any one of the first aspects above.

[0049] The DSA image and CT image registration method, system, storage medium, and electronic device provided in this application construct a three-dimensional model from preoperative CT images and register it with DSA images in real time during surgery. This avoids frequent operation of the DSA device during surgery, reduces patient radiation dose and economic burden, and improves surgical efficiency. By segmenting the rib structure from preoperative CT images and establishing a three-dimensional rib model, and registering and locating the ribs with DSA images based on rib shape, it avoids the differences between preoperative and intraoperative vascular morphology and the dependence on the clarity of blood vessels in DSA images when registering based on vascular shape. This method can improve registration stability and accuracy, reduce patient radiation dosage and contrast agent usage, and enhance robustness. By rendering a three-dimensional rib model to obtain a two-dimensional rendered image, the final values ​​of camera pose parameters are determined based on the two-dimensional rendered image and the DSA image. This effectively overcomes the problem of accuracy difference in the registration of three-dimensional and two-dimensional images. By iteratively updating the camera pose parameters, the difference between the two-dimensional rendered image and the DSA image is gradually reduced. The final values ​​of the camera pose parameters are then used to project the three-dimensional rib model, thereby achieving high-precision image registration, which helps to improve the safety and success rate of the surgery.

[0050] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0052] Figure 1 This is a schematic diagram of the system structure of the DSA image and CT image registration method and the DSA image and CT image registration system that can be applied to the embodiments of this application;

[0053] Figure 2 A flowchart illustrating the DSA image and CT image registration method provided in this application embodiment;

[0054] Figure 3 A schematic diagram of a 2D rendered image and a DSA image rendered based on the initial values ​​of camera pose parameters;

[0055] Figure 4 This is a schematic diagram of the 2D rendered image and the DSA image generated during the iteration process;

[0056] Figure 5 This is a schematic diagram of the structure of the DSA image and CT image registration system provided in the embodiments of this application;

[0057] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] To make the technical solutions and beneficial effects of the present invention more apparent and understandable, the technical solutions in the embodiments of this application are clearly and completely described below by listing specific examples. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] To make the technical solution and beneficial effects of the present invention more apparent and understandable, a detailed description is provided below by listing specific embodiments. The accompanying drawings are not necessarily drawn to scale, and local features may be enlarged or reduced to more clearly show the details of the local features; unless otherwise defined, the technical and scientific terms used herein have the same meanings as those in the technical field to which this application pertains.

[0060] The embodiments in this application are not exhaustive, but merely illustrative of some embodiments, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0061] In each embodiment of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0062] The terminology used in the embodiments of this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this disclosure.

[0063] In the embodiments of this application, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun after the article can be understood as either a singular expression or a plural expression.

[0064] In the embodiments of this application, "multiple" refers to two or more.

[0065] In some embodiments, the terms “at least one of”, “one or more”, “a plurality of”, “multiple”, etc., may be used interchangeably.

[0066] In some embodiments, the notation "at least one of A and B", "A and / or B", "A in one case, B in another", "A in one case, B in another", etc., may include the following technical solutions depending on the situation: in some embodiments, A (A is executed regardless of B); in some embodiments, B (B is executed regardless of A); in some embodiments, execution is selected from A and B (A and B are selectively executed); in some embodiments, A and B (both A and B are executed). The same applies when there are more branches such as A, B, C, etc.

[0067] In some embodiments, the notation "A or B" may include the following technical solutions, depending on the situation: in some embodiments, A (execution of A regardless of B); in some embodiments, B (execution of B regardless of A); in some embodiments, execution is selected from A and B (A and B are selectively executed). The same applies when there are more branches such as A, B, C, etc.

[0068] The prefixes "first," "second," etc., used in the embodiments of this application are merely for distinguishing different descriptive objects and do not impose restrictions on the position, order, priority, value, or content of the descriptive objects. The description of the descriptive objects should be found in the claims or the context of the embodiments, and the use of prefixes should not constitute unnecessary restrictions. For example, if the descriptive object is a "field," the ordinal numbers preceding "field" in "first field" and "second field" do not restrict the position or order of the "fields." "First" and "second" do not restrict whether the "fields" they modify are in the same message, nor do they restrict the order of "first field" and "second field." Similarly, if the descriptive object is a "level," the ordinal numbers preceding "level" in "first level" and "second level" do not restrict the priority between "levels." Furthermore, the value of the descriptive object is not limited by ordinal numbers and can be one or more. For example, in "first device," the value of "device" can be one or more. Furthermore, the objects modified by different prefixes can be the same or different. For example, if the object being described is "device", then "first device" and "second device" can be the same device or different devices, and their types can be the same or different. Similarly, if the object being described is "information", then "first information" and "second information" can be the same information or different information, and their content can be the same or different.

[0069] In some embodiments, “including A,” “containing A,” “for indicating A,” and “carrying A” can be interpreted as directly carrying A or indirectly indicating A.

[0070] In some embodiments, terms such as “…”, “determine…”, “in the case of…”, “when…”, “when…”, “if…”, etc. can be used interchangeably.

[0071] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”.

[0072] In some embodiments, devices, etc., can be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. Terms such as “device”, “equipment”, “circuit”, “network element”, “node”, “function”, “unit”, “section”, “system”, “network”, “chip”, “chip system”, “entity”, and “subject” can be used interchangeably.

[0073] like Figure 1 As shown, the system includes a server 10 and terminal devices 20 that communicate with the server 10. The server 10 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 20 can be various electronic devices with displays, including but not limited to desktop computers, laptops, smartphones, and tablets.

[0074] In this embodiment of the application, the DSA image and CT image registration method can be implemented by server 10. The DSA image and CT image registration system can be set in server 10. Terminal device 20 receives various instructions from users and displays the registration results accordingly to the user.

[0075] Figure 2 A flowchart illustrating the DSA image and CT image registration method provided in this application embodiment. Figure 2 As shown, the DSA image and CT image registration method of this application embodiment includes:

[0076] S201: Acquire preoperative CT images and input the CT images into the first image processing model to obtain a three-dimensional rib model;

[0077] S202: Acquire intraoperative DSA images;

[0078] S203: Determine the initial values ​​of camera pose parameters based on DSA images and a 3D rib model. The camera is used to project the 3D rib model onto a 2D plane.

[0079] S204: Based on the initial values ​​of the camera pose parameters, render the 3D rib model to obtain a 2D rendered image;

[0080] S205: Based on the DSA image and the 2D rendered image, update the camera pose parameters to obtain the final values ​​of the camera pose parameters;

[0081] S206: Based on the final values ​​of the camera pose parameters, adjust the registration of the 3D rib model with the DSA image and display it in real time.

[0082] By constructing a 3D model from preoperative CT images and registering it with DSA images in real time during surgery, frequent operation of the DSA equipment is avoided, reducing patient radiation dosage and economic burden, and improving surgical efficiency. By segmenting the rib structure from preoperative CT images and establishing a 3D rib model, registration and localization based on rib shape and DSA images avoids the problems of preoperative and intraoperative differences in vascular morphology, reliance on vascular clarity in DSA images, and low vascular segmentation accuracy encountered when registering based on vascular shape. This reduces segmentation difficulty, improves robustness, reduces patient radiation dosage and contrast agent usage, and improves registration accuracy. By rendering a 2D rendered image from the 3D rib model, the final values ​​of camera pose parameters are determined based on the 2D rendered image and DSA image, effectively overcoming the accuracy difference problem in 3D-to-2D image registration. Through multiple iterations of updating the camera pose parameters, the difference between the 2D rendered image and the DSA image is gradually reduced. The final values ​​of the camera pose parameters are used to project the 3D rib model, achieving high-precision image registration, which contributes to improving surgical safety and success rate.

[0083] It should be understood that, although Figure 1 The steps are displayed sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; moreover, these steps are not necessarily executed sequentially. This application does not exclude the possibility that at least two steps are completed at the same time or are completed alternately. The execution order between steps should still be determined according to the specific logical relationship.

[0084] Digital subtraction angiography (DSA) is a medical imaging technique used to observe blood vessels and their blood flow. Its core principle is to reduce interference from background tissue through digital subtraction, thereby clearly displaying lesions within the blood vessels. The specific steps include: taking a base image (mask image); injecting a contrast agent into the patient's blood vessels, followed by taking a filling image; and then subtracting the mask image from the filling image using a computer to obtain the vascular image. Therefore, in DSA images, vascular structures, due to the presence of contrast agent, efficiently absorb X-rays, resulting in the highest contrast. Bone structures have a strong ability to attenuate X-rays, resulting in moderate contrast, and are usually presented as shadows after subtraction. Soft tissue structures, such as muscle and fat, have a weak ability to attenuate X-rays, resulting in the lowest contrast, and are completely removed during the subtraction process.

[0085] Considering the characteristics of DSA images, current CT / DSA registration techniques typically use vascular structures as the registration benchmark. Specifically, after acquiring preoperative CT images, vascular structures are segmented from the CT images to create a three-dimensional vascular model, which is then registered with the vascular structures in the DSA images. However, several problems exist in this process: First, blood vessels may undergo morphological changes due to lesions (stenosis, aneurysm) or physiological states (contraction, dilation), and the amount of contrast agent input can also affect the presentation of vascular structures in DSA images, potentially leading to differences between the morphology of vascular structures in the preoperative three-dimensional vascular model and those in the intraoperative DSA images, making accurate registration difficult. Second, the small size and complex structure of blood vessels make segmentation from CT images challenging and inaccurate, resulting in an inaccurate three-dimensional vascular model. Third, the numerous and complex branches of blood vessels also increase the difficulty of registering the three-dimensional vascular model with the DSA images.

[0086] Based on this, this application proposes a CT / DSA registration method based on skeletal structure, specifically a CT / DSA registration method based on ribs. Utilizing the significantly higher contrast of skeletal structures in CT images compared to vascular structures, this method reduces segmentation difficulty, improves segmentation accuracy, and ensures the accuracy of the established 3D model. Skeletal structures are less affected by pathological and physiological factors, are essentially static, and have strong anti-interference capabilities. The skeletal structure in preoperative CT images is largely consistent with the skeletal structure in intraoperative DSA images, resulting in more stable registration. Furthermore, skeletal structures are relatively simple, relatively large, and have clear geometric contours, making it easier to extract feature points or edges, which simplifies the registration algorithm and improves computational efficiency. Therefore, by leveraging the relatively fixed anatomical positions of skeletal and vascular structures, the registration result of skeletal structure indirectly maps the location of blood vessels, reducing errors caused by the deformation of the blood vessels themselves. Of course, this registration method can also be applied to other scenarios requiring skeletal structure registration, such as lung intervention and tumor localization, and is not limited to vascular intervention scenarios.

[0087] First, step S201 is executed to acquire preoperative CT images. These CT images are then input into the first image processing model to obtain a three-dimensional rib model. Compared to segmenting vascular structures from CT images, ribs, as skeletal structures, have higher contrast, simpler structure, and clearer contours in CT images, making segmentation easier and more accurate.

[0088] Before inputting the CT image into the first image processing model, the method may further include preprocessing the CT image. Preprocessing may include denoising and contrast enhancement. Thus, denoising removes random noise from the CT image, improving its quality; contrast enhancement increases the grayscale difference between the rib image and the surrounding environment in the CT image, thereby enhancing the rib's boundary contour and making it stand out from the surrounding tissue, which is beneficial for distinguishing the rib image from the environment.

[0089] Optionally, the first image processing model includes an image segmentation model and a 3D reconstruction algorithm; the image segmentation model is used to segment CT images to obtain rib segmentation results, and the 3D reconstruction algorithm is used to construct a 3D rib model based on multiple rib segmentation results. The image segmentation model may include a machine learning model, which can be pre-trained and used to segment rib structures from input CT images. The image segmentation model and the 3D reconstruction algorithm can be any suitable model and algorithm well known in the art, and this application does not limit them.

[0090] In some embodiments, after performing step S202, the method may further include processing the three-dimensional rib model to make the file format of the three-dimensional rib model a mesh format.

[0091] Understandably, the 3D rib model established in step S201 by segmenting CT images is typically in medical image format, such as DICOM tomographic data or voxel segmentation masks. This type of data exists in the form of voxels (3D pixels) or point clouds, lacking obvious surface topology and geometric description, making it difficult to directly use for computer graphics rendering and projection calculations. Therefore, by processing the 3D rib model to present its file format as a mesh format, specifically a standardized polygon mesh (triangular or quadrilateral mesh), containing geometric information of points, edges, and faces, it can be directly read by the computer graphics engine for rendering, providing an image foundation for subsequent rendering.

[0092] In this embodiment, the three-dimensional rib model is specifically an STL (Stereo Lithography) format file. However, this application does not exclude the possibility that the three-dimensional rib model may be in other formats, such as PLY (Polygon) or OBJ (Object) formats. This embodiment only uses the STL format as an example for illustration.

[0093] Next, step S202 is performed to acquire intraoperative DSA images. In the subsequent registration process, the rib structure is used as the registration reference. The rib structure in the 3D rib model is registered with the rib structure in the DSA image, eliminating the need to utilize the vascular structure in the DSA image. Therefore, the amount of contrast agent used during the procedure can be appropriately reduced, minimizing harm to the patient.

[0094] It should be noted that step S202 can be executed before step S201 or simultaneously with step S201, and this application does not limit this.

[0095] After acquiring intraoperative DSA images, the method may further include preprocessing the DSA images. Preprocessing may include denoising and contrast enhancement. Thus, denoising removes random noise from the DSA images, improving their quality; contrast enhancement increases the grayscale difference between the rib images and surrounding images (vascular images, soft tissue images) in the DSA images, thereby enhancing the rib's boundary contour and making it stand out from the surrounding tissues, which is beneficial for distinguishing the rib images from the surrounding images.

[0096] Optionally, after acquiring the intraoperative DSA images, the method further includes: binarizing the DSA images to form binary images. In DSA images, the contrast of vascular structures is usually higher than that of bone and soft tissue structures, and the outline of bone structures is less clear. By binarizing the DSA images, the outline of bone structures can be made clearer, enhancing their visibility and improving the accuracy of subsequent registration. Binarization simplifies the image to black and white, reducing the amount of information in the image, making subsequent image processing steps more efficient and reducing computational complexity.

[0097] In some embodiments, binarization of DSA images can be achieved using a second image processing model. The second image processing model includes a deep learning model, which can be pre-trained and used to convert the input DSA image into a binarized image. This method is suitable for image processing in complex backgrounds and has strong adaptability. The second image processing model can be any suitable deep learning model well-known in the art, and this application does not limit it.

[0098] In some other embodiments, binarization of DSA images can be achieved using a binarization algorithm. Specifically, a pixel threshold is set and input into the binarization algorithm. The pixel values ​​of pixels in the image are compared with the threshold; pixels with values ​​higher than the threshold are set to 255, and pixels with values ​​lower than the threshold are set to 0. This allows for rapid binarization.

[0099] Next, step S203 is executed, which determines the initial values ​​of the camera pose parameters based on the DSA image and the three-dimensional rib model. The camera is used to project the three-dimensional rib model onto a two-dimensional plane.

[0100] The primary function of a camera is to project objects or scenes in three-dimensional space onto a two-dimensional plane, thereby generating a two-dimensional image. Camera pose parameters describe the camera's position and orientation in three-dimensional space, determining its position and orientation relative to a reference coordinate system. Camera pose parameters typically include camera position parameters, describing the camera's position in three-dimensional space, usually represented by X, Y, and Z coordinate values; and camera attitude parameters, describing the camera's orientation and tilt, usually represented by rotation matrices, Euler angles, or quaternions. By continuously adjusting the camera pose parameters—that is, adjusting the camera's position and orientation relative to the DSA image coordinate system—the position and angle of the 3D rib model in the DSA image coordinate system are adjusted, allowing the 3D rib model to be registered with the DSA image, or in other words, aligning the rib structure in the 3D rib model with the rib structure in the DSA image.

[0101] Optionally, the initial values ​​of the camera pose parameters are determined based on the DSA image and the 3D rib model. This includes: the user manually adjusting the registration between the 3D rib model and the DSA image through a visualization window to determine the initial values ​​of the camera pose parameters. Both the 3D rib model and the DSA image can be presented in the visualization window. The 3D rib model is available for manual adjustment by the user, who can touch to move (change camera position parameters) and rotate (change camera pose parameters). During this process, when the rib structure in the 3D rib model closely approximates the rib structure in the DSA image, the current camera pose parameters are used as the initial values. Obtaining the initial values ​​of the camera pose parameters through manual registration can reduce the difference between the initial and final values ​​of the camera pose parameters, reduce the number of subsequent iterations, and improve efficiency and accuracy.

[0102] Users manually adjust the registration between the 3D rib model and the DSA image through a visual window to determine the initial values ​​of the camera pose parameters. Specifically, this includes: users manually adjusting the 3D rib model through the visual window and obtaining the rotation angle and translation of the 3D rib model in real time; users confirming the registration between the 3D rib model and the DSA image and obtaining the current rotation angle and translation; and calculating the initial values ​​of the camera pose parameters based on the current rotation angle and translation.

[0103] When manually adjusting the 3D rib model, its movement can be understood as the movement of an intensifier used to acquire DSA images. Therefore, the rotation angle can be interpreted as the intensifier's projection angle. In other words, calculating the initial values ​​of the camera pose parameters based on the current rotation angle and translation amount can be understood as calculating the initial values ​​of the camera pose parameters based on the current projection angle and translation amount. The intensifier projects X-rays onto the object according to the set projection angle value, thus achieving imaging. Projection angle values ​​can include LAO (Left Anterior Oblique), RAO (Right Anterior Oblique), CRA (Cranial), and CAU (Caudal). LAO refers to the intensifier being positioned above and to the left of the patient, with X-rays projected from the patient's right rear to the left front. RAO refers to the intensifier being positioned above and to the right of the patient, with X-rays projected from the patient's left rear to the right front. CRA refers to the intensifier being close to the patient's head, with X-rays projected from the patient's feet to the head. CAU refers to the intensifier being close to the patient's feet, with X-rays projected from the patient's head to the feet. LAO and RAO are usually combined into a single angle value, which can be understood as the projection angle relative to the x-axis in a coordinate system. CRA and CAU are usually combined into a single angle value, which can be understood as the projection angle relative to the y-axis in a coordinate system. During angiography, by adjusting the LAO / RAO and CRA / CAU angle values, DSA images of different parts of the structure are projected.

[0104] The projection angle value can include a first angle value θ1 and a second angle value θ2, where the first angle value θ1 represents the projection angle of the camera relative to the x-axis, and the second angle value θ1 represents the projection angle of the camera relative to the y-axis; the initial values ​​of the camera pose parameters are calculated based on the current projection angle value and the current translation amount, and can include:

[0105] Construct the default rotation matrix The default rotation matrix is ​​the identity matrix, indicating that no rotation has occurred at the default position;

[0106] Construct the default translation vector The default translation vector is the zero vector, indicating that the position is at the default location and no translation has occurred.

[0107] Construct a rotation matrix, R = R1·R2, where R1 is the first rotation matrix. This represents a rotation about the x-axis relative to the default position, where R2 is the second rotation matrix. This indicates a rotation about the y-axis relative to the default position;

[0108] Construct translation vector x represents the translation amount along the x-axis relative to the default position, y represents the translation amount along the y-axis relative to the default position, and z represents the translation amount along the z-axis relative to the default position. z is a fixed value, usually twice the width of the 3D rib model.

[0109] Constructing the homogeneous transformation matrix This represents the camera pose parameters.

[0110] The first angle value can be an LAO / RAO angle value, and the second angle value can be a CRA / CAU angle value. The homogeneous transformation matrix is ​​the complete representation of the camera pose parameters, which can describe the complete pose of the camera relative to the world coordinate system (or other reference coordinate system). Understandably, during surgery, the camera only changes relative to the x-axis and y-axis, so there is no need to consider the rotation angle and translation relative to the z-axis. Alternatively, the projection angle value also includes a third angle value θ3, which represents the projection angle of the camera relative to the z-axis, relative to the third rotation matrix. Specifically, for the identity matrix, the formula R = R1·R2 still holds true.

[0111] Optionally, adjusting the registration between the 3D rib model and the DSA image includes: weighting the pixel values ​​of the 3D rib model and the pixel values ​​of the DSA image so that the pixel values ​​of the overlapping portion of the 3D rib model and the DSA image registration are less than or equal to 255.

[0112] Understandably, the range of displayable pixel values ​​for an image is 0-255. When two images are superimposed, the pixel values ​​of the overlapping portion may exceed 255 and cannot be displayed. Therefore, the pixel values ​​of the 3D rib model and the DSA image are weighted, assigning appropriate weights to both, so that the pixel values ​​of the overlapping portion are less than or equal to 255, thus achieving visualization. In this embodiment, the weight of the pixel values ​​of the 3D rib model is 0.5, and the weight of the pixel values ​​of the DSA image is 0.5.

[0113] Next, step S204 is executed, where the 3D rib model is rendered based on the initial values ​​of the camera pose parameters to obtain a 2D rendered image. This process can be understood as simulating the imaging process of a DSA device, using perspective projection to map the 3D model onto a 2D plane to obtain a 2D rendered image. Therefore, the 2D rendered image can also be understood as simulating a DSA image.

[0114] In some embodiments, the 3D rib model is rendered based on initial values ​​of camera pose parameters to obtain a 2D rendered image. This includes: rendering and binarizing the 3D rib model based on the initial values ​​of camera pose parameters to obtain a 2D rendered image, which is a binarized image. This results in a clearer rib structure in the image. A suitable rendering library can be selected to simultaneously perform rendering and binarization.

[0115] In some embodiments, after obtaining the two-dimensional rendered image, the method further includes: binarizing the two-dimensional rendered image. After rendering the three-dimensional rib model using a rendering library to obtain the two-dimensional rendered image, the two-dimensional rendered image is then post-processed to obtain a binarized image.

[0116] Next, step S205 is executed to update the camera pose parameters based on the DSA image and the 2D rendered image to obtain the final values ​​of the camera pose parameters.

[0117] In some embodiments, steps S204 and S205 may include:

[0118] Based on the initial values ​​of the camera pose parameters, the 3D rib model is rendered in a differentiable manner to obtain a 2D rendered image.

[0119] Calculate the mean square error between pixels based on DSA images and 2D rendered images;

[0120] Using mean squared error as the loss function, the gradient of the loss function with respect to the camera pose parameters is calculated using the backpropagation algorithm;

[0121] Update camera pose parameters based on gradients;

[0122] Repeat the above steps for n iterations until the number of iterations reaches a threshold or the mean square error reaches a threshold, to obtain the final values ​​of the camera pose parameters; where n≥1 and n is an integer.

[0123] Thus, through multiple iterations, the gap between the DSA image and the 2D rendered image is gradually narrowed, and the final values ​​of the camera pose parameters are obtained, achieving accurate registration.

[0124] Understandably, traditional rendering, i.e., non-differentiable rendering, typically employs discretized hard decisions during the rendering stage, such as rounding coordinates to the nearest pixel position. This leads to non-differentiability of operations, gradient breakage, and the inability to backpropagate gradients using the chain rule. If iteration is required, methods such as mesh search and genetic iteration are usually needed, but the direction of each iteration is uncertain and may not be along the optimal direction, resulting in poor iteration performance and low efficiency. Differentiable rendering, on the other hand, replaces discretized decisions with probabilistic or smoothing operations, ensuring that the mathematical expression of each rendering step is differentiable. The entire calculation path from the 3D model coordinates to the final pixel value is differentiable, allowing backpropagation to calculate gradients. In other words, it is possible to differentiate the changes in rendering parameters (geometry, camera pose, lighting parameters, etc.) and calculate the gradient of the rendering output relative to the rendering parameters. This enables automatic gradient calculation, and the direction of iterative optimization is controllable, achieving the desired iterative effect.

[0125] The process of performing differentiable rendering of a 3D rib model based on camera pose parameters to obtain a 2D rendered image is as follows:

[0126] I render =R(M, P) Formula (I)

[0127] Among them, I render For a 2D rendered image, R is the rendering function, M is the 3D model containing geometric information such as vertices and faces, and P is the camera pose parameter. The camera pose parameter P is specifically represented as a homogeneous transformation matrix, containing a rotation matrix R and a parallel vector t. It should be noted that, since it is differentiable rendering, the derivative of equation (I) can be taken to obtain the two-dimensional rendered image I. render The derivative with respect to the camera pose parameters. The mean square error between the pixels of the DSA image and the pixels of the 2D rendered image is specifically:

[0128]

[0129] Among them, I DSA Let N be the total number of pixels in the DSA image, and i be the i-th pixel in the image. The mean squared error is used as the loss function, and the loss function Loss relative to the 2D rendered image I can be calculated. render The derivative of .

[0130] The formula for calculating the gradient of the loss function with respect to the camera pose parameters is:

[0131]

[0132] Therefore, the loss function Loss relative to the two-dimensional rendered image I can be calculated using formula (II).render derivative Rendering image I using formula (I) render derivative with respect to camera pose parameters Then the derivative of the loss function with respect to the camera pose parameters is obtained. The gradient is passed from the pixel value to the camera pose parameter, indicating the optimization path of the camera pose parameter.

[0133] Please refer to Figure 3 The 3D rib model is rendered based on the initial values ​​of the camera pose parameters, with the iteration count (iter) at 0. The mean squared error (loss value) between the pixels of the 2D rendered image and the DSA image is calculated to be 52105.49. A significant difference between the 2D rendered image and the DSA image can be observed in the figure. Next, the camera pose parameters are updated using gradients, and the 3D rib model is rendered using the updated camera pose parameters to obtain a 2D rendered image. The loss function and its gradient with respect to the camera pose parameters are then calculated, and the camera pose parameters are updated again. These steps are repeated for multiple iterations until the number of iterations reaches a threshold. At this point, iteration stops, and the camera pose parameter with the smallest mean squared error during the iterations is retained as the final value, or the mean squared error reaches the threshold, and iteration stops, using the current camera pose parameter as the final value. Please refer to [reference needed]. Figure 4 At this point, the iteration count is 350, and the calculated mean square error (loss value) between the pixels of the 2D rendered image and the pixels of the DSA image is 1822.58, compared to... Figure 3 The mean square error is greatly reduced, and compared with Figure 3 and Figure 4 It can also be observed that the 2D rendered image is basically "aligned" with the DSA image.

[0134] In this embodiment, the rendering library is specifically PyTorch3d. PyTorch3d can perform differentiable rendering on mesh files and can directly render binarized images.

[0135] Finally, step S206 is executed, adjusting the registration between the 3D rib model and the DSA image based on the final values ​​of the camera pose parameters and displaying the results in real time. Thus, the 3D rib model is rotated and translated according to the camera pose parameters, ultimately aligning the rib structure in the 3D rib model with the rib structure in the DSA image, and the registration result is displayed in real time for the doctor's observation. Since the anatomical positions of ribs and blood vessels are relatively fixed, rib registration can indirectly map the position of blood vessels, facilitating the location of the surgical area and reducing errors caused by deformation of the blood vessels themselves.

[0136] Optionally, adjusting the registration between the 3D rib model and the DSA image further includes: weighting the pixel values ​​of the 3D rib model and the pixel values ​​of the DSA image so that the pixel values ​​of the overlapping portion of the 3D rib model and the DSA image are less than or equal to 255. This ensures that the overlapping portion can be displayed correctly.

[0137] This application also provides a DSA image and CT image registration system. Please refer to [link / reference]. Figure 5 The system includes:

[0138] The image acquisition module is used to acquire preoperative CT images and intraoperative DSA images;

[0139] The first image processing module is used to process the CT image to obtain a three-dimensional rib model;

[0140] A camera pose parameter determination module is used to determine the initial values ​​of camera pose parameters based on the DSA image and the three-dimensional rib model, wherein the camera is used to project the three-dimensional rib model onto a two-dimensional plane;

[0141] The rendering module is used to render the three-dimensional rib model based on the initial values ​​of the camera pose parameters to obtain a two-dimensional rendered image.

[0142] The camera pose parameter update module is used to update the camera pose parameters based on the DSA image and the two-dimensional rendered image to obtain the final value of the camera pose parameters.

[0143] An image registration module is used to adjust the registration of the three-dimensional rib model with the DSA image based on the final value of the camera pose parameters;

[0144] The interactive module is used to perform real-time visualization of the registration between the three-dimensional rib model and the DSA image.

[0145] Therefore, by constructing a 3D model from preoperative CT images and registering it with DSA images in real time during surgery, frequent operation of the DSA equipment during surgery is avoided, reducing the patient's radiation dose and economic burden, and improving surgical efficiency. By segmenting the rib structure from preoperative CT images and establishing a 3D rib model, registration and positioning based on the rib shape and DSA images can avoid the problems of differences in preoperative and intraoperative vascular morphology, reliance on the clarity of blood vessels in DSA images, and low vascular segmentation accuracy when registering based on vascular shape. This reduces segmentation difficulty, improves robustness, reduces the patient's radiation dose and contrast agent usage, and improves registration accuracy. By rendering the 3D rib model to obtain a 2D rendered image, the final values ​​of camera pose parameters are determined based on the 2D rendered image and the DSA image. This effectively overcomes the problem of accuracy difference when registering 3D and 2D images. By iteratively updating the camera pose parameters, the difference between the 2D rendered image and the DSA image is gradually reduced. The final values ​​of the camera pose parameters are used to project the 3D rib model, thereby achieving high-precision image registration, which is conducive to improving the safety and success rate of surgery.

[0146] In some embodiments, the system may further include a preprocessing module for preprocessing CT images and / or DSA images. Preprocessing may include denoising and contrast enhancement. Denoising removes random noise from the image, improving image quality; contrast enhancement increases the grayscale difference between the rib image and the surrounding environment, thereby enhancing the rib's boundary contour and making it stand out from the surrounding tissue, which helps distinguish the rib image from the environment.

[0147] A first image processing module is used to process the CT image to obtain a three-dimensional rib model. The first image processing module may include an image segmentation model and a three-dimensional reconstruction algorithm; the image segmentation model is used to segment the CT image to obtain rib segmentation results, and the three-dimensional reconstruction algorithm is used to construct a three-dimensional rib model based on multiple rib segmentation results. The image segmentation model may include a machine learning model, which can be pre-trained and used to segment the rib structure from the input CT image. The image segmentation model and the three-dimensional reconstruction algorithm can be any suitable model and algorithm well known in the art, and this application does not limit them.

[0148] In some embodiments, the system may further include a model processing module for processing the three-dimensional rib model to make the file format of the three-dimensional rib model a mesh format.

[0149] Understandably, 3D rib models are typically in medical image formats, such as DICOM tomographic data or voxel segmentation masks. This type of data exists in the form of voxels (3D pixels) or point clouds, lacking clear surface topology and geometric description, making it difficult to directly use for computer graphics rendering and projection calculations. Therefore, the 3D rib model is processed by a model processing module to present its file format as a mesh, specifically a standardized polygonal mesh (triangular or quadrilateral patches), containing geometric information of points, edges, and faces. This mesh can be directly read by computer graphics engines for rendering, providing the image foundation for subsequent rendering.

[0150] In this embodiment, the three-dimensional rib model is specifically an STL (Stereo Lithography) format file. However, this application does not exclude the possibility that the three-dimensional rib model may be in other formats, such as PLY (Polygon) or OBJ (Object) formats. This embodiment only uses the STL format as an example for illustration.

[0151] Optionally, the system further includes a second image processing module for binarizing the DSA image to form a binarized image.

[0152] In some embodiments, the second image processing module includes a deep learning model, which can be pre-trained and used to convert the input DSA image into a binarized image. This method is suitable for image processing in complex backgrounds and has strong adaptability. The second image processing model can be any suitable deep learning model well known in the art, and this application does not limit it.

[0153] In some other embodiments, the second image processing module includes a binarization algorithm. Specifically, a pixel threshold is set and input into the binarization algorithm, which compares the pixel values ​​of pixels in the image with the threshold. Pixel values ​​higher than the threshold are set to 255, and pixel values ​​lower than the threshold are set to 0. This allows for rapid binarization.

[0154] Optionally, the interactive module also allows users to manually adjust the registration of the 3D rib model with the DSA image through a visual window; the camera pose parameter determination module is specifically used to determine the initial values ​​of the camera pose parameters when the user manually adjusts the registration of the 3D rib model with the DSA image. Users can touch the 3D rib model to move (change camera position parameters) and rotate (change camera pose parameters). During this process, when the rib structure in the 3D rib model approaches the rib structure in the DSA image, the current camera pose parameters are used as the initial values. Obtaining the initial values ​​of the camera pose parameters through manual registration can reduce the difference between the initial and final values ​​of the camera pose parameters, reduce the number of subsequent iterations, and improve efficiency and accuracy.

[0155] Specifically, the camera pose parameter determination module is used to: acquire the rotation angle and translation of the 3D rib model in real time when the user manually adjusts the 3D rib model through the visualization window; acquire the current rotation angle and translation when the user confirms that the 3D rib model is registered with the DSA image; and calculate the initial values ​​of the camera pose parameters based on the current rotation angle and translation.

[0156] When manually adjusting the 3D rib model, the movement of the 3D rib model can be understood as the movement of the intensifier used to acquire DSA images. Therefore, the rotation angle value can be understood as the projection angle value of the intensifier. In other words, calculating the initial value of the camera pose parameters based on the current rotation angle value and the current translation amount can be understood as calculating the initial value of the camera pose parameters based on the current projection angle value and the current translation amount.

[0157] The projection angle value can include a first angle value θ1 and a second angle value θ2, where the first angle value θ1 represents the projection angle of the camera relative to the x-axis, and the second angle value θ1 represents the projection angle of the camera relative to the y-axis; the initial values ​​of the camera pose parameters are calculated based on the current projection angle value and the current translation amount, and can include:

[0158] Construct the default rotation matrix The default rotation matrix is ​​the identity matrix, indicating that no rotation has occurred at the default position;

[0159] Construct the default translation vector The default translation vector is the zero vector, indicating that the position is at the default location and no translation has occurred.

[0160] Construct a rotation matrix, R = R1·R2, where R1 is the first rotation matrix. This represents a rotation about the x-axis relative to the default position, where R2 is the second rotation matrix. This indicates a rotation about the y-axis relative to the default position;

[0161] Construct translation vector x represents the translation amount along the x-axis relative to the default position, y represents the translation amount along the y-axis relative to the default position, and z represents the translation amount along the z-axis relative to the default position. z is a fixed value, usually twice the width of the 3D rib model.

[0162] Constructing the homogeneous transformation matrix This represents the camera pose parameters.

[0163] The first angle value can be an LAO / RAO angle value, and the second angle value can be a CRA / CAU angle value. The homogeneous transformation matrix is ​​the complete representation of the camera pose parameters, which can describe the complete pose of the camera relative to the world coordinate system (or other reference coordinate system). Understandably, during surgery, the camera only changes relative to the x-axis and y-axis, so there is no need to consider the rotation angle and translation relative to the z-axis. Alternatively, the projection angle value also includes a third angle value θ3, which represents the projection angle of the camera relative to the z-axis, relative to the third rotation matrix. Specifically, for the identity matrix, the formula R = R1·R2 still holds true.

[0164] Optionally, the interaction module is also used to weight the pixel values ​​of the 3D rib model and the pixel values ​​of the DSA image so that the pixel values ​​of the overlapping parts of the 3D rib model and the DSA image are less than or equal to 255.

[0165] Understandably, the range of displayable pixel values ​​for an image is 0-255. When two images are superimposed, the pixel values ​​of the overlapping portion may exceed 255 and cannot be displayed. Therefore, the pixel values ​​of the 3D rib model and the DSA image are weighted, assigning appropriate weights to both, so that the pixel values ​​of the overlapping portion are less than or equal to 255, thus achieving visualization. In this embodiment, the weight of the pixel values ​​of the 3D rib model is 0.5, and the weight of the pixel values ​​of the DSA image is 0.5.

[0166] Optionally, the rendering module and the camera pose parameter update module are specifically used for:

[0167] Based on the initial values ​​of the camera pose parameters, the 3D rib model is rendered in a differentiable manner to obtain a 2D rendered image.

[0168] Calculate the mean square error between pixels based on DSA images and 2D rendered images;

[0169] Using mean squared error as the loss function, the gradient of the loss function with respect to the camera pose parameters is calculated using the backpropagation algorithm;

[0170] Update camera pose parameters based on gradients;

[0171] Repeat the above steps for n iterations until the number of iterations reaches a threshold or the mean square error reaches a threshold, to obtain the final values ​​of the camera pose parameters; where n≥1 and n is an integer.

[0172] Thus, through multiple iterations, the gap between the DSA image and the 2D rendered image is gradually narrowed, and the final values ​​of the camera pose parameters are obtained, achieving accurate registration.

[0173] The rendering module performs differentiable rendering of the 3D rib model based on camera pose parameters to obtain a 2D rendered image. The specific process is as follows:

[0174] I render =R(M, P) Formula (I)

[0175] Among them, I render For a 2D rendered image, R is the rendering function, M is the 3D model containing geometric information such as vertices and faces, and P is the camera pose parameter. The camera pose parameter P is specifically represented as a homogeneous transformation matrix, containing a rotation matrix R and a parallel vector t. It should be noted that, since it is differentiable rendering, the derivative of equation (I) can be taken to obtain the two-dimensional rendered image I. render The derivative with respect to the camera pose parameters.

[0176] In the camera pose parameter update module, the formula for calculating the mean square error between the pixels of the DSA image and the pixels of the 2D rendered image is:

[0177]

[0178] Among them, I DSA Let N be the total number of pixels in the DSA image, and i be the i-th pixel in the image. The mean squared error is used as the loss function, and the loss function Loss relative to the 2D rendered image I can be calculated. render The derivative of .

[0179] The formula for calculating the gradient of the loss function with respect to the camera pose parameters is:

[0180]

[0181] Therefore, the loss function Loss relative to the two-dimensional rendered image I can be calculated using formula (II). render derivative Rendering image I using formula (I) render derivative with respect to camera pose parameters Then the derivative of the loss function with respect to the camera pose parameters is obtained. The gradient is passed from the pixel value to the camera pose parameter, indicating the optimization path of the camera pose parameter.

[0182] Please refer to Figure 3 The 3D rib model is rendered based on the initial values ​​of the camera pose parameters, with the iteration count (iter) at 0. The mean squared error (loss value) between the pixels of the 2D rendered image and the DSA image is calculated to be 52105.49. The figure shows a significant difference between the 2D rendered image and the DSA image. Next, the camera pose parameters are updated using gradients, and the 3D rib model is rendered using the updated camera pose parameters to obtain the 2D rendered image. The loss function and its gradient with respect to the camera pose parameters are then calculated, and the camera pose parameters are updated again. These steps are repeated for multiple iterations until the number of iterations reaches a threshold. At this point, iteration stops, and the camera pose parameter with the smallest mean squared error during the iteration is retained as the final value, or the mean squared error reaches the threshold, and iteration stops, using the current camera pose parameter as the final value.

[0183] Please refer to Figure 4 At this point, the iteration count is 350, and the calculated mean square error (loss value) between the pixels of the 2D rendered image and the pixels of the DSA image is 1822.58, compared to... Figure 3 The mean square error is greatly reduced, and compared with Figure 3 and Figure 4 It can also be observed that the 2D rendered image is basically "aligned" with the DSA image.

[0184] In some embodiments, the rendering module is specifically used to: render and binarize the three-dimensional rib model based on the initial values ​​of the camera pose parameters to obtain the two-dimensional rendered image, which is a binarized image. This simultaneously achieves rendering and binarization.

[0185] In this embodiment, the rendering library is specifically PyTorch3d. PyTorch3d can perform differentiable rendering on mesh files and can directly render binarized images.

[0186] In some other embodiments, the system may further include a third image processing module for binarizing the two-dimensional rendered image. After rendering the three-dimensional rib model using a rendering library to obtain a two-dimensional rendered image, the two-dimensional rendered image is then post-processed to obtain a binarized image.

[0187] The image registration module is used to adjust the registration of the 3D rib model with the DSA image based on the final values ​​of the camera pose parameters. Thus, the 3D rib model is rotated and translated according to the camera pose parameters, ultimately aligning the rib structure in the 3D rib model with the rib structure in the DSA image. The registration result is displayed in real time through an interactive module for the doctor's observation. Since the anatomical positions of ribs and blood vessels are relatively fixed, rib registration can indirectly map the position of blood vessels, facilitating surgical area location and reducing errors caused by blood vessel deformation.

[0188] This application also provides a computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the DSA image and CT image registration method described above.

[0189] Embodiments of this application may be systems, methods, and / or computer program products. A computer program product may include a computer-readable storage medium on which computer-readable program instructions are loaded to cause a processor to implement various aspects of this application. The computer program product may be written in any combination of one or more programming languages ​​to perform operations of embodiments of this application. Programming languages ​​include object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code may execute entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0190] Computer-readable storage media can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A computer-readable storage medium is a tangible device capable of holding and storing instructions for use by an instruction execution device. A readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combinations thereof. The computer-readable storage medium as used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0191] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0192] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0193] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0194] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0195] This application also provides an electronic device. Figure 6 The figure shows a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown, the electronic device 600 includes: one or more processors 601 and a memory 602; the memory 602 stores computer-executable instructions; the processor 601 is used to execute the computer-executable instructions to implement the above-described DSA image and CT image registration method.

[0196] The processor 601 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0197] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1501 may execute the program instructions to implement the steps in the text recognition methods of the various embodiments of this application described above, and / or other desired functions.

[0198] In one example, the electronic device 600 may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0199] In addition, input devices may include, for example, a keyboard, a mouse, a microphone, etc. Output devices can output various information to the outside, and may include, for example, a monitor, speakers, a printer, and communication networks and their connected remote output devices, etc.

[0200] Of course, for the sake of simplicity, Figure 6 Only a portion of the components of the electronic device 600 relevant to this application are shown in this illustration; components such as buses and input / output interfaces are omitted. In addition, the electronic device 600 may include any other suitable components depending on the specific application.

[0201] It should be noted that the training method embodiment, apical chamber segmentation method embodiment, apical chamber segmentation device embodiment, computer-readable storage medium embodiment, and electronic device embodiment provided in this application belong to the same concept; the technical features in the technical solutions described in each embodiment can be arbitrarily combined without conflict.

[0202] It should be understood that the above embodiments are exemplary and are not intended to encompass all possible implementations included in the claims. Various modifications and changes can be made to the above embodiments without departing from the scope of this disclosure. Similarly, the various technical features of the above embodiments can be arbitrarily combined to form other embodiments of the present invention that may not be explicitly described. Therefore, the above embodiments only illustrate several implementations of the present invention and do not limit the scope of protection of this patent.

Claims

1. A method for registering subtraction angiography DSA images and CT images, characterized in that, The method includes: Acquire preoperative CT images and input the CT images into a first image processing model to obtain a three-dimensional rib model; Acquire intraoperative DSA images; The initial values ​​of the camera pose parameters are determined based on the DSA image and the three-dimensional rib model, and the camera is used to project the three-dimensional rib model onto a two-dimensional plane; Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered to obtain a two-dimensional rendered image; Based on the DSA image and the 2D rendered image, the camera pose parameters are updated to obtain the final values ​​of the camera pose parameters; Based on the final values ​​of the camera pose parameters, the three-dimensional rib model is adjusted to be registered with the DSA image and displayed in real time.

2. The DSA image and CT image registration method according to claim 1, characterized in that, The three-dimensional rib model is rendered based on the initial values ​​of the camera pose parameters to obtain a two-dimensional rendered image. Based on the DSA image and the 2D rendered image, the camera pose parameters are updated to obtain the final values ​​of the camera pose parameters, including: Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered in a differentiable manner to obtain the two-dimensional rendered image. Based on the DSA image and the two-dimensional rendered image, calculate the mean square error between pixels; Using the mean squared error as the loss function, the gradient of the loss function with respect to the camera pose parameters is calculated using the backpropagation algorithm; The camera pose parameters are updated based on the gradient. Repeat the above steps for n iterations until the number of iterations reaches a threshold or the mean square error reaches a threshold, to obtain the final value of the camera pose parameters; where n≥1 and n is an integer.

3. The DSA image and CT image registration method according to claim 2, characterized in that, After acquiring the DSA image during the procedure, the method further includes: performing binarization processing on the DSA image to form a binarized image.

4. The DSA image and CT image registration method according to claim 2, characterized in that, The process of rendering the three-dimensional rib model based on the initial values ​​of the camera pose parameters to obtain a two-dimensional rendered image includes: Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered and binarized to obtain the two-dimensional rendered image, which is a binarized image.

5. The DSA image and CT image registration method according to claim 1, characterized in that, Determining the initial values ​​of camera pose parameters based on the DSA image and the three-dimensional rib model includes: the user manually adjusting the registration of the three-dimensional rib model with the DSA image through a visualization window to determine the initial values ​​of the camera pose parameters.

6. The DSA image and CT image registration method according to claim 1 or 5, characterized in that, The adjustment of the registration between the three-dimensional rib model and the DSA image includes: The pixel values ​​of the three-dimensional rib model and the pixel values ​​of the DSA image are weighted so that the pixel values ​​of the overlapping parts of the three-dimensional rib model and the DSA image are less than or equal to 255.

7. A DSA image and CT image registration system, characterized in that, The system includes: The image acquisition module is used to acquire preoperative CT images and intraoperative DSA images; The first image processing module is used to process the CT image to obtain a three-dimensional rib model; A camera pose parameter determination module is used to determine the initial values ​​of camera pose parameters based on the DSA image and the three-dimensional rib model, wherein the camera is used to project the three-dimensional rib model onto a two-dimensional plane; The rendering module is used to render the three-dimensional rib model based on the initial values ​​of the camera pose parameters to obtain a two-dimensional rendered image. The camera pose parameter update module is used to update the camera pose parameters based on the DSA image and the two-dimensional rendered image to obtain the final value of the camera pose parameters. An image registration module is used to adjust the registration of the three-dimensional rib model with the DSA image based on the final value of the camera pose parameters; The interactive module is used to perform real-time visualization of the registration between the three-dimensional rib model and the DSA image.

8. The DSA image and CT image registration system according to claim 7, characterized in that, The rendering module and the camera pose parameter update module are specifically used for: Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered in a differentiable manner to obtain the two-dimensional rendered image. Based on the DSA image and the two-dimensional rendered image, calculate the mean square error between pixels; Using the mean squared error as the loss function, the gradient of the loss function with respect to the camera pose parameters is calculated using the backpropagation algorithm; The camera pose parameters are updated based on the gradient. Repeat the above steps for n iterations until the number of iterations reaches a threshold or the mean square error reaches a threshold, to obtain the final value of the camera pose parameters; where n≥1 and n is an integer.

9. The DSA image and CT image registration system according to claim 8, characterized in that, The system further includes a second image processing module for binarizing the DSA image to form a binary image.

10. The DSA image and CT image registration system according to claim 8, characterized in that, The rendering module is specifically used for: Based on the initial values ​​of the camera pose parameters, the three-dimensional rib model is rendered and binarized to obtain the two-dimensional rendered image, which is a binarized image.

11. The DSA image and CT image registration system according to claim 7, characterized in that, The interactive module is also used to allow users to manually adjust the registration of the three-dimensional rib model with the DSA image through a visual window; The camera pose parameter determination module is specifically used to: determine the initial value of the camera pose parameter when the user manually adjusts the registration of the three-dimensional rib model with the DSA image.

12. The DSA image and CT image registration system according to claim 7 or 11, characterized in that, The interaction module is further configured to perform weighted processing on the pixel values ​​of the three-dimensional rib model and the pixel values ​​of the DSA image, so that the pixel values ​​of the overlapping parts of the three-dimensional rib model and the DSA image are less than or equal to 255.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method described in any one of claims 1 to 6.

14. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store executable instructions for a computer; The processor is configured to execute the computer-executable instructions to perform the method described in any one of claims 1 to 6.

Images