Single perspective image and CT image registration method based on multi-modal joint optimization

Abstract

The application discloses a single perspective image and CT image registration method based on multi-modal joint optimization, which comprises the following steps: S1, obtaining a first processed image; S2, obtaining a second processed image; S3, generating a digital reconstructed radiograph; S4, calculating the normalized cross-correlation coefficient of the digital reconstructed radiograph and the second processed image, and converting it into a first function which can be minimized; S5, obtaining single-layer skin surface point cloud data; S6, constructing a second function which can be minimized according to the depth camera point cloud data and the single-layer skin surface point cloud data; S7, constructing a target function based on the first function and the second function, minimizing the target function by using a gradient descent optimization algorithm, and updating the pose transformation matrix step by step until the set condition is met, obtaining the minimized target function value, and finally obtaining the target pose transformation matrix of the preoperative three-dimensional image and the intraoperative two-dimensional perspective image; the application has the advantages of avoiding the need to shoot multiple perspective images, improving the registration efficiency, and saving the operation time.

Description

Technical Field

[0001] This invention belongs to the field of image registration method technology, specifically relating to a method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization. Background Technology

[0002] Surgical navigation systems based on preoperative CT images and intraoperative X-ray fluoroscopy images utilize the registration of X-ray and CT images to map the preoperative surgical plan to the intraoperative period, thereby achieving automatic surgical navigation and path guidance, which can greatly improve surgical accuracy and save surgical time.

[0003] Registration of X-ray fluoroscopy images and CT images typically employs an iterative optimization method based on digital reconstructed radiographs (DRR). This technique generates DRR from CT 3D volumetric data using a ray casting algorithm, establishes an objective function based on grayscale similarity, and iteratively adjusts spatial transformation parameters (including translation, rotation, and scaling degrees of freedom) using an optimization algorithm to maximize the objective function value. To address the issue of insufficient parameter constraints due to limited spatial information provided by single-view X-ray images, orthogonal dual-view (anteroposterior / lateral) imaging modes are often used in clinical practice. Establishing multi-plane constraint equations effectively improves the accuracy of 3D spatial registration.

[0004] However, traditional methods suffer from insufficient accuracy and robustness in registering single fluoroscopic images with CT images. Multiple fluoroscopic images are usually required to improve the registration effect, which not only increases the registration time but also prolongs the operation time. Therefore, a single fluoroscopic image and CT image registration method based on multimodal joint optimization is designed to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for registering single fluoroscopic images and CT images based on multimodal joint optimization, which solves the problems of insufficient accuracy and robustness in registering single fluoroscopic images and CT images, avoids the need to take multiple fluoroscopic images, improves registration efficiency, and saves surgical time.

[0006] The technical solution of the present invention is as follows:

[0007] A method for registering single fluoroscopic images and CT images based on multimodal joint optimization includes the following steps:

[0008] S1. Preoperative scanning of the patient's target area to obtain a preoperative three-dimensional image, which is then preprocessed to obtain the first processed image;

[0009] S2. Intraoperative scanning of the patient's target area to obtain intraoperative two-dimensional fluoroscopic images and depth camera point cloud data; preprocessing of the intraoperative two-dimensional fluoroscopic images to obtain a second processed image.

[0010] S3. Based on the patient's position relative to the two-dimensional fluoroscopic imaging device, obtain the pose transformation matrix between the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image, so as to transform the first processed image, and then generate a digitally reconstructed radiographic image through a ray projection algorithm.

[0011] S4. Calculate the normalized cross-correlation coefficient between the digitally reconstructed radiographic image and the second processed image, and convert it into a minimizeable first function;

[0012] S5. Threshold segmentation is performed on the preoperative three-dimensional image to obtain skin region image, and then point cloud data of single-layer skin surface is obtained.

[0013] S6. Transform the depth camera point cloud using the pose transformation matrix, and construct a second function that minimizes both the depth camera point cloud data and the single-layer skin surface point cloud data.

[0014] S7. Construct a target function based on the first function and the second function, and use the gradient descent optimization algorithm to minimize the target function. Update the pose transformation matrix one by one until the set conditions are met to obtain the minimized target function value, and finally obtain the target pose transformation matrix of the three-dimensional image and the two-dimensional perspective image.

[0015] Further, in step S5, threshold segmentation is performed on the preoperative three-dimensional image to obtain a skin region image, thereby acquiring single-layer skin surface point cloud data, including the following steps:

[0016] S51. The skin region image is obtained by segmenting the preoperative three-dimensional image based on a set threshold, and a morphological closing operation is performed on it to obtain a closed skin region image. The closed skin region image is then denoised using a mean filtering method.

[0017] S52. An edge detection operator is used to perform convolution operation on the denoised skin region image to obtain the initial edge image of the skin region. Then, non-maximum suppression is applied to it to extract the single-layer skin surface point cloud data.

[0018] Furthermore, the first processed image in step S1 is an image obtained by magnifying the original voxel values ​​of the skeletal region in the preoperative three-dimensional image by a set factor.

[0019] Furthermore, the preprocessing in step S2 includes the following steps:

[0020] S21. After smoothing the intraoperative two-dimensional fluoroscopic image, logarithmic mapping is performed to obtain the logarithmic mapping result. The negative image is generated by subtracting the logarithmic mapping result from the logarithmic value of the initial energy.

[0021] S22. The negative image is linearly normalized to a set range and adjusted to a set size by bilinear interpolation to obtain a second processed image.

[0022] Furthermore, after obtaining the depth camera point cloud through step S2 and the single-layer skin surface point cloud through step S5, the two are downsampled to obtain the downsampled depth camera point cloud and the downsampled skin surface point cloud.

[0023] Furthermore, step S4 specifically includes the following steps:

[0024] S41. Calculate the normalized cross-correlation coefficient between the digitally reconstructed radiographic image and the second processed image. The calculation formula is as follows:

[0025]

[0026] in, and , respectively, are the pixel mean values ​​of the second processed image and the digitally reconstructed radiographic image, <, > are the inner product, |·| is the second norm of the vector, and the value range of NCC is [0, 1];

[0027] S42. Construct a minimizeable first function based on the normalized cross-correlation coefficient in step S41, as follows:

[0028] L1 = 1 - NCC.

[0029] Furthermore, the minimizeable second function in step S6 is specifically a bidirectional nearest neighbor distance metric function between the depth camera point cloud data and the single-layer skin surface point cloud data, calculated as follows:

[0030]

[0031] Where P = {p i}, Q={q i For each point in P, find the nearest point in Q, calculate the squared distance and take the average. Repeat the same operation for each point in Q. The bidirectional nearest neighbor distance is the sum of these two.

[0032] Furthermore, in step S7, the target function constructed based on the first function and the second function is as follows:

[0033] Loss = α L1 + β L2

[0034] Loss=α(1-NCC(I Xray I DRR ))+βD charmfer (P, Q)

[0035] Where α and β are the weighting coefficients of the balance function, with α taking the value of 1 and β taking the value of 0.001.

[0036] Furthermore, the threshold value set in step S51 is selected based on the HU value of the skin, and the value range is [-50, 100].

[0037] Furthermore, the value range of the set multiple is [2.5, 4.5].

[0038] Furthermore, the setting range in step S22 is [0, 1], and the setting size is 128*128 or 64*64.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] 1. This invention introduces X-ray fluoroscopic images and depth camera point cloud data into a C-arm X-ray machine equipped with a depth camera. By adding depth camera point cloud data to the registration algorithm and extracting single-layer skin surface point cloud data from preoperative 3D images, it solves the problem of easily getting trapped in local optima caused by registration using only preoperative 3D images and intraoperative 2D fluoroscopic image data, thus improving the registration success rate.

[0041] 2. This invention constructs a first function and a second function related to the pose transformation matrix of preoperative 3D images and intraoperative 2D fluoroscopic images. It uses a gradient optimization descent algorithm to calculate the objective function composed of the first function and the second function. Finally, the values ​​of the first function, the second function, and the objective function reach the minimum value, and the pose transformation matrix is ​​no longer updated. The optimal target pose transformation matrix is ​​obtained, which ensures the accuracy of registration between preoperative 3D images and intraoperative 2D fluoroscopic images.

[0042] 3. This invention only requires registration of a single fluoroscopic image acquired during the operation with a CT image acquired before the operation, which solves the problem of long operation time caused by the need to take multiple fluoroscopic images in traditional methods. It also reduces the patient's exposure time to radiation during the operation, and ensures the accuracy and robustness of registration while optimizing the registration efficiency.

[0043] In summary, this invention solves the problems of insufficient accuracy and robustness in registering fluoroscopic images with CT images, and has the advantages of avoiding the need to take multiple fluoroscopic images, improving registration efficiency, and saving surgical time. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to the present invention;

[0045] Figure 2These are example images of the human body model of the present invention, wherein (a) is a CT image, (b) is a point cloud map of the skin surface of the human body model extracted using a skin estimation algorithm, and (c) is a point cloud map of a depth camera mounted on a C-arm machine;

[0046] Figure 3 The images are of the lumbar spine and sacral region of the present invention, wherein (a) is an X-ray fluoroscopic image, (b) is an X-ray fluoroscopic image after 2D preprocessing, and (c) is a DRR image generated using the transformation relation T after registration. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] like Figures 1 to 3 As shown, this invention proposes a method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization, including the following steps:

[0049] S1. Preoperative scanning of the patient's target area to obtain a preoperative three-dimensional image, which is then preprocessed to obtain the first processed image;

[0050] Specifically, the original voxel values ​​(i.e., HU values ​​of the bone region) in the preoperative three-dimensional image are magnified by a set factor to obtain the first processed image.

[0051] In this invention, voxel points with a HU value greater than 800 in the preoperative three-dimensional image are multiplied by a set factor.

[0052] In a specific embodiment of the present invention, the value range of the multiple is set to [2.5, 4.5], preferably 3.5.

[0053] In this invention, 3D preprocessing is performed, namely, voxel remapping, to improve the grayscale value of the skeletal region in the digitally reconstructed radiographic image generated based on the first processed image, enhance the skeletal structural features, and improve the distinguishability of image registration.

[0054] In this invention, the preoperative three-dimensional image is a CT image obtained by scanning the target area of ​​the patient using a CT scanner before surgery, such as... Figure 2 As shown in (a).

[0055] In a specific embodiment of the present invention, the target area for the patient is the lumbar spine and sacral region of the patient;

[0056] S2. Intraoperative scanning of the patient's target area to obtain intraoperative two-dimensional fluoroscopic images and depth camera point cloud data; preprocessing of the intraoperative two-dimensional fluoroscopic images to obtain the second processed image.

[0057] This invention acquires intraoperative two-dimensional fluoroscopic images of the patient's target area and point cloud data from the depth camera using a C-arm X-ray machine equipped with a depth camera. The intraoperative two-dimensional fluoroscopic images are then preprocessed to obtain a second processed image. Specifically, the invention includes the following steps:

[0058] S21. Smooth the intraoperative two-dimensional fluoroscopic images to suppress noise interference and reduce image detail differences, thereby improving the robustness of subsequent similarity calculations; then perform logarithmic mapping to obtain the logarithmic mapping result, and subtract the logarithmic mapping result from the logarithmic value of the initial energy to generate a negative image.

[0059] In this invention, a Gaussian filter with a kernel size of 5 and a standard deviation of 1.0 is selected to smooth the intraoperative two-dimensional images.

[0060] In this invention, a logarithmic mapping is performed to obtain a logarithmic mapping result. The logarithmic mapping result is then subtracted from the logarithmic value of the initial energy to generate a negative image. Specifically, the logarithmic value is calculated for each pixel value in the intraoperative two-dimensional fluoroscopic image to obtain a logarithmic mapping result, thereby enhancing the contrast of low grayscale areas (bone areas). The logarithmic value of the initial energy of the X-ray, 65536, is subtracted from the logarithmic value corresponding to each pixel value in the aforementioned intraoperative two-dimensional fluoroscopic image to generate a negative image, ensuring that the grayscale distribution of the image remains consistent with the subsequent digitally reconstructed radiographic image.

[0061] S22. Linearly normalize the negative image obtained in step S21 to a set range, and adjust it to a set size through bilinear interpolation to obtain the second processed image.

[0062] In a specific embodiment of the present invention, the linear normalization range of the negative image is set to [0, 1], and the size is set to 128*128 or 64*64. In order to ensure consistency with the subsequent digital reconstructed radiographic image (DRR) generated based on the preoperative three-dimensional image, the image resolution is unified, the amount of calculation is reduced, and the subsequent similarity matching step is adapted.

[0063] S3. Based on the patient's position relative to the two-dimensional fluoroscopic imaging device, obtain the pose transformation matrix between the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image, so as to transform the first processed image, and then generate a digitally reconstructed radiographic image through a ray projection algorithm.

[0064] The two-dimensional perspective imaging device in this invention is a C-arm X-ray machine (hereinafter referred to as a C-arm machine) equipped with a depth camera.

[0065] Specifically, the patient's position relative to the C-arm machine includes:

[0066] (1) The patient is in a prone position with the C-arm machine on the patient's right side;

[0067] (2) The patient is in a prone position with the C-arm machine on the patient's left side;

[0068] (3) The patient is supine with the C-arm machine on the patient's right side;

[0069] (4) The patient is supine with the C-arm machine on the patient's left side;

[0070] In this embodiment, based on the relative positional relationship between the patient and the C-arm machine, the initial value of the pose transformation matrix between the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image in step S3 is determined.

[0071] In a specific embodiment of the present invention, taking the patient lying prone and the C-arm machine on the patient's right side during fluoroscopy as an example, the initial value of the pose transformation matrix of the intraoperative two-dimensional fluoroscopic image relative to the preoperative three-dimensional image is:

[0072]

[0073] Where (cx, cy, cz) are the coordinates of the center point of the CT image, the initial value of the pose transformation matrix can be transformed into the form T0 = (-1.209, 1.209, -1.209, cx, cy, cz), where the first three elements are rotation vectors and the last three elements are translation vectors.

[0074] Similarly, when the patient is prone and the C-arm machine is on the patient's left side, the initial value of the pose transformation matrix of the intraoperative two-dimensional fluoroscopic image relative to the preoperative three-dimensional image is T0 = (1.209, -1.209, -1.209, cx, cy, cz).

[0075] When the patient is supine and the C-arm machine is on the patient's right side, the initial value of the pose transformation matrix of the intraoperative two-dimensional fluoroscopic image relative to the preoperative three-dimensional image is T0 = (1.209, 1.209, 1.209, cx, cy, cz).

[0076] When the patient is supine and the C-arm is positioned on the patient's left side, the initial value of the pose transformation matrix of the intraoperative two-dimensional fluoroscopic image relative to the preoperative three-dimensional image is T0 = (-1.209, -1.209, 1.209, cx, cy, cz).

[0077] In this invention, since the digital reconstructed radiographic image to be generated is a two-dimensional image, it is necessary to transform the preoperative three-dimensional image from the CT image coordinate system to the C-arm coordinate system of the intraoperative two-dimensional fluoroscopic image. Therefore, based on the current position of the patient relative to the C-arm machine, the initial value of the transformation relationship between the C-arm coordinate system and the CT image coordinate system is obtained, that is, the preoperative three-dimensional image is transformed to the C-arm coordinate system. The preoperative three-dimensional image is preprocessed to obtain the first processed image, and the first processed image is then used to generate the digital reconstructed radiographic image through a ray projection algorithm.

[0078] In this invention, the digitally reconstructed radiographic image generated above is linearly normalized to the range of [0, 1] to ensure consistency with the second processed image obtained in step S2.

[0079] S4. Calculate the normalized cross-correlation coefficient of the digitally reconstructed radiographic image and the second-processed image, and transform it into a minimizeable first function, specifically including:

[0080] In this invention, the image similarity between the digitally reconstructed radiographic image and the second-processed image is calculated. The similarity can be calculated using mutual information (MI) or normalized cross-correlation (NCC).

[0081] In this invention, normalized cross-correlation (NCC) is used as a similarity metric. The specific formula for calculating the normalized cross-correlation coefficient is as follows;

[0082]

[0083] in, and , respectively, are the pixel mean values ​​of the second processed image and the digitally reconstructed radiographic image, <, > are the inner product, |·| is the second norm of the vector, and the value range of NCC is [0, 1].

[0084] Based on the normalized cross-correlation coefficients obtained from the above steps, a minimizeable first function is constructed as follows:

[0085] L1 = 1 - NCC;

[0086] In this invention, in order to adapt to the gradient descent optimization framework, the normalized correlation coefficient needs to be transformed into a minimizeable objective function, where the NCC value ranges from [0, 1]. By transforming 1-NCC into a minimizeable objective, the result of maximizing similarity is equivalent to minimizing the first function value.

[0087] In this invention, the closer the value of NCC is to 1, the more similar the digitally reconstructed radiographic image and the second processed image are, and the smaller the value of the first function L1 = 1 - NCC will be, which is close to 0.

[0088] In this invention, the first function is a function of the second processed image and the digitally reconstructed radiographic image. The digitally reconstructed radiographic image is generated from the first processed image after transformation by the initial value T0 of the pose transformation matrix. Therefore, the first function is a function of the pose transformation matrix T.

[0089] S5. Threshold segmentation is performed on the preoperative 3D image to obtain the skin region image, and then point cloud data of the single-layer skin surface is obtained.

[0090] The specific steps include:

[0091] S51. Based on a set threshold, the preoperative three-dimensional image is segmented to obtain the skin region image. Morphological closing operation is performed on it to obtain the closed skin region image. The mean filtering method is used to reduce the noise of the closed skin region image.

[0092] In this invention, the skin region is segmented according to the HU value of the skin [-50, 100]. However, the HU value of some skin may not be within this range. Threshold segmentation will cause some holes in the skin region. Morphological closing operation (corrosion after linear expansion) is used to fill the holes to obtain a closed skin region image.

[0093] S52. An edge detection operator is used to perform convolution operation on the denoised skin region image to obtain the initial edge image of the skin region. Then, non-maximum suppression is performed on it to extract the single-layer skin surface point cloud data.

[0094] In this invention, a 3D edge detection operator is used to perform convolution operations on the denoised skin region image to extract edge information and obtain the initial edge image of the skin region.

[0095] In this invention, non-maximum suppression is used to refine the initial edge image of the skin region to a single-layer voxel thickness, thereby obtaining a single-layer skin surface point cloud.

[0096] In this invention, after obtaining the single-layer skin surface point cloud through step S5, the depth camera point cloud obtained in step S2 and the single-layer skin surface point cloud obtained in step S5 can be downsampled to obtain downsampled depth camera point cloud and downsampled skin surface point cloud, which helps to reduce the amount of data and subsequent computation while retaining the key features of the point cloud.

[0097] In this embodiment, the downsampling methods that can be used include, but are not limited to: uniform sampling, random sampling, and voxel grid-based sampling.

[0098] S6. Transform the depth camera point cloud using the pose transformation matrix, and construct a second function that minimizes both the depth camera point cloud data and the single-layer skin surface point cloud data.

[0099] In this embodiment, the minimizeable second function in step S6 is specifically a bidirectional nearest neighbor distance metric function that constructs the depth camera point cloud data and the single-layer skin surface point cloud data.

[0100] Specifically, to construct a differentiable bidirectional nearest neighbor distance (hamfer) metric function to implement the gradient descent-based optimization process, it is necessary to represent the depth camera point cloud data and single-layer skin surface point cloud data in a matrix format; whereby the single-layer skin surface point cloud data is defined as a matrix. Hollow R represents a set of three-dimensional points, and m represents the number of three-dimensional points contained in the point cloud data of a single layer of skin; the depth camera point cloud data is defined as a matrix. n represents the 3D point data of the depth camera point cloud. Based on this tensor representation, a differentiable bidirectional nearest neighbor distance metric function is established, ultimately achieving accurate registration between the depth camera point cloud and a single-layer skin point cloud.

[0101] The specific formula for calculating the bidirectional nearest neighbor distance metric is as follows:

[0102]

[0103] Where P = {p i}, Q={q i For each point in P, find the nearest point in Q, calculate the squared distance and take the average. For each point in Q, find the nearest point in P, calculate the squared distance and take the average. Then add the two to get the minimum value of the second function, which is the bidirectional nearest neighbor distance between the single-layer skin point cloud data and the depth camera point cloud.

[0104] In this invention, to more conveniently calculate the bidirectional nearest neighbor distance between the depth camera point cloud and the single-layer skin point cloud, it is necessary to transform the depth camera point cloud from the C-arm coordinate system to the CT image coordinate system. Therefore, using the initial value of the pose transformation matrix between the C-arm coordinate system and the CT image coordinate system obtained in step S2, the depth camera point cloud is transformed to the CT image coordinate system. Thus, the second function is a function of the pose transformation matrix T.

[0105] S7. Construct a target function based on the first function and the second function, and use the gradient descent optimization algorithm to minimize the target function. Update the pose transformation matrix one by one until the set conditions are met to obtain the minimized target function value, and finally obtain the target pose transformation matrix of the three-dimensional image and the two-dimensional perspective image.

[0106] In this invention, the target function is constructed based on the first function and the second function, as follows:

[0107] Loss = α L1 + β L2

[0108] Loss=α(1-NCC(I Xray I DRR ))+βD Charmfer (P, Q)

[0109] Here, α and β are weighting coefficients set to balance the first function and the second function. Since the range of NCC value is [0, 1], and the output range of the bidirectional nearest neighbor distance metric function is large, β is usually smaller than α. In this embodiment, α is set to 1 and β is set to 0.001.

[0110] In this invention, since both the first and second functions are functions of the pose transformation matrix T, the objective function is also a function of the pose transformation matrix T. The gradient descent optimization algorithm is used to optimize the objective function, which is actually a continuous optimization of the pose transformation matrix T. The pose transformation matrix is ​​updated once after each optimization is completed until the set conditions are met and the minimized objective function value is obtained. Finally, the target pose transformation matrix of the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image is obtained, that is, the registration of the two is completed.

[0111] In this invention, the Adam optimizer is used to optimize the objective function using gradient descent. The specific process of gradient descent optimization is described below:

[0112] a. Configure optimizer parameters: learning rate η = 0.01, step size of rotation component of pose transformation matrix T is 0.005, step size of translation component is set to 10, step size is decreased every 50 iterations, decay coefficient is 0.5. Momentum β1 = 0.9, β2 = 0.999;

[0113] b. Set the initial value T0 of the pose transformation matrix T based on the above-mentioned positional relationship between the patient and the C-arm machine;

[0114] c. Set the convergence threshold or maximum number of iterations;

[0115] d. Calculate the objective function Loss and find its gradient (i.e., derivative) with respect to the pose transformation matrix T. Then update the pose transformation matrix T. Use the updated pose transformation matrix T to transform the first processed image and generate the corresponding digitally reconstructed radiographic image. Repeat step S4 to calculate the normalized cross-correlation coefficient between the updated digitally reconstructed radiographic image and the second processed image. The first function is then updated. Use the updated pose transformation matrix T to transform the depth camera point cloud accordingly. Repeat step S6 to calculate the bidirectional nearest neighbor distance metric function between the single-layer skin surface point cloud data and the depth camera point cloud data. The second function is then updated, and the objective function is subsequently updated.

[0116] Determine whether the set conditions are met, and continue until the set conditions are met to obtain the minimized objective function value, and finally obtain the target pose transformation matrix of the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image.

[0117] The specific conditions are: compare whether the magnitude of the gradient of the objective function Loss with respect to the pose transformation matrix T is less than the convergence threshold, or whether the maximum number of iterations has been reached. If the conditions are not met, then step d is run again.

[0118] In this invention, the convergence threshold is set to 0.01, and the maximum number of iterations is set to 100 or 200.

[0119] In this invention, as the pose transformation matrix T is updated successively, it gets closer and closer to the optimal solution. The generated digitally reconstructed radiographic image becomes more and more similar to the second processed image (i.e., the value of the first function L1 gets closer and closer to 0), the depth camera point cloud becomes closer and closer to the single-layer skin surface point cloud (i.e., the value of the second function L2 gets closer and closer to 0), and the value of the target function Loss also gets closer and closer to 0, until the optimal solution is reached, and the pose transformation matrix that is no longer updated is obtained. That is, the target pose transformation matrix of the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image is obtained. At this time, the accurate registration of the two is completed through the target transformation matrix.

[0120] In this invention, registration can be completed using a single fluoroscopic image, which can save surgical time and avoid taking multiple fluoroscopic images.

[0121] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for registering a single fluoroscopic image with a CT image based on multimodal joint optimization, characterized in that: Including the following steps: S1. Preoperative scanning of the patient's target area to obtain a preoperative three-dimensional image, which is then preprocessed to obtain the first processed image; S2. Intraoperative scanning of the patient's target area to obtain intraoperative two-dimensional fluoroscopic images and depth camera point cloud data; preprocessing of the intraoperative two-dimensional fluoroscopic images to obtain a second processed image. S3. Based on the patient's position relative to the two-dimensional fluoroscopic imaging device, obtain the pose transformation matrix between the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image, so as to transform the first processed image, and then generate a digitally reconstructed radiographic image through a ray projection algorithm. S4. Calculate the normalized cross-correlation coefficient between the digitally reconstructed radiographic image and the second processed image, and convert it into a minimizeable first function; S5. Threshold segmentation is performed on the preoperative three-dimensional image to obtain a skin region image, and edge information is extracted from the skin region image to obtain an initial edge image of the skin region, thereby obtaining point cloud data of a single layer of skin surface. S6. Transform the depth camera point cloud using the pose transformation matrix, and construct a second function that minimizes both the depth camera point cloud data and the single-layer skin surface point cloud data. S7. Construct a target function based on the first function and the second function, and use the gradient descent optimization algorithm to minimize the target function. Update the pose transformation matrix one by one until the set conditions are met to obtain the minimized target function value, and finally obtain the target pose transformation matrix of the preoperative three-dimensional image and the intraoperative two-dimensional fluoroscopic image.

2. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 1, characterized in that: In step S5, threshold segmentation is performed on the preoperative three-dimensional image to obtain a skin region image, edge information is extracted from the skin region image to obtain an initial edge image of the skin region, and then single-layer skin surface point cloud data is obtained, including the following steps: S51. The skin region image is obtained by segmenting the preoperative three-dimensional image based on a set threshold, and a morphological closing operation is performed on it to obtain a closed skin region image. The closed skin region image is then denoised using a mean filtering method. S52. An edge detection operator is used to perform convolution operation on the denoised skin region image to obtain the initial edge image of the skin region. Then, non-maximum suppression is applied to it to extract the single-layer skin surface point cloud data.

3. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 1, characterized in that: The first processed image in step S1 is an image obtained by magnifying the original voxel values ​​of the skeletal region in the preoperative three-dimensional image by a set factor.

4. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 1, characterized in that: The preprocessing in step S2 includes the following steps: S21. After smoothing the intraoperative two-dimensional fluoroscopic image, logarithmic mapping is performed to obtain the logarithmic mapping result. The negative image is generated by subtracting the logarithmic mapping result from the logarithmic value of the initial energy. S22. The negative image is linearly normalized to a set range and adjusted to a set size by bilinear interpolation to obtain a second processed image.

5. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 1, characterized in that: After obtaining the depth camera point cloud in step S2 and the single-layer skin surface point cloud in step S5, downsampling is performed on both to obtain the downsampled depth camera point cloud and the downsampled skin surface point cloud.

6. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41. Calculate the normalized cross-correlation coefficient between the digitally reconstructed radiographic image and the second processed image. The calculation formula is as follows: NCC(I) Xray， I DRR )= ; in, and These are the average pixel values ​​of the second processed image and the digitally reconstructed radiographic image, respectively. It is the inner product. It is the second norm of the vector, and the numerical range of NCC is [0,1]. S42. Construct a minimizeable first function based on the normalized cross-correlation coefficient in step S41, as follows: L1 = 1 - NCC.

7. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 1, characterized in that: The minimizeable second function in step S6 is specifically a bidirectional nearest neighbor distance metric function between the depth camera point cloud data and the single-layer skin surface point cloud data, and its calculation formula is as follows: ； in For each point in P, find the nearest point in Q, calculate the squared distance and take the average. Repeat the same operation for each point in Q. The bidirectional nearest neighbor distance is the sum of these two.

8. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 1, characterized in that: In step S7, the target function constructed based on the first function and the second function is as follows: ； ； in, and These are the weighting coefficients of the balance function. The value is 1. The value is 0.

001.

9. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 2, characterized in that: The threshold value set in step S51 is selected based on the HU value of the skin, and the value range is [-50, 100].

10. The method for registering a single fluoroscopic image and a CT image based on multimodal joint optimization according to claim 3, characterized in that: The range of the set multiple is [2.5, 4.5].

Images