Three-dimensional c-arm-based 3D image automatic splicing method, system and terminal device

Abstract

The application discloses a 3D image automatic splicing method and system based on a three-dimensional C-shaped arm and a terminal device, and relates to the technical field of medical image processing. The application comprises the following steps: receiving a plurality of three-dimensional volume images, performing downsampling processing and local feature detection on the three-dimensional volume images, and obtaining three-dimensional volume image feature point coordinates and feature descriptors; based on the feature point coordinates and the feature descriptors, calculating an affine transformation matrix between adjacent images and determining a splicing direction between the images; then calculating the size of an output image after final splicing and the offset of each three-dimensional volume image to be spliced; and sequentially splicing each three-dimensional volume image into the final output image according to the offset of each three-dimensional volume image, and performing seamless fusion processing on the splicing overlap area. The splicing method provided by the application can overcome the defects of a small FOV of a three-dimensional C-shaped arm and inconvenience in observing global anatomical structures, and provide a wider field of view for doctors, thereby facilitating the doctors in diagnosing diseases.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method, system, and terminal device for automatic 3D image stitching based on a three-dimensional C-arm. Background Technology

[0002] Mobile 3D C-arm X-ray machines play an increasingly important role in modern clinical surgery, providing surgeons with real-time intraoperative 3D image navigation. Especially when used in conjunction with orthopedic surgical robots, they significantly improve surgical precision and safety. However, due to limitations in their physical structure, the field of view (FOV) of a single scan of a 3D C-arm is typically small, making it difficult to comprehensively image large anatomical structures such as the spine and pelvis in a single operation. This poses challenges for surgeons' diagnosis and surgical planning.

[0003] To address the limitation of field of view, stitching together reconstructed 3D volume images with overlapping areas from multiple scans to generate a panoramic 3D image with a wider coverage has become an essential technical requirement. In the field of 2D images, techniques based on local feature point detection and matching (such as SIFT, SURF, ORB, and BRIEF) have become the mainstream solution for image stitching, and related algorithms have been implemented in open-source libraries such as OpenCV, indicating relatively mature technology. In recent years, feature detection and matching methods based on deep learning have also demonstrated strong potential.

[0004] However, directly transferring 2D stitching technology to 3D volume image stitching faces significant challenges: First, the computational complexity and memory usage of 3D data increase cubically with size, placing extremely high demands on hardware for directly processing raw data, making it difficult to meet the real-time requirements during surgery; second, medical X-ray images inherently have high noise and low contrast, making feature extraction and matching far less stable than natural images; furthermore, some existing 3D registration methods (such as mutual information-based methods) can generally only handle pure translation transformations well, and their robustness and adaptability are poor for rotational changes that may occur during surgery due to slight patient movement or small changes in the C-arm angle, seriously affecting the success rate and practicality of stitching. Therefore, this invention proposes a 3D image automatic stitching method, system, and terminal device based on a 3D C-arm. Summary of the Invention

[0005] The purpose of this invention is to provide a method and process for automatic 3D image stitching that is simple to operate, computationally efficient, and robust, specifically for 3D C-arm systems, in order to overcome the inherent defect of small FOV and provide doctors with high-quality 3D images with global anatomical structure.

[0006] According to a first aspect of the present invention, in order to achieve the above-mentioned objective, the present invention provides the following technical solution: a 3D image automatic stitching method based on a three-dimensional C-arm, comprising:

[0007] The system receives multiple 3D volume images with overlapping regions, performs downsampling on the multiple 3D volume images, and performs local feature detection on the downsampled 3D volume images to obtain the coordinates of feature points and feature descriptors of the multiple 3D volume images.

[0008] Based on the feature point coordinates and feature descriptors of adjacent 3D volume images, the affine transformation matrix between adjacent images is calculated and the stitching direction between images is determined.

[0009] Based on the affine transformation matrix and the stitching direction, calculate the size of the final stitched output image and the offset of each three-dimensional volume image to be stitched in the output image coordinate system.

[0010] Based on the offset of each 3D volume image in the coordinate system of the output image, the 3D volume images are sequentially stitched into the final output image, and the overlapping areas are seamlessly merged.

[0011] Furthermore, multiple three-dimensional volumetric images with overlapping regions are acquired at different positions along a predetermined path using a three-dimensional C-arm. The predetermined path includes longitudinal or transverse movement along the operating table. The specific acquisition method is as follows:

[0012] (21) First, measure the total length L of the anatomical structure to be imaged and the size S of the maximum three-dimensional volume image that can be obtained by one reconstruction of the three-dimensional C-arm;

[0013] (22) Based on the principle of 40%-70% longitudinal overlap between adjacent maximum three-dimensional volume images, the number of reconstructions N and the distance M that the operating table or C-arm needs to move after each reconstruction are calculated according to the target length L and the reconstruction size S.

[0014] (23) Move the C-arm to the first planned imaging position, perform the first three-dimensional reconstruction at the current position, and obtain the first three-dimensional volume image;

[0015] (24) Based on the moving distance M, move the C-arm to the second position along the longitudinal or transverse direction of the operating table, perform the second reconstruction, and obtain the second three-dimensional volume image;

[0016] (25) Repeat steps (23) to (24) to obtain multiple three-dimensional volume images.

[0017] Furthermore, when downsampling multiple 3D volumetric images, the size of the received 3D volumetric images in each dimension is proportionally reduced to 1 / 2.

[0018] Furthermore, local feature detection is performed on the downsampled 3D volume image to obtain the coordinates of multiple 3D volume image feature points and feature descriptors, as follows:

[0019] By extending the two-dimensional SIFT, SURF, ORB, and BRIEF methods to three dimensions, the extracted features can also be local features trained using deep learning methods, resulting in keypoint coordinates and feature descriptors.

[0020] Furthermore, the affine transformation matrix between adjacent 3D volume images is calculated as follows:

[0021] (51) For two adjacent three-dimensional volumetric images, denoted as V1 and V2, their feature points are divided into two groups along the estimated stitching direction, as follows:

[0022] V1's feature points are divided into: left-side features key1_left, desc1_left and right-side features key1_right, desc1_right;

[0023] V2's feature points are divided into: left-side features key2_left, desc2_left and right-side features key2_right, desc2_right;

[0024] The matching of the right-side feature desc1_right of V1 with the left-side feature desc2_left of V2 is defined as a positive matching scenario, and the matching of the left-side feature desc1_left of V1 with the right-side feature desc2_right of V2 is defined as a negative matching scenario.

[0025] (52) Match the feature points of the two groups V1 and V2 respectively, and calculate the two candidate affine transformation matrices and the number of corresponding effective matching points, as follows:

[0026] (52.1) The nearest neighbor matching algorithm is adopted, and the FLANN-based descriptor matcher integrated by OpenCV is used to match the feature descriptors to obtain multiple matching pairs;

[0027] (52.2) Calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance, and set a threshold to retain matching pairs whose ratio is less than the set threshold;

[0028] (52.3) Use the estimateAffine3D method integrated in OpenCV to calculate the affine transformation matrices M1 and M2 of V1 and V2 respectively, and adjust the ransacThreshold parameter to adapt to medical image noise;

[0029] (53) Verify the validity of the calculated affine transformation matrix, including verifying the diagonal deviation and off-diagonal deviation of the rotation matrix R and the dominance of the translation matrix vector T in the splicing direction;

[0030] Set the diagonal deviation threshold threshR0 to 0.1 and the off-diagonal deviation threshold threshR1 to 0.2; for the translation matrix vector T, the translation along the splicing direction is much greater than the translation in other directions, so set the translation ratio threshold threshT to 1.5.

[0031] When the diagonal deviation is less than threshR0 and the off-diagonal deviation is less than threshR1; the translation vector T = (t1, t2, t3, 1), where t1, t2, and t3 correspond to translations in the x, y, and z directions respectively, and x is the horizontal direction; and when both t1 / t2 and t1 / t3 are greater than threshT, the affine transformation matrix is ​​considered valid.

[0032] (54) Based on the number of valid matching points and the verification results of the affine transformation matrix, determine the correct affine transformation matrix and splicing direction of two adjacent three-dimensional volume images.

[0033] Furthermore, based on the affine transformation matrix and the stitching direction, the size of the final stitched output image and the offset of each 3D volume image to be stitched in the output image coordinate system are calculated, as follows:

[0034] (61) Transform the vertex coordinates of each three-dimensional volume image to the coordinate system of the first image using its corresponding transformation matrix;

[0035] (62) Calculate the minimum and maximum values ​​of the coordinates of all transformed vertices;

[0036] (63) The output image is denoted as dstVolume, and the size dstSize of the output image is determined based on the difference between the minimum and maximum values;

[0037] (64) The minimum value is used as the offset move to translate each three-dimensional volume image to the correct position in the output image.

[0038] Furthermore, based on the offset of each 3D volumetric image in the output image coordinate system, the 3D volumetric images are sequentially stitched into the final output image, and the overlapping areas are seamlessly fused, as follows:

[0039] (71) Translate the first three-dimensional volume image and assign it to the output image dstVolume. Record the maximum value preRight and minimum value preLeft of the first three-dimensional volume image in the stitching direction after it has been translated to the output image.

[0040] (72) Seamlessly stitch the remaining three-dimensional volume images to dstVolume in sequence:

[0041] During stitching, first determine the maximum value currentRight and minimum value currentLeft in the stitching direction after the current 3D volume image is transformed into the output image. These two values ​​are determined by transforming the vertex coordinates of the current 3D volume image into the coordinate system of the output image, and then calculating the maximum and minimum values ​​in the stitching direction.

[0042] Extract slices from the output image along the stitching direction, with the slice range from minimum to maximum value. Transform the coordinates of this slice range to the coordinate system of the current 3D volume image, and filter using the range of the current 3D volume image. Points outside the coverage area of ​​the current 3D volume image are filtered out. The coordinates at this time are floating point type. The voxel value of the current point is obtained by linear interpolation using 8 integer vertices near the coordinates. This voxel value is then fused with the voxel value of the corresponding point in the output image to update the output image.

[0043] The overlapping region is determined using `currentLeft` and `preRight`, which form the two boundaries of the overlapping region. These two boundaries are then renamed `mergeLeft` and `mergeRight`. For voxels outside the overlapping region, the corresponding voxel value in the output image is 0, and they can be directly inserted. For voxels within the overlapping region, a fusion process is required during insertion. The specific steps of the fusion operation are as follows:

[0044] During fusion, the current voxel is multiplied by the coefficient alpha and then added to the multiplication of the corresponding output voxel. The multiplication of the corresponding output voxel is 1 – alpha, where alpha is the fusion coefficient of the gradient. The formula for calculating alpha is as follows:

[0045] alpha = (xx – mergeLeft) / (mergeRight – mergeLeft)

[0046] In the formula, xx is the coordinate of the current voxel in the splicing direction;

[0047] The stitching process is repeated sequentially. After each stitch is completed, currentLeft and currentRight are used to update preLeft and preRight, which in turn updates the fusion coefficient alpha.

[0048] According to a second aspect of the present invention, the present invention provides an automatic 3D image stitching system based on a three-dimensional C-arm, for implementing the automatic 3D image stitching method based on a three-dimensional C-arm described in the first aspect, comprising:

[0049] The local feature detection module is used to receive multiple three-dimensional volume images with overlapping regions, perform downsampling processing on the multiple three-dimensional volume images, and perform local feature detection on the downsampled three-dimensional volume images to obtain the coordinates of feature points and feature descriptors of the multiple three-dimensional volume images.

[0050] The first calculation module is used to calculate the affine transformation matrix between adjacent images and determine the stitching direction between images based on the feature point coordinates and feature descriptors of adjacent three-dimensional volume images.

[0051] The second calculation module is used to calculate the size of the final stitched output image and the offset of each three-dimensional volume image to be stitched in the output image coordinate system based on the affine transformation matrix and the stitching direction.

[0052] The stitching and fusion module is used to stitch the three-dimensional volume images sequentially into the final output image based on the offset of each three-dimensional volume image in the coordinate system of the output image, and to perform seamless fusion processing on the stitched overlapping areas.

[0053] According to a third aspect of the present invention, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor, and when the processor loads and executes the computer program, it employs the 3D image automatic stitching method based on a three-dimensional C-arm described in the first aspect.

[0054] According to a fourth aspect of the present invention, the present invention provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform the automatic 3D image stitching method based on a three-dimensional C-arm as described in the first aspect.

[0055] The present invention has at least the following beneficial effects:

[0056] 1. This invention clarifies the number of image acquisitions, locations, and movement distances through a quantitative preoperative planning process, ensuring a controllable and sufficient overlap area between adjacent three-dimensional volumetric images. This provides a reliable data foundation for subsequent high-precision automatic stitching and overcomes the drawbacks of traditional methods that rely on doctors' experience and lack standardized procedures.

[0057] 2. This invention employs a preprocessing step of downsampling the three-dimensional volume image, which significantly reduces the amount of data and reduces the memory usage to one-eighth of the original data. This significantly reduces the demand for hardware computing resources, making it possible to process large-scale three-dimensional data quickly on ordinary graphics processors and solving the bottleneck problem of high computational complexity in three-dimensional stitching algorithms.

[0058] 3. This invention extends mature two-dimensional feature extraction algorithms to three-dimensional space and combines them with graphics processors to accelerate computation, achieving efficient and rich three-dimensional local feature detection. This provides a key basis for subsequent accurate matching, balancing algorithm efficiency and feature representation capabilities.

[0059] 4. This invention adopts a combined registration strategy based on the bidirectional matching hypothesis, Lowe's ratio test, and random sampling consensus algorithm. It also innovatively introduces a physical rationality verification mechanism for the affine transformation matrix, which can automatically determine the relative orientation between volumetric images and effectively eliminate erroneous matches, greatly improving the accuracy of the registration algorithm and its robustness to the high noise characteristics of medical images.

[0060] 5. This invention automatically determines the size and offset of the final stitching result by calculating the spatial boundaries of all volumetric images in a unified coordinate system, and uses a weighted fusion method based on linear distance variation to seamlessly fuse overlapping areas, effectively eliminating stitching gaps and ensuring that the output results have good visual consistency and geometric accuracy.

[0061] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the splicing method described in this invention;

[0063] Figure 2 This is a schematic diagram showing the stitching direction of two adjacent three-dimensional volume images in this invention;

[0064] Figure 3 This is a schematic diagram of the process for calculating the affine transformation matrix in this invention;

[0065] Figure 4 This is a schematic diagram of the feature matching process in this invention;

[0066] Figure 5 This is a schematic diagram of the fusion coefficient alpha in this invention. Detailed Implementation

[0067] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0068] Example 1:

[0069] Please see Figures 1-5This invention provides a technical solution: an automatic 3D image stitching method based on a three-dimensional C-arm, comprising the following steps:

[0070] S1. Receive multiple three-dimensional volume images with overlapping regions, perform downsampling processing on the multiple three-dimensional volume images, and perform local feature detection on the downsampled three-dimensional volume images to obtain the coordinates of feature points and feature descriptors of the multiple three-dimensional volume images;

[0071] It should be noted that when downsampling multiple 3D volumetric images, the size of the received 3D volumetric images in each dimension is reduced to 1 / 2 proportionally in order to significantly reduce GPU usage. In this embodiment, the image is downsampled to ratio = 1 / 2 of the original size, and the GPU usage will be reduced to 1 / 8 of the original size.

[0072] S11. Multiple three-dimensional volumetric images with overlapping areas are acquired at different positions along a predetermined path using a three-dimensional C-arm. The predetermined path includes longitudinal or transverse movement along the operating table. By stitching images longitudinally, the surgeon can see a larger area of ​​the spine; by stitching images transversely, the surgeon can see a larger area of ​​the pelvis. The specific acquisition method is as follows:

[0073] (S11.1) First, measure the full length L of the anatomical structure to be imaged and the size S of the maximum three-dimensional volume image that can be obtained by a single reconstruction of the three-dimensional C-arm;

[0074] (S11.2) Based on the principle of 40%-70% longitudinal overlap between adjacent maximum three-dimensional volume images, the number of reconstructions N and the distance M that the operating table or C-arm needs to move after each reconstruction are calculated according to the target length L and the reconstruction size S.

[0075] The steps for calculating the number of reconstructions N and the distance M traveled are as follows:

[0076] 1) Define the overlap ratio as K. Since the overlap ratio is 40%-70%, 0.4≤K≤0.7;

[0077] 2) Model the multiple scanning process as a linear coverage problem:

[0078] The first scan covers a length of S. In each subsequent scan, by moving the operating table or C-arm, only a new length of coverage can be added. This new length is equal to the moving distance M.

[0079] After moving a distance M, the next scan starts from position M and covers M+S.

[0080] The overlapping region is the common part of the two scans. The first scan covers [0, S], and the second scan covers [M, M+S]. The length of the overlapping region is S - M, and the overlap ratio K is defined as the proportion of the overlapping region to the size S of a single scan. Therefore, K = (S - M) / S;

[0081] Therefore, the distance traveled is:

[0082] M = S(1 - K)

[0083] 3) Since the first scan covered the length S, subsequent scans from the 2nd to the Nth (a total of N-1 scans) will each add a region of length M to cover (because the overlapping part has already been calculated).

[0084] Therefore, the total length covered after N scans is:

[0085] Total coverage length = S + (N-1)M;

[0086] 4) To ensure that N scans completely cover the target length L, the following inequality must be satisfied:

[0087] S+(N-1)M≥L

[0088] Substituting M=S(1-K) into the above inequality, we get:

[0089] S+(N-1)S(1-K)≥L

[0090] After simplification, we get:

[0091] N≥1+(L / S-1) / (1-K)

[0092] Since the calculated result is not an integer, it needs to be rounded up to obtain the smallest integer, which is the number of reconstructions N.

[0093] (S11.3) Move the C-arm to the first planned imaging position, perform the first three-dimensional reconstruction at the current position, and obtain the first three-dimensional volume image;

[0094] (S11.4) According to the planned moving distance M, move the C-arm to the second position along the longitudinal or transverse direction of the operating table, perform the second reconstruction, and obtain the second three-dimensional volume image;

[0095] (S11.5) Repeat steps (S11.3) to (S11.4) to obtain multiple three-dimensional volume images;

[0096] S12. Perform local feature detection on the downsampled 3D volume image to obtain the coordinates of multiple 3D volume image feature points and feature descriptors, as follows:

[0097] Features extracted by extending two-dimensional methods such as SIFT, SURF, ORB, and BRIEF to three dimensions can also be local features trained using deep learning methods. Local feature detection yields keypoints and feature descriptors. The computational cost of three-dimensional stitching is significantly higher than that of two-dimensional stitching. Therefore, it is necessary to consider running the feature detection operation on the GPU to improve the stitching processing speed.

[0098] Extending 2D SIFT to 3D yields 3D SIFT features. Similarly, BRIEF features can be extended from 2D to 3D. This embodiment uses open-source 3D brief feature extraction code to obtain 3D feature points and feature descriptors. It should be noted that this embodiment does not rely on a specific 3D feature extraction method; it can be 3D SIFT, 3D brief, or other 3D features. This embodiment does not impose any specific limitations on these methods.

[0099] S2. Based on the feature point coordinates and feature descriptors of adjacent 3D volume images, calculate the affine transformation matrix between adjacent images and determine the stitching direction between images, as follows:

[0100] The affine transformation matrix between adjacent 3D volume images is calculated as follows:

[0101] S21. For two adjacent 3D volumetric images, denoted as V1 and V2, their feature points are divided into two groups along the estimated stitching direction, as follows:

[0102] V1's feature points are divided into: left-side features key1_left, desc1_left and right-side features key1_right, desc1_right;

[0103] V2's feature points are divided into: left-side features key2_left, desc2_left and right-side features key2_right, desc2_right;

[0104] The matching of the right-side feature desc1_right of V1 with the left-side feature desc2_left of V2 is defined as a positive matching scenario, and the matching of the left-side feature desc1_left of V1 with the right-side feature desc2_right of V2 is defined as a negative matching scenario.

[0105] Taking the stitching of a scene containing three 3D volume images (volumes) V1, V2, and V3 as an example, as described in the stitching workflow, V1, V2, and V3 are volumes reconstructed sequentially. The stitching direction is aligned with the left-right direction. The overlapping portion of V1 and V2 may be the right side of V1 and the left side of V2; this direction is considered positive. Conversely, if the overlapping area is located to the left of V1 and to the right of V2, it is considered negative. A schematic diagram of positive and negative stitching directions is shown below. Figure 2 As shown;

[0106] There are two ways to arrange V1 and V2: forward or reverse. Using desc1_left and desc2_right for registration, we get M1 and matchCnt1. Using desc1_right and desc2_left for registration, we get M2 and matchCnt2. In the forward scenario, matchCnt2 will be much larger than matchCnt1; conversely, in the reverse scenario, matchCnt1 will be much larger than matchCnt2.

[0107] S22. Match the feature points of the two sets of V1 and V2 respectively, and calculate the two candidate affine transformation matrices and the corresponding number of effective matching points, as follows:

[0108] (S22.1) The nearest neighbor matching algorithm is adopted, and the FLANN-based descriptor matcher integrated by OpenCV is used to match the feature descriptors to obtain multiple matching pairs;

[0109] (S22.2) Calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance and set a threshold. Retain matching pairs whose ratio is less than the set threshold. For the technical solution of this implementation, Lowe's ratio threshold is set to 0.75 in order to filter out better matches. This threshold filters out better matching results by the ratio of the nearest neighbor distance to the second nearest neighbor distance being less than the set threshold.

[0110] It should be noted that the distance in step (S22.2) uses Euclidean distance, which is the sum of squares of the differences between each component of the descriptor vector, and then the square root is taken.

[0111] (S22.3) Using the estimateAffine3D method integrated in OpenCV, calculate the affine transformation matrices M1 and M2 for V1 and V2 respectively, and adjust the ransacThreshold parameter to adapt to medical image noise, as follows:

[0112] In OpenCV, cv2.estimateAffine3D() is a function used to estimate the affine transformation matrix between two 3D point sets. It is mainly used for point set registration in 3D space. Its core function is to find an optimal 3D affine transformation (including rotation, scaling, translation, and shearing) that minimizes the error between the source point set and the target point set after the transformation.

[0113] In cv2.estimateAffine3D(), RANSAC (Random Sample Consensus) is used to robustly estimate the 3D affine transformation matrix, mainly for handling point sets containing noise, outliers, or mismatched points.

[0114] In this embodiment, OpenCV is directly called to adjust the parameters to adapt to medical image noise.

[0115] To address the high noise levels in medical images, the parameter `ransacThreshold` is appropriately lowered to more effectively identify and remove outliers, thus improving RANSAC performance in noisy environments. The affine transformation matrix calculation returns the transformation matrix `M` and the number of filtered matching points `matchCnt`. The affine transformation calculation process can be found in [link to documentation]. Figure 3 ;

[0116] like Figure 4 The following diagram shows the matching matrix and matching direction selection logic:

[0117] Determine the direction of a set of volumes. For a set of volumes, the directions must be consistent, otherwise the splicing failure will be returned. If the volume direction is reversed, the volume direction is reversed, and the matching matrix is ​​inverted to obtain the matching matrix after the direction is reversed.

[0118] S23. Verify the validity of the calculated affine transformation matrix, including verifying the diagonal deviation and off-diagonal deviation of the rotation matrix R, as well as the dominance of the translation matrix vector T in the splicing direction;

[0119] The affine transformation matrix is ​​a 3x4 matrix, denoted as P or [R|T], where R is the first 3 columns of P, representing rotation, and T is the 4th column of P, representing translation. In practical applications, there are translations and slight rotations between two adjacent reconstructions. In this case, the values ​​on the main diagonal of the R matrix are close to 1, and the values ​​on the off-diagonal are also within a certain range. Based on this characteristic, the effectiveness of the affine transformation can be verified. For the R matrix, the diagonal deviation threshold threshR0 is set to 0.1, and the off-diagonal deviation threshold threshR1 is set to 0.2. For the translation matrix T, the translation along the splicing direction is much greater than the translation in other directions. Therefore, the translation ratio threshold threshT can be set to 1.5 to verify that the absolute value of the translation distance along the splicing direction is not less than threshT times the absolute value along other directions. The two sets of calculated affine transformation matrices are verified separately, resulting in verification results check1 and check2.

[0120] When the diagonal deviation is less than threshR0 and the off-diagonal deviation is less than threshR1; the translation vector T = (t1, t2, t3, 1), where t1, t2, and t3 correspond to translations in the x, y, and z directions respectively, and x is the horizontal direction; and when both t1 / t2 and t1 / t3 are greater than threshT, the affine transformation matrix is ​​considered valid.

[0121] S24. Based on the number of valid matching points and the verification results of the affine transformation matrix, determine the correct affine transformation matrix and stitching direction for two adjacent three-dimensional volume images;

[0122] S3. Based on the affine transformation matrix and the stitching direction, calculate the size of the final stitched output image and the offset of each 3D volume image to be stitched in the output image coordinate system, as follows:

[0123] S31. Transform the vertex coordinates of each three-dimensional volume image to the coordinate system of the first image using their corresponding transformation matrices;

[0124] S32. Calculate the minimum and maximum values ​​of the coordinates of all transformed vertices;

[0125] S33. The output image is denoted as dstVolume, and the size dstSize of the output image is determined based on the difference between the minimum and maximum values;

[0126] S34. The minimum value is used as the offset move to translate each three-dimensional volume image to the correct position in the output image;

[0127] Specifically, for two adjacent 3D volume images, the transformation matrix from the latter 3D volume image to the former 3D volume image is obtained, and the volumes have been adjusted to be in the forward order. The transformation matrix is ​​homogeneous, and then the matrices are multiplied sequentially from back to front to obtain the transformation matrix from the current volume to the first volume.

[0128] Each volume has 8 vertices, and the coordinates of these 8 vertices in the current volume are fixed. Taking a cube with a side length of L as an example, the vertex coordinates are [0,0,0], [0,0,L], [L,L,L], etc. The vertex coordinates of the second to last volumes are transformed to the coordinate system of the first volume by multiplying the homogenized vertex coordinates by the transformation matrix. These vertex coordinates are then combined with the coordinates of the 8 vertices of the first volume. The minimum and maximum values ​​of all coordinates are calculated, and the difference between the maximum and minimum values ​​is used as the size of the output image. The minimum coordinate value is used as the offset value for all volumes.

[0129] When representing a volume, the coordinates cannot be negative. The purpose of setting the offset value is to translate the transformed volume to a valid coordinate position. The offset is a translation vector, which can be combined with the transformation matrix from the current volume to the first volume to form a new transformation matrix.

[0130] S4. Based on the offset of each 3D volume image in the output image coordinate system, stitch the 3D volume images sequentially into the final output image, and perform seamless fusion processing on the overlapping areas, as follows:

[0131] S41. Stitch together the first 3D volume image:

[0132] Translate the first 3D volume image (volume) and assign it to the output image (dstVolume). Record the maximum value (preRight) and minimum value (preLeft) of the first 3D volume image in the stitching direction after it has been translated to the output image.

[0133] S42. Seamlessly join the other volumes to dstVolume in sequence:

[0134] During splicing, first determine the maximum value currentRight and minimum value currentLeft in the splicing direction after the current volume is transformed to dstVolume. These two values ​​are determined by transforming the vertex coordinates of the current volume to the dstVolume coordinate system and then calculating the maximum and minimum values ​​in the splicing direction to approximate them.

[0135] Specifically, the vertex coordinates are represented as Pts_current, the transformation matrix is ​​M, a volume has 8 vertices, and the volume side length is s, so the vertex coordinates are ((0,0,0), (0,s,0)…(s,s,s)). When the current volume coordinates are transformed to the output volume coordinate system, then Pts_out = M Pts_current. Then, the maximum and minimum values ​​in the x-direction of the 8 points in Pts_out are calculated, which are currentRight and currentLeft.

[0136] Extract slices from `dstVolume` along the splicing direction, ranging from `currentLeft` to `currentRight`. Transform the coordinates of this slice range to the coordinate system of the current volume, and filter using the current volume's range. Points outside the current volume's coverage area are filtered out. Since the coordinates are now floating-point, linear interpolation is performed using the eight nearest integer vertices to obtain the voxel value of the current point. This voxel value is then fused with the corresponding voxel value in `dstVolume` to update `dstVolume`. This linear interpolation method effectively avoids the problem of some voxel values ​​not being filled, which would cause black lines to appear in `dstVolume`.

[0137] The overlapping region can be determined by currentLeft and preRight. currentLeft and preRight constitute the two boundaries of the overlapping region. These two boundaries are re-denoted as mergeLeft and mergeRight. For voxels outside the overlapping region, the corresponding voxel value of dstVolume is 0, and they can be directly inserted. For voxels within the overlapping region, merging processing is required during insertion.

[0138] During merging, the current voxel is multiplied by the coefficient alpha and then added to the voxel corresponding to dstVolume multiplied by (1 – alpha). To achieve seamless stitching, alpha needs to be a gradually changing coefficient. The formula for calculating alpha is as follows:

[0139] alpha = (xx – mergeLeft) / (mergeRight – mergeLeft)

[0140] Where xx is the coordinate of the current voxel in the splicing direction;

[0141] After completing one stitch, update preLeft and preRight using currentLeft and currentRight. The curve diagram of the blending coefficient alpha is shown below. Figure 5As shown, Figure 5 The horizontal axis represents the spatial position of each voxel in the overlapping area along the splicing direction, and the vertical axis represents the value of the fusion coefficient alpha corresponding to each position.

[0142] In summary, the splicing method proposed in this invention can overcome the pain points of small FOV of the three-dimensional C-arm and inconvenience in observing the global anatomical structure, providing doctors with a wider field of view and facilitating diagnosis.

[0143] Example 2:

[0144] This embodiment provides a 3D image automatic stitching system based on a three-dimensional C-arm, used to implement the 3D image automatic stitching method based on a three-dimensional C-arm described in Embodiment 1, including:

[0145] The local feature detection module is used to receive multiple three-dimensional volume images with overlapping regions, perform downsampling processing on the multiple three-dimensional volume images, and perform local feature detection on the downsampled three-dimensional volume images to obtain the coordinates of feature points and feature descriptors of the multiple three-dimensional volume images.

[0146] The first calculation module is used to calculate the affine transformation matrix between adjacent images and determine the stitching direction between images based on the feature point coordinates and feature descriptors of adjacent three-dimensional volume images.

[0147] The second calculation module is used to calculate the size of the final stitched output image and the offset of each three-dimensional volume image to be stitched in the output image coordinate system based on the affine transformation matrix and the stitching direction.

[0148] The stitching and fusion module is used to stitch the three-dimensional volume images sequentially into the final output image based on the offset of each three-dimensional volume image in the coordinate system of the output image, and to perform seamless fusion processing on the stitched overlapping areas.

[0149] Specifically, the aforementioned local feature detection module, first calculation module, second calculation module, and stitching fusion module can be embedded into a computer processing system. The computer, based on the aforementioned 3D image automatic stitching method based on a three-dimensional C-arm, calls the aforementioned modules to complete the task of seamlessly stitching orthopedic three-dimensional images. The aforementioned local feature detection module, first calculation module, second calculation module, and stitching fusion module can perform operations according to the specific steps given in the 3D image automatic stitching method based on a three-dimensional C-arm described in Embodiment 1.

[0150] It should be noted that the division of the various modules in the above system is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the local feature detection module can be a separate processing element or integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and its signal processing module functions can be called and executed by a processing element of the device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0151] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0152] Example 3:

[0153] The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it adopts the 3D image automatic stitching method based on a three-dimensional C-arm described in Embodiment 1.

[0154] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server, and the terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0155] Furthermore, the processor can be a central processing unit (CPU). Of course, depending on the actual use, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be used. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.

[0156] Example 4:

[0157] The present invention provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the automatic 3D image stitching method based on a three-dimensional C-arm described in Embodiment 1.

[0158] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0159] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0160] For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on," "mounted on," "fixed to," or "set on" another element, it may be directly on the other element or there may be an intermediate element present. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible embodiments.

[0161] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0162] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

Claims

1. A 3D image automatic stitching method based on a three-dimensional c-arm, characterized in that, The method comprises the following steps: Receiving a plurality of three-dimensional volume images with overlapping regions, downsampling the plurality of three-dimensional volume images, and detecting local features of the downsampled three-dimensional volume images to obtain three-dimensional volume image feature point coordinates and feature descriptors; Based on the feature point coordinates and feature descriptors of adjacent three-dimensional volume images, calculating an affine transformation matrix between adjacent images and determining the splicing direction between the images; According to the affine transformation matrix and the splicing direction, calculating the size of the final spliced output image and the offset of each three-dimensional volume image in the output image coordinate system; According to the offset of each three-dimensional volume image in the output image coordinate system, splicing each three-dimensional volume image into the final output image in sequence, and performing seamless fusion processing on the spliced overlapping region; The affine transformation matrix between adjacent three-dimensional volume images is calculated as follows: For two adjacent three-dimensional volume images, denoted as V1 and V2, the feature points are divided into two groups along the estimated splicing direction as follows: The feature points of V1 are divided into left feature key1_left,desc1_left and right feature key1_right,desc1_right; The feature points of V2 are divided into left feature key2_left,desc2_left and right feature key2_right,desc2_right; Defining the matching of the right feature desc1_right of V1 and the left feature desc2_left of V2 as a positive matching scenario, and defining the matching of the left feature desc1_left of V1 and the right feature desc2_right of V2 as a negative matching scenario; Match the two groups of feature points of V1 and V2 respectively, and calculate two candidate affine transformation matrices and the corresponding number of valid matching points as follows: Using the nearest neighbor matching algorithm, using the opencv integrated FLANN-based descriptor matcher to match the feature descriptors, obtaining a plurality of matching pairs; Calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance, and set a threshold to retain the matching pairs with a ratio less than the set threshold; Using the opencv integrated estimateAffine3D method to calculate the affine transformation matrices M1 and M2 of V1 and V2 respectively, and adjusting the ransacThreshold parameter to adapt to the noise of medical images; Performing validity check on the calculated affine transformation matrix, including checking the diagonal line deviation of the rotation matrix R, the non-diagonal line deviation, and the dominance of the translation matrix vector T in the splicing direction; Setting the diagonal line deviation threshold threshR0 to 0.1 and the non-diagonal line deviation threshold threshR1 to 0.2; for the translation matrix vector T, the translation along the splicing direction is much larger than the translation in other directions, and the translation ratio threshold threshT is set to 1.5; When the diagonal deviation is less than threshR0, the non-diagonal deviation is less than threshR1; the translation vector T=(t1, t2, t3, 1), t1, t2, t3 respectively correspond to x, y, z direction translation, wherein x is the horizontal direction; wherein t1 / t2, t1 / t3 are both greater than threshT, then it is determined that the affine transformation matrix has validity; According to the number of valid matching points and the checking result of the affine transformation matrix, the correct affine transformation matrix and the splicing direction of the adjacent two three-dimensional volume images are determined.

2. The 3D image automatic stitching method based on a three-dimensional c-arm according to claim 1, characterized in that: A plurality of three-dimensional volume images with overlapping regions are acquired at different positions along a predetermined path by a three-dimensional C-shaped arm, wherein the predetermined path includes moving along the longitudinal direction of the operating bed or moving along the transverse direction of the operating bed, and the specific acquisition method is as follows: (21) First, measure the full length L of the anatomical structure to be imaged and the size S of the maximum three-dimensional volume image that can be obtained by one reconstruction of the three-dimensional C-shaped arm; (22) According to the principle of 40%-70% longitudinal overlap of adjacent maximum three-dimensional volume images, the reconstruction times N and the distance M that the operating bed or the C-shaped arm needs to move after each reconstruction are calculated according to the target length L and the reconstruction size S; (23) Move the C-shaped arm to the first planned imaging position, perform the first three-dimensional reconstruction at the current position, and obtain the first three-dimensional volume image; (24) According to the moving distance M, move the C-shaped arm to the second position along the longitudinal direction or the transverse direction of the operating bed, perform the second reconstruction, and obtain the second three-dimensional volume image; (25) Repeat steps (23) to (24) to obtain a plurality of three-dimensional volume images.

3. The 3D image automatic stitching method based on a three-dimensional c-arm according to claim 2, characterized in that: When downsampling the plurality of three-dimensional volume images, the size of the received three-dimensional volume image in each dimension is reduced to 1 / 2.

4. The 3D image automatic stitching method based on a three-dimensional c-arm according to claim 3, characterized in that: Local feature detection is performed on the downsampled three-dimensional volume image to obtain a plurality of three-dimensional volume image feature point coordinates and feature descriptors, specifically as follows: By extending the two-dimensional BRIEF method to 3D, using the open source 3D brief feature extraction code, the feature point coordinates keyPoints and the feature descriptors descriptions are obtained.

5. The 3D image automatic stitching method based on three-dimensional c-arm according to claim 4, characterized in that: According to the affine transformation matrix and the splicing direction, the size of the final spliced output image and the offset of each three-dimensional volume image in the output image coordinate system are calculated, specifically as follows: (51) Convert the vertex coordinates of each three-dimensional volume image to the coordinate system of the first image through the corresponding transformation matrix; (52) Calculate the minimum value and the maximum value of all converted vertex coordinates; (53) Record the output image as dstVolume, and determine the size dstSize of the output image according to the difference between the minimum value and the maximum value; (54) Take the minimum value as the offset move for moving each three-dimensional volume image to the correct position in the output image.

6. The 3D image automatic stitching method based on three-dimensional c-arm according to claim 5, characterized in that: According to the offset of each three-dimensional volume image in the output image coordinate system, each three-dimensional volume image is sequentially spliced into the final output image, and seamless fusion processing is performed on the splicing overlap region, specifically as follows: (61) translating the first three-dimensional volume image to assign to an output image dstVolume, recording the maximum value preRight and the minimum value preLeft of the first three-dimensional volume image in the direction of splicing after the first three-dimensional volume image is translated to the output image; (62) sequentially splicing the remaining three-dimensional volume images to the dstVolume seamlessly: When splicing, the maximum value currentRight and the minimum value currentLeft in the direction of splicing of the current three-dimensional volume image after the current three-dimensional volume image is converted to the output image are determined first, and the determination of the two values is obtained by converting the vertex coordinates of the current three-dimensional volume image to the output image coordinate system and then calculating the maximum value and the minimum value in the direction of splicing; extracting a slice of the output image along the direction of splicing, the slice range being from the minimum value to the maximum value, converting the coordinate of the slice range to the coordinate system of the current three-dimensional volume image, and filtering using the range of the current three-dimensional volume image, the points exceeding the coverage range of the current three-dimensional volume image being filtered out, the coordinate at this time being a floating-point type, the voxel value of the current point being obtained by using linear interpolation method through the eight integer vertexes around the coordinate, and the voxel value being fused with the voxel value of the corresponding point of the output image to update the output image; determining the overlapping region through the currentLeft and the preRight, the currentLeft and the preRight constituting two boundaries of the overlapping region, the two boundaries being re-recorded as the mergeLeft and the mergeRight, the voxel value of the corresponding voxel of the output image being 0 for the voxels outside the overlapping region, and the voxel being directly inserted; for the voxels in the overlapping region, the fusion processing needs to be performed when the voxel is inserted, and the specific steps of the fusion operation are as follows: the current voxel is multiplied by a coefficient alpha and then fused with the corresponding voxel of the output, wherein the corresponding voxel of the output is multiplied by 1-alpha, alpha is a gradual fusion coefficient, and the alpha calculation formula is as follows: alpha = (xx - mergeLeft) / (mergeRight - mergeLeft) In the formula, xx is the coordinate of the current voxel in the direction of splicing. sequentially repeating the splicing, updating the preLeft and the preRight using the currentLeft and the currentRight after completing the splicing once, and further updating the fusion coefficient alpha.

7. A three-dimensional c-arm based 3D image automatic stitching system for implementing the three-dimensional c-arm based 3D image automatic stitching method according to any one of claims 1 to 6, characterized in that, Comprise: a local feature detection module, configured to receive a plurality of three-dimensional volume images having overlapping regions, perform down-sampling processing on the plurality of three-dimensional volume images, and perform local feature detection on the down-sampled three-dimensional volume images to obtain three-dimensional volume image feature point coordinates and feature descriptors of the plurality of three-dimensional volume images; a first calculation module, configured to calculate an affine transformation matrix between adjacent images and determine a splicing direction between the adjacent images based on the feature point coordinates and the feature descriptors of the adjacent images; a second calculation module, configured to calculate the size of an output image after final splicing and the offset of each three-dimensional volume image to be spliced in the output image coordinate system according to the affine transformation matrix and the splicing direction; The splicing and fusing module is used for splicing the three-dimensional volume images into a final output image in sequence according to the offset of each three-dimensional volume image in the output image coordinate system, and performing seamless fusion processing on the splicing overlapping area.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the three-dimensional c-arm-based 3D image automatic splicing method in any one of claims 1 to 6 is adopted.

9. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by the computer processor, are used to perform the three-dimensional c-arm-based 3D image automatic splicing method in any one of claims 1 to 6.

Images