A method and system for constructing 3D thoracic skeleton based on multi-viewpoint disordered X-rays

Abstract

The present invention discloses a method for constructing a thoracic skeleton three-dimensionally based on a multi-viewpoint disordered X-ray film, comprising: S1, scanning and photographing the thoracic cavity by using an X-ray device to obtain an initial image group, wherein the initial image group includes X-ray images generated by scanning at multiple angles; S2, preprocessing the initial image group, and simultaneously constructing a camera model based on the X-ray device to determine the internal and external parameters of the camera; S3, extracting features from the X-ray image and performing feature point matching; S4, constructing a sparse point cloud model by using a motion recovery structure algorithm; S5, extracting geometric information from the sparse point cloud model to construct a dense point cloud model; S6, obtaining a thoracic skeleton three-dimensional model having color, texture and structural characteristics through post-processing. The present invention also provides a system based on the construction method. The method is simple to operate, and only requires providing a group of X-ray images of a patient, so as to obtain a thoracic skeleton three-dimensional model with clear surface texture.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image generation, and in particular to a method and system for constructing three-dimensional thoracic skeleton based on multi-viewpoint disordered X-ray films. Background Art

[0002] Tumors pose a great threat to human health. With the development of medical technology, tumor puncture, which has the advantages of small incision, mild pain and less bleeding, has gradually become one of the latest important diagnosis and treatment methods for tumor patients. However, when doctors perform puncture surgery on tumor organs in the chest cavity, they usually need to use CT three-dimensional mapping before the operation, use ribs, intercostal spaces, body surface landmarks, etc. as reference points, select the needle insertion point and plan the needle insertion route based on the doctor's opinion, and insert the needle under the guidance of medical images during the operation.

[0003] However, since CT cannot generate images in real time, it is impossible to monitor in real time the movement of tissue and ribs in the patient's chest due to breathing or accidental shaking during surgery. There is a risk of incorrect positioning of the puncture needle and accidental contact with the dense nerves near the ribs.

[0004] With the continuous development of medical imaging three-dimensional reconstruction technology, computer-assisted medicine is being used more and more widely in clinical examination and treatment. Taking into account the shortcomings of CT imaging, such as slow imaging and high radiation exposure to patients, especially in puncture surgery, patients need to take CT images multiple times, which will cause certain physical damage and fatigue to patients. Therefore, X-ray detection with the characteristics of real-time monitoring, low radiation, and low cost is urgently needed for this purpose. It is undeniable that real-time X-rays lack anatomical information, especially the fuzzy imaging of soft tissues. Therefore, the present invention performs three-dimensional reconstruction of the thoracic skeleton with clear imaging in the X-ray film. The three-dimensional model of the thoracic skeleton obtained by rapid reconstruction can ensure that the distance between the puncture needle and the upper and lower ribs of the intercostal segment is visible, avoiding the risk of accidentally touching the intercostal nerves.

[0005] Patent document CN109223016A discloses a CT imaging method and imaging device, including: S1, scanning the imaging object with X-rays to obtain initial projection data, the initial projection data including data of at least one imaging geometric parameter; S2, performing parameter sampling on at least one imaging geometric parameter and then performing simulated CT projection to obtain a series of projection data; S3, searching for two projection data closest to the initial projection data in a series of projection data; S4, calculating and obtaining an updated value of at least one imaging geometric parameter through a difference algorithm; S5, performing image reconstruction based on the updated value to obtain a CT tomographic image of the imaging object; S6, evaluating whether the CT tomographic image meets the requirements; if it meets the requirements, outputting the results; if it does not meet the requirements, returning to step S2 and repeating the execution until a CT tomographic image that meets the requirements is obtained. This technical solution only uses the projection data closest to the initial projection data for reconstruction, and does not reuse the projection data under multiple viewing angles. The accuracy of bone reconstruction is not high, and it is necessary to consider the differences in the radiation attenuation coefficients of various tissues in the human body.

[0006] Patent document CN104282039A discloses a method for orthopedic brace plasticity based on 3D scanning, including using 3D scanning technology to accurately obtain surface point cloud data of human body parts, correcting the 3D model, and then using 3D printing technology to print in blocks and lines to produce orthopedic braces, and finally splicing the images. However, the 3D scanning technology used in this technical method is to scan the surface of the human body and then combine it with X-ray data, rather than directly scanning based on X-rays, so it is impossible to obtain an accurate bone reconstruction model. Summary of the invention

[0007] In order to solve the above problems, the present invention provides a method for constructing a three-dimensional thoracic skeleton based on multi-viewpoint disordered X-rays. The method is simple to operate and only requires providing a group of patient X-ray images to obtain a three-dimensional model of the thoracic skeleton with clear surface texture. Furthermore, simulation exercises can be performed using the three-dimensional model of the thoracic skeleton to avoid accidentally touching the intercostal nerves during actual surgery.

[0008] A method for constructing thoracic skeleton 3D based on multi-viewpoint disordered X-rays, comprising:

[0009] S1 scans and photographs the chest cavity using an X-ray device to obtain an initial image group, wherein the initial image group includes X-ray images generated by scanning at multiple angles;

[0010] S2 performs preprocessing on the initial image group, and builds a camera model based on the X-ray equipment to determine the internal and external parameters of the camera;

[0011] S3 extracts features from X-ray images and performs feature point matching;

[0012] S4 uses the motion recovery structure algorithm to construct a sparse point cloud model;

[0013] S5 extracts geometric information from the sparse point cloud model and constructs a dense point cloud model;

[0014] S6 obtains a 3D model of the thoracic skeleton with color, texture and structural characteristics through post-processing.

[0015] Preferably, in S1, scanning and photographing are performed based on the patient changing between two postures, namely, lying upright and lying vertically on the side, with photographs taken at least once every 20°, and photographs taken at least once every 15° at a multi-detail position.

[0016] Preferably, the preprocessing in S2: performing high-frequency modulation filtering and histogram equalization on the X-ray image, and performing low-order bit layer zeroing processing on the X-ray image after bit decomposition; using high-frequency enhancement filtering to highlight the bones and remove noise at the same time, and the constant term in the design of the filter can generally be greater than 3, so as to retain the blurred rib edge contour; enhancing the image contrast through histogram equalization, so as to highlight the ribs in the chest area.

[0017] Preferably, the feature extraction in S3 adopts the AKAZE feature algorithm, and constructs the scale space of the thoracic bones through nonlinear diffusion filtering, thereby solving the problem of blurry and dim ribs in X-rays; further, the AKAZE feature algorithm is a simplified algorithm, thereby reducing the time consumption of constructing the entire three-dimensional model.

[0018] Preferably, during the feature extraction process, the weight of the gradient value of the descriptor LDB in the vertical direction is adjusted to the maximum to adapt to the basic horizontal appearance of the ribs and the actual characteristics in the image.

[0019] Preferably, the specific process of S4 is:

[0020] S4.1 calibrates the camera, solves the intrinsic parameter matrix, extracts feature points and performs feature matching on every two images;

[0021] S4.2 selects the group with the most matching points as the starting image, and solves the essential matrix or basic matrix according to the epipolar geometry, where the essential matrix refers to the relationship between different cameras in the camera coordinate system, and the basic matrix refers to the relationship in the image coordinate system;

[0022] S4.3 triangulates the feature points matched by the above two images, continuously adds new images, and performs 3D-2D matching through the feature points matched by the new images;

[0023] S4.4 According to the BA algorithm, nonlinear optimization is performed on all camera poses and the three-dimensional coordinates of the thoracic skeleton in space to minimize the error obtained.

[0024] Preferably, the mismatched feature points in the feature point matching process of S3 are filtered by a RANSAC algorithm, thereby reducing the error rate of feature point matching.

[0025] Preferably, the post-processing in S7 adopts an implicit fitting method, specifically:

[0026] S7.1 Obtain an implicit equation representing the surface information described by the point cloud model by solving the Poisson equation;

[0027] S7.2 extracts isosurfaces from the equation to obtain a surface model with geometric entity information.

[0028] A system for implementing the above-mentioned thoracic skeleton three-dimensional construction method, comprising:

[0029] An image input module, used for inputting X-ray images;

[0030] Model output module, used to output the completed three-dimensional model;

[0031] Camera calibration module, used to obtain the three-dimensional coordinates of the scene;

[0032] An analysis module for analyzing X-ray image information and the camera posture of the X-ray device;

[0033] The generation module builds the model according to the analysis results of the analysis module;

[0034] The correction module corrects the image information during the model building process.

[0035] Doctors only need to import the patient's scanned X-rays into the system to obtain a three-dimensional model of the patient's chest bones.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) The A-KAZE algorithm is used to avoid the problem of target boundary information loss caused by Gaussian blur, and further improve the clarity of the chest rib surface texture.

[0038] (2) During the feature extraction process, the weight of the gradient value of the descriptor LDB in the vertical direction is adjusted to the maximum, so that the extracted image features are more consistent with the characteristics of the actual rib cross-section.

[0039] (3) This method can automatically generate the corresponding three-dimensional model by simply providing the image information of the X-ray machine, which is suitable for medical scenarios where time and imaging equipment are very scarce. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1It is a technical flow diagram of the method for constructing three-dimensional thoracic skeleton of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of a system for implementing the method for three-dimensionally constructing thoracic skeleton of the present invention;

[0042] Figure 3 It is a rendering of the thoracic skeleton three-dimensional model generated by the system of the present invention. DETAILED DESCRIPTION

[0043] like Figure 1 As shown, the patient's chest area is photographed by a C-arm X-ray machine. The patient only needs to change between lying upright and lying vertically on the side. The other acquisition angles are changed by the instrument. There is no strict requirement for the interval of shooting angles, which is between 15°-20°. More than 30 multi-viewpoint X-rays can be taken quickly.

[0044] S2 pre-processes the X-ray film:

[0045] High-frequency enhancement filtering is used to highlight the bones and remove noise at the same time. The constant term in the filter design can generally be greater than 3, in order to retain the blurred rib edge contour to prevent it from being discarded by the template filter.

[0046] For the low-order bits, the 0 and 1 positions are selected to be zero, and the blurred pixel components belonging to the soft tissues are directly discarded, thereby highlighting the ribs that are more blurred than the spine.

[0047] Histogram equalization is used: the cumulative distribution function (CDF) of the pixel level is obtained, and the gray levels are merged after rounding to enhance the image contrast and highlight the ribs in the chest area.

[0048] S3 extracts features from the preprocessed X-ray images, estimates camera pose, and matches feature points:

[0049] Extract the feature points between any two images in all X-ray images, and use the two images whose matching feature points reach a certain threshold as the image pair for subsequent operations.

[0050] The A-KAZE algorithm is used to extract features from all X-ray images. After setting the number of layers O and the number of towers S, the scale space is constructed according to the time ti through the diffusion function: σi(o,s)=2(o+s / S); the selected binary descriptor LDB uses the domain pixel intensity mean information and gradient information to increase the robustness of the binary description, among which the gradient operator is more sensitive to vertical boundaries, and is adapted to the basic horizontal appearance of the ribs and the actual characteristics in the image during feature extraction.

[0051] The position and attitude information of the X-ray device is obtained by the Iterative Closest Point (ICP) method, which is an optimal registration method based on the least squares method. This algorithm repeatedly selects corresponding point pairs and calculates the optimal rigid body transformation until the convergence accuracy requirement for correct registration is met. Given two three-dimensional point sets X1 and X2, the registration steps of the ICP method for these two point sets are as follows:

[0052] S3.1 Calculate the corresponding nearest points of each point in point set X2 in point set X1;

[0053] S3.2 Calculate the rigid body transformation that minimizes the average distance of the above corresponding point pairs, and obtain the translation parameters and rotation parameters;

[0054] S3.3 Apply the translation and rotation parameters obtained in the previous step to X2 to obtain a new transformed point set;

[0055] S3.4 If the average distance between the new transformed point set and the reference point set is less than a given threshold, stop the iterative calculation; otherwise, the new transformed point set will be used as the new X2 point set to continue the iteration until the requirements of the objective function are met.

[0056] Select the N images with the closest poses, where N is a natural number greater than or equal to 4. Then, use the nearest neighbor matching algorithm to match the unlabeled and already matched images. The algorithm calculates the nearest neighbor distance and the second nearest neighbor distance of the feature vectors and uses their ratio as the matching constraint condition for feature point matching. For example, for a certain eigenvector V1 in image I, calculate and filter out the two eigenvectors in the eigenvectors of another image J that are closest to this eigenvector V J1 and V J2 , define the distance ratio between them as Compare it with the preset threshold T. If d < T, it means the matching is successful; otherwise, it is a failure.

[0057] Usually, T ≥ 0.6. After completing the matching work, mark these two images as already matched.

[0058] Use the RANSAC algorithm for feature point filtering. Randomly select four matching point pairs from the feature point set to form a sample, calculate the transformation matrix, calculate the consensus that satisfies the current transformation matrix in combination with the error metric function, and judge whether it is the optimal consensus set according to the number of elements in it, and continue to update the consensus set.

[0059] S4 Use the Structure from Motion (SfM) algorithm to construct a sparse point cloud model:

[0060] S4.1 Calibrate the camera, solve the internal parameter matrix, extract feature points, and perform feature matching for every two images;

[0061] S4.2 selects the group with the most matching points as the starting image, and solves the essential matrix or basic matrix according to the epipolar geometry, where the essential matrix refers to the relationship between different cameras in the camera coordinate system, and the basic matrix refers to the relationship in the image coordinate system;

[0062] S4.3 triangulates the feature points matched by the above two images, continuously adds new images, and performs 3D-2D matching through the feature points matched by the new images;

[0063] S4.4 According to the BA algorithm, nonlinear optimization is performed on all camera poses and the three-dimensional coordinates of the thoracic skeleton in space to minimize the error obtained.

[0064] After S5 completes the construction of the sparse point cloud model, we convert the results into three types of information:

[0065] cameras.txt which contains all the image camera intrinsic parameters, images.txt which contains all the image poses and keypoints, and points.txt which contains all the reconstructed 3D points.

[0066] The dense point cloud is reconstructed using the PMVS algorithm based on patch-based 3D multi-view stereo vision. The patch similarity is mainly calculated through the gray consistency function, and the image set of two patches above a certain threshold is used as the subsequent patch extraction image pair. The gray consistency function is defined as follows:

[0067]

[0068] Where V(p) refers to the set of all visible patches in an image, R(p) is the corresponding reference image, and h is the grayscale consistency function. The patches are then optimized and filtered. The filtering relies on three filters. The first filter is filtered by visual consistency. Let U(p) represent the set of patches that are discontinuous with the current visual information. The so-called discontinuity means that the two patches p and p' are not neighbors, but exist in the same visible image block. For a patch p in U(p), if the following conditions are met, it will be filtered out:

[0069]

[0070] The second filter also considers visual consistency, but for each face p, the total number of visible images obtained by p through the depth test is calculated. If the number is less than the threshold, it is filtered out. For the third filter, for each face p, if the ratio of the number of faces in the eight-neighborhood of p to the number of faces collected is less than 0.375, p is considered an outlier and is filtered out. Finally, the face is extracted to obtain a dense 3D point cloud model.

[0071] The position of the point cloud represents the spatial position of the object surface. In Poisson reconstruction, given a region M and its boundary Define an indicator function X m , the X function value belonging to region M is set to 1, otherwise it is set to zero, and the boundary reconstruction problem is transformed into reconstructing the indicator function.

[0072] S6 adopts an implicit fitting method to obtain the implicit equation represented by the surface information described by the point cloud model by solving the Poisson equation. The reconstruction of the thoracic skeleton three-dimensional model is completed through the main steps of defining an octree, selecting spatial basis functions, solving the Poisson equation, and surface extraction. The specific process is the same as the existing technology and will not be repeated here.

[0073] The color, texture and other information are used to perform subsequent processing on the thoracic skeleton 3D model: including clearing the unwrapped points or surfaces in the Poisson reconstruction, smoothing the image according to the texture descriptor, thereby enhancing the visual effect of the skeletal 3D reconstruction model result, and finally obtaining a thoracic skeleton 3D model with clear surface texture.

[0074] like Figure 2 As shown, a system for implementing the above-mentioned thoracic skeleton three-dimensional construction method includes:

[0075] An image input module, used for inputting X-ray images;

[0076] Model output module, used to output the completed three-dimensional model;

[0077] Camera calibration module, used to obtain the three-dimensional coordinates of the scene;

[0078] An analysis module for analyzing X-ray image information and the camera posture of the X-ray device;

[0079] The generation module builds the model according to the analysis results of the analysis module;

[0080] The correction module corrects the image information during the model building process.

[0081] Doctors only need to import the patient's scanned X-rays into the system to obtain a three-dimensional model of the patient's chest bones.

[0082] Specific usage process: select the save path of the input image and the output result, perform feature matching and ISFM algorithm to obtain sparse point cloud; then start dense reconstruction, the models in the output file stores the dense point cloud in ply format, where txt stores the matrix information of each image: bundle.rd.out stores the sfm output file; then perform Poisson reconstruction, and finally get the obj format 3D model file, that is, the final reconstructed skeletal 3D model file, which can be viewed with the help of Meshlab. The actual 3D model effect is as follows Figure 3 shown.

Claims

1. A method for constructing 3D thoracic skeleton based on multi-viewpoint disordered X-rays, characterized in that: include: S1 inputs an initial image group, wherein the initial image group is obtained by scanning and photographing the chest cavity using an X-ray device, and the initial image group includes X-ray images generated by scanning at multiple angles; S2 preprocesses the initial image group and builds a camera model based on the X-ray device to determine the internal and external parameters of the camera; S3 extracts features from the X-ray image and performs feature point matching. The feature extraction uses the AKAZE feature algorithm and constructs the scale space of the thoracic bones through nonlinear diffusion filtering. During the feature extraction process, the weight of the gradient value of the descriptor LDB in the vertical direction is adjusted to the maximum; S4 uses the motion recovery structure algorithm to construct a sparse point cloud model. The specific process includes: S4.1 calibrates the camera, solves the intrinsic parameter matrix, extracts feature points and performs feature matching on every two images; S4.2 Select the group with the most matching points as the starting image, and solve the essential matrix or basic matrix according to the epipolar geometry, where the essential matrix refers to the relationship between different cameras in the camera coordinate system, and the basic matrix refers to the relationship in the image coordinate system; S4.3 triangulate the feature points matched by the above two images, continuously add new images, and perform 3D-2D matching through the feature points matched by the new images; S4.4 According to the BA algorithm, nonlinear optimization is performed on all camera poses and the three-dimensional coordinates of the thoracic skeleton in space to minimize the error obtained; S5 extracts geometric information from the sparse point cloud model and constructs a dense point cloud model; S6 obtains a 3D model of the thoracic skeleton with color, texture and structural characteristics through post-processing.

2. The method for constructing a thoracic skeleton in three dimensions according to claim 1, characterized in that: The X-ray images are obtained by scanning and photographing the patient in two postures, supine and vertically lying, at least once every 20°, and at least once every 15° in a multi-detail position.

3. The method for constructing a thoracic skeleton in three dimensions according to claim 1, characterized in that: The preprocessing in S2 includes: performing high-frequency modulation filtering and histogram equalization on the X-ray image, and performing low-order bit layer zeroing processing on the X-ray image after bit decomposition.

4. The method for constructing a thoracic skeleton in three dimensions according to claim 1, characterized in that: The post-processing in S6 adopts an implicit fitting method, specifically: S6.1 Obtain the implicit equation representing the surface information described by the point cloud model by solving the Poisson equation; S6.2 By performing isosurface extraction on the equation, a surface model with geometric entity information is obtained.

5. A system for implementing the method for constructing a thoracic skeleton in three dimensions according to any one of claims 1 to 4, comprising: An image input module, used for inputting X-ray images; Model output module, used to output the completed three-dimensional model; Camera calibration module, used to obtain the three-dimensional coordinates of the scene; An analysis module for analyzing X-ray image information and the camera posture of the X-ray device; The generation module builds the model according to the analysis results of the analysis module; The correction module corrects the image information during the model building process.

Images