Mapping method, device and imaging system for multi-angle two-dimensional images and three-dimensional models

Abstract

This invention provides a method for mapping multi-angle two-dimensional images to three-dimensional models, comprising: pre-generating two-dimensional reference keypoint sets based on original two-dimensional projected images from different angles; performing synchronous stitching and fusion processing on the original two-dimensional projected images and corresponding two-dimensional reference keypoints to obtain a stitched two-dimensional panoramic image and the coordinates of the deformed keypoint set; constructing a projection deformation field based on a comparison of the coordinates of the two-dimensional reference keypoints before and after stitching; receiving target points marked on the stitched two-dimensional panoramic image and using the projection deformation field to inversely map the target points back to the target two-dimensional coordinate positions in at least one original two-dimensional projected image; calculating the target three-dimensional coordinate positions of the target points in three-dimensional space based on the target two-dimensional coordinate positions and the corresponding imaging geometric calibration parameters. This establishes a reversible mapping relationship between the two-dimensional coordinates before and after stitching, thereby achieving accurate inversion from the stitched marked points to three-dimensional space.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more particularly to a method, apparatus and imaging system for mapping multi-angle two-dimensional images to three-dimensional models. Background Technology

[0002] CBCT (Cone Beam Computed Tomography) uses cone-beam projection imaging. When anatomical structures deviate from the imaging rotation center, due to ray divergence, the structures exhibit varying degrees of magnification at different viewpoints, causing spatially related proportional distortion. In 3D reconstruction, geometric correction models can eliminate this magnification difference. However, in multi-angle 2D stitched DR images, the image stitching process destroys the original projection geometry, rendering the gantry geometric correction file generated during single-loop CBCT acquisition invalid.

[0003] Clinically, measurements of the entire skeletal system, such as the lower limb HKA angle, acetabular anteversion angle, and spinal Cobb angle, often rely on long-format DR stitched images. However, due to geometric distortions caused by shearing, nonlinear distortion, and discontinuous scaling during the stitching process, the annotations on the stitched 2D images cannot be correctly extrapolated back to 3D space, resulting in measurement errors as high as several millimeters or even degrees. Although existing solutions, such as the EOS system, employ biorthogonal dual-view imaging for approximate 3D reconstruction, are expensive and cannot be applied to existing CBCT systems, nor can they be used for geometric error correction of stitched images. Summary of the Invention

[0004] The purpose of this invention is to provide a method, device and imaging system for mapping multi-angle two-dimensional images to three-dimensional models. By establishing a reversible mapping relationship between two-dimensional coordinates before and after image stitching, the accurate inversion of the labeled points of the stitched two-dimensional image to three-dimensional space can be achieved.

[0005] The technical solution provided by this invention is as follows: In a first aspect, the present invention provides a method for mapping multi-angle two-dimensional images to three-dimensional models, comprising the following steps: Based on the patient's original two-dimensional projection images from different angles, two-dimensional reference key point sets are pre-generated respectively; The original two-dimensional projection image and the corresponding two-dimensional reference key points are subjected to synchronous stitching and fusion processing to obtain the stitched two-dimensional panoramic image and the coordinates of the set of key points after deformation. Based on the comparison of the coordinates of the two-dimensional reference key points before and after splicing, a projection deformation field describing the geometric deformation in the two-dimensional plane during the splicing process is constructed. Receive the target points marked on the stitched two-dimensional panoramic image, and use the projection deformation field to inversely map the target points back to the target two-dimensional coordinate positions in at least one original two-dimensional projection image; Based on the target's two-dimensional coordinate position and the corresponding imaging geometric calibration parameters, the target point's three-dimensional coordinate position in three-dimensional space is calculated.

[0006] In some implementations, the step of pre-generating a set of two-dimensional reference key points in multiple original two-dimensional projection images of the patient from different angles includes the following steps: Obtain a sparse point cloud P of the standard three-dimensional model of the patient, wherein the sparse point cloud P contains a set of two-dimensional reference key points.

[0007] In some implementations, the step of performing synchronous stitching and fusion processing on the original two-dimensional projected image and the corresponding two-dimensional reference key points to obtain the stitched two-dimensional panoramic image and the coordinates of the deformed set of key points includes the following steps: Based on the patient's original two-dimensional projection image and the sparse point cloud P, the specific three-dimensional point cloud S is generated through feature detection and three-dimensional back projection processing. Each three-dimensional point in the specific three-dimensional point cloud S is projected onto the detector plane at the corresponding angle using a perspective projection algorithm to obtain the corresponding two-dimensional coordinates; The two-dimensional coordinates of the three-dimensional points are transformed into the final stitched panoramic image coordinate system using a stitching algorithm to obtain the coordinates of the deformed key point set.

[0008] In some implementations, the step of generating a patient-specific 3D point cloud S based on the patient's original 2D projection image and the sparse point cloud P through feature detection and 3D backprojection processing includes the following steps: Two-dimensional feature points corresponding to the sparse point cloud P are detected from the original two-dimensional projected image using a deep learning network or a model-based registration algorithm. By utilizing the geometric parameters of the imaging system or through the principle of multi-view triangulation, the two-dimensional feature points are back-projected into three-dimensional space to form a specific three-dimensional point cloud S for the patient.

[0009] In some implementations, receiving the target points marked on the stitched 2D panoramic image and using the projection deformation field to inversely map them back to the target's 2D coordinate position in at least one original 2D projected image includes the following steps: Find the nearest points of the target point in the specific three-dimensional point cloud S; The target two-dimensional coordinate position corresponding to each three-dimensional coordinate point in the specific three-dimensional point cloud S is obtained by interpolation calculation based on the projection deformation field.

[0010] In some implementations, the construction of a projected deformation field describing the geometric deformation in the two-dimensional plane during the splicing process, based on a comparison of the coordinates of the two-dimensional reference key points before and after splicing, includes the following steps: A projection mapping relationship is established from the specific three-dimensional point cloud S to the corresponding coordinate position in the stitched two-dimensional panoramic image to obtain the three-dimensional to two-dimensional projection deformation field.

[0011] In some implementations, establishing the projection mapping relationship from the specific 3D point cloud S to the corresponding coordinate positions in the stitched 2D panoramic image to obtain the 3D-to-2D projection deformation field includes the following steps: For each original two-dimensional projection image viewpoint, calculate the transformation relationship that maps the specific three-dimensional point cloud S to the corresponding position in the stitched two-dimensional panoramic image; For a target point in three-dimensional space, the two-dimensional projection position of the target point under this view is obtained by finding the neighboring points in the specific three-dimensional point cloud S and interpolating the transformation relationship corresponding to the neighboring points, so as to construct the three-dimensional to two-dimensional projection deformation field.

[0012] In some implementations, calculating the target point's three-dimensional coordinate position in three-dimensional space based on the target's two-dimensional coordinate position and the corresponding imaging geometric calibration parameters includes the following steps: Based on the target's two-dimensional coordinate position and the imaging geometric calibration parameters, the target point's three-dimensional coordinate position in three-dimensional space is calculated using a back-projection algorithm or a multi-view geometric inversion algorithm.

[0013] The imaging geometric calibration parameters include the geometric constraints provided by the projection deformation field or the specific three-dimensional point cloud S.

[0014] In some implementations, the present invention also provides a mapping device for multi-angle two-dimensional images and three-dimensional models, comprising: The generation module is used to pre-generate sets of two-dimensional reference key points based on the patient's original two-dimensional projection images from different angles. The processing module is used to perform synchronous stitching and fusion processing on the original two-dimensional projection image and the corresponding two-dimensional reference key points to obtain the stitched two-dimensional panoramic image and the coordinates of the set of key points after deformation. The construction module is used to construct a projection deformation field describing the geometric deformation in the two-dimensional plane during the splicing process, based on the comparison of the coordinates of the two-dimensional reference key points before and after splicing. The inverse mapping module is used to receive the target points marked on the stitched two-dimensional panoramic image and use the projection deformation field to inversely map the target points back to the target two-dimensional coordinate positions in at least one original two-dimensional projection image. The calculation module is used to calculate the target point's three-dimensional coordinate position in three-dimensional space based on the target's two-dimensional coordinate position and the corresponding imaging geometric calibration parameters.

[0015] In some implementations, the present invention also provides an imaging system, including the aforementioned mapping device for multi-angle two-dimensional images and three-dimensional models.

[0016] This invention explicitly restores the actual geometric deformation introduced into the two-dimensional image space by using the stitching and fusion algorithm, and establishes a reversible mapping relationship between the two-dimensional coordinates before and after stitching, thereby achieving accurate inversion of the stitched annotation points to the three-dimensional space. Attached Figure Description

[0017] The preferred embodiments will be described below in a clear and easy-to-understand manner, with reference to the accompanying drawings, to further explain the above-mentioned characteristics, technical features, advantages, and implementation methods of a mapping method and system for multi-angle two-dimensional images and three-dimensional models.

[0018] Figure 1 This is a schematic diagram of an embodiment of a mapping method between multi-angle two-dimensional images and three-dimensional models according to the present invention; Figure 2 This is a flowchart of an embodiment of a mapping system between multi-angle two-dimensional images and three-dimensional models according to the present invention. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or sets.

[0021] To keep the drawings concise, only the parts relevant to the invention are shown schematically in each figure, and they do not represent the actual structure of the product. Furthermore, for ease of understanding, in some figures, only one of components with the same structure or function is shown schematically, or only one is labeled. In this document, "one" can mean not only "only one" but also "more than one".

[0022] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0025] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0026] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0027] Gantry geometric correction files are like personalized error maps for a precision instrument (CT / Linac). They don't alter the physical errors of the hardware itself, but rather compensate in real-time through software algorithms, serving as a crucial bridge between "imperfect physical reality" and "perfect digital imaging / treatment." They are essential for ensuring the safety, effectiveness, and diagnostic / treatment quality of medical equipment. It is a data file (usually plain text or a specific format like .dat or .xml) containing a series of calibration parameters. It is used to compensate for and correct minute geometric errors in the gantry rotation system that occur during mechanical manufacturing, installation, and long-term use. It ensures that the positional relationships of key components such as the gantry rotation center, X-ray tube, and detectors in three-dimensional space perfectly match the design ideal values, thereby guaranteeing: Imaging quality: artifact-free images, high spatial resolution, and accurate distance measurements. Treatment accuracy: For radiotherapy equipment (such as linear accelerators), it ensures that the radiation beam accurately hits the tumor target area while protecting normal tissue.

[0028] Anterior tilt of the acetabulum: describes the direction of the acetabular opening plane in the horizontal plane (cross section). It refers to the angle between the acetabular opening plane and the coronal plane (anteroposterior plane) of the human body in the cross section (horizontal plane).

[0029] The Cobb angle is the standard measurement method used to quantify the severity of scoliosis on X-rays. It measures the coronal (anteroposterior) angle of a specific curvature (lateral curvature) of the spine, that is, the angle by which the spine deviates from the midline to the left or right. It is the internationally recognized gold standard for diagnosing scoliosis, assessing its progression, and determining treatment options.

[0030] In cone-beam computed tomography (CBCT) and multi-angle digital radiography (DR) imaging systems, 3D reconstruction typically relies on a calibrated imaging geometry model, using methods such as back projection to restore 2D projected coordinates to 3D spatial coordinates. However, in practical clinical applications, a single scan or single-circle imaging can only cover a local area of ​​the human body. To obtain images of the whole body in a standing position or of long bone structures, it is usually necessary to longitudinally stitch and fuse local 2D projected images obtained from multiple imaging sessions.

[0031] Because the projection magnification in cone-beam imaging is related to the spatial distance between the imaged object and the X-ray source, two-dimensional projection images obtained from different shooting positions have inherent scale differences in overlapping areas. Existing stitching and fusion algorithms often introduce nonlinear scaling, distortion, or displacement transformations into local regions to obtain a visually continuous overall image.

[0032] Specifically, CBCT has a cone-beam magnification effect. If a key object is not in the center of the image, its magnification will vary depending on its distance from the flat-panel radiation source, resulting in different magnifications in multi-angle fluoroscopic imaging. For example, in a test image, the magnification of the left femur may differ between the lateral and anteroposterior views. In 3D imaging, geometric correction can minimize the measurement error caused by this magnification difference. In multi-angle 2D imaging, measurements of the same point in space can be affected by different magnifications, preventing the coordinates from converging at different viewpoints.

[0033] In clinical practice, doctors often prefer to annotate and measure skeletal feature points only on the stitched overall 2D image, rather than manipulating multiple original local images. However, existing 3D measurement and modeling methods rely on the precise coordinate information of the original 2D image before stitching, creating a significant gap between the two. Furthermore, the stitched 2D image no longer strictly conforms to any single imaging geometry model, resulting in the inability to directly map the position of any point in the stitched 2D image back to its position in the original 2D projection image, and consequently, the inability to accurately invert its 3D spatial coordinates based on geometric calibration relationships.

[0034] To facilitate observation by doctors, they will not look at the individual images before stitching, but will directly view the stitched multi-angle 2D full-length image. However, because of the stitching process, the stitched multi-angle 2D full-length image has lost the geometric correction parameters of the original single-loop shooting, making it impossible to achieve accurate 3D coordinate regression of points.

[0035] To enable doctors to see exactly what they want, a spatial relationship must be established between the positional relationships of the 2D images before stitching and the multi-angle, full-length 2D images after stitching. Simple 2D-3D single-circle multi-angle photography, conforming to the spatial perspective relationships defined in the geometric correction file, can use simulation projection software to establish the positional relationships of points in 3D space relative to the 2D perspective images at various angles. However, the stitching algorithm disrupts this perspective relationship defined in the geometric file. Therefore, if a spatial relationship needs to be established between 3D space and the stitched 2D images, a non-rigid deformation field needs to be defined to reflect the complex spatial mapping relationships after perspective, stitching, and fusion.

[0036] Reference Appendix Figure 1 This invention provides a method for mapping multi-angle two-dimensional images to three-dimensional models, specifically including the following steps: S101 Based on the original two-dimensional projection images of the patient from different angles, pre-generate sets of two-dimensional reference key points respectively; S102 Perform synchronous stitching and fusion processing on the original two-dimensional projection images and the corresponding two-dimensional reference key points to obtain the stitched two-dimensional panoramic image and the coordinates of the deformed key point set; S103 Based on the comparison of the coordinates of the two-dimensional reference key points before and after stitching, construct a projection deformation field describing the geometric deformation in the two-dimensional plane during the stitching process; S104 Receive the target point marked on the stitched two-dimensional panoramic image, and use the projection deformation field to inversely map the target point back to the target two-dimensional coordinate position in at least one original two-dimensional projection image; S105 Based on the target two-dimensional coordinate position and the corresponding imaging geometric calibration parameters, calculate the target three-dimensional coordinate position of the target point in three-dimensional space.

[0037] This invention relates to the fields of medical image processing, orthopedic measurement, three-dimensional reconstruction and deep learning algorithms. It provides a technique for establishing a precise mapping relationship between multi-angle 2D stitched full-length images and three-dimensional skeletal models, thereby achieving a reversible conversion between the labeled points of the stitched 2D images and the real three-dimensional coordinates.

[0038] In one embodiment, the present invention also provides: 1. Key point layout before splicing In each original 2D projection image before stitching, a set of 2D reference key points is pre-generated.

[0039] Two-dimensional reference key points can be generated from two-dimensional grid points formed by rule sampling or from automated image analysis methods, and can be used as geometric reference markers.

[0040] 2. Key points are stitched together synchronously with the image. When performing 2D image stitching and fusion, 2D reference key points are input into the stitching algorithm along with the corresponding image as an independent layer or additional information, so that the 2D reference key points are scaled, distorted and shifted synchronously with the image content.

[0041] 3. Deformation Field Construction By comparing the two-dimensional coordinate positions of the same key point before and after splicing, the local geometric transformation relationship introduced in the two-dimensional plane during the splicing process is calculated, thereby constructing the two-dimensional deformation field corresponding to the splicing algorithm.

[0042] The deformation field explicitly records the actual geometric deformation applied to a two-dimensional image by the stitching and fusion algorithm (whether it is a rule-based algorithm or an AI-based algorithm).

[0043] 4. Inverse mapping of the labeled points after splicing When a doctor marks any target point on the stitched and fused two-dimensional image, based on the point's position in the deformation field, the marked point is mapped back to its corresponding coordinate position in each of the original two-dimensional projection images before stitching through the inverse transformation relationship of the deformation field.

[0044] For the annotation points located between key points, their displacement is obtained by interpolation using the deformation parameters of the adjacent key points.

[0045] 5. Three-dimensional spatial inversion Based on the two-dimensional projection coordinates obtained before stitching and the corresponding imaging geometric calibration parameters, the precise position of the marked points in three-dimensional space is inferred using back projection or geometric inversion methods, which can be used for three-dimensional vector calculation, distance measurement or skeleton modeling.

[0046] The method of the present invention establishes a mapping from a 3D template to a multi-angle 2D stitched image, thereby realizing the reverse calculation from 2D annotation points to 3D coordinates, effectively restoring the lost depth information in the stitched image, and improving the three-dimensional spatial accuracy of 2D image measurement.

[0047] In one embodiment, the present invention proposes a method for constructing a 3D-to-2D non-rigid mapping field constrained by sparse point clouds, comprising: 1. Construct a standard sparse point cloud model P for the skeleton, including a set of anatomical key points such as the hip, knee, and ankle.

[0048] Specifically, a sparse point cloud P of the patient's standard three-dimensional model is obtained, wherein the sparse point cloud P contains a set of two-dimensional reference key points.

[0049] 2. By combining multi-angle CBCT fluoroscopic images with deep learning keypoint detection and geometric backprojection, the patient-specific point cloud S is reconstructed.

[0050] Specifically, based on the patient's original two-dimensional projection image and the sparse point cloud P, the specific three-dimensional point cloud S is generated through feature detection and three-dimensional backprojection processing.

[0051] Specifically, two-dimensional feature points corresponding to the sparse point cloud P are detected from the original two-dimensional projected image using a deep learning network or a model-based registration algorithm. These two-dimensional feature points are then back-projected into three-dimensional space using the geometric parameters of the imaging system or through multi-view triangulation principles to form a patient-specific three-dimensional point cloud S.

[0052] 3. Calculate the projection of S from each viewpoint and perform image stitching to generate point cloud coordinates. ~ .

[0053] Specifically, each three-dimensional point in the specific three-dimensional point cloud S is projected onto the detector plane at the corresponding angle using a perspective projection algorithm to obtain the corresponding two-dimensional coordinates; the two-dimensional coordinates of the three-dimensional points are then transformed into the final stitched panoramic image coordinate system using an application stitching algorithm to obtain the coordinates of the deformed key point set.

[0054] 4. Construct a non-rigid spatial deformation field M using a Neural Warp Field (based on an MLP implicit field) such that M(S)=E holds true.

[0055] Specifically, for each of the original two-dimensional projected images, a transformation matrix is ​​calculated to map the points in the specific three-dimensional point cloud S to the corresponding positions in the stitched two-dimensional panoramic image.

[0056] 5. Use a local interpolation structure to realize the two-dimensional mapping Z=M(Y) of any three-dimensional point Y.

[0057] Specifically, for any point in three-dimensional space, by finding its neighboring points in the specific three-dimensional point cloud S and interpolating the transformation relationship corresponding to the neighboring points, the target two-dimensional coordinate position corresponding to each three-dimensional coordinate point in the specific three-dimensional point cloud S is obtained.

[0058] 6. Algorithms or doctors annotate multiple points on the stitched images. ~ The unique three-dimensional point Y is solved by minimizing the projection error.

[0059] Specifically, based on the target's two-dimensional coordinate position and the imaging geometric calibration parameters, the target point's three-dimensional coordinate position in three-dimensional space is calculated using a back-projection algorithm or a multi-view geometric inversion algorithm.

[0060] The imaging geometric calibration parameters include the geometric constraints provided by the projection deformation field or the specific three-dimensional point cloud S.

[0061] The specific method in this embodiment is described below: 1. Define a sparse point cloud P for the full-body 3D skeletal model. This point cloud represents the anatomical feature points and geometric points on the skeletal model, which can outline the shape of the skeleton.

[0062] Specifically, from a standard full-body 3D skeletal model (such as the average model reconstructed from CT), a set of key points is selected to form a sparse point cloud P. These points include anatomical feature points and geometric points, which together delineate the morphological contour of the skeleton. Delineating the skeletal morphology means that the contour and surface morphology of the skeleton can be approximately restored by connecting these points or by performing surface interpolation based on the point cloud (such as Delaunay triangulation and radial basis function reconstruction).

[0063] The point cloud P can be defined by manual annotation by experts or by automatic feature detection algorithms (such as methods based on curvature and shape context).

[0064] For example, anatomical feature points refer to points with clear anatomical significance, such as joint centers (femoral head center) and bony landmarks (femoral condyles, tibial tuberosities, etc.). Geometric points refer to points that describe the geometric features of bone morphology, such as extreme points of contour curvature, endpoints of axes, and surface concavity / convexity points.

[0065] 2. These skeletal feature points are detected from the single-loop 2D image using registration or deep learning methods, and the specific skeletal feature point cloud S of the current patient is calculated using a back-projection algorithm. Here, skeletal feature points refer to anatomical landmarks visible on the 2D image.

[0066] For example, first, acquire 2D X-ray images of the current patient from multiple angles (such as different angle segments of a full-length lower limb radiograph). Then, perform feature point detection: use registration methods or deep learning methods to detect corresponding 2D feature points in the images at each angle. Then, calculate the patient-specific 3D sparse point cloud S using a back-projection algorithm. Alternatively, obtain the deformed point cloud S directly by non-rigidly registering a 3D template to the 2D image.

[0067] Feature point detection via registration refers to model-based registration, projecting a 3D template onto a 2D image, and optimizing transformation parameters to match the projected contour with the image edges, thereby detecting feature points. Feature point detection via deep learning refers to using a convolutional neural network (CNN) to directly regress the coordinates of skeletal feature points in a 2D image.

[0068] The back-projection algorithm of this application for calculating a specific 3D point cloud S includes the following steps: using the geometric parameters of the imaging system (X-ray source position, detector attitude, etc.), the 2D feature points detected from multiple angles are used to calculate the corresponding 3D points through triangulation or a back-projection algorithm (such as direct linear transformation DLT) to form a patient-specific 3D point cloud S. If only a single view is available, back-projection can be performed in conjunction with template deformation constraints (such as statistical shape models).

[0069] 3. Using projection and stitching algorithms, calculate the coordinates of each point in S in the 2D full-length map at various angles after projection and stitching: arrive , where n represents different projection angles.

[0070] Specifically, the projection stitching coordinates at each angle are calculated: For each projection angle n (corresponding to a segment of the stitched image), the focal position of the X-ray source, the detector plane, and the stitching transformation (obtained through system calibration or image registration) are known. Each 3D point in the point cloud S is projected onto the detector plane at that angle using a perspective projection algorithm to obtain 2D coordinates; then, a stitching algorithm (such as translation, rotation, scaling, etc.) is applied to transform the point into the final stitched panoramic image coordinate system, resulting in a point set. Stitching algorithms may involve the alignment and fusion of multiple segments, and specific methods include feature-based matching (such as SIFT, ORB) or phase-correlation-based image registration.

[0071] 4. From 3D S to each A transformation matrix M from 3D point cloud to 2D point cloud can be calculated to realize complex projection transformations from 3D space to 2D projection, stitching, and fusion.

[0072] Specifically, solve for the transformation matrix from 3D to 2D. For each angle n, using the point cloud S and the corresponding 2D point set It is possible to estimate a transformation matrix from 3D to 2D. In homogeneous coordinates, this transformation can be expressed as: = * ,in Let n be the projection matrix (3×4). To construct a homography matrix (3×3). The solution can be obtained from at least 6 pairs of corresponding points using the Direct Linear Transformation (DLT) algorithm.

[0073] 5. For any point Y on the skeleton, it lies between four adjacent points in the point cloud S. If we assume that M is a deformation field from S to E, then obviously the spatial deformation of Y can be obtained by interpolation of M with respect to its neighboring S.

[0074] Specifically, establish a continuous deformation field: This is considered as a deformation field from 3D to 2D. For any 3D point Y on the skeleton that is not in S, find its four nearest neighbors in the point cloud of S (forming a tetrahedron), and calculate the centroid coordinates of Y within this tetrahedron. Then, use these centroid coordinates to perform weighted interpolation on the transformation of the neighboring points (i.e., their 2D projected coordinates) to obtain the 2D projection of Y at angle n. In this way, the deformation field is extended to the entire bone surface.

[0075] 6. Therefore, any 3D point Y can also be projected onto the 2D mosaic image via M (even though Y does not belong to S). Clearly, due to the special nature of 3D-2D projection, the relationship between the 2D coordinate point Z and any 3D point Y is not unique; multiple arbitrary 3D points Y can yield the same 2D coordinate point Z. However, if we consider 2D coordinate points Z at multiple angles, a unique 3D point Y in space can be determined through a reverse algorithm.

[0076] Specifically, multi-angle reverse localization of 3D points: Due to the ambiguity of single-angle projection, the same 2D point may correspond to all 3D points on a single projection line. Therefore, it is necessary to combine multi-angle information. If the doctor marks the 2D location of the same anatomical point on a stitched full-length image from multiple angles (at least two),... For each angle, a ray in 3D space can be obtained through the inverse (or back projection) of the deformation field (or by inferring possible 3D points through interpolation of the deformation field). Then, a unique 3D point Y is determined by inverse algorithms of multi-view geometry (such as least squares intersection).

[0077] Therefore, when an algorithm or doctor marks any point Z (from multiple angles) in the stitched 2D full-length image, we can obtain the near-real coordinates of the 3D point Z in 3D using this 3D-2D "deformation field." This eliminates the disruption of geometric relationships caused by the complex transformation of stitching. The technical solution in this embodiment supports 3D reconstruction of arbitrarily stitched DR images, achieving WYSIWYG (What You See Is What You Get) and has real-time performance.

[0078] In a real-world scenario, the patient stands with weight-bearing on both feet, and 120°–180° continuous fluoroscopic imaging is performed using a rotating CBCT. A keypoint detection network (such as HRNet / YOLOv8-Pose) is used to extract anatomical keypoints from each viewpoint; a 3D point cloud S is solved using multi-view geometric consistency backprojection; the projected images are stitched together to generate the final full-length 2D image, while simultaneously recording the S→E correspondence. Then, a neural implicit deformation field is used to fit the mapping M from S to E; the 3D coordinates of the points marked by the doctor on the stitched DR image are recovered through inverse 3D coordinate calculation; the 3D point coordinates are output, and clinical indicators such as the HKA angle and acetabular anteversion angle are automatically calculated.

[0079] The embodiments of the present invention can be applied to the following practical scenarios: full-length lower limb force line analysis (HKA, FTA, MAD), preoperative planning of total hip THA (acetabular anteversion and lordosis compensation analysis), full-length Cobb angle analysis and rotation assessment of the spine, 2D-3D surgical navigation system, and CBCT real-time posture compensation and alignment guidance.

[0080] The method of the present invention achieves a precise correlation between the stitched two-dimensional image and the original three-dimensional geometric model without relying on a specific stitching and fusion algorithm, enabling doctors to complete annotation operations on the overall two-dimensional image while meeting the needs of high-precision three-dimensional measurement and modeling.

[0081] The following describes specific methods for applying this invention in practical scenarios: 1. Preoperative 3D → stitched 2D prediction Given the patient's preoperative three-dimensional image model and imaging geometric parameters, the three-dimensional model can be projected to generate a two-dimensional projection image at the corresponding angle. Then, using the aforementioned deformation field model, its actual display position in the stitched and fused two-dimensional image can be predicted.

[0082] The method of this invention can be used in the preoperative stage to directly map the three-dimensional planning results onto the intraoperative or preoperative stitched two-dimensional images, realizing the visual alignment of three-dimensional information in the two-dimensional stitched images, and providing an intuitive reference for preoperative assessment and intraoperative decision-making.

[0083] 2. Surgical navigation By combining the stitched results of two-dimensional projection images acquired in real time or during surgery with the pre-constructed deformation field and its inverse mapping relationship, the real-time annotations or instrument positions made by doctors on the stitched two-dimensional images can be accurately mapped to a three-dimensional spatial coordinate system.

[0084] The technology of this invention can be used to construct a surgical navigation system based on stitched two-dimensional images, achieving precise positioning, path planning, and safe distance calculation in three-dimensional space without changing the doctor's two-dimensional operating habits.

[0085] 3. Standard skeleton template mapping By projecting a standard 3D bone template onto a multi-angle 2D space and combining it with the deformation field model obtained during the stitching process, the standard bone template can be accurately registered with the stitched 2D image of the patient.

[0086] The method of the present invention can be used for bone morphology alignment, force line analysis, parameter standardization measurement, and population statistical analysis, enabling standard bone models to maintain geometric consistency in two-dimensional images with splicing deformation.

[0087] This invention proposes a mapping method between a three-dimensional skeletal model based on a non-rigid neural warp field and a multi-angle stitched 2D image, which realizes image distortion compensation and reversible recovery of three-dimensional coordinates. This allows doctors to directly mark points on the stitched image to obtain its true three-dimensional coordinate values, eliminates the destruction of geometric relationships during the stitching process, and achieves millimeter-level accuracy similar to the EOS system.

[0088] Reference Appendix Figure 2 The present invention also provides a mapping device for multi-angle two-dimensional images and three-dimensional models, comprising: The generation module 101 is used to pre-generate a set of two-dimensional reference key points based on the patient's original two-dimensional projection images from different angles. Processing module 102 is used to perform synchronous stitching and fusion processing on the original two-dimensional projection image and the corresponding two-dimensional reference key points to obtain the stitched two-dimensional panoramic image and the coordinates of the set of key points after deformation. Module 103 is used to construct a projection deformation field describing the geometric deformation in the two-dimensional plane during the splicing process based on the comparison of the coordinates of the two-dimensional reference key points before and after splicing. The inverse mapping module 104 is used to receive the target points marked on the stitched two-dimensional panoramic image and use the projection deformation field to inversely map the target points back to the target two-dimensional coordinate positions in at least one original two-dimensional projection image. The calculation module 105 is used to calculate the target point's three-dimensional coordinate position in three-dimensional space based on the target's two-dimensional coordinate position and the corresponding imaging geometric calibration parameters.

[0089] Based on the above embodiments, the parts that are the same as those in the above embodiments will not be described again in this embodiment.

[0090] In one embodiment, the present invention also provides an imaging system, including the aforementioned mapping device for multi-angle two-dimensional images and three-dimensional models.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of program modules is merely an example. In practical applications, the above functions can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program units or modules to complete all or part of the functions described above. The program modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software program unit. Furthermore, the specific names of the program modules are only for easy differentiation and are not intended to limit the scope of protection of this application.

[0092] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0094] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces, and the indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0097] It should be noted that the above embodiments can be freely combined as needed. The above description is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for mapping multi-angle two-dimensional images to three-dimensional models, characterized in that, Including the following steps: Based on the patient's original two-dimensional projection images from different angles, two-dimensional reference key point sets are pre-generated respectively; The original two-dimensional projection image and the corresponding two-dimensional reference key points are subjected to synchronous stitching and fusion processing to obtain the stitched two-dimensional panoramic image and the coordinates of the set of key points after deformation. Based on the comparison of the coordinates of the two-dimensional reference key points before and after splicing, a projection deformation field describing the geometric deformation in the two-dimensional plane during the splicing process is constructed. Receive the target points marked on the stitched two-dimensional panoramic image, and use the projection deformation field to inversely map the target points back to the target two-dimensional coordinate positions in at least one original two-dimensional projection image; Based on the target's two-dimensional coordinate position and the corresponding imaging geometric calibration parameters, the target point's three-dimensional coordinate position in three-dimensional space is calculated.

2. The mapping method between multi-angle two-dimensional images and three-dimensional models according to claim 1, characterized in that, The step of pre-generating a set of two-dimensional reference key points in multiple original two-dimensional projection images of the patient from different angles includes the following steps: Obtain a sparse point cloud of a standard three-dimensional model of the patient, the sparse point cloud containing a set of two-dimensional reference key points.

3. The mapping method between multi-angle two-dimensional images and three-dimensional models according to claim 2, characterized in that, The process of simultaneously stitching and fusing the original two-dimensional projected image and its corresponding two-dimensional reference key points to obtain the stitched two-dimensional panoramic image and the coordinates of the deformed set of key points includes the following steps: Based on the patient's original two-dimensional projection image and the sparse point cloud, the specific three-dimensional point cloud is generated through feature detection and three-dimensional back projection processing. Each three-dimensional point in the specific three-dimensional point cloud is projected onto the detector plane at the corresponding angle using a perspective projection algorithm to obtain the corresponding two-dimensional coordinates; The two-dimensional coordinates of the three-dimensional points are transformed into the final stitched panoramic image coordinate system using a stitching algorithm to obtain the coordinates of the deformed key point set.

4. The mapping method between multi-angle two-dimensional images and three-dimensional models according to claim 3, characterized in that, The process of generating a patient-specific 3D point cloud based on the patient's original 2D projection image and the sparse point cloud through feature detection and 3D backprojection processing includes the following steps: Two-dimensional feature points corresponding to the sparse point cloud are detected from the original two-dimensional projection image using a deep learning network or a model-based registration algorithm. By utilizing the geometric parameters of the imaging system or through the principle of multi-view triangulation, the two-dimensional feature points are back-projected into three-dimensional space to form a specific three-dimensional point cloud for the patient.

5. The mapping method between multi-angle two-dimensional images and three-dimensional models according to claim 5, characterized in that, The step of receiving the target points marked on the stitched 2D panoramic image and using the projection deformation field to inversely map them back to the target's 2D coordinate position in at least one original 2D projection image includes the following steps: Find the nearest points of the target point in the specific 3D point cloud; The target two-dimensional coordinate position corresponding to each three-dimensional coordinate point in the specific three-dimensional point cloud is obtained by interpolation based on the projection deformation field.

6. The mapping method between multi-angle two-dimensional images and three-dimensional models according to claim 5, characterized in that, The construction of a projected deformation field describing the geometric deformation in the two-dimensional plane during the splicing process, based on the comparison of the coordinates of the two-dimensional reference key points before and after splicing, includes the following steps: A projection mapping relationship is established from the specific three-dimensional point cloud to the corresponding coordinate position in the stitched two-dimensional panoramic image to obtain the three-dimensional to two-dimensional projection deformation field.

7. The mapping method between multi-angle two-dimensional images and three-dimensional models according to claim 5, characterized in that, The step of establishing a projection mapping relationship from the specific 3D point cloud to the corresponding coordinate position in the stitched 2D panoramic image to obtain the 3D-to-2D projection deformation field includes the following steps: For each original 2D projection image viewpoint, calculate the transformation relationship that maps the specific 3D point cloud to the corresponding position in the stitched 2D panoramic image; For a target point in three-dimensional space, the two-dimensional projection position of the target point under this view is obtained by finding the neighboring points in the specific three-dimensional point cloud and interpolating the transformation relationship corresponding to the neighboring points, so as to construct the three-dimensional to two-dimensional projection deformation field.

8. The mapping method between multi-angle two-dimensional images and three-dimensional models according to any one of claims 1 to 7, characterized in that, The step of calculating the target point's three-dimensional coordinate position in three-dimensional space based on the target's two-dimensional coordinate position and the corresponding imaging geometric calibration parameters includes the following steps: Based on the target's two-dimensional coordinate position and the imaging geometric calibration parameters, the target point's three-dimensional coordinate position in three-dimensional space is calculated using a back projection algorithm or a multi-view geometric inversion algorithm. The imaging geometric calibration parameters include the geometric constraints provided by the projection deformation field or the specific three-dimensional point cloud.

9. A mapping device for multi-angle two-dimensional images and three-dimensional models, characterized in that, include: The generation module is used to pre-generate sets of two-dimensional reference key points based on the patient's original two-dimensional projection images from different angles. The processing module is used to perform synchronous stitching and fusion processing on the original two-dimensional projection image and the corresponding two-dimensional reference key points to obtain the stitched two-dimensional panoramic image and the coordinates of the set of key points after deformation. The construction module is used to construct a projection deformation field describing the geometric deformation in the two-dimensional plane during the splicing process, based on the comparison of the coordinates of the two-dimensional reference key points before and after splicing. The inverse mapping module is used to receive the target points marked on the stitched two-dimensional panoramic image and use the projection deformation field to inversely map the target points back to the target two-dimensional coordinate positions in at least one original two-dimensional projection image. The calculation module is used to calculate the target point's three-dimensional coordinate position in three-dimensional space based on the target's two-dimensional coordinate position and the corresponding imaging geometric calibration parameters.

10. An imaging system, characterized in that, It includes a mapping device for multi-angle two-dimensional images and three-dimensional models as described in claim 9.

Images