How to render soft tissue into the X-ray modality

Abstract

The present invention relates to a computer-implemented method for rendering soft tissue in an X-ray image, the method comprising the steps of: providing a joint 3D anatomical model of at least one joint of a body, the anatomical model having a rigid structure and at least one type of soft tissue, the rigid structure having at least two bones connected by the joint. The method further comprises the steps of providing an X-ray image of a joint of a patient, the joint of the patient corresponding to the joint of the anatomical model, and registering the anatomical model to the X-ray image. The method further comprises the steps of projecting the at least one type of soft tissue into an image domain of the X-ray image, and visualizing the at least one type of soft tissue in the X-ray image.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for rendering soft tissue in an X-ray image. [Background technology]

[0002] X-ray modalities such as diagnostic X-ray, fluoroscopy, and CT are widely used in clinical practice and have been continuously developed for more than 100 years since the discovery of X-ray. Recent research has aimed at introducing dark field X-ray (DAX) into clinical use, for example to visualize diagnostically relevant changes in the lung parenchyma. Contrast agents can help to increase the radiodensity in target tissues or structures. Summary of the Invention [Problem to be solved by the invention]

[0003] However, many types of tissue remain invisible under x-ray and therefore require additional imaging or surgery to be diagnosed.

[0004] Without visibility of certain tissues, such as cartilage or tendons, skeletal X-ray images and sequences are incomplete, hindering diagnostic confidence. Often, additional scans in other modalities are required to confirm findings or point out discrepancies. This places a burden on patients, who must repeat scans or visit specialized radiology, and on staff, who must repeat scans on the same patients. In addition, it places a burden on healthcare providers and may result in higher machine loads. Ideally, previously undiagnosable pathologies would be directly assessable from routine X-ray images.

[0005] Chen Hsin-Chen et al.'s paper "A Joint-Constraint Model-Based System for Reconstructing Total Knee Motion", IEEE Transactions on Biomedical Engineering, IEEE, USA, vol. 61, no.1, January 2014, pages 171-181, discloses a 2D-3D registration system based on a joint constraint model for reconstructing total knee motion. This model, with the bone geometry and joint mechanics, is iteratively registered to a single-plane fluoroscopic video to obtain a sequence of registered knee joint poses that exhibit a smooth and reasonable physiological pattern of motion.

[0006] Accordingly, the inventors have found that it would be advantageous to have a method for rendering soft tissue in an X-ray image that does not suffer from the above-mentioned drawbacks.

[0007] It is an object of the present invention to provide a method for rendering soft tissue in an X-ray image. [Means for solving the problem]

[0008] The object of the present invention is solved by the subject matter of the independent claims, further embodiments are incorporated in the dependent claims.

[0009] The described embodiments also relate to a computer-implemented method, a data processing apparatus, a computer program, and a computer-readable storage medium for rendering soft tissue in an X-ray image. Although not described in detail, synergistic effects may arise from different combinations of these embodiments.

[0010] Furthermore, it should be noted that all embodiments of the present invention relating to methods can be carried out in the order of steps as described, but that this is nevertheless not required to be the only and essential order of steps of the method. The methods presented herein can be carried out in other orders of steps as disclosed without departing from the respective method embodiments, unless otherwise stated below.

[0011] According to a first embodiment of the present invention, there is provided a computer-implemented method for rendering soft tissue in an X-ray image. The method comprises the steps of providing a three-dimensional anatomical model of at least one joint of a body, the model comprising a rigid structure and at least one type of soft tissue, the rigid structure comprising at least two bones connected by a joint. The method further comprises the steps of providing an X-ray image of a joint of a patient, the patient's joint corresponding to a joint of the anatomical model, and registering the anatomical model to the X-ray image. The method further comprises the steps of projecting the at least one type of soft tissue into an image domain of the X-ray image, and visualizing the at least one type of soft tissue in the X-ray image.

[0012] Thus, according to the method, an articular anatomical model having a set of at least one rigid structure and one non-rigid structure, as well as a medical X-ray image, is provided. The model and the X-ray image preferably cover the same area of ​​the patient's body, in particular the same joint, such as, for example, the knee joint. The anatomical model has at least a rigid structure and at least one type of soft tissue, where the rigid structure may have a number of bones connected to each other by joints. The soft tissue may be considered as a part of the patient's body that is not bone, or at least a part thereof. The soft tissue may be, for example, tendons, ligaments, skin, cartilage, muscle, etc. In particular, according to the invention, the soft tissue may be considered as a tissue having a relatively low X-ray absorption coefficient, so that the soft tissue is not clearly imaged in the X-ray image. Furthermore, the lack of difference in attenuation of different soft tissue types, such as muscle, fat, or cartilage, contributes to making it difficult to image soft tissue in skeletal X-ray imaging. A joint 3D anatomical model can be a computer-implemented representation of at least one joint of a body, the anatomical model including the relative positions and orientations of the bones and soft tissues of the rigid structure relative to each other in the articulation range of the joint. The joint range of motion and hence the postures of the rigid and soft parts of the model can be sampled as sampling / support points across the articulation range of the target anatomical structure, such as from straight leg to full flexion in the case of a knee joint.

[0013] In a step of registering the anatomical model to the X-ray image, the anatomical model can be adapted to the pose of the joints depicted in the X-ray image and the correct position and orientation of the model can be determined so that the rigid structures of the anatomical model correspond to the bones of the X-ray image. In a next step, at least one type of soft tissue is derived from the anatomical model and the soft tissue is forward projected into the image domain of the X-ray image. Thus, the soft tissue can be rendered and / or visualized in the X-ray image together with the bones of the X-ray image. This step can be repeated for multiple images of a time sequence of images in the examination under examination.

[0014] Anatomical 3D models of joints provide a good understanding of how these soft tissue parts look and behave in different poses and offer the possibility to model these soft tissue components using models that include both rigid and non-rigid structures. Registration based on rigid structures visible on radiography and rendering of non-rigid structures barely visible on radiography can form a powerful extension of classical X-ray images in diagnostic, fluoroscopy, tomosynthesis or (cone beam) CT. Images enhanced according to the invention can improve readability, especially for less well-trained radiologists, and improve diagnostic confidence. By mapping a statistical atlas for specific structures of interest onto the X-ray image and performing a local adaptation of the tone or color map, any deviations and anomalies can be visually highlighted.

[0015] The method can be applied, for example, as a software function in quality control, CAD, PACS, viewing stations, educational products, etc. However, the method can be applied to any X-ray imaging modality in 2D or 3D, in static or dynamic imaging.

[0016] In one embodiment of the present invention, the step of registering an anatomical model to an X-ray image comprises the steps of: segmenting the X-ray image to determine the positions of at least two bones in the X-ray image; aligning a rigid structure of the anatomical model with the positions of the at least two bones in the X-ray image by applying a transformation to the anatomical model, the transformation being defined by a rotation, a scaling, and a translation of the anatomical model and an adjustment to the posture of joints of the anatomical model; and determining the position of at least one type of soft tissue of the anatomical model after the transformation has been applied to the anatomical model.

[0017] Thus, the joint 3D model of the target anatomical structure can be statically or dynamically registered to the X-ray image. Segmentation of the bones in the X-ray image can be performed, for example, via model-based segmentation, fully convolutional neural networks, or other known approaches. Aligning the rigid parts of the joint model with the segmented bones can be performed by simultaneously optimizing the rotation, scale, translation, and preferably the joint motion and pose of each joint. In the case of two-dimensional images, this may require pose estimation of the joints in the image.

[0018] Selected soft tissue body parts can be extracted from the joint model and transformed using the same transformation to render such parts directly into the X-ray image domain, or a presence probability map of structures of interest from a statistical anatomical atlas can be mapped onto the X-ray to improve the visibility of these structures using locally adapted color or tone maps. Soft tissue components can be added by deforming the interpolated surface or volume of the soft tissue by a previously determined transformation. The soft tissue can then be forward projected into the image domain using either the CT or X-ray geometry and / or the previously estimated pose. Different methods of visualizing these additional soft tissue parts can be utilized.

[0019] Furthermore, if the images are acquired with a different modality, structures such as soft tissue can be mapped onto the X-ray image by adapting the joint model to this second image to compensate for possible different joints in both images.

[0020] In one embodiment of the present invention, providing a joint 3D anatomical model includes providing a plurality of joint 3D anatomical models, and aligning a rigid structure of the anatomical models with the positions of at least two bones in the X-ray image includes determining a residual error for each of the plurality of anatomical models, and selecting an anatomical model from the plurality of anatomical models having a minimum residual error.

[0021] It is therefore possible to provide an appropriate model for each patient. Providing multiple anatomical models ensures that the model that best corresponds to each patient's anatomy can be determined and used.

[0022] In one embodiment of the present invention, the plurality of anatomical models includes a male model and a female model.

[0023] In addition to male and female anatomical models, models for different ages of patients can also be provided, e.g. multiple anatomical models of children or teenagers. Furthermore, it is possible to provide respective models for patients with different body mass indexes, such as underweight, normal, obese or sports patients. The use of multiple models ensures the best results of the visualization of soft tissues in the X-ray images.

[0024] In one embodiment of the invention, the X-ray image is a two-dimensional image, and the step of aligning a rigid structure of the anatomical model with the positions of at least two bones in the X-ray image comprises estimating the posture of the patient's joints in the X-ray image.

[0025] Using two-dimensional x-ray images may require first determining the pose of the joint in the x-ray image before the rigid structure can be aligned with the bones in the x-ray image.

[0026] In one embodiment of the invention, the X-ray image is a three-dimensional X-ray image. Thus, for example images from a computer tomography examination can be used and enhanced with the method according to the invention.

[0027] In one embodiment of the invention, the joint is a hip, knee, ankle, shoulder, elbow, or wrist joint, and / or the soft tissue comprises a ligament, muscle, cartilage, tendon, or skin.

[0028] According to the invention, the joint 3D anatomical model can be a model of a human joint. The joint can be, but is not limited to, a hip joint, a knee joint, an ankle joint, a shoulder joint, an elbow joint, a wrist joint. The model has at least the main bones as rigid parts, potentially additional bones such as the patella of the knee joint, and at least one type of soft tissue, e.g., but is not limited to, ligaments, muscles, cartilage, skin.

[0029] In one embodiment of the present invention, a 3D anatomical model of the joint is constructed from a specialized or routine examination of the subject and / or from an anatomical atlas.

[0030] In one embodiment of the present invention, the three-dimensional anatomical model of a joint includes the position and shape of rigid structures and at least one type of soft tissue throughout the range of articulation of the joint.

[0031] In one embodiment of the present invention, the step of segmenting the X-ray image is performed by model-based segmentation or by a fully convolutional neural network.

[0032] Alternatively, the step of registering the anatomical model to the X-ray image can be performed via an end-to-end deep learning model, such that registration parameters such as rotation, scaling, translation, and pose can be derived from the deep learning model.

[0033] In one embodiment of the present invention, visualizing at least one type of soft tissue in the X-ray image includes determining an intensity change in the X-ray image and adapting the visualization of the at least one type of soft tissue based on the determined intensity change.

[0034] In this embodiment, the soft tissue parts are post-processed using the low level intensity variations in the original image and are therefore refined relative to the actual image content.

[0035] Furthermore, in one embodiment of the present invention, the enhancement of X-ray images with soft tissue structures estimated from the anatomical joint 3D model can be used for automatic region of interest (ROI) location. These ROIs can be used to draw the radiologist's attention to specific regions within the image or as a pre-processing step for computer-aided diagnosis applications.

[0036] In one embodiment of the invention, visualizing at least one type of soft tissue in the X-ray image comprises rendering the soft tissue as additional contrast, tone mapping, color overlay, or hue modulation.

[0037] Thus, soft tissue body parts can be visualized as an overlay with color or intensity modulation, or in a side-by-side view. To improve the dynamic range of the image, tone mapping can also be used to render the soft tissue parts. For image sequences, the soft tissue dynamics can be rendered in each frame.

[0038] According to another aspect of the invention there is provided a data processing apparatus having a processor configured to perform the method steps of any of the preceding embodiments.

[0039] The data processing apparatus may comprise multiple processors for executing the computer program according to the invention.

[0040] According to another aspect of the invention there is provided a computer program having instructions which, when executed by a computer, cause the computer to carry out the method steps of any of the preceding embodiments.

[0041] The computer program can be executed on one or more processing units which are instructed to carry out said method for rendering soft tissue in an X-ray image. Preferably, the program is stored in a system for rendering soft tissue in an X-ray image and the processing units which execute this program are part of said system. The computer program may be part of a computer program but may also be a whole program in itself. For example, the computer program can be used to update an existing computer program to become the present invention.

[0042] According to another aspect of the invention, there is provided a computer readable storage medium having instructions having instructions, which when executed by a computer, cause the computer to perform the method steps of any of the preceding embodiments.

[0043] The computer program can be stored in a computer readable storage medium, which can be regarded as a storage medium such as, for example, a USB stick, a CD, a DVD, a data storage device, a hard disk, or any other medium on which such a program can be stored.

[0044] Thus, advantages provided by any of the above aspects apply equally to all other aspects, and vice versa.

[0045] In summary, the present invention relates to a computer-implemented method for rendering soft tissue in an X-ray image. The method comprises the steps of providing a joint 3D anatomical model of at least one joint of a body, the model comprising a rigid structure and at least one type of soft tissue, the rigid structure comprising at least two bones connected by a joint. The method further comprises the steps of providing an X-ray image of a joint of a patient, the patient's joint corresponding to a joint of the anatomical model, and registering the anatomical model to the X-ray image. The method further comprises the steps of projecting the at least one type of soft tissue into an image domain of the X-ray image, and visualizing the at least one type of soft tissue in the X-ray image.

[0046] These aspects and embodiments will be apparent from and will be elucidated with reference to the exemplary embodiments described hereinafter. Exemplary embodiments of the invention are described below with reference to the following drawings, in which: [Brief description of the drawings]

[0047] [Figure 1] Diagram showing an articular 3D anatomical model of the rigid structure of the knee. [Figure 2A] Diagram showing ankle soft tissue types joint 3D anatomical model. [Figure 2B] Diagram showing an articulated 3D anatomical model of the rigid structure of the ankle. [Figure 3A] A diagram showing an X-ray image of the knee joint. [Figure 3B] A diagram showing the rigid structure of a 3D anatomical model of the joint registered to the bones of an x-ray image of a knee joint. [Figure 4] On the left is a lateral x-ray of the knee, and on the right is an x-ray of the knee highlighting the patella and soft tissues such as the quadriceps tendon and Hoffa's fat pad. [Diagram 5] 1 is a block diagram illustrating a method for rendering soft tissue in an X-ray image according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0048] FIG. 1 shows an example of an articular 3D anatomical model (120) of the rigid structures (130) of a knee joint (150) over the range of articulation of the knee joint. In a straight position, the bones (131) are highlighted with a black border to improve visibility of the femur, tibia, fibula and patella. Additionally, in this figure, the positions of the rigid structures (130) of the tibia, fibula and patella are shown over various flexion angles up to full flexion of the knee joint. These bones can be used for registration to the x-ray image domain.

[0049] Figure 2A shows an articulated 3D anatomical model (120) of a type of soft tissue (140) of an ankle joint (150). This is an example of articulation of the non-rigid part of the ankle, where a type of soft tissue, the skin surface, is visualized for different flexion angles of the ankle joint.

[0050] Figure 2B shows an articulated 3D anatomical model (120) of the rigid structure (130) of the ankle joint (150). Different bones (131) such as the tibia, fibula and part of the foot are visualized throughout the ankle joint range of motion. For example, knowing the shape of the skin surface according to the pose of the rigid parts allows morphing the skin surface to the bone constellation estimated in a given X-ray image. Kinematic models of other soft tissue structures such as ligaments or muscles that are connected to the bone and skin constellations allow predicting the spatial constellations of these other structures as well.

[0051] 3A shows an x-ray image 110 of a knee joint 150. The bones 132 are clearly visible in the x-ray image due to their higher specific x-ray absorption coefficient, but the soft tissue structures are not clearly visible in such an x-ray image.

[0052] Figure 3B shows a rigid structure (130) with bones (131) of a joint 3D anatomical model (120) registered to bones (132) of an X-ray image (110) of the knee joint (150) of Figure 3 A. The white areas indicate the bones (132) detected by the X-ray image, while the black borders indicate the surfaces of the bones (131) of the rigid structure (130) of the joint 3D anatomical model (120) of the knee joint (150) registered to the X-ray image (110).

[0053] FIG. 4 shows, on the left, a lateral X-ray image (110) of a knee joint (150), in which the femur, tibia, fibula and patella bones (132) are clearly visible, while the soft tissue is only slightly visible. On the right side of FIG. 4, the same X-ray image as on the left is shown. However, on the right side, the X-ray image has been enhanced according to the method of the present invention. The soft tissue structures (140) of the knee joint, such as the patella and the quadriceps tendon, and the Hoffa fat pad, are extracted from the registered anatomical model (120) and projected into the X-ray image domain. These soft tissue structures (140) are then visualized in the X-ray image as an additional overlay. In this example, in particular, the patellar tendon, which connects the patella to the tibia, is visualized with emphasis.

[0054] 5 shows a block diagram of a method for rendering soft tissue (140) in an X-ray image (110) according to an embodiment of the present invention. After a joint 3D anatomical model (120) of at least one joint (150) of a body is provided (S210) and an X-ray image (110) of each joint (150) is provided (S220), the anatomical model is registered to the X-ray image (S230). In a next step, the soft tissue (140) is projected into the image domain of the X-ray image (S240) and the soft tissue is visualized in the X-ray image (S250).

[0055] While the invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered exemplary or explanatory and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the dependent claims.

[0056] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be interpreted as limiting the scope. [Explanation of symbols]

[0057] 110:X-ray image 120: 3D anatomical model of joints 130: Rigid structure 131: Anatomical model bones 132: X-ray image of bones 140: Soft tissue 150: Joints

Claims

1. A computer implementation method for rendering soft tissue into an X-ray image, A step of providing a three-dimensional anatomical model of at least one joint of the body, wherein the anatomical model has a rigid structure and at least one type of soft tissue, and the rigid structure has at least two bones connected by the joint, A step of providing an X-ray image of a patient's joint, wherein the patient's joint corresponds to the joint of the anatomical model, The steps include registering the anatomical model to the X-ray image, The steps include projecting at least one type of soft tissue onto the image domain of the X-ray image, The steps include visualizing at least one type of soft tissue in the aforementioned X-ray image, It has, The three-dimensional anatomical model of the joint has the position and shape of the rigid structure and the at least one type of soft tissue over the range of motion of the joint. method.

2. The step of registering the anatomical model to the X-ray image is, The steps include segmenting the X-ray image to determine the location of at least two bones in the X-ray image, A step of aligning the rigid structure of the anatomical model with the positions of at least two bones in the X-ray image by applying a transformation to the anatomical model, wherein the transformation is defined by rotation, scaling, and translation of the anatomical model, and adjustment of the posture of the joints of the anatomical model. After the transformation is applied to the anatomical model, the steps include determining the location of at least one type of soft tissue in the anatomical model, The method according to claim 1, comprising:

3. The step of providing the three-dimensional anatomical model of the joint comprises the step of providing a plurality of three-dimensional anatomical models of the joint. The method according to claim 2, wherein the step of aligning the rigid structure of the anatomical model with the positions of the at least two bones in the X-ray image comprises the steps of determining a residual error for each of the plurality of anatomical models and selecting the anatomical model having the smallest residual error among the plurality of anatomical models.

4. The method according to claim 3, wherein the plurality of anatomical models include male and female models.

5. The method according to claim 2, wherein the X-ray image is a two-dimensional image, and the step of aligning the rigid structure of the anatomical model with the positions of the at least two bones in the X-ray image comprises the step of estimating the posture of the patient's joint in the X-ray image.

6. The method according to claim 1, wherein the X-ray image is a three-dimensional X-ray image.

7. The method according to claim 1, wherein the joint is a hip joint, knee joint, ankle joint, shoulder joint, elbow joint, or wrist joint, and / or the soft tissue includes ligaments, muscles, cartilage, tendons, or skin.

8. The method according to claim 1, wherein the three-dimensional anatomical model of the joint is constructed from a special or routine examination of the subject and / or from an anatomical atlas.

9. The method according to claim 1, wherein the step of segmenting the X-ray image is performed by model-based segmentation or by a fully convolutional neural network.

10. The method according to claim 1, wherein the step of visualizing the at least one type of soft tissue in the X-ray image comprises the steps of determining an intensity change in the X-ray image and adapting the visualization of the at least one type of soft tissue based on the determined intensity change.

11. The method according to claim 1, wherein the step of visualizing the at least one type of soft tissue in the X-ray image comprises the step of rendering the soft tissue as additional contrast, tone mapping, color overlay, or hue modulation.

12. A data processing device having a processor configured to perform the steps of the method according to any one of claims 1 to 11.

13. A computer program having an instruction that causes a computer to perform a step of the method according to any one of claims 1 to 11 when executed by the computer.

14. A computer-readable storage medium having, when executed by a computer, an instruction causing the computer to perform a step of the method according to any one of claims 1 to 11.

Images