Methods for modeling joints

Abstract

2. A method comprising: processing first 3D surface image data of the body part in a first pose by an automatic pose finding method; performing image registration between the first medical image data of the body part, first medical image data acquired simultaneously with the first 3D surface image data and depicting a first anatomical structure and a second anatomical structure, and second medical image data of the body part, where the image registration is performed separately for each of the first anatomical structure and the second anatomical structure; and determining one or more first transformations required to register the first anatomical structure in the first medical image data to the first anatomical structure in the second medical image data and / or one or more second transformations required to register the second anatomical structure in the first medical image data.

Description

[Technical field]

[0001] The present invention provides a computer-implemented method, a data processing system, a system comprising a data processing system, a computer program product, and a computer readable medium for modeling a joint. [Background technology]

[0002] Many recent developments in medical image analysis and processing require dynamic anatomical models, also called articulatory models, which represent postures such as flexion of a joint or inhalation / exhalation of the rib cage.

[0003] For example, 3D joint models may be used for quality assessment, preparation for diagnosis, and / or treatment planning. In particular, certain diagnostic questions require specific image acquisitions in which joint flexion is within a certain range, since deviations from this flexion may obscure certain anatomical features of diagnostic interest in the projection images.

[0004] Joint models can be used to assess object positioning based on image data alone and, when they are adapted to the available data, also make it possible to determine the normality of joint dynamics and (abnormal) ranges of motion.

[0005] Building these concatenated models across broad patient populations is laborious and time consuming.

[0006] Moreover, their construction requires the use of imaging modalities that, for example, use ionizing radiation that can be harmful to patients, and therefore volunteer scans outside of scheduled imaging sessions are never available in sufficient numbers as routine examinations of patients, which may also raise ethical concerns.

[0007] Routine imaging sessions, such as the routine examination of a patient, do not usually allow for significant workflow modifications, especially if they are time-consuming, e.g., if the imaging modality used for the imaging session is used to image joints in multiple poses. Clinical workflows usually do not allow for additional steps that would slow down the routine scheduling and throughput, such as performing dedicated measurements for different poses during the examination. For example, externally measuring flexion for multiple images obtained by the workflow's imaging modality in different poses, recording it to be stored together with the images, co-registering and sorting them according to the flexion angle, and building a model based thereon is difficult to incorporate into routine workflows.

[0008] Therefore, it can be difficult to build a complete model with sufficient accuracy. Summary of the Invention [Problem to be solved by the invention]

[0009] It is an object of the present invention to allow improved modelling of joints. [Means for solving the problem]

[0010] The invention provides a method for modelling a joint, a data processing system, a system comprising a data processing system, a computer program product and a computer readable medium according to the independent claims.

[0011] A method for modeling a joint according to the present invention includes processing first 3D planar image data of a body part in a first pose by an automatic pose search method to obtain first joint parameters representative of the first pose, the body part including a first anatomical structure and a second anatomical structure connected by a joint.

[0012] The method for modeling a joint according to the present disclosure further includes performing image registration between first medical image data of the body part, first medical image data acquired simultaneously with the first 3D surface image data and depicting the first anatomical structure and the second anatomical structure, and second medical image data of the body part in the second pose, the image registration being performed separately for each of the first anatomical structure and the second anatomical structure.

[0013] A method for modeling a joint according to the present disclosure further includes determining transformation data representing one or more first transformations required to register a first anatomical structure in the first medical image data to a first anatomical structure in the second medical image data and / or one or more second transformations required to register a second anatomical structure in the first medical image data to a second anatomical structure in the second medical image data.

[0014] The method for modeling a joint according to the present disclosure further includes generating and / or updating a statistical joint motion model based on at least the transformation data, the first joint motion parameter, and a second joint motion parameter representing the second pose.

[0015] In other words, the limitations of known methods can be addressed by relying on additional sensor data (in addition to data from the imaging modality), e.g., providing depth information from a surface scanner or range camera or other 3D camera system, e.g., an RGBD camera, and recording pose information during examination with the imaging modality. This allows relating poses and images from the imaging modality acquired in a routine workflow. Based on the relationships, a joint model can be built and / or can be built within the standard workflow of the radiology department.

[0016] In particular, the disclosed method allows for the acquisition of any posture without the need for additional recorded information. For example, user observations regarding the joint posture of a joint for a given acquisition are not required, and posture data does not need to be measured / tracked manually by a radiographer. This is particularly useful when using measurement devices to measure posture parameters, which would be a burden to the imaging modality, for example, in a CT bore due to space limitations, or in an MRI scanner due to the impossibility of using metal.

[0017] Therefore, the proposed approach does not require interruption of the imaging workflow, allowing the inclusion of images from the clinical routine workflow into the data used to build the joint model, and therefore allows large and diverse datasets to be used to build the model.

[0018] The present disclosure, as an example, enables the development of a joint model, where the model is generated using an MR scan of the joint where the joint is positioned a priori in one or more known poses or joint movements.

[0019] A potential application of the joint model is for skeletal X-ray image quality assessment.

[0020] As can be seen from the above, the claims involve generating and / or updating a joint model. For example, the method may involve using as a starting point any joint model, which may be empirical, semi-empirical or theoretical, and repeatedly applying the steps of the disclosure, in particular to one or more subjects and / or one or more imaging sessions of the same or different subjects, to improve the model. Improving the model may concern different aspects. For example, the model may become less general and more closely represent reality statistically, in particular across a wide range of subjects. Furthermore, the accuracy of the model may be improved, for example, by allowing the size of the data set to be significantly increased. Furthermore, the range of the model may be extended for extreme postures towards the end of the movement spectrum, such as postures representing very strong flexion and / or postures representing hyperextension of the joint.

[0021] In this disclosure, except in the discussion of the prior art, the term "joint model" is used synonymously with the term "statistical joint model" for the sake of readability.

[0022] A joint is a connection that movably connects bones in the body.

[0023] The body part may be, for example, an arm, wrist, shoulder, ankle, leg, hip, foot, hand, or any other part of the body that includes a joint.

[0024] In this disclosure, the term "anatomical structure" refers to a subsurface portion of the body. The anatomical structure may be, for example, bone, muscle, cartilage, ligament, etc. It should be noted that different anatomical structures may be rigid to different degrees, and the motion of some anatomical structures may have more degrees of freedom than the motion of other anatomical structures. For example, bone is essentially rigid and incompressible, and its motion can generally be described by translation and rotation. Tendons or cartilage may be elastically deformed, e.g., stretched, contracted, or compressed, and thus their motion may also include elasticity, deformation, and scaling, among others.

[0025] A pose is a particular relative configuration of a set of anatomical structures. For example, a pose may be defined by the relative configuration of two bones, e.g., by one or more angles that describe their relative configuration in one or more degrees of freedom.

[0026] The posture can be represented by joint parameters. The joint parameters can be any parameterization of the relative positioning of anatomical structures. For example, the joint parameters can include flexion angles.

[0027] It should be appreciated that the first and second poses are different, e.g., have at least one joint parameter different, e.g., the first and second poses may differ in having different flexion angles.

[0028] The automatic pose search can be any method known in the art, for example from gaming applications, that determines the pose of a body part based on 3D surface image data. The 3D information makes it possible to resolve directional ambiguities that would not otherwise be resolved.

[0029] It is noted that according to claim 1, second joint parameters are used which are representative of a second pose. The second joint parameters may be data derived from second 3D surface image data of the body part acquired using automatic pose search. Alternatively, the second joint parameters may be joint parameters derived from second medical image data and / or image data acquired using a different imaging modality and / or by direct pose measurement.

[0030] 3D surface image data is data that provides depth information, for example, 3D surface image data can be acquired by, for example, a 3D surface scanner, a stereoscopic camera, and / or an RGBD camera.

[0031] Medical image data in this disclosure refers to image data that shows subsurface structures of the body, for example image data obtained using ionizing radiation. Medical image data can include, for example, CT image data, MRI data, or X-ray image data. Medical image data can include 2D and / or 3D image data. 3D image data provides depth information of the imaged structure. 2D image data can, for example, show a projection of the imaged structure.

[0032] According to the present disclosure, the first 3D surface image data and the first medical image data are acquired simultaneously. This may include that during an imaging session, the first 3D surface image data is acquired continuously over a period of time including or at the time the first medical image data is acquired. In particular, the 3D surface image data from said period may be used to build a model in the absence of corresponding medical image data, e.g., acquired at a time between capturing medical images. For example, such image data may be used to improve automatic pose retrieval accuracy or improve accuracy when building a joint model by inferring key points of anatomy from surface image data or by helping to remove ambiguity, as described in more detail below.

[0033] Image registration may be performed using registration methods known in the art, including 2D / 3D registration, where 2D image data is registered with 3D image data and / or a mesh, and 3D image data is registered with 3D image data and / or a mesh.

[0034] According to the present disclosure, image registration is performed for each anatomical structure separately, meaning that the first and second anatomical structures are registered independently. Nevertheless, image registration may be performed in parallel for the first and second anatomical structures.

[0035] Transformations according to the present disclosure may include rotations and / or translations. In addition, transformations may include translations and scaling, for example, if the anatomical structure is non-rigid and / or compressible.

[0036] The joint model may be a model that represents the joint movements between different joints, and in particular the anatomical structure of the joints.

[0037] The joint model may be a 2D model that models the joint in projection, or a 3D model with depth information of the joint, especially the anatomical structures. For completeness, even if the joint model is a 2D model, the process of generating and / or updating the model includes some 3D data so that ambiguities can be avoided.

[0038] The statistical model represents data obtained from multiple imaging sessions, particularly from more than one subject. As an example, the same joint can be imaged for multiple subjects, and a model can be constructed to average out differences between the different subjects, such as differences in size and shape, differences in alignment of anatomical structures, etc. The statistical model may also be selectively based on data from multiple subjects sharing certain characteristics, as described in more detail below.

[0039] Generating the statistical joint model may require starting from a model of the overall joint, e.g. starting from individual models of the first and second anatomical structures, and / or starting from a rigid body model of the joint and / or a non-statistical model of the joint (articular or non-articular) to obtain the statistical joint model.

[0040] Updating the statistical joint model may include modifying a current version of the statistical joint model to account for the additional information, in particular to account for the transformation data, the first joint parameter, and the second joint parameter representing the second pose.

[0041] Therefore, generating and / or updating the statistical joint model based on the transformation data, the first joint parameters, and the second joint parameters representing the second pose may be considered as taking into account said data and parameters for modeling the joint, optionally together with existing data for modeling the joint.

[0042] According to the present disclosure, the statistical joint model may be based on transformation data, a first joint parameter, and a second joint parameter from image data acquired in multiple imaging sessions of the subject and / or from one or more imaging sessions of each of the multiple subjects.

[0043] The imaging session may be a routine examination and may in particular not involve altering a routine workflow, in particular not involve acquiring medical image data that is part of a routine examination and does not involve performing manual measurements of posture.By allowing data from different imaging sessions of the same or different subjects to be combined, larger data sets and therefore more accurate statistics can be performed, thereby allowing more accurate models.

[0044] Alternatively or additionally, according to the present disclosure, generating and / or updating the statistical joint model may include determining an average over a randomly selected subset of the population, and / or determining an average over a subset of the population that share a type of joint misalignment.

[0045] When using data obtained from multiple, particularly randomly selected, subjects, a statistical model can be provided that averages subject specific characteristics.

[0046] When only data acquired for a subset of the population that shares a type of misalignment is used in the statistical model, this may allow for determining the characteristics of the joints associated with the misalignment, which may help to better address the needs posed by the misalignment, but still has a large data set compared to only a single subject.

[0047] According to the present disclosure, the statistical joint model may be a 3D model, for example, 3D surfaces of each of the anatomical structures of the joint may be provided, particularly over the entire range of the model, which allows a particularly detailed study of the anatomical structures and their interactions.

[0048] The system of the present invention generates a constellation model C(Φ0 to Φ2) representing an arrangement of a first anatomical structure and an arrangement of a second anatomical structure for each of a plurality of postures including a first posture P1 and a second posture P2. n ) as a function of the joint parameters that represent each posture, in particular the joint flexion angle Φ i to obtain a plurality of constellation models as a function of . Such constellation models may be used as input data for generating and / or updating the statistical joint model. The constellations may be determined with respect to any suitable common reference.

[0049] According to the present disclosure, generating and / or updating the statistical joint model can be based on multiple constellation models, and in particular can include combining multiple constellation models. For example, the constellation models can be used to sort medical images in an image space to generate the joint model. Furthermore, the constellation models can be used together with interpolation to obtain a joint model that includes poses for which image data is not available.

[0050] According to the present disclosure, image registration can include, for example, applying a registration algorithm to register the image data based on landmarks in the image data and / or a mesh obtained by, for example, a segmentation algorithm based on one or more vertices of the mesh. In particular, the method of the present disclosure can include, prior to image registration, applying a segmentation algorithm to segment the first anatomical structure and / or the second anatomical structure to obtain a mesh. Any known segmentation and / or image registration method can be applied. Anatomical atlas data can be used.

[0051] The method of the present disclosure may include using the first joint motion parameter and the second joint motion parameter for pre-registration.

[0052] For example, the first and second joint motion parameters may be used to perform a transformation that approximately matches the anatomical structure before performing an image registration algorithm, which can reduce the resources required for image registration and can resolve ambiguities that image registration cannot resolve.

[0053] According to the present disclosure, a first anatomical structure may be used as a reference anatomical structure, and a statistical joint model may represent the positioning of a second anatomical structure relative to the first anatomical structure.

[0054] For example, the flexion angle may be expressed relative to a reference anatomical structure, which may be one of the bones or bones of the joint.

[0055] In accordance with the present disclosure, the statistical joint model may be a model representative of one or more selected motion types, and the method may include filtering data for one or more selected poses, and in particular one or more selected joint parameters, such that only data representative of the one or more selected motion types is used to generate and / or update the statistical joint model.

[0056] Joint models representing selected motion types can be advantageous for detailed characterization of joints, and can reduce unnecessary complexity when only one movement type is relevant for a given application.

[0057] For example, the movement type may be rotation and / or bending in a first plane only, or bending in a second plane only. A movement may be represented by a set of postures that are part of the movement. A movement type may, for example, be represented by only a subset of all possible postures of a joint. For example, a movement type may be represented only by postures that have the same bending angle in one direction and a different bending angle in another direction. Thus, the joint parameters representing a selected movement type may be joint parameters corresponding to the selected postures representing the selected movement type.

[0058] Filtering data for a selected pose may involve only data, e.g., image data and / or meshes, that correspond to the selected pose being used as input for generating the joint model.

[0059] The system of the present invention may include registering key points estimated from 3D planar image data of the body part for each anatomical structure to transform a reference segmentation of a first anatomical structure and / or a second anatomical structure from a first pose P1 to a third pose P3, and generating and / or updating a statistical joint model based on the third pose P3 and joint parameters of the transformed reference segmentation and / or the corresponding transformed data.

[0060] As an example, bones or other anatomical structures may affect the surface of the subject and may be identifiable, for example, through the skin and / or tissue. For example, protrusions may occur on the surface. Points at which anatomical structures are identifiable on the surface may be detected in the surface image data. The location of the detected points may then make it possible to estimate where one or more key points of the anatomical structures, for example on the mesh, are located, in particular based on the 3D image data and the medical image data in a given pose, optionally supplemented by general anatomical data. Based thereon, detecting points at which anatomical structures are identifiable on the surface in the 3D image data for other poses for which medical image data is not available makes it possible to infer the location of the key points. Based thereon, a transformation of the key points from one pose to the other can be determined, and the mesh can be transformed similarly based on the transformation of the key points. This allows additional data of the anatomical structures to be obtained even for poses for which only surface image data is available. Examples are described in more detail below.

[0061] The method of the present disclosure may include determining, in first 3D surface image data of the body part, one or more key points of a surface of the body part in the first 3D surface image data that correspond to key points of the first anatomical structure and / or the second anatomical structure in the first medical image data.

[0062] A surface keypoint may be a point on the surface of the object at which a portion of the anatomical structure is identifiable, e.g., by generating a protrusion on the surface. A corresponding keypoint of the anatomical structure may be a surface point at which a portion of the anatomical structure is identifiable on the surface, e.g., by generating a protrusion on the surface.

[0063] The system of the present invention may further include, for example, determining the position of each of one or more key points of the surface of the body part in the first pose P1 and the position of each of one or more key points of the first anatomical structure and / or the second anatomical structure by a reference segmentation of the first anatomical structure and / or a reference segmentation of the second anatomical structure.

[0064] The invention may further include processing a third 3D surface image data of the body part disposed in a third pose P3 to identify a position of each of the one or more key points of the surface of the body part in the third pose P3. It is to be understood that the third pose is different from the second pose and may be different from the first pose.

[0065] The method of the present invention may further include determining a position of each of the corresponding key points of the first anatomical structure and / or the second anatomical structure in the third pose P3 based on the position of each of the one or more key points of the surface of the body part in the third pose P3, the position of each of the one or more key points of the surface of the body part in the first pose P1, and the position of each of the one or more key points of the first anatomical structure and / or the second anatomical structure in the first pose P1.

[0066] The method of the present invention may further comprise determining keypoint transformation data representative of one or more third transformations required to match each of the one or more keypoints of the first anatomical structure and / or the second anatomical structure in the first pose P1 with each of the keypoints of the first anatomical structure and / or the second anatomical structure in the third pose P3.

[0067] The present invention may further include generating and / or updating a statistical joint model based on the keypoint transformation data and a third joint parameter representing a third pose P3.

[0068] Thus, surface image data for poses for which medical image data is not available, e.g., the third pose, can be used as input for the joint model. This allows for a broader set of input data for the model, which also increases the accuracy of the model.

[0069] The invention may include a step of checking for potential collisions between a reference segmentation of the first anatomical structure and a reference segmentation of the second anatomical structure when transforming from the first pose P1 to the third pose P3, and constraining the movement of the anatomical structures accordingly.

[0070] This allows extrapolation and / or other modeling errors to be avoided. Specifically, if the actual anatomical structures do not collide, the portions of the joint model that represent the collision will be inaccurate and therefore may be discarded.

[0071] The disclosed method may include simultaneously acquiring first 3D surface image data and first medical image data, i.e. the method may include a step of data acquisition of data used to generate and / or update the model, for example by the respective 3D surface imaging means and medical imaging means as described above.

[0072] The invention also provides a data processing system for performing and / or controlling any of the method steps of the present disclosure, in particular the data processing system may be configured to perform one or more, in particular all, of the computing steps of the present disclosure and / or to control the acquisition of the 3D surface image data and / or medical image data, for example by controlling a 3D surface imaging system configured to acquire 3D surface image data and / or an imaging system configured to acquire medical image data.

[0073] In particular, a data processing system may comprise one or more processors, in particular one or more computing devices and / or a distributed network of computing devices comprising one or more processors, where the one or more processors may be configured to perform and / or control the steps of the methods disclosed herein.

[0074] The invention also provides a system comprising the data processing system of the present disclosure and further comprising a 3D surface imaging system configured to acquire 3D surface image data comprising the first 3D surface image data and / or the second 3D surface image data and / or the third 3D surface image data. The system also comprises an imaging system, in particular a CT imaging system and / or an MRT imaging system and / or an X-ray imaging system, configured to acquire medical image data comprising the first medical image data and / or the second medical image data. The system may be configured to perform any of the methods of the present disclosure.

[0075] The present invention also provides a computer program product comprising instructions which, when executed by a computer, cause the computer to perform and / or control any of the method steps described in the present disclosure.

[0076] The present invention also provides a computer readable medium containing instructions which, when executed by a computer, cause the computer to perform and / or control any of the method steps described in this disclosure.

[0077] The present disclosure also provides for the use of statistical joint models for at least one of quality assurance of measurement data, proper positioning / placement of the subject, proper measurement of the subject, simulation and / or visualization of medical imaging devices, subject specific joints, deviations, and limitations.

[0078] The features and advantages outlined in the context of the method for modeling a joint apply equally to the data processing system, the system comprising the data processing system, the computer program product, and the computer readable medium of the present disclosure.

[0079] Further features, embodiments, and advantages will become apparent from the detailed description which refers to the accompanying drawings. [Brief description of the drawings]

[0080] [Figure 1] 1 shows a schematic diagram of a system according to the present disclosure. [Figure 2a] 1 shows a schematic, not-to-scale representation of two different types of joints. [Figure 2b] 1 shows a schematic, not-to-scale representation of two different types of joints. [Diagram 3] FIG. 3 shows a flow diagram illustrating a method according to the present disclosure. [Figure 4a] 1 shows different views of different types of joints. [Figure 4b] 1 shows different views of different types of joints. [Figure 5a] 1 shows an example of a 3D joint model and a segmented view of a knee. [Figure 5b] 1 shows an example of a 3D joint model and a segmented view of a knee. [Figure 5c] 1 shows an example of a 3D joint model and a segmented view of a knee. [Figure 6] 4 illustrates another exemplary flow chart of a method according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0081] FIG. 1 shows a schematic diagram of a system according to the present disclosure.

[0082] The system 1 comprises a data processing system 2 according to the present disclosure. The system may also comprise, as shown in this example, a 3D surface imaging system 3, e.g. a depth camera, and an imaging system 4 configured to acquire 2D and / or 3D medical image data, e.g. an MRI or CT scanner or an X-ray imager. The data processing system 2 may comprise one or more processors, in particular one or more computing devices and / or a distributed network of computing devices comprising one or more processors.

[0083] The system 1 is configured to execute the methods of the present disclosure, such as those specified in the claims or those described below in the context of Figures 3 and 4. In particular, the data processing system may be configured to execute and / or control the methods of the present disclosure, such as those specified in the claims or those described below.

[0084] Specifically, the data processing system may be configured to process the first 3D surface image data of the body part in the first pose by an automatic pose search method to obtain a first joint parameter representative of the first pose. Exemplary body parts, e.g., a leg and an ankle, are shown in and described in the context of Figs. 2a and 2b, respectively. The body part comprises a first anatomical structure and a second anatomical structure connected by joints, e.g., bones and joints, described in the context of Figs. 2a and 2b. The system may be further configured to perform image registration between the first medical image data of the body part and the second medical image data of the body part in the second pose. The first medical image data, e.g., CT, MRI, or X-ray image data, is image data acquired simultaneously with the first 3D surface image data and shows the first anatomical structure and the second anatomical structure. The system is configured to perform image registration for each of the first anatomical structure and the second anatomical structure separately. Furthermore, the system may be configured to determine transformation data representing one or more first transformations required to register a first anatomical structure in the first medical image data to a first anatomical structure in the second medical image data, and / or determine one or more second transformations required to register a second anatomical structure in the second medical image data to a second anatomical structure in the second medical image data, and determine one or more second transformations required to generate and / or update a statistical syntactic joint model based on at least the transformation data, the first syntactic parameters, and the second syntactic parameters representing the second pose. If the joint model is updated, the system may be configured to access, e.g., locally or remotely, a stored current statistical joint model and update the accessed model. The system may then store the generated and / or updated model, e.g., locally or remotely.

[0085] The 3D surface imaging system 3 may be configured for use in the method claims or to acquire 3D surface image data as described below. The medical imaging system may be configured for use in the method claims or to acquire medical image data as described below, e.g. CT, MRI or X-ray image data.

[0086] In addition to being configured to perform the above-mentioned steps, the data processing system may also be configured to control image acquisition by the 3D surface imaging system and / or the medical imaging system to obtain the respective image data. Alternatively or additionally, the processing system may be configured to retrieve previously stored image data and / or models, e.g., stored locally and / or remotely in one or more data storage devices 10. The data processing system may be connected to the imaging system and / or data storage system in which the image data and / or models are stored via a data connection 11, e.g., a wireless and / or wired data connection.

[0087] Figures 2a and 2b are schematic of two different types of joints and are not to scale.

[0088] 2a, by way of example, the body part 5 is a lower leg of a human subject, with the knee shown as an example of a joint 6. The first anatomical structure 7 may be the femur and the second anatomical structure 8 may be the tibia. Additionally, the patella is shown as a third anatomical structure 9.

[0089] In Fig. 2b, as an example, the body part is the ankle of a human subject, and the ankle joint is shown as an example of joint 6. The tibia, talus, and calcaneus are shown as examples of first, second, and third anatomical structures 7-9. Note that the knee has only one degree of freedom, and therefore there is only one flexion angle for each posture. The ankle joint has two independent degrees of freedom, and therefore there are two flexion angles for each posture.

[0090] As mentioned above in the general section of the description, a constellation model may be provided by the systems and methods of the present disclosure that is a collection of flexion angles for each of a number of postures. The constellation model may then be used to obtain the joint model.

[0091] In FIG. 3, a flow chart illustrates an exemplary method for modeling a joint according to the present disclosure.

[0092] At S11, first 3D planar image data of a body part in a first pose is processed by an automatic pose retrieval method to obtain first joint parameters representative of the first pose, the body part comprising a first anatomical structure and a second anatomical structure connected by a joint.

[0093] In step S12, image registration is performed between the first medical image data of the body part and the second medical image data of the body part in the second pose. The first medical image data is image data acquired simultaneously with the first 3D surface image data and shows a first anatomical structure and a second anatomical structure. The image registration is performed separately for each of the first anatomical structure and the second anatomical structure.

[0094] In this example, the medical image data may be 3D medical image data, for example 3D CT or MRI image data, so in this case 2D / 3D registration of the medical image data onto the mesh to obtain 3D information may not be required.

[0095] In step S13, transformation data is determined that represents one or more first transformations required to register a first anatomical structure in the first medical image data to a first anatomical structure in the second medical image data and / or one or more second transformations required to register a second anatomical structure in the first medical image data to a second anatomical structure in the second medical image data.

[0096] In step S14, a statistical joint model is generated and / or updated based on at least the transformation data, the first joint parameter, and the second joint parameter representing the second pose.

[0097] Optional steps S15 and S16 are shown in Fig. 3. In step S15, a registration, anatomical structure by anatomical structure, e.g. bone by bone, of key points of anatomical structures in the medical image data that can be inferred from the 3D surface image data of the body part can be performed. For example, the shape of the surface of the body part can indicate the shape and position of bones or other anatomical structures below the surface, so that detecting such shapes in the 3D surface image can be used to estimate the position of key points of the respective anatomical structures in the medical image data. A reference segmentation of the first anatomical structure and / or the second anatomical structure from the first pose to the third pose is transformed using the estimated positions of the key points.

[0098] In step S16, the joint model is updated based on the joint parameters of the third pose and the transformed reference segmentation and / or corresponding transformation data. Note that steps S15 and S16 may be performed after steps S11 to S14 or may be performed after them. If they are performed before steps S11 to S14, they may be used to generate the joint model, which may then be updated in step S14.

[0099] Optional steps S15 and S16 allow for information derived from the 3D surface image data to be incorporated into the joint model even when corresponding medical image data is not available, which can be particularly useful when the 3D surface image data within one imaging session is acquired at a higher rate than the medical image data.

[0100] It is noted that the method can optionally include a step S10 of simultaneously acquiring the 3D surface image data and the medical image data, however, alternatively, the image data may also be, at least in part, retrieved image data.

[0101] In the following, an exemplary method for modeling a joint according to the present disclosure is described, where the medical image includes a 2D medical image. Most of the steps may be the same as, for example, FIG. 3 and described above.

[0102] However, an additional step S11a is performed, where a 2D / 3D registration of the 2D first and second medical image data is performed before the registration step S12. For example, the 2D medical image data may be registered onto a common mesh representing the 3D shape of the surface of the respective anatomical structures, as will be explained in more detail below.

[0103] The registration S12 and transformation steps S13 may be performed using 2D / 3D registered medical image data.

[0104] As an example, the above method may be applied when a medical imaging modality is used that depicts anatomical structures such as bones in a 2D projection, e.g., a conventional X-ray imaging system. As mentioned above, the 2D medical image data may be used in the method according to the present disclosure for generating and / or updating a joint model by 2D / 3D registration of the first and second 2D medical image data onto a common mesh topology. This allows the 2D image data to be accurately registered and incorporated into the joint model. The 2D image data may also be used in particular to generate and / or update a 3D joint model by performing 2D / 3D registration.

[0105] The mesh may be obtained from an existing 3D model of each of the anatomical structures, for example an existing 3D joint model of a joint and / or an existing rigid body model of a joint or individual anatomical structure.

[0106] The same mesh is preferably used when repeating the 2D / 3D registration to incorporate additional image data into the joint model. As an example, since the joint model is a statistical model that can incorporate data from several different subjects, an average mesh of these subjects can be used, which can be updated with each new additional medical image data set. The initial mesh can be generated manually, semi-automatically, or automatically.

[0107] The use of the mesh for 2D / 3D registration of anatomical structures can be performed using known methods known in the art.

[0108] Figures 4a and 4b show different postures P1 to P4, in the case of Figure 4a showing the knee and in the case of Figure 4b showing the ankle.

[0109] In Figures 4a and 4b, the flexion angle Φ is shown for each posture. The knee has only one degree of freedom, therefore there is only one flexion angle for each posture. The ankle has two independent degrees of freedom, therefore there are two flexion angles for each posture. However, only the flexion of the upper ankle joint, i.e. dorsiflexion and plantarflexion, is shown for simplicity, i.e. the flexion angle of the lower ankle joint, i.e. inversion or abduction angle, remains constant.

[0110] Figures 5a and 5b show examples of 3D joint models of the knee, and Figure 5c shows a mesh, i.e., a segmented view of the knee, which may be used for different purposes for generating and / or updating the joint model, as described above.

[0111] 6 shows another exemplary flow chart of a method according to the present disclosure. A medical imaging modality and a depth camera are used to simultaneously record medical and depth images of a subject. This information is processed in a dedicated processing pipeline as shown in FIG. 6 in order to accurately co-register the medical images and generate or refine a joint model.

[0112] Exemplary method components include: 3D or range cameras that record objects during image formation, For example, the joints of the object, in particular the bending angles Φ to Φ n A pose search algorithm to derive a segmentation algorithm for segmenting bones in an image; A registration algorithm that uses the measured pose of each bone to register the image or mesh according to the previous (reference) bones; a registration algorithm that uses the measured pose of each bone to register all the other bones; translation / rotation transformations that describe the relative motion of each bone, and A constellation model C(Φ0 to Φ n ) output It is.

[0113] The above describes fully automatic joint model construction from (depth camera signal, medical image) pairs.

[0114] The registration in steps 4 and 5 may be based on segmented bones, or on both segmented bones and estimated pose. In both cases, depth signal-based pose estimation can be used to measure joint motion parameters and thus sort the cases. Alternatively, the joint parameters can be derived from the rotational degrees of freedom of the registration transformation (which encodes the rotation relative to a reference pose).

[0115] Depth-based pose estimation can be used to filter the dataset for specific joint movements, e.g., to select cases where only the ankle joint flexion angle is changed while the inversion / abduction angles are binned into a small fixed range.

[0116] Instead of segmenting bones in all images of a training cohort, bone segmentations of several reference cases can also be used to build the articular bone model. In this case, these reference segmentations (e.g., mesh formation) can be transformed to other poses by registering keypoints inferred from the depth signal for each bone. When transforming this pose to a different pose, the algorithm can check for potential collisions of the bone mesh and then constrain the bone motion accordingly.

[0117] While the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered as illustrative and not restrictive. The present invention is not limited to the disclosed embodiments. In view of the foregoing description and drawings, it will be apparent to those skilled in the art that various modifications may be made within the scope of the present invention, as defined by the claims. [Explanation of symbols]

[0118] System 1 Data Processing System 2 3D Surface Imaging System 3 Imaging System 4 Main body part 5 Joint 6 First Anatomy 7 Second anatomical structure 8f Third Anatomy 9 Data storage device 10 Data Connection 11 First posture P1 Second posture P2 Third posture P3 Fourth posture P4 Bending angle Φ

Claims

1. 1. A computer-implemented method for modeling a joint, the method comprising: processing first 3D surface image data of a body part in a first pose by an automatic pose search method to obtain first joint parameters representing the first pose, the body part having a first anatomical structure and a second anatomical structure connected by a joint; performing image registration between first medical image data of the body part and second medical image data of the body part in a second pose, the first medical image data being acquired simultaneously with the first 3D surface image data and depicting the first anatomical structure and the second anatomical structure, wherein the image registration is performed separately for each of the first anatomical structure and the second anatomical structure; determining transformation data representing one or more first transformations required to register a first anatomical structure in the first medical image data to a first anatomical structure in the second medical image data and / or one or more second transformations required to register a second anatomical structure in the first medical image data to a second anatomical structure in the second medical image data; generating and / or updating a statistical joint model based at least on the transformation data, the first joint parameters, and second joint parameters representing the second pose; 10. A computer-implemented method comprising:

2. the statistical joint model is based on image data acquired in multiple imaging sessions of the subject, and / or on the second joint parameter, the first joint parameter, and transformation data from one or more imaging sessions of each of the multiple subjects; and / or generating and / or updating the statistical joint model comprises determining an average over a randomly selected subset of the population and / or determining an average over subsets of the population that share a type of misalignment of the joint; The method of claim 1.

3. The method of claim 1 , wherein the statistical joint model is a 3D model.

4. generating, for each of a plurality of postures including the first posture and the second posture, a constellation model representing a configuration of the first anatomical structure and a configuration of the second anatomical structure as a function of joint parameters representing the respective postures, in particular as a function of flexion angles of the joints, so as to obtain a plurality of constellation models; the step of generating and / or updating the joint statistical model is based on the plurality of constellation models, and in particular comprises a step of combining the plurality of constellation models. The method of claim 1.

5. the image registration comprises applying a registration algorithm to register image data and / or meshes obtained by a segmentation algorithm, in particular the method comprises, prior to the image registration, applying a segmentation algorithm to segment the first anatomical structure and / or the second anatomical structure to obtain meshes; and / or the method comprising using the first joint parameter and the second joint parameter for pre-registration; The method of claim 1.

6. The method of claim 1 , wherein the first anatomical structure is used as a reference anatomical structure, and the statistical joint model represents the placement of the second anatomical structure relative to the first anatomical structure.

7. 2. The method of claim 1, wherein the statistical joint model is a model representative of one or more selected movement types, the method comprising the step of filtering data representative of the one or more selected movement types for one or more selected poses, in particular for the one or more selected joint parameters, such that only data representative of the one or more selected movement types are used to generate and / or update the statistical joint model.

8. performing an anatomical registration of keypoints inferred from 3D surface image data of the body part to transform the reference segmentation of the first anatomical structure and / or the second anatomical structure from the first pose to a third pose; generating and / or updating the statistical joint model based on the joint parameters of the third pose and the transformed reference segmentation and / or corresponding transformation data; The method of claim 1 further comprising:

9. determining, in first 3D surface image data of the body part, one or more key points of a surface of the body part in the first 3D surface image data that correspond to key points of the second anatomical structure and / or the first anatomical structure in the first medical image data; - determining a position of each of one or more key points on a surface of the body part and a position of each of one or more key points of the second anatomical structure and / or the first anatomical structure in the first pose by means of a reference segmentation of the first anatomical structure and / or a reference segmentation of the second anatomical structure; processing 3D surface image data of the body part positioned in a third pose to identify a position of each of one or more key points on the surface of the body part in the third pose; determining positions of each of the corresponding key points of the second anatomical structure and / or the first anatomical structure in the third pose based on positions of each of the one or more key points of the surface of the body part in the third pose, positions of each of the one or more key points of the surface of the body part in the first pose, and positions of each of the one or more key points of the second anatomical structure and / or the first anatomical structure in the first pose; determining keypoint transformation data representing one or more third transformations required to match each of one or more keypoints of the second anatomical structure and / or the first anatomical structure in the first pose with each of one or more keypoints of the second anatomical structure and / or the first anatomical structure in the third pose; generating and / or updating the statistical joint model based on the keypoint transformation data and third joint parameters representing the third pose; 2. The method of claim 1, comprising:

10. 9. The method of claim 8, further comprising checking for potential collisions of a reference segmentation of the first anatomical structure and a reference segmentation of the second anatomical structure when transforming from the first pose to the third pose, and constraining movement of the anatomical structure accordingly.

11. The method of claim 1 , comprising acquiring the first 3D surface image data and the first medical image data simultaneously.

12. A data processing system configured to perform the method steps of any one of claims 1 to 10 and / or to perform and / or control the method steps of claim 11.

13. A data processing system according to claim 12, a 3D surface imaging system configured to acquire 3D surface image data comprising the first 3D surface image data and / or the second 3D surface image data and / or the third 3D surface image data; an imaging system, in particular a CT imaging system and / or an MRT imaging system and / or an X-ray imaging system, configured to acquire medical image data including the first medical image data and / or the second medical image data; and In particular, the system is configured to carry out the method according to any one of claims 1 to 11. system.

14. A computer program product comprising instructions which, when executed by a computer, cause said computer to carry out the method according to any one of claims 1 to 10 and / or to carry out and / or control the method steps according to claim 11.

15. A computer readable medium having instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 10 and / or to perform and / or control the method steps of claim 11.

Images