Modular data processing for spine workflow

Abstract

The disclosed method supplements spine surgery by data processing. The method is modular using one or more data processing blocks related to one or more tasks of a spine surgery workflow, depending on the desired level of support by the data processing method. Each data processing block has input data and / or output data and processing logic that transforms the input data into the output data or generates the output data. Embodiments of the invention relate to different combinations of processing blocks. In the maximum scope, the data processing method covers all stages of the spine surgery workflow from pre-operative data acquisition to post-operative evaluation of the outcome.

Description

[0001] Brainlab AG

[0002] Attorney’s File: P101315WO XV

[0003] MODULAR DATA PROCESSING FOR SPINE WORKFLOW

[0004] FIELD OF THE INVENTION

[0005] The present invention relates to a computer-implemented method of data processing along at least a part of a medical spine surgery workflow, a corresponding computer program, a computer-readable storage medium storing such a program and a computer executing the program, as well as a medical system comprising an electronic data storage device and the aforementioned computer.

[0006] TECHNICAL BACKGROUND

[0007] Spine surgery is a complex medical procedure. It benefits from accompanying data processing, for example for planning, simulating, tracking, navigation or documentation.

[0008] The spine consists of multiple vertebrae, each having a vertebral body and a vertebral arch. An intervertebral disc, or spinal disc, is located between the bodies of neighboring vertebrae. Each vertebra has a foramen, which is an opening which accommodates the spinal cord comprising nervous tissue and extending along the spine passing through the vertebrae.

[0009] There are two main types of spine surgery. A first type removes part of a vertebra or other tissue, like tumor tissue, bulged spinal disc material or parts of the spinal bones to achieve decompression of the nervous tissue. A second type involves placement of one or more implants to achieve stabilization and correct or restore the spinal alignment. Implants can be cages replacing intervertebral discs, plates introduced from a ventral direction or pedicle screws and rods to align two or more neighboring vertebrae. The pedicle screws are attached to the pedicles of the vertebrae and the rod connects the pedicle screws to adjust the curvature of the spine in the treated area. The rod has a curvature to achieve a desired curvature of the spine when the pedicle screws are connected by the rod. Both types of spine surgery can be combined.

[0010] The purpose of the second type of spine surgery is to induce a fusion of adjacent vertebrae to prevent motion induced pain and also to obtain a desired or target alignment of the spine. The alignment of the spine is defined by the relative positions of all vertebrae, for example in terms of angles between neighboring vertebrae. In combination with other properties of the patient, like musculature and / or weight, the alignment of the spine results in a particular pose of the patient.

[0011] Both types of surgery can be performed independently or combined.

[0012] The medical spine surgery workflow does not only comprise the surgery itself, but also preceding aspects like planning, including simulations or predictions, and subsequent aspects like documentation and evaluation. The outcome of the spine surgery depends on many factors, including, but not limited to the quality of the plan of the surgery and the accuracy when the plan is executed.

[0013] The present invention can be used for spine surgery procedures e.g. in connection with systems such as the Spine Planning Element, the Planning Elements, the Spine & Trauma Navigation Software, the Cirq Robot and the Loop-X Imaging Robot, all products of Brainlab AG.

[0014] Aspects of the present invention, examples and exemplary steps and their embodiments are disclosed in the following. Different exemplary features of the invention can be combined in accordance with the invention wherever technically expedient and feasible.

[0015] EXEMPLARY SHORT DESCRIPTION OF THE INVENTION

[0016] In the following, a short description of the specific features of the present invention is given which shall not be understood to limit the invention only to the features or a combination of the features described in this section. The disclosed method supplements spine surgery by data processing. The method is modular using multiple data processing blocks related to one or more tasks of a spine surgery workflow, depending on the desired level of support by the data processing method. Each data processing block has input data and / or output data and processing logic that transforms the input data into the output data, generates the output data or otherwise processes the input data. Embodiments of the invention relate to different combinations of processing blocks. In the maximum scope, the data processing method covers all stages of the spine surgery workflow from pre-operative data acquisition to post-operative evaluation of the outcome to integrate as much knowledge as possible into a fully integrated approach that enables the best-possible outcome not only for the current patient, but also for subsequent other patients. However, the invention also accompanies partial aspects of the spine surgery workflow only.

[0017] GENERAL DESCRIPTION OF THE INVENTION

[0018] In this section, a description of the general features of the present invention is given for example by referring to possible embodiments of the invention.

[0019] Data Processing Blocks

[0020] The invention combines two or more data processing blocks, sometimes simply referred to as blocks, into a data processing method. While each data processing block may have input and / or output data, a combination of two blocks does not necessarily resemble a sequence of invoking independent functions. It is possible to seamlessly combine two or more data processing blocks. It is also possible to merge two or more data processing blocks such that they are not executed consecutively, but in an intertwined manner, for example by iteratively executing parts of the blocks multiple times.

[0021] Each data processing block may perform one or more processes, for example depending on the data input to the respective block or accessible for the respective block, and thus available to the respective block, or depending on the combination of data processing blocks, since a process might generate output data that is not processed by any other data processing block of the data processing method.

[0022] For each block, the input data are defined. However, if particular data is mentioned as input data, this does not necessarily mean that all of said particular data are input. It is also possible that only a part of said particular data is input to the respective block.

[0023] Depending on the time or the step of the spine surgery workflow at which a data processing block is executed, there may be different input data to the respective processing block. This means that some of the following explanations in one processing block refer to data generated by a second processing block not yet described. However, after reading the description of all data processing blocks, it becomes apparent how a particular data processing block processes the data.

[0024] The different data processing blocks are described below.

[0025] Data Acquisition Block

[0026] The data acquisition block acquires the data to be processed by the other data processing blocks. The optional input data to the data acquisition block is patient identification data which identifies a patient to whom the spine surgery is to be applied. The patient identification data may optionally identify a particular surgery to be applied if the patient has more than one associated surgery. The output data of the data acquisition block are patient data. The patient identification data is not necessary if the data to be acquired can be unambiguously identified by other means, for example if the data is retrieved from an imaging device.

[0027] The data acquisition block for example queries one or more databases and / or devices like storages to retrieve the patient data which corresponds to the patient identification data. The data acquisition block may directly acquire the patient data from the database or device or may first acquire a resource locator pointing to the patient data, which is then acquired in a second step. Different parts of the patient data can be obtained from different sources. The data acquisition block may for example access an electronic file of the patient which comprises patient data or links to patient data.

[0028] The patient data comprises one or more of image data, previous plans, physiological data, diagnosis data and existing documentation data. In this document, a plan comprises planning data, which can be alignment planning data, including simulation data, and / or decompression planning data and / or implant selection, sizing and placement planning data explained in more detail below. The expressions “plan” and “planning data” are thus used as generic terms covering any kind of planning data.

[0029] The image data comprise one or more of 3D image data, like CT image data, MR image data or pseudo-CT data generated from MR image data, and 2D image data, like x-ray image data. The image data shows all or part of the spine of the patient. The image data can be stored in a storage or retrieved from an imaging system which captures the image data in a “live” manner. The latter is for example the case if image data captured immediately before surgery or during surgery is to be processed. “Immediately before surgery” means that the patient is already situated in the operating room.

[0030] A single one of the 3D image data and the 2D image data can be sufficient in a basic implementation of the medical spine surgery workflow. For example, a lateral 2D x-ray image can be used to define a target alignment of the spine, for example for determining a target curvature of a rod. 3D image data is particularly useful for planning the implants, like cages or pedicle screws, in relation to one or more vertebrae. However, using information comprised in different kinds of image data can improve the quality of the method.

[0031] If it is stated in this document that image data comprises a structure, like a vertebra or an implant, this means that the image data comprises image elements, like pixels or voxels, which represent or show said structure. In analogy, if it is stated that a structure is found in image data, this means that image elements of the image data which represent or show said structure are found. The x-ray image data for example includes standing x-ray image data showing the patient in a standing pose such that the spine is shown in a relevant pose of the patient. Due to technical reasons, the 3D image data is typically generated when the patient is lying on a table, such that the vertebrae exhibit relative positions which are different from the relative positions at the standing pose. The x-ray data may comprise multiple x-ray data for different regions of the spine and / or different poses of the patient.

[0032] The 2D image data is typically used to determine possible shape outcomes of the spine after surgery since they represent multiple vertebrae of the spine and thus the relationship between multiple vertebrae. This might be used for determining the rod curvature. The 3D image data represent details of the shapes of the vertebrae and might thus be used for planning the locations of implants or of osteotomies relative to the vertebrae. 3D image data is typically provided with physical dimensions. This means that the voxels of the 3D image data represent a known size in the real world. If other data, such as 2D image data, have no assigned physical size, they can be annotated with physical dimensions based on, e.g. by a registration of the other data and the 3D image data.

[0033] The image data may comprise the latest image data of the patient, captured for example one month, one week or less before performing the data processing method, but also old image data up to several years or even decades old. The old image data might be used for tracking changes of the patient over time.

[0034] Previous plans may comprise plans of previous spine surgeries and / or plans of surgeries other than spine surgeries. The previous plans thus give additional information on the patient and previous treatments, for example.

[0035] The physiological data means additional data about the patient, like gender, age, height, weight, BMI (body mass index) and the like.

[0036] The diagnosis data represents information about a diagnosis made by a doctor and for example identifies one or more vertebrae that have to be treated by the surgery. The existing documentation data represents a documentation of data collected or actions performed before the data acquisition block is executed.

[0037] The patient identification data is optional input data to the data acquisition block. If the patient identification data is not input, the data acquisition block receives user input data representing one or more information items, or parts thereof, of the patient. Examples of information items are first name, last name, date of birth, place of birth or address of the patient. Using the user input data, the acquisition block queries a database of patients which associates information items with patient identification data. If there is a single patient in the database to which all input information items match, the associated patient identification data is retrieved and the data acquisition block uses those patient identification data. If there are multiple patients in the database to which all input information items match, a list of those patients is provided to the user and the user can select one of the patients. The patient identification data associated with the selected patient is retrieved and used by the data acquisition block.

[0038] Data Analysis Block

[0039] The data analysis block analyzes the patient data, for example to make the data processable by subsequent data processing blocks. The input data to the data analysis block are one or more of patient data, the patient identification data, planning data and post-operative data. One or more of the processes described below might be performed by the data analysis block.

[0040] In one process, the data analysis block performs segmentation and classification of the image data, for example by invoking a universal patient model. The universal patient model, which can also be referred to as anatomical patient model, comprises data about the patient, in particular as many data as available. The universal patient model is a kind of digital twin of the patient. The universal patient model is provided with input data, such as some or all of the patient data and / or the patient identification data. Other input data defines the information to be delivered. In the case of segmentation and classification, the universal patient model is for example invoked with the image data and the patient identification data. The universal patient model identifies the digital twin based on the patient identification data, and segments and classifies the image data. The universal patient model for example uses segmented and classified structures of the spine, or at least a part of the spine, and optionally of other anatomical structures. The other anatomical structures may include one or more of the spinal cord, nerves and vessels. The segmentation is performed on the 2D image data, the 3D image data or both. The universal patient model returns segmentation data and classification data.

[0041] The segmentation data defines one or more of: outlines, sets of pixels, sets of voxels, and reference structures like reference landmarks of one or more of the structures mentioned above found in the image data. This process might be referred to as segmentation process and generates segmentation data.

[0042] Reference landmarks are particular points of a structure, like the “upper left corner of the L5 vertebra”. By correlating reference landmarks between different imaging modalities, scaling and co-registrations may technically by achieved between such data. Reference landmarks can be detected for example in 2D image data using artificial intelligence or in 3D image data using the universal patient model.

[0043] The classification data defines types and properties of one or more of the structures mentioned above found in the image data. The classification data may include tissue types like ‘bone’, ‘nerves’, ‘disc’, ‘vessel’, and / or organ types like ‘spinal vertebrae’, ‘lumbar vertebra’, ‘thoracic vertebra’, ‘cervical vertebra’, ‘sacrum’, ‘ilium’, ‘spinal cord’, or common names (‘vertebrae labels’) like C1 to C7, Th1 to Th12, L1 to L5, Os sacrum and Coccyx, ilium, cranium.

[0044] In the universal patient model, the segmentation and classification might be automatic as described above, manual based on user input data or a combination of both. At manual segmentation, the image data, or part of the image data, is displayed, for example using the display block described below, and the user might input delineation data by drawing the outline of a segment over the image data. The user input might further include a classification, like a label of a segment. Instead of drawing the outline, the user might manipulate a generic model of a vertebra obtained from the universal patient model, for example by one or more of scaling, moving, rotating, shearing or adding non-affine transformations (warp) to the generic vertebra model such that it matches actual vertebra in the image data. The combination of both automatic and manual segmentation and classification might comprise automatic segmentation and classification followed by manual correction of the segmentation and classification by user input.

[0045] The universal patient model may use a generic model obtained by analysis of image data of multiple patients and aggregation of the result of the analysis or by using machine learning utilizing convolutional neural networks (CNNs). It is for example used to enable segmentation and classification based on the analysis of image data of multiple patients. It further comprises additional data, like classification information identifying different segments in the generic model and optional labels assigned to the segments, or desired implant positions, like optimal pedicle screw positions.

[0046] The generic model is either matched onto the image data, for example using rigid or elastic image fusion, to adapt the generic model to the actual image data such that the segments specified in the generic model can be identified in the image data, and / or one or more trained CNNs are used to segment and classify outlines, sets of pixels, sets of voxels and reference landmarks of a specific patient. Both the matching and the use of CNNs may be used exclusively, subsequently, iteratively or parallel for all of or sub-parts of the imaging data to achieve segmentation and classification. For example, a CNN may be used to classify the correct body region and to create correct scaling, then a subsequent matching of the generic model on the 3D image data may be used to fully segment and classify the 3D image data.

[0047] In one process, the data analysis block invokes the universal patient model to identify which vertebrae, or spinal levels, are shown by the image data. For example, the common labels (C1 to C7, Th1 to Th12, L1 to L5, Os sacrum, Coccyx, ilium, cranium) are assigned to the vertebrae identified in the image data, for example after or as a part of segmentation and classification. The labels might be taken from the generic model. In one example heuristics and adjacent anatomical structure are additionally used to check consistent consecutive labelling for several adjacent vertebrae for a specific body region. This process might be referred to as vertebra identification process and generates vertebra identification data. Instead of automatically identifying the vertebrae, the vertebra identification process might involve receiving user input representing a label of a vertebra.

[0048] In one process, the data analysis block invokes the universal patient model to identify one or more organs at risk (OAR), like spinal cord, nerves or vessels, in the image data. Organs at risk are anatomical structures of the patient which must not be damaged during surgery or by an implant after surgery. The organs at risk might be identified based on the segmentation and an indication in the universal patient model that a particular structure is an organ at risk. This process might be referred to as OAR identification process and generates OAR identification data.

[0049] The OAR identification might be automatic as described above, manual based on user input data or a combination of both. At manual OAR identification, the image data, or part of the image data, is displayed, for example using the display block described below, and the user might input identification data by pointing to an organ at risk or drawing the outline of an organ at risk over the image data. The user input might further include a label of an organ at risk. The combination of both automatic and manual OAR identification might comprise automatic OAR identification followed by manual correction of the OAR identification by user input. In addition or as an alternative, user input points at the OAR and automatic OAR identification searches for an OAR at the position indicated by the user input.

[0050] In one process, the data analysis block identifies structures of interest, like a vertebra, one or more parts of a vertebra (for example a pedicle), an intervertebral disc or a disc space of an intervertebral disc, or multiple of said structures. The identification is for example based on the patient data, in particular the diagnosis data. This process might be referred to as structure identification process and generates structure identification data.

[0051] The structures of interest are for example structures on which surgery is to be performed, or is likely to be performed.

[0052] The structure identification might be automatic as described above, manual based on user input data or a combination of both. At manual structure identification, the image data, or part of the image data, is displayed, for example using the display block described below, and the user might input delineation data by drawing the outline of a structure over the image data. The user input might further include a label of a structure. Instead of drawing the outline, the user might manipulate a generic model of a structure, for example by one or more of scaling, moving, rotating, shearing or by adding non-affine transformations (warp) to the generic structure such that it matches the structure in the image data. The combination of both automatic and manual structure identification might comprise automatic structure identification followed by manual correction of the structure identification by user input.

[0053] In one implementation, the structure identification might use patient data to identify a structure of interest. For example, a structure identified in a previous plan, diagnosis data or existing documentation data might be identified as a structure of interest. In another example, a structure shown in image data which only covers a small part of the spine is identified as a structure of interest.

[0054] In one process, the data analysis block detects abnormalities in the image data. Abnormalities may include one or more of prolapse, spinal cord compression, tumors and the like. This process might be referred to as abnormality detection process and generates abnormality data.

[0055] Abnormality detection might be automatic, manual or a combination of both. In one example of automatic abnormality detection, a difference between the matched structure of the generic model and the structure in the image data to which it was matched might be analyzed to identify an abnormality. In another example, a generic representation of an abnormality, such as a 2D representation or a 3D model, is searched for in the image data. In another example a trained CNN is used to identify the abnormality.

[0056] In manual abnormality detection, the image data, or part of the image data, is displayed, for example using the display block described below, and the user might input delineation data by drawing the outline of an abnormality over the image data. The user input might further include a label of an abnormality. The combination of both automatic and manual abnormality detection might comprise automatic abnormality detection followed by manual correction of the abnormality by user input.

[0057] In one process, the data analysis block determines the severity of an abnormality. In one implementation, the severity is input by a user. In another implementation, the severity is determined automatically, for example using a CNN or querying a database which associates abnormalities with severities. The severity is output as severity data.

[0058] In one process, the data analysis block detects existing implants in the image data. Existing implants might involve one or more of cages, screws or rods. The implant detection involves detection of the type of implant and / or the position of the implant. The type of implant may mean the general type, such as “screw” or “cage”, and the particular type, such as which screw or which cage. This process might be referred to as existing implant detection process and generates existing implant data.

[0059] The image data in which existing implants are detected can be pre-operative image data in the patient data. However, the image data can also be post-operative image data in the post-operative data. In this case, the actual position of an implant after surgery can be detected, for example for documentation purposes. The actual position is for example a position of an implant relative to a vertebra.

[0060] Existing implant detection might be automatic, manual or a combination of both. In one example of automatic existing implant detection, a generic representation of an implant, such as a 3D model, is searched for in the image data.

[0061] The generic representation of the implant might further identify the type of implant.

[0062] In another example, a trained CNN is used to identify the implant.

[0063] In manual existing implant detection, the image data, or part of the image data, is displayed, for example using the display block described below, and the user might input delineation data by drawing the outline of an existing implant over the image data. The user input might further include a label of an existing implant, for example for identifying the type of implant. The combination of both automatic and manual existing implant detection might comprise automatic existing implant detection followed by manual correction of the existing implant by user input. In addition or as an alternative, user input points at the implant and automatic existing implant detection searches for an implant at the position indicated by the user input.

[0064] In one process, the data analysis block derives numerical values corresponding to the spinal anatomy. Depending on the input data to the data analysis block, the numerical values can be derived from one or more of pre-operative image data in the patient data, the planning data and post-operative image data in the post-operative data. The numerical values can for example be angles between neighboring vertebrae, distances between neighboring vertebrae or heights of vertebrae.

[0065] The output of the data analysis block is analysis data. Depending on which processes are performed by the data analysis block, the analysis data comprises one or more of segmentation data, classification data, vertebra identification data, OAR identification data, structure identification data, abnormality data, existing implant data, seventy data and numerical data.

[0066] Data Fusion Block

[0067] The data fusion block generates additional data about the patient by combining two or more parts of the input data to the data fusion block. The input data to the data fusion block are one or more of the patient data, or parts thereof, the analysis data, or parts thereof, alignment planning data, or parts thereof, decompression planning data, intraoperative data and post-operative data. One or more of the processes described below might be performed by the data fusion block.

[0068] In one process, the data fusion block generates standing 3D image data from the standing x-ray image data and the 3D image data comprised in the patient data. The vertebrae represented by the 3D image data are registered with the vertebrae in the standing x-ray image data. Each segment of the 3D image data representing one or more vertebrae is registered individually with the standing x-ray image data. Registration might use known 3D-2D registration algorithms, like those using digitally reconstructed radiographs. This process aligns the 3D image data of one or more vertebrae with the standing x- ray image data of the corresponding vertebrae, and thus aligns the 3D image data of said vertebrae with each other. This results in a virtual 3D model of the spine, or a part of the spine, in the standing pose of the patient. This might use the segmentation data, reference landmarks and / or the vertebra identification data in the analysis data to find corresponding vertebrae in the 3D image data and the standing x-ray image data and / or to align image data with each other, for example 3D image data with 2D image data or 3D image data with other 3D image data.

[0069] The standing 3D image data may comprise the aligned segmented vertebrae of the 3D image data only or also additional structures from the 3D image dataset, such as spinal discs, spinal cord, nerves or vessels. The standing 3D image data may further comprise existing implants, like cages, screws or rods.

[0070] In one process, the data fusion block compares the spinal anatomy, for example the alignment, of the patient at different points in time to generate comparison data. The data fusion block for example aligns image data of different points in time with each other, for example two or more of pre-operative image data, post-operative image data and predicted image data representing a simulation of an image of the spine if the surgery is performed as planned. This might use a particular vertebra in all considered image data as a reference, which means that parts of the image data which show said vertebra are aligned and the other structures shown by the image data maintain their relative position to said vertebra. The aligned image data may be displayed side by side or in an overlaid manner by the display block to visualize the anatomy at the different points in time.

[0071] One or more points in time can be future points in time, such that the corresponding image data is predicted image data. The predicted image data for example shows the predicted post-operative spinal anatomy depending on a particular surgical plan, such that the expected change to the anatomy by the surgery can be visualized.

[0072] The points in time can for example be two or more of some months to years before the surgery, days to weeks before the surgery, during surgery, some weeks after surgery and months to years after surgery, for example if pain to the patient becomes worse again.

[0073] The comparison data may comprise image data as explained above. However, the comparison data can also comprise numerical data which represent changes in an intervertebral angle between neighboring vertebrae between two or more points in time. The comparison data can be a combination of the image data and a visual representation of the numerical data.

[0074] The data fusion block may compare multiple predicted image data. The multiple predicted image data may be image data representing the predicted spinal anatomy after two or more surgeries in a surgical plan involving multiple surgeries, thus representing the outcome of each of the surgeries. In this example, the expected change to the anatomy can be visualized over the sequence of surgeries. However, predicted image data can also represent an expected change to the anatomy due to aging or degeneration of the spine. The predicted image data may also represent the anatomy after a combination of one or more surgeries and aging / degeneration. The predicted image data may also represent the predicted post-operative anatomy for different surgical plans according to different planning data.

[0075] In one process, the data fusion block performs matching of one piece of image data onto another piece of image data, for example by rigid or elastic image fusion. The matched pieces of image data can have different imaging modalities. The one piece of image data of one modality is transformed such that it matches the second piece of image data of potentially another modality. The fusion for example achieves an exact matching of the vertebrae and a transformation of soft tissue which is anatomically correct. In one example, 3D image data of the patient data, which is for example MRT data, is matched onto intra-operative image data, which is for example x-ray image data. This process generates matched image data. Matched image data can also be generated by matching image data to other intra-operative data, such as positions sampled using a pointer.

[0076] In one process, the data fusion block enhances image data with additional data, such as bone density, bone strength, muscle attachment points, inflammation areas, osteophytes or impingements, to generate enhanced data. The additional data is for example requested from the universal patient model.

[0077] In one process, the data fusion block generates fused image data which combine different data. The fused image data may comprise image data of different points in time and / or of different modalities. The fused image data may comprise a representation of comparison data, like numerical values. The fused image data may comprise a representation of one or more of the other patient data, like all or parts of the physiological data, the diagnosis data and previous plans, of the planning data or of the analysis data.

[0078] In one example, the fused image data comprises a pre-operative standing x-ray image, a predicted post-operative image, which can be a 2D or a 3D image, and a postoperative standing x-ray image.

[0079] In one example, the data fusion block combines information comprised in 3D image data of different modalities. In one example, image data showing particular structure are taken from the modality which shows this structure best and are added to other image data. This process also generates fused image data.

[0080] The output data of the data fusion block are fusion data comprising one or more of the standing 3D image data, the comparison data, the enhanced data, the predicted image data, the matched image data and the fused image data.

[0081] Model Generation Block

[0082] The model generation block generates a biomechanical model of the patient, in particular of the spine system of the patient. The biomechanical model represents the spinal column, or a part thereof, including the vertebrae and the spinal discs and optionally one or more of ligaments, muscles directly and indirectly acting on the spinal column, the ribcage, the abdomen, part of the or the whole pelvis, hip joints, knees and feet. It can be used for simulation of loads and the kinematics of the spine in different poses and / or for different kinds of motion. The input data to the model generation block include one or more of the image data within the patient data, the segmentation data within the analysis data, the fusion data, the planning data described below, the analysis data and optionally further data, like data representing other parts of the musculoskeletal system. The planning data may comprise alignment planning data and / or decompression planning data. Based on the planning data, the biomechanical model can be generated to represent the predicted situation after surgery. One or more of the processes described below might be performed by the model generation block.

[0083] There are different kinds of biomechanical models. Statistical models comprise collections of "if-then" measurements derived from past surgeries of a large number of patients. Musculoskeletal simulations are for example multi-rigid-body models. Further models are based on stress / force simulations, like finite element models (FEM).

[0084] In the case of statistical models, formulas or tables can be used to predict the change in posture of a patient. The statistical model for example indicates that stiffening of a patient with a measured angle of X to obtain a planned angle Y actually results in an average angle Z.

[0085] In the case of musculoskeletal simulations, for example, simulated muscle forces, or muscle activation energy, are minimized to find out postures the patient is likely to adopt. Via changes in boundary conditions, like simulated vertebral stiffnesses, a surgical outcome (in this case, a patient posture) can be simulated, for example. Also, for different selected postures, corresponding to different planned surgical strategies, the different forces and activation energies can be simulated. This can then be used in the planning phase to evaluate different surgical strategies against each other. In particular, these models are good for dynamic modulations defining how the structures move in different poses (like sitting or standing) or in dynamic transitions (like walking or boarding). The output of this dynamic simulation can again be the input of an FEM simulation.

[0086] From an FEM model, forces in tissue structures (bones=pressure I stress on adjacent vertebrae, endplates, intervertebral discs, vertebral joints, screw insertion areas, ...) or metal structures (implants) can be simulated. A biomechanical model, or plural models in combination, can be used for many purposes. In one use case, the differences between a geometric planning of posture corrections on the image data set and the actual surgical result can be simulated. If, for example, the patient has a high BMI and soft bones, the whole screw-rod assembly will elastically bend under the load of body weight in different postures, additionally the cages will 'plastically' sink into the endplates. All of this can be simulated.

[0087] In one use case, a faster degeneration of other structures can be predicted. For example, many unfavorable forces on the ligament sheath and the joint to the next untreated vertebra when walking could indicate a fast continuation of degeneration to this next untreated vertebra. This is also called adjacent level disease.

[0088] In one use case, possible implant failure can be predicted. High strain or stress on a particular highly bent area in the rod could mean that the rod breaks faster and high strain or stress on a particular screw could lead to early loosening of the implant.

[0089] A biomechanical model is calibrated using gold standard data representing a known plan associated with a known outcome and then tested on other gold standard data to see how strong the predictive power is for verse diseases, populations, etc.

[0090] The model generated by the model generation block can comprise two or more models, and another block uses the kind of model suited best for its purpose. If it is to be analyzed if a certain treatment (screw and rod fixation) will induce the adjacent disc to degenerate faster (= the likelihood that the patient develops an “adjacent level disease”), a local FEM force model may be used. If it is to be analyzed in which body posture the patient will end up after a certain rod is inserted, a multi-body model and muscle activation force minimization may be used.

[0091] In one embodiment, the model(s) is / are fitted (initialized) to the specific patient. An example without initialization would be a statistical formula that predicts the change in posture with a stiffening of segments x to y and that is used universally for all patients.

[0092] Initialization can be done using patient data (age, BMI, height, disease, planned treatment type, ...). This means that a different statistical formula is used depending on age and gender or a musculoskeletal model is scaled differently depending on age, height and BMI.

[0093] In the most elaborate form, all relevant analysis data from the data analysis block are used, like segmentation data, reference landmarks, etc. to fit the model exactly to the individual patient. For example, the exact shape and internal condition of a vertebral body is 'recreated' in an FEM model.

[0094] In a multi-rigid-body model, the vertebral body representatives (rigid bodies) and their connecting disc and muscle representatives (non-linear spring and damping elements) can be adjusted in size, shape and / or ratio to each other. Using further information from the patient data and treatment data (like age, BMI or osteoporosis) or the image data (e.g. from a segmented muscle), their simulated material properties (=viscoelastic properties) can be adapted to the real patient.

[0095] It shall be noted that any modeling can also be directly replaced by machine learning with enough available gold standard data. So instead of simulating by means of a deterministic model (in which assumptions about cause-effect are stored by means of mathematical relationships), the biomechanical effect of a treatment can also be learned. If the CNN has all analysis data from the data analysis block available and knows the planning and the result, it can directly "predict" all things described above after "learning" from sufficiently large data sets.

[0096] The output data of the model initialization block are model data describing the biomechanical model of the patient.

[0097] Alignment Planning Block

[0098] The alignment planning block generates a plan which relates to the pose of the patient’s spine, that is especially the relation of vertebrae to each other that represent the post-operative situation, for example in one or more typical poses or activities of the patient, like neutral standing, standing in flexion, standing in extension, neutral sitting, walking, running and so on. These poses are typically described by defined angular and distance values or relations and classification values processing such values and relations into groups (Lumbar Cobb angles, SVA, Spondylosthesis grade after XYZ). This may involve planning the position and relation of one or more implants and optionally also the selection of an implant from a set of available implants. The selection of the implant includes the size and the shape of the implant. The plan may further or instead involve planning of one or more osteotomies. The alignment aims at the restoration of the spine’s pose and fixations of vertebrae. The output of the alignment planning block are alignment planning data and optional relative position data.

[0099] The input data to the alignment planning block are patient data, in particular the image data, the analysis data, the fusion data and optionally one or ore of the model data, user input data, feasibility data, decompression planning data, and an existing plan. Implants can for example be cages, screws or rods. One or more of the processes described below might be performed by the alignment planning block.

[0100] Planning a screw as an example of an implant involves planning of one or more of the local position relative to the vertebra, the length, the diameter, the material and the type of the screw.

[0101] Planning a rod as another example of an implant involves planning of one or more of the shape and the size of the rod. The planning may be limited to readily available or standard rods defined in a rod database. Planning the rod may consider forces acting on the rod once it is implanted, which can be derived from the patient data, like data about the musculoskeletal system, for example. It might then optionally further consider the stability of the rod, which results in possible deformations of the rod due to the forces acting on the rod.

[0102] Planning a rod is for example based on standing x-ray data in the 2D image data. The standing x-ray data is typically a lateral view which shows a side view of the spine, which is the best starting point for planning the rod. The shape of the rod can, but not necessarily must, be solely based on the standing x-ray image data.

[0103] Planning a cage as yet another example of an implant involves planning of one or more of the local position, the size, the height, the material and the type of the cage. The properties of screws and / or cages are for example planned to support the rod and to restrict relative movement of neighboring vertebrae to stop pain. Interactions between screws, cages and / or rods may be considered. For example, save, doable, convenient and / or non-skiving implant positions may cause use of a different rod from a perfect rod.

[0104] Planning of screws, cages and / or rods may consider the access to the site of implantation. There must be, for example, an access path large enough for the implant and any tool used for positioning the implant. The objectives might be one or more of save, non-colliding, cosmetically preferable and / or convenient access to the implantation site.

[0105] Planning an osteotomy involves planning of one or more of the local position, the size, the angulation, the shape and the volume of the osteotomy.

[0106] The alignment planning block optionally considers decompression planning data described below when generating the alignment planning data. Removal of tissue for decompression might have an impact on the alignment of the spine, such that alignment planning takes this into account.

[0107] The alignment planning block optionally assumes tissue removal, for example of an intervertebral disc to make space for a cage, or due to laminectomy, even if the tissue to be removed is still shown in the patient data since the tissue is removed before the implant is applied.

[0108] In a first process, the alignment planning block utilizes the user input data indicating one or more implants and / or one or more osteotomies. Indicating an implant includes the identification of an implant and the position of the implant. The implant is for example positioned relative to the 2D or 3D image data or model data of a vertebra, indicating an osteotomy includes the position, the shape and the size of the osteotomy, for example in 2D or 3D image data or model data of a vertebra. The user input data may further indicate one or more rods. The alignment planning data output by the alignment planning block comprise the user input data. Based on the user input data, the alignment planning block calculates relative position data based on the user input data and the model data. The relative position data represents the relative positions of the vertebrae if the implants and / or osteotomies are applied according to the user input data, potentially taking biomechanical simulations or CNNs into account to predict the real spinal alignment, and thus indicate the predicted outcome of the surgery.

[0109] In a second process, the alignment planning block plans implants and / or osteotomies based on user input data indicating the desired alignment of the spine after surgery. The user input data for example defines target relative positions of the vertebrae. The user input data may be generated by manual manipulation of the vertebrae based on the 3D image data and the segmentation data or based on a generic 2D or 3D model of the spine. The segmented vertebrae, or the vertebrae of the generic model, are displayed using the display block described below and can be moved (or arranged) by a user, using an input device like a mouse, a touch screen or a data glove, to generate or modify the user input data. The user can also input target angles between neighboring vertebrae.

[0110] Based on the user input data, the alignment planning block calculates one or more implants and / or one or more osteotomies as the alignment planning data based on the user input data and the model data. Calculating an implant includes the identification or selection of an implant and the position of the implant. The implant is for example positioned relative to the 3D image data of a vertebra or another implant.

[0111] The second process can query an outcome database comprising information on the outcome and the corresponding plan of past surgeries. The outcome database may be part of the registry. A difference between a planned outcome and an actual outcome stored in the outcome database can be used for compensation of systematic errors or biases during planning. Data in the outcome database corresponding to patients similar to the current patient for whom planning is performed might be found and used. The compensation involves modifying one or more of the implants and osteotomies. In one example of using the outcome database, the dataset in which the outcome coincides best with the desired alignment while the pre-operative status coincides sufficiently with the pre-operative status of the patient, for example as represented by the image data in the patient data, is determined and the information on the implants and / or osteotomies of the plan stored in this dataset are used.

[0112] The second process could use a CNN to generate the alignment planning data or could test a plurality of candidate alignment planning data to find the one which comes closest to the user input data indicating the desired alignment of the spine after surgery.

[0113] In a third process, the alignment planning block automatically plans cages and / or osteotomies. This planning is based on medical knowledge, which can be generalized medical knowledge, for example how many cages and / or osteotomies are to be added in total. This medical knowledge is for example retrieved from a heuristics database or is input by a user.

[0114] The third process can query an outcome database comprising information on the outcome and the corresponding plan of past surgeries. The outcome database may be part of the registry. A difference between a planned outcome and an actual outcome stored in the outcome database can be used for compensation of systematic errors or biases during planning. Data in the outcome database corresponding to patients similar to the current patient for whom planning is performed might be found and used. The compensation involves modifying one or more of the cages or the rod shape.

[0115] In one example of using the outcome database, the dataset in which the pre-operative status coincides best with the pre-operative status of the patient, for example as represented by the image data in the patient data, while the patient satisfaction associated with this dataset is good enough is determined and the information on the implants and / or osteotomies of the plan stored in this dataset are used. The patient satisfaction indicates how satisfied the user on whom the dataset is based was with his surgery.

[0116] The process may involve acquisition of user input data representing a manual correction of the plan. The manual correction comprises one or more of a correction of the position of the cage and / or the osteotomy and selecting another size, height, type or material of the cage. The rod may be updated automatically to reflect the manual correction. The user input data typically overrides an automatic planning.

[0117] In a fourth process, the alignment planning block automatically plans one or more screws. This planning is based on medical knowledge, which can comprise a screw insertion philosophy. This medical knowledge is for example retrieved from a heuristics database or is input by a user. Planning a screw involves selecting a screw, for example from a database of available screws, and / or defining a position of the screw relative to a vertebra, in particular relative to the vertebra to which the screw is to be attached.

[0118] The process can involve planning of one or more screws based on the shape of a rod connecting the screw or screws. The screw(s) can for example be planned such that the curvature of the rod is minimized or an available pre-bent rod can be used. The process can for example access a database of available rods, including properties like shape and size of the rods.

[0119] The process can involve planning of a screw in order to minimize skiving. Skiving means a displacement of a drill because it slides on the surface of a bone. One criterion might be to make the axis of the screw as perpendicular as possible to the surface of the vertebra at the insertion position of the screw.

[0120] The process can involve planning of a screw in order to minimize screw loosening. Screw loosening means a process in which a screw loses its stable connection to the surrounding bony structure. One criterion might be to make the diameter of the screw as big as possible in comparison to the possible implantation paths in the vertebra or to change the implantation path such that the screw will cut through the stable out part of the vertebrae more than once (‘bicortical’ screws, ‘in-out-in’-screws).

[0121] The process can query an outcome database comprising information on the outcome and the corresponding plan of past surgeries. The outcome database may be part of the registry. A difference between a planned outcome and an actual outcome stored in the outcome database can be used for compensation of systematic errors or biases during planning. Data in the outcome database corresponding to patients similar to the current patient for whom planning is performed might be found and used. The compensation involves modifying one or more of the screws.

[0122] In one example of using the outcome database, the dataset in which the pre-operative status coincides best with the pre-operative status of the patient, for example as represented by the image data in the patient data, while the patient satisfaction is good enough is determined and the information on screws of the plan stored in this dataset are used. The patient satisfaction indicates how satisfied the user on whom the dataset is based was with his surgery.

[0123] In a fifth process, the alignment planning block automatically plans a set of one or more screws, one or more cages, one or more osteotomies and one or more rods. However, the result of the planning can be that one or more of said components are not to be used.

[0124] The planning in this process for example obtains an initial plan based on heuristics and then automatically corrects, varies or optimizes the initial plan.

[0125] In one aspect, the planning in this process uses medical knowledge, like the standard lordosis, and geometric constraints for easy insertion, like of a screw or a cage.

[0126] In one aspect, the planning in this process optimizes the rod, preferably including the optimization of the position(s) of one or more screws and the shape of the rod. The rod shall have a shape, and after implantation also a position, such that all screws can be connected to it. In addition, the rod shall induce and fixate the desired alignment correction after fixation of the screws to the rod.

[0127] One goal for optimization of the rod may be that the rod is as straight as possible when viewed in the anterior-posterior direction, which means when it is projected into the coronal or frontal plane. If it is not possible to have a rod with this property, the goal might be having a rod with parts as large as possible which have this property and there is a curvature only in a part as small as possible. This curvature is preferably uniform. In a lateral viewing direction, that is when projected into the sagittal or longitudinal plane, the rod is advantageously curved continuously and / or uniformly per section, wherein a section preferably is an anatomical section, like the lumbal section, thoracic section or cervical section.

[0128] The process can query an outcome database comprising information on the outcome and the corresponding plan of past surgeries. The outcome database may be part of the registry. A difference between a planned outcome and an actual outcome stored in the outcome database can be used for compensation of systematic errors or biases during planning. Data in the outcome database corresponding to patients similar to the current patient for whom planning is performed might be found and used. The compensation involves modifying one or more members of the set of one or more screws, one or more cages, one or more osteotomies and one or more rods.

[0129] In one example of using the outcome database, the dataset in which the pre-operative status coincides best with the pre-operative status of the patient, for example as represented by the image data in the patient data, while the patient satisfaction is good enough is determined and the information on the implants, the rod and / or osteotomies of the plan stored in this dataset are used. The patient satisfaction indicates how satisfied the user on whom the dataset is based was with his surgery.

[0130] In the processes mentioned above, querying the outcome database may be replaced by training a CNN with the outcome database, presenting the relevant data of the corresponding process to the trained CNN and letting the trained CNN find a suitable plan. Using a CNN in this manner and querying the outcome database are two examples of using knowledge about past surgeries for generating the alignment planning data.

[0131] In any of the processes mentioned above, user input data may be acquired. This user input data describes the clinical approach selected by the surgeon and / or available instruments. An example of a clinical approach is a lateral approach in which cages are only inserted laterally. The process then only selects cages which can be inserted this way. The planning may consider user preferences like implant / screw insertion philosophies or fixation philosophies. Common screw insertion philosophies are for example “lateral mass vs. pedicle screw” or “ilieum vs. S2-AI screws”. These screw insertion philosophies can come from presets, can be user selected or can be suggested (heuristics, outcome database). The fixation philosophies concern the number of spinal levels to be fixated. Common philosophies involve “one level above and one level below”, “two-above, one below” or “two-above, two below”.

[0132] The philosophy to be considered by the alignment planning block may be pre-set, user selected, selected using heuristics or from an outcome database or learned by a CNN from outcome database data.

[0133] The alignment planning block can combine two or more of the processes. For example, the alignment planning data can be calculated automatically using the third, fourth or fifth process and the outcome with those planning data is displayed using the display block. The user can modify the alignment planning data and the alignment planning data are used as user input data for the first process of the alignment planning block. The user may for example modify one or more implants and / or one or more osteotomies. The outcome based on the modified implants and / or osteotomies is thus generated and for example displayed.

[0134] The user may for example move one or more vertebrae, thus generating user input data. The second process of the alignment planning block is then performed based on this user input data. This results in revised alignment planning data in terms of implants and / or osteotomies.

[0135] In the second to fifth process, the alignment planning block may further calculate relative position data based on the calculated alignment planning data and the model data. The relative position data represents the relative positions of the vertebrae if the plan is implemented as defined by the alignment planning data, and thus indicate the predicted outcome of the surgery. The output data of the alignment planning block are alignment planning data. The alignment planning data may represent information on at least one of one or more implants and one or more osteotomies. Information on an implant comprises the type of implant and / or the position of the implant. Information on an osteotomy comprises the shape and / or the position of the osteotomy. The position of an implant or an osteotomy is for example defined relative to a segment of the 3D image data which represents the vertebra to which the implant is to be implanted or to which the osteotomy is to be made. The alignment planning data can be the user input data of the first process or the calculated data of the second to fifth process.

[0136] The second to fifth processes may consider limitation data comprised in the user input data. The limitation data represents one or more of a positive selection of an implant, the negative exclusion of an implant, a maximum or minimum size of an implant, an exclusion of a vertebra from an osteotomy or from receiving an implant, a maximum or minimum size of an osteotomy, position constraints of an implant or an osteotomy or an area of the spine excluded from receiving implants and / or osteotomies. The limitation data means constraints for the actual planning.

[0137] A positive selection of an implant means that the user input data specifies a particular implant to be used by the process. A negative exclusion of an implant means that the user input data specifies a particular implant not to be used by the process. If an implant is available in different sizes, some of those sizes can be excluded from consideration by the process according to the maximum or minimum size of the implant.

[0138] An area excluded from receiving implants and / or osteotomies may be a part of the spine, for example in terms of two or more adjacent vertebrae. An area excluded from receiving an implant may for example be a disc space, such that the plan does not comprise replacing the disc in this disc space by a cage.

[0139] The exclusion of a vertebra from an osteotomy means that the process must not plan an osteotomy in the specified vertebra. A maximum or minimum size of an osteotomy limits the size of an osteotomy to be planned by the process. Position constraints positively define areas in which an implant or an osteotomy might be placed or negatively define areas in which an implant or an osteotomy must not be placed by the process.

[0140] If the alignment planning block performs one of the second to fifth processes, the alignment planning data may additionally comprise the relative position data. In case of the second process, the relative position data can be the user input data, that is the basis for the planning, or relative position data calculated based on the model data and the alignment planning data calculated by the second process.

[0141] It shall be noted that the relative position data may be calculated by the display block rather than by the alignment planning block.

[0142] Any process of the alignment planning block may use the analysis data, or parts thereof. Planning of implants and / or osteotomies can for example be limited to structures indicated by the structure identification data.

[0143] Segments indicating pedicles may be used to plan the size and / or position of an implant, in particular of a screw in a screw-rod-system.

[0144] Segments indicating an intervertebral disc, or an intervertebral disc space, may be used to plan the complete removal of the spinal disc and to plan the size and / or position of a cage.

[0145] Segments indicating vertebrae may be used to perform biomechanical modelling of intervertebral forces, to predict the global post-operative alignment of the spine and to plan corrections of said alignment.

[0146] If existing implant data are input to the alignment planning block, the planning can be based on the existing implants. The planning block may for example plan a rod to which existing screws can be fixed or plan new screws such that they can be fixed to an existing rod. In addition or as an alternative, the planning block may use existing implants for the plan. Adapting the relative position between two or more vertebrae by using a different rod or modifying an existing screw might be chosen over adding a new screw to another vertebra. If feasibility data is input to the alignment planning block, this means that there is an existing plan for which the feasibility has been determined. The existing plan might have been generated by the alignment planning block in a previous execution and the feasibility data might have been generated by the feasibility test block described below. The existing plan might be stored within the alignment planning block or might have been input together with the corresponding feasibility data. However, the plan and the corresponding feasibility data might have been retrieved from a data source and might have been created by some person or another planning tool.

[0147] If the feasibility data indicates that the plan is not feasible, the alignment planning block creates a new plan including new alignment planning data and, optionally, new relative position data. The new plan is different from the non-feasible plan. If the feasibility data indicates which implants or osteotomies are non-feasible, the new plan might maintain some or all of the other implants and osteotomies and only avoids the non-feasible implants and osteotomies.

[0148] Decompression Planning Block

[0149] The decompression planning block generates a plan which relieves compressed nervous tissue. Decompression can involve laminectomy in which a section of bone is removed from a vertebra or discectomy in which a section of a damaged disc is removed. Decompression planning involves planning the position and the size of tissue to be removed, wherein the term “tissue” in this context also comprises bone material, for example.

[0150] The input data to the decompression planning block are patient data, in particular the image data, the analysis data and optionally one or more of the fusion data, the alignment planning data and the feasibility data.

[0151] Based on the segmentation data and the image data, compressed nervous tissue is identified. The shape of the uncompressed nervous tissue is calculated and bone tissue occupying target space of the uncompressed nervous tissue, and thus to be removed, is identified. The abnormality data in the analysis date may indicate a prolapse. In this case, the parts of the prolapse which reach into the spinal canal can be planned to be removed.

[0152] The abnormality data in the analysis data may indicate spinal cord compression. Spinal cord compression may cause prickling, pain, numbness or even complete loss of sensation or control. The decompression planning block may plan tissue removal to remove the spinal cord compression.

[0153] The abnormality data in the analysis data may indicate a tumor. The decompression block may plan removal of the tumor, either as a part of removal of the spinal cord compression or a primary resection target in its own right.

[0154] The decompression planning block optionally receives user input data. In one example, the user input data indicates tissue to be removed. The user input data may be drawn in image data displayed by the display block. In another example, the user input data indicates changes to automatically determined position and shape of the tissue to be removed. The user input data may be drawn in image data displayed by the display block in combination with the automatically determined position and shape of the tissue to be removed.

[0155] If feasibility data is input to the decompression planning block, this means that there is an existing plan for which the feasibility has been determined. The existing plan might have been generated by the decompression planning block in a previous execution and the feasibility data might have been generated by the feasibility test block described below. The existing plan might be stored within the decompression planning block or might have been input together with the corresponding feasibility data. However, the plan and the corresponding feasibility data might have been retrieved from a data source and might have been created by some person or another planning tool.

[0156] If the feasibility data indicates that the plan is not feasible, the decompression planning block creates a new plan including new decompression planning data. The new plan is different from the non-feasible plan. If the feasibility data indicates which tissue removals are non-feasible, the new plan might maintain some or all of the other tissue removals and only avoids the non-feasible tissue removals.

[0157] The output data of the decompression planning block are decompression planning data representing information on the position and the shape of the tissue to be removed.

[0158] Feasibility Test Block

[0159] The feasibility test block checks whether or not the plan comprising the alignment planning data and / or the decompression planning data is feasible. In particular, it checks whether the implants in the alignment planning data can be implanted, the osteotomies in the alignment planning data can be made or the tissue indicated by the decompression planning data can be removed. This in particular considers the accessibility of the corresponding site, for example by considering the access path from outside of the patient to the corresponding site.

[0160] The input data to the feasibility test block are the patient data, in particular the 3D image data, and one or both of the alignment planning data and the decompression planning data. Optional input data are analysis data, in particular one or more of segmentation data, OAR data and structure identification data.

[0161] If alignment planning data representing an implant or an osteotomy are input to the feasibility test block, a virtual object may be assigned to the implant or osteotomy. The virtual object defines a geometrical region relative to the implant or osteotomy, respectively, required for placing the implant or making the osteotomy, respectively. The virtual object is for example retrieved from a database associating virtual objects to implants or osteotomies, respectively. The shape of the virtual object can for example be a cone or a cylinder. The virtual object is for example a backwards prolongation of the implant, wherein “backwards” means the opposite direction to the insertion direction of the implant.

[0162] The implant or the osteotomy, together with the assigned virtual object, is aligned with the 3D image data according to the alignment planning data. The feasibility test block then analyzes the 3D image data within the boundaries of the virtual object. For example, it is determined whether or not predetermined body parts, like organs at risk or bones, or predetermined physical objects, like instruments used during surgery or existing implants, are located at least partly within the boundaries of the virtual object. If this is the case, the plan is not feasible. Predetermined body parts may be identified in the 3D image data based on the segmentation data, the OAR data and / or the structure identification data.

[0163] If decompression planning data are input to the feasibility test block, it is determined whether or not the tissue to be removed is accessible, for example without affecting structures at risk of the patient, and / or whether or not a structure, like a vertebra, would still be stable after the planned tissue is removed. The structure is for example still considered to be stable if it would not break due to the weight or movement of the patient.

[0164] The output data of the feasibility test block are feasibility data indicating whether or not the plan is feasible. The feasibility data may indicate which implantation or osteotomy is not feasible. Block

[0165] The surgery companion block is executed parallel to the surgery. It adapts the plan to the current situation, provides guidance information to the surgeon and / or tracks implants. The input data to the surgery companion block are the planning data and the patient data, in particular the 3D image data. Optional input data are the analysis data, in particular the segmentation data. It shall be noted that the surgery companion block solely relates to data processing and does in itself not mean or include any physical interaction with any living being.

[0166] The surgery companion block acquires intra-operative patient data, like intra-operative image data and / or point clouds. A point cloud comprises locations of points, like landmarks, of the patient. The locations are for example used to register the patient to a spatial coordinate system of a medical tracking system. The locations are for example sampled using a tracked instrument. The instrument is for example a trackable pointer. Image data typically comprises one or more x-ray images (2D image data) or tomography image data (3D image data) captured using a tracked imaging system or trackable imaging phantoms. Tracking is typically performed using a medical tracking system such that the intra-operative patient data are defined in a reference system of the medical tracking system. Trackable imaging-phantoms are a combination of radiopaque markers visible in the image data and optical or electromagnetic markers detectable by the medical tracking system. Since the position of the radiopaque markers relative to the optical or electromagnetic markers is known, the positions of the radiopaque markers, or fiducials, in the reference system of the medical tracking system is known and the radiopaque markers detected in the image data can be registered with the optical or electromagnetic markers.

[0167] If the intra-operative image data is captured using a calibrated and tracked imaging system or a trackable imaging phantom in the scan, both can also be used to register the patient to the spatial coordinate system of the medical tracking system.

[0168] The plan is then transferred to the intra-operative situation, in particular to the position of the patient in the reference system of the medical tracking system, by registering pre-operative image data, like the image data within the patient data or the enhanced data within the fusion data, with the intra-operative patient data. In particular, the preoperative image data are registered individually, for example segment by segment, with the intra-operative image data since the intra-operative posture of the patient typically differs from the one when the pre-operative image data were captured. This can be implemented using multi rigid fusion in which parts of the image data are registered with each other in a rigid manner and independently of the other part of the image data. In one example, parts of the image data showing different vertebrae are registered to the intra-operative patient data individually to virtually position the vertebrae and the associated parts of the plan in the reference system of the tracking system.

[0169] At the same time, the planning data is adapted to the intra-operative situation since it is defined relative to the pre-operative image data. As a result, the target positions of implants and osteotomies are known in the reference system of the medical tracking system. The surgery companion block calculates and outputs guidance information which aids the surgeon in positioning an implant, in particular a screw or a cage, in making an osteotomy or in removing tissue for decompression. In this context, a medical instrument is tracked and the tracked position of the medical instrument is compared to a target position which is based on the adapted planning data.

[0170] The medical instrument can be a drill, a saw, a milling tool, a guide for one or more of the aforementioned instruments or a positioning tool attached to an implant for handling the implant during positioning. The position of the implant relative to the positioning tool is known, and for example predetermined or measured.

[0171] The surgery companion block optionally determines implant positioning data representing the actual position of an implant and / or osteotomy positioning data representing the actual position of an osteotomy. The position can for example be determined from the position of the implant positioning tool when the implant is at its final position, by tracking the drill, the saw, the corresponding guide or the milling tool while making the osteotomy or by sampling points on the vertebra and or the implant, as applicable. In another example, the position of an implant or an osteotomy is determined from intra-operative images captured after positioning the implant or making the osteotomy. While the position is initially determined in the reference system of the medical tracking system, it is transferred to the 3D image data using an inverse of the transformation used when adapting the planning data to the intra-operative situation. The result are implant positioning data representing the position of an implant relative to a segment of the 3D image data representing a vertebra or osteotomy positioning data representing the position of an osteotomy relative to a segment of the 3D image data representing a vertebra.

[0172] The surgery companion block optionally determines intra-operative tissue removal data representing the volume of the tissue removed for decompression. The intraoperative tissue removal data can be based by tracking the instrument(s) used for removing the tissue. The surgery companion block optionally determines progress data indicating the progress of an osteotomy or a tissue removal. This can be based on the comparison of a planned osteotomy with the osteotomy positioning data or the comparison of planned tissue removal with the intra-operative tissue removal data. The progress data can be calculated at different points in time to represent the progress over time. The progress data can simply indicate a numerical value of the progress between 0 percent and 100 percent or be image data showing both the planned osteotomy or tissue removal and the part of the osteotomy already performed or the tissue already removed.

[0173] The surgery companion block optionally outputs a stop command to a tool used for performing the osteotomy if the osteotomy is complete or the tool leaves the planned area of the osteotomy. In analogy, the surgery companion block optionally outputs a stop command to a tool used for performing the tissue removal if the tissue removal is complete or the tool leaves the planned area of the tissue removal.

[0174] The surgery companion block optionally calculates and outputs guidance information which aids the surgeon in aligning a vertebra to a rod or a rod to a screw.

[0175] Regarding alignment of a vertebra to a rod, the aim is to induce additional corrections for the alignment by mechanically displacing or moving anatomical parts (vertebral bodies or parts of vertebral bodies, for example in case of osteotomies) before the positions are "frozen" by fixing the screws on the rod.

[0176] This process can be navigated or supported by a navigation system. For example, a gap caused by wedge osteotomy (like for example a pedicle subtraction osteotomy) would first be closed or opened after tissue removal (to increase or decrease the spinal curvature) and then screwed tight to the rod.

[0177] This closing and opening or other tilting, rotating and lateral movements can be navigated (or integrated into a planning), wherein the system indicates whether one or more anatomical parts are already correctly placed (relative to each other or to the rod) - or what effects the current position would have on the outcome. This could be done "live", if, for example, individual anatomical structures, like vertebrae, are tracked. Another option is an iterative approach of measuring the positions of the anatomical parts, for example by digitizing using a tracked instrument or imaging, of determining the deviation and of displaying the correction (or updating the plan).

[0178] The positions of anatomical structures are represented by structure position data, which can be output by the surgery companion block.

[0179] Aligning a rod to a screw means positioning the rod such that the screw can be fixed to the rod. It is preferably possible to "navigate" a rod. This makes sense if the surgery is not performed openly and the end of the screw which is to be fixed to the rod is not visible.

[0180] In minimally invasive surgeries, the rod is inserted through the tissue in such a way that one screw head after another is "threaded" onto the rod. In this step, the rod tip can be navigated to the positions of the screw heads. Screw head positions must of course be known, for example from planning, or can be detected in an intra-operative image or measured by tracking.

[0181] The surgery companion block optionally determines instrument position data, which is data representing the position of an instrument during surgery. The instrument position data can represent a single position of the instrument at a particular instant in time or a sequence of positions over time. The position of an instrument is for example determined using a medical tracking system.

[0182] The surgery companion block may decide or may be instructed by user input to repeat the planning process based on the intra-operative patient data, and in particular the intra-operative image data. The planning process is repeated with the intra-operative image data as the image data, starting for example with the data analysis block.

[0183] The repeated planning process may create a new plan from scratch or modify the previous plan based on the intra-operative image data. The output data of the surgery companion block are intra-operative data, including one or more of the intra-operative patient data, the implant positioning data, the osteotomy positioning data, the tissue removal data, the structure position data, the intraoperative image data, the instrument position data and the progress data.

[0184] Post-OP Control Block

[0185] The post-op control block enables post-operative control of the surgery. Input data to the post-op control block are the planning data and optionally one or more of patient data, in particular the 3D image data, analysis data, in particular the segmentation data, model data and intra-operative data, in particular the structure position data.

[0186] The post-op control block can be executed immediately after completion of the surgery or a step of the surgery. In addition, or alternatively, it can be executed once or multiple times after surgery, for example days, weeks, months or years after surgery. The time between different executions of the post-op control block can be set identical or different from each other. The post-op control block is preferably executed in combination with the registry supplementing block to add the post-operative data to the registry described below.

[0187] The post-op control block acquires one or both of post-operative image data and postoperative structure position data. The post-operative image data can be 2D image data, 3D image data or a combination thereof. The post-operative structure position data for example represents a point cloud consisting of surface points of an anatomical structure or of an implant or the position of an anatomical structure or an implant. If intra-operative data is input to the post-op control block, the structure position data of the intra-operative data can be used as the post-operative structure position data.

[0188] The post-operative image data is for example acquired from an x-ray imaging system. Vertebrae and / or implants and / or osteotomies are identified in the post-operative image data and the positions of implants and osteotomies relative to the corresponding vertebrae are determined as post-operative implant data and post-operative osteotomy data, respectively. The determined relative positions are compared to the relative positions in the planning data to generate deviation data representing if and in how far an actual relative position differs from a planned relative position.

[0189] Based on the post-operative implant data, post-operative alignment data can be calculated which represents the post-operative alignment of the spine. This calculation may utilize the model data. The post-operative alignment data can be image data or numerical data. The image data can be 2D image data, like x-ray image data, or 3D image data, like tomography image data. The post-op control block may cause image data to be captured or may adapt image data in the patient data to the post-operative status of the spine. The numerical data can represent numerical values of angles between neighboring vertebrae.

[0190] The image data in the patient data and the segmentation data may be used when identifying a vertebra in the post-operative image data. A segment of the 3D image data corresponding to a vertebra can be registered with the post -operative image data to identify a vertebra and to determine the position of the vertebra in the reference system of the post-operative image data.

[0191] The post-op control block may calculate post-operative tissue removal data representing the removed tissue for achieving decompression by comparing postoperative image data of a vertebra with image data in the patient data, which is preoperative image data, of the same vertebra.

[0192] The output data of the post-op control block are post-operative data which include one or more of the deviation data, the post-operative implant data, the post-operative osteotomy data, the post-operative tissue removal data, the post-operative image data, the post-operative alignment data and the post-operative structure position data.

[0193] Documentation Block

[0194] The documentation block creates a documentation of the planning process and in particular documents the final plan and / or the current (updated) plan. It may also contain the measured result of the surgery and the deviations to the plan. Input data to the documentation block are one or more of patient data, analysis data, standing 3D image data, comparison data, enhanced data, model data, user input data, alignment planning data, decompression planning data, feasibility data, display data, existing documentation data and deviation data.

[0195] Further input data to the documentation might be intra-operative data and / or postoperative data.

[0196] The output data of the documentation block are documentation data. The documentation data can comprise DICOM data and / or human readable data, like text or graphics.

[0197] The documentation block may receive feedback data. The feedback data may be received from a doctor or the patient and represents the subjective satisfaction with the surgery.

[0198] In minimum documentation, the documentation data only represents the planning data, and can thus be understood as a list of implants, osteotomies and tissue removals. In full documentation, the documentation data represents all data collected or generated by the data processing method. Any kind of documentation between those extremes is possible. The documentation data in particular represents all data provided as input data to the documentation block during execution of the data processing method.

[0199] If multiple plans are generated, for example by executing the alignment planning block and / or the decompression planning block multiple times, the documentation data may comprise the data of each of those plans, including the corresponding feasibility data and the corresponding display data. The documentation data may indicate the order in which multiple plans have been generated and added to the documentation data.

[0200] The documentation block is for example executed after the first execution of one or both of the alignment planning block and of the decompression planning block. In this case, the documentation data is based on all data input to the documentation block. After a subsequent execution of one or both of the alignment planning block and of the decompression planning block, the documentation block receives the planning data generated by said subsequent execution and adds them to the documentation data. If existing implant data generated by the data analysis block is based on post-operative image data, the documentation block may add those existing implant data to the documentation data as data which documents the actual positions of the implants after surgery. In this case, said existing implant data represents implants which were implanted during surgery. At the time of documentation, those implants are existing implants.

[0201] The documentation block either stores the documentation data internally such that the documentation block can supplement any stored documentation data upon subsequent execution of the documentation block or receives previously generated documentation data as input data in terms of existing documentation data.

[0202] Display Block

[0203] The display block displays data. In terms of data processing, displaying means that display data comprising content for visual display is generated. The input data to the display block are patient data and optionally one or more of analysis data, fusion data, planning data, feasibility data, intra-operative data and post-operative data. The output data of the display block are the display data.

[0204] The display data can be two-dimensional display data, for example for display on a two-dimensional display apparatus like a screen, or three-dimensional display data, for example for display on a stereoscopic display apparatus, like a headset, a 3D screen or a 3D projector. In case of two-dimensional display data representing 3D image data, a suitable projection of the 3D image data into two dimensions is calculated. In case of three-dimensional display data representing 2D image data, the 2D image data might be presented at a predetermined depth. Several layers of 2D image data can be positioned at different depths. Additional information, like rendered text, can be treated like 2D image data.

[0205] One or more of the processes described below might be performed by the display block. In a first process, the display block displays some or all of the image data comprised in the patient data. The first process optionally also displays additional data, like some or all of the fusion data, the physiological data in the patient data or the analysis data. The additional data is for example rendered into image data and positioned relative to the image data. Graphics indicating to which part of the image data the additional data belongs might be added to the display data.

[0206] The display data might indicate the boundaries of vertebrae based on the segmentation data in the analysis data. The display data might indicate labels of vertebrae based on the vertebrae identification data. The display data might indicate organs at risk based on the OAR identification data and / or anatomical structures based on the structure identification data.

[0207] The display data might represent some or all enhanced data. The bone density and / or the bone strength might be represented by colors. Muscle attachment points might be indicated by marks or arrows. Inflammation areas might be indicated by colors or textures. Osteophytes and / or impingements might be indicated by colors, boundaries or textures. In this context, an indication by color means that image data is colored rather than having grey values.

[0208] The display data might represent the matched image data.

[0209] The display data might comprise representations of implants and / or osteotomies. A representation of an implant is a visual representation, like a generic x-ray representation showing how the implant would be depicted by an x-ray image. The representation is selected based on the type of implant and positioned in the display data according to the alignment planning data. The planned implants are thus displayed together for example with the 2D image data. A representation of an osteotomy is also a visual representation, like a colored area. The representation of an implant and / or an osteotomy could be a pictogram, like a symbol, for example a generic pictogram like a cross, a geometric shape or an exclamation mark. The pictogram simply indicates that an implant and / or osteotomy is planned, without necessarily indicating the exact position thereof. The pictogram is for example partly or fully overlaid over the vertebra, pedicle or intervertebral space which is to receive the implant and / or osteotomy.

[0210] In the first process, the image data in the patient data might be modified using the relative position data output by the alignment planning block and the segmentation data. This means that the vertebrae are isolated using the segmentation data and rearranged using the relative position data such that the modified image data represents the expected shape of the spine after surgery. This might involve using the model data. The modified image data can be displayed instead of or together with the image data, for example as an overlay or side by side. Representations of implants and / or osteotomies are preferably added to the modified image data only.

[0211] When displaying 2D image data, in particular x-ray image data, the image data may comprise numerical values representing one or more angles or distances between vertebrae.

[0212] When displaying of 3D image data on a 2D display apparatus, viewpoint input data provided by a user can be used to calculate a two-dimensional view of the 3D image data for the display data. The viewpoint input data describes the viewing direction onto the 3D image data. When displaying 3D image data on a 3D display apparatus, the viewpoint input data is used to virtually align the 3D image data with the 3D display apparatus to have a desired viewing direction onto the 3D image data.

[0213] Displaying 3D image data on a 2D display may additionally or alternatively involve using multiplanar reconstructions (MPR) or generating digitally reconstructed radiographs (DRR).

[0214] Upon displaying of image data, delineation data may be received from a user to be used in manual segmentation, OAR identification, structure identification, abnormality detection or existing implant detection.

[0215] Upon displaying the 3D image data, user input may be received which modifies the position of a vertebra, which is referred to as modified vertebra, relative to another vertebra. This may be used to generate the target relative positions of the vertebrae for the second process of the alignment planning block. The 3D image data is displayed again based on the modified relative position such that the new anatomy of the spine can be recognized in real time. When modifying the position of a vertebra relative to another vertebra, the remaining vertebrae behind the modified vertebra may be assumed as being rigid with the modified vertebra such that their relative positions are also modified along with the modified vertebra.

[0216] Two image data from different points in time can be displayed based on the comparison data.

[0217] It is possible to display two or more of 2D image data, 3D image data and spine model data. The spine model data represent a generic image or a mesh model of a spine. The spine model data is preferably adapted to the actual situation of the patient using intra-operative data. For example, the spine model data is matched onto intraoperative image data or a point cloud representing locations of landmarks.

[0218] 2D and 3D image data might be displayed side by side or overlaid, for example in terms of so-called 2.5D images. In 2.5D images, objects protrude from a 2D background.

[0219] A 3D image can be reduced to a 2D image, for example using MPR or DRR, and then be displayed side by with or overlaid with a 2D image.

[0220] The segmentation data can be used to isolate an anatomical structure from 2D or 3D image data and to overlay the isolated anatomical structure over 3D or 2D image data in the display data.

[0221] It is possible to generate display data including a visualization of the spine, or of parts thereof, which is assembled from different 2D image data and / or 3D image data, for example if those 2D image data and 3D image data do not overlap or only overlap slightly. This visualization can be adapted to the current status of the patient as explained above.

[0222] Image data of different points in time can be displayed side by side, as an animation or in any other suitable way. The image data at different points I time can be of differing modalities. It is possible to generate intermediate images between two images, thus morphing from the first image to the second image. Depending on the available image data, the animation can comprise image data of one or more pre-operative points in time, an intra-operative point in time and / or one or more post-operative points in time. Image data of post-operative points in time can be calculated from the planning data and the model data and may represent post-operative changes of the spine over time. Image data of post-operative points in time can be post-operative image data.

[0223] The display data may comprise a diagram which for example represents the development of a numerical value over time. This numerical value is for example an angle or a distance between two vertebrae.

[0224] In a second process, the display block displays feasibility data. In one example, the display block adds an information item, like text or a symbol, indicating whether or not the plan is feasible, to the display data. In another example, the display block indicates one, several or all implants or osteotomies in the display data which are not feasible. This indication might involve a particular color of a non-feasible implant or osteotomy, adding a text stating the non-feasible implants and osteotomies or otherwise marking non-feasible implants and osteotomies, like adding an arrow pointing to the non- feasible implant or osteotomy. It is also possible to highlight collision areas in which a medical instrument and the anatomy would collide, for example using a predetermined colorization or hatching.

[0225] In a third process, which is performed concurrently to the surgery companion block, the display block displays the current position of an implant relative to one or more segments of the 3D image data for example based on the tracked position of the positioning tool attached to the implant or the intra-operative data provided by the surgery companion block.

[0226] In a fourth process, the display block displays a planned spine anatomy by using the relative position data to arrange the segments of the 3D image data accordingly. The segments are thus displayed with relative positions therebetween which correspond to the predicted outcome of the surgery according to the plan. The relative position data may be input data to the display block which are output by the alignment planning block or may be calculated within the display block based on the alignment planning data and the model data.

[0227] In a fifth process, which is performed if the display block is executed after the post-op control block, the display block displays one or more of the post-operative implant data, the post-operative osteotomy data and the post-operative tissue removal data. This may mean displaying text representing parameters, like coordinates, of the postoperative position, and optionally also of the planned position. In addition or alternatively, this may mean that the display block displays a segment of the 3D image data together with a representation of an implant according to the post-operative implant data, a representation of an osteotomy according to the post-operative osteotomy data or a representation of removed tissue according to the post-operative tissue removal data. The display block optionally also displays a representation of an implant or of an osteotomy according to the alignment planning data to visualize both the planned and the actual position of an implant or an osteotomy relative to the corresponding vertebra.

[0228] Yet in addition or alternatively, the display block uses the model data, the segmentation data and one or both of the post-operative implant data and the post-operative osteotomy data to calculate a visualization of the virtual post-operative spine, or parts thereof, from the 3D image data. The display data comprise this visualization, and optionally a visualization of the pre-operative spine, or parts thereof, and / or a visualization of the planned spine, or parts thereof. The visualization of the preoperative spine can be taken from the 2D image data in the patient data, be taken from the standing 3D image data or be generated from the 3D image data in the patient data. The visualization of the planned spine is calculated from the 3D image data and the planning data and optionally also from the model data.

[0229] In a sixth process, the display block displays the standing 3D image data together with at least the 2D image data showing a lateral view of the spine in a standing pose of the patient. This allows a verification of the standing 3D image data with the actual preoperative spinal alignment in the standing pose. Instead of or in addition to the standing 3D image data, the spine model data can be displayed. The spine model data is for example matched to the 2D image data in the patient data to represent the preoperative spinal alignment.

[0230] In a seventh process, the display block displays a visual representation of the model data, for example of the model data only, which means none of the image data in the patient data. The seventh process in particular displays a visual representation of the spinal column, or a part thereof, in the model data. The model data is preferably fitted to the patient, such that the visual representation of the model data represents the preoperative situation of the patient. The model data is for example matched to the 2D image data in the patient data to represent the pre-operative spinal alignment. In a modification of this process, the spine model data instead of the model data is displayed, wherein the spine model data is matched to image data in the patient data which represents the pre-operative situation of the patient.

[0231] In an eighth process, the display block displays a representation of the reported outcome data stored in the registry, for example if the display block is executed during post-operative evaluation of the surgery.

[0232] In a ninth process, the display block displays indication information which indicates an abnormality represented by the abnormality data. The indication information can be of any suitable kind, such as text, highlighting a part of image data showing the abnormality (for example using color, a border around the abnormality, a pointer to the abnormality or a combination thereof) or displaying the abnormality enlarged compared to the rest of the image data.

[0233] In a tenth process, the display block displays seventy information which represent a risk assessment, like how severe the abnormality is. This severity information is for example retrieved as the severity data. The severity can for example be displayed as text or by colorizing the abnormality in an image using a color scheme.

[0234] One or more processes, or parts of processes, of the display block might be performed iteratively with one or more processes, or part of processes, of the alignment planning block. In one example, the alignment planning block performs the first process and calculates relative position data based on user input data. The display block then displays the predicted outcome. If the user modifies the user input data, or inputs new input data, the first process of the alignment planning block is repeated with the new or modified user input data.

[0235] In one example, the alignment planning block performs one or more of the second to fifth processes and the display block displays the predicted outcome. If the user inputs limitation data or modifies limitation data, the process of the alignment planning block is repeated based on the new limitation data and the new outcome is displayed by the display block.

[0236] The output data of the display block is the display data. The display data is to be displayed on a suitable device.

[0237] In many constellations, it is advantageous if the display block is executed concurrently to another block, like for example the data analysis block, data fusion block, the model generation block, the alignment planning block, the decompression planning block, the feasibility test block or the surgery companion block. In this case, the data generated by the other block, or parts of those data, can be displayed in real time or virtually in real time such that a user can see what the other block currently does or to enable user input to be processed by the other block. Block

[0238] The registry supplementing block stores information about the surgery in a registry, which is a database of surgeries. The registry comprises information about one or more surgeries of one or more patients. Input data to the registry supplementing block are one or more of patient data, planning data, deviation data, post-operative image data and documentation data.

[0239] The post-operative image data may be the data output by the post-op control block, image data of at least a part of the spine of the patient at a point in time well after the surgery or a combination of both. The registry supplementing block may store the documentation data in the registry to have a full documentation of the surgery. The registry supplementing block may make sure that no data is stored in the registry twice. So if, for example, the documentation data comprises data already stored in the registry, this data is removed from the documentation data before the documentation data is stored in the registry or this data is maintained in the documentation data, but the already stored data is removed from the registry.

[0240] The registry supplementing block optionally acquires reported outcome data representing statements of the patient and / or medical personnel on the outcome of the surgery, and for example represents an assessment of pain and / or mobility of the patient at one or more points in time after the surgery.

[0241] In the registry, each surgery has a unique identifier, for example based on the identity of the patient and the date of the surgery.

[0242] If there is no entry for the surgery in the registry yet, the registry supplementing block creates a new entry including the data input to the registry supplementing block. If an entry for the surgery already exists, the registry supplementing block updates said entry with the data input to the registry supplementing block. This is for example the case if new post-operative image data are captured or new reported outcome data are acquired.

[0243] Embodiments of the Invention

[0244] Now embodiments of the invention combining multiple data processing blocks are described.

[0245] In a first aspect, the present invention relates to a computer-implemented method of data processing along at least a part of a medical spine surgery workflow.

[0246] In a first embodiment, the method comprises execution of the data acquisition block, the data analysis block, the model generation block, a planning block and the display block. The planning block can be one or more of the alignment planning block and the decompression planning block.

[0247] Patient identification data which uniquely identifies the patient for whom the surgery is to be performed is provided to the data acquisition block. The data acquisition block acquires data corresponding to the patient and outputs them as patient data. The patient data includes image data comprising one or both of 2D x-ray image data and 3D image data.

[0248] The data analysis block performs segmentation of the 2D image data and / or the 3D image data as present such that segments thereof represent the vertebrae. The model generation block generates a spine model tailored to the patient based on the image data in the patient data.

[0249] The planning block plans the surgery by determining planning data. The planning relates to alignment, decompression or both. In alignment planning, the planning block plans one or more implants, like cages, screws or rods, and / or one or more osteotomies. In decompression planning, the planning block plans tissue removal to relieve compressed nervous tissue.

[0250] Then the display block displays the planned implants, osteotomies and tissue removals together with the part or all of the image data.

[0251] In a first aspect, the invention is directed to a computer-implemented medical method of data processing along at least a part of a medical spine surgery workflow, comprising execution of a data acquisition block, a data analysis block, a model generation block, a planning block and a display block. The method comprises executing, on at least one processor of at least one computer (for example at least one computer being part of a navigation system), the following set of blocks which are executed by the at least one processor.

[0252] The method of the first aspect executes the data acquisition block to acquire patient data, the data analysis block to analyze the patient data to make the data processable by subsequent blocks, the model generation block to generate a biomechanical model of the patient, in particular of the spine system of the patient, a planning block to generate planning data of the surgery, and the display block to display data.

[0253] This set of blocks is the minimum set of blocks to accompany a spinal intervention in terms of data processing.

[0254] In one embodiment, the planning block is one or more of the alignment planning block and the decompression planning block, wherein the alignment planning block generates a plan which relates to the alignment of the patient’s spine and the decompression planning block generates a plan which relieves compressed nervous tissue.

[0255] In one embodiment, the method further executes the data fusion block that generates additional data about the patient by combining two or more parts of the patient data.

[0256] In one embodiment, the method executes the feasibility test block that checks whether or not the plan according to the planning data is feasible.

[0257] In one embodiment, the method executes the surgery companion block that performs at least one of adapting the planning data to the current situation, providing guidance information and tracking implants.

[0258] In one embodiment, the method executes the post-op control block that enables postoperative control of the surgery.

[0259] In one embodiment, the method executes the documentation block that creates a documentation of a planning process.

[0260] In one embodiment, the method executes the registry supplementing block that stores information about the surgery in a database.

[0261] In one embodiment, the method executes one, multiple or all of the data fusion block, both the alignment planning block and the decompression planning block, the feasibility test block, the surgery companion block, the post-OP control block, the documentation block and the registry supplementing block.

[0262] In any possible combination of blocks executed by the method, a block being executed receives all or some of the output data output by the block(s) already executed and performs processing according to the data input to said block. So even if the same combination of blocks is executed in the same order in two different applications, the processing within the blocks within the two applications might be different depending on the data available to or processed by the blocks of said application.

[0263] In a second aspect, the invention is directed to a computer program comprising instructions which, when the program is executed by at least one computer, causes the at least one computer to carry out method according to the first aspect. The invention may alternatively or additionally relate to a (physical, for example electrical, for example technically generated) signal wave, for example a digital signal wave, such as an electromagnetic carrier wave carrying information which represents the program, for example the aforementioned program, which for example comprises code means which are adapted to perform any or all of the steps of the method according to the first aspect. The signal wave is in one example a data carrier signal carrying the aforementioned computer program. A computer program stored on a disc is a data file, and when the file is read out and transmitted it becomes a data stream for example in the form of a (physical, for example electrical, for example technically generated) signal. The signal can be implemented as the signal wave, for example as the electromagnetic carrier wave which is described herein. For example, the signal, for example the signal wave is constituted to be transmitted via a computer network, for example LAN, WLAN, WAN, mobile network, for example the internet. For example, the signal, for example the signal wave, is constituted to be transmitted by optic or acoustic data transmission. The invention according to the second aspect therefore may alternatively or additionally relate to a data stream representative of the aforementioned program, i.e. comprising the program.

[0264] In a third aspect, the invention is directed to a computer-readable storage medium on which the program according to the second aspect is stored. The program storage medium is for example non-transitory. In a fourth aspect, the invention is directed to at least one computer (for example, a computer), comprising at least one processor (for example, a processor), wherein the program according to the second aspect is executed by the processor, or wherein the at least one computer comprises the computer-readable storage medium according to the third aspect.

[0265] In a fifth aspect, the invention is directed to a medical system, comprising: a) the at least one computer according to the fourth aspect; b) at least one electronic data storage device storing at least some of the patient data.

[0266] In particular, the disclosed method is not a method for treatment of the human or animal body by surgery or therapy. For example, the invention does not involve or in particular comprise or encompass an invasive step which would represent a substantial physical interference with the body reguiring professional medical expertise to be carried out and entailing a substantial health risk even when carried out with the reguired professional care and expertise. The disclosed method is solely foreseen to operate a data processing device or system to process data as described above.

[0267] DEFINITIONS

[0268] In this section, definitions for specific terminology used in this disclosure are offered which also form part of the present disclosure.

[0269] Definitions of Specific Terms data Represents the complete plan of the surgery, including alignment planning and decompression planning.

[0270] Alignment planning data: Represents the plan regarding the alignment of the spine. It can define the selection and / or position of an implant (cage, screw, rod) or osteotomies. The position of a cage or a screw is defined relative to a vertebra, the position of a rod is defined relative to a screw. The alignment planning data can optionally comprise relative position data.

[0271] Relative data Represents the predicted alignment of the spine after surgery, for example in terms of relative positions between vertebrae. Can be calculated based on the alignment planning data. data Represent the plan regarding decompression of the spine. Identifies, for example, material to be removed.

[0272] Patient data Represents data about the patient, like image data (3D, 2D), previous plans, physiological data (gender, age, height, weight, BMI), diagnosis data, existing documentation data. is data Represents the result of the data analysis block. It comprises one or more of segmentation data, classification data, vertebra identification data, OAR identification data, structure identification data, abnormality data, existing implant data, numerical data and severity data.

[0273] OAR identification data Identifies an organ at risk.

[0274] Structure identification data Identifies an anatomical structure of interest, like a vertebra. A structure of interest can for example be a structure on which surgery is to be performed.

[0275] Fusion data Represents the result of the data fusion block. It can comprise one or more of standing 3D image data (3D image data matched to a standing 2D image), comparison data (compare different states of the spine; can be image data or numerical data), enhanced data (image data with additional information, like bone density, bone strength, muscle attachment points, inflammation areas, osteophytes or impingements), predicted image data (shows the predicted post-operative spinal anatomy depending on a particular surgical plan), matched image data (image data matched to other data), fused image data (image data assembled from multiple data) Model data Describes the biomechanical model of the patient. resents whether or not the plan is feasible. Represents data obtained during surgery, like intra-operative patient data (image data or point clouds), implant positioning data (actual position of an implant), osteotomy positioning data (actual position of an osteotomy), intraoperative tissue removal data (data on tissue actually removed), structure position data

[0276] (actual position of an anatomical structure), intra-operative image data, instrument position data (actual position of a medical instrument) or progress data (indicating the progress of an osteotomy or a tissue removal).

[0277] Post-i data Represents data obtained after surgery, like deviation data

[0278] (represents if and in how far an actual relative position of a vertebra, implant or osteotomy differs from a planned relative position), post-operative implant data (position of an implant relative to the corresponding vertebrae after surgery), postoperative osteotomy data (position of an osteotomy relative to the corresponding vertebrae after surgery), post-operative tissue removal data (represents actually removed tissue), post-operative image data, post-operative alignment data (represents the post-operative alignment of the spine), post-operative structure position data (represents the position of an anatomical structure after surgery)

[0279] Documentation data Represents data of the surgery to save for documentation purposes.

[0280] Computer-implemented method

[0281] The method in accordance with the invention is for example a computer-implemented method. For example, all the steps or merely some of the steps (i.e. less than the total number of steps) of the method in accordance with the invention can be executed by a computer (for example, at least one computer). An embodiment of the computer implemented method is a use of the computer for performing a data processing method. An embodiment of the computer implemented method is a method concerning the operation of the computer such that the computer is operated to perform one, more or all steps of the method.

[0282] The computer for example comprises at least one processor and for example at least one memory in order to (technically) process the data, for example electronically and / or optically. The processor being for example made of a substance or composition which is a semiconductor, for example at least partly n- and / or p-doped semiconductor, for example at least one of II-, III-, IV-, V-, Vl-sem iconductor material, for example (doped) silicon and / or gallium arsenide. The calculating or determining steps described are for example performed by a computer. Determining steps or calculating steps are for example steps of determining data within the framework of the technical method, for example within the framework of a program. A computer is for example any kind of data processing device, for example electronic data processing device. A computer can be a device which is generally thought of as such, for example desktop PCs, notebooks, netbooks, etc., but can also be any programmable apparatus, such as for example a mobile phone or an embedded processor. A computer can for example comprise a system (network) of "sub-computers", wherein each sub-computer represents a computer in its own right. The term "computer" includes a cloud computer, for example a cloud server. The term computer includes a server resource. The term "cloud computer" includes a cloud computer system which for example comprises a system of at least one cloud computer and for example a plurality of operatively interconnected cloud computers such as a server farm. Such a cloud computer is preferably connected to a wide area network such as the world wide web (WWW) and located in a so-called cloud of computers which are all connected to the world wide web. Such an infrastructure is used for "cloud computing", which describes computation, software, data access and storage services which do not require the end user to know the physical location and / or configuration of the computer delivering a specific service. For example, the term "cloud" is used in this respect as a metaphor for the Internet (world wide web). For example, the cloud provides computing infrastructure as a service (laaS). The cloud computer can function as a virtual host for an operating system and / or data processing application which is used to execute the method of the invention. The cloud computer is for example an elastic compute cloud (EC2) as provided by Amazon Web Services™. A computer for example comprises interfaces in order to receive or output data and / or perform an analogue-to-digital conversion. The data are for example data which represent physical properties and / or which are generated from technical signals. The technical signals are for example generated by means of (technical) detection devices (such as for example devices for detecting marker devices) and / or (technical) analytical devices (such as for example devices for performing (medical) imaging methods), wherein the technical signals are for example electrical or optical signals. The technical signals for example represent the data received or outputted by the computer. The computer is preferably operatively coupled to a display device which allows information outputted by the computer to be displayed, for example to a user. One example of a display device is a virtual reality device or an augmented reality device (also referred to as virtual reality glasses or augmented reality glasses) which can be used as "goggles" for navigating. A specific example of such augmented reality glasses is Google Glass (a trademark of Google, Inc.). An augmented reality device or a virtual reality device can be used both to input information into the computer by user interaction and to display information outputted by the computer. Another example of a display device would be a standard computer monitor comprising for example a liquid crystal display operatively coupled to the computer for receiving display control data from the computer for generating signals used to display image information content on the display device. A specific embodiment of such a computer monitor is a digital lightbox. An example of such a digital lightbox is Buzz®, a product of Brainlab AG. The monitor may also be the monitor of a portable, for example handheld, device such as a smart phone or personal digital assistant or digital media player.

[0283] The invention also relates to a computer program comprising instructions which, when on the program is executed by a computer, cause the computer to carry out the method or methods, for example, the steps of the method or methods, described herein and / or to a computer-readable storage medium (for example, a non-transitory computer- readable storage medium) on which the program is stored and / or to a computer comprising said program storage medium and / or to a (physical, for example electrical, for example technically generated) signal wave, for example a digital signal wave, such as an electromagnetic carrier wave carrying information which represents the program, for example the aforementioned program, which for example comprises code means which are adapted to perform any or all of the method steps described herein. The signal wave is in one example a data carrier signal carrying the aforementioned computer program. The invention also relates to a computer comprising at least one processor and / or the aforementioned computer-readable storage medium and for example a memory, wherein the program is executed by the processor.

[0284] Within the framework of the invention, computer program elements can be embodied by hardware and / or software (this includes firmware, resident software, micro-code, etc.). Within the framework of the invention, computer program elements can take the form of a computer program product which can be embodied by a computer-usable, for example computer-readable data storage medium comprising computer-usable, for example computer-readable program instructions, "code" or a "computer program" embodied in said data storage medium for use on or in connection with the instructionexecuting system. Such a system can be a computer; a computer can be a data processing device comprising means for executing the computer program elements and / or the program in accordance with the invention, for example a data processing device comprising a digital processor (central processing unit or CPU) which executes the computer program elements, and optionally a volatile memory (for example a random access memory or RAM) for storing data used for and / or produced by executing the computer program elements. Within the framework of the present invention, a computer-usable, for example computer-readable data storage medium can be any data storage medium which can include, store, communicate, propagate or transport the program for use on or in connection with the instruction-executing system, apparatus or device. The computer-usable, for example computer-readable data storage medium can for example be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device or a medium of propagation such as for example the Internet. The computer-usable or computer-readable data storage medium could even for example be paper or another suitable medium onto which the program is printed, since the program could be electronically captured, for example by optically scanning the paper or other suitable medium, and then compiled, interpreted or otherwise processed in a suitable manner. The data storage medium is preferably a non-volatile data storage medium. The computer program product and any software and / or hardware described here form the various means for performing the functions of the invention in the example embodiments. The computer and / or data processing device can for example include a guidance information device which includes means for outputting guidance information. The guidance information can be outputted, for example to a user, visually by a visual indicating means (for example, a monitor and / or a lamp) and / or acoustically by an acoustic indicating means (for example, a loudspeaker and / or a digital speech output device) and / or tactilely by a tactile indicating means (for example, a vibrating element or a vibration element incorporated into an instrument). For the purpose of this document, a computer is a technical computer which for example comprises technical, for example tangible components, for example mechanical and / or electronic components. Any device mentioned as such in this document is a technical and for example tangible device.

[0285] Acquiring data

[0286] The expression "acquiring data" for example encompasses (within the framework of a computer implemented method) the scenario in which the data are determined by the computer implemented method or program. Determining data for example encompasses measuring physical quantities and transforming the measured values into data, for example digital data, and / or computing (and e.g. outputting) the data by means of a computer and for example within the framework of the method in accordance with the invention. A step of “determining” as described herein for example comprises or consists of issuing a command to perform the determination described herein. For example, the step comprises or consists of issuing a command to cause a computer, for example a remote computer, for example a remote server, for example in the cloud, to perform the determination. Alternatively or additionally, a step of “determination” as described herein for example comprises or consists of receiving the data resulting from the determination described herein, for example receiving the resulting data from the remote computer, for example from that remote computer which has been caused to perform the determination. The meaning of "acquiring data" also for example encompasses the scenario in which the data are received or retrieved by (e.g. input to) the computer implemented method or program, for example from another program, a previous method step or a data storage medium, for example for further processing by the computer implemented method or program. Generation of the data to be acquired may but need not be part of the method in accordance with the invention. The expression "acquiring data" can therefore also for example mean waiting to receive data and / or receiving the data. The received data can for example be inputted via an interface. The expression "acquiring data" can also mean that the computer implemented method or program performs steps in order to (actively) receive or retrieve the data from a data source, for instance a data storage medium (such as for example a ROM, RAM, database, hard drive, etc.), or via the interface (for instance, from another computer or a network). The data acquired by the disclosed method or device, respectively, may be acquired from a database located in a data storage device which is operably to a computer for data transfer between the database and the computer, for example from the database to the computer. The computer acquires the data for use as an input for steps of determining data. The determined data can be output again to the same or another database to be stored for later use. The database or database used for implementing the disclosed method can be located on network data storage device or a network server (for example, a cloud data storage device or a cloud server) or a local data storage device (such as a mass storage device operably connected to at least one computer executing the disclosed method). The data can be made "ready for use" by performing an additional step before the acquiring step. In accordance with this additional step, the data are generated in order to be acquired. The data are for example detected or captured (for example by an analytical device). Alternatively or additionally, the data are inputted in accordance with the additional step, for instance via interfaces. The data generated can for example be inputted (for instance into the computer). In accordance with the additional step (which precedes the acquiring step), the data can also be provided by performing the additional step of storing the data in a data storage medium (such as for example a ROM, RAM, CD and / or hard drive), such that they are ready for use within the framework of the method or program in accordance with the invention. The step of "acquiring data" can therefore also involve commanding a device to obtain and / or provide the data to be acquired. In particular, the acquiring step does not involve an invasive step which would represent a substantial physical interference with the body, requiring professional medical expertise to be carried out and entailing a substantial health risk even when carried out with the required professional care and expertise. In particular, the step of acquiring data, for example determining data, does not involve a surgical step and in particular does not involve a step of treating a human or animal body using surgery or therapy. In order to distinguish the different data used by the present method, the data are denoted (i.e. referred to) as "XY data" and the like and are defined in terms of the information which they describe, which is then preferably referred to as "XY information" and the like.

[0287] Image registration

[0288] Image registration is the process of transforming different sets of data into one coordinate system. The data can be multiple photographs and / or data from different sensors, different times or different viewpoints. It is used in computer vision, medical imaging and in compiling and analyzing images and data from satellites. Registration is necessary in order to be able to compare or integrate the data obtained from these different measurements.

[0289] Marker

[0290] It is the function of a marker to be detected by a marker detection device (for example, a camera or an ultrasound receiver or analytical devices such as CT or MRI devices) in such a way that its spatial position (i.e. its spatial location and / or alignment) can be ascertained. The detection device is for example part of a navigation system. The markers can be active markers. An active marker can for example emit electromagnetic radiation and / or waves which can be in the infrared, visible and / or ultraviolet spectral range. A marker can also however be passive, i.e. can for example reflect electromagnetic radiation in the infrared, visible and / or ultraviolet spectral range or can block x-ray radiation. To this end, the marker can be provided with a surface which has corresponding reflective properties or can be made of metal in order to block the x-ray radiation. It is also possible for a marker to reflect and / or emit electromagnetic radiation and / or waves in the radio frequency range or at ultrasound wavelengths. A marker preferably has a spherical and / or spheroid shape and can therefore be referred to as a marker sphere; markers can however also exhibit a cornered, for example cubic, shape.

[0291] Marker device

[0292] A marker device can for example be a reference star or a pointer or a single marker or a plurality of (individual) markers which are then preferably in a predetermined spatial relationship. A marker device comprises one, two, three or more markers, wherein two or more such markers are in a predetermined spatial relationship. This predetermined spatial relationship is for example known to a navigation system and is for example stored in a computer of the navigation system.

[0293] In another embodiment, a marker device comprises an optical pattern, for example on a two-dimensional surface. The optical pattern might comprise a plurality of geometric shapes like circles, rectangles and / or triangles. The optical pattern can be identified in an image captured by a camera, and the position of the marker device relative to the camera can be determined from the size of the pattern in the image, the orientation of the pattern in the image and the distortion of the pattern in the image. This allows determining the relative position in up to three rotational dimensions and up to three translational dimensions from a single two-dimensional image.

[0294] The position of a marker device can be ascertained, for example by a medical navigation system. If the marker device is attached to an object, such as a bone or a medical instrument, the position of the object can be determined from the position of the marker device and the relative position between the marker device and the object. Determining this relative position is also referred to as registering the marker device and the object. The marker device or the object can be tracked, which means that the position of the marker device or the object is ascertained twice or more over time.

[0295] Pointer

[0296] A pointer is a rod which comprises one or more - advantageously, two - markers fastened to it and which can be used to measure off individual co-ordinates, for example spatial co-ordinates (i.e. three-dimensional co-ordinates), on a part of the body, wherein a user guides the pointer (for example, a part of the pointer which has a defined and advantageously fixed position with respect to the at least one marker attached to the pointer) to the position corresponding to the co-ordinates, such that the position of the pointer can be determined by using a surgical navigation system to detect the marker on the pointer. The relative location between the markers of the pointer and the part of the pointer used to measure off co-ordinates (for example, the tip of the pointer) is for example known. The surgical navigation system then enables the location (of the three-dimensional co-ordinates) to be assigned to a predetermined body structure, wherein the assignment can be made automatically or by user intervention.

[0297] Reference star

[0298] A "reference star" refers to a device with a number of markers, advantageously three markers, attached to it, wherein the markers are (for example detachably) attached to the reference star such that they are stationary, thus providing a known (and advantageously fixed) position of the markers relative to each other. The position of the markers relative to each other can be individually different for each reference star used within the framework of a surgical navigation method, in order to enable a surgical navigation system to identify the corresponding reference star on the basis of the position of its markers relative to each other. It is therefore also then possible for the objects (for example, instruments and / or parts of a body) to which the reference star is attached to be identified and / or differentiated accordingly. In a surgical navigation method, the reference star serves to attach a plurality of markers to an object (for example, a bone or a medical instrument) in order to be able to detect the position of the object (i.e. its spatial location and / or alignment). Such a reference star for example features a way of being attached to the object (for example, a clamp and / or a thread) and / or a holding element which ensures a distance between the markers and the object (for example in order to assist the visibility of the markers to a marker detection device) and / or marker holders which are mechanically connected to the holding element and which the markers can be attached to.

[0299] Surgical tracking system

[0300] A tracking system, such as a surgical tracking system, is understood to mean a system which can comprise: at least one marker device; a transmitter which emits electromagnetic waves and / or radiation and / or ultrasound waves; a receiver which receives electromagnetic waves and / or radiation and / or ultrasound waves; and an electronic data processing device which is connected to the receiver and / or the transmitter, wherein the data processing device (for example, a computer) for example comprises a processor (CPU) and a working memory and advantageously an indicating device for issuing an indication signal (for example, a visual indicating device such as a monitor and / or an audio indicating device such as a loudspeaker and / or a tactile indicating device such as a vibrator) and a permanent data memory, wherein the data processing device processes navigation data forwarded to it by the receiver and can advantageously output guidance information to a user via the indicating device. The navigation data can be stored in the permanent data memory and for example compared with data stored in said memory beforehand.

[0301] Landmarks

[0302] A landmark is a defined element of an anatomical body part which is always identical or recurs with a high degree of similarity in the same anatomical body part of multiple patients. Typical landmarks are for example the epicondyles of a femoral bone or the tips of the transverse processes and / or dorsal process of a vertebra. The points (main points or auxiliary points) can represent such landmarks. A landmark which lies on (for example on the surface of) a characteristic anatomical structure of the body part can also represent said structure. The landmark can represent the anatomical structure as a whole or only a point or part of it. A landmark can also for example lie on the anatomical structure, which is for example a prominent structure. An example of such an anatomical structure is the posterior aspect of the iliac crest. Another example of a landmark is one defined by the rim of the acetabulum, for instance by the center of said rim. In another example, a landmark represents the bottom or deepest point of an acetabulum, which is derived from a multitude of detection points. Thus, one landmark can for example represent a multitude of detection points. As mentioned above, a landmark can represent an anatomical characteristic which is defined on the basis of a characteristic structure of the body part. Additionally, a landmark can also represent an anatomical characteristic defined by a relative movement of two body parts, such as the rotational center of the femur when moved relative to the acetabulum.

[0303] Atlas / Atlas segmentation

[0304] Atlas data describes (for example defines, more particularly represents and / or is) a general three-dimensional shape of the anatomical body part. The atlas data therefore represents an atlas, also referred to as generic model, of the anatomical body part. An atlas typically consists of a plurality of generic models of objects, wherein the generic models of the objects together form a complex structure. For example, the atlas constitutes a statistical model of a patient’s body (for example, a part of the body) which has been generated from anatomic information gathered from a plurality of human bodies, for example from medical image data containing images of such human bodies. In principle, the atlas data therefore represents the result of a statistical analysis of such medical image data for a plurality of human bodies. This result can be output as an image - the atlas data therefore contains or is comparable to medical image data. Such a comparison can be carried out for example by applying an image fusion algorithm which conducts an image fusion between the atlas data and the medical image data. The result of the comparison can be a measure of similarity between the atlas data and the medical image data. The atlas data comprises image information (for example, positional image information) which can be matched (for example by applying an elastic or rigid image fusion algorithm) for example to image information (for example, positional image information) contained in medical image data so as to for example compare the atlas data to the medical image data in order to determine the position of anatomical structures in the medical image data which correspond to anatomical structures defined by the atlas data.

[0305] The human bodies, the anatomy of which serves as an input for generating the atlas data, advantageously share a common feature such as at least one of gender, age, ethnicity, body measurements (e.g. size and / or mass) and pathologic state. The anatomic information describes for example the anatomy of the human bodies and is extracted for example from medical image information about the human bodies. The atlas of a femur, for example, can comprise the head, the neck, the body, the greater trochanter, the lesser trochanter and the lower extremity as objects which together make up the complete structure. The atlas of a brain, for example, can comprise the telencephalon, the cerebellum, the diencephalon, the pons, the mesencephalon and the medulla as the objects which together make up the complex structure. One application of such an atlas is in the segmentation of medical images, in which the atlas is matched to medical image data, and the image data are compared with the matched atlas in order to assign a point (a pixel or voxel) of the image data to an object of the matched atlas, thereby segmenting the image data into objects. For example, the atlas data includes information of the anatomical body part. This information is for example at least one of patient-specific, non-patient-specific, indication-specific or non-indication-specific. The atlas data therefore describes for example at least one of a patient-specific, non-patient-specific, indication-specific or non-indication-specific atlas. For example, the atlas data includes movement information indicating a degree of freedom of movement of the anatomical body part with respect to a given reference (e.g. another anatomical body part). For example, the atlas is a multimodal atlas which defines atlas information for a plurality of (i.e. at least two) imaging modalities and contains a mapping between the atlas information in different imaging modalities (for example, a mapping between all of the modalities) so that the atlas can be used for transforming medical image information from its image depiction in a first imaging modality into its image depiction in a second imaging modality which is different from the first imaging modality or to compare (for example, match or register) images of different imaging modality with one another.

[0306] Elastic fusion, image fusion / morphing, rigid

[0307] Image fusion can be elastic image fusion or rigid image fusion. In the case of rigid image fusion, the relative position between the pixels of a 2D image and / or voxels of a 3D image is fixed, while in the case of elastic image fusion, the relative positions are allowed to change.

[0308] In this application, the term "image morphing" is also used as an alternative to the term "elastic image fusion", but with the same meaning.

[0309] Elastic fusion transformations (for example, elastic image fusion transformations) are for example designed to enable a seamless transition from one dataset (for example a first dataset such as for example a first image) to another dataset (for example a second dataset such as for example a second image). The transformation is for example designed such that one of the first and second datasets (images) is deformed, for example in such a way that corresponding structures (for example, corresponding image elements) are arranged at the same position as in the other of the first and second images. The deformed (transformed) image which is transformed from one of the first and second images is for example as similar as possible to the other of the first and second images. Preferably, (numerical) optimization algorithms are applied in order to find the transformation which results in an optimum degree of similarity. The degree of similarity is preferably measured by way of a measure of similarity (also referred to in the following as a "similarity measure"). The parameters of the optimization algorithm are for example vectors of a deformation field. These vectors are determined by the optimization algorithm in such a way as to result in an optimum degree of similarity. Thus, the optimum degree of similarity represents a condition, for example a constraint, for the optimization algorithm. The bases of the vectors lie for example at voxel positions of one of the first and second images which is to be transformed, and the tips of the vectors lie at the corresponding voxel positions in the transformed image. A plurality of these vectors is preferably provided, for instance more than twenty or a hundred or a thousand or ten thousand, etc. Preferably, there are (other) constraints on the transformation (deformation), for example in order to avoid pathological deformations (for instance, all the voxels being shifted to the same position by the transformation). These constraints include for example the constraint that the transformation is regular, which for example means that a Jacobian determinant calculated from a matrix of the deformation field (for example, the vector field) is larger than zero, and also the constraint that the transformed (deformed) image is not self-intersecting and for example that the transformed (deformed) image does not comprise faults and / or ruptures. The constraints include for example the constraint that if a regular grid is transformed simultaneously with the image and in a corresponding manner, the grid is not allowed to interfold at any of its locations. The optimizing problem is for example solved iteratively, for example by means of an optimization algorithm which is for example a first-order optimization algorithm, such as a gradient descent algorithm. Other examples of optimization algorithms include optimization algorithms which do not use derivations, such as the downhill simplex algorithm, or algorithms which use higher-order derivatives such as Newton-like algorithms. The optimization algorithm preferably performs a local optimization. If there is a plurality of local optima, global algorithms such as simulated annealing or generic algorithms can be used. In the case of linear optimization problems, the simplex method can for instance be used.

[0310] In the steps of the optimization algorithms, the voxels are for example shifted by a magnitude in a direction such that the degree of similarity is increased. This magnitude is preferably less than a predefined limit, for instance less than one tenth or one hundredth or one thousandth of the diameter of the image, and for example about equal to or less than the distance between neighboring voxels. Large deformations can be implemented, for example due to a high number of (iteration) steps.

[0311] The determined elastic fusion transformation can for example be used to determine a degree of similarity (or similarity measure, see above) between the first and second datasets (first and second images). To this end, the deviation between the elastic fusion transformation and an identity transformation is determined. The degree of deviation can for instance be calculated by determining the difference between the determinant of the elastic fusion transformation and the identity transformation. The higher the deviation, the lower the similarity, hence the degree of deviation can be used to determine a measure of similarity.

[0312] A measure of similarity can for example be determined on the basis of a determined correlation between the first and second datasets.

[0313] Medical Workflow

[0314] A medical workflow comprises a plurality of workflow steps performed during a medical treatment and / or a medical diagnosis. The workflow steps are typically, but not necessarily performed in a predetermined order. Each workflow step for example means a particular task, which might be a single action or a set of actions. Examples of workflow steps are capturing a medical image, positioning a patient, attaching a marker, performing a resection, moving a joint, placing an implant and the like.

[0315] Convolutional Neural Network, CNN

[0316] A convolutional neural network, or CNN, is an example of a neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1 -D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D or 3-D grid of pixels. The name “convolutional neural network” indicates that the network employs the mathematical operation of convolution. Convolution is a linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers. There are several variants on the convolution function that are widely used in practice for neural networks. In general, the operation used in a convolutional neural network does not correspond precisely to the definition of convolution as used in other fields, such as engineering or pure mathematics.

[0317] BRIEF DESCRIPTION OF THE DRAWINGS

[0318] In the following, the invention is described with reference to the appended figures which give background explanations and represent specific embodiments of the invention. The scope of the invention is however not limited to the specific features disclosed in the context of the figures, wherein

[0319] Fig. 1 illustrates a system for implementing the method;

[0320] Figs. 2a and 2b illustrate an embodiment using all data processing blocks;

[0321] Fig. 3 shows adaptation of a generic model to the image data;

[0322] Fig. 4 shows segmentation and labelling;

[0323] Fig. 5 shows generation of standing 3D image data;

[0324] Fig. 6 shows alignment planning;

[0325] Fig. 7 shows examples of a cage and a rod;

[0326] Fig. 8 shows fine planning of screws and rod;

[0327] Fig. 9 shows decompression planning;

[0328] Fig. 10 shows screw insertion tracking;

[0329] Fig. 11 illustrates an embodiment using a minimal set of data processing blocks;

[0330] Fig. 12 illustrates the embodiment of figure 11 with an additional data fusion block;

[0331] Fig. 13 illustrates the embodiment of Figure 12 with an additional model generation block; and

[0332] Fig. 14 illustrates an embodiment using a typical set of data processing blocks. DESCRIPTION OF EMBODIMENTS

[0333] Figure 1 illustrates a medical system 1 comprising a computer 2 connected to in input unit 6, an output unit 7, a medical tracking system 8 and an electronic data storage device 9. The output unit 7 can for example be any display device, such as a monitor, a projector or VR or AR goggles. The input unit can for example be a keyboard, a mouse, a trackball, a touch sensitive surface or a combination thereof. The computer 2 comprises a central processing unit (CPU) 3, a memory 4 and an interface 5. By the interface 5, the computer 2 can be connected to external devices, such as any one or more of the input unit 6, the output unit 7, the medical tracking system 8 and the electronic data storage device 9.

[0334] The medical tracking system 8 can be any system capable of tracking e.g. one or more markers attached to an object. The medical tracking system 8 can in particular comprise a light source, a stereoscopic camera and a processing unit. However, the CPU 3 of the computer 2 can implement the functionality of the processing unit. The medical tracking system 8 is optional. It can for example be omitted if only preoperative blocks are executed.

[0335] The electronic data storage device, which is also referred to just as storage device 9, can not only be a single device, but also a device comprising multiple sub-units, wherein the sub-units can be spatially distributed. The sub-units may be servers or other storages in the internet or an intranet.

[0336] Figures 2a and 2b illustrate an embodiment of the present invention in which the method executes all data processing blocks described above. This embodiment is thus also referred to as full embodiment or maximum embodiment.

[0337] The method first executes the data acquisition block at S01 . This block receives patient identification data, for example from the input unit 6 or via the interface 5. The data acquisition block uses the patient identification data to retrieve patient data, for example from the storage device 9. If no patient identification data is input to the data acquisition block, the block receives user input data representing features of the patient, like one or more of first name, last name, date of birth, place of birth or address of the patient. The block uses those features to query a database for the patient identification data. This database can be stored in the memory 4 or the storage device 9, for example.

[0338] The patient data includes 2D image data in terms of a lateral x-ray image of the spine and a frontal (anterior-posterior, AP) x-ray image of the spine, 3D image data in terms of CT data of the spine, MRT data of the spine, previous plans for spine surgery, physiological data, like gender, age, height, weight and BMI (body mass index), diagnosis data and existing documentation data of previous spine surgeries.

[0339] The data acquisition block outputs the patient data as output data.

[0340] The method then executes the documentation block at S02. The output data of the data acquisition block is the input data to the documentation block. There are no existing documentation data since the documentation block is executed for the first time. The documentation block thus generates documentation data from the input data and outputs them as output data.

[0341] The method then executes the data analysis block at S03. The input data to the data analysis block are the patient data and the patient identification data. The data analysis block generates analysis data as output data, wherein the analysis data comprise segmentation data, classification data, vertebra identification data, OAR identification data, structure identification data, abnormality data, existing implant data, numerical data and seventy data.

[0342] The data analysis block analyzes the 2D and 3D image data by invoking the universal patient model to segment and classify the image data. The universal patient model matches a generic model to the image data to be segmented and classified. In the example shown in Figure 3, the generic model, or synthetic model, is matched to the 2D image data representing a lateral view of the spine. The method executes the display block at S04 to display some or all of the output data of the data analysis block. Figure 4 shows an exemplary screenshot in which the frontal (AP) x-ray image of the spine is segmented into vertebrae and each vertebra is labelled.

[0343] The method either returns from the display block to the data analysis block for further processing before the documentation block is executed again or directly proceeds to the documentation block from the display block.

[0344] The method then executes the documentation block at S05. The documentation block supplements the documentation data with the output data of the data analysis block.

[0345] The method then executes the data fusion block at S06. The input data to the data fusion block are the patient data and the analysis data. The data fusion block generates fusion data in terms of co-registered 3D Data, standing 3D image data from coregistration of 2D and 3D data, comparison data and enhanced data (bone density, bone strength, muscle attachment points, inflammation areas, osteophytes or impingements).

[0346] Figure 5 shows an example of matching the 3D image data of the spine taken at a lying position of the patient to 2D image data of the spine taken at a standing position of the patient. Both image data are segmented and labelled, such that they can be matched vertebra by vertebra.

[0347] The method executes the display block at S07 to display some or all of the output data of the data fusion block. The method either returns from the display block to the data fusion block for further processing before the documentation block is executed again or directly proceeds to the documentation block from the display block.

[0348] The method then executes the documentation block at S08. The documentation block supplements the documentation data with the fusion data output by the data fusion block. The method then executes the model generation block at S09. Input data to the model generation block are the image data, the segmentation data, the fusion data, the analysis data and further data, like data representing other parts of the musculoskeletal system. The model generation block outputs model data describing the biomechanical model of the spine.

[0349] The method executes the display block at S10 to display some or all of the model data output by the model generation block. The method either returns from the display block to the model generation block for further processing before the documentation block is executed again or directly proceeds to the documentation block from the display block.

[0350] The method then executes the documentation block at S11 . The documentation block supplements the documentation data with the model data output by the model generation block.

[0351] The method then executes the alignment planning block at S12. Input data to the alignment planning block are the patient data, the analysis data, the fusion data, the model data, user input data, the feasibility data and an existing plan. The alignment planning block generates alignment planning data and relative position data. The alignment planning data represents one or more implants and one or more osteotomies. An implant is for example a cage, a pedicle screw or a rod. The existing plan as input data is optional. In this case, the planning starts from scratch based on the other input data.

[0352] Depending on the type of alignment planning, the alignment planning block queries for user input to be considered when generating the alignment planning data.

[0353] Figures 6 to 8 show an example of the alignment planning. As shown in Figure 6, the display block displays 2D image (X-ray data) data and model data representing a lateral view of the spine. The user inputs a desired pose of the spine after surgery. In the present case, the user defines a target angle of -30.0 degrees between the vertebrae L1 and L5 and a target angle of 32.6 degrees between the vertebrae Th12 and Th1 as shown in Figure 6. The alignment planning block plans pedicle screws for the vertebrae L1 to L5 and two rods to be connected to the pedicle screws. Figure 7 shows possible implants in more detail. In the left part of this figure, a cage 10 is planned to replace the intervertebral disc between two adjacent vertebrae, for example if the intervertebral disc is worn out. The right part of Figure 7 shows how a rod 11 is fixed to pedicle screws 12 and 13 which are screwed into respective vertebrae. This rod 11 fixates the vertebrae.

[0354] Figure 8 shows fine planning of the pedicle screws 12a to 12e. Planning a pedicle screw means selecting a pedicle screw and setting the position of the pedicle screw relative to the corresponding vertebra either manually, automatically or both interdependently. As can be seen from the list of pedicle screws in the screenshot of Figure 8, two pedicle screws are planned for each of the vertebrae L1 to L5. Fine planning is made using 3D image data of the vertebrae to place the pedicle screws. Figure 8 further shows the rod 11 to be attached to the pedicle screws 12a to 12e.

[0355] The CT inline windows show the position of a pedicle screw relative to the corresponding vertebra in 2D slices of the 3D image data. In the present exemplary view, pedicle screw 12c is shown in the planned position relative to the corresponding vertebra. The slices are typically orthogonal to each other.

[0356] The method then executes the display block at S13 to display some or all of the alignment planning data and the relative position data output by the alignment planning block. The method either returns from the display block to the alignment planning block for further processing before the documentation block is executed again or directly proceeds to the documentation block from the display block.

[0357] The method then executes the documentation block at S14. The documentation block supplements the documentation data with the alignment planning data and the relative position data output by the alignment planning block.

[0358] The method then executes the decompression planning block at S15. Input data to the decompression planning block are the patient data, the analysis data, the fusion data and the alignment planning data. The decompression planning block generates decompression planning data. The method then executes the display block at S16 to display the decompression planning data output by the decompression planning block. The display block for example shows a visualization of the decompression planning data as shown in Figure 9. This visualization comprises an image of a vertebrae, wherein the part of the vertebra to be removed according to the decompression planning data is indicated as a hatched area 13.

[0359] The method either returns from the display block to the decompression planning block for further processing before the documentation block is executed again or directly proceeds to the documentation block from the display block.

[0360] The method then executes the documentation block at S17. The documentation block supplements the documentation data with the decompression planning data output by the decompression planning block.

[0361] The method then executes the data analysis block again at S18. This time, the alignment planning data and the decompression planning data is also input to the data analysis block, which now generates and outputs numerical data calculated from the input data.

[0362] The method then executes the display block at S19 to display the numerical data output by the data analysis block. The method either returns from the display block to the data analysis block for further processing before the documentation block is executed again or directly proceeds to the documentation block from the display block.

[0363] The method then executes the documentation block at S20. The documentation block supplements the documentation data with the numerical data and any other data output by the data analysis block this time.

[0364] The method then executes the feasibility test block at S21. Input data are the patient data, the alignment planning data, the decompression planning data and the analysis data. The feasibility test block determines whether or not the plan as defined by the alignment planning data and the decompression planning data is feasible and outputs the result as feasibility data.

[0365] The method then optionally executes the display block at S22 to display the feasibility data output by the feasibility test block. The method proceeds to execute the documentation block at S23 from the display block or directly from the feasibility test block. The documentation block supplements the documentation data with the feasibility data output by the feasibility test block.

[0366] If the plan is not feasible, the method returns and executes the alignment planning block and the decompression planning block again from S12 to generate a new plan in terms of new alignment planning data and decompression planning data. The display block and the documentation block are also executed again, wherein the documentation block adds the new alignment planning data and the new decompression planning data to the documentation data.

[0367] The method then executes the feasibility test block again to determine whether or not the new plan is feasible. The feasibility test block generates and outputs feasibility data for the new plan. The method executes the display block to display the feasibility data and the documentation block to supplement the documentation data with the feasibility data.

[0368] If the feasibility data indicates that the plan is feasible, the method executes the model generation block again at S24. This time, the alignment planning data and the decompression planning data are also input to the model generation block. The model generation block updates the model data based on the alignment planning data and the decompression planning data and outputs the model data.

[0369] The method then executes the display block at S25 to display the model data output by the model generation block. The method either returns from the display block to the model generation block for further processing before the documentation block is executed again or directly proceeds to the documentation block from the display block. The method then executes the documentation block at S26. The documentation block supplements the documentation data with the model data output by the model generation block.

[0370] The method then executes the data fusion block again at S27. This time, the input data to the data fusion block also comprises the alignment planning data and the decompression planning data.

[0371] The data fusion block now uses the latest model data, the alignment planning data, the decompression planning data and the image data to generate comparison data, predicted image data and fused image data as output data. The data fusion block can now generate a visualization of the predicted outcome of the surgery, for example.

[0372] The predicted image data represents a predicted view of the spine after surgery if the plan as represented by the alignment planning data and the decompression planning data is implemented.

[0373] The comparison data represents image data including the predicted image data and image data comprised in the patient data. It further comprises numerical data which represent changes in an intervertebral angle between neighboring vertebrae between the state at which the image data in the patient data was captured and the postoperative state predicted according to the alignment planning data and the decompression planning data.

[0374] The fused image data comprises a representation of comparison data, like numerical values. The fused image data also comprises a representation of one or more of the other patient data, like all or parts of the physiological data, the diagnosis data and previous plans, of the planning data or of the analysis data.

[0375] The method then executes optionally the display block at S28 to display the comparison data, the predicted image data and the fused image data output by the data fusion block. The method then executes the documentation block at S29 which supplements the documentation data with the comparison data, the predicted image data and the fused image data output by the data fusion block. The method then executes the surgery companion block at S30. The input data to this block are the planning data, the patient data and the analysis data. The block generates intra-operative data as output data. The intra-operative data include intraoperative patient data, implant positioning data, osteotomy positioning data, intraoperative tissue removal data, structure position data, intra-operative image data and instrument position data.

[0376] The surgery companion block adapts the plan to the current situation of the patient, for example by registering the image data on which the plan is based and relative to which the plan is defined to intra-operative patient data. This for example locates implants and / or osteotomies in the reference system of the medical tracking system 8.

[0377] The surgery companion block does not only generate the above output data, but also aids in navigating objects like pedicle screws, cages and rods. For this purpose, the object to be navigated or a tool with which the object is handled is tracked using the medical tracking system 8.

[0378] Figure 10 shows an example in which a pedicle screw 12a is tracked during insertion. The pedicle screw 12a is applied using a tool 14 carrying markers 15. The markers 15 are tracked using the medical tracking system 9. This also tracks the pedicle screw 12a since it is in a known relative position to the tool 14. The final position of the pedicle screw 12a can be output in the implant positioning data.

[0379] The osteotomy positioning data and the intra-operative tissue removal data can be generated by tracking a tool for performing the osteotomy or for removing tissue, like a drill, a saw or a milling tool.

[0380] The surgery companion block generates progress data indicating the progress of the osteotomy or the tissue removal. If the osteotomy or tissue removal is complete, the surgery companion block can cause the tool used for this action to stop.

[0381] The surgery companion block may also stop the tool if it leaves the planned area of the osteotomy or the tissue removal. The surgery companion block may determine, based on the intra-operative data, whether or not the plan is still feasible and / or can be implemented. If this is not the case, the method may return to the documentation block at S02 to create a new plan based on the intra-operative data.

[0382] The surgery companion block may determine, based on the intra-operative data, whether or not an implant is set and / or an osteotomy is made as planned. If the surgery companion block determines a deviation from the plan, it may output his such that the implant and / or osteotomy can be corrected. In addition or as an alternative, the surgery companion block may cause the rest of the plan to be adapted to compensate for the deviation, for example by returning to the documentation block at S02.

[0383] The method then executes the display block at S31 to display data output by the surgery companion block. The display block for example outputs an image showing the position of an implant relative to the patient by fusing a visual representation of the implant with image data, like image data in the patient data.

[0384] The method either returns from the display block to the surgery companion block for further processing, like further tracking of an implant, before the documentation block is executed again or directly proceeds to the documentation block from the display block.

[0385] The display block can be executed in parallel, or virtually parallel by quickly switching, to the surgery companion block, for example to display the data generated by the surgery companion block in real time. In this manner, the position of an implant or a medical instrument can be displayed in real time and / or the progress of the osteotomy or the tissue removal can be displayed in real time. The display block can for example display the planned osteotomy or tissue removal relative to image data of the area of the spine to be treated together with the area in which the osteotomy is already performed or the tissue is already removed.

[0386] The display block may display in indication if the progress data indicates that the osteotomy or the tissue removal is complete. The method then executes the documentation block at S32. The documentation block supplements the documentation data with the intra-operative data output by the surgery companion block. The method may return to the surgery companion block, for example if the surgery is not yet completed and the documentation block was only executed to add intermediate intra-operative data to the documentation data.

[0387] The method then executes the data fusion block again at S33. The input data to the data fusion block now comprises the intra-operative data output by the surgery companion block. This block matches 3D image data of the patient data onto intraoperative image data. This process generates matched image data, which is output by the data fusion block.

[0388] The method then optionally executes the display block at S34 to display the matched image data output by the data fusion block. The method then executes the documentation block again at S35. The documentation block supplements the documentation data with the matched image data output by the data fusion block.

[0389] The method then executes the post-op control block at S36. The input data to the postop control block are the planning data, the patient data, the analysis data, the model data and the intra-operative data.

[0390] The post-op control block acquires one or both of post-operative image data and postoperative structure position data which both represent information about the status of the patient after the surgery. The post-operative structure position data can be the structure position data of the intra-operative data.

[0391] The post-op control block compares the actual state of the patient, in particular of the spine, with the pre-operative state and / or the planned state. The post-op control block outputs post-operative data in terms of deviation data, post-operative implant data, post-operative osteotomy data, post-operative tissue removal data, post-operative image data, post-operative alignment data and post-operative structure position data. The method then executes the data analysis block again at S37. The input data to the data analysis block now also comprises the post-operative data output by the post-op control block. The data analysis block detects existing implants in the post-operative image data in the post-operative data. In this case, the actual position of an implant after surgery can is detected and output as existing implant data. The data analysis block further derives numerical data from the post-operative data and outputs them.

[0392] The method then executes the data fusion block again at S38. The input data to the data fusion block now also comprises the post-operative data output by the post-op control block. The data fusion block generates comparison data from the postoperative data as well as the existing implant data and the numerical data output by the data analysis block based on the post-operative data. The data fusion block may use any other data for generating the comparison data.

[0393] The data fusion block further generates fused image data comprising the postoperative image data and one or more of pre-operative image data and predicted postoperative image data.

[0394] The method then optionally executes the display block at S39 to display the postoperative data output by the post-op control block, the existing implant data and the numerical data output by the data analysis block and the comparison data and the fused image data output by the data fusion block.

[0395] The method then executes the documentation block again at S40. The documentation block supplements the documentation data with the post-operative data output by the post-op control block, the existing implant data and the numerical data output by the data analysis block and the comparison data and the fused image data output by the data fusion block.

[0396] The method then executes the registry supplementing block at S41 . The input data to the registry supplementing block are the patient data, the planning data, the deviation data, the post-operative image data and the documentation data. The registry supplementing block acquires reported outcome data representing statements of the patient and / or medical personnel on the outcome of the surgery, and for example represents an assessment of pain and / or mobility of the patient at one or more points in time after the surgery.

[0397] The registry supplementing block stores the input data and the reported outcome data in the registry, which comprises information about one or more surgeries of one or more patients.

[0398] The post-op control block, the data analysis block, the data fusion block, the display block, the documentation block and the registry supplementing block can be executed several times to track the outcome of the surgery over time.

[0399] If applicable, the display block may receive user input data and adapt the display data based on the user input data. The user input data may for example instruct a change of the viewing direction onto 3D image data, of a zoom factor or of a panning of the image data. The display block itself may adapt the image data or provide the user input data to the block which was executed immediately before the display block. The user input data may comprise other input that is provided to the block which was executed immediately before the display block, for example input indicating that the plan shall be modified.

[0400] Figure 11 illustrates an embodiment of the present invention in which the method executes the smallest set of data processing blocks described above. This embodiment is thus also referred to as smallest embodiment or minimum embodiment.

[0401] The method of this embodiment first executes the data acquisition block. The input data to the data acquisition block are patient identification data. The data acquisition block acquires 2D image data in terms of a lateral x-ray image of the spine of the patient as patient data. The data acquisition block outputs the patient data in terms of the 2D image data as output data.

[0402] The method then executes the data analysis block. The input data to the data analysis block are the 2D image data. The data analysis block performs segmentation of the 2D image data by invoking the universal patient model to generate analysis data in terms of segmentation data. The data analysis block in particular outlines each vertebra in the segmentation data.

[0403] The method then executes the alignment planning block. The input data to the alignment planning block are the 2D image data.

[0404] In one example, the alignment planning block receives user input defining a desired alignment of two or more of the vertebrae. The alignment planning block then plans pedicle screws, rods and optionally cages to achieve the desired alignment and outputs the result as alignment planning data.

[0405] Instead of or in addition to executing the alignment planning block, the method may execute the decompression planning block.

[0406] The method then executes the display block. The input data to the display block are the 2D image data and the alignment planning data. The display block displays the 2D image data together with the pedicle screws, rods and cages (if applicable).

[0407] Any embodiment between the minimum embodiment and the maximum embodiment is encompassed by this document. This may mean that not all blocks and / or not all data of the maximum embodiment are used.

[0408] Figure 12 illustrates another embodiment of the present invention in which the embodiment of Figure 11 is supplemented by execution of the data fusion block.

[0409] The method of this embodiment first executes the data acquisition block. The input data to the data acquisition block are patient identification data. The data acquisition block acquires 2D image data in terms of a lateral x-ray image of the spine of the patient and 3D image data representing a three-dimensional image of one or more vertebrae. The data acquisition block outputs the patient data in terms of the 2D image data and the 3D image data as output data.

[0410] The method then executes the data analysis block. The input data to the data analysis block are the 2D image data and the 3D image data. The data analysis block performs segmentation and classification of the 2D and 3D image data by invoking the universal patient model to generate analysis data in terms of segmentation data and classification data. The data analysis block in particular outlines each vertebra in the segmentation data and labels the segmented vertebrae.

[0411] The method then executes the data fusion block. The input data to the data fusion block are the 2D image data, the 3D image data and the analysis data. The data fusion block uses the analysis data to align the 3D image data with the 2D image data to generate standing 3D image data. The standing 3D image data represents the spine of the patient in a standing pose of the patient and is output as fusion data.

[0412] The method then executes the alignment planning block. The input data to the alignment planning block are the patient data and the fusion data.

[0413] In one example, the alignment planning block receives user input defining a desired alignment of two or more of the vertebrae. The alignment planning block then plans implants in terms of pedicle screws, rods and optionally cages to achieve the desired alignment and outputs the result as alignment planning data. The positions of the pedicle screws and optional rods are planned relative to the corresponding vertebrae and the positions of the rods are planned relative to the pedicle screws.

[0414] Instead of or in addition to executing the alignment planning block, the method may execute the decompression planning block.

[0415] The method then executes the display block. The input data to the display block are the patient data, the fusion data and the alignment planning data. The display block displays at least the standing 3D image data together with the pedicle screws, rods and cages (if applicable).

[0416] Figure 13 illustrates another embodiment of the present invention in which the embodiment of Figure 12 is supplemented by execution of the model generation block.

[0417] The model generation block is executed between the data fusion block and the alignment planning block. The input data to the model generation block are the patient data, the analysis data and the fusion data. The model generation block generates a biomechanical model of the patient. In one example, the model generation block adapts a generic model to the current patient using the analysis data and the fusion data. The model generation block outputs the biomechanical model as model data.

[0418] The method then continues with the alignment planning block (or the decompression planning block), which in the present embodiment also receives the model data as input data and utilizes the model data for planning.

[0419] Figure 14 illustrates a typical embodiment using some, but not all of the blocks.

[0420] In this embodiment, the method first executes the data acquisition block. The input data to the data acquisition block are patient identification data. The data acquisition block acquires 2D image data in terms of a lateral x-ray image of the spine of the standing patient and 3D image data in terms of a CT image of the spine of the lying patient as patient data. The data acquisition block outputs the patient data as output data.

[0421] The method then executes the data analysis block. The input data to the data analysis block are the patient data. The data analysis block performs segmentation and classification of the 2D image data and the 3D image data by invoking the universal patient model to generate analysis data in terms of segmentation data and classification data. The segmentation data defines the outlines of the vertebrae in the 2D image data and the 3D image data, while the classification data assigns labels, in particular the names of the vertebrae, to the segments represented by the segmentation data. The data analysis block outputs the segmentation data and the classification data es analysis data.

[0422] The method then executes the data fusion block. The input data to the data fusion block are the patient data and the analysis data.

[0423] The data fusion block matches the 3D image data to the 2D image data using the segmentation data and the classification data to generate standing 3D image data. The standing 3D image data comprises the vertebrae of the 3D image data, but in the arrangement as defined by the 2D image data. The data fusion block outputs the standing 3D image data as fusion data.

[0424] The method then executes the model generation block. The input data to the model generation block are the patient data, the analysis data and the fusion data. The model generation block generates a biomechanical model of the patient. In one example, the model generation block adapts a generic model to the current patient using the segmentation data. The model generation block outputs the biomechanical model as model data.

[0425] The method then executes the alignment planning block. The input data to the alignment planning block are the patient data, the analysis data and the model data.

[0426] In one example, the alignment planning block receives user input defining a desired alignment of two or more of the vertebrae. The alignment planning block then plans pedicle screws, rods and optionally cages and / or osteotomies to achieve the desired alignment and outputs the result as alignment planning data.

[0427] Planning pedicle screws and rods does not only include selection of suitable pedicle screw and rods, but also positioning of the pedicle screws in the vertebrae. This positioning is planned using the 3D image data to find the positions of the pedicle screws relative to the vertebrae.

[0428] Instead of or in addition to executing the alignment planning block, the method may execute the decompression planning block.

[0429] In the following, the term “planning data” encompasses the alignment planning data if the method executes the alignment planning block and the decompression planning data if the method executes the decompression planning block.

[0430] The method then executes the data fusion block again. The input data to the data fusion block this time comprise the planning data. The data fusion block generates predicted image data from the planning data and outputs the predicted image data as fusion data. The predicted image data include a two-dimensional lateral view on the spine and a view of the vertebrae affected by the planned implants or tissue removals. A vertebra is affected by an implant if the implant is a cage and the vertebra contacts the cage or if the implant is a pedicle screw and the pedicle screw is planned to be attached to the vertebra.

[0431] The data fusion block can now generate a visualization of the predicted outcome of the surgery, for example.

[0432] The method then executes the display block. The input data to the display block are the patient data, the fusion data and the planning data. The display block displays the planning data and the predicted image data.

[0433] The method then executes the surgery companion block. The input data to the surgery companion block are the planning data and the patient data.

[0434] The surgery companion block tracks implants, like pedicle screws, cages and rods, in real time using the medical tracking system 8 and interacts with the display block to display the tracked implant relative to the patient. The combination of the surgery companion block and the display block in particular displays the implant together with one or more vertebrae, for example including at least the vertebrae to which the tracked implant is to be attached.

[0435] The surgery companion block outputs intra-operative data. The intra-operative data comprises implant positioning data if the alignment planning data defines at least one implant and osteotomy positioning data if the alignment planning data defines at least one osteotomy. The intra-operative data comprises intra-operative tissue removal data if tissue was removed according to decompression planning data. The intra-operative data thus defines how implants are implanted, osteotomies are made and tissue is removed at the end of the surgery.

[0436] The method then executes the documentation block. The input data to the documentation block are the patient data, the analysis data, the fusion data, the planning data and the intra-operative data. The documentation block generates documentation data for documentation of the surgery. The documentation data is then stored.

Claims

1. Page 89 of 91Brainlab AGAttorney’s File: P101315WO XVCLAIMS1 . A computer-implemented method of data processing along at least a part of a medical spine surgery workflow, comprising execution of a data acquisition block, a data analysis block, a planning block and a display block, wherein- the data acquisition block acquires patient data,- the data analysis block analyses the patient data to make the data processable by subsequent blocks,- the planning block generates planning data of the surgery, and- the display block displays data.

2. The method according to the preceding claim, wherein the planning block is one or more of an alignment planning block and a decompression planning block, wherein the alignment planning block generates a plan which relates to the alignment of the patient’s spine and the decompression planning block generates a plan which relieves compressed nervous tissue.

3. The method according to any one of the preceding claims, further comprising execution of a data fusion block that generates additional data about the patient by combining two or more parts of the patient data.

4. The method of claim 3, further comprising execution of a model generation block that generates a biomechanical model of the patient, in particular of the spine system of the patient.

5. The method according to any one of the preceding claims, further comprising execution of a feasibility test block checks whether or not the plan according to the planning data is feasible.

6. The method according to any one of the preceding claims, further comprising executing a surgery companion block that performs at least one of adapting thePage 90 of 91 planning data to the current situation, providing guidance information and tracking implants.

7. The method according to any one of the preceding claims, further comprising execution of a post-op control block that enables post-operative control of the surgery.

8. The method according to any one of the preceding claims, further comprising execution of a documentation block that creates a documentation of a planning process.

9. The method according to any one of the preceding claims, further comprising execution of a registry supplementing block that stores information about the surgery in a database.

10. A computer program comprising instructions which, when the program is executed by a computer (2), cause the computer (2) to carry out the method according to any one of the preceding claims.

11. A computer-readable storage medium on which the program according to claim 9 is stored.

12. A computer (2) comprising at least one processor (3), wherein the program according to claim 9 is executed by the processor (3) or the computer (2) comprises the storage medium according to claim 10.

13. A data carrier signal carrying the program of claim 9 or a data stream comprising the program of claim 9.

14. A medical system (1 ) comprising an electronic data storage device (9) and the computer (2) of claim 11 .

Images