Automated pre-check to assess medical imaging data for suitability for surgical planning purposes
By using an automated image inspection suite, machine learning models and shape models are employed to evaluate medical imaging data, addressing the data deficiencies in orthopedic joint replacement surgery planning and improving the accuracy and reliability of the planning.
Patent Information
- Application Number
- CN202580012448.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-01-30
- Filing Date
- 2025-01-30
- Publication Date
- 2026-11-10
AI Technical Summary
In the current orthopedic joint replacement surgery planning process, medical imaging data may contain defects or errors, leading to inaccurate segmentation and registration, which affects the accuracy and integrity of planning and surgery.
An automated image examination suite is provided that automatically assesses the quality and accuracy of medical imaging data through machine learning and shape models, including classification of anatomical structure types, lateralization, shape model fitting, anatomical landmark comparison, implant selection, and configuration setting checks, to ensure that the data is suitable for surgical planning.
It improves the accuracy of medical imaging data assessment, reduces human error, ensures the accuracy and completeness of surgical planning, and enhances data reliability.
Smart Images

Figure CN123076908A_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims priority and all benefits to U.S. Provisional Application No. 63 / 626,742, filed January 30, 2024, the entire contents of which are incorporated herein by reference. Background Technology
[0003] Surgical planning for orthopedic joint replacement surgery typically begins with acquiring medical imaging data of the patient's anatomical joints. A treatment planning request is submitted to the surgical planning team. The team reviews patient information and manually evaluates the imaging data. A segmentation specialist manually performs the segmentation process to outline the bone across multiple slices of the imaging data. The output of the segmentation process is a virtual model of the bone. The surgical planning team or surgeon uses this virtual model as a reference to virtually plan the type, size, and location of the joint replacement implant, as well as the proposed surgical workflow. The virtual surgical plan, incorporating the virtual model, is then registered with the physical bone during surgery to provide intraoperative guidance to the surgeon.
[0004] Typical review and planning processes have several drawbacks. One difficulty is that the imaging data input to the segmentation team may be defective or exhibit errors. Therefore, such imaging data is unsuitable for segmentation, bone model creation, or intraoperative purposes, such as visualizing the bone model on a monitor or registering the bone model with the actual bone. For example, if the image of a bone was not acquired with an appropriate scanner configuration, fails to contain the required detail of the bone, has a portion of the target bone cropped, and / or is not large enough for navigation visualization purposes, the imaging data may be unsuitable for planning or intraoperative surgical purposes. Another potential drawback may be that the imaging data fails to show what is expected. For example, patient information may indicate a request for surgical planning for the left knee, but the medical image may be of the right knee. Other errors may include imaging data showing an existing implant or metal (indicating revision surgery), while patient information instead indicates a request for initial surgery. Imaging data may also exhibit noise or blurriness, which will render the image quality insufficient for segmentation.
[0005] If such image errors are ignored, the problem will propagate downstream, potentially impacting the accuracy and completeness of aspects described in the planning process and surgical workflow. The fact that hundreds of thousands of patients require surgical planning each year exacerbates these issues. Summary of the Invention
[0006] The present invention provides a simplified description of a series of concepts further described in the detailed embodiments below. This invention is not intended to limit the scope of the claimed subject matter, nor is it intended to identify key or essential features of the claimed subject matter.
[0007] According to a first aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for that target anatomical structure by being configured to: perform automated inspections to determine whether: the medical imaging data was scanned according to acceptable configuration settings; whether the target anatomical structure in the medical imaging data is acceptablely captured within the boundaries of the medical imaging data; whether the medical imaging data exhibits a motion bar visible above a visibility threshold level; whether the medical imaging data acceptably shows the target anatomical structure of the expected type and the expected surgical side of the target anatomical structure; and, in response to determining that any one or more of these automated inspections produce unacceptable results, automatically reject the medical imaging data as unacceptable for facilitating surgical planning.
[0008] According to a second aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically evaluate medical imaging data of a target anatomical structure by being configured to: receive the medical imaging data as input, wherein at input, the type and laterality of the target anatomical structure in the medical imaging data are unclassified; automatically classify the type of the target anatomical structure in the medical imaging data using a first machine learning model; select a second machine learning model using the classified type of the target anatomical structure, the second machine learning model being specifically trained to classify the laterality of the classified type of the target anatomical structure; automatically classify the laterality of the target anatomical structure in the medical imaging data using the second machine learning model; and generate a computer-generated output to identify the classified type and classified laterality of the target anatomical structure in the medical imaging data.
[0009] According to a third aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure by being configured to: automatically identify a shape model and fit the shape model to the target anatomical structure in the medical imaging data; automatically compare features of the shape model with the boundaries of the medical imaging data; and automatically reject the medical imaging data as unacceptable for facilitating surgical planning in response to determining that the features of the shape model exceed the boundaries of the medical imaging data.
[0010] According to a fourth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for that target anatomical structure by being configured to: automatically compare the medical imaging data with a statistical cohort of medical imaging data including other anatomical structures to identify anatomical landmarks required to be visible in the medical imaging data for comparison with the target anatomical structure; automatically assess the medical imaging data to determine whether the anatomical landmark is visible in the medical imaging data; and automatically reject the medical imaging data as unacceptable for facilitating surgical planning in response to determining that the anatomical landmark is not visible in the medical imaging data.
[0011] According to a fifth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure by being configured to: automatically acquire measurements of the target anatomical structure in the medical imaging data; automatically select an implant from a plurality of implant options based on the measurements of the target anatomical structure in the medical imaging data, and automatically acquire implant measurements of the selected implant; automatically determine, based on the implant measurements, a required feature of the target anatomical structure that must be fully captured in the medical imaging data to acceptably facilitate surgical planning of the selected implant relative to the target anatomical structure; and automatically reject the medical imaging data as unacceptable for facilitating surgical planning in response to the determination that the required feature of the target anatomical structure is not fully captured in the medical imaging data.
[0012] According to a sixth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure by being configured to: automatically obtain measurements of the target anatomical structure in the medical imaging data; automatically select an implant from a plurality of implant options based on the measurements of the target anatomical structure in the medical imaging data, and automatically obtain implant measurements of the selected implant; automatically compare the implant measurements with the boundaries of the medical imaging data; and automatically reject the medical imaging data as unacceptable for facilitating surgical planning in response to determining that the implant measurements exceed the boundaries of the medical imaging data or fail to be spaced apart from the boundaries of the medical imaging data by a threshold distance.
[0013] According to a seventh aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning of the target anatomical structure by being configured to: obtain preoperative patient data associated with the medical imaging data, the preoperative patient data including information indicating expected parameters, including: (1) the expected type of the target anatomical structure and (2) the expected surgical side of the target anatomical structure; automatically apply the medical imaging data to a machine learning model to analyze the medical imaging data to output predicted parameters, including: (1') the predicted type of the target anatomical structure and (2') the predicted surgical side of the target anatomical structure; automatically compare each expected parameter with its corresponding predicted parameter; and automatically approve the medical imaging data as acceptable for facilitating surgical planning in response to determining that each expected parameter acceptably matches its corresponding predicted parameter.
[0014] According to the eighth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically classify a CT volume of an anatomical joint by being configured to: automatically generate a plurality of digitally reconstructed radiographs from the CT volume, the plurality of digitally reconstructed radiographs being two-dimensional and capturing the structure of the anatomical joint; and automatically apply each of these digitally reconstructed radiographs to a convolutional neural network configured to analyze each of these digitally reconstructed radiographs to output (1') a predicted type of anatomical joint, (2') a predicted surgical side of the anatomical joint, and (3') a predicted presence or absence of an existing implant for the anatomical joint.
[0015] According to a ninth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for that target anatomical structure by being configured to: automatically acquire one or more configuration settings that define how the medical imaging data is scanned by an imaging device; automatically compare the one or more configuration settings with one or more acceptable configuration settings; and automatically reject the medical imaging data as unacceptable for facilitating surgical planning in response to determining that the one or more configuration settings do not correspond to the one or more acceptable configuration settings.
[0016] According to a tenth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for that target anatomical structure by being configured to: automatically identify a motion lever in a volume of the medical imaging data; automatically assess the volume of the medical imaging data to determine whether the volume acceptably displays the motion lever; and automatically reject the medical imaging data as unacceptable for facilitating surgical planning in response to determining that the volume fails to acceptably display the motion lever.
[0017] According to the eleventh aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically: identify a shape model and fit the shape model to the target anatomical structure in the medical imaging data; compare the shape model with the boundary of the medical imaging data; determine that a portion of the shape model extends beyond the boundary of the medical imaging data; and modify the medical imaging data to capture the portion of the shape model that extends beyond the boundary of the medical imaging data.
[0018] According to a twelfth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implanted method is provided to automatically: obtain measurements of the target anatomical structure in the medical imaging data; select an implant from a plurality of implant options based on the measurements of the target anatomical structure in the medical imaging data, and obtain implant measurements of the selected implant; determine, based on the implant measurements, a required feature of the target anatomical structure that must be fully captured in the medical imaging data to acceptably facilitate surgical planning of the selected implant relative to the target anatomical structure; determine that the required feature of the target anatomical structure is not fully captured in the medical imaging data; identify a shape model and fit the shape model to the target anatomical structure in the medical imaging data, wherein a portion of the shape model extends beyond the boundaries of the medical imaging data; and modify the medical imaging data to acquire the portion of the shape model that extends beyond the boundaries of the medical imaging data.
[0019] According to a thirteenth aspect, an automated image inspection suite, a non-transitory computer-readable computer program product, or a computer-implemented method is provided to automatically: identify an implant shape model and fit the implant shape model to the target anatomical structure in the medical imaging data; compare the implant shape model with the boundary of the medical imaging data; determine that a portion of the implant shape model extends beyond the boundary of the medical imaging data; and modify the medical imaging data to capture the portion of the implant shape model that extends beyond the boundary of the medical imaging data.
[0020] Any of the above aspects can be utilized individually or in combination.
[0021] Some aspects of the above-mentioned aspects can be combined with other aspects of other aspects and utilized.
[0022] Any of the above aspects can be utilized individually or in combination with any one or more of the following implementations: The feature of the shape model may include the boundary of the shape model. The boundary of the shape model may be compared to the boundary of the medical imaging data. In response to determining that the boundary of the shape model exceeds the boundary of the medical imaging data, the medical imaging data may be rejected because it is unacceptable for facilitating surgical planning; the feature of the shape model may include one or more points located on the boundary of the shape model. The one or more points of the shape model may be compared to the boundary of the medical imaging data. In response to determining that the one or more points of the shape model exceed the boundary of the medical imaging data, the medical imaging data may be rejected as unacceptable for facilitating surgical planning. The one or more points on the shape model may correspond to predetermined anatomical landmarks of the target anatomy. In response to determining that the feature of the shape model is within the boundary of the medical imaging data, the medical imaging data may be automatically approved as acceptable for facilitating surgical planning. The shape model may be derived from a statistical population of images of other anatomical structures comparable to the target anatomy. The shape model may include any one or more of the following: statistical shape model, active shape model, active appearance model, and active contour model. Output can be automatically generated in response to the rejection of medical imaging data as unacceptable. This output may be a notification or alert generated to a graphical user interface (GUI) informing the user that the medical imaging data has been rejected. The output may display the medical imaging data and the shape model on the GUI to show that a feature of the shape model exceeds the boundary of the medical imaging data. The output may generate a recommendation for corrective action based on the rejection of the medical imaging data. The medical imaging data may be a CT volume. The target anatomical structure may be a skeletal structure. Assessing whether the medical imaging data is acceptable for facilitating surgical planning for the target anatomical structure can be defined as assessing whether the medical imaging data is acceptable for facilitating any one or more of the following: segmentation of the medical imaging data; creation of a virtual model of the target anatomical structure; virtual implant planning relative to the target anatomical structure; registration of the target anatomical structure to facilitate surgical navigation; and visualization of the medical imaging data to facilitate surgical navigation. In response to the determination that the required features of the target anatomical structure are fully captured in the medical imaging data, the medical imaging data can be automatically approved as acceptable for facilitating surgical planning. Measurements of the target anatomical structure can be automatically obtained. A shape model can be automatically identified and fitted to the target anatomical structure in the medical imaging data. Measurements of the target anatomical structure can be automatically obtained from the shape model. Distances between points on the shape model can be automatically measured. Each point can correspond to a pre-defined anatomical landmark of the target anatomical structure. Implants can be automatically selected based on these measurements of the target anatomical structure.The implant can be automatically selected and appropriately sized for the measurement of the target anatomy. The required features of the target anatomy can be the required measurement and / or required markers. Output can be automatically generated in response to the medical imaging data being rejected as unacceptable. This output may include displaying the medical imaging data on a graphical user interface along with the required features of the target anatomy that were not fully captured in the medical imaging data. The output may also include displaying the medical imaging data on a graphical user interface along with the selected implant or the implant measurement. In response to determining that the implant measurement is within or separated from the boundary of the medical imaging data by a threshold distance, the medical imaging data can be automatically approved as acceptable for facilitating surgical planning. An implant shape model can be automatically obtained from the selected implant. Measurements of the selected implant can be obtained from the implant shape model. The implant shape model can be fitted to the target anatomy in the medical imaging data. The implant shape model may include boundaries or points. The boundaries or points of the implant shape model can be automatically compared to the boundaries of the medical imaging data. In response to determining that the boundary or point of the implant shape model exceeds the boundary of the medical imaging data or fails to be separated from the boundary of the medical imaging data by the threshold distance, the medical imaging data can be automatically rejected as unacceptable for facilitating surgical planning. In response to determining that any one or more of these expected parameters fail to acceptably match their corresponding predicted parameters, the medical imaging data can be automatically rejected as unacceptable for facilitating surgical planning. The medical imaging data may include a CT volume. From the CT volume, a two-dimensional digitally reconstructed radiograph can be automatically generated. The digitally reconstructed radiograph can be automatically applied to a machine learning model to analyze the digitally reconstructed radiograph to output the predicted parameters. The machine learning model may be a convolutional neural network. The medical imaging data can be automatically applied to the convolutional neural network to analyze the medical imaging data to output the predicted parameters. Preoperative patient data associated with the medical imaging data can be automatically obtained. The preoperative patient data may include information indicating expected parameters such as the expected type of anatomical joint and the expected surgical side of the anatomical joint. The medical imaging data can be automatically applied to the machine learning model to analyze it and output predicted parameters such as the predicted type of anatomical joint and the predicted surgical side of that anatomical joint. The medical imaging data may include the CT volume of the anatomical joint. The machine learning model may be a lightweight 2D convolutional neural network. From the CT volume, a first digitally reconstructed radiograph, which is two-dimensional and captures the skeletal structure of the anatomical joint, can be automatically generated.From the CT volume, a second digitally reconstructed radiograph, which is two-dimensional and captures the soft tissue structure of the anatomical joint, can be automatically generated. The first digitally reconstructed radiograph can be automatically applied to a lightweight 2D convolutional neural network to analyze it and output the predicted type of anatomical joint. The second digitally reconstructed radiograph can be automatically applied to the lightweight 2D convolutional neural network to analyze it and output the predicted surgical side of the anatomical joint. Preoperative patient data including information indicating the expected presence or absence of existing implants for the target anatomy can be obtained. The medical imaging data can be automatically applied to the machine learning model to analyze it and further output predicted parameters such as the predicted presence or absence of existing implants for the target anatomy. The expected presence or absence of existing implants can be automatically compared with the predicted presence or absence of existing implants. In response to the failure of the determination of the anticipated presence or absence of an existing implant to acceptablely match the predicted presence or absence of the existing implant, the medical imaging data can be automatically rejected as unacceptable for facilitating surgical planning. From the CT volume, digitally reconstructed radiographs capturing the metallic structure within the anatomical joint can be automatically generated. These digitally reconstructed radiographs can be automatically applied to a convolutional neural network to analyze them and output the predicted presence or absence of an existing implant for the target anatomical structure. The digitally reconstructed radiographs can be automatically applied to a convolutional neural network to analyze them and output the predicted presence of the existing implant and one or more of the following: the predicted manufacturer of the existing implant; the predicted model number of the existing implant; and the predicted size or type of the existing implant. Preoperative patient data including information indicating anticipated parameters such as the anticipated type of procedure to be planned for the target anatomical structure can be obtained. The medical imaging data can be automatically applied to the machine learning model to analyze it and further output predicted parameters such as the predicted type of procedure to be planned for the target anatomical structure. The expected type of procedure can be automatically compared to the predicted type of procedure. In response to determining that the expected type of procedure fails to acceptablely match the predicted type of procedure, the medical imaging data can be automatically rejected as unacceptable for facilitating surgical planning. In response to determining that one or more configuration settings correspond to one or more acceptable configuration settings, the medical imaging data can be automatically approved as acceptable for facilitating surgical planning.
[0023] The medical imaging data can be a 3D CT volume. The target anatomical structure can be an anatomical joint. A first machine learning model can automatically classify the type of the anatomical joint in the 3D CT volume into one of the following: unilateral hip, unilateral knee, unilateral ankle, bilateral hip, bilateral knee, or bilateral ankle. A confidence score can be generated to indicate the accuracy of the classification of that type of anatomical joint in the 3D CT volume. The confidence score can be compared to a threshold. In response to the confidence score exceeding the threshold, the type of anatomical joint can be automatically classified. The classified type of anatomical joint can be used to automatically select the second machine learning model. Multiple confidence scores can be generated, indicating the accuracy of classifying the type of anatomical joint in the 3D CT volume into each of the following: unilateral hip, unilateral knee, unilateral ankle, bilateral hip, bilateral knee, or bilateral ankle. The highest confidence score among the multiple confidence scores can be identified. The anatomical joint of this type can be automatically classified based on the highest confidence score. The classified anatomical joint type can be used to automatically select the second machine learning model. The classified anatomical joint type can only be used to automatically select the second machine learning model if the type of the anatomical joint in the 3D CT volume is automatically classified as one of the following: unilateral hip, unilateral knee, or unilateral ankle. In response to the automatic classification of the anatomical joint type in the 3D CT volume as unilateral hip, a second machine learning model specifically trained to classify whether the unilateral hip joint is left or right can be selected. Similarly, in response to the automatic classification of the anatomical joint type in the 3D CT volume as unilateral knee, a second machine learning model specifically trained to classify whether the unilateral knee joint is left or right can be selected. In response to the automatic classification of the anatomical joint type in the 3D CT volume as a unilateral ankle joint, a second machine learning model specifically trained to classify the unilateral ankle joint as either a left or right ankle joint can be selected. A confidence score can be generated, indicating the classification accuracy of the lateral aspect of the anatomical joint in the 3D CT volume. This confidence score can be compared to a threshold. In response to the confidence score exceeding the threshold, the lateral aspect of the anatomical joint can be automatically classified. The classified lateral aspect of the anatomical joint can be used to automatically generate a computer-generated output identifying the classified lateral aspect. A first confidence score can indicate the classification accuracy of the anatomical joint as a left-sided joint. A second confidence score can indicate the classification accuracy of the anatomical joint as a right-sided joint.The highest confidence score among these first and second confidence scores can be identified as the basis for automatically classifying the lateral aspect of the anatomical joint based on that highest confidence score. A confidence score indicating the accuracy of classification of the target anatomical structure in the medical imaging data for that type can be generated, and in response to determining that the confidence score fails to meet an acceptable threshold, the medical imaging data can be automatically rejected as unacceptable for facilitating surgical planning. These first and second machine learning models can be deep learning models including convolutional neural networks.
[0024] Medical imaging examiners or automated imaging suites can be implemented in various ways, such as Software as a Medical Device (SaMD) (software designed for use for one or more medical purposes, which can perform these purposes without being part of a hardware medical device), Software as a Service (SaaS) (software accessed via the internet without downloading), and so on. The imaging data can include any type of imaging data, such as, but not limited to, 2D or 3D CT scans or volumes, X-rays, fluoroscopy image data, ultrasound imaging data, digital reconstructed radiographs (DRR), etc. This medical imaging data may have been previously classified or unclassified before being processed using the techniques described herein.
[0025] Extrapolation of the target anatomical structure, shape model, implant shape model, or medical image can be used in any of these ways. The medical imaging data can be modified to visualize the shape model (including the portion of the shape model that extends beyond the boundary of the medical imaging data). An extrapolated region can be generated that includes the portion of the shape model that extends beyond the boundary of the medical imaging data. The extrapolated region may be adjacent to the boundary of the medical imaging data, or the medical imaging data may be extended to include the extrapolated region. Within the extrapolated region, features of the target anatomical structure within the portion of the shape model that extends beyond the boundary of the medical imaging data can be artificially recreated. Soft tissue regions adjacent to the portion of the shape model that extends beyond the boundary of the medical imaging data can also be artificially recreated. Color coding and / or annotation can be used to identify the portion of the shape model that extends beyond the boundary of the medical imaging data.
[0026] Any of the examinations described herein can be used to visualize any type of anatomical structure, and these examinations can be used to assess and / or classify any type of anatomical structure, including but not limited to: bone; skeleton; soft tissue; ligaments; cartilage; osteophytes; shoulder joint bones or features, such as the glenoid fossa, scapula, or humerus, or any part thereof; spinal bones, such as vertebral bodies, intervertebral discs or cartilage, pedicles, or any part thereof; knee joint bones, such as the femur, tibia, patella, ligaments, cartilage, or any part thereof; hip joint bones, such as the femur, acetabulum, pelvis, cartilage, ligaments, or any part thereof; skull bones; facial bones; ankle bones; wrist bones; bone fragments or fractured parts. Other types of anatomical structures are envisioned. Furthermore, any of the examinations described herein can be used to visualize any type of foreign body or existing implant, and these examinations can be used to assess and / or classify by any of these examinations. Attached Figure Description
[0027] The advantages of the present invention will be readily understood, as they will become better understood when considered in conjunction with the accompanying drawings and by referring to the following detailed description.
[0028] Figure 1 It is a block diagram of an image processing workflow that includes evaluating medical imaging data using an automated image inspection suite, according to one implementation method.
[0029] Figure 2 Sample screenshots show a graphical user interface that can be used with automated image inspection suites.
[0030] Figure 3 This is a flowchart illustrating example steps of automated image parameter pre-checking according to one implementation method.
[0031] Figure 4 This is a flowchart illustrating example steps of an automated image view pre-check according to one implementation method.
[0032] Figure 5 This is an example illustration of medical imaging data of a target anatomical structure, in which automated image view pre-examination assesses the shape model or anatomical landmarks of the boundaries of the medical imaging data.
[0033] Figure 6 This is a flowchart illustrating example steps of automated planning pre-checks based on one implementation method.
[0034] Figure 7 It is a diagram of medical imaging data related to a target anatomical structure according to one implementation method, wherein automated planning of pre-examination assessment determines the shape model or markers of the boundaries of the medical imaging data.
[0035] Figure 8 It is a side view of an example implant with corresponding measurements that can be utilized by automated planning pre-examination, according to one implementation method.
[0036] Figure 9 This is a flowchart illustrating example steps of an automated image classification pre-check based on one implementation method.
[0037] Figure 10 This is a flowchart illustrating example steps of an automated motion pre-check according to one implementation method.
[0038] Figure 11 It is a combination of flowcharts and block diagrams illustrating example components and steps of an automated capacity classifier and side pre-checking according to one implementation.
[0039] Figure 12 It is a block diagram of an image processing workflow that includes the evaluation of medical imaging data using an automated image inspection suite, according to another implementation method.
[0040] Figure 13 Example screenshots show a graphical user interface (GUI) according to one implementation, which can be used in conjunction with any aspect of an automated image inspection suite to provide imaging inspection reports.
[0041] Figure 14 Another example screenshot shows a graphical user interface based on an alternative implementation, which can be used in conjunction with any aspect of an automated image inspection suite to provide imaging inspection reports.
[0042] Figure 15 Example screenshots show a graphical user interface based on an alternative implementation, which can be used in conjunction with an automated image inspection suite to provide a case file dashboard with image inspection status updates. Detailed Implementation
[0043] This article describes systems, computer-implemented methods, software programs, non-transitory computer-readable media, and / or techniques for the automatic evaluation of medical imaging data.
[0044] refer to Figure 1The Automated Imaging Inspection Suite (AICS) is provided for the automated evaluation of medical imaging data (MID). For ease of drafting, the term "Automated Imaging Inspection Suite (AICS)" is used only to organize the various aspects of the systems / methods / software / techniques described herein. The Automated Imaging Inspection Suite (AICS) can be implemented on a local computer and / or a remote server (such as a cloud computing server). Aspects or features of the Automated Imaging Inspection Suite (AICS) can be implemented on any number of controllers, processors, computers, software modules, or servers. The Automated Imaging Inspection Suite (AICS) may include non-transitory memory for storing instructions that, when executed by one or more processors, execute software, modules, or programs capable of performing any of the aspects described herein. The processes described herein can be implemented by any control system, one or more controllers, or one or more processing devices. None of the processes described herein need to be implicitly or explicitly executed by software designed or named "Automated Imaging Inspection Suite." For example, the software may be referred to as a "medical imaging inspector."
[0045] exist Figure 1 The document illustrates a sample workflow involving the Automated Imaging Review Suite (AICS). Treatment planning requests for one or more patients are generated by a hospital, organization, or surgeon. Treatment planning requests can be transmitted using any suitable technology or medium. According to one example, the AICS receives medical imaging data (MID) for a patient in accordance with the treatment planning request. In another example, as described below, a treatment planning request can be created after successful processing by the AICS. The medical imaging data (MID) can be provided simultaneously with or separately from the treatment planning request. For example, the AICS can receive imaging data (MID) from a PACS (Picture Archiving and Communication System) server or any centralized computing system, enabling healthcare professionals to share patient medical images and reports across various locations. Imaging data (MID) can be transferred to the AICS using any suitable method, such as transmission over the Internet, a physical connection via a data storage device (such as a flash drive), etc. Imaging data (MID) can be downloaded or retrieved in batches for multiple patients or as needed on a single patient basis.
[0046] Imaging data MIDs can be transferred to the Automated Imaging Examination Suite (AICS) using any suitable method (such as transmission over the Internet, physical connection via a data storage device, such as a flash drive, etc.). Imaging data MIDs can be downloaded or retrieved in batches for multiple patients or, as needed, on a single patient basis.
[0047] Imaging data (MID) is associated with the target anatomical structure (TA) for one or more patients. The techniques described herein are fully applicable to any type of target anatomical structure TA that may require surgical planning, such as bone, soft tissue, etc. For example, the target anatomical structure TA can be the hip joint or hip bone, knee joint or knee bone, shoulder joint or shoulder bone, ankle joint or ankle bone, vertebrae, or vertebrae, skull, etc. Surgical planning can facilitate any type of surgery, including: orthopedic surgery such as partial or total knee replacement surgery, partial or total hip replacement surgery, partial or total shoulder replacement surgery, anatomical or reverse shoulder surgery, arthroplasty, arthroplasty, discectomy, laminectomy, disc arthroplasty, trauma or fracture repair surgery, wrist repair, ankle repair, craniofacial surgery; cardiac surgery; oncological surgery; dental surgery, etc. The surgical planning process can be used to plan for any suitable type of implant or prosthesis required for implantable surgery. For example, implants or prostheses can be hip and knee implants, including single knee implants, double knee implants or total knee implants, femoral stem implants, acetabular cup implants, orthopedic screws, pedicle screws, orthopedic plates, etc.
[0048] Imaging data MID can include any type of imaging data, such as, but not limited to, CT scans or volumetric images, X-rays, fluoroscopic images, MRI scans, PET scans, ultrasound imaging data, digital reconstructed radiographs (DRR), single-photon emission computed tomography (SPECT) images, arthrography, etc. Imaging data MID can also be in any suitable file format, such as Analyze, the Neuroimaging Information Technology Initiative (Nifti), Minc, and Digital Imaging and Communications in Medicine (DICOM). Imaging data MID can be 3D volumetric image data or any number of 2D layers.
[0049] In addition to the visual image of the target anatomical structure (TA), the imaging data MID may further include data or metadata. For example, the data may include textual or digital information related to the patient or the imaging data MID. Either textual or digital information may be included on the image scan itself and / or provided as an electronic file along with these image scans. In one example, the imaging data MID may include data related to scanner configuration settings, such as, but not limited to: radiation dose, detector configuration (beam collimation), pixel size, resolution, window width, window level, number of slices, slice thickness, image density, intensity, automatic exposure control (AEC) parameters (such as noise figure or mA level during tube current modulation), signal-to-noise ratio, tube potential (kV), gantry rotation, patient positioning, scan range, slice thickness and pitch, scan time, etc. Imaging data MID may also include (e.g., preoperative) patient-specific information or surgical information, such as, but not limited to: patient name, date of birth, sex, the patient's planned surgical side, data related to the treatment request, surgeon's name, type of surgery to be performed, type of planned implant, and planned date of surgery.
[0050] like Figure 1 and Figure 2 As shown, the Automated Image Inspection Suite (AICS) can optionally be implemented with a visual or graphical user interface (GUI), which can be provided on any suitable display device (DD). Figure 2 The GUI can be a data review screen to display any suitable information related to the imaging data, such as, but not limited to: imaging data MID from different planes or layers, slice thickness, slice spacing, field of view of the displayed imaging data, scan parameters (such as pixel size, resolution, voltage, current, scan date), patient information (such as name, date of birth, gender, etc.), the patient's planned surgical side, data related to treatment requests, surgeon's name, type of surgery to be performed, type of planned implant, planned date of surgery, etc. The GUI can be used to optionally enable technicians to: review the data described above, review the output of automated functionality implemented by the Automated Imaging Review Suite (AICS), visualize segmentation or anatomical models, and / or (before or after the automated functionality implemented by the Automated Imaging Review Suite (AICS)) perform supplemental manual review.
[0051] A. Image pre-examination
[0052] refer to Figure 1The Automated Image Inspection Suite (AICS) can perform one or more automated image pre-inspections on imaging data MID. As used herein, the term "pre-inspection" defines the initial examination of received imaging data MID before it is further processed for use in any downstream surgical planning (e.g., segmentation, anatomical model creation, implant planning) or aspects of a surgical workflow dependent on surgical planning or medical imaging (e.g., surgical navigation visualization, anatomical registration, etc.). For this purpose, the Automated Image Inspection Suite (AICS) is configured to preoperatively evaluate medical imaging data (i.e., prior to the virtual surgical planning process and before the actual surgery).
[0053] As described in the following sections, the Automated Image Inspection Suite (AICS) is configured to perform any one or more of the following automated pre-checks: Automated Image Parameter Pre-check (IPPC), Automated Image View Pre-check (IVPC), Automated Planning Pre-check (PPC), Automated Image Classification Pre-check (ICPC), Automated Motion Pre-check (MRC), and Automated Capacity Classifier and Side Classification Pre-check (VLC). The AICS can perform any of these automated image pre-checks in several ways. In one example, any one or more of the automated pre-checks can be performed individually or independently of the other pre-checks. Alternatively, automated pre-checks can be performed simultaneously, jointly, or together. Additionally, the AICS can perform partial aspects of any of the pre-checks described herein, or combine these partial aspects with features of other pre-checks. Furthermore, any two or more of these automated pre-checks can be combined into a single inspection operation. Additionally, any two or more of these automated pre-checks can be performed in priority order or any other order. In some instances, the AICS can selectively determine which automated image pre-check to perform or not perform. For example, the Automated Image Examination Suite (AICS) can make this determination in response to automatically identifying missing information about a patient in the imaging data MID. The priority of examinations can be predefined based on the statistical or predictive probability of such erroneous identification. The AICS can output a single report containing information about the results of any one examination, or a combined report containing information about the results of multiple examinations. Furthermore, this document describes techniques for extrapolating medical images and / or target anatomical structures beyond the original boundaries of the medical image, for example, to provide methods to salvage scans that would otherwise fail to pass the corresponding examination due to the failure to capture the necessary amount of target anatomical structures.
[0054] Using these image pre-checks, the Automated Image Inspection Suite (AICS) can automatically, quickly, and accurately identify defects or errors in imaging data mid-injection (MID). In doing so, AICS can automatically determine whether the imaging data MID is suitable for downstream surgical planning (e.g., segmentation, anatomical model creation, implant planning) or for various aspects of surgical workflows that rely on surgical planning or medical imaging (e.g., surgical navigation visualization, anatomical registration, etc.). AICS can process the described pre-checks in seconds, thereby reducing the time-consuming burden on surgical planning teams to manually review imaging data MIDs to identify potential errors. Consequently, AICS can significantly reduce the labor costs and human error associated with manually reviewing imaging data.
[0055] 1. Automated image parameter pre-check
[0056] refer to Figure 3 The Automated Image Inspection Kit (AICS) can perform automated image parameter pre-checking (IPPC), i.e., pre-checking to determine whether the imaging data (MID) was acquired with acceptable image parameters or scanner configuration settings. Acceptable image parameters can be those deemed satisfactory by the AICS supplier or those acceptable according to industry standards. Settings can include, but are not limited to: radiation dose, detector configuration (beam collimation), pixel size, resolution, window width, window level, image density, intensity, automatic exposure control (AEC) parameters (such as noise figure or mA level during tube current modulation), signal-to-noise ratio, tube potential (kV), gantry rotation, patient positioning, scan range, slice thickness and pitch, scan time, etc. The AICS can include a list of acceptable image setting values or ranges. For example, these values can be stored in a lookup table. Figure 3 The diagram illustrates an example method 100 for performing an image parameter pre-examination IPPC and includes a step 102 of receiving imaging data MID of the target anatomical structure TA. The received imaging data MID includes image parameters, which may be encoded in a DICOM file, for example. For this examination, the received imaging data MID may or may not include the actual image scan of the patient. At step 104, the Automated Image Examination Suite (AICS) can automatically identify and extract the image parameters from the received imaging data MID. This process may include text or string search or matching algorithms, etc. At step 106, the Automated Image Examination Suite (AICS) can automatically compare the extracted image parameters with stored acceptable image parameter values.
[0057] At step 108, a comparison can be used to determine whether the extracted value is acceptable or falls within a predetermined range or threshold. If the extracted value is acceptable, at step 110, the Automated Image Parameter Pre-Check (IPPC) can automatically generate a response to confirm acceptability. In one example, the response is to send an acknowledgment to the GUI so that a technician can see that the pre-check has passed or the settings are correct. In another example, the response is to allow the Automated Image Inspection Suite (AICS) to continue processing additional pre-checks (e.g., in an ordered series of checks). Other responses are envisioned, such as not generating a response unless an error is detected.
[0058] If the extracted values are unacceptable, at step 112, the Automated Image Parameter Pre-Check IPPC can automatically generate a response relating to the unacceptability. In one example, the response is to send an alert or notification to the GUI so that technicians can see that the pre-check has failed, or to identify which / which settings are incorrect. In another example, the response is to have the Automated Image Inspection Suite (AICS) stop processing additional pre-checks (e.g., in an ordered inspection series). The response can also be to output a message requesting a new scan with appropriate parameters.
[0059] 2. Automated image view pre-check
[0060] refer to Figure 4 and Figure 5 The Automated Image Inspection Suite (AICS) can perform an image view pre-examination (IVPC). The IVPC automatically determines whether the image from the imaging data MID includes the required detail of the target anatomical structure TA (e.g., bone); whether the image is not large enough; whether the target anatomical structure TA is adequately captured within the image view; whether the target anatomical structure TA is cropped; and / or whether any required portion of the target anatomical structure extends beyond the boundaries of the image MID. For each patient, the IVPC can be repeated for any number or all layers of the medical imaging data MID that include the patient's target anatomical structure TA.
[0061] Image-view pre-checking (IVPC) can be relevant to surgical planning purposes, such as segmentation or anatomical model creation, where ensuring the required amount of target anatomical structures is captured within the image data is crucial. Otherwise, the segmentation or anatomical model may be incomplete or contain errors. IVPC can also be relevant to navigation purposes. For example, ensuring the required amount of target anatomical structures is captured within the image data ensures the accuracy of the visualization of the image data referenced by the surgeon during navigational surgery, as well as the accuracy of the anatomical registration process, where the anatomical model is registered to the physical target anatomical structure (TA).
[0062] Figure 4 An example method 200 for performing automated image view pre-examination (IVPC) is illustrated, including step 202 of receiving imaging data MID of a target anatomical structure TA. At step 204, the Automated Image Examination Suite (AICS) automatically identifies a shape model SSM and fits the shape model to the target anatomical structure TA in the medical imaging data MID. The term "shape model" as used herein can include any one or more of the following: statistical shape model, active shape model, active appearance model, active contour model, or any other suitable type of shape model. In one example, the shape model SSM can be understood as a mesh, shape, or contour with adjustable nodes to deform the mesh, shape, or contour to substantially conform to the shape or contour of the target anatomical structure TA in the medical imaging data MID. The shape model can be initially derived or generated from a population of other patient images exhibiting anatomical structures similar to the target anatomical structure TA. This population can exhibit similar characteristics of the subjects, such as age, sex, ethnicity, body size, or other physical or demographic data. For example, when the target anatomical structure TA is bone, the shape model SSM can be derived from a statistical representation of images of bone from a comparable anatomical origin, derived from a group of patients known to have normal or “healthy” skeletal anatomy. The shape data used to derive the shape model SSM can include geometric features of the bone, such as markers AL, surfaces, boundaries, geometric features, or intensity information of the target anatomical structure TA. The shape model can be provided from analyses of images from other patients and / or from point clouds acquired from normal bone from comparable anatomical origins. The Automated Image Inspection Suite (AICS) can select one shape model from multiple shape models that best fits the target anatomical structure. The best-fit shape model may or may not need to be morphed to the target anatomical structure. Alternatively, the Automated Image Inspection Suite (AICS) can utilize a generic shape model (for a specific anatomical structure type) and morph it to the target anatomical structure. In any implementation described herein, the shape model can be a single shape model or can be implemented as multiple shape model instances.
[0063] A shape model SSM can be utilized by employing an algorithm that automatically segments medical imaging data (MID). In one implementation, the SSM can represent a “healthy” anatomical structure that does not necessarily correspond precisely to a target anatomical structure in the subject patient. The segmentation algorithm can perform image processing such as aligning the coordinates of the target anatomical structure TA in the image data with the SSM. Alignment can be based on key landmarks on the SSM and the target anatomical structure TA in the image data. The segmentation algorithm can then variant, deform, or scale the SSM until the SSM and the target anatomical structure TA in the image data are registered. This fitting process can be performed using any optimization technique, such as a least-squares optimizer. Registration can include adjusting the size and shape of the SSM to match the target anatomical structure TA in the image data, and adjusting the position of the SSM to align with the target anatomical structure TA in the image data. The result of registration can be an approximate shape model of the target anatomical structure TA in the image data (e.g., optionally with or without osteophytes or abnormal morphology).
[0064] In another implementation, the automated segmentation algorithm performs an initial segmentation process on the image data associated with the target anatomical structure TA using a first shape model to generate an initial segmentation of the target anatomical structure TA. Automated segmentation may optionally perform a fine segmentation process on the image regions of the image data associated with the target anatomical structure TA using a neural network, which takes the image data of the target anatomical structure TA and the output of the initial segmentation as input. The first shape model may be mapped to the output of the fine segmentation process. Optionally, anatomical landmarks AL of the target anatomical structure TA may be identified. An example of such automated segmentation can be implemented as described in U.S. Provisional Patent Application No. 63 / 505,466, filed June 1, 2023, entitled “Segmentation of Bony Structures” (Attorney General’s File No. 060939.01029), the entire contents of which are hereby incorporated by reference.
[0065] like Figure 5 As shown, the shape model SSM (derived from any of the techniques described above) is represented as a shape that approximates the contour of the target anatomical structure TA (e.g., knee joint bone) relative to the medical imaging data MID. The process of visualizing the shape model SSM relative to an image is optional and not necessary for this automated pre-examination. If useful, a technician can manually retrieve and review any shape model relative to any corresponding image on the GUI.
[0066] exist Figure 5 In, and as Figure 4As shown in step 206, the Automated Image View Pre-Check (IVPC) performs an automated comparison of the shape model SSM with the boundary MIB of the medical imaging data MID. The parameters of the boundary MIB of the medical imaging data MID can be derived from scanner configuration settings or image parameters, and / or by automatically measuring the size of the boundary MIB. In most cases (but not always), the boundary MIB of the medical imaging data MID will be rectangular.
[0067] In one implementation, the comparison is a boundary-to-boundary comparison, specifically a comparison between the boundary SSM-B of the shape model SSM and the boundary MIB of the medical imaging data MID. The shape model boundary SSM-B can be determined from the parameters or size of the shape model SSM and / or by automatically measuring the boundary SSM-B. In some instances, the shape model SSM and the medical imaging data MID are registered to a common coordinate system and compared within that system, which can be the image coordinate system, the shape model coordinate system, or any arbitrary coordinate system. The coordinate system used to measure these boundaries can be larger than the boundary MIB of the medical imaging data MID to enable detection of the shape model SSM extending beyond the image boundary MIB. Boundary comparisons can be performed relative to all or some of the boundaries of the shape model SSM. Either side of this comparison can be a backend (non-visualization process) and does not necessarily need to be visible to the user.
[0068] In another implementation, the automated image view pre-checking (IVPC) performs point-to-boundary comparisons, rather than boundary-to-boundary evaluations. For example, one or more anatomical landmarks (ALs) of a target anatomical structure (TA) identified during the automated segmentation process can be mapped to a shape model (SSM) (such as...). Figure 5(As shown in the diagram). Automated image view pre-checking (IVPC) assesses whether anatomical landmarks (ALs) fall within or beyond the image boundary MIB. In one example, ALs can be derived from clinical data that identifies which landmarks need to be visible to enable surgical planning and execution. In another example, ALs can be derived from those points required for anatomical registration during intraoperative navigation surgery. Alternatively, anatomical landmarks (ALs) can be one or more points automatically selected based on the likelihood that the ALs extend beyond the image boundary MIB. For example, ALs can be the anterior, posterior, medial, lateral, highest, or lowest point of the target anatomical structure (TA). Alternatively, Automated image view pre-checking (IVPC) can compare medical imaging data (MID) with a statistical population rather than a shape model of medical images that include other anatomical structures for comparison to identify the required anatomical landmarks (ALs) visible in the medical imaging data (MID). Automated image view pre-checking (IVPC) can automatically assess medical imaging data (MID) to determine whether the required anatomical landmarks (ALs) are visible in the medical imaging data (MID).
[0069] Point-to-point evaluation is also envisioned. The shape of the SSM can be interpolated using multiple points with coordinates in a coordinate system. The Medical Imaging Data (MID) can be interpolated into a grid of points or pixels. Automated Image View Pre-Check (IVPC) can check whether the coordinates of points in the SSM correspond to or overlap with the coordinates of points or pixels in the Medical Imaging Data (MID). Automated Image View Pre-Check (IVPC) can be combined with any of the described techniques for comparing the SSM to the Medical Imaging Data (MID).
[0070] At step 208, boundary comparisons determine whether the medical imaging data MID is acceptable. Figure 5 In the example, the Automated Image View Pre-Check (IVPC) identifies that the outer right contour of the Shape Model B (SSM-B) extends beyond the image boundary (MIB). Therefore, the IVPC detects one or more of the following: the image fails to include the required detail of the target anatomical structure (TA); the image is not large enough; the target anatomical structure (TA) is not fully captured within the image view; the target anatomical structure (TA) is cropped; and / or the required portion of the target anatomical structure extends beyond the boundary of the image MID.
[0071] If, as a result of this comparison, the medical imaging data MID is determined to be unacceptable, the Automated Imaging Suite (AICS) can automatically generate a response relating to the unacceptability at step 210. In one example, the response is to send an alert or notification to the GUI so that the technician can see that the pre-examination has failed, or to identify which image layers have failed, or optionally to display in the GUI the corresponding layer and SSM showing the boundary comparison of the failure. The response could also be to output a message requesting the patient to obtain a larger scan. In another example, the response is to have the Automated Imaging Suite (AICS) stop processing additional pre-examinations (e.g., in an ordered series of examinations).
[0072] On the other hand, as a result of this comparison, medical imaging data MID can be determined to be acceptable. For example, in Figure 5 In the example, if the outer right contour of the shape model boundary SSM-B is (hypothetically) captured inside the image boundary MIB, the Automated Image View Pre-inspection (IVPC) detects one or more of the following: the image successfully includes the required detail of the target anatomical structure TA; the image is large enough; the target anatomical structure TA is fully captured within the image view; the target anatomical structure TA is not cropped; and / or the required portion of the target anatomical structure falls within the boundary of the image MID. If acceptable, the Automated Image Inspection Suite (AICS) can automatically generate a response at step 212 to confirm acceptability. In one example, the response is to send an acknowledgment to the GUI so that the technician can see that the pre-inspection has passed. In another example, the response is to allow the Automated Image Inspection Suite (AICS) to continue processing additional pre-inspections (e.g., in an ordered series of inspections). Other responses are envisioned, such as not generating a response unless an error is detected.
[0073] 2A. Anatomical Structure / Image Extrapolation
[0074] refer to Figure 4 and Figure 5 Automated Image Inspection Suite (AICS) and / or Image View Pre-Inspection Programming (IVPC) can be equipped with the functionality to predictively extrapolate or extend the target anatomical structure (TA) and / or the image midpoint (MID). For example, if it is determined that the required amount of the target anatomical structure (TA) is not captured in the image midpoint, the AICS can artificially extend the target anatomical structure (TA) beyond the original boundary (MIB) of the image midpoint. Furthermore, the shape model SSM is used to assess the required length of the target anatomical structure (TA) in the image midpoint; here, the same shape model SSM or a different shape model can be further used to extend the anatomical structure to the required length or size for capture.
[0075] To achieve this functionality, the Automated Image Inspection Suite (AICS) automatically identifies the shape model SSM and fits this shape model to the target anatomical structure TA in the medical imaging data MID. Figure 4 Step 204). In Figure 5 In, and as Figure 4 As shown at step 206, an automated comparison is performed between the shape model SSM and the boundary MIB of the image MID. At 208, this comparison determines that the shape model SSM (or a portion thereof) extends beyond the boundary MIB (and would otherwise be unacceptable). For example, in Figure 5 In the example, the outer right contour portion of the identified shape model boundary SSM-B extends beyond the image boundary MIB. However, an extrapolation technique can be used to retain the image MID instead of rejecting it at step 210, thus avoiding the need to rescan the image MID. That is, at step 220, the image MID can be automatically annotated, modified, regenerated, or reconstructed to include information about the shape model SSM, and specifically, the portion of the shape model that extends beyond the image boundary MIB. This extrapolation can be performed in a manner that allows the image MID to be automatically approved (at 212). Similarly, the shape model SSM used to check boundary limits (at steps 204, 206, 208) can be the same as or different from the shape model SSM used for extrapolation (at step 220).
[0076] To capture portions of the shape model that extend beyond the image boundary MIB, the Automated Image Inspection Suite (AICS) can generate or identify extrapolated regions (ERs), such as... Figure 5 As shown in the example, the extrapolation region (ER) can be an artificial extension of the image MID. The extrapolation region (ER) can modify the size of the boundary MIB or can be identified as a region independent of the original image MID. Inside the extrapolation region (ER), the shape model SSM portion can be visually preserved to simulate the required amount of target anatomical structure TA. For example, when retrieving images later for planning purposes, the image MID can show the boundary SSM-B of the shape model SSM, which extends beyond the original boundary MIB of the image data MID and optionally lies within the extrapolation region (ER).
[0077] In some cases, the target anatomical structure (TA) within the extrapolation region (ER) can be automatically populated with features derived from the shape model SMM or other statistical data. For example, the region within the boundary SSM-B can be artificially populated with anatomical features (such as bone features that would have been captured if the initial scan image was larger) instead of simply showing the boundary SSM-B of the shape model SSM in the extrapolation region (ER). Similarly, tissue surrounding the target anatomical structure (TA) can be artificially populated within the extrapolation region (ER) with anatomical features, such as surrounding soft tissue that has already been captured and has a larger image at the time of the initial scan. In some cases, tissue or bone regions within the extrapolation region (ER) can be artificially populated using scan data from adjacent layers that may include missing features.
[0078] Alternatively, the entire image MID can be artificially regenerated or reconstructed to include the extrapolated region (ER). Using this technique, there will be no visible boundary between the boundary MIB of the image MID and the extrapolated region (ER). Instead, the two regions will be seamlessly merged into a single image MID. This will allow the image MID to appear as if it included the necessary amount of target anatomical structures (TA) despite being originally scanned.
[0079] Any textual or visual indicator can be used to distinguish between the extrapolated region (ER) or features within the extrapolated region (ER). For example, the shape model SSM portion can be annotated, labeled, or provided with a message indicating that the portion is extrapolated, predicted, or otherwise not present in the initially scanned image MID. Visually, the shape model SSM portion in the extrapolated region (ER) can be color-coded, for example, to differ from the rest of the image or shape model SMM. Such markings can be used to convey to the reviewer that the target anatomical structure TA in the region (ER) is extrapolated and may not perfectly represent the actual scanned anatomical structure. The further the extrapolation is from the original image boundary MIB, the greater the likelihood of a prediction error. Thus, the image MID can also include a tiered level of textual or visual indicators that indicates the greater the likelihood of error as the extrapolation moves further from the original image boundary MIB. For example, in the extrapolation region (ER), a gradient color scheme can be used to highlight the portion of the target anatomical structure closest to the original image boundary (MIB) using green, and the green can transition to orange and finally to red to highlight the portion of the target anatomical structure furthest from the original image boundary (MIB).
[0080] The described extrapolation technique can offer numerous benefits to surgical procedures where medical imaging data often does not include the necessary amount of target anatomical structures (TA). For example, for revision total knee arthroplasty (TKA), planning for a 50 mm longer stem requires a longer CT scan, but the actual CT scan length may vary. The described extrapolation technique can utilize current CT scan images and extrapolate the required additional length based on the shape model (SM). For TKA revision cases, the stem length can be 50 mm, 100 mm, or 150 mm. To plan for 100 mm and 150 mm stems, the bone requires scan lengths of 150 mm and 200 mm, respectively. To avoid having two different CT scan protocol lengths, and to avoid potentially shorter scan lengths than needed, the described technique can be used to extrapolate the length for the initial procedure to 100 mm and the length for the revision procedure to 200 mm. In this way, there will be no need to refuse a CT scan due to scan length, and the patient will not need to undergo another CT scan.
[0081] This is beneficial for short CT scans (even for initial cases), eliminating the need to reject scans based on length. For initial TKA cases, CT scan protocols specify scans should be taken within ±100 mm of the joint line. Often, scans are much shorter than this distance. There are minimum length requirements for generating surfaces to map bone registration points and for making notch calculations for the anterior cut. The minimum scan length depends on the implant size, but the implant size cannot be estimated before the scan is segmented. Sometimes, scans must be rejected due to violations of the minimum scan length. If rejected, notification usually occurs more than one day after the scan. This then requires the patient to return to the CT scan facility for another scan. To prevent rejection, shape models (SMs) can be trained to accurately extrapolate the femoral / tibial shaft to ±100 mm, thus avoiding image rejection and the need for rescanning.
[0082] 3. Automated planning pre-check
[0083] refer to Figures 6 to 8The Automated Imaging Review Suite (AICS) can perform automated pre-planning examination (PPC). The primary purpose of PPC is to assess whether medical imaging data (MID) is suitable for downstream surgical planning purposes related to an implant targeting the target anatomical structure (TA). The implant used in this examination is not natively identified in the imaging data MID, but rather evaluated by comparing it to the MID. The implant can be a planned implant, a proposed implant, a best guess implant, or a theoretical implant. While PPC can utilize data from the surgeon's planning, it is not intended to replace the surgeon's planning. Instead, PPC extracts implant-related data in advance to assess whether the medical imaging data MID adequately represents the range of the target anatomical structure (TA) required for the implant planning purpose.
[0084] Accordingly, the planning pre-examination PPC can automatically determine: whether the image from the imaging data MID includes the required detail of the target anatomical structure TA (e.g., bone) needed for implant planning; whether the image is not large enough to capture the required portion of the target anatomical structure TA, considering the implant; whether the portion of the target anatomical structure to which the implant will be positioned cannot be captured within the image view; whether the portion of the target anatomical structure to which the implant will be positioned is cropped; and / or whether any required portion of the target anatomical structure required for implant planning extends beyond the boundaries of the image MID. For each patient, the planning pre-examination PPC can be repeated for any number or all layers of the medical imaging data MID that include the patient's target anatomical structure TA. The planning pre-examination PPC can be combined with the image view pre-examination IVPC in a single examination. In addition, the planning pre-examination PPC can be performed simultaneously on many target anatomical structures TA (e.g., relative bones such as anatomical joints) in the medical imaging data MID.
[0085] Figure 6 An example method 300 for performing automated planning pre-inspection (PPC) is shown, and includes a step 302 of receiving imaging data MID of a target anatomical structure TA. At step 304, the Automated Image Inspection Suite (AICS) automatically performs measurements AM on the target anatomical structure TA found within the image MID. In one example, this can be accomplished by identifying markers AL of the target anatomical structure TA and measuring the distances between these markers AL. For example, as... Figure 7As shown, when the image is of the knee joint, the marker AL can be a distal feature of the femur (such as the condylar surface) and the nearest lateral point of the femur at the boundary MIB. This measurement can represent the length of the femoral shaft visible in the medical imaging data MID. Of course, depending on the image, the type of the target anatomical structure TA, or procedural preferences, the marker AL can be any appropriate point or surface on the target anatomical structure TA. The Automated Image Inspection Suite (AICS) can use a digital ruler to measure the distance between marker ALs. Any appropriate number of marker ALs can be used.
[0086] Alternatively, at step 304, the Automated Image Inspection Suite (AICS) automatically obtains the target anatomical structure measurement (AM) by identifying the shape model SSM and fitting that shape model to the target anatomical structure TA in the medical imaging data MID. Any of the techniques described above for method 200 can be used to identify, generate, and / or fit the shape model SSM. Therefore, the specific details and various implementations of the shape model SSM are fully incorporated into this section and will not be repeated for the sake of simplicity. Figure 7 In the example, the shape model SSM is fitted to the target anatomical structure TA of the femur. As described above, the shape model SSM can extend beyond the boundaries of the image MID. The parameters of the shape model SSM's boundaries include measurements of the shape model SSM. The Automated Image Inspection Suite (AICS) can automatically obtain the target anatomical structure measurement AM based on the SSM parameters and by comparing these parameters with the medical image boundary MIB.
[0087] Regardless of whether the marker AL and / or shape model SSM are used, the measurement of the target anatomical structure TA can include any one or more measurements, such as length, width, area, volume, perimeter, height, depth, thickness, orientation, position, inversion, eversion, version, back tilt, and tilt angle. Figure 7 In this study, the recorded measurement AM is the length of the femur from the condylar surface to the boundary of the image MIB, which is 87 mm in this example and partially represents the femoral shaft length.
[0088] Method 300 includes step 306 identifying the implant based on the target anatomical structure measurement AM acquired at step 304, or the implant measurement SI. Similarly, the implant or implant measurement SI used in this step can be a planned implant, a proposed implant, a best guess implant, or a theoretical implant. In one example, the Automated Image Inspection Suite (AICS) can automatically select from a database an implant or implant measurement SI that has been appropriately sized for the parameters of the target anatomical structure TA, based on the target anatomical structure measurement AM. The implant or implant measurement SI selected from the database can be represented as a virtual model or SSM of the implant. Alternatively, or additionally, the selected implant or implant measurement SI can include measurement data without any graphical or virtual elements. Based on the target anatomical structure measurement AM, the Automated Image Inspection Suite (AICS) can select an implant with appropriate size, orientation, biometric or mechanical fit, type, or configuration. The implant or implant measurement can be selected based on manufacturer, model, size identifier, and / or based on anatomical measurements.
[0089] According to the first technique, at step 308, the Automated Image Inspection Suite (AICS) can determine parameters of the required feature RF of the target anatomical structure based on the identified implant or implant measurement SI (from step 306). In one example, the required feature RF defines, based on the selected implant SI, the required quantity, measurement, extent, or landmark of the target anatomical structure that must be visible in the medical imaging data MID. For example, if the selected implant is a size 6 femoral knee joint assembly, the implant may include data defining the geometric measurements of the implant (such as AP size, ML size, overhang length, etc.). The selected implant SI may also require the femur to include the required shaft length to properly accommodate the selected femoral assembly (in image space). For example, as... Figure 8 The example shown is for illustrative purposes only, illustrated in a sagittal view of the selected implant SI. The selected implant SI can be represented by numerical or textual data and does not need to be displayed. In this example, the selected implant SI is the femoral component of the knee prosthesis derived from the target anatomical measurement AM. The selected implant SI includes implant measurements IM, such as the total length of the implant, for example, from the proximal tip of the anterior edge to the distal point of the condylar contact surface. In one example, based on the selected size of the implant, the illustrated implant measurement IM height could be 72 mm. Of course, any other measurements can be obtained as needed.
[0090] Implant measurements retrieved from the database for the selected implant SI may include a 72 mm implant size and parameters of the required characteristic RF for the target anatomy. The parameters of the required characteristic RF for the target anatomy can be obtained from various sources, such as from the implant manufacturer, minimum threshold measurements (e.g., 20% larger than the implant measurement), implant coverage thresholds, safety tolerances, clinical or statistical data, surgeon preferences, etc. Figure 8 In the example shown, the required feature RF of the target anatomical structure (femur) can be a measurement of 92 mm, which can be understood as including a measurement of the required femoral shaft length necessary to accommodate the selected implant SI. Of course, depending on the type of target anatomical structure, the type of protocol being planned, the specific implant measurement being selected, and the specific tolerance of the required anatomical structure, step 308 can produce different results. Alternatively, as... Figure 8 As shown, the required feature can be defined at the end of the required measurement, or a point or marker RF' defined based on a predefined point on the femoral shaft.
[0091] Continuing with the first technique, at step 310, the Automated Image Inspection Suite (AICS) can automatically compare the requested feature RF (obtained from step 308) with the anatomical measurement AM and / or the medical image boundary MIB. In one example, this step can be performed by adding the target anatomical structure measurement AM originally obtained at step 304. Alternatively, step 310 may include remeasuring the target anatomical structure TA in the image MID based on the requested feature RF. For example, the Automated Image Inspection Suite (AICS) can automatically register or correlate reference coordinates with corresponding markers of the target anatomical structure on the image data MID from which implant measurements are taken. For example, reference... Figure 8 For example, in the image MID, the required feature RF can be evaluated starting from the farthest point of the condylar contact surface. Continuing with the example shown, the target anatomical structure TA is originally measured to be 87 mm within the image boundary MIB, and the required feature RF is measured to be 92 mm. Figure 7 In this case, the Automated Image Inspection Suite (AICS) automatically measured the required Feature RF and determined that the required Feature RF extended beyond the image boundary MIB. Therefore, the image MID failed to include the required 5 mm of the target anatomical structure TA.
[0092] Alternatively, in another technique, the Automated Image Inspection Suite (AICS) can perform automated planning pre-examination (PPC) by evaluating the selected implant SI or implant measurement IM relative to the target anatomical structure TA in the image. In other words, in this technique, the implant parameters themselves are evaluated, and it is not necessary to obtain or evaluate the required feature RF (derived from the implant). Specifically, as shown at step 312, the Automated Image Inspection Suite (AICS) can evaluate whether any portion of the identified implant SI or implant measurement IM extends beyond the medical image boundary MIB. In one example, this could be a check to determine whether the space between the identified implant SI or implant measurement IM and the image boundary is separated by a threshold distance. For example, the threshold distance could be based on the difference between: the implant measurement IM and the required feature RF; the implant measurement IM and the anatomical measurement AM; or the anatomical measurement AM and the required feature RF. Alternatively, the threshold distance could be a predefined distance (e.g., derived from statistical data). The Automated Image Inspection Suite (AICS) can automatically perform this comparison, for example, by registering or relating reference coordinates to corresponding markers of the target anatomical structure (TA) on the image data (MID), from which implant measurements (IM) are taken. For example, reference... Figure 8 For example, in the image MID, the implant SI or implant measurement IM can be assessed starting from the distal lateral point of the condylar contact surface (shown at marker AL). In some instances, in the case of the native articular surface of the target anatomical structure TA in the articular surface offset image of the planned implant surface, the offset implant distance can be included. Continuing with... Figure 7 In the example shown, the anatomical measurement AM, limited by the image, is 89 mm, and the implant measurement IM height is 72 mm. Based on these measurements, the Automated Image Inspection Suite (AICS) automatically determines that the selected implant SI or implant measurement IM is within the image boundary MIB. Therefore, the image MID fully encompasses the required image of the target anatomical structure TA.
[0093] Alternatively, to compare the implant or implant measurement against the medical image boundary MIB, the selected implant SI can be represented as an implant shape model SSM-I. The Automated Image Inspection Suite (AICS) can fit the implant shape model SSM-I to the target anatomical structure TA in the image MID at the planned location. The AICS automatically determines whether any portion of the boundary of the implant shape model SSM-I extends beyond the image boundary MIB. Figure 7 In the example, the implant shape model SSM-I is captured within the image MID.
[0094] After applying one or two of the techniques described above, (in) Figure 6 Step 314) Automated planning pre-check (PPC) assesses whether the medical imaging data (MID) is acceptable. For example, in Figure 7 In the example, the Automated Image View Pre-Check (IVPC) identifies that the required feature RF extends beyond the image boundary MIB. Assuming this is the only type of check, the Automated Planning Pre-Check (PPC) at step 314 will determine that the image is unacceptable. Reasons for rejection can be characterized as follows: the image fails to include the required detail of the target anatomical structure TA (e.g., bone) necessary for implant planning; the image is not large enough to capture the required portion of the target anatomical structure TA considering the implant, or is not large enough to capture the area required by the planned implant; the portion of the target anatomical structure to which the implant will be positioned is not captured within the image view; the portion of the target anatomical structure to which the implant will be positioned is cropped; the portion of the target anatomical structure required for implant planning extends beyond the image boundary; and / or the planned implant for the target anatomical structure will extend beyond the image boundary, or is not spaced from the image boundary by a threshold distance.
[0095] At step 316, if the Medical Imaging Data (MID) is determined to be unacceptable as a result of the Automated Planning Pre-examination (PPC), a response relating to the unacceptability can be automatically generated. In one example, the response is to send an acknowledgment to the GUI so that the technician can see that the pre-examination has failed, or to identify which image layers have failed, or optionally to display in the GUI the corresponding layers and measurements showing a boundary comparison of the failure. The response could also be to output a message requesting the patient to obtain a larger scan. In another example, the response is to have the Automated Imaging Examination Suite (AICS) stop processing additional pre-examinations (e.g., in an ordered series of examinations).
[0096] On the other hand, as a result of this comparison, medical imaging data MID can be determined to be acceptable. For example, in Figure 7 In the example, the shape model implant SSM-I and / or implant measurement IM are captured inside the image boundary MIB. Assuming these are the only types of checks, the automated planning pre-check PPC at step 314 will determine that the image is acceptable. Reasons for acceptability can be characterized as follows: the image includes the required detail of the target anatomical structure TA (e.g., bone) necessary for implant planning; the image is large enough to capture the required portion of the target anatomical structure TA taken into account for the implant, or large enough to capture the planned implant; the portion of the target anatomical structure to which the implant will be positioned is captured within the image view; the portion of the target anatomical structure to which the implant will be positioned is not cropped; and / or the planned implant for the target anatomical structure will not extend beyond the image boundary, or be spaced at a threshold distance from the image boundary.
[0097] If acceptable, the Automated Planning Pre-Check (PPC) at step 318 can automatically generate a response to confirm acceptability. In one example, the response is to send an acknowledgment to the GUI so that technicians can see that the pre-check has passed. In another example, the response is to allow the Automated Image Inspection Suite (AICS) to continue processing additional pre-checks (e.g., in an ordered series of checks). Other responses are envisioned, such as not generating a response unless an error is detected.
[0098] 3A. Anatomical structures / images / implant extrapolation
[0099] As described in Reference Section (2A) above, the Automated Planning Pre-examination (PPC) is similarly equipped with the ability to predictively extrapolate or extend target anatomical structures (TA), implants, and / or image MIDs, such as in Figure 6 As shown at step 320. If it is determined that the required amount of the target anatomical structure TA required for the planned implant is not captured in the image MID, the Automated Image Inspection Suite (AICS) can artificially extend the target anatomical structure TA and / or the implant beyond the original boundary MIB of the image MID. As described above, this technique can be incorporated, and for simplicity, the details are not fully repeated. This extrapolation can be performed on the target anatomical structure, the implant, and / or the image after the comparison in step 310 (which compares the required features of the target anatomical structure to the medical image boundary) or after the comparison in step 312 (which compares the implant measurement to the medical image boundary). After step 310, the target anatomical structure can be extended using a shape model SM, as described above. After 312, the target anatomical structure and / or the implant can be extended using the shape model SM and the implant shape model SSM-I, respectively, as described above.
[0100] This technique helps to prevent images from being rejected and avoids rescanning images that would otherwise fail the pre-planning screening. For example, for primary or revision total knee arthroplasty (TKA), the described extrapolation technique can utilize current CT scan images and extrapolate the additional length required by the anatomical shape model SM and / or implant shape model SSM-I to account for the desired length of the planned stem, such as 100 mm or 200 mm.
[0101] 4. Automated image classification pre-check
[0102] refer to Figure 9The Automated Image Inspection Suite (AICS) performs automated image classification pre-inspection (ICPC). The primary purpose of ICPC is to assess whether a median image detail (MID) represents what the imaging data MID is intended to represent. If the imaging data MID fails to represent the expected content, the imaging data may adversely affect the accuracy and / or completeness of any downstream surgical planning (e.g., segmentation, anatomical model creation, implant planning) or aspects of surgical workflows that rely on surgical planning or medicine (e.g., surgical navigation visualization, anatomical registration, etc.). Automated ICPC efficiently and accurately detects these discrepancies, thereby improving the accuracy and completeness of downstream surgical planning.
[0103] Figure 9 The document demonstrates an example implementation of the Automated Image Classification Pre-Check (ICPC). Figure 9 Some features or aspects shown may be optional and will be described as such. The Automated Image Classification Pre-Check (ICPC) is not strictly limited to the steps described. Some features of the Automated Image Classification Pre-Check (ICPC) can be implemented independently or without other features. For example, the Automated Image Classification Pre-Check (ICPC) can perform classification or prediction without necessarily performing the described pre-checks.
[0104] At step 402, the Automated Image Inspection Suite (AICS) receives the imaging data MID of the target anatomical structure TA. For Figure 9 For example, the imaging data is a volumetric (3D) scan, and more specifically a CT volumetric scan (402a). However, the imaging data is not limited to CT scans or volumetric scans. As described above, the imaging data MID can include any type of image and scan, whether 3D or 2D.
[0105] In addition to CT volume, the imaging data MID further includes data (at 402b), namely, textual or numerical data, metadata, and / or information of any type indicating expected parameters related to the patient, the target anatomical structure TA, and / or the imaging data MID. Expected parameters indicate alleged features, aspects, or conclusions related to the target anatomical structure scanned in the image. Automated Image Classification Pre-screening (ICPC) checks the accuracy of these expected parameters. For example, in Figure 9In this context, the expected parameter data 402b may include: (1) data relating to the expected target anatomical structure allegedly scanned in the image, such as the expected type of joint (e.g., knee, hip, shoulder, etc.) or expected parameters of that joint (e.g., joint geometry, kinematics, dynamics, bone mineral density, disease status); (2) expected images or planning data, such as the expected surgical side of the anatomical structure of the patient to which the alleged scan was obtained, or the expected surgical side on which the patient's surgery is planned (e.g., left joint, right joint, bilateral joint, medial, etc.). (3) Expected procedure, such as whether the procedure is a total joint replacement (e.g., total knee TKA, total hip THA, etc.), a partial joint replacement (e.g., partial knee, partial hip, etc.), a primary surgery or a revision surgery; the expected implant for the patient (e.g., a total knee implant, a partial hip implant, etc.), or the expected manufacturer, model or size of the implant; the expected presence / absence of existing metals, foreign bodies, or implants in the target anatomy; and / or the expected type of existing metals, foreign bodies, or implants in the target anatomy.
[0106] At step 404, the Automated Image Inspection Suite (AICS) optionally converts or generates one or more digitally reconstructed radiographic images (DRRs) from the imaging data MID (e.g., CT volume). Each of the one or more DRRs is a 2D digital analog projection of the CT volume data. At step 404, the Automated Image Inspection Suite (AICS) generates one or more DRRs of a target anatomical structure TA, along with specific features optionally associated with that target anatomical structure TA. For example, the Automated Image Inspection Suite (AICS) may generate one or more DRRs of: (1) the skeletal structure of the target anatomical structure TA; (2) soft tissue (e.g., including skin) on or around the target anatomical structure; and / or (3) metal or foreign bodies located in the target anatomical structure TA. In one instance, DRRs may be generated indiscriminately for each layer of the volume data. Alternatively, the Automated Image Inspection Suite (AICS) may initially identify these features using the shape modeling techniques described above and generate DRRs specifically to capture these initially identified features. Other techniques can also be used (e.g., material density (Hu) classification, voxel analysis, brightness or contrast comparison, region of interest classification, etc.) to quickly identify bones, soft tissues, or metals.
[0107] An automated image inspection suite (AICS) is envisioned that could utilize 2D images (e.g., X-rays or 2D CT slices) to identify specific features related to the target anatomical structure (TA) (e.g., bone, soft tissue, metal, etc.), instead of generating a DRR at step 404. It is also envisioned that 3D volumetric data (such as raw CT volumes) could be utilized (without reducing the volume to a 2D image or DRR). Downsampling of the 3D image data is possible. Although the various steps described below are explained in the context of DRR, it should be understood that these individual steps can be implemented using any other type of imaging data format.
[0108] At step 406, the Automated Image Inspection Suite (AICS) further performs Automated Image Classification Pre-Inspection (ICPC) by automatically applying one or more DRRs to the Machine Learning Model (MLM). The MLM analyzes one or more DRRs to automatically classify or identify objects or features (such as bone, soft tissue, or metal) within those DRRs. The MLM can be configured to detect and classify these objects or features using shape or contour recognition, sign detection, pattern recognition, bounding boxes, or shape modeling techniques described above. In some instances, the MLM can perform image segmentation on the DRRs. Before being input into the MLM, the DRRs can be normalized to provide a consistent format prior to classification.
[0109] In one example, the machine learning model can be a neural network, such as a convolutional neural network (CNN) or an artificial neural network (ANN). To quickly process DRRs, the machine learning model can be configured as a 2D lightweight convolutional neural network. The machine learning model can include any number of convolutional layers and connection layers. When using a convolutional neural network, the architecture can include several nodes organized into layers. The DRR is input to the input layer and filtered through various interconnected processing layers. Based on the training data, the connections between nodes are assigned weight values, which can be adjusted. The output of the neural network is based on the sum of these weight values. Throughout this process, features within the DRR can be progressively segmented, classified, or distinguished.
[0110] Machine learning models (MLMs) can be trained on any medical imaging data, such as medical imaging data related to other patients. For example, training data can be based on imaging data related to one or more of the following: patients with characteristic target anatomy; patients with characteristic surgical procedures; patients with characteristic lesion anatomy or healthy anatomy; patients with procedures of characteristic type; postoperative images of patients with characteristic implant type, size, or manufacturer; patients with revision surgery; patients with primary surgery; patients with characteristic age, sex, ethnic origin, weight, and / or height, etc.
[0111] The machine learning model MLM can also incorporate either the described preoperative patient data or the expected parameters (provided along with the medical imaging data). The machine learning model MLM can do this as a method to accelerate the classification process. For example, if the medical imaging data identifies the medical images as knee images and the planned procedure is total knee replacement, the machine learning model MLM can incorporate this information to adjust the classification algorithm to use training data based on knee-based medical images (e.g., rather than hip-based images). Similarly, other patient data such as age, sex, ethnic origin, weight, and height can be fed into the machine learning model MLM.
[0112] In one example, the convolutional neural network (CNN) employs a model-switching architecture that selects the model used for segmentation based on the input DRR (Diagnosis Related to Restricted Registry). For instance, if the input DRR is a skeletal structure, the CNN can select a model for classifying skeletal structures. If the input DRR is a soft tissue structure, the CNN can select a second model for classifying skeletal structures. If the input DRR is a metallic structure, the CNN can select a third model for classifying metallic structures. This model selection technique increases classification accuracy and speed while reducing computational load. Furthermore, the CNN can be adapted to include additional models beyond those described. For example, if classification of existing implant types is required later, a fourth model for classifying implant types can be trained, and the CNN can select this model based on the DRR representing the implant. In one implementation, for any given imaging dataset, three independent DRRs can be generated for bone, soft tissue, and metal, and the machine learning model (MLM) simultaneously evaluates these three independent DRRs. The MLM can select one model to evaluate one or more DRRs, or select one or more models to evaluate several DRRs. Other types of machine learning models can be used, such as deep learning models configured to classify the contents of medical image data or CT volumes (including anatomical structure type, side, treatment or procedure type, existing implants, etc.).
[0113] At step 408, after analyzing the medical imaging data (or DRR) using the machine learning model MLM, the MLM outputs predicted parameters. The predicted parameters may include textual, numerical, metadata, and / or any type of information indicating a classification prediction related to the patient, the target anatomical structure (TA), and / or the imaging data (MID). The predicted parameters are provided to confirm or refute the expected parameter features, aspects, or conclusions (from step 402b) related to the target anatomical structure scanned in the image and / or derived from the patient data. In other words, the Automated Image Classification Pre-Check (ICPC) uses the predicted parameters to assess whether the expected parameters are accurate.
[0114] The predicted parameters may include: (1) data predicting the target anatomical structure scanned in the image, such as predicting the type of joint (e.g., knee, hip, shoulder, etc.), or predicting parameters of the joint (e.g., joint geometry, kinematics, dynamics, bone density, disease status); (2) prediction of image or planning data, such as predicting the surgical side of the patient for the anatomical structure scanned, or predicting the surgical side of the patient for the planned surgery (e.g., left joint, right joint, bilateral joint, medial, lateral, etc.); (3) prediction of surgical procedure, such as predicting whether the procedure is a total joint replacement (e.g., total knee, total hip, etc.), a partial joint replacement (e.g., partial knee, partial hip, etc.), a primary surgery or a revision surgery; predicting the implant for the patient (e.g., a total knee implant, a partial hip implant, etc.), predicting the manufacturer, model or size of the implant; predicting the presence or absence of existing metals, foreign bodies, or implants in the target anatomical structure; and / or predicting the type of existing metals, foreign bodies, or implants in the target anatomical structure. For example, when medical imaging data is of an anatomical joint, the machine learning model MLM can predict that the joint is a knee joint, the surgical side should be the left knee joint, and no existing metal objects are found in the image of the anatomical joint.
[0115] At step 410, the Automated Image Inspection Suite (AICS) may optionally output the classification / prediction determined at step 408. This output may be generated before performing a comparison with expected parameters, as described in more detail below. Output 410 may be provided for various purposes. For example, output 410 may be provided to provide a classification report of medical imaging data. For example, the classification report may include any of the predictions already described above related to the target anatomical structure or procedure. The classification report may be provided on a data review screen or GUI for review by a technician. For example, a technician may want to review whether the imaging data indicates the presence or absence of an existing implant for the target anatomical structure, and if present, review the predicted manufacturer or model of the existing implant. The report may be provided for clinical or surgical planning purposes. Alternatively, at step 406, output 410 may also be fed into a machine learning model (MLM) to provide additional input to the training data and / or to improve the training data.
[0116] At step 412, the Automated Image Classification Pre-Check (ICPC) automatically compares each expected parameter with its corresponding predicted parameter. For example, it compares the expected type of the target anatomy with its predicted type, or the expected surgical side of the target anatomy with its predicted surgical side. The comparator module can be implemented to receive and organize the expected parameters from the received medical imaging data 402, and to receive and organize the predicted parameters output by the machine learning model (MLM) (from steps 406, 408). The comparator can organize the corresponding expected and predicted parameters into a lookup table. In some cases, the Automated Image Classification Pre-Check (ICPC) may know in advance what the expected parameters are and attempt to obtain specific predicted parameters to confirm / refute the corresponding expected parameters. In other cases, the Automated Image Classification Pre-Check (ICPC) may not know in advance what the expected parameters are. Instead, the Automated Image Classification Pre-Check (ICPC) determines which predicted parameters to output and checks whether the corresponding expected parameters are provided.
[0117] Comparisons can be made by determining whether the corresponding expected parameter and the predicted parameter are exactly the same, match, consistent, substantially similar, or otherwise acceptablely matched. For an acceptable match, the corresponding expected parameter and the predicted parameter can be consistent within an acceptable threshold tolerance. In one example, the threshold tolerance could be greater than a 95% confidence score. For instance, the expected parameter might indicate that the scanned anatomy is the left knee joint, while the corresponding predicted parameter might indicate a 97% confidence score that the scanned anatomy is the left knee joint. Because the prediction is larger than the threshold, the Automated Image Classification Pre-examination (ICPC) can confirm that the intended target anatomy is actually scanned. In other examples, for an acceptable match, the corresponding expected parameter and the predicted parameter must be exactly the same, with a tolerance of zero. For example, the expected parameter might indicate that the surgery is for the right hip joint. For an acceptable match to the prediction, the prediction must indicate a 100% confidence score that the surgery is for the right hip joint. The level of confidence or threshold tolerance can be selectively adjusted depending on the criticality of the examination. In other examples, such specific accuracy can be used to train a machine learning model (MLM) to a level that does not involve confidence or threshold tolerance. For example, the expected parameter could indicate that the target anatomy does not include an existing implant. If the predicted parameter indicates that the target anatomy does not include an existing implant, the result can be considered accurate regardless of the accuracy score. In other examples, if the corresponding expected parameter is missing from the medical imaging data, the comparison may produce erroneous results, but the predicted parameter can still be output at 410. Imagine using any other technique or method not specifically described herein to perform comparisons between any expected parameter and any predicted parameter described above.
[0118] After applying the comparisons described above, the Automated Image Classification Pre-Check (ICPC) assesses whether the medical imaging data MID is acceptable. In other words, the ICPC determines, based on the output predictions, whether the medical imaging data MID represents what it is intended to represent.
[0119] Medical imaging data (MID) can be approved in response to determining that any expected parameter is acceptablely matched with its corresponding predicted parameter. Approval may depend on the acceptable matching of several expected parameters with their corresponding predicted parameters. For example, approval may require that the predicted joint type, predicted surgical side, and predicted procedure type match the corresponding expected joint type, expected surgical side, and expected procedure type. Automated Image Classification Pre-screening (ICPC) can be configured to select which parameters or groups of parameters must be acceptablely matched.
[0120] If acceptable, the Automated Image Classification Pre-Check (ICPC) determines, based on the output predictions, that the Medical Imaging Data's Mid-ID (MID) represents what it is intended to represent. At step 414, the ICPC can automatically generate a response to confirm acceptability. In one example, the response is to send confirmation to the GUI so that a technician can see that the pre-check has passed. In another example, the response is to allow the Automated Image Inspection Suite (AICS) to continue processing additional pre-checks (e.g., in an ordered series of checks). Other responses are envisioned, such as not generating a response unless a classification difference is detected.
[0121] On the other hand, in response to the determination that any one or more of the expected parameters and their corresponding predicted parameters fail to match acceptablely, the Medical Imaging Data MID may be rejected. If rejected, the Automated Image Classification Pre-examination (ICPC) determines, based on the output predictions, that the Medical Imaging Data MID fails to represent what it is intended to represent. If, as a result of the Automated Image Classification Pre-examination (ICPC), the Medical Imaging Data MID is determined to be unacceptable, a response can be automatically generated at step 416 to confirm the unacceptability. In one example, the response is to send an alert or notification to the GUI so that the technician can see that the pre-examination has failed, or to identify which expected parameters have failed and why, or optionally to display the predicted features from the DRR in the GUI. The response can also be to output a message requesting the patient to obtain another scan or to review preoperative patient data. In another example, the response is to have the Automated Imaging Examination Suite (AICS) stop processing additional pre-examinations (e.g., in an ordered series of examinations).
[0122] 5. Automated motion (lever) inspection
[0123] Another pre-inspection that can be performed by the Automated Image Inspection Suite (AICS) is Automated Motion Inspection (MRC), such as... Figure 1 As shown in and as Figure 10 Method 500 describes motion testing. MRC automatically evaluates medical imaging data MID to determine whether the patient moved during the scan. Figure 10 As shown in the example medical image layer, a motion bar (MR) exists in the image along with the target anatomical structure (TA). The motion bar MR is included as part of the scanning protocol and is a physical rod attached to a patient's anatomical structure or limb (e.g., leg) during the scan to keep that structure stationary. For example, in preparation for a scan targeting the knee joint, the motion bar MR may be strapped to the patient's leg, extending from the hip region to the knee region and down to the ankle region. The motion bar MR is radiopaque and visible within the scanned image. During the scan, the motion bar MR should remain stationary to ensure scan accuracy. If movement of the bar MR is detected, this indicates that the patient has moved during the scan, which will require rejection or a repeat scan.
[0124] Accordingly, in order to achieve automated motion detection (MRC), Figure 10Example method 500 includes step 502 of receiving imaging data MID comprising a target anatomical structure TA and a moving rod MR. For this examination, the received imaging data MID is advantageously provided as a 3D CT volume. However, in other implementations, the imaging data MID may be a CT slice. At step 504, the automated motion examination MRC may automatically identify the moving rod MR within the 3D volume of the medical imaging data MID. This process may involve the automated motion examination MRC receiving known parameters of the moving rod MR, such as rod diameter, radius, length, density, etc. The automated motion examination MRC may additionally or alternatively (with or without known information about the moving rod) utilize object detection algorithms or machine learning algorithms to detect the moving rod MR. The machine learning algorithm may be trained on the dataset to automatically distinguish the moving rod MR from the target anatomical structure TA or other objects such as existing implants. At step 504, once the moving rod MR has been initially identified, the automated motion examination MRC automatically assesses the entire 3D volume of the medical imaging dataset to determine whether the moving rod MR is visible. Step 506 can be performed simultaneously with step 504 (i.e., within a single timeframe). If layers are used, step 506 may include a layer-by-layer assessment of the visibility of the motion bar. To determine whether the motion bar MR is visible, automated motion examination MRC can assess voxels or utilize object detection or machine learning algorithms to detect the presence of the motion bar MR within a 3D volume. Similarly, if available, the process may involve known parameters of the motion bar MR. Additionally, automated motion examination MRC can create a shape model (e.g., a cylinder or circle) to fit the motion bar MR or its cross-section (if layers are used). For volumetric analysis, automated motion examination MRC can check for the presence of a complete cylinder within a 3D volume that matches the scanned motion bar parameters. A complete cylinder indicates that the motion bar MR is stationary and therefore the volume was taken when the patient was not moving. If a geometry other than a complete cylinder is identified within the volume (e.g., a partial cylinder, or an ambiguous geometry), the automated motion detection MRC automatically identifies that the motion lever MR has moved during the scan and therefore takes the scan when the patient moves. If slices are used, the automated motion detection MRC can check if a complete circle exists in the slice at the location of the motion lever MR. A complete circle indicates that the motion lever MR is stationary and therefore the slice is taken when the patient is not moving. If a geometry other than a complete circle is identified within a particular slice (e.g., a partial circle, or an ambiguous geometry), the automated motion detection MRC automatically identifies that the motion lever MR has moved during the scan and therefore takes that slice when the patient moves.
[0125] At step 508, as a result of the evaluation of the medical imaging data MID for the motion bar MR, the automated motion examination MRC automatically determines whether the evaluation results in an acceptable value falling within a predetermined range or threshold. For example, acceptability may mean that the volume or each layer exhibits a fully (100%) visible motion bar MR. Alternatively, acceptability may mean that the volume or each layer exhibits a motion bar MR that is visible above a threshold (e.g., greater than 95%). An unacceptable result may mean that the volume or at least one layer fails to exhibit a fully visible motion bar MR. Alternatively, an unacceptable result may mean that the volume or at least one layer fails to exhibit a motion bar MR that is visible above a threshold. The thresholds for acceptability and unacceptability may be the same or different.
[0126] If the assessment is acceptable, the Automated Motion Inspection MRC can automatically generate a response at step 510 to confirm acceptability. In one example, the response is to send an confirmation to the GUI so that the technician can see that the pre-examination has passed, or that the scan was correctly obtained without patient movement. In another example, the response is to allow the Automated Imaging Inspection Suite (AICS) to continue processing additional pre-examinations (e.g., in an ordered series of examinations). Other responses are envisioned, such as not generating a response unless an error is detected.
[0127] If the assessment is unacceptable, the Automated Motion Examination MRC can automatically generate a response regarding the unacceptability at step 512. In one example, the response is to send an alert or notification to the GUI so that the technician can see that the pre-examination has failed, or be able to identify which layer (if applicable) failed. In another example, the response is to have the Automated Imaging Examination Suite (AICS) stop processing additional pre-examinations (e.g., in an ordered examination series). The response may also output a message requesting the retrieval of a new scan if the patient is motionless.
[0128] 6. Automated capacity classifier and side pre-check
[0129] Another module that can be implemented by the Automated Image Inspection Suite (AICS) is the Automated Capacity Classifier and the Side Pre-inspection or Classifier (VLC), such as... Figure 1 As shown in and as referenced Figure 11 And as described in Method 600. In addition to any of the described pre-checks, a VLC pre-check may be included, or a VLC pre-check may be used as a sub-feature of any of the described pre-checks.
[0130] VLC pre-examination includes a pipeline that can identify the volume type contained within a given 3D image volume of medical imaging data (MID). VLC pre-examination can also identify the laterality of unilateral volumes of anatomical structures. Based on this pipeline, VLC pre-examination outputs a final label identifying the volume type and laterality.
[0131] refer to Figure 11 At step 602, the medical imaging data MID is input into the VLC pre-examination. The medical imaging data MID can be a 3D volume, such as a 3D CT volume from the DICOM series. The medical imaging data MID can be any type of anatomical structure, including but not limited to: knee joint, ankle joint, hip joint, shoulder joint, spine, etc. In one example, the CT volume is unclassified before being input into the VLC pre-examination. In other words, the CT volume may not include information to identify the type of anatomical volume being imaged, nor the lateral aspect of the imaged anatomical volume.
[0132] Medical imaging data (MID) can be input into a volume type classifier (VTC) implemented as the first part of the VLC pre-examination pipeline. The VTC can be implemented as a deep learning model that computes the probability that a 3D volume is one of several volume types. Volume types include, but are not limited to: unilateral hip, unilateral knee, unilateral ankle, bilateral hip, bilateral knee, or bilateral ankle. The most probable term among these is considered the output of the volume type classifier. For example, as... Figure 11 As shown, the output of the Volume Type Classifier (VTC) is an identifier for medical imaging data (MID) containing a unilateral knee joint. Confidence values can be logged or presented on a GUI or in a report. Here, the VTC produces a confidence score of 0.9998 for a unilateral knee joint. In one example, if the confidence score is higher than a threshold, such as greater than 95%, 98%, or 99%, the VTC can output the result at 604 to the next step in the VLC pre-examination pipeline, namely the lateral or lateral classifier (LC). Alternatively, the highest confidence score can be considered the output of the VTC.
[0133] In one implementation, the volume type classifier (VTC) can only output the result to the lateral classifier (LC) in response to classifying the volume integral as a "unilateral" skeletal structure (hip, knee, ankle, etc.). This is because unilateral skeletal structures exhibit higher sensitivity to lateralization confusion when compared to bilateral structures because there is no corresponding left or right side of the anatomical structure to provide a comparison. In such cases, if the volume type classifier (VTC) classifies the volume integral as a "bilateral" skeletal structure, then at position 605, the volume type classifier (VTC) can bypass the lateral classifier (LC) and at position 606, output the result directly to the final labeled output. Of course, it is conceivable that the volume type classifier (VTC) could also output the result to the lateral classifier (LC) in response to classifying the volume integral as a "bilateral" skeletal structure.
[0134] In one example, the deep learning model used for the volumetric type classifier (VTC) is the densely connected convolutional network architecture DenseNet-264. Training images are rescaled to preserve the aspect ratio of the initial CT images, and voxel intensities are cropped to a specific range of Henle units. During each training epoch, these images are randomly sampled based on their labels (“left hip” or “right hip”) to ensure the model is trained on approximately equal numbers of each type of image. Voxel intensity scaling is applied to compensate for variability in the CT scanner and its calibration. Affine transformations (shearing and rotation) are applied to the training images to compensate for changes in patient position within the scanner, and scaling is included to compensate for variability in patient height and weight. Other types of machine learning models (such as convolutional neural networks, random forest classifiers, decision tree classifiers, K-nearest neighbor classifiers, Naive Bayes classifiers, support vector machines, etc.) can be used to classify medical image data or the contents of CT volumes.
[0135] To implement the lateral classifier LC, VLC pre-examination can include a CT hip lateral view classifier, which can be implemented as a deep learning model, calculating the probability that the 3D volume belongs to the left or right hip joint. The most probable term is considered the output of the classifier. VLC pre-examination can also include a CT knee lateral view classifier, which can be implemented as a deep learning model, calculating the probability that the 3D volume belongs to the left or right knee joint. The most probable term is considered the output of the classifier. VLC pre-examination can also include a CT ankle lateral view classifier, which can be implemented as a deep learning model, calculating the probability that the 3D volume belongs to the left or right ankle joint. The most probable term is considered the output of the classifier.
[0136] These lateral classifiers (LCs) can be implemented individually or combined into a single classifier. For example... Figure 11As shown, when the lateral classifier LC is separated, the output from the volume type classifier VTC is directly input into the corresponding lateral classifier LC associated with the classified volume. For example, in Figure 11 In this model, the output of the volume type classifier (VTC) is for a single knee joint. Therefore, the VLC pre-examination has successfully identified the type of anatomical structure in the image, but has not yet identified the lateral aspect of the knee joint. Accordingly, the output of the volume type classifier (VTC) can be directly fed into a CT knee lateral view classifier, which is specifically trained to detect the lateral aspect of the knee joint in the imaging volume. This design choice of dividing the lateral classifier by bone type reduces computation time and enables the processing of large amounts of imaging data.
[0137] like Figure 11 As shown, the output of the lateral classifier LC is the lateral identifier of the anatomical structure from the medical imaging data MID. Confidence values can be logged or presented on a GUI or in a report. In the example shown, the lateral classifier LC produces a confidence score of 0.9995 indicating the knee joint as the right knee joint. In one example, if the confidence score is higher than a threshold, such as greater than 95%, 98%, or 99%, the lateral classifier LC can output the result to the final labeled output at 606. Alternatively, the highest confidence score can be considered as the output of the lateral classifier LC. For example, if the confidence score for the left knee joint is 0.05 and the confidence score for the right knee joint is 0.95, the lateral classifier LC can consider the right knee joint as the correct lateral identifier for the final labeled output at 606. The final labeled output 606 (e.g., "right knee joint") can be provided on a report or GUI, such as those shown and described herein.
[0138] In one example, the deep learning model used for any one or more of the lateralization classifiers (LC) is the densely connected convolutional network architecture DenseNet-121. Training images are rescaled to preserve the aspect of the initial CT images, and voxel intensities are cropped to a specific range of Henle units. During each training epoch, these images are randomly sampled based on their labels (“left hip” or “right hip”) to ensure the model is trained on approximately equal numbers of each type of image. Voxel intensity scaling is applied to compensate for variability in the CT scanner and its calibration. Affine transformations (shearing and rotation) are applied to the training images to compensate for changes in patient position within the scanner, and scaling is included to compensate for variability in patient height and weight. Other types of machine learning models (such as convolutional neural networks, random forest classifiers, decision tree classifiers, K-nearest neighbor classifiers, Naive Bayes classifiers, support vector machines, etc.) can be used to classify the lateralization of medical image data or CT volumes.
[0139] VLC pre-screening can use the labeled output 606 to perform automated checks to determine whether medical imaging data (MID) is acceptable. This check can be performed automatically in several ways. For example, VLC pre-screening can automatically assess the confidence scores for output type and side to determine if those scores are above a threshold. If either value is below the threshold, the check can output a failure result. If both values are above the threshold, the check can output a pass result. Additionally, outputs from other pre-screenings from AICS can be used to feed the confidence scores from the VLC pre-screening. In some instances, the volume type and side outputs can be presented for manual review by technicians.
[0140] B. Example Planning Workflow
[0141] Return to reference Figure 1 If the Automated Image Inspection Suite (AICS) outputs a successfully approved Medical Imaging Data Identifier (MID) after performing one or more of the described pre-inspections, the treatment planning request (including the approved MID) can be passed downstream for use in other procedures or processes involving surgical planning. For example, at the SG, an automated or manual segmentation procedure can be initiated based on the approved MID. At the MC, an anatomical model or virtual model can be created based on the segmented image data or the approved MID. At the SP, surgical planning can be implemented against the approved MID or the anatomical or virtual model. Surgical planning can be implant planning and can be based on any of the predicted parameters described above. At the RPL, after the surgical planning is completed, the treatment planning request can be returned to the appropriate requester. After the planning request is returned, the approved MID, using the techniques described herein, can be further relied upon for surgical workflow purposes.
[0142] exist Figure 12This document provides another example workflow involving the Automated Image Inspection Suite (AICS). In this example, a treatment planning request is generated after the medical imaging data (MID) has been initially processed by the AICS. For example, a medical image from a PACS server is initially input into the AICS. This input can be automated and uploaded to the server running the AICS. The AICS then performs one or more of the automated image pre-examinations described herein. If the automated image pre-examinations are unsuccessful, for example, one or more of these examinations have failed, the AICS will automatically reject the scan. Feedback can be automatically provided to the appropriate representative or technician, as described below. The feedback can be provided in a GUI and can include detailed reports summarizing the reasons for the rejection, recommendations on how to correct the scan, instructions on requesting a rescan, etc. If the automated image pre-examinations are successful, the workflow continues to create a treatment planning request. Knowing that the medical imaging has been initially pre-examined and is therefore initially suitable for planning purposes, the appropriate representative or technician can then create a treatment planning request.
[0143] Continue to refer to Figure 12After a treatment plan is created, the plan, which includes medical image data, can be re-entered into the Automated Image Inspection Suite (AICS). The AICS again performs the "post-planning" processing of any one or more of the automated pre-screenings described herein. Here, the screenings may be the same as or different from those performed before the treatment plan request was created. If the (post-planning) automated pre-screening is unsuccessful—for example, one or more of these screenings have failed—a representative or technician from the segmentation team can be automatically informed of a rejection, and the output can be reviewed. Data review may include reviewing the rejection report, identifying potential problems, and implementing corrective actions. Feedback can be similarly provided to the appropriate segmentation representative or technician via a GUI to facilitate data review. On the other hand, if the (post-planning) automated pre-screening is successful, the workflow continues with automated pre-segmentation of the target anatomical structure (TA) in the medical imaging data. Automated pre-segmentation may perform coarse segmentation of the target anatomical structure (TA), which will later be fine segmented by a technician. Alternatively, automated pre-segmentation may perform full segmentation, which will later be reviewed by a technician. At this step, automated pre-segmentation can be robustly performed based on the confidence level that the medical imaging has passed two automated pre-examination iterations, and is therefore suitable for segmentation purposes. Based on the automated pre-segmentation, a representative or technician from the segmentation team can be informed of the segmentation output and can conduct a data review of the output. Data review may include examining the segmentation accuracy, the pre-examination data, identifying potential problems, and implementing corrective actions. If the data review process is successful, the segmentation output can be improved (automatically or manually by the segmentation technician). If the data review process produces a negative result, the case can be rejected, with an optional case note indicating the reasons for rejection.
[0144] C. Graphical User Interface Output
[0145] refer to Figures 13 to 15 Additional features / outputs / interfaces that can be utilized with the Automated Image Inspection Suite (AICS) are provided for the GUI. Figure 13 and Figure 14 The document provides a sample GUI screen demonstrating a Report RP that can be automatically generated by the Automated Image Inspection Suite (AICS). Such reports can be generated regardless of whether the results of one or more of the pre-inspections are acceptable or unacceptable. The Report RP can be automatically generated for the results of any one or more of the steps 110, 112, 210, 212, 316, 318, 414, 416, 510, 512, 604, and 606 described above for various pre-inspections. For example, in response to... Figure 1 and Figure 12The pre-planning or post-planning review performed by AICS as described in the workflow can also automatically generate a Report RP. Report RPs can provide various types of information to assist technicians in performing image reviews, such as preparing for segmentation or planning purposes. Information about a Report RP may include, for example, any of the following: the data that generated the report; a report overview, including the specific reasons for rejection when compared to acceptable thresholds (if applicable); recommendations on how to correct the scan; recommendations for rescanning; an image or representation of a specific layer of the medical imaging data MID (such as those layers that may have already led to rejection); attributes of the image or target anatomy, such as volume type, size, spacing, height; patient name; patient's DOB; name of the requesting surgeon; the expected type of procedure (e.g., THA, TKA); an overview of examinations performed or not performed by AICS, and whether such examinations were passed or failed, and the reasons; confidence or accuracy scores for any examination; recommendations for modifications to the segmentation or planning process, etc.
[0146] For example, a failure in the report could be explained as a scan being rejected because the anatomical structure region was not included in the image. The report could provide recommendations to correct the problem, such as having the scanning facility re-burn an optical disc containing all included regions. Additionally, the report could include predictions or suggestions to identify the root cause of the problem, such as: the observed pixel size of the anatomical structure is smaller than the expected pixel size; the observed slice thickness is not within the expected range; the slice spacing is inconsistent with the protocol (e.g., the observed slice spacing is 2.5 mm, but the maximum slice spacing must be less than 1.1 mm, with no gaps or overlaps). These parameters can be checked, for example, by an automated image parameter checker such as IPPC, as referenced. Figure 3 Shown and described.
[0147] For example, as in Figure 14 As shown, the additional information provided in the report includes imaging acquisition parameters or DICOM tag information, such as modality (e.g., CT), number of samples per pixel (e.g., 1), bit depth (e.g., 16), gantry tilt (e.g., 0), patient image orientation (e.g., 1\0\0\0\1\0), etc. The report can provide the image's square resolution, volumetric height, number of slices, pixel size in X and Y, slice thickness, and Z-spacing. These parameters can be checked, for example, by an automated image parameter checker (IPPC), such as a reference. Figure 3 Shown and described.
[0148] refer to Figure 15The dashboard (DB) can be implemented via a GUI to provide organization and management of patient case files. The DB can be dynamically updated with various information about the case files, such as patient name, the layer number of the medical imaging data (MID), surgeon's name, etc. Buttons can be provided to provide immediate access, download, or retrieval of the medical imaging data (MID) for viewing. Advantageously, the DB is provided with status checks. These status checks are dynamically updated based on the continuous evaluation of medical imaging data across various case files using the Automated Image Inspection Suite (AICS). Status checks can include icons graphically presented on the GUI and are dynamically updated based on status changes. For example, icons can indicate that one or more inspections are being processed, completed, failed, or passed. Icons can be checkmarks (if successful), crosses (if failed), or loading graphs (to indicate that an inspection is in progress). AICS can be integrated with the dashboard, for example, via API or other means, to provide dynamic status updates for each case file. Additionally, AICS can automatically load relevant annotations regarding status, such as whether one or more examinations are being processed, completed, failed, or passed, as well as any information available in the Report RP described above, such as: the specific reason for rejection compared to an acceptable threshold (if applicable); recommendations on how to correct the scan; recommendations for rescanning; images or representations of specific layers in the Medical Imaging Data MID, such as layers that may have led to rejection; an overview of examinations performed or not performed by AICS; the confidence or accuracy score for any examination; recommendations for modifications to the segmentation or planning process, etc. Furthermore, the Dashboard DB can provide direct links to the corresponding case files to access / download / view any Report RP, as described above. Moreover, between the Report RP and the Dashboard, the system provides an exceptionally user-friendly imaging examination management level that offers immediate insight into the automated imaging examination activities performed by AICS.
[0149] D. Example technical solutions and advantages
[0150] As is evident from the detailed description and accompanying figures, the pre-examination provided by the Automated Image Inspection Suite (AICS) offers significant advantages and technical solutions by automatically, rapidly, and accurately identifying defects or errors in imaging data mid-injection (MID). In doing so, the AICS can automatically determine whether the MID is suitable for downstream post-processing and / or surgical planning (e.g., segmentation, anatomical model creation, implant planning) or for various aspects of surgical workflows that rely on surgical planning or medical imaging (e.g., surgical navigation visualization, anatomical registration, etc.). The AICS can process the described pre-examination almost instantly within seconds, thereby reducing the time-consuming burden on technical teams to manually review imaging data mid-injection to identify potential errors. Furthermore, the AICS can significantly reduce the labor costs and human error involved in manually reviewing imaging data. The AICS can be advantageously performed before and / or after the creation of treatment request plans, thus providing additional confidence prior to planning and segmentation. By performing automated inspections early in the workflow (e.g., before treatment planning requests or segmentation), the amount of manual inspections performed by technicians is significantly reduced, freeing up technicians' time to handle a significantly larger number of cases. Additionally, automated inspections can provide feedback earlier in the process, avoiding unnecessary delays and wasted downstream resources. The Automated Image Inspection Suite (AICS) can perform automated scan rejections and provide automated reports and overviews to quickly help technicians process large volumes of patient data, resulting in significant improvements in reducing human error and labor costs. Furthermore, by providing automated feedback (e.g., through dashboard reports), AICS and the GUI can provide a summary overview for educating scanning technicians to potentially reduce image rejection rates in the future. Additionally, the AICS suite can process and classify unclassified images without relying on pre-classification performed by technicians or radiologists, which can be prone to human error. Moreover, the classifier can be specifically trained using imaging datasets for certain types of anatomical structures (e.g., knee, hip, ankle joints), providing a fast and robust method for processing large numbers of medical images. Other advantages not specifically described herein will readily be appreciated by those skilled in the art.
[0151] Several embodiments have been discussed in the foregoing description. However, the embodiments discussed herein are not intended to be exhaustive or to limit the invention to any particular form. The terminology used is intended to be descriptive rather than restrictive in nature. In view of the foregoing teachings, many modifications and variations are possible, and the invention may be practiced in ways other than those specifically described.
[0152] Many features and advantages of the invention will be apparent from the detailed description, and therefore the appended claims are intended to cover all such features and advantages that fall within the true spirit and scope of the invention. Furthermore, since many modifications and variations will readily occur to those skilled in the art, it is not intended to limit the invention to the precise constructions and operations shown and described, and accordingly, all suitable modifications and equivalents falling within the scope of the invention may be adopted.
Claims
1. An automated image review suite configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for said target anatomical structure, the automated image review suite comprising a non-transitory computer-readable medium including instructions configured, said instructions, when executed by one or more processors, to: Perform automated checks to determine: Whether the medical imaging data was scanned according to acceptable configuration settings; Whether the target anatomical structure in the medical imaging data is acceptablely captured within the boundaries of the medical imaging data; Does the medical imaging data show a moving rod visible above a visibility threshold level? Whether the medical imaging data can acceptablely represent the expected type of target anatomical structure and the expected lateralization of the target anatomical structure; and In response to determining that any one or more of the automated examinations produce unacceptable results, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
2. The automated image examination suite of claim 1, wherein, in order to determine whether the medical imaging data was scanned according to the acceptable configuration settings, the instructions, when executed by one or more processors, are configured to: Automatically obtain one or more configuration settings that define how the medical imaging data is scanned by the imaging device; Automatically compare the one or more configuration settings with one or more acceptable configuration settings; and In response to determining that one or more configuration settings do not correspond to one or more acceptable configuration settings, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
3. The automated image inspection suite of claim 2, wherein the instructions are configured, when executed by the one or more processors, to: In response to determining that the one or more configuration settings correspond to the one or more acceptable configuration settings, the medical imaging data is automatically approved as acceptable for facilitating surgical planning.
4. The automated image inspection suite of claim 1, wherein, in order to determine whether the target anatomical structure in the medical imaging data is acceptablely captured within the boundaries of the medical imaging data, the instructions, when executed by one or more processors, are configured to: Automatically identify shape models and fit the shape models to the target anatomical structures in the medical imaging data; The features of the shape model are automatically compared with the boundaries of the medical imaging data; as well as In response to determining that the features of the shape model exceed the boundaries of the medical imaging data, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
5. The automated image inspection suite of claim 1, wherein, in order to determine whether the target anatomical structure in the medical imaging data is acceptablely captured within the boundaries of the medical imaging data, the instructions, when executed by one or more processors, are configured to: The medical imaging data is automatically compared with a statistical cohort of medical imaging data that includes other anatomical structures to identify anatomical landmarks of the target anatomical structure that are visible in the medical imaging data and can be compared with the target anatomical structure. The medical imaging data is automatically evaluated to determine whether the anatomical landmarks are visible in the medical imaging data; as well as In response to determining that the anatomical landmarks are not visible in the medical imaging data, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
6. The automated image inspection suite of claim 1, wherein, in order to determine whether the medical imaging data exhibits a motion bar visible above the visibility threshold level, the instructions, when executed by one or more processors, are configured to: The motion rod in the volume of the medical imaging data is automatically identified; Automatically assess the volume of the medical imaging data to determine whether the volume of the medical imaging data can acceptably represent the motion lever; and In response to determining that the volume of the medical imaging data fails to unacceptably represent the motion lever, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
7. The automated image inspection suite of claim 6, wherein, in order to automatically identify the motion lever, the instructions are configured, when executed by one or more processors, to: Implement an object detection algorithm or machine learning model to automatically identify the moving rod in the volume and distinguish the moving rod from other features exhibited in the volume.
8. The automated image inspection suite of claim 7, wherein, in order to automatically assess the volume of the medical imaging data to determine whether the volume acceptablely represents the motion lever, the instructions, when executed by one or more processors, are configured to: Implement the object detection algorithm or the machine learning model to automatically determine whether the moving rod exhibits a complete cylinder in the volume.
9. The automated image examination suite of claim 1, wherein, in order to determine whether the medical imaging data acceptably represents the expected type and the expected laterality of the target anatomy, the instructions, when executed by one or more processors, are configured to: The medical imaging data is received as input, wherein at the time of input, the type and laterality of the target anatomical structure in the medical imaging data are unclassified; The type of the target anatomical structure in the medical imaging data is automatically classified using a first machine learning model; A second machine learning model is selected using the classified type of the target anatomical structure, the second machine learning model being specifically trained to classify the lateral aspect of the classified type of the target anatomical structure; as well as The second machine learning model is used to automatically classify the lateral aspect of the target anatomical structure in the medical imaging data.
10. The automated image inspection suite of claim 9, wherein the instructions are configured, when executed by one or more processors, to: Generate a confidence score, which indicates the classification accuracy of the type of the target anatomical structure in the medical imaging data; and In response to the determination that the confidence score fails to meet an acceptable threshold, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
11. The automated image inspection suite of claim 9, wherein the instructions are configured, when executed by one or more processors, to: Generate a confidence score, which indicates the classification accuracy of the lateral aspect of the target anatomical structure in the medical imaging data; and In response to the determination that the confidence score fails to meet an acceptable threshold, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
12. A computer-implemented method for automatically assessing whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for said target anatomical structure, said computer-implemented method comprising: Perform automated checks to determine: Whether the medical imaging data was scanned according to acceptable configuration settings; Whether the target anatomical structure in the medical imaging data is acceptablely captured within the boundaries of the medical imaging data; Does the medical imaging data show a moving rod visible above a visibility threshold level? Does the medical imaging data acceptablely show the expected type of target anatomical structure and the expected lateralization of the target anatomical structure? as well as In response to determining that any one or more of the automated examinations produce unacceptable results, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
13. A non-transitory computer-readable medium including instructions configured to automatically evaluate medical imaging data of a target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: The medical imaging data is received as input, wherein at the time of input, the type and laterality of the target anatomical structure in the medical imaging data are unclassified; The first machine learning model is used to automatically classify the type of the target anatomical structure in the medical imaging data; A second machine learning model is selected based on the classified type of the target anatomical structure, the second machine learning model being specifically trained to classify the lateral aspects of the classified type of the target anatomical structure; The second machine learning model is used to automatically classify the lateral aspect of the target anatomical structure in the medical imaging data; as well as The computer generates output to identify the classified type and classified laterality of the target anatomical structure in the medical imaging data.
14. The non-transitory computer-readable medium of claim 13, wherein the medical imaging data is a 3D CT volume and the target anatomical structure is an anatomical joint, and wherein the instructions are configured, when executed by one or more processors, to: The first machine learning model is used to automatically classify the type of anatomical joint in the 3D CT volume into one of the following: unilateral hip joint, unilateral knee joint, unilateral ankle joint, bilateral hip joint, bilateral knee joint, or bilateral ankle joint.
15. The non-transitory computer-readable medium of claim 14, wherein the instructions are configured, when executed by one or more processors, to: Generate a confidence score, which indicates the classification accuracy of the type of the anatomical joint in the 3D CT volume; The confidence score is compared with a threshold. In response to the confidence score exceeding the threshold, the anatomical joints of the type are automatically classified; as well as The second machine learning model is automatically selected using the classified types of anatomical joints.
16. The non-transitory computer-readable medium of claim 14, wherein the instructions are configured, when executed by one or more processors, to: Multiple confidence scores are generated, which indicate the accuracy of classifying the type of the anatomical joint in the 3D CT volume as each of the following: unilateral hip joint, unilateral knee joint, unilateral ankle joint, bilateral hip joint, bilateral knee joint, or bilateral ankle joint. The highest confidence score among the multiple confidence scores is identified. The anatomical joints of this type are automatically classified based on the highest confidence score. as well as The second machine learning model is automatically selected using the classified types of anatomical joints.
17. The non-transitory computer-readable medium of claim 14, wherein the instructions are configured, when executed by one or more processors, to: The second machine learning model is automatically selected only in response to the automatic classification of the type of anatomical joint in the 3D CT volume into one of the following: unilateral hip joint, unilateral knee joint, or unilateral ankle joint.
18. The non-transitory computer-readable medium of claim 17, wherein the instructions are configured, when executed by one or more processors, to: In response to automatically classifying the type of the anatomical joint in the 3D CT volume as a unilateral hip joint, a second machine learning model specifically trained to classify whether the side of the unilateral hip joint is a left or right hip joint is selected. or In response to automatically classifying the type of the anatomical joint in the 3D CT volume as a unilateral knee joint, a second machine learning model specifically trained to classify whether the side of the unilateral knee joint is a left or right knee joint is selected. or In response to automatically classifying the type of the anatomical joint in the 3D CT volume as a unilateral ankle joint, a second machine learning model specifically trained to classify whether the unilateral ankle joint is a left or right ankle joint is selected.
19. The non-transitory computer-readable medium of claim 18, wherein the instructions are configured, when executed by one or more processors, to: Generate a confidence score, which indicates the classification accuracy of the lateral aspect of the anatomical joint in the 3D CT volume; The confidence score is compared with a threshold. In response to the confidence score exceeding the threshold, the laterality of the anatomical joint is automatically classified; as well as The computer-generated output identifies the classified lateral anatomical joints automatically.
20. The non-transitory computer-readable medium of claim 18, wherein the instructions are configured, when executed by one or more processors, to: A first confidence score is generated, which indicates the accuracy of the classification that the laterality of the anatomical joint is a left-sided joint; A second confidence score is generated, which indicates the accuracy of the classification that the lateral aspect of the anatomical joint is a right-sided joint; The highest confidence score between the first confidence score and the second confidence score is identified. The laterality of the anatomical joint is automatically classified based on the highest confidence score; as well as The computer-generated output identifies the classified lateral anatomical joints automatically.
21. The non-transitory computer-readable medium of claim 13, wherein the instructions, when executed by one or more processors, are configured to automatically assess whether the medical imaging data of the target anatomy is acceptable for facilitating surgical planning for the target anatomy by being configured to: Generate a confidence score, which indicates the classification accuracy of the type of the target anatomical structure in the medical imaging data; and In response to the determination that the confidence score fails to meet an acceptable threshold, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
22. The non-transitory computer-readable medium of claim 13, wherein the instructions, when executed by one or more processors, are configured to automatically assess whether the medical imaging data of the target anatomy is acceptable for facilitating surgical planning for the target anatomy by being configured to: Generate a confidence score, which indicates the classification accuracy of the lateral aspect of the target anatomical structure in the medical imaging data; and In response to the determination that the confidence score fails to meet an acceptable threshold, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
23. The non-transitory computer-readable medium of claim 13, wherein each of the first machine learning model and the second machine learning model is a deep learning model comprising a convolutional neural network.
24. A computer-implemented method for automatically evaluating medical imaging data of a target anatomical structure, the computer-implemented method comprising: The medical imaging data is received as input, wherein at the time of input, the type and laterality of the target anatomical structure in the medical imaging data are unclassified; The first machine learning model is used to automatically classify the type of the target anatomical structure in the medical imaging data; The classified type of the target anatomical structure is used to select a second machine learning model, which is specifically trained to classify the lateral aspects of the classified type of the target anatomical structure. The second machine learning model is used to automatically classify the lateral aspect of the target anatomical structure in the medical imaging data; as well as A computer-generated output is generated, which identifies the classified type and laterality of the target anatomical structure in the medical imaging data.
25. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: Automatically identify shape models and fit the shape models to the target anatomical structures in the medical imaging data; The features of the shape model are automatically compared with the boundaries of the medical imaging data; as well as In response to determining that the features of the shape model exceed the boundaries of the medical imaging data, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
26. The non-transitory computer-readable medium of claim 25, wherein the features of the shape model include the boundaries of the shape model, and wherein the instructions, when executed by the one or more processors, are further configured to: Compare the boundary of the shape model with the boundary of the medical imaging data; and In response to determining that the boundary of the shape model exceeds the boundary of the medical imaging data, the medical imaging data is rejected as unacceptable for facilitating surgical planning.
27. The non-transitory computer-readable medium of claim 25, wherein the features of the shape model include one or more points located on the boundary of the shape model, and wherein the instructions, when executed by the one or more processors, are further configured to: Compare the one or more points of the shape model with the boundary of the medical imaging data; and In response to determining that one or more points of the shape model exceed the boundary of the medical imaging data, the medical imaging data is rejected as unacceptable for facilitating surgical planning.
28. The non-transitory computer-readable medium of claim 27, wherein the one or more points on the shape model correspond to predetermined anatomical landmarks of the target anatomical structure.
29. The non-transitory computer-readable medium of claim 25, wherein the instructions are further configured, when executed by the one or more processors, to: In response to determining that the features of the shape model are within the boundaries of the medical imaging data, the medical imaging data is automatically approved as acceptable for facilitating surgical planning.
30. The non-transitory computer-readable medium of claim 25, wherein the instructions, when executed by the one or more processors, automatically generate output in response to rejecting the medical imaging data as unacceptable by being configured to: Generate a notification or alert to the graphical user interface to inform that the medical imaging data has been rejected; Display the medical imaging data and the shape model on a graphical user interface to show that the features of the shape model extend beyond the boundaries of the medical imaging data; or Based on the rejection of the medical imaging data, recommendations for corrective actions are generated.
31. The non-transitory computer-readable medium of claim 25, wherein the shape model is derived from a statistical population of images of other anatomical structures comparable to the target anatomical structure.
32. The non-transitory computer-readable medium of claim 25, wherein the shape model comprises any one or more of the following: a statistical shape model, an active shape model, an active appearance model, and an active contour model.
33. The non-transitory computer-readable medium of claim 25, wherein the medical imaging data is a CT volume and the target anatomical structure is a skeletal structure.
34. A computer-implemented method for automatically assessing whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for said target anatomical structure, the computer-implemented method comprising: Automatically identify shape models and fit the shape models to the target anatomical structures in the medical imaging data; The features of the shape model are automatically compared with the boundaries of the medical imaging data; as well as In response to determining that the features of the shape model exceed the boundaries of the medical imaging data, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
35. The computer-implemented method of claim 34, wherein the features of the shape model include the boundary of the shape model, and the computer-implemented method further comprises: The boundary of the shape model is compared with the boundary of the medical imaging data; as well as In response to determining that the boundary of the shape model exceeds the boundary of the medical imaging data, the medical imaging data is rejected as unacceptable for facilitating surgical planning.
36. The computer-implemented method of claim 34, wherein the features of the shape model include one or more points located on the boundary of the shape model, and the computer-implemented method further comprises: Compare the one or more points of the shape model with the boundary of the medical imaging data; as well as In response to determining that one or more points of the shape model exceed the boundary of the medical imaging data, the medical imaging data is rejected as unacceptable for facilitating surgical planning.
37. The computer-implemented method of claim 36, wherein the one or more points on the shape model correspond to predetermined anatomical landmarks of the target anatomical structure.
38. The computer-implemented method of claim 34, further comprising, in response to determining that the features of the shape model are within the boundaries of the medical imaging data, automatically approving the medical imaging data as acceptable for facilitating surgical planning.
39. The computer-implemented method of claim 34, further comprising automatically generating output in response to rejecting the medical imaging data as unacceptable, wherein the output includes one or more of the following: Generate a notification or alert to the graphical user interface to inform that the medical imaging data has been rejected; Display the medical imaging data and the shape model on a graphical user interface to show that the features of the shape model extend beyond the boundaries of the medical imaging data; and Based on the rejection of the medical imaging data, recommendations for corrective actions are generated.
40. The computer-implemented method of claim 34, wherein the shape model is derived from a statistical population of images of other anatomical structures that can be compared with the target anatomical structure.
41. The computer-implemented method of claim 34, wherein the shape model comprises any one or more of the following: a statistical shape model, an active shape model, an active appearance model, and an active contour model.
42. The computer-implemented method of claim 34, wherein the medical imaging data is a CT volume and the target anatomical structure is a skeletal structure.
43. The computer-implemented method of claim 34, wherein assessing whether the medical imaging data is acceptable for facilitating surgical planning for the target anatomy is further defined as assessing whether the medical imaging data is acceptable for facilitating any one or more of the following: The segmentation of the medical imaging data; Virtual model creation of the target anatomical structure; Virtual implant planning relative to the target anatomical structure; The registration of the target anatomical structures facilitates surgical navigation; as well as The visualization of the medical imaging data facilitates surgical navigation.
44. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: The medical imaging data is automatically compared with a statistical cohort of medical imaging data that includes other anatomical structures to identify anatomical landmarks of the target anatomical structure that are visible in the medical imaging data and can be compared with the target anatomical structure. The medical imaging data is automatically evaluated to determine whether the anatomical landmarks are visible in the medical imaging data; as well as In response to determining that the anatomical landmarks are not visible in the medical imaging data, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
45. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to automatically: Identify shape models and fit the shape models to the target anatomical structures in the medical imaging data; Compare the shape model with the boundaries of the medical imaging data; It is determined that a portion of the shape model extends beyond the boundary of the medical imaging data; as well as The medical imaging data is modified to capture the portion of the shape model that extends beyond the boundaries of the medical imaging data.
46. The non-transitory computer-readable medium of claim 45, wherein the instructions, when executed by one or more processors, modify the medical imaging data by being configured to capture portions of the shape model that extend beyond the boundaries of the medical imaging data: Generate an extrapolated region that includes the portion of the shape model.
47. The non-transitory computer-readable medium of claim 46, wherein the instructions, when executed by one or more processors, are configured to generate the extrapolation region adjacent to the boundary of the medical imaging data.
48. The non-transitory computer-readable medium of claim 46, wherein the instructions, when executed by the one or more processors, are configured to extend the boundaries of the medical imaging data to include the extrapolated region.
49. The non-transitory computer-readable medium of claim 46, wherein the instructions, when executed by one or more processors, are configured to artificially recreate features of the target anatomical structure within the extrapolation region, the features being within the portion of the shape model that extends beyond the boundary of the medical imaging data.
50. The non-transitory computer-readable medium of claim 49, wherein the instructions, when executed by one or more processors, are configured to artificially recreate a soft tissue region within the extrapolation region, the soft tissue region being adjacent to the portion of the shape model that extends beyond the boundary of the medical imaging data.
51. The non-transitory computer-readable medium of claim 45, wherein the instructions, when executed by one or more processors, are configured to provide color coding that identifies portions of the shape model that extend beyond the boundaries of the medical imaging data.
52. The non-transitory computer-readable medium of claim 45, wherein the instructions, when executed by one or more processors, modify the medical imaging data by being configured to capture portions of the shape model that extend beyond the boundaries of the medical imaging data: This enables the medical imaging data to be visualized together with the shape model, including the portion of the shape model that extends beyond the boundaries of the medical imaging data.
53. The non-transitory computer-readable medium of claim 45, wherein the instructions are further configured, when executed by the one or more processors, to: The medical imaging data is annotated to identify the portion of the shape model that extends beyond the boundaries of the medical imaging data.
54. The non-transitory computer-readable medium of claim 45, wherein the shape model is derived from a statistical population of images of other anatomical structures comparable to the target anatomical structure.
55. The non-transitory computer-readable medium of claim 45, wherein the shape model comprises any one or more of the following: a statistical shape model, an active shape model, an active appearance model, and an active contour model.
56. The non-transitory computer-readable medium of claim 45, wherein the medical imaging data is a CT volume and the target anatomical structure is a skeletal structure.
57. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: Automatically obtain measurements of the target anatomical structure in the medical imaging data; Based on the measurements of the target anatomical structure in the medical imaging data, an implant is automatically selected from multiple implant options, and implant measurements of the selected implant are automatically obtained; Based on the implant measurements, the required features of the target anatomy are automatically determined, and these required features must be fully captured in the medical imaging data to acceptablely facilitate surgical planning of the selected implant relative to the target anatomy; and In response to the failure of the required features for determining the target anatomical structure to be fully captured in the medical imaging data, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
58. The non-transitory computer-readable medium of claim 57, wherein the instructions are configured, when executed by the one or more processors, to: In response to the determination that the required features of the target anatomical structure are fully captured in the medical imaging data, the medical imaging data is automatically approved as acceptable for facilitating surgical planning.
59. The non-transitory computer-readable medium of claim 57, wherein the instructions, when executed by the one or more processors, automatically obtain the measurement of the target anatomical structure by being configured to perform the following: Automatically identify shape models and fit the shape models to the target anatomical structures in the medical imaging data; and The measurements of the target anatomical structure are automatically obtained from the shape model.
60. The non-transitory computer-readable medium of claim 59, wherein the instructions, when executed by the one or more processors, are configured to automatically obtain the measurement of the target anatomical structure from the shape model by: The distance between points on the shape model is automatically measured, where each point corresponds to a predetermined anatomical landmark of the target anatomical structure.
61. The non-transitory computer-readable medium of claim 59, wherein the shape model is derived from a statistical population of images of other anatomical structures comparable to the target anatomical structure.
62. The non-transitory computer-readable medium of claim 59, wherein the shape model comprises any one or more of the following: a statistical shape model, an active shape model, an active appearance model, and an active contour model.
63. The non-transitory computer-readable medium of claim 57, wherein the instructions, when executed by the one or more processors, are further configured to automatically select the implant based on the measurements of the target anatomy: The implant is automatically selected and its size is appropriately determined for the measurement of the target anatomical structure.
64. The non-transitory computer-readable medium of claim 57, wherein the claimed feature of the target anatomical structure is the claimed measurement and / or the claimed marker.
65. The non-transitory computer-readable medium of claim 57, wherein the instructions, when executed by the one or more processors, automatically generate output in response to rejecting the medical imaging data as unacceptable by being configured to: Generate a notification or alert to the graphical user interface to inform that the medical imaging data has been rejected; The medical imaging data and the required features of the target anatomical structure that were not fully captured in the medical imaging data are displayed on a graphical user interface. The medical imaging data and the selected implant or the implant measurement are displayed on the graphical user interface; or Based on the rejection of the medical imaging data, recommendations for corrective actions are generated.
66. A computer-implemented method for automatically assessing whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for said target anatomical structure, the computer-implemented method comprising: Automatically obtain measurements of the target anatomical structure in the medical imaging data; Based on the measurements of the target anatomical structure in the medical imaging data, an implant is automatically selected from multiple implant options, and implant measurements of the selected implant are automatically obtained; Based on the implant measurements, the required features of the target anatomy are automatically determined, and these required features must be fully captured in the medical imaging data to acceptablely facilitate surgical planning of the selected implant relative to the target anatomy. as well as In response to the failure of the required features for determining the target anatomical structure to be fully captured in the medical imaging data, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
67. The computer-implemented method of claim 66, further comprising, in response to determining that the required features of the target anatomical structure are fully captured in the medical imaging data, automatically approving the medical imaging data as acceptable for facilitating surgical planning.
68. The computer-implemented method of claim 66, wherein the measurement of the target anatomical structure is further defined as automatically: Identify shape models and fit the shape models to the target anatomical structures in the medical imaging data; and The measurements of the target anatomical structure are obtained from the shape model.
69. The computer-implemented method of claim 68, wherein the measurement of automatically obtaining the target anatomical structure from the shape model is further defined as: The distance between points on the shape model is automatically measured, where each point corresponds to a predetermined anatomical landmark of the target anatomical structure.
70. The computer-implemented method of claim 68, wherein the shape model is derived from a statistical population of images of other anatomical structures that can be compared with the target anatomical structure.
71. The computer-implemented method of claim 68, wherein the shape model comprises any one or more of the following: a statistical shape model, an active shape model, an active appearance model, and an active contour model.
72. The computer-implemented method of claim 66, wherein automatically selecting the implant based on the measurement of the target anatomy is further defined as automatically selecting the implant, the implant being appropriately sized for the measurement of the target anatomy.
73. The computer-implemented method of claim 66, wherein the required features of the target anatomical structure are the required measurements and / or the required markers.
74. The computer-implemented method of claim 66, further comprising automatically generating output in response to rejecting the medical imaging data as unacceptable, wherein the output includes one or more of the following: Generate a notification or alert to the graphical user interface to inform that the medical imaging data has been rejected; The medical imaging data and the required features of the target anatomical structure that were not fully captured in the medical imaging data are displayed on a graphical user interface. Display the medical imaging data and the selected implant or measurements of the implant in a graphical user interface; and Based on the rejection of the medical imaging data, recommendations for corrective actions are generated.
75. The computer-implemented method of claim 66, wherein assessing whether the medical imaging data is acceptable for facilitating surgical planning for the target anatomy is further defined as assessing whether the medical imaging data is acceptable for facilitating any one or more of the following: The segmentation of the medical imaging data; Virtual model creation of the target anatomical structure; Virtual implant planning relative to the target anatomical structure in the medical imaging data; The registration of the target anatomical structures facilitates surgical navigation; as well as The visualization of the medical imaging data facilitates surgical navigation.
76. The computer-implemented method of claim 66, wherein the medical imaging data is a CT image and the target anatomical structure is a skeletal structure.
77. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to automatically: Measurements of the target anatomical structure obtained from the medical imaging data; Based on the measurements of the target anatomical structure in the medical imaging data, an implant is selected from multiple implant options, and measurements of the selected implant are obtained; Based on the implant measurements, the required features of the target anatomy are determined, and these required features must be fully captured in the medical imaging data to acceptablely facilitate surgical planning of the selected implant relative to the target anatomy. The required features for determining the target anatomical structure were not fully captured in the medical imaging data; Identify a shape model and fit the shape model to the target anatomical structure in the medical imaging data, wherein a portion of the shape model extends beyond the boundaries of the medical imaging data; as well as The medical imaging data is modified to capture the portion of the shape model that extends beyond the boundaries of the medical imaging data.
78. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: Automatically obtain measurements of the target anatomical structure in the medical imaging data; Based on the measurements of the target anatomical structure in the medical imaging data, an implant is automatically selected from multiple implant options, and implant measurements of the selected implant are automatically obtained; The boundaries of the implant measurements are automatically compared with those of the medical imaging data; as well as In response to determining that the implant measurement exceeds the boundary of the medical imaging data or fails to be spaced from the boundary of the medical imaging data by a threshold distance, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
79. The non-transitory computer-readable medium of claim 78, wherein the instructions are configured, when executed by the one or more processors, to: In response to determining that the implant measurement is within or separated from the boundary of the medical imaging data by the threshold distance, the medical imaging data is automatically approved as acceptable for facilitating surgical planning.
80. The non-transitory computer-readable medium of claim 78, wherein, in order to automatically obtain the measurement of the selected implant, the instructions are configured to automatically, when executed by the one or more processors: Obtain an implant shape model for the selected implant; and The measurements of the selected implant are obtained from the implant shape model.
81. The non-transitory computer-readable medium of claim 80, wherein the instructions, when executed by the one or more processors, are configured to: fit the implant shape model to the target anatomical structure in the medical imaging data.
82. The non-transitory computer-readable medium of claim 81, wherein the implant shape model includes boundaries or points, and wherein, in order to automatically compare the implant measurements with the boundaries of the medical imaging data, the instructions are configured to automatically: Compare the boundaries or points of the implant shape model with the boundaries of the medical imaging data; and In response to determining that the boundary or point of the implant shape model exceeds the boundary of the medical imaging data or fails to be spaced from the boundary of the medical imaging data by the threshold distance, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
83. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to automatically: Identify the implant shape model and fit the implant shape model to the target anatomical structure in the medical imaging data; Compare the implant shape model with the boundaries of the medical imaging data; Determine that a portion of the implant shape model extends beyond the boundary of the medical imaging data; as well as The medical imaging data is modified to capture the portion of the implant shape model that extends beyond the boundaries of the medical imaging data.
84. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: Obtain preoperative patient data associated with the medical imaging data, the preoperative patient data including information indicating expected parameters, the expected parameters including: (1) the target anatomical structure of the expected type and (2) the expected surgical side of the target anatomical structure; The medical imaging data is automatically applied to a machine learning model to analyze the medical imaging data and output predicted parameters, the predicted parameters including: (1') the predicted type of the target anatomical structure and (2') the predicted surgical side of the target anatomical structure; Automatically compare each expected parameter with its corresponding predicted parameter; and In response to determining that each expected parameter is acceptablely matched with its corresponding predicted parameter, the medical imaging data is automatically approved as acceptable for facilitating surgical planning.
85. The non-transitory computer-readable medium of claim 84, wherein the instructions are configured, when executed by the one or more processors, to: In response to determining that any one or more of the expected parameters fail to match their corresponding predicted parameters acceptablely, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
86. The non-transitory computer-readable medium of claim 84, wherein the medical imaging data includes CT volumes, and wherein the instructions are configured, when executed by the one or more processors, to: Digitally reconstructed radiographic images are automatically generated from the CT volume; the digitally reconstructed radiographic images are two-dimensional; and The digitally reconstructed radiographic images are automatically applied to the machine learning model to analyze the digitally reconstructed radiographic images and output the predicted parameters.
87. The non-transitory computer-readable medium of claim 84, wherein the machine learning model is a convolutional neural network, and wherein the instructions are configured, when executed by the one or more processors, to: The medical imaging data is automatically applied to the convolutional neural network to analyze the medical imaging data and output the predicted parameters.
88. The non-transitory computer-readable medium of claim 84, wherein the target anatomical structure is an anatomical joint, and wherein the instructions are configured, when executed by the one or more processors, to: Obtain preoperative patient data associated with the medical imaging data, the preoperative patient data including information indicating expected parameters, the expected parameters including: (1) the type of anatomical joint and (2) the intended surgical side of the anatomical joint; as well as The medical imaging data is automatically applied to the machine learning model to analyze the medical imaging data and output predicted parameters, the predicted parameters including: (1') the predicted type of anatomical joint and (2') the predicted surgical side of the anatomical joint.
89. The non-transitory computer-readable medium of claim 88, wherein the medical imaging data includes the CT volume of the anatomical joint, and the machine learning model is a lightweight 2D convolutional neural network, and wherein the instructions are configured, when executed by the one or more processors, to: A first digitally reconstructed radiograph is automatically generated from the CT volume. The first digitally reconstructed radiograph is two-dimensional and captures the skeletal structure of the anatomical joint. A second digitally reconstructed radiograph is automatically generated from the CT volume. The second digitally reconstructed radiograph is two-dimensional and captures the soft tissue structure of the anatomical joint. The first digitally reconstructed radiograph is automatically applied to the lightweight 2D convolutional neural network to analyze the first digitally reconstructed radiograph to output (1') the predicted type of anatomical joint; and The second digitally reconstructed radiograph is automatically applied to the lightweight 2D convolutional neural network to analyze the second digitally reconstructed radiograph to output (2') the predicted surgical side of the anatomical joint.
90. The non-transitory computer-readable medium of claim 84, wherein the instructions are configured, when executed by the one or more processors, to: Obtain preoperative patient data that includes information further indicative of expected parameters, said expected parameters including: (3) The anticipated presence or absence of existing implants for the target anatomical structure; The medical imaging data is automatically applied to the machine learning model to analyze the medical imaging data in order to further output predicted parameters, including: (3') the predicted presence or absence of existing implants for the target anatomical structure; Automatically compare the expected presence or absence of existing implants with the predicted presence or absence of existing implants; and In response to the determination that the expected presence or absence of an existing implant fails to acceptablely match the predicted presence or absence of an existing implant, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
91. The non-transitory computer-readable medium of claim 90, wherein the target anatomical structure is an anatomical joint, the medical imaging data includes the CT volume of the anatomical joint, and the machine learning model is a convolutional neural network, and wherein the instructions are configured, when executed by the one or more processors, to: Automatically generate digitally reconstructed radiographic images capturing metallic structures within the anatomical joint from the CT volume; and The digitally reconstructed radiographs are automatically applied to the convolutional neural network to analyze the digitally reconstructed radiographs to output (3') the predicted presence or absence of existing implants for the target anatomical structure.
92. The non-transitory computer-readable medium of claim 91, wherein the instructions are configured, when executed by the one or more processors, to: The digitally reconstructed radiographic images are automatically applied to the convolutional neural network to analyze the images and output the predicted presence of the existing implant and one or more of the following: the predicted manufacturer of the existing implant; the predicted model number of the existing implant; and the predicted size or type of the existing implant.
93. The non-transitory computer-readable medium of claim 84, wherein the instructions are configured, when executed by the one or more processors, to: Obtain preoperative patient data that includes information further indicative of expected parameters, said expected parameters including: (4) Procedures to be planned for the intended type of the target anatomical structure; The medical imaging data is automatically applied to the machine learning model to analyze the medical imaging data in order to further output predicted parameters, the predicted parameters including: (4') a procedure of a predicted type to be planned for the target anatomical structure; Automatically compare the procedure for the expected type with the procedure for the predicted type; and In response to a procedure that determines the expected type of procedure failing to acceptablely match the predicted type of procedure, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
94. A non-transitory computer-readable medium comprising instructions configured to automatically classify CT volumes of anatomical joints, wherein the instructions are configured, when executed by one or more processors, to: Multiple digitally reconstructed radiographs are automatically generated from the CT volume, the multiple digitally reconstructed radiographs being two-dimensional and capturing the structure of the anatomical joints; and Each of the digitally reconstructed radiographs is automatically applied to a convolutional neural network configured to analyze each of the digitally reconstructed radiographs to output (1') the predicted type of anatomical joint, (2') the predicted surgical side of the anatomical joint, and (3') the predicted presence or absence of an existing implant for the anatomical joint.
95. The non-transitory computer-readable medium of claim 94, wherein the instructions, when executed by one or more processors, are configured to automatically apply each of the digitally reconstructed radiographs to the convolutional neural network to analyze each of the digitally reconstructed radiographs to further output (4') a procedure to be planned for the predicted type of the anatomical joint.
96. The non-transitory computer-readable medium of claim 94, wherein the instructions are configured, when executed by the one or more processors, to: Each of the digitally reconstructed radiographs is automatically applied to the convolutional neural network to analyze each of the digitally reconstructed radiographs to further output the predicted presence of the existing implant and one or more of the following: the predicted manufacturer of the existing implant; the predicted model number of the existing implant; and the predicted size or type of the existing implant.
97. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: Automatically obtain one or more configuration settings that define how the medical imaging data is scanned by the imaging device; Automatically compare the one or more configuration settings with one or more acceptable configuration settings; and In response to determining that one or more configuration settings do not correspond to one or more acceptable configuration settings, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
98. The non-transitory computer-readable medium of claim 97, wherein the instructions, when executed by the one or more processors, are configured to: automatically approve the medical imaging data as acceptable for facilitating surgical planning in response to determining that the one or more configuration settings correspond to the one or more acceptable configuration settings.
99. A non-transitory computer-readable medium comprising instructions configured to automatically assess whether medical imaging data of a target anatomical structure is acceptable for facilitating surgical planning for the target anatomical structure, wherein the instructions, when executed by one or more processors, are configured to: Automatically identify moving rods within the volume of the medical imaging data; The volume of the medical imaging data is automatically evaluated to determine whether the volume can acceptably represent the motion lever; and In response to determining that the volume fails to unacceptably represent the motion lever, the medical imaging data is automatically rejected as unacceptable for facilitating surgical planning.
100. The non-transitory computer-readable medium of claim 99, wherein, in order to automatically identify the motion lever, the instructions are configured to, when executed by one or more processors: Implement an object detection algorithm or machine learning model to automatically identify the moving rod in the volume and distinguish the moving rod from other features exhibited in the volume.
101. The non-transitory computer-readable medium of claim 100, wherein, in order to automatically assess the volume of the medical imaging data to determine whether the volume acceptablely represents the motion lever, the instructions, when executed by one or more processors, are configured to: Implement the object detection algorithm or the machine learning model to automatically determine whether the moving rod exhibits a complete cylinder in the volume.
