Automated pre-screening to evaluate whether medical imaging data is suitable for surgical planning purposes

Abstract

An automated imaging suite, software program, and method for automatically evaluating whether medical imaging data of an anatomical part is acceptable to facilitate surgical planning for a target anatomical part. The automated imaging suite is configured to perform one or more automated inspections to determine, for example, whether the medical imaging data was scanned according to acceptable configuration settings; whether the target anatomical part is acceptablely captured within the boundaries of the medical imaging data in the medical imaging data; whether the patient moved during the scan; whether essential features of the target anatomical part that must be fully captured within the medical imaging data to acceptablely facilitate surgical planning of a selected implant for the target anatomical part are acceptablely captured within the boundaries of the medical imaging data; and / or whether the medical imaging data acceptablely shows the intended type of the target anatomical part and the intended surgical side of the target anatomical part.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims the priority and all benefits of U.S. Provisional Application No. 63 / 626,742 (filed January 30, 2024, the whole of which is incorporated herein by reference). Background Technology

[0003] The surgical planning process for orthopedic joint replacement surgery typically begins with acquiring medical imaging data of the patient's anatomical joint. A treatment planning request is submitted to the surgical planning team. The team reviews the patient information and manually evaluates the imaging data. A segmentation specialist manually performs a segmentation process to extract the contours of the bone(s) from multiple image slices of the data. The output of this segmentation process is a virtual model of the bone(s). The surgical planning team or the surgeon uses this virtual model as a reference to virtually plan the type, size, and location of the joint replacement implant(s) for the bone(s) and the proposed surgical workflow. The virtual surgical plan, including the virtual model, is aligned with the actual bone during surgery to provide the surgeon with guidance during the operation.

[0004] The general review and planning process has several drawbacks. One difficulty is that the image data input to the segmentation team may be incomplete or contain errors. Consequently, such image data is unsuitable for segmentation, bone model generation, or intraoperative purposes, such as visualizing bone models on a display or aligning bone models to actual bones. For instance, if bone images were not acquired with appropriate scanner configuration settings, lack essential bone details, have portions of the target bone removed, or are not large enough for navigational visualization, the image data may not be suitable for surgical planning or intraoperative purposes. Another potential issue is when the image data fails to show what was intended. For instance, patient information might indicate a need for surgery on the patient's left knee, but the medical imaging might show an image of the right knee. Another error could occur where the image data shows existing implants or metal (suggesting revision surgery), while the patient information specifies a request for a primary surgery. Additionally, noise or blurring may appear in the image data, making the image quality insufficient for segmentation.

[0005] If these imaging errors are overlooked, a chain reaction of problems can occur, potentially impacting the accuracy and completeness of the described aspects of the planning process and surgical workflow. This issue is exacerbated by the fact that hundreds of thousands of patients require surgical planning every year.

[0006] The content section of the invention provides a brief introduction to some of the concepts further described in detail in the detailed description below. The content section is not intended to limit the scope of the claimed invention, nor is it intended to identify the core or essential features of the claimed invention.

[0007] According to a first embodiment, an automated image checking suite, a non-transient computer-readable computer program product, or a computer-implemented method is provided to automatically evaluate whether medical image data of a target anatomical part is acceptable to facilitate surgical planning for the target anatomical part by being configured to automatically reject the medical image data as unacceptable to facilitate surgical planning in response to any one or more of the automated checks being determined to produce unacceptable results.

[0008] According to a second embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to perform the following steps: receiving medical image data as input, wherein at the time of input, the type and laterality of the target anatomical part in the medical image data are unclassified; automatically classifying the type of the target anatomical part in the medical image data using a first machine learning model; selecting a second machine learning model specially trained to classify the laterality of the classified type of the target anatomical part using the classified type of the target anatomical part; automatically classifying the laterality of the target anatomical part in the medical image data using the second machine learning model; and generating a computer-generated output that identifies the classified type and the classified laterality of the target anatomical part in the medical image data, thereby being provided to automatically evaluate medical image data of a target anatomical part.

[0009] According to a third embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to perform the steps of: automatically identifying a target anatomical part in medical image data and fitting a shape model to the target anatomical part; automatically comparing features of the shape model with the boundaries of the medical image data; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that features of the shape model exceed the boundaries of the medical image data, thereby being provided to automatically evaluate whether the medical image data of the target anatomical part is acceptable for facilitating surgical planning for the target anatomical part.

[0010] According to a fourth embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to perform the steps of: automatically comparing medical image data with a statistical group of medical image data containing other anatomical parts similar to the target anatomical part to identify anatomical landmarks of the target anatomical part required to be visible in the medical image data; automatically evaluating the medical image data to determine whether the anatomical landmarks are visible in the medical image data; and automatically rejecting the medical image data as unacceptable to facilitate surgical planning in response to the determination that the anatomical landmarks are not visible in the medical image data, thereby being provided to automatically evaluate whether the medical image data of the target anatomical part is acceptable to facilitate surgical planning for the target anatomical part.

[0011] According to the fifth embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to perform the steps of: automatically acquiring measurements of a target anatomical part in medical image data; automatically selecting an implant from a plurality of implant options based on the measurements of the target anatomical part in medical image data and automatically acquiring implant measurements of the selected implant; automatically determining, based on the implant measurements, essential features of the target anatomical part that must be fully captured in the medical image data to enable acceptable planning of the selected implant for the target anatomical part; and automatically rejecting the medical image data as unacceptable for enabling surgical planning in response to the determination that the essential features of the target anatomical part are not fully captured in the medical image data, thereby enabling automatic evaluation of whether the medical image data of the target anatomical part is acceptable for enabling surgical planning for the target anatomical part.

[0012] According to the sixth embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to perform the steps of: automatically acquiring measurements of a target anatomical part in medical image data; automatically selecting an implant from a plurality of implant options based on measurements of the target anatomical part in medical image data and automatically acquiring implant measurements of the selected implant; automatically comparing the implant measurements with the boundaries of the medical image data; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the implant measurements exceed the boundaries of the medical image data or are not separated by a threshold distance from the boundaries of the medical image data, thereby providing for automatically evaluating whether the medical image data of the target anatomical part is acceptable for facilitating surgical planning for the target anatomical part.

[0013] According to the seventh embodiment, an automated image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to perform the step of acquiring preoperative patient data related to medical image data, wherein the preoperative patient data includes information representing intended parameters, such as (1) an intended type of target anatomical part and (2) an intended surgical side of the target anatomical part; the step of analyzing the medical image data and automatically applying the medical image data to a machine learning model to output predicted parameters, such as (1') a predicted type of target anatomical part and (2') a predicted surgical side of the target anatomical part; the step of automatically comparing each intended parameter with a corresponding predicted parameter; and the step of automatically approving the medical image data as acceptable to facilitate surgical planning in response to a determination that each intended parameter is acceptablely matched with a corresponding predicted parameter, thereby being provided to automatically evaluate whether the medical image data of the target anatomical part is acceptable to facilitate surgical planning for the target anatomical part.

[0014] According to the eighth embodiment, an automated imaging suite, a non-transient computer-readable computer program product, or a computer-implemented method is provided to automatically classify a CT volume of an anatomical joint by performing the steps of: automatically generating a plurality of two-dimensional digitally-reconstructed radiographic images capturing the structure of an anatomical joint from a CT volume; and automatically applying each digitally-reconstructed radiographic image to a convolutional neural network configured to analyze each digitally-reconstructed radiographic image and output (1') a predicted type of anatomical joint, (2') a predicted surgical side of the anatomical joint, and (3') the predicted presence or absence of an existing implant in the anatomical joint.

[0015] According to the ninth aspect, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to automatically acquire one or more configuration settings that define how an image device scans medical image data; automatically compare one or more configuration settings with one or more acceptable configuration settings; and automatically reject the medical image data as unacceptable for facilitating surgical planning in response to a determination that one or more configuration settings do not correspond to one or more acceptable configuration settings, thereby being provided to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning of the target anatomical part.

[0016] According to the tenth embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is configured to perform the steps of: automatically identifying a motion load within a volume of medical image data; automatically evaluating a volume of medical image data to determine whether the volume shows the motion load in an acceptable manner; and automatically rejecting the medical image data as unacceptable to facilitate surgical planning in response to the determination that the volume does not show the motion load in an acceptable manner, thereby being provided to automatically evaluate whether medical image data of a target anatomical part is acceptable to facilitate surgical planning for the target anatomical part.

[0017] According to the 11th embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is provided to automatically perform the steps of: identifying a target anatomical part in medical image data and fitting a shape model to the target anatomical part; comparing the shape model with the boundaries of the medical image data; determining that a part of the shape model exceeds the boundaries of the medical image data; and modifying the medical image data to capture the part of the shape model that exceeds the boundaries of the medical image data.

[0018] According to the 12th embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is provided to automatically perform the steps of: acquiring measurements of a target anatomical part in medical image data; selecting an implant from a plurality of implant options based on the measurements of the target anatomical part in medical image data and acquiring implant measurements of the selected implant; determining, based on the implant measurements, essential features of the target anatomical part that must be fully captured in the medical image data to allowably facilitate the surgical planning of the selected implant for the target anatomical part; determining that essential features of the target anatomical part are not fully captured in the medical image data; identifying the target anatomical part in medical image data and fitting a shape model to the target anatomical part, wherein a part of the shape model extends beyond the boundaries of the medical image data; and modifying the medical image data to capture the part of the shape model that extends beyond the boundaries of the medical image data.

[0019] According to the 13th embodiment, an automatic image inspection suite, a non-transient computer-readable computer program product, or a computer-implemented method is provided to automatically perform the steps of: identifying a target anatomical part in medical image data and fitting an implant shape model to the target anatomical part; comparing the implant shape model with the boundaries of the medical image data; determining that a part of the implant shape model exceeds the boundaries of the medical image data; and modifying the medical image data to capture the part of the implant shape model that exceeds the boundaries of the medical image data.

[0020] Any of the above modes may be used individually or in combination.

[0021] Some of the above specific embodiments may be used in combination with other parts of other embodiments.

[0022] Any of the above embodiments may be used individually or in combination with any one or more of the following embodiments:

[0023] Features of the shape model may include the boundaries of the shape model. The boundaries of the shape model may be compared with the boundaries of the medical image data. In response to a determination that the boundaries of the shape model exceed the boundaries of the medical image data, the medical image data may be rejected as unacceptable for facilitating surgical planning; features of the shape model may include one or more points located on the boundaries of the shape model. One or more points of the shape model may be compared with the boundaries of the medical image data. In response to a determination that one or more points of the shape model exceed the boundaries of the medical image data, the medical image data may be rejected as unacceptable for facilitating surgical planning. One or more points on the shape model may correspond to predetermined anatomical landmarks of the target anatomical part. In response to a determination that features of the shape model are within the boundaries of the medical image data, the medical image data may be automatically approved as acceptable for facilitating surgical planning. The shape model may be derived from a statistical group of images of other anatomical parts similar to the target anatomical part. The shape model may include any one or more of a statistical shape model, an active shape model, an active outline model, and an active contour model. The output result may be automatically generated in response to medical image data being rejected as unacceptable. The output result may be generating a notification or warning in a graphical user interface to indicate the rejection of the medical image data. The output result may be displaying the medical image data and the geometric model in a graphical user interface to show that the features of the geometric model exceed the boundaries of the medical image data. The output result may be generating recommendations for corrective measures based on the rejection of the medical image data. The medical image data may be a CT volume. The target anatomical part may be a bone structure.Evaluating whether medical imaging data is acceptable for establishing a surgical plan for a target anatomical part may be defined as evaluating whether the medical imaging data is acceptable for facilitating any one or more of the following: subsequent segmentation of the medical imaging data; creation of a virtual model of the target anatomical part; establishment of a virtual implant plan for the target anatomical part; alignment of the target anatomical part to facilitate surgical navigation; and visualization of the medical imaging data to facilitate surgical navigation. In response to a determination that the essential features of the target anatomical part are fully captured in the medical imaging data, the medical imaging data may be automatically approved as acceptable for establishing a surgical plan. Measurements of the target anatomical part may be automatically acquired. The target anatomical part may be automatically identified in the medical imaging data and a shape model may be fitted to the target anatomical part. Measurements of the target anatomical part may be automatically acquired from the shape model. Distances between points on the shape model may be automatically measured. Each point may correspond to a predetermined anatomical landmark of the target anatomical part. Implants may be automatically selected based on the measurements of the target anatomical part. The implant can be automatically selected to be appropriately sized for the measurements of the target anatomical part. Essential features of the target anatomical part may be essential measurements and / or essential landmarks. Output results can be automatically generated in response to medical imaging data being rejected as unacceptable. Output results may include medical imaging data and displaying essential features of the target anatomical part that were not fully captured in the medical imaging data in a graphical user interface. Output results may include medical imaging data and displaying the selected implant or implant measurements in a graphical user interface.Medical imaging data may be automatically accepted as acceptable to facilitate surgical planning in response to a determination that implant measurements are within the boundaries of the medical imaging data and / or are located at a threshold distance from the boundaries of the medical imaging data. An implant geometry model may be automatically generated for the selected implant. Measurements of the selected implant may be obtained from the implant geometry model. The implant geometry model may be fitted to the target anatomical part in the medical imaging data. The implant geometry model may include boundaries or points. The boundaries or points of the implant geometry model may be automatically compared with the boundaries of the medical imaging data. Medical imaging data may be rejected as unacceptable to facilitate surgical planning in response to a determination that the boundaries or points of the implant geometry model exceed the boundaries of the medical imaging data or are not located at a threshold distance from the boundaries of the medical imaging data. Medical imaging data may be automatically rejected as unacceptable to facilitate surgical planning in response to a determination that any one or more of the intended parameters do not agree acceptablely with the corresponding predicted parameters. Medical imaging data may include a CT volume. A two-dimensional digitally reconstructed radiographic image may be automatically generated from the CT volume. Digitally reconstructed radiographic images can be automatically applied to a machine learning model to analyze the images and output predicted parameters. The machine learning model may be a convolutional neural network. Medical imaging data can be automatically applied to a convolutional neural network to analyze the medical imaging data and output predicted parameters. Preoperative patient data related to the medical imaging data can be automatically acquired. Preoperative patient data may include information indicating intended parameters, such as the intended type of anatomical joint and the intended side of the anatomical joint.Medical imaging data can be automatically applied to a machine learning model to analyze the medical imaging data and output predicted parameters, for example, the predicted type of anatomical joint and the predicted surgical side of the anatomical joint. The medical imaging data may include CT volumes of the anatomical joint. The machine learning model may be a lightweight 2D convolutional neural network. From the CT volume, a first 2D digitally reconstructed radiographic image capturing the bone structure of the anatomical joint may be automatically generated. From the CT volume, a second 2D digitally reconstructed radiographic image capturing the soft tissue structure of the anatomical joint may be generated. The first digitally reconstructed radiographic image can be automatically applied to a lightweight 2D convolutional neural network to analyze the first digitally reconstructed radiographic image and output the predicted type of the anatomical joint. The second digitally reconstructed radiographic image can be automatically applied to a lightweight 2D convolutional neural network to analyze the second digitally reconstructed radiographic image and output the predicted surgical side of the anatomical joint. Preoperative patient data may be acquired including information indicating intended parameters, such as the intended presence or absence of existing implants in the target anatomy. Medical imaging data may be automatically applied to a machine learning model to analyze the medical imaging data and further output predicted parameters, such as the predicted presence or absence of existing implants in the target anatomical part. The intended presence or absence of existing implants may be automatically compared with the predicted presence or absence of existing implants. Medical imaging data may be automatically rejected as unacceptable to facilitate surgical planning in response to a determination that the intended presence or absence of existing implants does not acceptablely match the predicted presence or absence of existing implants. Digitally reconstructed radiographic images capturing metal structures in the anatomical joint may be automatically generated from CT volumes.Digitally reconstructed radiographic images can be automatically applied to a convolutional neural network to analyze the digitally reconstructed radiographic images and output the predicted presence or absence of existing implants in the target anatomical region. Digitally reconstructed radiographic images can be automatically applied to a convolutional neural network to analyze the digitally reconstructed radiographic images and output not only the predicted presence of existing implants but also the predicted manufacturer of the existing implants; the predicted model number of the existing implants; and one or more of the predicted size or type of the existing implants. Preoperative patient data may be acquired, including information indicating intended parameters such as the type of intended procedure planned for the target anatomical region. Medical imaging data can be automatically applied to a machine learning model to analyze the medical imaging data and output additional predicted parameters, such as the type of procedure to be planned for the target anatomical region. The intended procedure type can be automatically compared with the predicted procedure type. Medical imaging data can be automatically rejected as unacceptable to facilitate surgical planning in response to a determination that the intended procedure type does not acceptablely match the predicted procedure type. Medical imaging data may be automatically approved as acceptable to facilitate surgical planning in response to a determination that one or more configuration settings correspond to one or more acceptable configuration settings.

[0024] Medical imaging data may be a 3D CT volume. The target anatomical part may be an anatomical joint. A first machine learning model can automatically classify the type of anatomical joint in the 3D CT volume into one of unilateral hip joint, unilateral knee joint, unilateral ankle joint, bilateral hip joint, bilateral knee joint, or bilateral ankle joint. A confidence score may be generated to indicate the classification accuracy of the type of anatomical joint in the 3D CT volume. The confidence score may be compared to a threshold. In response to the confidence score exceeding the threshold, the type of anatomical joint may be automatically classified. A second machine learning model may be automatically selected using the classified type of anatomical joint. Multiple confidence scores may be generated to indicate the classification accuracy of classifying the type of anatomical joint in the 3D CT volume into unilateral hip joint, unilateral knee joint, unilateral ankle joint, bilateral hip joint, bilateral knee joint, or bilateral ankle joint, respectively. The highest confidence score among the multiple confidence scores may be identified. Based on the highest confidence score, the type of anatomical joint may be automatically classified. A second machine learning model can be automatically selected using the classified types of anatomical joints. The classified types of anatomical joints can be used to automatically select a second machine learning model in response only to the automatic classification of the type of anatomical joint in the 3D CT volume as one of unilateral hip joint, unilateral knee joint, or unilateral ankle joint. In response to the automatic classification of the type of anatomical joint in the 3D CT volume as unilateral hip joint, a second machine learning model specially trained to classify the unilaterality of the unilateral hip joint as the left hip joint or the right hip joint may be selected. In response to the automatic classification of the type of anatomical joint in the 3D CT volume as unilateral knee joint, a second machine learning model specially trained to classify the unilaterality of the unilateral knee joint as the left knee joint or the right knee joint may be selected.In response to the automatic classification of the anatomical joint type as a unilateral ankle joint in the 3D CT volume, a second machine learning model specially trained to classify the unilaterality of the unilateral ankle joint as a left ankle joint or a right ankle joint may be selected. A confidence score representing the classification accuracy of the unilaterality of the anatomical joint in the 3D CT volume may be generated. The confidence score may be compared to a threshold. In response to the confidence score exceeding the threshold, the unilaterality of the anatomical joint may be automatically classified. The classified unilaterality of the anatomical joint may be used to automatically generate computer-generated output that identifies the classified unilaterality. The first confidence score may represent the classification accuracy where the unilaterality of the anatomical joint is a left joint. The second confidence score may represent the classification accuracy where the unilaterality of the anatomical joint is a right joint. The highest confidence score among the first and second confidence scores may be identified, and the unilaterality of the anatomical joint may be automatically classified based on the highest confidence score. A confidence score representing the classification accuracy for classifying the type of target anatomical part in medical image data may be generated, and in response to a determination that the confidence score does not meet an acceptable threshold, the medical image data may be automatically rejected as unacceptable for facilitating surgical planning. A confidence score representing the classification accuracy for classifying the laterality of the target anatomical part in medical image data may be generated, and in response to a determination that the confidence score does not meet an acceptable threshold, the medical image data may be automatically rejected as unacceptable for facilitating surgical planning. The first and second machine learning models may be deep learning models including convolutional neural networks.

[0025] Medical imaging devices or automated imaging suites may be implemented in various ways, such as Software as a Medical Device (SaMD, software intended to be used for one or more medical purposes that performs these purposes without part of a hardware medical device), Software as a Service (SaaS, access to software via the Internet without downloading), etc. Imaging data may include any type of imaging data, including but not limited to 2D or 3D CT scans or volumes, X-rays, fluoroscopic imaging data, ultrasound imaging data, digitally reconstructed radiographic images (DRR), etc. Medical imaging data may have been previously classified or may not have been classified before being processed using the techniques described herein.

[0026] The target anatomical part, shape model, implant shape model, or extrapolation of the medical image may be used in any modality. The medical image data may be modified to visualize the shape model, including a portion of the shape model that extends beyond the boundaries of the medical image data. An extrapolation region may be created that includes the portion of the shape model that extends beyond the boundaries of the medical image data. The extrapolation may be adjacent to the boundaries of the medical image data, or the medical image data may be extended to include the extrapolation region. Within the extrapolation region, features of the target anatomical part located in the portion of the shape model that extends beyond the boundaries of the medical image data may be artificially reproduced. Additionally, a soft tissue region adjacent to the portion of the shape model that extends beyond the boundaries of the medical image data may be artificially reproduced. Color coding and / or annotations may be used to identify the portion of the shape model that extends beyond the boundaries of the medical image data.

[0027] Any of the examinations described herein may be used on images representing any type of anatomical part, and such examination may be used to evaluate and / or classify any type of anatomical part including but not limited to: bone; bones; soft tissue; ligament; cartilage; osteophyte; shoulder joint bones or features such as the glenoid cavity of the scapula, the scapula or humerus, or any part thereof; vertebrae such as vertebral bodies, discs or cartilage, pedicles, or any part thereof; knee joint bones such as the femur, tibia, patella, ligament, cartilage, or any part thereof; hip joint bones such as the femur, acetabulum, pelvis, cartilage, ligament, or any part thereof; skull; facial bones; ankle bones; wrist bones; bone fragments or fractured parts. Other types of anatomical parts are also considered. Additionally, any of the examinations described herein may be used on images containing any type of foreign body or existing implant, and such examination may be used to evaluate and / or classify by any examination. Brief explanation of the drawing

[0028] The advantages of the present invention will be more easily understood by referring to the following detailed description in conjunction with the accompanying drawings. FIG. 1 is a block diagram of an image processing workflow including evaluating medical image data through an automated image inspection suite according to one embodiment. Figure 2 illustrates an exemplary screenshot of a graphical user interface that can be used through an automatic image inspection suite. FIG. 3 is a method flowchart illustrating exemplary steps of automatic image parameter pre-inspection according to one embodiment. FIG. 4 is a method flowchart illustrating exemplary steps of an automatic image viewing pre-inspection according to one embodiment. FIG. 5 is an exemplary diagram of medical image data of a target anatomical part, illustrating that an automated image viewing pre-examination evaluates shape models or anatomical landmarks on the boundaries of the medical image data. FIG. 6 is a method flowchart illustrating exemplary steps of an automatic plan pre-inspection according to one embodiment. FIG. 7 illustrates an example of medical image data related to a target anatomical part according to one embodiment, wherein an automatic planning pre-examination evaluates a shape model or landmark on the boundary of the medical image data. FIG. 8 is a side view of an exemplary implant including corresponding measurements that can be used in an automatic planning pre-inspection according to one embodiment. FIG. 9 is a method flowchart illustrating exemplary steps of an automatic image classification pre-inspection according to one embodiment. FIG. 10 is a method flowchart illustrating exemplary steps of automatic motion pre-inspection according to one embodiment. FIG. 11 illustrates a combination of a method flowchart and a block diagram showing example components and steps of an automatic volume classifier and a unilateral pre-inspection according to one embodiment. FIG. 12 is a block diagram of an image processing workflow including evaluating medical image data through an automated image inspection suite according to another embodiment. FIG. 13 illustrates an exemplary screenshot of a graphical user interface that can be used to provide an image inspection report in any aspect of an automatic image inspection suite according to one embodiment. FIG. 14 illustrates another exemplary screenshot of a graphical user interface that can be used to provide an image inspection report in any aspect of an automatic image inspection suite according to one embodiment. FIG. 15 illustrates an exemplary screenshot of a graphical user interface that can be used to provide a case file dashboard including image inspection status updates in an automatic image inspection suite according to one embodiment. Specific details for implementing the invention

[0029] This specification describes a system, computer-implemented method, software program, non-transient computer-readable medium and / or technique for automatically evaluating medical image data.

[0030] Referring to FIG. 1, an Automatic Imaging Inspection Suite (AICS) is provided for implementing the automatic evaluation of medical imaging data (MID). The term “Automatic Imaging Inspection Suite (AICS)” is used merely to organize the various aspects of the system / method / software / technique described herein for the sake of simplicity in drafting. The Automatic Imaging Inspection Suite (AICS) may be implemented on a local computer and / or on a remote server, such as a cloud computing server. An aspect or function of the Automatic Imaging Inspection Suite (AICS) may be implemented on any number of controllers, processors, computers, software modules, or servers. The Automatic Imaging Inspection Suite (AICS) may include non-transient memory for storing instructions to execute software, modules, or programs capable of performing any aspect of the aspects described herein when executed by one or more processors. The processes described herein may be implemented by any control system, one or more controllers, or any one or more processing devices. Any process among the processes described herein does not need to be performed explicitly or implicitly by software designed or named as the “Automatic Imaging Inspection Suite.” For example, this software can be called a "medical imaging scanner."

[0031] In FIG. 1, an exemplary workflow including an Automated Imaging Suite (AICS) is illustrated. A hospital, institution, or surgeon generates a treatment planning request for one or more patients. The treatment planning request may be transmitted using any suitable technique or medium. In one example, upon the treatment planning request, the Automated Imaging Suite (AICS) receives the patient's medical imaging data (MID). As described below, in another example, the treatment planning request may be generated after it has been successfully processed by the Automated Imaging Suite (AICS). The medical imaging data (MID) may be provided simultaneously with or separately from the treatment planning request. For example, the Automated Imaging Suite (AICS) may receive the imaging data (MID) from a Picture Archiving and Communication System (PACS) server or any centralized computing system so that medical professionals can share patient medical images and reports from multiple locations. The imaging data (MID) may be transmitted to the Automated Imaging Suite (AICS) using any suitable method, such as transmission over the Internet or a physical connection via a data storage device such as a flash drive. Image data (MID) can be downloaded or retrieved in batches for many patients, or downloaded or retrieved for individual patients as needed.

[0032] Image data (MID) can be transmitted to an Automated Imaging Inspection Suite (AICS) using any suitable method, such as transmission over the Internet or a physical connection via a data storage device like a flash drive. Image data (MID) can be downloaded or retrieved in batches for many patients, or downloaded or retrieved individually for each patient as needed.

[0033] Image data (MID) relates to target anatomical parts (TA) for one or more patients. The techniques described herein are fully applicable to any type of target anatomical part (TA) that may require surgical planning, such as bone, soft tissue, etc. For example, the target anatomical part (TA) may be the hip joint or hip bone, the knee joint or knee bone, the shoulder joint or shoulder joint bone, the ankle joint or ankle joint bone, a vertebra or vertebrae, the skull, etc. Surgical planning is intended to facilitate any type of surgery, including orthopedic surgery such as, for example, partial or total knee replacement, partial or total hip replacement, partial or total shoulder replacement, anatomical or reverse shoulder joint surgery, arthrodesis, arthroscopic surgery, arthroplasty, discectomy, laminectomy, disc arthroplasty, trauma or fracture repair surgery, wrist repair surgery, ankle repair surgery, craniofacial surgery, cardiac surgery, oncology surgery, dental surgery, etc. The surgical planning process may be used to plan the implantation of any suitable type of implant or prosthesis required for surgery. For example, implants or prostheses may include hip and knee implants, including single-compartment, double-compartment, or total knee implants, femoral stem implants, acetabular cup implants, orthopedic screws, intervertebral screws, orthopedic plates, etc.

[0034] Image data (MID) may include any type of image data, including but not limited to CT scans or volumes, X-rays, fluoroscopic image data, MRI scans, PET scans, ultrasound image data, digitally reconstructed radiographic images (DRR), single-photon emission computed tomography (SPECT) images, arthrography, etc. Image data (MID) may also be any suitable file format such as Analyze, NIFTI (Neuroimaging Informatics Technology Initiative), MINC, and DICOM (Digital Imaging and Communications in Medicine). Image data (MID) may be 3D volume image data or any number of 2D image cross-sections.

[0035] In addition to the visual image of the target anatomical part (TA), the image data (MID) may additionally include data or metadata. For example, the data may include text or numeric information related to the patient or the image data (MID). Any of the text or numeric information may be included in the image scan itself and / or in an electronic file provided with the image scan. In one example, the image data (MID) may include data related to scanner configuration settings, including but not limited to radiation dose, detector configuration (beam collimation), pixel size, resolution, window width, window level, number of image sections, image section thickness, image density, intensity, automatic exposure control (AEC) parameters such as the noise figure or mA level during tube current modulation, signal-to-noise ratio, tube voltage (kV), gantry rotation, patient positioning, scan range, image section thickness and pitch, scan time, etc. In addition, imaging data (MID) may include patient-specific information or surgical information (e.g., pre-operative), including but not limited to the patient's name, date of birth, gender, the patient's planned surgical side, data related to treatment requests, the surgeon's name, the type of surgery to be performed, the type of planned implant, and the scheduled date of surgery.

[0036] As illustrated in FIGS. 1 and 2, the Automatic Imaging Inspection Suite (AICS) may be implemented through a graphical user interface (GUI) that may optionally be provided to a visualization function or any suitable display device (DD) (Fig. 2). The GUI may be a data review screen that displays any suitable information related to the imaging data, including but not limited to image data of various planes or imaging sections (MID), image section thickness, image section interval, field of view of the displayed image data, scan parameters (e.g., pixel size, resolution, voltage, current, scan date), patient information (e.g., name, date of birth, gender), the patient's planned operating side, data related to treatment requests, the name of the surgeon, the type of surgery to be performed, the type of implant planned, the scheduled date of surgery, etc. The GUI may optionally be used to allow a technician to review the data described above, review the output results of automatic functions implemented in the Automatic Imaging Inspection Suite (AICS), visualize the output results of the segmentation or anatomical model, or perform additional manual review (before or after the execution of automatic functions implemented in the Automatic Imaging Inspection Suite (AICS)).

[0037] A. Video Pre-test

[0038] Referring to FIG. 1, an automated image inspection suite (AICS) may perform one or more automated image pre-inspections on image data (MID). As used herein, the term “pre-inspection” defines an initial inspection performed on received image data (MID) prior to any subsequent surgical planning (e.g., segmentation, anatomical model generation, implant planning) or any aspect of a surgical workflow that relies on surgical planning or medical imaging (e.g., surgical navigation visualization, anatomical registration, etc.). To this end, the automated image inspection suite (AICS) is configured to evaluate medical image data prior to surgery, that is, prior to the virtual surgical planning process and prior to the execution of surgery.

[0039] As described in the following sections, the Automatic Image Inspection Suite (AICS) is configured to perform any one or more of automatic pre-inspections, including Automatic Image Parameter Pre-inspection (IPPC), Automatic Image View Pre-inspection (IVPC), Automatic Planning Pre-inspection (PPC), Automatic Image Classification Pre-inspection (ICPC), Automatic Motion Pre-inspection (MRC), and Automatic Volume Classifier and Unilaterality Pre-inspection (VLC). The Automatic Image Inspection Suite (AICS) may execute any of these automatic image pre-inspections in various ways. In one example, any one or more of the automatic pre-inspections are executed individually or independently of other pre-inspections. Alternatively, automatic pre-inspections may be executed simultaneously, collectively, or together. Additionally, the Automatic Image Inspection Suite (AICS) may execute some aspects of any of the pre-inspections described herein, or combine these partial aspects with the functions of other pre-inspections. Furthermore, any two or more of these automatic pre-inspections may be combined into a single inspection operation. Additionally, any two or more of these automatic pre-inspections may be performed according to priority or in any order. In some cases, the Automated Imaging Inspection Suite (AICS) can selectively determine which tests to perform and which not to perform during automated imaging pre-inspection. For example, the Automated Imaging Inspection Suite (AICS) may make such a decision in response to automatically identifying that specific patient information is missing from the imaging data (MID). The priority of tests may be predetermined based on the statistical or predicted probability that these tests will identify errors. The Automated Imaging Inspection Suite (AICS) may output a single report containing information on the results of any test or an integrated report containing information on the results of multiple tests.Furthermore, the present specification describes a technique for providing means to rescue scan data that would have failed to pass each examination by, for example, by extrapolating a medical image and / or a target anatomical part beyond the original boundaries of the medical image, for example, by failing to capture an essential part of the target anatomical part.

[0040] By utilizing these pre-inspections, the Automated Imaging Inspection Suite (AICS) can automatically, rapidly, and accurately identify defects or errors in imaging data (MID). This allows the AICS to automatically determine whether the imaging data (MID) is suitable for subsequent surgical planning (e.g., segmentation, anatomical model generation, implant planning) or for modes of surgical workflows that rely on surgical planning or medical imaging (e.g., surgical navigation visualization, anatomical registration, etc.). Since the AICS can process the described pre-inspections within seconds, it can reduce the time-consuming burden on surgical planning teams of manually reviewing imaging data (MID) to identify potential errors. As a result, the AICS can significantly reduce labor costs and human error associated with the manual review of imaging data.

[0041] 1. Automatic image parameter pre-check

[0042] Referring to FIG. 3, the Automatic Imaging Inspection Suite (AICS) can perform Automatic Imaging Parameter Pre-check (IPPC), that is, perform a pre-check to determine whether image data (MID) has been acquired with acceptable image parameters or scanner configuration settings. Acceptable image parameters may be parameters considered satisfactory by the vendor of the Automatic Imaging Inspection Suite (AICS) or acceptable according to industry standards. Settings may include, but are not limited to, radiation dose, detector configuration (beam collimation), pixel size, resolution, window width, window level, image density, intensity, Automatic Exposure Control (AEC) parameters such as the noise figure or mA level during tube current modulation, signal-to-noise ratio, tube voltage (kV), gantry rotation, patient positioning, scan range, image cross-sectional thickness and pitch, scan time, etc. The Automatic Imaging Inspection Suite (AICS) may include a list of values ​​or ranges for acceptable image settings. These values ​​may be stored, for example, in a lookup table. FIG. 3 illustrates an exemplary method (100) for performing an image parameter pre-inspection (IPPC), which includes the step (102) of receiving image data (MID) of a target anatomical part (TA). The received image data (MID) contains image parameters, which may be encoded, for example, in a DICOM file. For this inspection, the received image data (MID) may or may not include actual image scans of the patient. In step (104), an automated image inspection suite (AICS) may automatically identify and extract image parameters from the received image data (MID). This process may include text or string search or matching algorithms, etc. In step (106), the automated image inspection suite (AICS) may automatically compare the extracted image parameters with stored acceptable image parameter values.

[0043] In step (108), a comparison may be performed to determine whether the extracted value is an acceptable value or falls within a predetermined range or threshold. If the extracted value is acceptable, in step (110), the Automatic Image Parameter Pre-check (IPPC) may automatically generate a response to verify the acceptability. In one example, the response is to send a confirmation message to the GUI so that a technician can see that this pre-check has passed or that the settings are correct. In another example, the response is for the Automatic Image Inspection Suite (AICS) to continue processing additional pre-checks (e.g., a series of sequential checks). Other responses are also considered, such as not responding unless an error is detected.

[0044] If the extracted value is unacceptable, the automatic image parameter pre-check (IPPC) of step (112) may automatically generate a response regarding the unacceptability. In one example, the response is to send a warning or notification to the GUI so that a technician can see that this pre-check failed or identify incorrect setting(s). In another example, the response is for the automatic image check suite (AICS) to stop processing additional pre-checks (e.g., a series of sequential checks). The response may also be to output a message requesting a new scan with appropriate parameters.

[0045] 2. Automatic video viewing pre-check

[0046] Referring to FIGS. 4 and 5, the automated image inspection suite (AICS) can perform an image view pre-inspection (IVPC). The image view pre-inspection (IVPC) can automatically determine whether the image of the image data (MID) contains essential details (e.g., bone) of the target anatomical part (TA); whether the image size is insufficient; whether the target anatomical part (TA) is acceptablely captured within the image view; whether the target anatomical part (TA) is cropped; and / or whether any essential part of the target anatomical part extends beyond the boundaries of the image (MID). For each patient, the image view pre-inspection (IVPC) may be repeated for any number of image sections or all image sections of the medical image data (MID) containing the patient's target anatomical part (TA).

[0047] Intravenous Pre-Viewing Tests (IVPC) may be relevant to surgical planning, such as for the purpose of creating segments or anatomical models where it is important to ensure that essential parts of the target anatomical region are captured in the imaging data. Otherwise, the segments or anatomical models may be incomplete or contain errors. IVPC may also be relevant for navigation purposes. For example, ensuring that essential parts of the target anatomical region are captured in the imaging data guarantees the visualization accuracy of the imaging data that the surgeon can refer to during navigation surgery, and guarantees the accuracy of the anatomical registration process that aligns the anatomical model with the actual target anatomical region (TA).

[0048] An exemplary method (200) for performing an automated image viewing pre-examination (IVPC) is illustrated in FIG. 4, which includes the step (202) of receiving image data (MID) of a target anatomical part (TA). In step (204), an automated image examination suite (AICS) automatically identifies the target anatomical part (TA) in the medical image data (MID) and fits a shape model (SSM) to the target anatomical part. As used herein, the term “shape model” includes any one or more of a statistical shape model, an active shape model, an active appearance model, an active contour model, or any other suitable type of shape model. In one example, the shape model (SSM) may be understood as a mesh, shape, or contour having adjustable nodes capable of adjusting the mesh, shape, or contour to substantially correspond to the shape or contour of the target anatomical part (TA) in the medical image data (MID). The shape model may be initially derived or generated from a group of other patient images having anatomical parts similar to the target anatomical part (TA). This group may possess characteristics similar to the target patients (e.g., age, sex, race, physique, and other physical or demographic data). For example, if the target anatomical part (TA) is bone, the shape model (SSM) may be derived from a statistical representation of bone images of similar anatomical origin obtained from a group of patients known to have normal or "healthy" bone structures. The shape data used to derive the shape model (SSM) may include the geometric characteristics of the bone, such as landmarks (AL), surfaces, boundaries, geometric features, or intensity information of the target anatomical part (TA). The shape data may be provided from point clouds acquired from other patient image analyses and / or normal bones of similar anatomical origin. The Automated Imaging Inspection Suite (AICS) may select the optimal shape model for the target anatomical part from among multiple shape models.The optimal shape model may or may not be modified to fit the target anatomical part. Alternatively, the Automated Imaging Inspection Suite (AICS) may use a single general shape model (for a specific anatomical type) and modify it to fit the target anatomical part. In any embodiment described herein, the shape model may be a single shape model or may be implemented as multiple shape model instances.

[0049] A shape model (SSM) can be utilized using an algorithm that automatically segments medical image data (MID). In one embodiment, the SSM may represent "healthy" anatomical parts that may not exactly correspond to the target anatomical part of the subject patient. The segmentation algorithm may perform image processing, such as aligning the coordinates of the target anatomical part (TA) in the image data to the SSM. This alignment may be based on key marker points of the target anatomical part (TA) in the image data and the SSM. The segmentation algorithm may then deform, adjust, or scale the SSM until the SSM and the target anatomical part (TA) in the image data are aligned. This alignment process may be performed using any optimization technique, such as at least squared optimization. The alignment process may include adjusting the size and shape of the SSM to fit the target anatomical part (TA) in the image data and adjusting the position of the SSM to align with the target anatomical part (TA) in the image data. The registration result may be a shape model that approximates the target anatomical part (TA) in the image data (e.g., optionally including or excluding osteophytes or abnormal shapes).

[0050] In another embodiment, the automatic segmentation algorithm performs an initial segmentation process using a first shape model on image data associated with a target anatomical part (TA) to generate an initial segmentation result of the target anatomical part (TA). The automatic segmentation may optionally further perform a fine segmentation process on an image region of the image data associated with the target anatomical part (TA) using a neural network that takes as input the image data of the target anatomical part (TA) and the output of the initial segmentation result. The first shape model may be mapped to the output of the fine segmentation process. Optionally, anatomical landmarks (ALs) of the target anatomical part (TA) may be identified. An example of such automatic segmentation may be implemented in the manner described in U.S. Provisional Patent Application No. 63 / 505,466 (Application Date: June 1, 2023, Title of Invention: "Segmentation of Bony Structures", Agent Document No. 060939.01029, the entire contents of which are incorporated herein by reference).

[0051] As illustrated in FIG. 5, a shape model (SSM) (derived from any of the techniques described above) is represented as a shape that approximates the contour of a target anatomical part (TA) (e.g., knee bone(s)) in medical imaging data (MID). Visualizing the shape model (SSM) for the image is optional and is not essential for this automated pre-inspection. Where useful, a technician can manually search for and review any shape model for any corresponding image in the GUI.

[0052] As illustrated in FIG. 5 and step (206) of FIG. 4, the automatic image viewing pre-inspection (IVPC) performs an automatic comparison between the shape model (SSM) and the boundary (MIB) of the medical image data (MID). The parameters of the boundary (MIB) of the medical image data (MID) can be derived from scanner configuration settings or image parameters and / or by automatic measurement of the boundary (MIB) size. In most cases, but not always, the boundary (MIB) of the medical image data (MID) is rectangular.

[0053] In one embodiment, the comparison is a boundary-to-boundary comparison, that is, a comparison between the boundary (SSM-B) of the shape model (SSM) and the boundary (MIB) of the medical image data (MID). The shape model boundary (SSM-B) may be determined from the parameters or size of the shape model (SSM) and / or by automatic measurement of the boundary (SSM-B). In some cases, the shape model (SSM) and the medical image data (MID) are compared by being aligned to a common coordinate system, which may be the image, the shape model's coordinate system, or any coordinate system. The coordinate system in which these boundaries are measured may be larger than the boundary (MIB) of the medical image data (MID) so that the shape model (SSM) can be detected beyond the image boundary (MIB). The boundary comparison may be performed based on all or some boundaries of the shape model (SSM). Any aspect of this comparison may be a backend (unvisualized process) and is not necessarily visualized to the user.

[0054] In another embodiment, instead of boundary-to-boundary evaluation, an automated image viewing pre-check (IVPC) performs a point-to-boundary comparison. For example, one or more anatomical landmarks (ALs) of the target anatomical part (TA) identified during the automated segmentation process can be mapped to a shape model (SSM) (as illustrated in FIG. 5). The automated image viewing pre-check (IVPC) evaluates whether the landmarks (ALs) are within the image boundary (MIB) or extend beyond this area boundary. In one example, the landmarks (ALs) may be derived from clinical data identifying landmarks that need to be shown for surgical planning and execution. In another example, the landmarks (ALs) may be derived from points required to perform anatomical registration during intraoperative navigation surgery. Additionally, or alternatively, the anatomical landmarks (ALs) may be one or more points automatically selected based on the likelihood of the landmarks (ALs) extending beyond the image boundary (MIB). For example, the landmarks (ALs) may be the most anterior, posterior, medial, lateral, superior, or inferior points of the target anatomical part (TA) structure. Alternatively, instead of a shape model, Automated Intravenous Pre-screening (IVPC) can identify anatomical landmarks (ALs) of the target anatomical part required to be visible in the medical imaging data (MID) by comparing the medical imaging data (MID) with a statistical population of medical images containing other anatomical parts similar to the target anatomical part. Automated Intravenous Pre-screening (IVPC) can automatically evaluate the medical imaging data (MID) to determine whether the required anatomical landmarks (ALs) are visible in the medical imaging data (MID).

[0055] Point-to-point evaluation is also considered. The shape of the SSM can be interpolated using multiple points with coordinates in a coordinate system. Medical imaging data (MID) can be interpolated into a grid of points or pixels. Automated Intravenous Pre-screening (IVPC) can examine whether the coordinates of points in the SSM correspond to or overlap with the coordinates of points or pixels in the medical imaging data (MID). Automated Intravenous Pre-screening (IVPC) can combine any of the described techniques to compare the SSM and the medical imaging data (MID).

[0056] In step (208), a boundary comparison is used to determine whether the medical image data (MID) is acceptable. In the example of FIG. 5, the automated image view pre-inspection (IVPC) identifies that the right outer contour of the shape model boundary (SSM-B) extends beyond the image boundary (MIB). Thus, the automated image view pre-inspection (IVPC) detects one or more of the following: the image does not contain essential details of the target anatomical part (TA); the image size is insufficient; the target anatomical part (TA) is not fully captured within the image view; the target anatomical part (TA) is cropped; and / or essential parts of the target anatomical part extend beyond the boundaries of the image (MID).

[0057] If, as a result of such comparison, the medical imaging data (MID) is determined to be unacceptable, the automated imaging inspection suite (AICS) may automatically generate a response regarding the unacceptability in step (210). In one example, the response may send a warning or notification to the GUI so that a technician can see that this preliminary inspection has failed or identify the failed image section, or optionally display an SSM and each image section indicating the boundary comparison failure in the GUI. Additionally, the response may be to output a message requesting the patient to undergo a larger scan. In another example, the response is for the automated imaging inspection suite (AICS) to stop processing additional preliminary inspections (e.g., a series of sequential inspections).

[0058] Meanwhile, medical image data (MID) may be determined to be acceptable based on the results of this comparison. For example, in the example of FIG. 5, assuming that the right outer contour of the shape model boundary (SSM-B) is (virtually) captured within the image boundary (MIB), the automated image view pre-check (IVPC) detects one or more of the following: whether the essential details of the target anatomical part (TA) are successfully included in the image; whether the image size is sufficient; whether the target anatomical part (TA) is fully captured within the image view; whether the target anatomical part (TA) is not cropped; and / or whether the essential parts of the target anatomical part are included within the boundaries of the image (MID). If acceptable, the automated image check suite (AICS) may automatically generate a response to confirm the acceptability in step (212). In one example, the response is to send a confirmation message to the GUI so that a technician can see that this pre-check has passed. In another example, the response is for the automated image check suite (AICS) to continue processing additional pre-checks (e.g., a series of sequential checks). Unless an error is detected, other responses, such as not responding at all, are also considered.

[0059] 2A. Anatomical Parts / Image Extrapolation

[0060] Referring to FIGS. 4 and 5, the automated image inspection suite (AICS) and / or image viewing pre-inspection (IVPC) may be capable of extrapolating or extending the target anatomical part (TA) and / or image (MID) by prediction. For example, if it is determined that the required portion of the target anatomical part (TA) is not captured in the image (MID), the automated image inspection suite (AICS) may artificially extend the target anatomical part (TA) beyond the original boundary (MIB) of the image (MID). Although the required length of the target anatomical part (TA) in the image (MID) was evaluated using a shape model (SSM) above, the same shape model (SSM) or other shape models may be additionally used to extend the anatomical part to the length or size required to capture it.

[0061] To implement this function, the Automatic Image Inspection Suite (AICS) automatically identifies a target anatomical part (TA) in the medical image data (MID) and fits a shape model (SSM) to the target anatomical part (step (204) in FIG. 4). As illustrated in FIG. 5 and step (206) in FIG. 4, an automatic comparison is performed between the shape model (SSM) and the boundary (MIB) of the image (MID). In step (208), the comparison determines that the shape model (SSM) (or part thereof) extends beyond the boundary (MIB) (otherwise it would be unacceptable). For example, in the example of FIG. 5, the inspection determines that the right outer contour portion of the shape model boundary (SSM-B) extends beyond the image boundary (MIB). However, instead of rejecting the image (MID) in step (210), the image (MID) can be preserved using an extrapolation technique, so there is no need to rescan the image (MID). That is, in step (220), the image (MID) can be automatically annotated, modified, duplicated, or reproduced to include information about the shape model (SSM), in particular, information about the portion of the shape model that extends beyond the image boundary (MIB). This extrapolation can be performed so that the image (MID) can be automatically approved (in step (212)). Again, the shape model (SSM) used to inspect the boundary limits (in steps (204, 206, 208)) may be the same or different from the shape model (SSM) used for extrapolation (in step (220)).

[0062] To capture portions of the shape model that extend beyond the image boundary (MIB), the Automated Image Inspection Suite (AICS) can create or identify an extrapolation region (ER) as illustrated in FIG. 5. In one example, the extrapolation region (ER) may be an artificial extension of the image (MID). The extrapolation region (ER) may modify the size of the boundary (MIB) or be identified as a region distinct from the original image (MID). Within the extrapolation region (ER), portions of the shape model (SSM) may be visually preserved to mimic essential parts of the target anatomical part (TA). Later, when retrieving the image for planning purposes, the image (MID) may show, for example, that the boundary (SSM-B) of the shape model (SSM) extends beyond the original boundary (MIB) of the image data (MID) and optionally, for example, into the extrapolation region (ER).

[0063] In some cases, the target anatomical part (TA) within the extrapolation region (ER) can be automatically filled with features of the anatomical part (TA) derived from the shape model (SMM) or other statistical data. For example, instead of displaying only the boundary (SSM-B) of the shape model (SSM) in the extrapolation region (ER), the area inside the boundary (SSM-B) can be artificially filled with anatomical features, such as bone features, that would have been captured if the image size had been larger at the initial scan. Similarly, the tissue surrounding the target anatomical part (TA) in the extrapolation region (ER) can be artificially filled with anatomical features, such as surrounding soft tissue, that would have been captured if the image size had been larger at the initial scan. In some cases, tissue or bone regions within the extrapolation region (ER) can be artificially filled using scan data from adjacent image sections that may contain missing features.

[0064] Additionally or alternatively, the entire image (MID) can be artificially reconstructed or regenerated to include the extrapolated region (ER). Using this technique, there is no visible boundary between the boundary of the image (MID) (MIB) and the extrapolated region (ER). Instead, these two regions are seamlessly merged into a single image (MID). As a result, the image (MID) appears to include all essential parts of the target anatomical region (TA), just as if it were originally scanned.

[0065] The extrapolation area (ER) or features within the extrapolation area (ER) can be distinguished using arbitrary text or visual indicators. For example, parts of the feature model (SSM) can be indicated via annotations, labels, or messages as having been extrapolated, predicted, or not present in the initially scanned image (MID). Visually, parts of the feature model (SSM) within the extrapolation area (ER) can be color-coded to be distinct from the rest of the image or the feature model (SSM). Using this identification, reviewers can recognize that the target anatomical part (TA) within this area (ER) is extrapolated and may not perfectly represent the actual scanned anatomical part. The further the extrapolation area is from the original image boundary (MIB), the greater the likelihood of prediction errors. Therefore, the image (MID) may include step-level text or visual indicators indicating that the likelihood of error increases as it moves further away from the original image boundary (MIB). For example, in the extrapolation region (ER), a stepwise color scheme can be used to highlight the target anatomical part closest to the original image boundary (MIB) in green, transition from green to orange, and finally change to red to highlight the target anatomical part furthest from the original image boundary (MIB).

[0066] The described extrapolation technique can provide significant advantages in surgical procedures where medical imaging data often does not include essential parts of the target anatomical part (TA). For example, in the case of revision total knee arthroplasty (TKA), planning a stem longer than 50 mm requires a longer CT scan, but the actual CT scan length can be highly variable. The described extrapolation technique can extrapolate the necessary additional length based on the shape model (SM) using the current CT scan image. For TKA revision, the stem length can be 50 mm, 100 mm, or 150 mm. To plan 100 mm and 150 mm stems, the bone requires scan lengths of 150 mm and 200 mm, respectively. To avoid having two different CT scan protocol lengths and to avoid scan lengths shorter than necessary, the described technique can be used to extrapolate to 100 mm for the primary surgery and to 200 mm for the revision surgery. In this way, CT scans are no longer rejected due to scan length, and patients do not need to undergo additional CT scans.

[0067] This is also useful for short CT scans, even in the case of primary surgery, eliminating the need to reject CT scans based on length. For primary TKA surgery, the CT scan protocol requires the scan to be within ±100 mm of the articular line. Often, the scan is much shorter than this. There are minimum length requirements necessary to generate surfaces for mapping bone alignment points and performing notching calculations for the anterior incision. The minimum scan length depends on the implant size, but the implant size cannot be estimated until the scan is segmented. Occasionally, a scan may be rejected due to a violation of the minimum scan length. If rejected, notification is typically provided more than one day after the scan. In this case, the patient must revisit the CT scanning facility for a rescan. To prevent rejection, the geometry model (SM) is trained to accurately extrapolate the femoral / tibial shaft to ±100 mm, thereby avoiding image rejection and the need for a rescan.

[0068] 3. Automated Plan Pre-check

[0069] Referring to FIGS. 6 through 8, the automated imaging inspection suite (AICS) can perform an automated pre-planning examination (PPC). The primary purpose of the pre-planning examination (PPC) is to evaluate whether medical imaging data (MID) is suitable for a subsequent surgical plan involving an implant to be inserted into a target anatomical part (TA). The implant used in this examination is not originally present in the imaging data (MID) but is compared with the imaging data (MID) for evaluation. The implant may be a planned implant, a proposed implant, an optimal-estimated implant, or a theoretical implant. The pre-planning examination (PPC) may utilize the surgeon's planning data, but it is not intended to replace the surgeon's plan. Instead, the pre-planning examination (PPC) retrieves data related to the implant in advance to evaluate whether the medical imaging data (MID) sufficiently shows the extent of the target anatomical part (TA) required for the implant plan.

[0070] Accordingly, the Planning Pre-test (PPC) can automatically determine whether the images in the imaging data (MID) contain details of the target anatomical parts (TA) (e.g., bone) required for implant planning; whether the image size is insufficient to capture the essential portion of the target anatomical part (TA) when considering the implant; whether the target anatomical part where the implant is to be placed is not captured within the image view; whether the target anatomical part where the implant is to be placed is cropped; and / or whether any essential portion of the target anatomical part required for implant planning extends beyond the boundaries of the image (MID). For each patient, the Planning Pre-test (PPC) may be repeated for any number of image sections or all image sections of the medical imaging data (MID) containing the patient's target anatomical part (TA). The Planning Pre-test (PPC) may be performed as a single test in combination with the Image View Pre-test (IVPC). Additionally, the Planning Pre-test (PPC) may be performed simultaneously for multiple target anatomical parts (TA) (e.g., opposing bones of an anatomical joint) within the medical imaging data (MID).

[0071] An exemplary method (300) for performing an automated planning pre-examination (PPC) is illustrated in FIG. 6, and the method includes the step (302) of receiving image data (MID) of a target anatomical part (TA). In step (304), an automated image examination suite (AICS) automatically performs measurements (AM) of the target anatomical part (TA) found within the image (MID). In one example, this may be performed by identifying landmarks (AL) of the target anatomical part (TA) and measuring the distance between the landmarks (AL). For example, as illustrated in FIG. 7, if the image is of a knee joint, the landmarks (AL) may be distal features of the femur, such as the proximal point of the femur at the boundary (MIB) and the articular condyle. These measurements may represent the length of the femoral shaft visible in the medical image data (MID). Of course, depending on the image, the type of target anatomical part (TA), or the programmed settings, the landmark (AL) can be any appropriate point or surface on the target anatomical part (TA). The Automated Imaging Inspection Suite (AICS) can measure the distance between the landmarks (AL) using a digital ruler. Any appropriate number of landmarks (AL) may be used.

[0072] Additionally or alternatively, in step (304), the automated imaging inspection suite (AICS) automatically acquires measurements (AM) of the target anatomical part by identifying the target anatomical part (TA) in the medical imaging data (MID) and fitting a shape model (SSM) to the target anatomical part. The shape model (SSM) may be identified, created, and / or customized using any of the techniques described above in relation to the method (200). Thus, specific details and various embodiments of the shape model (SSM) are fully included in this section and are therefore not repeated for convenience of explanation. In the example of FIG. 7, the shape model (SSM) is fitted to the target anatomical part (TA), which is the femur. As described above, the shape model (SSM) may extend beyond the boundaries of the image (MID). The parameters of the boundaries of the shape model (SSM) include the measurements of the shape model (SSM). The Automated Imaging Inspection Suite (AICS) can automatically acquire target anatomical measurements (AM) based on SSM parameters and compare these parameters with medical imaging boundaries (MIB).

[0073] Regardless of whether landmarks (AL) and / or shape models (SSM) are used, measurements of the target anatomical part (TA) may include any one or more measurement(s), such as length, width, area, volume, perimeter, height, depth, thickness, orientation, position, varus, valgus, version, retroversion, inclination, etc. In FIG. 7, the recorded measurement (AM) is the length from the articular eminence to the image boundary (MIB), which is 87 mm in this example and partially represents the length of the femoral shaft.

[0074] The method (300) includes a step (306) of identifying an implant or implant measurement (SI) based on the target anatomical measurement (AM) acquired in step (304). In other words, the implant or implant measurement (SI) used in this step may be a planned implant, a proposed implant, an optimal-estimated implant, or a theoretical implant. In one example, an automated imaging suite (AICS) may automatically select an implant or implant measurement (SI) from a database that has parameters of appropriate size for the target anatomical part (TA) based on the target anatomical measurement (AM). The implant or implant measurement (SI) selected from the database may be represented as a virtual model or SSM of the implant. Alternatively or additionally, the selected implant or implant measurement (SI) may include measurement data without any graphic or virtual elements. Based on the target anatomical measurement (AM), the automated imaging suite (AICS) may select an implant having appropriate size, orientation, biometric or mechanical compatibility, type, or configuration. The implant or implant measurement may be selected based on the manufacturer, model, size identifier, and / or anatomical measurement.

[0075] According to the first technique, in step (308), the automated imaging inspection suite (AICS) may determine parameters of essential features (RF) of the target anatomical part based on the implant or implant measurement (SI) identified (from step (306). In one example, the essential features (RF) define the essential parts, measurements, ranges, or landmark(s) of the target anatomical part that must be shown in the medical imaging data (MID) based on the selected implant (SI). For example, if the selected implant is a size 6 femoral knee joint component, the implant may include data defining the geometric measurements of the implant (e.g., AP dimension, ML dimension, protrusion length, etc.). The selected implant (SI) may also be required to include the femoral shaft length necessary to adequately accommodate the selected femoral component (in imaging space). For example, as illustrated in FIG. 8, an exemplary selected implant (SI) is shown in a sagittal view only for exemplary purposes. The selected implant (SI) may be represented as numeric or text data and is not required to be displayed. In this example, the selected implant (SI) is the femoral component of the knee prosthesis derived from the target anatomical measurement (AM). The selected implant (SI) includes an implant measurement (IM), such as the entire length of the implant from the proximal end of the anterior flange to the most distal point of the articular condyle contact surface. In one example, the height of the exemplified implant measurement (IM) based on the selected implant size may be 72 mm. Of course, any other measurement may be obtained as needed.

[0076] The implant measurements retrieved from the database for this selected implant (SI) may include not only the 72 mm implant dimension but also parameters of the essential features (RF) of the target anatomical part required for this implant. The parameters of the essential features (RF) of the target anatomical part may be obtained from various sources, such as implant manufacturers, minimum threshold measurements (e.g., a value 20% greater than the implant measurement), implant covering thresholds, safety tolerances, clinical or statistical data, surgeon preferences, etc. In the example illustrated in FIG. 8, the essential features (RF) of the target anatomical part (femur) may be a measurement of 92 mm, which can be understood as a measurement including the length of the femoral shaft required to accommodate the selected implant (SI). Of course, depending on the type of target anatomical part, the type of planned procedure, the specific selected implant measurement, and the specific tolerance of the essential anatomical part, this step (308) may produce different results. Alternatively, as illustrated in FIG. 8, the essential feature may be a point or landmark (RF') defined at the final point of the essential measurement, or a point or landmark defined based on a predetermined point on the femoral shaft.

[0077] Continuing with the first technique, in step (310), the automated imaging inspection suite (AICS) can automatically compare the essential feature (RF) (acquired in step (308)) with the anatomical measurement (AM) and / or medical image boundary (MIB). In one example, this step can be performed by augmenting the target anatomical measurement (AM) initially acquired in step (304). Additionally, or alternatively, step (310) may include a step of re-measuring the target anatomical part (TA) in the image (MID) based on the essential feature (RF). For example, the automated imaging inspection suite (AICS) can automatically align or correlate the reference coordinates from which the implant measurement was taken with the corresponding landmark of the target anatomical part on the image data (MID). For example, referring to the example in FIG. 8, the essential feature (RF) can be evaluated in the image (MID) starting from the furthest point of the articular condyle contact surface. Continuing to examine the example, the target anatomical region (TA) was initially measured at 87 mm within the image boundary (MIB), and the measurement of the essential feature (RF) was 92 mm. In Fig. 7, the Automated Imaging Inspection Suite (AICS) automatically measures the essential feature (RF) and determines that the essential feature (RF) extends beyond the image boundary (MIB). Consequently, the image (MID) does not contain the essential 5 mm of the target anatomical region (TA).

[0078] Additionally or alternatively, in other techniques, an automated imaging inspection suite (AICS) may perform an automated planning pre-inspection (PPC) by evaluating selected implants (SI) or implant measurements (IM) for a target anatomical part (TA) in the image. In other words, in this technique, the implant parameters themselves are evaluated, and there is no need to acquire or evaluate essential features (RF) (derived from the implant). Specifically, as illustrated in step (312), the automated imaging inspection suite (AICS) may evaluate whether any part of the identified implant (SI) or implant measurement (IM) extends beyond the medical imaging boundary (MIB). In one example, this may be an inspection to determine whether the identified implant (SI) or implant measurement (IM) is located a threshold distance from the imaging boundary. The threshold distance may be based, for example, between the implant measurement (IM) and the essential feature (RF); between the implant measurement (IM) and the anatomical measurement (AM); or between the anatomical measurement (AM) and the essential feature (RF). Alternatively, the critical distance may be a predetermined distance derived, for example, from statistical data. An automated imaging inspection suite (AICS) can automatically perform this comparison, for example, by aligning or correlating the reference coordinates from which the implant measurement (IM) was taken to the corresponding landmarks of the target anatomical part (TA) on the image data (MID). For example, referring to the example in Fig. 8, the implant (SI) or implant measurement (IM) may be evaluated in the image (MID) starting from the most distal point of the articular condyle contact surface (indicated as Landmark (AL)). In some cases, an offset implant distance may be included if the articular surface of the planned implant surface is offset from the original articular surface of the target anatomical part (TA) in the image. Continuing to the example illustrated in Fig. 7, the anatomical measurement (AM) enclosed by the image boundary was 89 mm, and the height of the implant measurement (IM) was 72 mm.Based on these measurements, the Automated Imaging Inspection Suite (AICS) automatically determines that the selected implant (SI) or implant measurement (IM) is within the image boundary (MIB). Therefore, the image (MID) sufficiently includes the essential image of the target anatomical part (TA).

[0079] Additionally or alternatively, to compare the implant or implant measurements with the medical image boundary (MIB), the selected implant (SI) can be represented as an implant shape model (SSM-I). The automated imaging suite (AICS) can fit the implant shape model (SSM-I) to the target anatomical part (TA) in the image (MID) of the planned location. The automated imaging suite (AICS) automatically determines whether any part of the boundary of the implant shape model (SSM-I) exceeds the image boundary (MIB). In the example of Fig. 7, the implant shape model (SSM-I) is captured within the image (MID).

[0080] After applying one or two of the techniques described above, the Automated Planning Pre-check (PPC) (at step (314) in FIG. 6) evaluates whether the medical image data (MID) is acceptable. For example, in the example in FIG. 7, the Automated Image View Pre-check (IVPC) identified that essential features (RF) extended beyond the image boundary (MIB). Assuming this is the only type of check, the Automated Planning Pre-check (PPC) would determine at step (314) that the image is unacceptable. Reasons for rejection include: the image not containing essential details of the target anatomical part (TA) (e.g., bone) required for the implant plan; the image size being insufficient to capture the essential part of the target anatomical part (TA) when considering the implant, or insufficient to capture the essential area for the planned implant; the target anatomical part where the implant is to be placed not being captured within the image view; the target anatomical part where the implant is to be placed being truncated; or the essential part of the target anatomical part required for the implant plan exceeding the image boundary. and / or it can be characterized as a case where the implant planned for the target anatomical part extends beyond the image boundary or is not located at a critical distance from the image boundary.

[0081] If the medical imaging data (MID) is determined to be unacceptable as a result of the automatic planning pre-examination (PPC), a response regarding the unacceptability may be automatically generated in step (316). In one example, the response may send a confirmation message to the GUI so that the technician can see that this pre-examination failed or identify the failed image section, or optionally display each image section and a measurement indicating a boundary comparison failure in the GUI. Additionally, the response may be to output a message requesting the patient to undergo a larger scan. In another example, the response is for the automatic imaging inspection suite (AICS) to stop processing additional pre-examinations (e.g., a series of sequential examinations).

[0082] Meanwhile, medical imaging data (MID) may be determined to be acceptable based on the results of this comparison. For example, in the example of FIG. 7, the shape model implant (SSM-I) and / or implant measurements (IM) are captured within the image boundary (MIB). Assuming this is the only type of inspection, the automated planning pre-inspection (PPC) will determine that the image is acceptable in step (314). Reasons for acceptance may be characterized by: the image containing details of the target anatomical part (TA) (e.g., bone) required for the implant plan; the image being large enough to capture the essential part of the target anatomical part (TA) when considering the implant, or the image being large enough to capture the planned implant; the target anatomical part where the implant is to be placed being captured within the image view; the target anatomical part where the implant is to be placed being not cropped; and / or the planned implant for the target anatomical part not extending beyond the image boundary or being located at a critical distance from the image boundary.

[0083] If acceptable, the automatic planning pre-check (PPC) may automatically generate a response to verify acceptability in step (318). In one example, the response is to send a confirmation message to the GUI so that a technician can see that this pre-check has passed. In another example, the response is for the automatic image inspection suite (AICS) to continue processing additional pre-checks (e.g., a series of sequential checks). Other responses are also considered, such as not responding unless an error is detected.

[0084] 3A. Anatomical Parts / Imaging / Implant Extrapolation

[0085] As described above with reference to Section (2A), the automated planning pre-check (PPC) may have the capability to predictively extrapolate or extend the target anatomical part (TA), implant, and / or image (MID), as similarly illustrated in step (320) of FIG. 6. If it is determined that the essential part of the target anatomical part (TA) required for the implant plan is not captured in the image (MID), the automated image check suite (AICS) may artificially extend the target anatomical part (TA) and / or implant beyond the original boundary (MIB) of the image (MID). This technique may be applied as described above and will not be repeated for the sake of brevity. Such extrapolation of the target anatomical part, implant, and / or image may be performed after the comparison in step (310) (the step of comparing the essential features of the target anatomical part with the medical image boundary) or after the comparison in step (312) (the step of comparing the implant measurements with the medical image boundary). After step (310), the target anatomical part may be expanded using the shape model (SM) as described above. After step (312), the target anatomical part and / or the implant may be expanded using the shape model (SM) and the implant shape model (SSM-I), respectively, as described above.

[0086] This technique is useful for preventing image rejection and avoiding rescanning of images that may fail during pre-planning examinations. For example, in the case of primary or revision total knee arthroplasty (TKA), the described extrapolation technique can take a current CT scan image to consider the desired length of the planned stem (e.g., 100 mm or 200 mm) and extrapolate the necessary additional length of the anatomical shape model (SM) and / or implant shape model (SSM-I).

[0087] 4. Automatic Image Classification Pre-check

[0088] Referring to Fig. 9, the Automatic Image Inspection Suite (AICS) can perform Automatic Image Classification Pre-inspection (ICPC). The primary purpose of the Automatic Image Classification Pre-inspection (ICPC) is to evaluate whether the image data (MID) properly represents what it is intended to represent. If the image data (MID) does not properly represent what it is intended to represent, the image data may negatively affect the accuracy and / or completeness of any subsequent surgical plan (e.g., segmentation, anatomical model generation, implant planning) or the characteristics of surgical workflows that depend on the surgical plan or medical information (e.g., surgical navigation visualization, anatomical registration, etc.). The Automatic Image Classification Pre-inspection (ICPC) efficiently and accurately detects these discrepancies to improve the accuracy and completeness of the subsequent surgical plan.

[0089] An exemplary implementation of an Automatic Image Classification Pre-check (ICPC) is illustrated in FIG. 9. The specific functions or aspects illustrated in FIG. 9 may be optional, and if so, will be described separately. The Automatic Image Classification Pre-check (ICPC) is not strictly limited to the exact steps described. Specific functions of the Automatic Image Classification Pre-check (ICPC) may be implemented independently or without other functions. For example, the Automatic Image Classification Pre-check (ICPC) may perform classification or prediction without performing the described pre-check.

[0090] In step (402), the automated imaging inspection suite (AICS) receives image data (MID) of the target anatomical part (TA). In the example of FIG. 9, the image data is a volumetric (3D) scan, more specifically a CT volumetric scan (402a). However, the image data is not limited to CT scans or volumetric scans. As described above, the image data (MID) may include any type of image or scan, whether 3D or 2D.

[0091] In addition to the CT volume, the imaging data (MID) further includes data (in 402b), namely text or numeric data, metadata, and / or any type of information representing intended parameters related to the patient, the target anatomical part (TA), and / or the imaging data (MID). The intended parameters represent estimated features, modalities, or conclusions related to the target anatomical part scanned in the image. An automated image classification pre-check (ICPC) checks whether these intended parameters are accurate. For example, in FIG. 9, the intended parameter data (402b) includes (1) data related to the intended target anatomical part presumed to have been scanned in the image, e.g., the intended joint type (e.g., knee joint, hip joint, shoulder joint, etc.) or intended joint parameters (e.g., joint geometry, kinematics, kinematics, bone density, disease state); (2) intended image or planning data, e.g., the intended surgical side of the patient's anatomical part where the scan is presumed to have been acquired or the intended surgical side where the patient's surgery is planned (e.g., left joint, right joint, bilateral joint, inner, outer, etc.), (3) intended procedure, e.g., whether the procedure is total joint replacement (e.g., total knee replacement (TKA), total hip replacement (THA), etc.), partial joint replacement (e.g., partial knee replacement, partial hip replacement, etc.), primary surgery or revision surgery; the intended implant for the patient (e.g., total knee implant, partial hip implant, etc.) or the intended manufacturer, model, or size of the implant; the intended presence or absence of metal, foreign material, or implant present in the target anatomical part; and / or the intended type of metal, foreign material, or implant present in the target anatomical part.

[0092] In step (404), the automated imaging suite (AICS) optionally converts or generates one or more digitally reconstructed radiographic images (DRRs) from imaging data (MID) (e.g., CT volume). Each of the one or more DRRs is a 2D digitally simulated projection of the CT volume data. In this step (404), the automated imaging suite (AICS) generates one or more DRRs of the target anatomical part (TA), and optionally, specific features associated with the target anatomical part (TA). For example, the automated imaging suite (AICS) may generate one or more DRRs for (1) bone structures of the target anatomical part (TA), (2) soft tissues on or around the target anatomical part (e.g., skin), and / or (3) metal or foreign material located in the target anatomical part (TA). In this case, DRRs may be generated indiscriminately for each imaging cross-section of the volume data. Alternatively, an Automated Imaging Inspection Suite (AICS) can pre-identify these features using the shape modeling techniques described above and generate DRRs that specifically capture these pre-identified features. For example, bone, soft tissue, or metal can be easily identified using other techniques such as material density (Hu) classification, voxel analysis, brightness or contrast comparison, and region of interest classification.

[0093] Instead of generating DRR in step (404), the automated imaging suite (AICS) is considered to use 2D images (e.g., X-ray or 2D CT cross-sections) to identify specific features associated with the target anatomical part (TA) (e.g., bone, soft tissue, metal, etc.). It is also considered to use 3D volumetric data (e.g., original CT volume) (without reducing the volume to 2D images or DRR). The 3D image data may be downsampled. Although the various steps described below have been explained in the context of DRR, it is understood that the various steps may be implemented using any other type of image data format.

[0094] In step (406), the Automatic Image Inspection Suite (AICS) automatically applies one or more DRRs to a Machine Learning Model (MLM) to further perform Automatic Image Classification Pre-inspection (ICPC). The Machine Learning Model (MLM) analyzes one or more DRRs to automatically classify or identify objects or features (e.g., bone, soft tissue, or metal) within the DRRs. The Machine Learning Model (MLM) may be configured to detect and classify these objects or features using shape or contour recognition, landmark detection, pattern recognition, bounding boxes, or the shape modeling techniques described above. In some cases, the Machine Learning Model (MLM) may perform image segmentation of the DRRs. The DRRs may be normalized before being input into the Machine Learning Model (MLM) to provide a consistent format before classification.

[0095] For example, the machine learning model can be a neural network such as a Convolutional Neural Network (CNN) or an Artificial Neural Network (ANN). To process DRR quickly, the machine learning model can be composed of a lightweight 2D Convolutional Neural Network. The machine learning model can include an arbitrary number of convolutional layers and connection layers. When using a Convolutional Neural Network, the architecture can include multiple nodes composed of layers. DRR is input into the input layer and filtered through various interconnected processing layers. Connections between nodes can be assigned weight values ​​and adjusted based on training data. The output of the neural network is based on the sum of these weight values. Through this process, features within the DRR can be progressively segmented, classified, or discriminated.

[0096] Machine learning models (MLMs) can be trained based on arbitrary medical imaging data, such as medical imaging data related to other patients. For example, training data may be based on imaging data related to one or more of the following: patients with a specific target anatomical part; patients who have undergone a procedure on a specific surgical side; patients showing an anatomical part of a specific disease or a healthy anatomical part; patients with a specific type of procedure; postoperative images of patients with a specific implant type, size, or manufacturer; patients who have undergone revision surgery; patients who have undergone primary surgery; patients with a specific age, gender, ethnicity, weight, and / or height; or similar patients.

[0097] Machine learning models (MLMs) can also use either the described preoperative patient data or intended parameters (provided with medical imaging data) as input. Machine learning models (MLMs) can perform these tasks to accelerate the classification process. For example, if medical imaging data identifies that the medical image is of the knee joint and the planned procedure is total knee replacement, the machine learning model (MLM) can use this information as input to adjust the classification algorithm to use training data based on medical images of the knee joint (e.g., instead of the hip joint). Similarly, other patient data such as age, gender, ethnicity, weight, and height can also be input into the machine learning model (MLM).

[0098] In one example, a convolutional neural network is based on a model switching architecture that selects a model for segmentation based on the input DRR. For instance, if the input DRR is a bone structure, the convolutional neural network may select a model that classifies bone structures. If the input DRR is a soft tissue structure, the convolutional neural network may select a second model that classifies bone structures. If the input DRR is a metal structure, the convolutional neural network may select a third model that classifies metal structures. This model selection technique improves classification accuracy and speed while reducing computational load. Additionally, the convolutional neural network can be adapted to include additional models beyond those described above. For instance, if there is a need to classify existing implant types in the future, a fourth model that classifies implant types can be trained, and the convolutional neural network may select this model based on the DRR representing the implant. In one embodiment, for any given image dataset, three separate DRRs for bone, soft tissue, and metal may be generated, and a machine learning model (MLM) evaluates these three separate DRRs simultaneously. Machine learning models (MLMs) may select one model to evaluate one or more DRRs, or select one or more models to evaluate multiple DRRs. Other types of machine learning models may also be used, such as deep learning models configured to classify the contents of medical imaging data or CT volumes, including anatomical type, unilaterality, type of treatment or procedure, existing implants, etc.

[0099] In step (408), after the machine learning model (MLM) analyzes the medical image data (or DRR), the machine learning model (MLM) outputs predicted parameters. The predicted parameters may include text, numerical data, metadata, and / or any type of information representing predictions or classifications related to the patient, the target anatomical part (TA), and / or the image data (MID). The predicted parameters are provided to confirm or refute intended parameter features, modalities, or conclusions related to the target anatomical part scanned in the image and / or derived from the patient data (from step (402b)). In other words, the automated image classification pre-test (ICPC) uses the predicted parameters to evaluate whether the intended parameters are accurate.

[0100] The predicted parameters include (1) data predicting the target anatomical part scanned in the image (e.g., predicting joint type (e.g., knee joint, hip joint, shoulder joint, etc.) or predicting joint parameters (e.g., joint shape, kinematics, biomechanics, bone density, disease state); (2) prediction of image or planned data (e.g., prediction of the surgical side of the patient's anatomical part scanned or prediction of the surgical side where the patient's surgery is planned (e.g., left joint, right joint, bilateral joint, medial, lateral, etc.)); (3) prediction of whether the procedure is, for example, total joint replacement (e.g., total knee replacement, total hip replacement, etc.), partial joint replacement (e.g., partial knee replacement, partial hip replacement, etc.), primary surgery, or revision surgery; prediction of implants for the patient (e.g., total knee implant, partial hip implant, etc.), and prediction of the manufacturer, model, or size of the implants; and prediction of the presence or absence of metal, foreign substances, or implants in the target anatomical part. It may include predictions of surgical procedures, such as predicting the types of metals, foreign bodies, or implants present in the target anatomical part. For example, when medical imaging data concerns an anatomical joint, a machine learning model (MLM) can predict that the joint is the knee joint, the surgical side is the left knee joint, and that no metallic objects were found in the image of the anatomical joint.

[0101] In step (410), the automated image inspection suite (AICS) may optionally output the classification / prediction results determined in step (408). This output may be generated before performing a comparison with the intended parameters, which is described in more detail below. This output (410) may be provided for various purposes. For example, the output (410) may be provided to provide a classification report of medical image data. For example, the classification report may include any of the predictions related to the target anatomical part or procedure described above. The classification report may be provided on a data review screen or GUI so that a technician can review it. For example, a technician may want to review whether the image data indicates the presence or absence of an implant in the target anatomical part, and if so, the predicted manufacturer or model of the implant present. This report may be provided for clinical purposes or for surgical planning purposes. Additionally or alternatively, the output (410) may be used as input to a machine learning model (MLM) in step (406) to provide additional input to the training data and / or to improve the training data.

[0102] In step (412), the automatic image classification pre-test (ICPC) automatically compares each intended parameter with a corresponding predicted parameter. For example, the intended type of the target anatomical part is compared with the predicted type of the target anatomical part, or the intended surgical side of the target anatomical part is compared with the predicted surgical side of the target anatomical part, and so on. The comparison module may be implemented to receive and organize the intended parameters from the medical image data (402) received (from steps (406, 408)) and to receive and organize the predicted parameters output from the machine learning model (MLM). The comparator may organize the corresponding intended parameters and predicted parameters into a lookup table. In some cases, the automatic image classification pre-test (ICPC) may know in advance what the intended parameter is and attempt to obtain a specific predicted parameter to confirm / refute the corresponding intended parameter. In other cases, the automatic image classification pre-test (ICPC) may not know in advance what the intended parameter is. Instead, the Automatic Image Classification Pre-check (ICPC) determines which predicted parameters were output and checks whether the corresponding intended parameters were provided.

[0103] Comparison can be implemented by determining whether the corresponding intended parameter and predicted parameter are identical, consistent, corresponding, substantially similar, or otherwise acceptable. For acceptable match, the corresponding intended parameter and predicted parameter can match within an acceptable threshold error range. In one example, the threshold error range may be greater than a 95% confidence score. For instance, the intended parameter may indicate that the scanned anatomical part is the left knee joint, and the corresponding predicted parameter may indicate a 97% confidence score that the scanned anatomical part is the left knee joint. Since the predicted value is greater than the threshold, the Automatic Image Classification Pre-check (ICPC) can confirm that the target anatomical part to be scanned was indeed scanned. In another example, for acceptable match, the corresponding intended parameter and predicted parameter must be completely identical without any tolerance. For instance, the intended parameter may indicate that the surgery is a right hip joint surgery. For the prediction to be acceptable match, the prediction must indicate a 100% confidence score that the surgery is a right hip joint. The confidence level or threshold tolerance may be optionally adjusted according to the importance of the examination. In another example, a machine learning model (MLM) may be trained with such high accuracy that it is not necessary to consider the confidence level or threshold tolerance. For example, the intended parameter may indicate that there are no implants present in the target anatomical part. If the predicted parameter indicates that there are no implants present in the target anatomical part, this result may be considered accurate regardless of the accuracy score. In another example, if the corresponding intended parameter is missing from the medical imaging data, the comparison result may be erroneous, but the predicted parameter may nevertheless be output in (410).It is also considered to perform a comparison between any intended parameter and a predicted parameter described above using any other technique or method not specifically described in this specification.

[0104] After applying the comparison described above, the Automatic Image Classification Pre-test (ICPC) evaluates whether the Medical Image Data (MID) is acceptable. In other words, the Automatic Image Classification Pre-test (ICPC) determines whether the Medical Image Data (MID) properly represents what it is intended to represent based on the output prediction values.

[0105] Medical imaging data (MID) may be approved upon determining that any intended parameter(s) are in acceptable agreement with their corresponding predicted parameter(s). Approval may be conditioned on multiple intended parameters being in acceptable agreement with their corresponding predicted parameters. For example, approval may require that the predicted joint type, predicted surgical side, and predicted procedure type match their respective intended joint type, intended surgical side, and intended procedure type. An automated image classification pre-check (ICPC) may be configured to select the parameters or groups of parameters that must be in acceptable agreement.

[0106] If acceptable, the Automatic Image Classification Pre-check (ICPC) determines, based on the output prediction, that the medical image data (MID) correctly represents what it is intended to represent. The Automatic Image Classification Pre-check (ICPC) may automatically generate a response to verify acceptability in step (414). In one example, the response is to send a confirmation message to the GUI so that the technician can see that this pre-check has passed. In another example, the response is for the Automatic Image Inspection Suite (AICS) to continue processing additional pre-checks (e.g., a series of sequential checks). Other responses are considered, such as not responding unless a classification error is detected.

[0107] Meanwhile, medical imaging data (MID) may be rejected in response to a determination that any one or more of the intended parameters do not acceptablely match the corresponding predicted parameters. If rejected, the Automatic Image Classification Pre-test (ICPC) determines, based on the output prediction, that the medical imaging data (MID) does not properly represent what was intended. If the result of the Automatic Image Classification Pre-test (ICPC) determines that the medical imaging data (MID) is unacceptable, a response to confirm the unacceptability may be automatically generated in step (416). In one example, the response may send a warning or notification to the GUI to allow the technician to see that this pre-test has failed, to identify which intended parameters failed and the reason, or optionally display the features predicted from the DRR on the GUI. Additionally, the response may be to output a message requesting that the patient requires additional scans or that pre-operative patient data be reviewed. In another example, the response is for the Automatic Image Inspection Suite (AICS) to stop processing additional pre-tests (e.g., a series of sequential tests).

[0108] 5. Automatic Motion (Load) Inspection

[0109] Another pre-examination that can be implemented in an automated imaging inspection suite (AICS) is an automated motion inspection (MRC), as illustrated in FIG. 1 and described in the method (500) of FIG. 10. The motion inspection (MRC) automatically evaluates medical imaging data (MID) to determine whether the patient moved during the scanning process. As can be seen in the exemplary medical imaging cross-section of FIG. 10, a motion rod (MR) is present in the image next to the target anatomical part (TA). The motion rod (MR) is included as part of the scanning protocol and is a physical rod attached to the patient's anatomical part or limb (e.g., leg) to immobilize the anatomical part during the scan. For example, during the preparation for a scan for knee surgery, a motion rod (MR) can be attached to the patient's leg extending from the hip area to the knee and ankle joints. The motion rod (MR) is radiopaque and visible in the scanned image. During the scan, the motion rod (MR) must remain stationary to obtain accurate scan results. If motion is detected in the Motion Load (MR), the patient is considered to have moved during the scan, and the scan must be refused or repeated.

[0110] Accordingly, to implement automatic motion inspection (MRC), the exemplary method (500) of FIG. 10 includes the step (502) of receiving image data (MID) of a target anatomical part (TA) and a motion rod (MR). For this inspection, it is advantageous for the received image data (MID) to be provided as a 3D CT volume. However, in other embodiments, the image data (MID) may be a CT image cross-section. In step (504), the automatic motion inspection (MRC) can automatically identify the motion rod (MR) in the 3D volume of the medical image data (MID). This process may include the automatic motion inspection (MRC) receiving known parameters of the motion rod (MR), such as the diameter, radius, length, density, etc. The automatic motion inspection (MRC) may additionally or alternatively use an object detection algorithm or a machine learning algorithm to detect the motion rod (MR) (regardless of the presence or absence of known information about the motion rod). Machine learning algorithms can be trained using a dataset to automatically distinguish between motion loads (MR) and other objects, such as target anatomical parts (TA) or existing implants. In step (504), when a motion load (MR) is first identified, the automatic motion inspection (MRC) automatically evaluates the entire 3D volume of the medical image dataset to determine whether the motion load (MR) is visible. This step (506) can be performed simultaneously with step (504) (i.e., at once). If image sections are used, step (506) may include an evaluation of the visibility of the motion load per image section. To determine whether the motion load (MR) is visible, the automatic motion inspection (MRC) may evaluate voxels or use object detection or machine learning algorithms to detect the presence of the motion load (MR) within the 3D volume. Again, this process may utilize known parameters of the motion load (MR) if available.Additionally, Automatic Motion Check (MRC) can generate shape models (e.g., cylinders or circles) corresponding to the motion rod (MR) or image cross-section (if image cross-sections are used). For volumetric analysis, Automatic Motion Check (MRC) can check whether a complete cylinder exists in the 3D volume and matches the parameters of the scanned motion rod. A complete cylinder indicates that the volume was captured while the motion rod (MR) was stationary and the patient was not moving. If shapes other than a complete cylinder (e.g., partial cylinders or blurry shapes) are identified in the volume, Automatic Motion Check (MRC) automatically identifies that the motion rod (MR) moved during the scan and that the scan was taken while the patient was moving. If image cross-sections are used, Automatic Motion Check (MRC) can check whether a complete circle exists in the image cross-section at the position of the motion rod (MR). A complete circle indicates that the image cross-section was captured while the motion rod (MR) was stationary and the patient was not moving. If a shape other than a perfect circle (e.g., a partial circle or a blurry shape) is identified in a specific image cross-section, the Automatic Motion Check (MRC) automatically identifies that the motion rod (MR) moved during the scan and that the image cross-section was captured while the patient was moving.

[0111] In step (508), as a result of evaluating medical imaging data (MID) for motion load (MR), the automatic motion check (MRC) automatically determines whether the evaluation result is an acceptable value or within a predetermined range or threshold. For example, acceptable may mean that the motion load (MR) is fully (100%) visible in the volume or all imaging sections. Alternatively, acceptable may mean that the motion load (MR) is visible exceeding a threshold (e.g., exceeding 95%) in the volume or all imaging sections. Unacceptable result may mean that the motion load (MR) is not fully visible in the volume or at least one imaging section. Alternatively, unacceptable result may mean that the motion load (MR) is not visible exceeding a threshold in the volume or at least one imaging section. The thresholds for acceptable and unacceptable may be the same or different.

[0112] If the evaluation is acceptable, the automatic motion check (MRC) in step (510) may automatically generate a response to confirm the acceptability. In one example, the response is to send a confirmation message to the GUI so that the technician can see that this preliminary check has passed or that the scan was properly acquired without patient motion. In another example, the response is for the automatic image inspection suite (AICS) to continue processing additional preliminary checks (e.g., a series of sequential checks). Other responses are also considered, such as not responding unless an error is detected.

[0113] If the evaluation is unacceptable, the automatic motion check (MRC) in step (512) may automatically generate a response regarding the unacceptability. In one example, the response is to send a warning or notification to the GUI so that the technician can see that this preliminary check has failed, or, if applicable, identify the failed image section. In another example, the response is for the automatic image check suite (AICS) to stop processing additional preliminary checks (e.g., a series of sequential checks). Additionally, the response may be to output a message requesting that a new scan be taken while the patient is stationary.

[0114] 6. Automatic Volume Classifier and Unilateral Pre-check

[0115] Another module that can be implemented in the Automatic Image Inspection Suite (AICS) is an automatic volume classifier and a unilateral pre-inspection or classifier (VLC), as illustrated in FIG. 1 and described with reference to FIG. 11 and method (600). The VLC pre-inspection may be included in any of the described pre-inspections or used as a sub-function thereof.

[0116] The VLC pre-scan includes a pipeline capable of identifying the volume type contained within a given 3D image volume of medical imaging data (MID). The VLC pre-scan can also identify the laterality of anatomical parts for volumes of a unilateral type. Based on this pipeline, the VLC pre-scan outputs a final label identifying the volume type and laterality.

[0117] Referring to FIG. 11, in step (602), medical imaging data (MID) is input into the VLC pre-examination. The medical imaging data (MID) may be a 3D volume, for example, a 3D CT volume of the DICOM series. The medical imaging data (MID) may represent any type of anatomical part, including but not limited to the knee joint, ankle joint, hip joint, shoulder joint, spine, etc. In one example, the CT volume is in an unclassified state before being input into the VLC pre-examination. In other words, the CT volume may not contain information identifying the type of anatomical volume being photographed or the laterality of the anatomical volume being photographed.

[0118] Medical imaging data (MID) can be input into a Volume Type Classifier (VTC) implemented in the first part of the VLC pre-examination pipeline. The Volume Type Classifier (VTC) can be implemented as a deep learning model that calculates the probability that a 3D volume is one of multiple volume types. Volume types include, but are not limited to, unilateral hip joints, unilateral knee joints, unilateral ankle joints, bilateral hip joints, bilateral knee joints, or bilateral ankle joints. The most likely type among these is taken as the output of the Volume Type Classifier. For example, as illustrated in Fig. 11, the output of the Volume Type Classifier (VTC) is an identification result indicating that the medical imaging data (MID) contains a unilateral knee joint. The confidence value can be recorded or presented in a GUI or report. Here, the Volume Type Classifier (VTC) generates a confidence score of 0.9998 for the unilateral knee joint. In one example, if the confidence score exceeds a threshold (e.g., exceeding 95%, 98%, or 99%), the volume type classifier (VTC) can output the result at (604) and send it to the next stage of the VLC pre-test pipeline, namely the lateral or unilateral classifier (LC). Additionally, or alternatively, the highest confidence score is taken as the output of the volume type classifier (VTC).

[0119] In one embodiment, the volume type classifier (VTC) may output results to the unilateral classifier (LC) only in response to classifying the volume as a "unilateral" bone structure (hip joint, knee joint, ankle joint, etc.), because the unilateral bone structure lacks a left or right anatomical part that provides a reference comparison, making the possibility of confusion for unilaterality higher compared to bilaterality. In this case, if the volume type classifier (VTC) classifies the volume as a "bilateral" bone structure, the volume type classifier (VTC) may bypass the unilateral classifier (LC) in (605) and output results directly to the final label output in (606). Of course, outputting results to the unilateral classifier (LC) in response to the volume type classifier (VTC) classifying the volume as a "bilateral" bone structure is also considered.

[0120] In one example, the deep learning model used for the Volume Type Classifier (VTC) is DenseNet-264, a densely connected convolutional network architecture. Training images were scaled to maintain the shape of the original CT images, and voxel intensities were restricted to a specific range of Hounsfield units. During each training epoch, images were randomly sampled according to each label ("left hip" or "right hip") to ensure that the model was trained with nearly the same number of images for each label. Intensity scaling was performed on the voxels to compensate for variability in CT scanners and CT scanner calibrations. Affine transformations (shearing and rotation) were applied to the training images to compensate for changes in patient position within the scanner, and scaling was included to compensate for variability in patient height and weight. Other types of machine learning models, such as convolutional neural networks, random forest classifiers, decision tree classifiers, K-nearest neighbor classifiers, Naive Bayes classifiers, and support vector machines, can be used to classify medical image data or the contents of CT volumes.

[0121] To implement a unilateral classifier (LC), the VLC pre-test may include a CT hip lateral classifier, which can be implemented as a deep learning model that calculates the probability that the 3D volume is the left or right hip joint. The most likely of these left or right hip joints is taken as the output of the classifier. The VLC pre-test may include a CT knee lateral classifier, which can be implemented as a deep learning model that calculates the probability that the 3D volume is the left or right knee joint. The most likely of these left or right knee joints is taken as the output of the classifier. The VLC pre-test may include a CT ankle lateral classifier, which can be implemented as a deep learning model that calculates the probability that the 3D volume is the left or right ankle joint. The most likely of these left or right ankle joints is taken as the output of the classifier.

[0122] These unilateral classifiers (LCs) can be implemented individually or combined into a single classifier. When the unilateral classifiers (LCs) are separated, as illustrated in FIG. 11, the output of the volume type classifier (VTC) is directly input to each unilateral classifier (LC) associated with the classified volume. For example, in FIG. 11, the output of the volume type classifier (VTC) is a unilateral knee joint. Thus, the VLC pre-examination successfully identified the type of anatomical part in the image, but the unilaterality of this knee joint has not yet been identified. Therefore, the output of the volume type classifier (VTC) can be directly input to a CT knee lateral classifier specifically trained to detect the unilaterality of the knee joint in the image volume. This design approach of separating unilateral classifiers by bone type reduces computation time and enables the processing of large amounts of image data.

[0123] As illustrated in FIG. 11, the output of the unilaterality classifier (LC) is the result of identifying the unilaterality of an anatomical part from medical imaging data (MID). The confidence value may be recorded or presented in a GUI or report. In the illustrated example, the unilaterality classifier (LC) generates a confidence score of 0.9995 to indicate that the knee joint is the right knee joint. In one example, if the confidence score exceeds a threshold (e.g., exceeding 95%, 98%, or 99%), the unilaterality classifier (LC) may output the result to the final label output in (606). Additionally, or alternatively, the highest confidence score is taken as the output of the unilaterality classifier (LC). For example, if the confidence for the left knee joint is 0.05 and the confidence for the right knee joint is 0.95, the unilaterality classifier (LC) may take the right knee joint as the correct unilaterality identification for the final label output in (606). The final label output (606), for example, "right knee joint", may be provided on a report or GUI as described and illustrated in this specification.

[0124] In one example, the deep learning model used for any one or more of the laterality classifiers (LC) is DenseNet-121, a densely connected convolutional network architecture. Training images were scaled to maintain the shape of the original CT images, and voxel intensities were restricted to a specific range of Hounsfield units. During each training epoch, images were randomly sampled according to each label ("left hip" or "right hip") to ensure that the model was trained with nearly the same number of images for each label. Intensity scaling was performed on the voxels to compensate for variability in CT scanners and CT scanner calibrations. Affine transformations (shearing and rotation) were applied to the training images to compensate for changes in patient position within the scanner, and scaling was included to compensate for variability in patient height and weight. Other types of machine learning models, such as convolutional neural networks, random forest classifiers, decision tree classifiers, K-nearest neighbor classifiers, Naive Bayes classifiers, and support vector machines, can be used to classify laterality in medical image data or CT volumes.

[0125] VLC pre-testing can perform an automated inspection using label output (606) to determine whether medical imaging data (MID) is acceptable. This inspection can be performed automatically in several ways. For example, VLC pre-testing can automatically evaluate the reliability scores of output type and unilaterality to determine whether the reliability scores exceed a threshold. If either of the two values ​​is below the threshold, the inspection can output a failure result. If both values ​​exceed the threshold, the inspection can output a pass result. Additionally, the results of other pre-tests from AICS can be used for the reliability scores of the VLC pre-testing. In some cases, the output results of volume type and unilaterality may be presented for manual review by a technician.

[0126] B. Exemplary Planning Workflow

[0127] Referring again to FIG. 1, when the Automated Imaging Inspection Suite (AICS) outputs successful approval or medical imaging data (MID) after performing any one or more of the aforementioned pre-inspections, a treatment planning request including the approved medical imaging data (MID) may be forwarded to a sub-stage for other procedures or processes related to surgical planning. For example, in (SG), an automatic or manual segmentation procedure may be initiated based on the approved medical imaging data (MID). In (MC), an anatomical model or virtual model may be generated based on the segmented imaging data or the approved medical imaging data (MID). In (SP), a surgical plan may be implemented on the approved medical imaging data (MID) or the anatomical model or virtual model. The surgical plan may be an implant plan and may be based on any of the predicted parameters described above. Once the surgical plan is completed, the treatment planning request may be returned to the appropriate requester in the RPL. After the planning request is returned, the approved medical imaging data (MID) may be further utilized for surgical workflow purposes using the techniques described herein.

[0128] In FIG. 12, another exemplary workflow including an automated imaging suite (AICS) is provided. In this example, a treatment plan request is generated after medical imaging data (MID) is initially processed by the automated imaging suite (AICS). The medical images are initially input into the automated imaging suite (AICS), for example, from a PACS server. This input can be automatically performed and uploaded, for example, to a server running the automated imaging suite (AICS). The automated imaging suite (AICS) then performs any one or more of the automated imaging pre-examinations described herein. If the results of the automated imaging pre-examination are unsuccessful, for example, if one or more examinations fail, the AICS automatically rejects the scan. As described below, feedback may be automatically provided to the appropriate person in charge or technician. The feedback may be provided in a GUI and may include a detailed report summarizing the reason for rejection, recommendations for scan correction methods, instructions for requesting a rescan, etc. If the results of the automated imaging pre-examination are successful, the workflow proceeds to the treatment plan request generation step. Then, the appropriate person in charge or technician can generate a treatment plan request after determining that the medical image has passed the initial automated pre-screening and is suitable for use in early planning.

[0129] Referring again to FIG. 12, after a treatment plan is generated, the plan, including medical imaging data, can be entered once again into the Automated Imaging Inspection Suite (AICS). The Automated Imaging Inspection Suite (AICS) performs "post-planning" processing of any one or more of the automated imaging pre-inspections described herein. Here, the inspection may be the same or different from the inspection performed before the creation of the treatment plan request. If the results of the (post-planning) automated imaging pre-inspection are unsuccessful, for example, if one or more inspections fail, the person in charge or technician of the segmentation team is automatically notified of the rejection and can perform a review of the output data. The data review may include reviewing the rejection report, identifying potential issues, and performing corrective actions. Similarly, feedback can be provided to the appropriate segmentation person or technician via a GUI to facilitate the data review. Meanwhile, if the results of the (post-planning) automated imaging pre-inspection are successful, the workflow proceeds to the automated pre-segmentation step of the target anatomical part (TA) in the medical imaging data. The automated pre-segmentation roughly segments the target anatomical part (TA), after which a technician can perform a precise segmentation later. Alternatively, a technician may review the complete segmentation later after performing it through automated pre-segmentation. At this stage, automated pre-segmentation can be reliably executed based on the confidence that the medical image has passed the automated pre-segmentation twice and is suitable for use in segmentation. After automated pre-segmentation, a member of the segmentation team or a technician is notified of the segmentation results and can review the resulting data. The data review may include reviewing segmentation accuracy, examining the pre-segmentation output, identifying potential issues, and performing corrective actions. If the data review process is successful, the segmentation output can be fine-segmented (automatically or manually by a segmentation technician).If a negative result is obtained during the data review process, this case may be rejected with a case memo indicating the reason for rejection.

[0130] C. Graphic User Interface Output

[0131] Referring to FIGS. 13 through 15, additional functions / outputs / interfaces are provided for a GUI that can be used with the Automatic Image Inspection Suite (AICS). FIGS. 13 and 14 provide exemplary screens of a GUI showing a Report (RP) that can be automatically generated by the Automatic Image Inspection Suite (AICS). Such a report can be generated regardless of whether the results of any one or more of the pre-inspections during the pre-inspection are acceptable or unacceptable. The Report (RP) can be automatically generated as a result of any one or more of the steps (110, 112, 210, 212, 316, 318, 414, 416, 510, 512, 604, 606) described above for various pre-inspections. The Report (RP) may also be automatically generated in response to a pre-plan or post-plan review by AICS, for example, as described in the workflow of FIGS. 1 and 12. The Report (RP) may provide various types of information to assist technicians performing image reviews, for example, when preparing for segmentation or planning. The information contained in the Report (RP) may include, for example, the data used to generate the report; a report summary (including specific reasons for rejection relative to acceptable thresholds, where applicable); recommendations for scan correction methods; recommendations for rescans; images or representations such as specific image sections of medical imaging data (MID) (e.g., the image section that caused the rejection); attributes of the image or target anatomical part (e.g., volume type, dimensions, interval, height, etc.); the patient's name; the patient's date of birth (DOB); the name of the requesting surgeon; the type of intended procedure (e.g., THA, TKA); a summary of tests performed or not performed at AICS, whether these tests passed or failed and the reasons therefor; reliability or accuracy scores for any tests; recommendations for modifying the segmentation or planning process, etc.

[0132] The failure description in the report may include, for example, that the scan was rejected because an anatomical region was not included in the image. The report may provide recommendations for resolving the problem, for example, that the scanning facility should re-burn the disc to include all regions. Additionally, the report may include predictions or suggestions to identify the root cause of the problem, for example, that the observed pixel size of the anatomical region is smaller than the expected pixel size; that the observed image cross-sectional thickness is not within the expected range; or that the image cross-sectional spacing does not comply with the protocol (for example, the observed image cross-sectional spacing is 2.5 mm, but the image cross-sectional spacing should be less than 1.1 mm at most and there should be no gaps or overlaps). These parameters may be checked via automated image parameter inspection (IPPC), for example, as illustrated in Fig. 3 and described with reference thereto.

[0133] Additional information provided in the report includes image acquisition parameters or DICOM tag information (e.g., modality (e.g., CT), samples per pixel (e.g., 1), allocated bits (e.g., 16), gantry tilt (e.g., 0), patient image orientation (e.g., as shown in FIG. 14). (etc.) may be included. The report may provide the square resolution of the image, volume height, number of image sections, pixel size on the X and Y axes, image section thickness, and Z-axis interval. These parameters may be checked through automatic image parameter inspection (IPPC), as illustrated in, for example, FIG. 3 and described with reference thereto.

[0134] Referring to FIG. 15, a dashboard (DB) can be implemented using a GUI to provide organization and management of patient case files. The dashboard (DB) can be dynamically updated with various information about the case file, such as the patient name, the number of image sections of the medical imaging data (MID), and the surgeon's name. Buttons may be provided to provide immediate access, download, or search for the medical imaging data (MID) to be viewed. Advantageously, the dashboard (DB) is provided with a status check function. The status check is dynamically updated as the Automated Imaging Inspection Suite (AICS) continuously evaluates the medical imaging data for various case files. The status check may include icons that are graphically displayed in the GUI and are dynamically updated based on changes in status. The icons may indicate, for example, that an examination or examinations are in progress, completed, failed, or passed. The icons may be a check mark (if successful), an X mark (if unsuccessful), or a loading graphic (indicating that the examination is in progress). The AICS may be integrated with the dashboard, for example, via an API or in other ways, to provide dynamic status updates for each case file. Additionally, AICS can automatically load relevant notes regarding the status of a test or tests, such as whether the tests are in progress, completed, failed, or passed, and any information that may be provided in the report (RP) described above, e.g., specific reasons for rejection relative to acceptable thresholds (where applicable); recommendations for scan correction methods; recommendations for rescans; images or representations of specific image sections of medical imaging data (MID) (e.g., image sections that caused the rejection); summaries of tests performed or not performed in AICS; reliability or accuracy scores for any test; recommendations for modifying the segmentation or planning process, etc.Additionally, the dashboard (DB) provides direct links to each case file, enabling access, downloading, and viewing of arbitrary reports (RPs) as described above. As a result, between the reports (RPs) and the dashboard, this system provides immediate insights into automated image inspection activities performed in AICS and offers a highly user-friendly level of image inspection management.

[0135] D. Exemplary Technical Solutions and Advantages

[0136] As is evident from the detailed description and drawings, the pre-inspection provided by the Automated Imaging Inspection Suite (AICS) offers significant advantages and technical solutions by automatically, rapidly, and accurately identifying defects or errors in imaging data (MID). This enables the Automated Imaging Inspection Suite (AICS) to automatically determine whether the imaging data (MID) is suitable for subsequent post-processing and / or surgical planning (e.g., segmentation, anatomical model generation, implant planning), or for modes of surgical workflows that rely on surgical planning or medical imaging (e.g., surgical navigation visualization, anatomical registration, etc.). The Automated Imaging Inspection Suite (AICS) can process the described pre-inspection almost immediately—that is, within seconds—thereby reducing the time-consuming burden on technical teams to manually review imaging data (MID) to identify potential errors. As a result, the Automated Imaging Inspection Suite (AICS) can significantly reduce labor costs and human error associated with manual review of imaging data. The Automated Imaging Inspection Suite (AICS) can be advantageously performed before and / or after the creation of a treatment plan request, providing additional reliability prior to planning and segmentation. By performing automated checks early in the workflow (e.g., before treatment planning requests or splitting), the workload on technicians for manual checks can be significantly reduced, freeing up time to process a much larger number of cases. Additionally, automated checks provide feedback early in the process, preventing unnecessary delays and subsequent resource waste. Automated Image Inspection Suite (AICS) can contribute significantly to reducing human error and labor costs by automatically rejecting scans and providing automated reports and summaries, enabling technicians to rapidly process large volumes of patient data. Furthermore, by providing automated feedback, for example through dashboard reports, AICS and GUI can educate scan technicians to provide comprehensive summary information that can potentially reduce future image rejection rates.Additionally, the Automated Imaging Inspection Suite (AICS) processes and classifies unclassified images, thereby avoiding reliance on pre-classification by technicians or radiologists, which are prone to human error. Furthermore, the classifier can be specially trained using image datasets specialized for specific types of anatomical parts (e.g., knee, hip, ankle joints), providing a means to process numerous medical images quickly and reliably. Other advantages not specifically described herein will be readily apparent to those skilled in the art.

[0137] Various embodiments have been discussed in the foregoing description. However, the embodiments discussed herein are not intended to limit the invention to any specific form or to provide all embodiments. The terms used are intended to describe the invention, not to limit it. Various modifications and variations are possible based on the foregoing, and the invention may be practiced in ways other than those specifically described.

[0138] Since the various features and advantages of the present invention are evident from the detailed description, the appended claims are intended to encompass all such features and advantages that fall within the true spirit and scope of the invention. Furthermore, since a person skilled in the art can readily conceive of various modifications and variations, the invention is not intended to be limited to the exact structure and operation illustrated and described, and therefore all suitable modifications and equivalents should be interpreted as being included within the scope of the invention.

Claims

Claim 1 An automated image checking suite configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the automated image checking suite comprises a non-transient computer-readable medium containing instructions, and the instructions, when executed by one or more processors, perform an automated check, Whether the above medical image data was scanned according to acceptable configuration settings; Whether the target anatomical part in the medical image data is acceptablely captured within the boundaries of the medical image data; Whether the above medical imaging data indicates a motion load visible beyond a critical visibility level; and Whether the above medical imaging data acceptablely shows the intended type of the target anatomical part and the intended laterality of the said anatomical part An automated imaging inspection suite configured to perform the step of executing the automated inspection to determine; and the step of automatically rejecting the medical imaging data as unacceptable to facilitate surgical planning in response to a determination that any one or more of the automated inspections generate unacceptable results. Claim 2 An automatic image inspection suite according to claim 1, wherein, to determine whether the medical image data was scanned according to the acceptable configuration settings, the instruction is configured to perform the steps of: automatically acquiring one or more configuration settings that define the manner in which the image device scanned the medical image data when executed by one or more processors; automatically comparing the one or more configuration settings with one or more acceptable configuration settings; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the one or more configuration settings do not correspond to the one or more acceptable configuration settings. Claim 3 An automated imaging suite according to paragraph 2, wherein the instruction is configured to automatically approve the medical imaging data as acceptable to facilitate surgical planning in response to a determination that the one or more configuration settings correspond to the one or more acceptable configuration settings when executed by the one or more processors. Claim 4 An automated image inspection suite according to claim 1, wherein, for determining whether the target anatomical part in the medical image data is acceptablely captured within the boundaries of the medical image data, the instruction is configured to perform the steps of: automatically identifying the target anatomical part in the medical image data and fitting a shape model to the target anatomical part when executed by one or more processors; automatically comparing the features of the shape model with the boundaries of the medical image data; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to determining that the features of the shape model exceed the boundaries of the medical image data. Claim 5 An automated image inspection suite according to claim 1, wherein, to determine whether the target anatomical part in the medical image data is acceptablely captured within the boundaries of the medical image data, the instruction is configured to perform the steps of: automatically comparing the medical image data with a statistical group of medical image data containing other anatomical parts similar to the target anatomical part to identify anatomical landmarks of the target anatomical part required to be visible in the medical image data when executed by one or more processors; automatically evaluating the medical image data to determine whether the anatomical landmarks are visible in the medical image data; and automatically rejecting the medical image data as unacceptable to facilitate surgical planning in response to the determination that the anatomical landmarks are not visible in the medical image data. Claim 6 An automated image inspection suite according to claim 1, configured to perform the following steps to determine whether the medical image data represents a motion load that appears to exceed a threshold visibility level: the instruction, when executed by one or more processors, automatically identifying the motion load within the volume of the medical image data; automatically evaluating the volume of the medical image data to determine whether the volume of the medical image data represents the motion load in an acceptable manner; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to the determination that the volume of the medical image data does not represent the motion load in an acceptable manner. Claim 7 An automatic image inspection suite according to claim 6, wherein, for automatically identifying the motion load, the instruction is configured to perform the step of implementing an object detection algorithm or a machine learning model to automatically identify the motion load within the volume and to distinguish the motion load from other features appearing within the volume when executed by one or more processors. Claim 8 An automatic image inspection suite according to claim 7, wherein the instruction is configured to implement the object detection algorithm or the machine learning model to automatically determine whether the motion load shows a complete cylinder within the volume when executed by one or more processors, for automatically evaluating the volume of the medical image data to determine whether the volume acceptablely represents the motion load. Claim 9 An automatic image inspection suite configured to perform, in order to determine whether the medical image data acceptablely shows the intended type of the target anatomical part and the intended laterality of the target anatomical part, the instruction, when executed by one or more processors, the receiving step of receiving the medical image data as input, wherein at the time of input the type and laterality of the target anatomical part in the medical image data are unclassified; the receiving step of automatically classifying the type of the target anatomical part in the medical image data using a first machine learning model; the selecting step of selecting a second machine learning model specifically trained to classify the laterality of the classified type of the anatomical part using the classified type of the target anatomical part; and the step of automatically classifying the laterality of the target anatomical part in the medical image data using the second machine learning model. Claim 10 An automated image inspection suite according to claim 9, wherein the instruction is configured to perform the steps of: generating a confidence score representing the classification accuracy of the type of the target anatomical part in the medical image data when executed by one or more processors; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the confidence score does not meet an acceptable threshold. Claim 11 An automated image inspection suite according to claim 9, wherein the instruction is configured to perform the steps of: generating a confidence score representing the classification accuracy of the laterality of the target anatomical part in the medical image data when executed by one or more processors; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the confidence score does not meet an acceptable threshold. Claim 12 A computer-implemented method for automatically evaluating whether medical imaging data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, comprising the step of performing an automatic examination, Whether the above medical image data was scanned according to acceptable configuration settings; Whether the target anatomical part in the medical image data is acceptablely captured within the boundaries of the medical image data; Whether the above medical imaging data indicates a motion load visible beyond a critical visibility level; and Whether the above medical imaging data acceptablely shows the intended type of the target anatomical part and the intended laterality of the said target anatomical part A computer-implemented method comprising: a step of performing the above-mentioned automatic inspection to determine; and a step of automatically rejecting the medical image data as unacceptable to facilitate surgical planning in response to a determination that any one or more of the above-mentioned automatic inspections generate unacceptable results. Claim 13 A non-transient computer-readable medium comprising instructions configured to automatically evaluate medical image data of a target anatomical part, wherein the instructions are configured to perform the following steps when executed by one or more processors: receiving the medical image data as input, wherein at the time of input, the type and laterality of the target anatomical part in the medical image data are not classified; automatically classifying the type of the target anatomical part in the medical image data using a first machine learning model; selecting a second machine learning model specifically trained to classify the laterality of the classified type of the target anatomical part using the classified type of the target anatomical part; automatically classifying the laterality of the target anatomical part in the medical image data using the second machine learning model; and generating a computer-generated output that identifies the classified type and the classified laterality of the target anatomical part in the medical image data. Claim 14 A non-transient computer-readable medium, wherein, in paragraph 13, the medical imaging data is a 3D CT volume, the target anatomical part is an anatomical joint, and the instruction is configured to perform the step of automatically classifying the type of anatomical joint within the 3D CT volume into one of a unilateral hip joint, a unilateral knee joint, a unilateral ankle joint, a bilateral hip joint, a bilateral knee joint, or a bilateral ankle joint using the first machine learning model when executed by one or more processors. Claim 15 A non-transient computer-readable medium configured such that, when executed by one or more processors, the above instruction performs the steps of: generating a confidence score representing the classification accuracy of the type of anatomical joint in the 3D CT volume; comparing the confidence score with a threshold; automatically classifying the type of anatomical joint in response to the confidence score exceeding the threshold; and automatically selecting the second machine learning model using the classified type of anatomical joint. Claim 16 A non-transient computer-readable medium configured such that, when executed by one or more processors, the above instruction performs the steps of: generating a plurality of reliability scores representing classification accuracy for classifying the type of anatomical joint in the 3D CT volume into a unilateral hip joint, a unilateral knee joint, a unilateral ankle joint, a bilateral hip joint, a bilateral knee joint, or a bilateral ankle joint, respectively; identifying the highest reliability score among the plurality of reliability scores; automatically classifying the type of anatomical joint based on the highest reliability score; and automatically selecting the second machine learning model using the classified type of anatomical joint. Claim 17 A non-transient computer-readable medium configured such that, in the 14th paragraph, the instruction is configured to perform the step of automatically selecting the second machine learning model in response only to automatically classifying the type of the anatomical joint in the 3D CT volume as one of a unilateral hip joint, a unilateral knee joint, or a unilateral ankle joint using the classified type of the anatomical joint when executed by one or more processors. Claim 18 A non-transient computer-readable medium configured to perform the following steps when executed by one or more processors: selecting the second machine learning model specifically trained to classify the unilaterality of the unilateral hip joint as a left hip joint or a right hip joint in response to the automatic classification of the type of the anatomical joint in the 3D CT volume as a unilateral hip joint; or selecting the second machine learning model specifically trained to classify the unilaterality of the unilateral knee joint as a left knee joint or a right knee joint in response to the automatic classification of the type of the anatomical joint in the 3D CT volume as a unilateral knee joint; or selecting the second machine learning model specifically trained to classify the unilaterality of the unilateral ankle joint as a left ankle joint or a right ankle joint in response to the automatic classification of the type of the anatomical joint in the 3D CT volume as a unilateral ankle joint. Claim 19 A non-transient computer-readable medium configured such that, when executed by one or more processors, the above instruction performs the steps of: generating a confidence score representing the classification accuracy of the unilaterality of the anatomical joint in the 3D CT volume; comparing the confidence score with a threshold; automatically classifying the unilaterality of the anatomical joint in response to the confidence score exceeding the threshold; and automatically generating a computer-generated output identifying the classified unilaterality using the classified unilaterality of the anatomical joint. Claim 20 A non-transient computer-readable medium configured such that, when executed by one or more processors, the above instructions perform the steps of: generating a first confidence score indicating a classification accuracy in which the unilaterality of the anatomical joint is a left joint; generating a second confidence score indicating a classification accuracy in which the unilaterality of the anatomical joint is a right joint; identifying the highest confidence score among the first and second confidence scores; automatically classifying the unilaterality of the anatomical joint based on the highest confidence score; and automatically generating a computer-generated output that identifies the classified unilaterality using the classified unilaterality of the anatomical joint. Claim 21 A non-transient computer-readable medium configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for the target anatomical part, by performing the steps of: generating a confidence score indicating the classification accuracy of the type of the target anatomical part in the medical image data when executed by one or more processors; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the confidence score does not meet an acceptable threshold. Claim 22 A non-transient computer-readable medium configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for the target anatomical part, by performing the steps of: generating a confidence score representing the classification accuracy of the laterality of the target anatomical part in the medical image data when executed by one or more processors; and automatically rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the confidence score does not meet an acceptable threshold. Claim 23 In paragraph 13, a non-transient computer-readable medium wherein each of the first and second machine learning models is a deep learning model comprising a convolutional neural network. Claim 24 A computer implementation method for automatically evaluating medical image data of a target anatomical part, comprising: a receiving step of receiving said medical image data as input, wherein at the time of input, the type and laterality of said target anatomical part in said medical image data are not classified; a receiving step of automatically classifying the type of said target anatomical part in said medical image data using a first machine learning model; a second machine learning model specially trained to classify the laterality of the classified type of said target anatomical part using the classified type of said target anatomical part; a second machine learning model of automatically classifying the laterality of said target anatomical part in said medical image data using the second machine learning model; and a computer implementation method comprising a step of generating a computer-generated output representing the classified type and laterality of said target anatomical part in said medical image data. Claim 25 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to perform the steps of: automatically identifying said target anatomical part in said medical image data and fitting a shape model to said target anatomical part; automatically comparing features of said shape model with boundaries of said medical image data; and automatically rejecting said medical image data as unacceptable for facilitating surgical planning in response to a determination that features of said shape model exceed boundaries of said medical image data. Claim 26 A non-transient computer-readable medium according to claim 25, wherein the feature of the shape model includes the boundary of the shape model, and the instruction is further configured to perform the step of comparing the boundary of the shape model with the boundary of the medical image data when executed by the one or more processors; and the step of rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the boundary of the shape model exceeds the boundary of the medical image data. Claim 27 A non-transient computer-readable medium according to claim 25, wherein the feature of the shape model comprises one or more points located on the boundary of the shape model, and the instruction is further configured to perform the step of comparing one or more points of the shape model with the boundary of the medical image data when executed by the one or more processors; and the step of rejecting the medical image data as unacceptable for facilitating surgical planning in response to determining that one or more points of the shape model exceed the boundary of the medical image data. Claim 28 In paragraph 27, a non-transient computer-readable medium in which one or more points on the shape model correspond to predetermined anatomical landmarks of the target anatomical part. Claim 29 A non-transient computer-readable medium, wherein the above instruction is further configured to automatically approve the medical image data as acceptable for facilitating surgical planning in response to a determination that the features of the shape model are within the boundaries of the medical image data when executed by the one or more processors. Claim 30 A non-transient computer-readable medium that automatically generates an output in response to rejecting the medical image data as unacceptable, wherein the above instruction is configured to perform the steps of: generating a notification or warning in a graphical user interface to indicate the rejection of the medical image data when executed by the one or more processors; displaying the medical image data and the shape model in the graphical user interface to show that the features of the shape model exceed the boundaries of the medical image data; or generating a recommendation for corrective action based on the rejection of the medical image data. Claim 31 In paragraph 25, the shape model is a non-transient computer-readable medium derived from a statistical group of images of other anatomical parts similar to the target anatomical part. Claim 32 In paragraph 25, the shape model comprises any one or more of a statistical shape model, an active shape model, an active appearance model, and an active contour model, in a non-transient computer-readable medium. Claim 33 In paragraph 25, the medical imaging data is a CT volume, and the target anatomical part is a bone structure, a non-transient computer-readable medium. Claim 34 A computer-implemented method for automatically evaluating whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, comprising: automatically identifying the target anatomical part in said medical image data and fitting a shape model to said target anatomical part; automatically comparing the features of said shape model with the boundaries of said medical image data; and automatically rejecting said medical image data as unacceptable for facilitating surgical planning in response to a determination that the features of said shape model exceed the boundaries of said medical image data. Claim 35 A computer-implemented method according to claim 34, further comprising: a step of comparing the boundary of the shape model with the boundary of the shape model, wherein the feature of the shape model includes the boundary of the shape model; and a step of rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that the boundary of the shape model exceeds the boundary of the medical image data. Claim 36 A computer-implemented method according to claim 34, wherein the features of the shape model include one or more points located on the boundary of the shape model, and further comprising the step of comparing one or more points of the shape model with the boundary of the medical image data; and the step of rejecting the medical image data as unacceptable for facilitating surgical planning in response to a determination that one or more points of the shape model exceed the boundary of the medical image data. Claim 37 A computer implementation method according to claim 36, wherein one or more points on the shape model correspond to predetermined anatomical landmarks of the target anatomical part. Claim 38 A computer-implemented method according to claim 34, further comprising the step of automatically approving the medical image data as acceptable to facilitate surgical planning in response to a determination that the features of the shape model are within the boundaries of the medical image data. Claim 39 A computer-implemented method according to claim 34, comprising the step of automatically generating an output in response to the rejection of the medical image data as unacceptable, wherein the output comprises: generating a notification or warning in a graphical user interface to indicate the rejection of the medical image data; displaying the medical image data and the shape model in a graphical user interface to show that the features of the shape model exceed the boundaries of the medical image data; and generating a recommendation for corrective measures based on the rejection of the medical image data. Claim 40 In paragraph 34, a computer implementation method wherein the shape model is derived from a statistical group of images of other anatomical parts similar to the target anatomical part. Claim 41 A computer-implemented method according to claim 34, wherein the shape model comprises any one or more of a statistical shape model, an active shape model, an active appearance model, and an active contour model. Claim 42 A computer-implemented method according to claim 34, wherein the medical imaging data is a CT volume and the target anatomical part is a bone structure. Claim 43 A computer-implemented method according to claim 34, wherein the step of evaluating whether the medical image data is acceptable for facilitating surgical planning for the target anatomical part is further defined as evaluating whether the medical image data is acceptable for facilitating any one or more of the following: segmentation of the medical image data; creation of a virtual model of the target anatomical part; virtual implant planning for the target anatomical part; alignment of the target anatomical part to facilitate surgical navigation; and visualization of the medical image data to facilitate surgical navigation. Claim 44 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to perform the steps of: automatically comparing said medical image data with a statistical group of medical image data containing other anatomical parts similar to said target anatomical part to identify anatomical landmarks of said target anatomical part required to be visible in said medical image data; automatically evaluating said medical image data to determine whether said anatomical landmarks are visible in said medical image data; and automatically rejecting said medical image data as unacceptable for facilitating surgical planning in response to a determination that said anatomical landmarks are not visible in said medical image data. Claim 45 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to automatically perform the steps of: identifying said target anatomical part in said medical image data and fitting a shape model to said target anatomical part; comparing said shape model with the boundaries of said medical image data; determining that a portion of said shape model exceeds the boundaries of said medical image data; and modifying said medical image data to capture the portion of said shape model exceeding the boundaries of said medical image data. Claim 46 In paragraph 45, a non-transient computer-readable medium that modifies medical image data to capture a portion of the shape model that extends beyond the boundaries of the medical image data, wherein the instruction is configured to generate an extrapolation region including a portion of the shape model when executed by one or more processors. Claim 47 In paragraph 46, the above instruction is configured to generate the extrapolation region adjacent to the boundary of the medical image data when executed by one or more processors, in a non-transient computer-readable medium. Claim 48 In paragraph 46, the above instruction is configured to extend the boundaries of the medical image data to include the extrapolated region when executed by one or more processors, in a non-transient computer-readable medium. Claim 49 In paragraph 46, the above instruction is configured to artificially reproduce the features of the target anatomical part within the part of the shape model that extends beyond the boundary of the medical image data within the extrapolation area, when executed by one or more processors, in a non-transient computer-readable medium. Claim 50 In paragraph 49, the above instruction is configured to artificially reproduce a soft tissue region adjacent to a portion of the shape model that exceeds the boundary of the medical image data within the extrapolation region when executed by one or more processors, a non-transient computer-readable medium. Claim 51 In paragraph 45, the above instruction is configured to provide a color code that identifies a portion of the shape model that extends beyond the boundaries of the medical image data when executed by one or more processors, in a non-transient computer-readable medium. Claim 52 A non-transient computer-readable medium for modifying medical image data to capture a portion of a shape model that extends beyond the boundaries of the medical image data, wherein the instruction is configured to perform a step of enabling the medical image data to be visualized together with the shape model, including a portion of the shape model that extends beyond the boundaries of the medical image data when executed by one or more processors. Claim 53 In paragraph 45, the above instruction is further configured to annotate the medical image data to identify a portion of the shape model that exceeds the boundaries of the medical image data when executed by one or more processors, in a non-transient computer-readable medium. Claim 54 In paragraph 45, the shape model is a non-transient computer-readable medium derived from a statistical group of images of other anatomical parts similar to the target anatomical part. Claim 55 In paragraph 45, the shape model comprises any one or more of a statistical shape model, an active shape model, an active appearance model, and an active contour model, in a non-transient computer-readable medium. Claim 56 In paragraph 45, the medical imaging data is a CT volume, and the target anatomical part is a bone structure, a non-transient computer-readable medium. Claim 57 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical imaging data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to perform the steps of: automatically acquiring measurements of said target anatomical part in said medical imaging data; automatically selecting an implant from a plurality of implant options based on measurements of said target anatomical part in said medical imaging data and automatically acquiring implant measurements of the selected implant; automatically determining, based on said implant measurements, essential features of said target anatomical part that must be fully captured in said medical imaging data to acceptablely facilitate surgical planning of said selected implant for said target anatomical part; and automatically rejecting said medical imaging data as unacceptable for facilitating surgical planning in response to a determination that the essential features of said target anatomical part are not fully captured in said medical imaging data. Claim 58 In paragraph 57, the above instruction is configured to automatically approve the medical image data as acceptable for facilitating surgical planning in response to a determination that, when executed by the one or more processors, the essential features of the target anatomical part are fully captured in the medical image data. Claim 59 A non-transient computer-readable medium that automatically obtains measurements of a target anatomical part by means of the above instructions, wherein, when executed by the one or more processors, the above instructions are configured to perform the steps of: automatically identifying the target anatomical part in the medical image data and fitting a shape model to the target anatomical part; and automatically obtaining measurements of the target anatomical part from the shape model. Claim 60 In claim 59, the above instruction is configured to automatically measure the distance between points corresponding to each predetermined anatomical landmark of the target anatomical part in the shape model when executed by the one or more processors, thereby automatically obtaining a measurement of the target anatomical part from the shape model. Claim 61 In paragraph 59, the shape model is a non-transient computer-readable medium derived from a statistical set of images of other anatomical parts similar to the target anatomical part. Claim 62 In paragraph 59, the shape model comprises any one or more of a statistical shape model, an active shape model, an active appearance model, and an active contour model, in a non-transient computer-readable medium. Claim 63 In paragraph 57, the above instruction is further configured to automatically select the implant that is of an appropriate size for the measurement of the target anatomical part when executed by the one or more processors, thereby automatically selecting the implant based on the measurement of the target anatomical part, a non-transient computer-readable medium. Claim 64 In paragraph 57, a non-transient computer-readable medium in which the essential features of the target anatomical part are essential measurements and / or essential landmarks. Claim 65 A non-transient computer-readable medium that automatically generates an output in response to the medical image data being rejected as unacceptable, wherein the instruction is configured to perform the steps of: generating a notification or warning in a graphical user interface to indicate the rejection of the medical image data when executed by the one or more processors; displaying the medical image data and essential features of the target anatomical part not fully captured in the medical image data in the graphical user interface; displaying the medical image data and the selected implant or the implant measurement values ​​in the graphical user interface; or generating a recommendation for corrective action based on the rejection of the medical image data. Claim 66 A computer-implemented method for automatically evaluating whether medical imaging data of a target anatomical part is acceptable for facilitating a surgical plan for the target anatomical part, comprising: a step of automatically acquiring measurements of the target anatomical part in the medical imaging data; a step of automatically selecting an implant from a plurality of implant options based on the measurements of the target anatomical part in the medical imaging data and automatically acquiring measurements of the selected implant; a step of automatically determining, based on the implant measurements, essential features of the target anatomical part that must be fully captured in the medical imaging data to acceptablely facilitate a surgical plan for the selected implant for the target anatomical part; and a step of automatically rejecting the medical imaging data as unacceptable for facilitating a surgical plan in response to a determination that the essential features of the target anatomical part are not fully captured in the medical imaging data. Claim 67 A computer-implemented method according to claim 66, further comprising the step of automatically approving the medical image data as acceptable to facilitate surgical planning in response to a determination that the essential features of the target anatomical part are fully captured in the medical image data. Claim 68 A computer-implemented method according to claim 66, wherein the step of automatically acquiring measurements of the target anatomical part is further defined as automatically performing the steps of: identifying the target anatomical part in the medical image data and fitting a shape model to the target anatomical part; and acquiring measurements of the target anatomical part from the shape model. Claim 69 A computer-implemented method according to claim 68, wherein the step of automatically obtaining measurements of the target anatomical part from the shape model is further defined as the step of automatically measuring the distance between points corresponding to each predetermined anatomical landmark of the target anatomical part in the shape model. Claim 70 In paragraph 68, a computer implementation method wherein the shape model is derived from a statistical group of images of other anatomical parts similar to the target anatomical part. Claim 71 A computer-implemented method according to claim 68, wherein the shape model comprises any one or more of a statistical shape model, an active shape model, an active appearance model, and an active contour model. Claim 72 A computer implementation method according to claim 66, wherein automatically selecting the implant based on the measurement of the target anatomical part is further defined as automatically selecting an implant of a size suitable for the measurement of the target anatomical part. Claim 73 In paragraph 66, a computer implementation method in which the essential features of the target anatomical part are essential measurements and / or essential landmarks. Claim 74 A computer-implemented method according to claim 66, comprising the step of automatically generating an output in response to the rejection of the medical image data as unacceptable, wherein the output comprises: generating a notification or warning in a graphical user interface to indicate the rejection of the medical image data; displaying the medical image data and essential features of the target anatomical part not fully captured in the medical image data in the graphical user interface; displaying the medical image data and the selected implant or the implant measurement value in the graphical user interface; and generating a recommendation for corrective action based on the rejection of the medical image data. Claim 75 A computer-implemented method according to claim 66, wherein the step of evaluating whether the medical image data is acceptable for facilitating surgical planning for the target anatomical part is further defined as evaluating whether the medical image data is acceptable for facilitating any one or more of the following: segmentation of the medical image data; creation of a virtual model of the target anatomical part; virtual implant planning for the target anatomical part in the medical image data; alignment of the target anatomical part to facilitate surgical navigation; and visualization of the medical image data to facilitate surgical navigation. Claim 76 A computer-implemented method according to claim 66, wherein the medical imaging data is a CT image and the target anatomical part is a bone structure. Claim 77 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical imaging data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, include: a step of obtaining measurements of said target anatomical part in said medical imaging data; a step of selecting an implant from a plurality of implant options based on the measurements of said target anatomical part in said medical imaging data and obtaining implant measurements of the selected implant; a step of determining, based on the implant measurements, essential features of said target anatomical part that must be fully captured in said medical imaging data to acceptablely facilitate surgical planning of said selected implant for said target anatomical part; a step of determining that essential features of said target anatomical part are not fully captured in said medical imaging data; and a step of identifying said target anatomical part in said medical imaging data and fitting a shape model to said target anatomical part, wherein a portion of said shape model extends beyond the boundaries of said medical imaging data. A non-transient computer-readable medium configured to automatically perform the step of modifying the medical image data to capture a portion of the shape model that exceeds the boundaries of the medical image data. Claim 78 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to perform the steps of: automatically acquiring a measurement of said target anatomical part in said medical image data; automatically selecting an implant from a plurality of implant options based on the measurement of said target anatomical part in said medical image data and automatically acquiring an implant measurement of the selected implant; automatically comparing said implant measurement with a boundary of said medical image data; and automatically rejecting said medical image data as unacceptable for facilitating surgical planning in response to a determination that said implant measurement exceeds the boundary of said medical image data or is not separated by a threshold distance from the boundary of said medical image data. Claim 79 A non-transient computer-readable medium according to claim 78, wherein the instruction is configured to perform the step of automatically approving the medical image data as acceptable for facilitating surgical planning in response to a determination that the implant measurement is within the boundary of the medical image data or is separated from the boundary of the medical image data by the threshold distance when executed by the one or more processors. Claim 80 In claim 78, a non-transient computer-readable medium configured to automatically perform the steps of: obtaining an implant shape model for the selected implant when executed by the one or more processors; and obtaining a measurement value of the selected implant from the implant shape model, in order to automatically obtain a measurement value of the selected implant. Claim 81 In paragraph 80, the above instruction is configured to fit the implant shape model to the target anatomical part in the medical image data when executed by the one or more processors, in a non-transient computer-readable medium. Claim 82 A non-transient computer-readable medium according to claim 81, wherein the implant shape model includes a boundary or point, and to automatically compare the implant measurement value with the boundary of the medical image data, the instruction is configured to automatically perform the step of comparing the boundary or point of the implant shape model with the boundary of the medical image data when executed by the one or more processors; and the step of rejecting the medical image data as unacceptable for facilitating surgical planning in response to determining that the boundary or point of the implant shape model exceeds the boundary of the medical image data or is not separated by the threshold distance from the boundary of the medical image data. Claim 83 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to automatically perform the steps of: identifying said target anatomical part in said medical image data and fitting an implant shape model to said target anatomical part; comparing said implant shape model with the boundaries of said medical image data; determining that a portion of said implant shape model exceeds the boundaries of said medical image data; and modifying said medical image data to capture the portion of said implant shape model that exceeds the boundaries of said medical image data. Claim 84 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical imaging data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to perform the steps of: acquiring preoperative patient data associated with said medical imaging data, wherein said preoperative patient data includes information representing intended parameters, such as (1) an intended type of the target anatomical part and (2) an intended operative side of said target anatomical part; analyzing said medical imaging data and automatically applying said medical imaging data to a machine learning model to output predicted parameters, such as (1') a predicted type of said target anatomical part and (2') a predicted operative side of said target anatomical part; automatically comparing each intended parameter with a corresponding predicted parameter; and automatically approving said medical imaging data as acceptable for facilitating surgical planning in response to a determination that each intended parameter is acceptablely matched with a corresponding predicted parameter. Claim 85 A non-transient computer-readable medium configured such that, in response to a determination that any one or more of the intended parameters do not acceptablely match the corresponding predicted parameters when executed by the one or more processors, the instruction automatically rejects the medical image data as unacceptable to facilitate surgical planning. Claim 86 A non-transient computer-readable medium according to claim 84, wherein the medical imaging data comprises a CT volume, and the instruction is configured to perform the steps of: automatically generating a two-dimensional digitally reconstructed radiographic image from the CT volume when executed by the one or more processors; and automatically applying the digitally reconstructed radiographic image to the machine learning model to analyze the digitally reconstructed radiographic image and output the predicted parameters. Claim 87 A non-transient computer-readable medium according to claim 84, wherein the machine learning model is a convolutional neural network, and the instruction is configured to perform the step of automatically applying the medical image data to the convolutional neural network to analyze the medical image data and output the predicted parameters when executed by the one or more processors. Claim 88 A non-transient computer-readable medium configured to perform the steps of: acquiring preoperative patient data associated with medical imaging data when executed by the one or more processors, wherein the preoperative patient data includes information indicating intended parameters including (1) an intended type of anatomical joint and (2) an intended surgical side of the anatomical joint; and analyzing the medical imaging data and automatically applying the medical imaging data to the machine learning model to output predicted parameters including (1') a predicted type of anatomical joint and (2') a predicted surgical side of the anatomical joint. Claim 89 A non-transient computer-readable medium configured to perform the following steps when executed by the one or more processors: (1') automatically generating a first digitally reconstructed radiographic image in two dimensions that captures the bone structure of the anatomical joint from the CT volume; (2') automatically generating a second digitally reconstructed radiographic image in two dimensions that captures the soft tissue structure of the anatomical joint from the CT volume; (1') analyzing the first digitally reconstructed radiographic image and automatically applying the first digitally reconstructed radiographic image to the lightweight 2D convolutional neural network to output a predicted type of the anatomical joint; and (2') analyzing the second digitally reconstructed radiographic image and automatically applying the second digitally reconstructed radiographic image to the lightweight 2D convolutional neural network to output a predicted surgical side of the anatomical joint. Claim 90 A non-transient computer-readable medium configured to perform, when executed by one or more processors, the steps of: (3) acquiring preoperative patient data including information further indicating intended parameters including the intended presence or absence of an existing implant for the target anatomical part; analyzing the medical imaging data and (3') automatically applying the medical imaging data to the machine learning model to further output predicted parameters including the predicted presence or absence of an existing implant for the target anatomical part; automatically comparing the intended presence or absence of the existing implant with the predicted presence or absence of the existing implant; and automatically rejecting the medical imaging data as unacceptable to facilitate surgical planning in response to a determination that the intended presence or absence of the existing implant does not acceptablely match the predicted presence or absence of the existing implant. Claim 91 A non-transient computer-readable medium configured to perform the steps of: (3') analyzing the digitally reconstructed radiographic image and automatically applying the digitally reconstructed radiographic image to the convolutional neural network to output the predicted presence or absence of an existing implant for the target anatomical part, wherein, in paragraph 90, the target anatomical part is an anatomical joint, the medical imaging data includes a CT volume of the anatomical joint, the machine learning model is a convolutional neural network, and the instruction is executed by the one or more processors to automatically generate a digitally reconstructed radiographic image capturing a metal structure in the anatomical joint from the CT volume; and analyzing the digitally reconstructed radiographic image and (3') automatically applying the digitally reconstructed radiographic image to the convolutional neural network to output the predicted presence or absence of an existing implant for the target anatomical part. Claim 92 A non-transient computer-readable medium configured such that, in paragraph 91, the instruction is configured to perform the step of analyzing the digitally-reconstructed radiographic image when executed by the one or more processors and automatically applying the digitally-reconstructed radiographic image to the convolutional neural network to output the predicted presence of the existing implant as well as the predicted manufacturer of the existing implant; the predicted model number of the existing implant; and the predicted size or type of the existing implant. Claim 93 A non-transient computer-readable medium configured to perform, when executed by one or more processors, (4) acquiring preoperative patient data including information further indicating intended parameters including an intended type of procedure to be planned for the target anatomical part; analyzing the medical imaging data and (4') automatically applying the medical imaging data to the machine learning model to further output predicted parameters including a predicted type of procedure to be planned for the target anatomical part; automatically comparing the intended type of procedure with the predicted type of procedure; and automatically rejecting the medical imaging data as unacceptable to facilitate surgical planning in response to a determination that the intended type of procedure does not acceptablely match the predicted type of procedure. Claim 94 A non-transient computer-readable medium comprising instructions configured to automatically classify CT volumes of anatomical joints, wherein the instructions, when executed by one or more processors, automatically generate a plurality of two-dimensional digitally-reconstructed radiographic images capturing the structure of said anatomical joint from said CT volumes; and a non-transient computer-readable medium configured to perform the steps of analyzing each of said digitally-reconstructed radiographic images and automatically applying each of said digitally-reconstructed radiographic images to a convolutional neural network configured to output (1') a predicted type of anatomical joint, (2') a predicted surgical side of said anatomical joint, and (3') a predicted presence or absence of an existing implant of said anatomical joint. Claim 95 In paragraph 94, the above instruction is configured to analyze each of the digitally reconstructed radiographic images when executed by one or more processors, and (4') to automatically apply each of the digitally reconstructed radiographic images to the convolutional neural network to further output a predicted type of procedure to be planned for the anatomical joint, a non-transient computer-readable medium. Claim 96 A non-transient computer-readable medium configured such that, when executed by the one or more processors, the instruction analyzes each of the digitally-reconstructed radiographic images and automatically applies each of the digitally-reconstructed radiographic images to the convolutional neural network to further output the predicted presence of the existing implant as well as the predicted manufacturer of the existing implant; the predicted model number of the existing implant; and one or more of the predicted size or type of the existing implant. Claim 97 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical image data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions are configured to perform, when executed by one or more processors, the step of automatically acquiring one or more configuration settings that define the manner in which an imaging device scanned said medical image data; the step of automatically comparing said one or more configuration settings with one or more acceptable configuration settings; and the step of automatically rejecting said medical image data as unacceptable for facilitating surgical planning in response to a determination that said one or more configuration settings do not correspond to said one or more acceptable configuration settings. Claim 98 In paragraph 97, the above instruction is configured to automatically approve the medical image data as acceptable to facilitate surgical planning in response to a determination that the one or more configuration settings correspond to the one or more acceptable configuration settings when executed by the one or more processors. Claim 99 A non-transient computer-readable medium comprising instructions configured to automatically evaluate whether medical imaging data of a target anatomical part is acceptable for facilitating surgical planning for said target anatomical part, wherein the instructions, when executed by one or more processors, are configured to perform the steps of: automatically identifying a motion load within a volume of said medical imaging data; automatically evaluating the volume of said medical imaging data to determine whether said volume acceptablely shows said motion load; and automatically rejecting said medical imaging data as unacceptable for facilitating surgical planning in response to a determination that said volume does not acceptablely show said motion load. Claim 100 In claim 99, a non-transient computer-readable medium configured such that, for automatically identifying the motion load, the instruction is configured to implement an object detection algorithm or a machine learning model to automatically identify the motion load within the volume and to distinguish the motion load from other features appearing within the volume when executed by one or more processors. Claim 101 A non-transient computer-readable medium according to claim 100, wherein the instruction is configured to implement the object detection algorithm or the machine learning model to automatically determine whether the motion load shows a complete cylinder within the volume when executed by one or more processors, for automatically evaluating the volume of the medical image data to determine whether the volume shows the motion load in an acceptable manner.

Images