Method and system for image analysis

Abstract

To provide a computer-implemented image analysis method.SOLUTION: A method of the present invention includes: a step for quantizing one or more features segmented and identified from a medical image of a subject; a step for extracting clinically related features from non-image data on the subject; a step for evaluating the quantized features segmented from the medical image and the features extracted from the non-image data with a trained machine learning model; and a step for outputting one or more results on the evaluation of the features.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method and system for image analysis, in particular for (e.g. musculoskeletal) imaging. The present invention relates to applications in medical imaging, such as diagnosing and monitoring diseases and conditions in the medical imaging field. The region of interest is a 2D region within such a medical image. Possible medical imaging applications include computed tomography (CT ), magnetic resonance (MR), ultrasound, and lesion scanner imaging.

[0002] This application is a joint venture with U.S. Patent Application No. 16 / 448,460 (filed June 21, 2019). No. 6,313,635, filed on Dec. 13, 2003, and which claims priority thereto, the entire contents of which are incorporated herein by reference at the time of filing. Be taken in. [Background technology]

[0003] The morbidity, mortality and costs associated with the development of musculoskeletal disorders are on the rise. Part of the reason is that longer life spans are increasing the proportion of the elderly in the population. Early detection of musculoskeletal disorders may facilitate interventions that may reduce disease progression or prevent malignancy. Consequences (eg, fractures) may be minimized.

[0004] In the field of musculoskeletal imaging, various techniques are used to detect musculoskeletal disorders. For example, bone biopsy It can be used to detect osteosarcoma or other bone diseases and can accurately measure tissue characteristics. However, bone biopsies are invasive and can result in pain and scarring.

[0005] Other techniques involve the detection of musculoskeletal diseases by analyzing musculoskeletal images to identify or quantify abnormalities. The images are acquired using a variety of imaging modalities to assess the patient’s condition. Modalities include: DXA (Dual-energy X-ray Absorptiometry) X-ray absorptiometry), CT (Computed Tomography), MRI (Magnetic Resonance Imaging) , and X-ray scanners. Each modality is used to treat (e.g., bone fragility, osteoarthritis, rheumatoid arthritis) Specific long-term measures for screening and evaluation of musculoskeletal disorders (e.g., osteomalacia, osteoporosis, and bone deformities) It has a place.

[0006] For example, HRpQCT (High-Resolution peripheral Quantitative Computed Tomography) PHY (High Resolution Peripheral Quantitative Computed Tomography) is a high resolution, low radiation, and non-invasive method. It is a common technique used to evaluate the three major determinants of bone quality: microarchitecture. , mineralization status, and biomechanical properties.

[0007] There are manual, semi-manual, and automated methods for evaluating musculoskeletal disorders, and a variety of imaging modalities. For example, the BMB (Bone Marrow Burden) score is MRI scoring method for manual assessment of bone marrow involvement in Xenograft disease. Technologists measure the BMB score on MRI images of the lumbar spine and femur. This is done using signal strength and distribution according to the King criteria. For example, According to the system, a score ranging from 0 to 8 is given for the lumbar spine and 0 for the femur. A score ranging from 0 to 8 is given, and an overall score ranging from 0 to 16. The BMB score indicates more severe bone marrow involvement.

[0008] When measuring bone mineral density (BMD) using DXA images, a semi-manual method is used. . The spine or lumbar region is scanned by DXA. A radiation technician or doctor selects the region of interest (R OI) (for example, different vertebral sections of a spine scan or the femoral neck of the thigh in a lumbar scan, etc.). The bone mineral density of the selected region is determined based on a predefined density comparison formula. The measured density value is converted to a T-score, and this is done by comparing it with measurements from a group of age-matched young adults with peak bone mass. A T-score of ≧1 is considered normal; a T-score of -2.5 < T < -1 is classified as osteopenia; a T-score of ≦ -2. 5 is defined as osteoporosis. The T-score is considered by clinicians when examining whether to evaluate the risk of fracture occurrence and recommend treatment. However, a major concern regarding manual or semi-manual musculoskeletal imaging methods is their labor intensity and reproducibility. Due to the subjectivity associated with the measurement and its evaluation, accuracy and reproducibility are not guaranteed.

[0009] However, a significant concern regarding manual or semi-manual musculoskeletal imaging methods is their labor intensity and reproducibility. Due to the subjectivity associated with the measurement and its evaluation, accuracy and reproducibility are not guaranteed.

[0010] One automated method for evaluating musculoskeletal images is disclosed in Patent Document 1 titled "Method and System for Image Analysis of Selected Tissue Structures". This method automatically analyzes and evaluates musculoskeletal images such as wrist CT scans. The method can be used automatically, extracting the radius from a wrist CT scan and segmenting the radius into microstructures (for example, cortical bone, transition zones, and trabecular regions), and quantifying cortical porosity and trabecular density. can be done. bone, transition zones, and trabecular regions), and quantifying cortical porosity and trabecular density. can be done.

[0011] Known methods have focused individually on each measurement value, but this way specific musculoskeletal attributes in diseased patients For example, B does not provide any useful insight into how the disease can be differentiated from non-disease individuals. The MB score evaluates the degree of bone marrow involvement only in relation to Gaucher disease, and the BMD score evaluated bone mineral density only in relation to fracture risk, and cortical porosity was not evaluated as an important bone attribute. Although these measurements provide a quantitative estimate of bone microarchitecture, they do not provide complete information about bone microarchitecture. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] U.S. Patent No. 906432 Summary of the Invention

[0013] According to a first aspect of the present invention there is provided a computer implemented image analysis method comprising the steps of: The method includes the steps of: One or more features (e.g., structures and materials) that are segmented and identified from a medical image of the subject (including) quantifying the features; extracting non-image data relating to the subject from one or more non-image data sources; Extracting clinically relevant features from non-image data relating to the subject; Quantified features segmented from medical images and features extracted from non-image data evaluating the trained machine learning model; Outputting one or more results of the evaluation of the features.

[0014] Therefore, machine learning algorithms (e.g., deep learning algorithms) are used to analyze image data. By combining multiple features extracted from both data and non-image data, The invention enables diagnosis and monitoring of diseases and conditions in medical images, such as musculoskeletal images. To tighten.

[0015] In an embodiment, a method includes receiving an image and segmenting one or more features from the image. and identifying segmented features from the image.

[0016] In an embodiment, a method includes receiving an image with features segmented therefrom; and identifying segmented features from the image.

[0017] In an embodiment, the segmentation and identification is performed using a machine learning algorithm trained segmentation. The method is implemented by a recognition and discrimination model that extracts features from an image. For example, a trained segmenter may be used to segment and identify The application and discrimination models include deep convolutional neural network trained models. .

[0018] In an embodiment, the trained machine learning model comprises a disease classification model.

[0019] In an embodiment, the trained machine learning model is based on features extracted from patient data and disease or comprises a model trained with labels or annotations indicative of non-disease.

[0020] In embodiments, the trained machine learning model may be a deep learning neural network or other Machine learning algorithms (Support Vector Machines (SVM), Decision Trees, AdaBoost, etc.) Prepare.

[0021] In embodiments, the trained machine learning model is adapted to diagnose and treat one or more (e.g., musculoskeletal) disorders. and / or a model trained to monitor.

[0022] In an embodiment, the method includes the steps of: (i) training a trained machine learning model; and / or (ii) new or newly analyzed additional labeling derived from subject data; and (e.g., continuously) updating the trained machine learning model with the new data. Including etc.

[0023] In an embodiment, the result comprises one or more disease classifications and / or probabilities.

[0024] In an embodiment, the method further comprises generating a report based at least on the results. For example, the method may provide a report that is additionally based on information from a domain knowledge database. The method includes generating a notification.

[0025] In an embodiment, the assessment comprises a bone fragility assessment.

[0026] In an embodiment, the results include one or more fracture risk scores.

[0027] According to a second aspect of the present invention there is provided an image analysis system comprising: It has the following: Segment and identify one or more features (structure or material) from the subject's medical image a feature quantifier configured to quantify the Extract non-imaging data about the subject from one or more non-imaging data sources and clinically Non-image data configured to extract relevant features from non-image data relating to a subject. A processor; Features segmented from medical images and features extracted from non-image data are trained a feature evaluation unit configured to evaluate using a machine learning model; An output configured to output one or more results of the evaluation of the features.

[0028] In an embodiment, a system includes receiving an image, segmenting one or more features from the image, and a segmenter and a classifier configured to identify segmented features from the image; In one example, a segmenter and classifier extracts features from an image to create segments and Segmentation and identification models (i.e., machine learning algorithms) configured to The segmentation and classification model is trained using the rhythm. The mentation and discrimination models are deep convolutional neural network trained models. It may comprise:

[0029] In an embodiment, the trained machine learning model comprises a disease classification model.

[0030] In an embodiment, the trained machine learning model is based on features extracted from patient data and disease or comprises a model trained with labels or annotations indicative of non-disease.

[0031] In embodiments, the trained machine learning model may be a deep learning neural network or other Equipped with machine learning algorithms.

[0032] In embodiments, the trained machine learning model is adapted to diagnose and treat one or more (e.g., musculoskeletal) disorders. and / or a model trained to monitor.

[0033] In an embodiment, the system performs a step of generating a new or newly analyzed subject data derived Updating a trained machine learning model (e.g., continuously) with additional labeled data The method further comprises:

[0034] In an embodiment, the result comprises one or more disease classifications and / or probabilities.

[0035] In an embodiment, the system is configured to generate a report based at least on the results. The report generator may further include a report generator for generating a report based on the data (e.g., of the system). configured to generate reports based additionally on information from the main knowledge database; There are.

[0036] In an embodiment, the feature assessor is configured to assess bone fragility.

[0037] In an embodiment, the results include one or more fracture risk scores.

[0038] According to a second aspect of the present invention, when executed by one or more processors, and a computer program code comprising instructions configured to carry out the image analysis method of This aspect includes computer program code as described above (non-volatile). In one embodiment, a computer-readable medium may be provided.

[0039] Any of the various individual features of each of the above-described aspects of the invention, as well as the claims Any of the various individual features of the embodiments described herein, including those described in the claims, can be combined in any suitable and desired manner. [Brief description of the drawings]

[0040] In order that the present invention may be more clearly defined, reference will now be made, by way of example only, to the accompanying drawings, in which: The form will be explained below.

[0041] [Figure 1] 1 is a schematic diagram of a medical image analysis system according to an embodiment of the present invention; [Diagram 2]2A and 2B are schematic diagrams of exemplary image data stored in a memory of the medical image analysis system of FIG. [Diagram 3] Figure 3A is a schematic diagram of an example unpopulated, non-image data store of the medical image analysis system of Figure 1. Figure 3B is a schematic diagram of an example populated, non-image data store of the medical image analysis system of Figure 1. [Figure 4] FIG. 2 is a schematic diagram of the operation of the image data processor of the medical image analysis system of FIG. 1 in a bone fragility application. [Diagram 5] 2 is a schematic diagram of the operation of the image data processor of the medical image analysis system of FIG. 1 on exemplary non-image data including structured and unstructured data. [Figure 6] FIG. 2 is a schematic diagram of offline training of a machine learning model of the feature evaluation unit of the medical image analysis system of FIG. 1. [Figure 7] FIG. 2 is a schematic diagram of a report generator of the medical image analysis system of FIG. 1. [Figure 8] FIG. 2 illustrates an exemplary operation of the medical image analysis system of FIG. 1 in a bone fragility application, where the image data includes a wrist HRpQCT scan and the non-image data includes patient information. [Figure 9] 2 is a schematic diagram of an exemplary report generated by the report generator of the medical image analysis system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] FIG. 1 is a block diagram of an image processing system provided as a medical image analysis system 10 in accordance with an embodiment of the present invention. FIG. 1 is a schematic diagram of an analysis system.

[0043] Referring to FIG. 1, the system 10 includes an image analysis controller 12 (including a GUI 16). and a user interface 14, which typically includes one or more The above displays (at least one of which can display a GUI16), keyboard and The image analysis controller 12 includes at least one The system 10 includes a processor 18 and a memory 20. The system 10 may be implemented, for example, on a computer. As a combination of software and hardware (e.g., personal computers and mobile computers) Either implemented as a dedicated image segmentation system The system 10 may optionally be distributed; for example , all or some components of the memory 20 may be located remotely from the processor 18. The user interface 14 may be configured with a memory 20 and / or a processor. It can be located remotely from the server 18, and in fact it can be accessed through a web browser and and mobile device applications.

[0044] The memory 20 is in data communication with the processor 18 and is typically volatile. and non-volatile memory (and one or more of each memory type). (which may include a RAM (random access memory), a ROM, and one or more This includes mass storage devices.

[0045] As will be described in more detail below, the processor 18 includes a segmenter and classifier 24 and a An image data processor 22 (including a feature quantification unit 25) and a non-image data processor 26 a feature evaluator 28 (including a precedent comparator 29); a machine learning model trainer 30; and a report generator. The memory 2 includes a memory unit 32, an I / O interface 34, and a result section which is a result output 36. 0 is a program code 38, image data 40, non-image data 42, training data 44, training The image analysis controller 12 includes a trained machine learning model 46 and domain knowledge 48. A processor executing program code 38, at least in part, from memory 20. It is implemented by 18.

[0046] Generally speaking, the I / O interface 34 receives image data relating to a subject or patient. and non-image data (e.g., DICOM format) in the memory 20. The image data processor 22 is configured to read or receive the image data into the image data processor 42. A segmenter and classifier 24 and a feature quantifier 25 are used to segment features from the image data. The non-image data processor 26 extracts (i.e., extracts) and quantifies the features from the non-image data. The feature evaluator 28 processes the features with one or more machine learning models 46, and The I / O interface 34 transmits the results of the analysis to, for example, a result output 36 and / or the GUI 16. Output.

[0047] System 10 is a machine learning algorithm that uses artificial intelligence (e.g., deep neural networks) and other machine learning algorithms. Feature extraction and image feature quantification using algorithms and computer vision algorithms This is particularly useful in musculoskeletal image analysis, as it automates the process of performing skeletal muscle analysis accurately and reproducibly. The results are suitable for use in localizing the affected area of ​​musculoskeletal disorders, diagnosing musculoskeletal disorders, and monitoring disease progression. This will become the case from now on.

[0048] Referring to FIG. 1, the system 10 receives two types of subject or patient data: Image data (e.g., one or more imaging modalities) Medical images acquired at the site of interest) and non-image data (structure of medical history data, interview data, etc.) System 10 is a system that collects and stores patient data (structured patient data, as well as unstructured patient data, such as physician notes and voice recordings). These data are stored in image data 40 and non-image data 42, respectively. The various data formats and structures govern the behavior of the non-image data processor 26 .

[0049] The image data processor 22 comprises two components: segmentation a segmenter and classifier 24 configured to perform feature quantification and classification; The image data 40 is processed by the image data processor 22. The images are then processed using a segmenter and classifier 24 to identify clinically relevant structures, features or materials. In this embodiment, the method automatically segments and identifies a medical image of a subject or patient. The segmentation and classification are performed from the feature quantification unit 25. and quantification of identified clinically relevant structures, features, materials, or combinations thereof. However, in some embodiments, the image is received already segmented. It should be noted that the system 10 can be configured Optionally, the segmented features may already be identified. The vessel 24 may be omitted or its use eliminated.

[0050] In this embodiment, segmentation and classification are performed using conventional image processing methods (i.e., non- It is implemented using machine learning (ML) algorithms, such as thresholding, contour detection, blob detection, and rice. Patent No. 9,064,320 (title: "Image Analysis of Selected Tissue Structures") "Method and system for

[0051] However, in some other embodiments, the segmenter and classifier 24 is implemented using machine learning algorithms. A segmentation and discrimination model trained on rhythms can be provided, which can be used to The method can be configured to segment and identify structures or features of interest from For example, in bone fragility applications, deep convolutional neural networks are used to train Such a model was developed to segment and identify the radius from wrist HRpQCT scans. The training data can be used to develop algorithms that distinguish between voxels of the radius, ulna, and surrounding tissue. It can be an annotated wrist HRpQCT scan.

[0052] The non-image data is processed by a non-image data processor 26, which converts the non-image data into clinical data. Specifically, relevant features are extracted.

[0053] The quantitative features obtained from both the image data and the non-image data are evaluated by a feature evaluation unit 28. The labeled training data is input into the machine learning model 46. It is pre-trained with training data, i.e. annotations that serve as the correct answer for machine learning. For example, to train a bone segmentation model, the training data is typically The system is equipped with a set of images and their corresponding ground truth images, i.e., images with bones annotated. To train a disease classification model, training data is typically extracted from patient data. The system is equipped with the features and the corresponding ground truth images, which indicate disease or non-disease. Label / annotation. (The terms "label" and "annotation" are essentially interchangeable.) Although the term is interchangeable with "label," it is generally used to refer to diseases and conditions. "Annotation" generally refers to image segmentation / classification machine learning models. It is used.)

[0054] In this embodiment, the processor 18 includes a machine learning model trainer 30. The trainer uses training data 44 (which may include new subject data) to train a machine learning model 46 46 (and retraining or updating the machine learning model 46, as described below). However, in other embodiments, the machine learning trainer 30 may train the machine learning model 46 as It may be configured or used only for retraining or refresher training.

[0055] Various machine learning algorithms (using features from both image and non-image data) The machine learning model 46 used in this embodiment is trained using a can be used to diagnose and monitor, for example using deep learning neural networks (preferred) or support It can involve other machine learning algorithms such as Support Vector Machines (SVM), decision trees, and AdaBoost. do.

[0056] In one embodiment, the pre-trained machine learning model 46 is adapted to perform the following steps: It is continually updated with additional labeled data derived from the data.

[0057] Therefore, the feature evaluation unit 28 uses a machine learning model 46 to evaluate and assess the features. and also transmits the results (e.g., in the form of classifications or probabilities) to a report generator 32. The report generator 32 generates a report based on these results. The processor 32 can obtain additional information from domain knowledge 48 and combine that information with the results. and both can be presented in the report.

[0058] For example, in a bone fragility assessment application embodiment, the feature evaluator 28 may use a machine learning model 46 to The report generator 32 may use this score to output results including a fracture risk score. The information may be obtained from domain knowledge 48, The information may be, for example, to aid in the interpretation of the scores. Both data and information may be presented in the report.

[0059] The final report is output via the results output 36 and / or the user interface 14. .

[0060] 2A and 2B are schematic diagrams of exemplary image data stored in image data 40. The image data includes clinically relevant medical images of a subject or patient. Typically, images are acquired at one or more regional locations using one or more imaging modalities. The image is the target.

[0061] For example, in the bone fragility example, as shown in FIG. 2A, the input image data may include a wrist In another example, both the peripheral and central skeletons may be analyzed. As shown in FIG. 2B, it may be desirable to input image data including wrist HRpQ. CT scan 50, leg HRpQCT scan 52, spine DXA scan 54, and lumbar DXA scan56 (and bone marrow information, if optionally integrated) and lumbar MRI scan58).

[0062] 3A and 3B show the results for non-image data with and without data, respectively. FIG. 1 is a schematic diagram of an example of non-image data 42 received by the system 10. The non-image data stored in the database contains a significant amount of useful data that may indicate musculoskeletal disorders. As shown in Figure 3A, non-image data may come from a variety of structured and unstructured data sources. This may include patient information collected, for example, during the examination or treatment of a subject or patient. Structured data includes basic information about subjects, such as gender, age, weight, and height, and Laboratory test results (e.g. blood test results, DNA test results, etc.) and This may include medical data (e.g., medical history, alcohol history, fracture history, etc.) and medical history data (e.g., smoking history, alcohol history, fracture history, etc.). Unstructured data includes text documents of clinical test results, doctor's notes, radiology reports, etc. It may include.

[0063] As shown in FIG. 3A, the non-image data in non-image data 42 may have various formats, such as: It can be structured video 62, text 64, numeric 66, and / or The data includes audio 68, as well as unstructured video 72, text 74, numbers 76 and / or is audio 78. FIG. 3B illustrates an example embodiment in which the non-image data may include: Structured text 64 in the form of medical history data and interview data 80, and patient measurements 82 66 structured numeric values, 84 unstructured videos in the form of patient walking videos, and 72 in the form of doctor notes. and unstructured audio 78 in the form of a patient record 88.

[0064] The non-image data processor 26 uses different data processing techniques to process the non-image data. The present invention is configured to have and use a method and a feature extraction method. , in each case according to the structure and format of each piece of data.

[0065] FIG. 4 is a schematic representation of the operation of the image data processor 22 in a bone fragility test. 0. In this example, we use a pre-trained deep neural network model to The bones were identified and segmented from the wrist HRpQCT scan. The radius was The segmenter and classifier 24 distinguishes between dense cortex 94, transition zone 96, and trabecular bone region 98. The different structures and groups of structures are then segmented and identified by the feature quantification unit 25. The combined attributes are quantified into the following features: vBMD (volumetric bone mineral density of the whole radius) ) 102, cortical porosity (percentage of pores within bone volume) 104, transition zone vBMD (transition zone vBMD) trabecular bone mineral density (vBMD) (volumetric bone mineral density of trabecular bone region) 106; trabecular bone mineral density (vBMD) (volumetric bone mineral density of trabecular bone region) 108; Matrix mineralization level (average percentage of mineralized material) 110, medullary hyperlipid status (trabecular area medullary fat density (mean thickness of the dense cortex)112, cortical thickness (mean thickness of the dense cortex)114, and trabecular tissue Separation (the average separation distance of trabecular tissue)116. These features indicate bone fragility and fracture risk. In other applications, features related to the state of interest may be chosen similarly. Select.

[0066] FIG. 5 illustrates an exemplary non-image data processor 26 including structured and unstructured data. 1 is a schematic representation 120 of operations on data 42. 26 extracts features from non-image data, and various tools have different structures and The present invention is adapted for extracting non-image data types and formats.

[0067] Typically, the structured data 122 is stored in a database table, a .json file, or a . Stored and maintained in structured data stores such as .xml or .csv files The non-image data processor 26 extracts features from the structured data, if necessary. It does this by querying 124 the parameters and attributes to be processed from the data source, This extracts information of interest from structured data sources. The data may be complete and of interest in itself, making query unnecessary. do.

[0068] Unstructured data 126 may include physician notes, audio recordings, graphical reports, etc. Therefore, prior to extraction, the non-image data processor 26 typically processes the unstructured data 126 as The conversion method used by the non-image data processor 26 is For example, physician notes are structured data. To convert the image data into non-image data, the non-image data processor 26 uses a trained optical character reader. By including or utilizing OCR, the notes can be processed by the system 10. 128 can be converted into recognizable text by a non-image data processor. 26 is a structured response to text (using keywords such as "fracture," "pain," and "fall"). Perform sentence analysis.

[0069] As another example, the non-image data processor 26 may include a trained speech recognition model. By using the voice recording of the interview, the system 10 can convert the voice recording into a text that can be recognized by the system 10. The non-image data processor 26 can then convert the original interview data into Segment and organize the converted text into structured data by referencing the question .

[0070] In another exemplary embodiment, for example, a pattern related to gait is associated with fracture risk. Non-image data may include video, as research has shown that non-image data Processor 26 processes the video to extract gait features therefrom.

[0071] Once converted 128 into structured data, the non-image data processor 26 (previously The aim is to extract features from structured data (which was previously unstructured data), and not from the data source. It does this by querying 124 the necessary parameters and attributes from As mentioned above, currently structured data is complete and of interest. In the above walking video example, the video is (structured data) After conversion to gait features (constituting gait feature vectors 124), such queries 124 are not necessary.

[0072] Ultimately, the non-image data processor 26 may include a number of different types of information, such as gender 132, age 134, smoking history 136, and the like. Features 130 such as fracture history 138, treatment history 140, and gait 142 are extracted.

[0073] FIG. 6 is a schematic representation of the (offline) training of a model in a machine learning model 46. First, in S146, the training data is accessed or the data of the training data 44 is The training data may be selected from the subject image data and / or subject It may include non-image data. Data with various attributes may be selected for training to target Using the above method, the training data is as follows: Extraction of features; quantification of features extracted in S150. Then, the training data is converted into a set of quantitative features. This will result in the company having a

[0074] In S152, the training data is annotated (by a human operator) In S154, the training data represented by the extracted features is diagnosed. In combination with the fragment labels, one or more machine learning algorithms of the machine learning model trainer 30 One or more models are trained by feeding them into the dataset. Note that black box algorithms include machine learning algorithms. It can be a system (e.g., a neural network) whose decision process is beyond the scope of human knowledge. Alternatively, it can be a white-box algorithm (e.g., decision The model can be a tree, support vector machine, or linear discriminant analysis, and is modeled by humans. The decision-making process of the algorithm can be interpreted. It is also possible to create a hybrid algorithm by combining black box algorithms. can.

[0075] In S156, the currently trained model is deployed for use, typically by a machine. This will be stored in the learning model 46.

[0076] When used, features extracted from new patient data are applied to the currently trained machine. The features are then fed into the machine learning model 46. As described above, the machine learning model 46 evaluates the features It outputs one or more results, which can be a binary classification, a score, or a probability. In some embodiments, (optional step S158) the new patient data is used as training data. In addition to 44, it can be used to retrain or update the model 46.

[0077] The training of the model and its usefulness are domain specific. Each model 46 is modeled on the basis of the data and target conditions (e.g., For example, diseases such as bone fragility and osteoporosis, and infections such as certain bacterial infections. In an embodiment, one of the models 46 is trained and used to diagnose one or more diseases or conditions. This can be achieved by generating one or more disease probability scores. In this embodiment, multiple models 46 can be trained and used to diagnose each disease. Similarly, in another embodiment, one of the models 46 can be used to measure one or more symptoms, e.g. A model 46 can be trained to generate a prediction of the likelihood of occurrence of a fracture (fracture) or train multiple models 46 and use them to generate respective predictions about the likelihood of each symptom occurring. It is possible.

[0078] FIG. 7 is a schematic diagram of the report generator 32, which uses the machine learning model 46 The report generator 32 generates a report based at least on the results 162 obtained by the report generator 32. In this case, information stored in the domain knowledge 48 can be used. The characteristic evaluation unit 28 may include information that may be useful to a physician in interpreting the results output by the characteristic evaluation unit 28. For example, a system for fracture assessment may be developed (e.g., guidelines for diagnosis and treatment). 10, providing clinicians with treatment guidelines for different levels of fracture risk. The report can then include the results (subject or patient) generated by the feature evaluation unit 28. characterizing a subject's bones, such as a fragility score, porosity score, or trabecular bone score; etc.) as well as information to aid in the interpretation of those scores and selection of treatments.

[0079] This information may optionally be based on previous subject(s) with comparable scores or may include the diagnosis of a cohort of patients and / or the treatment or therapy prescribed for that cohort. Indeed, in some embodiments, feature evaluator 28 includes an optional precedent comparator 29 so that This is because the results generated by machine learning models46 (especially numerical values ​​such as scores) 48 such information from domain knowledge (if the subject has a disease) Based on the diagnosis and comparison (by assigning a probability or probability to each) Automatically generate treatment / therapy recommendations and output these to a report generator 32 for inclusion in a report. It is configured to be included.

[0080] Reports may also include results of disease progression monitoring and treatment efficacy. Treatment was successful if follow-up examinations demonstrated a reduction in fracture risk. It can be regarded as.

[0081] Reports may also display similar or opposite cases. For the purposes of the decision, it may be useful to present similar or opposite cases. Presenting examples can help train novice users or improve the performance of models46 It helps users to verify the results and to study similar or opposite cases. By doing so, it is possible to learn how previous subjects or patients responded to different treatment options. This will provide insight into the effectiveness of treatments (or the estimated effectiveness of proposed treatments) for current patients. This will allow medical professionals to evaluate the effectiveness of the treatment.

[0082] All of these results and information are included in the report 166 by the report generator 32. The report generator 32 then outputs the report 166 (via the results output 36) to the user interface 1 4 (e.g., a web browser, a PC application, or a mobile device app) Deliver it to the user for viewing.

[0083] 8 illustrates an exemplary embodiment of the system 10 in a bone fragility application. Image Data The non-image data includes a wrist HRpQCT scan 172 and the non-image data includes patient demographics 174 . As shown in FIG. 8, the segmenter and classifier 24 (trained segmentation and classification Using model 46, we identify and extract 176 the radius and divide it into structures (dense cortex, transition zone , and trabecular bone regions). Based on this structural segmentation, The feature quantifier 25 determines the cortical porosity 180 and trabecular bone density 182 .

[0084] The non-image data processor 26 uses the query to compile non-image data basic patient information 174 The feature evaluation unit 28 extracts the four features, cortical perforation, The characteristic evaluation unit 28 receives the sex 180, the trabecular bone density 182, the gender 186, and the age 188. Using another pre-trained model 46'' (in this example, the bone collapse model), we compared the four features mentioned above. A structural vulnerability score of 190 was calculated by assessing the characteristics of Alternatively, the bone collapse model46' was developed by Zebaze et al. ty and Reduced Trabecular Density Are Not Necessarily Synonymous With Bone Loss “Increased cortical porosity and decreased trabecular density are associated with "This is not necessarily synonymous with bone loss and microarchitectural deterioration" (JBMR Plus (2018)) The training data 44 can be trained using the algorithm described herein. These represent the four characteristics mentioned above180,182,186,188, and allow for a correct diagnosis of bone fragility. It has been annotated accordingly.

[0085] The vulnerability scores 190 and other information are used by the report generator 32 to generate the reports 166. This ultimately results in a user interface 14 (in this example The output is sent to a web browser.

[0086] FIG. 9 illustrates an example report 200 generated by report generator 32 of system 10. Report 200 includes several sections, including: Subject / Patient Details section 202, Bone Volume Fraction map field 204, score field 206 and score graph field 208. Subject / Patient Details Column 202 typically includes bibliographic information (e.g., name, date of birth, sex, and age), physician Name (or names of physicians), date of data acquisition, and date of data processing (if such date is reported) (It also serves as the date of Proclamation 200.)

[0087] The bone volume fraction map section 204 is a false color 3D reconstruction of the radial bone volume of the patient in this example. Example 210 and the corresponding reference example (typically healthy) radial volume false color 3D It includes an example reconstruction 212 and a false color legend 214. (Note that the colors in FIG. 9 are reconstructed in grayscale.) (Note that the false color is used to represent "bone volume fraction.") , which is the volume of mineralized bone per unit volume of sample. 210,212 are presented side by side to allow medical personnel to assess the extent of (in this case, bone loss) This allows the degree and distribution to be easily evaluated.

[0088] The score column 206 contains the scores generated by the feature evaluator 28. In this example, These scores were a fragility score of 216, a porosity score of 218, and a trabecular score of 220. The fragility score 216 is expressed as a percentage of the total cortical and trabecular bone. The porosity score 218 indicates the degree of porosity of the cortical bone. The trabecular bone score 220 indicates the degree of porosity of the bone. This is a score indicating the density of the beam region.

[0089] Optionally, these scores 216, 218, 220 are graphically represented by respective bars 222, 223. For example, the score can be expressed on the sliders 228, 230, and 232. In this example, each bar is labeled with three values: Lower and upper values ​​that indicate core, and decreased and increased risk (or normal and abnormal) For example, for the vulnerability score 216 in the example shown, In bar 222, they are 20%, 90% and 70%, respectively. For the bar 224 for the core 218, the values ​​are 20%, 60%, and 46%, respectively. In the example shown, the bars 226 for the trabecular bone score 220 are 0%, 10%, and 2%. It is.

[0090] These three values ​​(the lower limit, the upper limit, and the cutoff between decreasing and increasing risk) are , for example, from the training data 44 or precedent data contained within the domain knowledge 48. Alternatively, the system 10 may access one or more external databases to perform all or part of the Precedent data can be obtained.

[0091] Optionally, the bars 222, 224, 226 are colored or shaded. 234 and areas of increased risk. This will result in the sliders 228, 230, and 232 (and therefore the score) 216,218,220) have been associated with precedent data in high- or low-risk subjects. In FIG. 9, it is possible to immediately and visually clarify whether the data corresponds to the Darker shading is used to indicate areas of higher risk.

[0092] Thus, in score field 206, report 200 yields the following score: Vulnerability score 2 75% as 16 (indicating increased risk); 37% as porosity score (indicating decreased risk) ); and trabecular bone score of 1% (indicating increased risk). These scores alone or in combination However, they do not constitute a diagnosis because they do not identify a specific disease or affliction. However, these may vary depending on the symptoms and how pronounced they are. Inform the user of the information that is available.

[0093] The score graph section 208 displays the fragility score (FS((%)), the porosity score (PS(( %)), and trabecular bone score (TS((%)) plots 240, 242, 244 Each of them changes with the passage of time T. In the example shown in the figure, Axial divisions are typically 1-2 days long, but in general each division represents the time between scans. (That is, it can be said that the divisions in a single graph do not necessarily represent a constant time difference. In FIG. 9, plots 240, 242, and 244 first plot the scores in the score column 206. The scores are presented in Table 1, followed for each case by the scores obtained at a series of follow-up examinations. Therefore, plots 240, 242, and 244 allow medical personnel to quickly detect changes occurring in subjects. This allows rapid evaluation of changes, including spontaneous changes and changes in response to treatment. Including changes and other changes.

[0094] Those skilled in the art will recognize that many modifications can be made without departing from the scope of the invention. In particular, certain features of the embodiments of the present invention may be used to provide further embodiments. Note that it is possible.

[0095] If any references to prior art are made in this specification, such references are not to be construed as limiting the scope of the invention. It should be noted that a prior art application does not constitute an admission that the prior art forms part of the public knowledge in any country. I want to.

[0096] In the appended claims and the foregoing detailed description of the invention, Therefore, unless the context otherwise requires, the terms "have" and "have" (third person singular Terms such as "has" and "has" are used in a comprehensive sense (i.e., the presence of the declared feature). Although the term "presence" is used to indicate the presence of additional features in various embodiments of the invention, This does not preclude the inclusion or addition of other elements. [Explanation of symbols]

[0097] 12 Image Analysis Controller 14 User Interface 16 GUI 18 Processors 20 Memory 40 Image data storage unit 44 Training Data 48 Domain knowledge 200 reports

Claims

1. 1. A method of performing computer-implemented image analysis on a computer, the method comprising: receiving a medical image, or segmenting one or more features from the medical image, or receiving a medical image with one or more features segmented therefrom; identifying the one or more features segmented from the medical image; and quantifying the one or more features segmented and identified from the medical image. extracting non-image data relating to the subject from one or more non-image data sources; extracting clinically relevant features from the non-image data relating to the subject; evaluating the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data with a trained machine learning model; and outputting one or more results regarding an evaluation of the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data.

2. A method as described in claim 1, wherein the segmenting and identifying is implemented by a segmentation and identification model trained with a machine learning algorithm, and the identification model is configured to segment and identify the one or more features from the medical image.

3. The method of claim 1, wherein the trained segmentation and discrimination model comprises a deep convolutional neural network trained model.

4. In the method according to claim 1 or 2, the trained machine learning model A method comprising a disease classification model, a model trained using features extracted from patient data and labels or annotations indicating disease or non-disease, and / or a deep learning neural network or other machine learning algorithm.

5. A method according to any one of claims 1 to 3, wherein the trained machine learning model is a model trained to diagnose and / or monitor one or more musculoskeletal disorders.

6. A method as described in any one of claims 1 to 3, further comprising performing, by the processor, (i) a step of training the trained machine learning model, and / or (ii) a step of updating the trained machine learning model with new or newly analyzed additional labeled data derived from subject data.

7. A method according to any one of claims 1 to 3, wherein the results include (i) one or more disease classifications; (ii) one or more disease probabilities; and / or (iii) one or more fracture risk scores.

8. A method according to any one of claims 1 to 3, wherein the assessment includes a bone fragility assessment.

9. A system for image analysis, comprising: a segmenter and classifier configured to receive a medical image of a subject, receive one or more features segmented from the medical image, and identify the one or more features segmented from the medical image; a feature quantifier configured to quantify one or more features segmented and identified from the medical image; a non-image data processor configured to extract non-image data relating to the subject from one or more non-image data sources and to extract clinically relevant features from the non-image data relating to the subject; a feature evaluator configured to use a trained machine learning model to form an evaluation of the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data; an output unit configured to output one or more results of an evaluation of the one or more features segmented and identified from the medical image and the clinically relevant features extracted from the non-image data.

10. The system of claim 9, wherein the segmenter and classifier are provided with a segmentation and identification model trained with a machine learning algorithm, and the identification model is configured to segment and identify the one or more features from the medical image.

11. The system of claim 10, wherein the trained segmentation and discrimination model comprises a deep convolutional neural network trained model.

12. A system as described in any one of claims 9 to 11, wherein the trained machine learning model comprises a disease classification model, a model trained using features extracted from patient data and labels or annotations indicating disease or non-disease, and / or a deep learning neural network or other machine learning algorithm.

13. A system as described in any one of claims 9 to 11, wherein the trained machine learning model is a model trained to diagnose and / or monitor one or more musculoskeletal disorders.

14. A system as described in any one of claims 9 to 11, further comprising a machine learning model trainer configured to update the trained machine learning model with additional labeled data derived from new or newly analyzed subject data.

15. A system as described in any one of claims 9 to 11, wherein the results include (i) one or more disease classifications; (ii) one or more disease probabilities; and / or (iii) one or more fracture risk scores.

16. A system according to any one of claims 9 to 11, wherein the feature evaluation unit is configured to evaluate bone fragility.

17. Computer program code comprising instructions configured to, when executed by one or more computing devices, implement the image analysis method of any one of claims 1 to 8.

18. A computer-readable medium comprising the computer program code of claim 17.

Images