Method and system for machine learning classification based on structure or material segmentation in image

Abstract

To provide a system and method for classifying a structure or material in an image of a subject.SOLUTION: The system comprises: a segmenter configured to form one or more segmentations of a structure or material in an image and generate from the segmentations one or more segmentation maps of the image including categorizations of pixels or voxels of the segmentation maps assigned from one or more respective predefined sets of categories; a classifier that implements a trained classification machine learning model configured to generate, based on the segmentations maps, one or more classifications and to assign to the classifications respective scores indicative of a likelihood that the structure or material, or the subject, falls into the respective classifications; and an output for outputting a result indicative of the classifications and scores.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention classifies images (e.g., medical images) based on structure or material segmentation. Use machine learning, especially deep neural networks, to measure (and monitor) changes in Possible medical imaging applications include computer vision systems and methods. Computed tomography (CT), magnetic resonance (MR), ultrasound, HRpQCT (High-Resolution Photon Quantum Computed Tomography) eripheral Quantitative Computed Tomography (high-resolution peripheral bone quantitative CT)), and This includes lesion scanner imaging, etc.

[0002] This application is a joint venture with U.S. Patent Application No. 16 / 448,474 (filed June 21, 2019). No. 6,313,935, filed on Dec. 13, 2003, and which claims priority thereto, the entire contents of which are incorporated herein by reference at the time of filing. Be taken in. [Background technology]

[0003] Computer vision and image processing techniques have been applied to medical image analysis. The computer-assisted system performs the analysis in two steps: segmentation. Segmentation is the process of dividing pixels in an image (2D image) or Identifying structures or objects in images, such as medical images, by distinguishing voxels (3D images) Based on segmentation, Quantitative features such as volume, shape, and density are calculated. For example, brain CT and MRI scans. After the lesion is segmented, the size and shape of the lesion are calculated; A (dual-energy x-ray absorptiometry) scan After the femoral neck has been segmented, bone mineral density can be calculated. Such calculations are compared with healthy controls. After comparison with the reference data, the physician can make a diagnosis or treatment decision.

[0004] For example, a T-score is given to a patient's bone mineral density when compared to a young normal reference mean. The World Health Organization (WHO) defines osteoporosis as a T score of -2.5 or less. That is, bone density 2.5 standard deviations below the mean for a healthy 30-year-old man / woman. This corresponds to the degree.

[0005] Segmentation can be accomplished manually, semi-manually, or automatically. In one example, a user operates a computer to draw a rectangular box on the lumbar DXA scanner. Move over the scan, thereby selecting the area of ​​the femoral neck.

[0006] Semi-manual segmentation is an image processing program that utilizes user initialization or input. For example, the user can operate the computer to approximate the bone boundaries of the wrist. It can be written onto a CT scan, and the program can then extract the tissue from the surrounding area. The approximate boundary is then adjusted to a contour that segments the bone.

[0007] Automatic segmentation is based on features of objects of interest (e.g., signal intensity values, edges, and shapes). One existing example is the voxel-based A thresholding technique can be used to segment bone from surrounding tissue in a CT scan. Gram uses machine learning algorithms to segment abnormal tissue in medical images. Train the classifier, for example by using tumor and normal images as training data. Feature-based machine learning algorithms (e.g., Support Vector Machines (SVMs) and decision A trained classifier is used for each "window" of new images. The image is scanned to segment any image regions that contain tumors.

[0008] Machine learning algorithms have shown promising accuracy and efficiency in this field. However, collecting sufficient training data and annotating the training data is a challenge. Both of these are quite difficult. Training images must be annotated by experts. This is a tedious and time-consuming process. Even for experts, it is extremely difficult or almost impossible to accurately annotate training images. For example, in bone quality assessment, the cortical and A transition zone consisting of both marrow and trabecular bone is present in every sample. It consists of an inner cortex adjacent to the canals and trabecular bone attached to the cortex and adjacent to the inner cortical surface. In the transition zone, active bone remodeling is taking place. It is important to identify and segment the image area, but due to the limitations of image resolution, it is difficult for even an expert to do so. Accurately and consistently annotating the emoko domain is a virtually impossible task. If there are no annotated images as training data, segmentation I can't train the model.

[0009] In the past few years, machine learning or deep neural networks have been applied to many applications such as natural image classification. It has surpassed human performance in many visual recognition tasks. In one embodiment, the network The network has an input layer, a hidden layer, and an output layer. An image is fed into the network through the input layer. The image is sampled and a convolution operation is applied to generate the hidden layer. The output of each layer is The output layer is fully connected at the end and is used as the input for the next layer in the network. , the classification results are output. The training data are images with classification labels. The training process obtains the parameters of the neural network. After the training is completed, A new image is processed by the neural network with the acquired parameters. For example, OCT (Optical Coherence Tomography) Method) Determine the status of diabetic retinopathy (e.g., none, mild, moderate, severe, etc.) from the images. We use deep learning neural network algorithms to train models on It can be used.

[0010] However, this end-to-end solution poses two problems for clinical practice: First, end-to-end solutions are black boxes. The input is a medical image and the output is a classification of the disease or condition. The process by which the network arrives at its decisions is difficult to interpret, which makes the classification results less reliable. Second, this solution requires a significant amount of As mentioned above, in medical applications, the training data requires Labeling and grading is a tedious and time-consuming task. Therefore, collecting sufficient training data for each type of classification outcome is It will be quite a test. Summary of the Invention

[0011] According to a first aspect of the present invention, a system for classifying structures or materials in an image of a subject is provided. The present invention provides a system comprising: A segmenter that identifies structures or materials in an image (including, for example, a medical image). Form a segmentation of the image above and also extract one The segmentation map above, each of which includes one or more sets of predefined categories. The category of the pixel or voxel in the segmentation map assigned from a segmenter that generates a segmentation map including a categorization of the image; A trained classification machine learning model that generates one or more and determine which structures, substances or subjects fall into each classification. a trained classification machine configured to assign a score to each classification indicating the likelihood that the classification A classifier that implements the learning model; and an output unit for outputting a result indicative of the classification and score.

[0012] In an embodiment, the classifier is based on a segmentation map and non-image data relating to the subject. ,generates one or more classifications based on the,system.

[0013] The system can be configured to train a classification machine learning model.

[0014] In an embodiment, the segmenter: i) for pixels or voxels that are assigned from a predefined set of structural categories, The method is configured to generate a structural segmentation map including all of the categorizations. Structural segmenter, ii) For pixels or voxels that are assigned from a predefined set of material categories configured to generate a material segmentation map including a categorization of The selected substance segmenter, and / or iii) Pixels or voxels assigned from a predefined set of abnormal / normal categories. To generate an anomaly segmentation map that includes categorization for the cells. The present invention includes an anomaly segmenter configured as follows:

[0015] For example, the structural segmenter uses a structural segmentation machine learning model to extract structural information. The material segmenter is configured to generate a segmentation map. mentation to generate material segmentation maps using machine learning models The anomaly segmenter uses the anomaly segmentation model to perform anomaly segmentation. The structural segmenter is configured to generate a structural segmentation map. The material segmenter may be configured to train a machine learning model for detecting the material. The method may be configured to train a quality segmentation machine learning model to detect anomalies. The segmenter is configured to train an anomaly segmentation machine learning model. This can be done.

[0016] In an embodiment, the system performs a segmentation map before the segmentation map is input by the classifier. A segmentation map processor configured to process the segmentation map For example, the segmentation map processor may further include a segmentation map processor. The 3D filter is configured to downsample the annotation map.

[0017] In an embodiment, the classification machine learning model is a neural network, a support vector machine, For example, classification machine learning models include SVM, decision trees, or a combination of these. The model includes convolutional neural networks and fully connected neural network layers. It may include a neural network.

[0018] In an embodiment, the image is a medical image and the classification is based on whether the structure or material or subject is Each corresponds to the probability of developing a particular condition or symptom within a certain time period. For example: In other words, the period is divided into a short-term period, a long-term period, and a small period between the short-term period and the long-term period. In another example, the condition or symptom is a bone fracture.

[0019] In an embodiment, the image is a medical image and the classification is based on whether the structure or material or subject is Each condition or symptom corresponds to a probability of occurring. For example, a condition or symptom is Have a bone-related condition.

[0020] In an embodiment, the image is a medical image and the classification is for progression of a disease or condition. Each speed corresponds to a probability. For example, the classifications are stable, slightly degraded, and accelerating. The classification may include classifications corresponding to any one or more of the degradations.

[0021] In an embodiment, the images are medical images and the classification is based on the effectiveness of each treatment option. For example, treatment options may include antiresorptive and / or anabolic therapy. It may include.

[0022] In an embodiment, the images are medical images and the classifications correspond to respective medical conditions. For example, the medical condition may include one or more of osteomalacia, tumor, osteonecrosis, and infection. obtain.

[0023] In an embodiment, the classification machine learning model is adapted to analyze image data and non-image data associated with a training subject. A model is trained with data related to a subject (typically one or more images) Each score is generated based on image data and non-image data.

[0024] According to a second aspect of the present invention, a computer-aided method for classifying structures or materials in an image of a subject is provided. A computer-implemented method is provided, the method comprising: forming one or more segmentations of structures or materials in the image; The segmentation produces one or more segmentation maps for the image. Each predefined set of structure / material categories is assigned a segmentation A segmentation map containing the categorization of the pixels or voxels in the segmentation map is generated. generating Using a trained classification machine learning model, one or more and each classification indicating the likelihood that a structure, substance or subject falls into each classification. assigning a score to the classification; and outputting the results indicative of the classification and score.

[0025] In an embodiment, a classification machine learning model is used to generate a segmentation map for a subject. and generating one or more classifications based on the group and non-image data.

[0026] The method may include training a classification machine learning model.

[0027] In an embodiment, the step of forming the one or more segmentations comprises: i) for pixels or voxels that are assigned from a predefined set of structural categories, generating a structural segmentation map including all of the categorizations; ii) For pixels or voxels that are assigned from a predefined set of material categories generating a material segmentation map including a categorization of the material; and / or iii) Pixels or voxels assigned from a predefined set of abnormal / normal categories. A process for generating an anomaly segmentation map including categorization of cells. Includes top.

[0028] For example, the method may include performing structural segmentation using a structural segmentation machine learning model. generating a material segmentation map; and detecting the material segmentation map using a material segmentation machine learning model. generating a segmentation map; and and generating an anomaly segmentation map using the structure. Segmentation machine learning model, material segmentation machine learning model, and / or may include training an anomaly segmentation machine learning model.

[0029] According to a third aspect of the present invention there is provided a computer implemented diagnostic method comprising: The method includes the method of the second aspect.

[0030] According to a fourth aspect of the present invention, a classifier for classifying structures or materials in an image of a subject is provided. A computer-implemented method for training a machine learning model is provided, the method comprising: The annotated segmentation map and the annotated non-image data are a step of dividing the training set and the test set into a training set and a test set; Each test set contains some annotated segmentation maps and some non-annotated image data), and one or more annotated segmentation maps. and (a) implementing a classification machine learning model, a step of initializing a parameter of the driver; (b) A classification machine learning model is created by running a learning algorithm on the training data. updating parameters of the rule; (c) testing the classification machine learning model on the test data; and (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) through (d) if performance is insufficient in step (d); and Output the classification machine learning model as a trained classification machine learning model for deployment, or flagging the classification machine learning model as a trained classification machine learning model; Includes.

[0031] This aspect may be used in conjunction with or in combination with (or as part of) the second aspect. which may be used, for example, to train a classification machine learning model of the second aspect.

[0032] The method (e.g., in generating annotated segmentation maps) This may include segmenting one or more images.

[0033] In an embodiment, the method further comprises the steps of: Receive and use more image and non-image data to train machine learning models This includes:

[0034] Classification models can be trained using a variety of machine learning algorithms, so e.g. , neural networks, support vector machines (SVMs), decision trees, or It may include a combination of the above.

[0035] Thus, in one embodiment, the classification machine learning model comprises artificial neurons. It has a neural network with multiple layers, and the parameters are the number of layers, neurons The number of neurons, neuron weights, and neuron function parameters. Testing a learned model involves testing a classification machine learning model on test data. nothing.

[0036] In an embodiment, updating the parameters includes determining the slope of the loss function. do.

[0037] In an embodiment, the images are medical images and the non-image data includes medical records.

[0038] In an embodiment, the method comprises: Dividing the scanned image data into a training set, a development set, and a test set; Development data can be used to examine learning procedures and tune parameters. (Also, if the classification machine learning model has a neural network, the layers (This includes training).

[0039] According to this aspect, a classification machine learning model for classifying structures or materials in an image of a subject is provided. A system for training a user is provided, the system comprising a processor. and the processor: The annotated segmentation map and the annotated non-image data are , a training set and a test set, A station map is obtained by segmenting one or more images. , steps, (a) implementing a classification machine learning model, a step of initializing a parameter of the driver; (b) A classification machine learning model is created by running a learning algorithm on the training data. updating parameters of the rule; (c) testing the classification machine learning model on the test data; and (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) through (d) if performance is insufficient in step (d); and Output the classification machine learning model as a trained classification machine learning model for deployment, or flagging the classification machine learning model as a trained classification machine learning model; The present invention is configured to:

[0040] According to a fifth aspect, the invention provides a method and apparatus for performing the method on one or more computing devices. A program code configured to implement any one or more of the second to fourth aspects. In this aspect, a computer program comprising such a code is provided. A computer readable medium having a computer program thereon is also provided.

[0041] Any of the various individual features of each of the above-described aspects of the invention, as well as the claims Any of the various individual features of the embodiments described herein, including those described in the claims, can be combined in any suitable and desired manner. [Brief description of the drawings]

[0042] In order that the present invention may be more clearly defined, reference will now be made, by way of example only, to the accompanying drawings, in which: The form will be explained below.

[0043] [Figure 1] FIG. 1 is a schematic diagram of a classification system according to an embodiment of the present invention. [Diagram 2] 2 is a high level schematic diagram illustrating the operation of the classification system of FIG. 1 in calculating a subject's fracture risk from medical image scans. [Diagram 3] FIG. 2 is a schematic diagram illustrating the operation of a segmenter of the classification system of FIG. 1. [Figure 4] FIG. 2 is a schematic diagram illustrating the operation of a classifier of the classification system of FIG. [Diagram 5] 2 is a flow diagram illustrating training of the classification neural network of the classification system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] FIG. 1 illustrates a method for identifying structures or materials in a medical image (structure and material segmentation) according to an embodiment of the present invention. FIG. 1 is a schematic diagram of a classification system 10 for classification based on orientation.

[0045] Referring to FIG. 1, the system 10 includes a classification controller 12 (including a GUI 16). and a user interface 14. The user interface 14 displays information to a user. It is provided to provide a means to communicate with users and to receive input (including feedback) from users. Typically, there is one or more displays (possibly displaying one or more GUIs 16). a web browser, a keyboard and mouse, and optionally a printer. The classification controller 12 includes at least one processor 18 and a memory 20. The system 10 may be implemented, for example, as a combination of software and hardware on a computer. (e.g., as a personal computer or mobile computing device) or The system 10 may be implemented as an image segmentation system. For example, some or all of the components of memory 20 may be distributed among the various components; The user interface may be located remotely from the processor 18; The interface 14 may be located remotely from the memory 20 and / or the processor 18. For example, system 10 may be implemented within a service-oriented architecture. The components can be used in communication networks such as LANs, WANs, and the Internet. The system 10 can be deployed in the cloud. The use of the service can be shared by users in different locations.

[0046] In certain other embodiments, the system 10 (in terms of software and hardware) Implemented as a stand-alone system or executable by a computer It can be implemented as a standalone software and deployed in one location. For example, the system 10 may be deployed in hospitals, medical clinics, and other clinical locations.

[0047] The memory 20 is in data communication with the processor 18 and is typically volatile. and non-volatile memory (and one or more of each memory type). (which may include a RAM (random access memory), a ROM, and one or more This includes mass storage devices.

[0048] As described in more detail below, processor 18 includes a segmentor 22 (which includes a structure segmentor 24a, a material segmentor 24b, and an anomaly segmentor such as an abnormal material segmentor 24c), a segmentation map processor 26, and a non-image data processor 28. Processor 18 also includes a classifier 30, an I / O interface 32, and a results output 34.

[0049] The memory 20 includes program code 36, image data 38, non-image data 40, segmentation models 42 (which in this example include a structural segmentation model 44, a material segmentation model 46, and an anomaly segmentation model as an abnormal material segmentation model 48), and a segmentation map 50 (which in this example include a structural segmentation map 52, a material segmentation map 54, and an anomaly segmentation map as an abnormal material segmentation map 56). The structural segmenter 24a, the material segmenter 24b, and the abnormal material segmenter 24c train their respective segmentation models 44, 46, 48 and perform segmentation on the incoming image using the segmentation models 44, 46, 48 to generate the structural segmentation map 52, the material segmentation map 54, and the abnormal material segmentation map 56, respectively.

[0050] The memory 20 also includes a classification machine learning model as a classification neural network 58. , which is then trained so that the classifier 30 uses this, the segmentation map 50 and the non-image data 40. The classification controller 12 performs classification at least in part (in some implementations In one embodiment, the processor 20 executes program code 36 from memory 20 (completely). Implemented by SA18.

[0051] It should be noted that since this embodiment relates to classification of structures and / or materials in medical images, abnormal materials Segmenter 24c may also be referred to as an abnormal tissue segmenter and may also be referred to as an abnormal material segmenter. The segmentation map 56 may also be referred to as an abnormal tissue segmentation map.

[0052] Generally speaking, the I / O interface 32 receives image data and non-image data relating to the subject. The image data is read or received, and the data is divided into image data 38 and non-image data. The embodiment is configured to store the data 40 in the memory 20 for processing. The image data 38 is typically in the form of a medical image, e.g. The non-image data 40 typically includes a variety of structured and unstructured images. Contains subject or patient information from data sources, including at the time of subject examination, treatment, and Subject structured data is collected over the course of follow-up visits. Basic information on subjects such as gender, age, weight, height, etc.; blood test results, DNA test results, etc. clinical test results, e.g., clinical outcome; treatment data, e.g., drug type and dosage; and smoking history and alcohol intake. Interview data such as drinking history and fracture history. Subject unstructured data is a text document of clinical test results. Non-image data 40 may include, for example, numerical data, medical records, physician notes, radiology reports, etc. This information can come in a variety of forms, including text, audio, and video.

[0053] The segmenter 22 processes image data 38 (which may comprise one or more medical images) to generate structural segments. Segmentation model 44, material segmentation model 46, and abnormal material segmentation model A structural segmentation map 5 is generated from the image data 38 using the spatial segmentation model 48. 2, a material segmentation map 54 and an abnormal material segmentation map 56. Each of these generates a different set of image data 38, each of which characterizes the image data 38 in a different way. The classifier 30 uses the resulting segmentation map 50 and the non-image data 40 and generate the result as a classification output therefrom. In this embodiment, the classification output is , which is then presented to the user or used for further analysis, which is I / O intensive. This is done via the interface 32, which outputs the results 34 and / or the user interface 1 This will be done at 4.

[0054] Classifier 30 (generated in this embodiment with classification neural network 58) The classification output may be for each of one or more classifications (and preferably for each of a plurality of possible classifications). Each score is calculated based on the number of subjects who fit into the corresponding category. In this embodiment of the bone fragility assessment, The classifications are as follows: "negligible fracture risk", "imminent fracture risk", "medium-term "Fracture Risk," and "Long-term Fracture Risk." The classification outputs are described in more detail below.

[0055] In an alternative embodiment, the classifier may include a respective one or more state progression stages The output is a disease progression score for each condition. Each score represents the predicted progression from the current condition to another condition. For example, when assessing bone fragility, the degree of disease progression is expressed as "stable," These may include "mild aging," "accelerated aging," and "slight aging."

[0056] In yet another embodiment, the classifier comprises a respective treatment for each of a plurality of treatment options. Outputs a score. Each score represents the degree to which the treatment is predicted to be most efficient for the patient. For example, in bone fragility assessment, the treatment options are "antiresorptive drugs" , "anabolic agent therapy," and "antiresorptive agent plus anabolic agent therapy."

[0057] In further embodiments, the classification output includes a score for each of one or more possible classifications that correspond to known medical conditions or pathologies. For example, in bone fragility assessment, these classifications may include: "osteomalacia", "tumor", "osteonecrosis", and "infection". In such embodiments, the resulting score represents the degree to which the subject's sample (e.g., bone) matches the classification / condition. If only one classification has a significant score, or if one classification has a score significantly greater than all others, then that classification may be considered as a diagnosis or suggested diagnosis for the corresponding condition or pathology.

[0058] In certain embodiments, the classification output may be such (selected from the examples above or other examples) Contains two or more sets of scores.

[0059] Returning to FIG. 1, and as will be appreciated by those skilled in the art, one or more medical imaging images may be included. Image data 38 may be acquired using various medical imaging modalities (e.g., HRpQCT or High-Resolution Quantitative Computed Tomography). n peripheral Quantitative Computed Tomography (high-resolution quantitative CT for peripheral bones, etc.) This includes data generated by one or more medical imaging devices ( Each of these devices (whether in vivo or not) is implemented by a HRpQCT scanner. Scanning a sample (whether in vitro or not) to obtain a visual image of the interior of the specimen, typically A medical image may be, for example, a representation of a part of the human body or of the entire subject (e.g., the brain, lower back, or A medical image may represent the same sample or body part using different imaging modalities. Different imaging modalities can be acquired by scanning with the same sample. The same imaging modality for different body parts may reveal different characteristics of the same pattern or body part. Medical images can be acquired using different imaging techniques, with different body parts of the same patient being imaged differently. This could provide new insights into diseases and conditions, such as bone fragility. For the assessment of sexuality, both the wrist and leg of the patient were scanned with an HRpQCT scanner ( Scanning different samples and body parts using different imaging modalities ), which can provide information for assessing the subject's bone quality.

[0060] The image data 38 may be a 2D (two-dimensional) image or a may include a 3D (three-dimensional) image that may be represented as a three-dimensional array of voxels. For convenience, the medical images described below are 3D images that can be represented as a 3D array of voxels. be.

[0061] As described above, one or more received medical images stored in the image data 38 are segmented by the segmenter 22 into respective segmentation maps 50 using the trained segmentation model 42. Each segmentation map 52, 54, 56 characterizes each medical image differently. The structure segmentation map 52 represents the medical image as one or more different anatomical structures from a predefined structure set. For example, a wrist CT scan can be segmented into dense cortex, transition zone, and trabecular bone region. The material segmentation map represents the medical image with multiple different materials from a predefined material set. For example, a wrist CT scan can be segmented into mineralized material, fully mineralized material, red bone marrow in the trabecular bone region, and yellow bone marrow in the trabecular bone region. The abnormal material segmentation map 56 represents the medical image with normal and abnormal materials (in this example, normal and abnormal tissue). For example, from a wrist CT scan, a tumor or fracture can be segmented and represented as “abnormal” in the abnormal material segmentation map 56 .

[0062] The segmentation map 50 is combined with the non-image data 40 and input to the classifier 30. The classifier 30 performs a first classification based on the segmentation map 50 and the non-image data 40. The input data to the classifier 30 is generally multidimensional, so The classifier 30 may be, for example, a neural network, a support vector machine, a decision tree, or the like. It is implemented using machine learning algorithms such as combinations of these.

[0063] In this embodiment, the classifier 30 uses a classification neural network 58 or The neural network is implemented as a convolutional neural network58 in this example. The network layer and the fully connected neural network layer are included (in other embodiments, other (A machine learning algorithm such as the above is also possible.) Classification neural networks58 are Then, it is trained on the training data.

[0064] As mentioned above, the final classification output is provided by the system 10 as a result output 34 or a user output. The classification output is output to a user via an interface 14. The classification output may optionally include one or more For example, the result may include a visual representation of the corresponding segmentation map 50. Segmentation is useful when it helps users interpret the classification output, such as when assessing the reliability of a segment A location map 50 can be presented.

[0065] Figure 2 shows a medical imaging scan, in this example a wrist HRpQCT scan62 (its negative image) 62' is also shown) for calculating the subject's risk of fracture. 2 is a high level schematic diagram 50 illustrating the operation of the system 10. Receive a wrist HRpQCT scan 62 including slices of As shown, an HRpQCT scan may have 100 or more slices, but for simplicity, (Please note that only 4 slices are shown for convenience.)

[0066] The segmenter 22 segments the scan 62 into a structural segmentation map 52. The scan was segmented into the dense cortex, the outer transition zone, the inner transition zone, and the trabecular bone region. The segmenter 22 segments the scan 62 into a material segmentation map 54. Scan 62 is for the surrounding muscle, surrounding fat, yellow bone marrow hyperlipid areas, The red bone marrow is segmented into hyperlipidated areas. Segmentation maps 52, 54 and The abnormal material segmentation map 56 and non-image data 40 (e.g., clinical data including gender and age) The data 64 including the phenotypic and phenotypic factors is processed by the trained classifier 30 to produce a classification output. The classification output includes segmentation maps 52, 54 and tables or reports 66. The table or report 66 presents a numerical estimate of the probability of fracture in each of the following categories of fracture risk: and / or include in graphic form: Imminent risk of fracture 68a (fracture within 2 years: t<2y ), medium-term fracture risk 68b (fractures within 2 to 5 years: 2≦t≦5y), and long-term Fracture risk 68c (fractures within 5 to 10 years: 5≦t≦10y) and negligible bone Risk of fracture 68d. In the example shown, the following probabilities are seen: Risk of fracture of wrist within 2 years The risk of wrist fracture in 2 to 5 years was 2.4%, and the risk of wrist fracture in 5 to 10 years was 1.0%. 1.6% risk and 0.3% negligible risk of wrist fracture. (When combining cases where the risk of neck fracture is negligible and cases where there is a long-term risk of fracture) The probability that a subject would not experience a wrist fracture over the next 5 years is only 4.4%. Although 6 does not include a diagnosis (that the subject has osteomalacia), these probabilities are very valuable. I want people to understand that the value of their treatment is high, and for example, by receiving medical examinations and consultations, This may include prompting the patient to seek a diagnosis.

[0067] FIG. 3 is a schematic diagram 70 of the operation of segmenter 22. Receive an input including one or more medical images (from image data 38), process the images, and segment The medical images are generated using the same imaging modality. This can be obtained by scanning different body parts of the patient using a In some applications to assess bone quality, the patient's hand is scanned in an HRpQCT scanner for evaluation. Both the neck and legs can be scanned. Medical images are taken of the patient using different imaging modalities. These can be obtained by scanning the same or different body parts. In some applications, patients undergo wrist HRpQCT scans and lumbar DXA scans for evaluation. (Recall again that other imaging modalities may be used to acquire different samples.) Medical images can be obtained by pulling a lens or scanning a body part.

[0068] Referring to FIG. 3, the segmenter 22 processes medical images 1 to n (38 1 ,...,3 8 n One or more processing branches 1 to n (labeled 72 in the figure) corresponding to 1 ,...,72 n When there are multiple processing branches, , Medical Imaging 38 1 ,...,38 n For example, different imaging modalities (74 1 ,..., 74 n This may be caused by a For example, different body parts, different scans of a single body part, or two or more of these Combinations of the above are also possible. 1 ,...,72 n is an image and then, according to the image type (e.g., with different program codes 36), The input image is segmented to generate a branch output. (configures application map 50).

[0069] To process each input medical image, the segmenter 22 first determines the type of input image. The segmenter 22 is configured to select a processing branch from processing branches 1 to n according to determines the image type according to: the sample (e.g. the scanned body part) ) and imaging modality, information that can be determined from each image, such as the header file of the medical image. The information can be derived from metadata stored in the file and / or from the file type. For example, Information about the scanned body part and imaging modality is accessible from the metadata It is.

[0070] Each input medical image 1-n is processed by three segmentation programs (i.e., structural segmentation, 24a, material segmenter 24b, and abnormal material segmenter 24c). The images are then processed and transformed into the corresponding segmentation maps 52, 54, and 56. The segmenter 22 includes segmenters 24a, 24b, and 24c (respectively, 24a 1 ,...,twenty four a n , 24b 1 ,...,24b n , and 24c 1 ,...,24c n Labeled as Since a maximum of n instances of each However, for each segmenter 24a, 24b, 24c (which in each case ranges from 0 to n), The number of instances for each can vary.

[0071] The structure segmenter 24a, the material segmenter 24b, and the abnormal material segmenter 24c each Processing branch 72 1 ,...,72 nEach segmentation map can be generated in For example, in FIG. 3, structure segmenter 24a, material segmenter 24b, and anomaly material segmenter 24c are The mentor 24c includes segmentation maps corresponding to the medical imaging modalities 1 to n. The resulting structure segmentation map, material segmentation map, The segmentation map and the abnormal tissue segmentation map are correspondingly transformed into structural segmentation maps. Station Map 52 1 ,...,52 n , Material Segmentation Map 54 1 ,.. .,54 n , and abnormal tissue segmentation map 56 1 ,...,56 n Labeled with However, for some applications, it is necessary to generate all three segmentation maps. Note that it may not be possible or desirable to do so. This can be, for example, a limitation of the image, the imaging modality, or the segmenters 24a, 24b, 24c. This may be due to limitations in the segmenter training data, for example.

[0072] The structure segmenter 24a, the material segmenter 24b, and the abnormal material segmenter 24c Each voxel of these segmentation maps 50 is assigned a predefined type (or category). One or more "types" (or "categories") are assigned to a category from a set of categories. In this embodiment, the structure segmenter 24a uses a predefined set of structure types Assign each structure type to each voxel in the scan. For example, wrist HRp The QCT scan is segmented into a structural segmentation map 52. Each voxel in the can is assigned a structure type (or category) from the set are assigned: "periphery," "dense cortex," "transition zone," and "trabecular region."

[0073] The material segmenter 24b segments each material type (from a set of predefined material types). For example, in this embodiment, material segmenter 24b assigns a hand Segment the neck HRpQCT scan into a material segmentation map54 Each voxel in the scan is assigned a material type from the set consisting of: The following are considered: "calcified material", "fully calcified material", "red marrow hyperlipidated material", and "yellow marrow hyperlipidated material". "Bone marrow hyperlipid state substance."

[0074] The abnormal material segmenter 24c assigns each voxel an abnormal / normal type ("normal" and (from a set of predefined abnormal / normal types, such as a set containing "abnormal" and "abnormal"). For example, in this embodiment, the abnormal material segmenter 24c assigns the wrist HRpQCT Segment the scan into an abnormal tissue segmentation map 54. Each voxel in is assigned a "normal" or "abnormal" status. In an embodiment, the anomaly material segmenter 24c can distinguish between different types of anomalies and can be configured to use predefined The set of abnormal / normal types determined (in addition to "normal") is It may contain one or more specific abnormalities that are specific to the sample. If the sample is bone, these may include For example, a "fracture crack" or a "bone tumor" may be included. In such an embodiment, the abnormal substance In cases where segmenter 24c is unable to determine the specific anomaly type, the "anomaly" is set. can be optionally included in

[0075] In some embodiments, a structure segmenter 24a, a material segmenter 24b, and an anomaly material segmenter 24c are The grammenter 24c assigns each type of probability along with a confidence limit or probability to each voxel in the medical image. In some other embodiments, the structure segmenter 24a, the material segmenter 24 b, and abnormal material segmenter 24c for each voxel in the medical image (optionally each Multiple types (with associated confidence limits or probabilities) can be assigned. For example, Segmenter 24a uses ambiguous vowels to segment wrist HRpQCT scans. A cell can be assigned both the "transition zone" and "trabecular region" types. This is done with respective confidence limits or probabilities (which are typically different).

[0076] The segmenter 22 uses a trained segmentation model 42 (which includes a structural segmentation segmentation model, material segmentation model, and abnormal material segmentation model A segmentation map 50 is generated using the segmentation Models (e.g., neural networks, support vector machines (SVMs), decision (a) trained using machine learning algorithms; In these embodiments, the segmentation model is a deep learning neural network with multiple layers. The neural networks used in the study include convolutional neural networks. The neural network layer, the fully connected neural network layer, the normalization layer, and the multiplication layer.

[0077] Segmenter 22 may additionally (or alternatively) use non-machine learning based techniques to perform segmentation. For example, methods based on the location of edges, corners, and transition slopes can be used. Alternatively, there may be a method based on a wide range of features such as the histogram or brightness of an image. For example, U.S. Patent Publication No. 2012 / 0232375 ("Image Analysis of Selected Tissue Structures") The present invention relates to a method and system for detecting the transition between dense cortical and trabecular bone regions in a wrist CT scan. A method is disclosed for segmenting zones based on local and global features of bone. and in many applications it will be appropriate for segmenter 22 to use this technique.

[0078] FIG. 4 is a schematic diagram of the operation 80 of the classifier 30. The classifier 30 22) and the non-image subject data 40. and configured to process the input to generate one or more classification results. There are.

[0079] However, both the segmentation map 50 and the non-image subject data 40 are classified. The segmentation map processor 26 and the non-image data processor 30 are then respectively For example, in some embodiments, the classification may be performed by a classifier 30. Downsample the segmentation map 50 to enable faster image processing. It may be advantageous to convert the image into a low resolution map, and such processing may be desirable. If necessary, this is done by the segmentation map processor 26. In some embodiments, the segmentation map processor 26 More than one type was assigned (with different confidence limits or probabilities) for a particular segment. This sets the type of any voxel (in the annotation map), e.g. It does this by assigning to the voxel the type that has the highest or highest probability.

[0080] The non-image data may include structured data and unstructured data. 28 is configured to mobilize various techniques to process any structured data. This is done by extracting features from the data, and in each case This is done according to the structure and format of the data. For example, structured data is typically , database tables, .json files, .xml files, .csv files, etc. The non-image data processor 28 is required Structured data by querying key parameters and attributes from each source of data Extract features from.

[0081] The non-image data processor 28 processes the unstructured data in two steps: first, Converting it into structured data and then extracting features from the converted data. The conversion method used by non-image data processor 28 is unique for each source. For example, to convert physician notes into structured data, the non-image data processor 28 may: Using trained Optical Character Recognition (OCR), your notes can be read by your computer. and can be converted to recognizable text by a non-image data processor. 28 applies keywords such as “fracture”, “pain” and “fall” to the converted text. Once the unstructured data is converted to structured data, A non-image data processor 28 then extracts features from the now structured data. .

[0082] The processed non-image data 40 and the segmentation map 22 are passed to a classifier 30. The classifier uses these inputs to generate a classification score (e.g., disease status score 82, disease progression score 83). A classification output having a row score 84 and / or a treatment score 86 is generated.

[0083] FIG. 5 illustrates a classification system for a classifier 30 with a deep neural network according to an embodiment of the present invention. 9 shows a flow diagram 90 for training the class neural network 58. As shown in the figure, As such, in S92, data, including medical image data and non-image data, are used for training and testing. In S94, the collected images are segmented by the segmenter 22. The resulting images are then processed to generate a segmentation map (see above).

[0084] In S96, the image data 38 and the non-image data 40 are annotated with labels. The training classifiers are then trained by qualified experts with domain knowledge. The output may be determined based on the subject's clinical record. For example, the classification output may be a fracture probability score. If included, the training output is the timing (after the scan) of the fracture, which is related to the subject's pathology. If no fracture is evident, the risk is considered to be "negligible" (or none at all). The classification output is recorded as a category that corresponds to a known medical condition. If the scores are to be included, the training output is the actual disease state of the subject, which is also based on the clinical record of the subject. It is required from the records.

[0085] In S98, the data (including the segmentation map 50 and the non-image data 40) is A training set, a development set (abbreviated as Dev set, which may be omitted in some embodiments), and a test set. The training set is the set used for learning, and the experimental set is the set used for testing. The data on which the algorithm is run; the Dev set is used for parameter tuning. The test set is the data that should be used to evaluate the trained model. do.

[0086] The S100 implements layers of a deep learning neural network. Each layer is an artificial neural network. An artificial neuron is a neural network that receives one or more inputs and outputs a sum of them. A mathematical function that produces a force, usually weighted separately for each input. Sam is passed through a nonlinear function. As the neural network learns, The weights of the model are adjusted, and the adjustment is done to reduce the error (network output) This is done according to the difference between the force and the annotation, and the error cannot be reduced any further. Adjustments will be made until this is achieved.

[0087] In S102, the number of layers, the number of neurons, the neuron weights, and the performance of the neuron functions are determined. S10 Initialize the parameters of the classification neural network 58, including the parameters, etc. In step 4, a learning algorithm is run on the training dataset to generate a classification neural network. The parameters of the work 58 are updated, for example by determining the slope of the loss function The loss function is calculated based on the labeled classification output and the classification neural network. The development data is calculated based on the output generated by the network 58. can be used to explore and tune layers and parameters.

[0088] In S106, the classifier 30 provided with the classification neural network 58 performs the test In S108, the performance of the classifier 30 is evaluated to see if it is sufficient. If performance is insufficient, the process returns to step 92 and more training data is added. The data is collected.

[0089] If the performance is deemed sufficient in S108, the process continues to S110, where the trained classifier The device 30 is output, flagged for deployment, or released for use. And the process ends.

[0090] Those skilled in the art will recognize that many modifications can be made without departing from the scope of the invention. In particular, certain features of the embodiments of the present invention may be used to provide further embodiments. Note that it is possible.

[0091] If any references to prior art are made in this specification, such references are not to be construed as limiting the scope of the invention. It should be noted that a prior art application does not constitute an admission that the prior art forms part of the public knowledge in any country. I want to.

[0092] In the appended claims and the foregoing detailed description of the invention, Therefore, unless the context otherwise requires, the terms "have" and "have" (third person singular Terms such as "has" and "has" are used in a comprehensive sense (i.e., the presence of the declared feature). Although the term "presence" is used to indicate the presence of additional features in various embodiments of the invention, This does not preclude the inclusion or addition of other elements. [Explanation of symbols]

[0093] 12 Classification Controller 14 User Interface 16 GUI 18 Processors 20 Memory 30 classifier

Claims

1. For training a classification machine learning model to classify structures or materials in an image of a subject.

1. A computer-implemented method, comprising: The annotated segmentation map and the annotated non-image data are , a training set and a test set, A mentation map is obtained by segmenting one or more images. There are steps, (a) implementing a classification machine learning model, the step comprising: a step including initializing the parameters of the learning model; (b) detecting the classification machine by executing a learning algorithm on the training data; updating parameters of the learning model; (c) testing the classification machine learning model on the test data; and (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) through (d) if performance is insufficient in step (d); and Output the classification machine learning model as a trained classification machine learning model for deployment or flagging the classification machine learning model as a trained classification machine learning model. and

2. 10. The method of claim 1, wherein the classification machine learning model is further adapted to perform the classification process if the performance is deemed insufficient. collecting more image and non-image data to train the

3. 3. The method of claim 1 or 2, wherein the classification machine learning model comprises artificial neurons. The present invention includes a neural network having multiple layers that can The parameters include the number of layers, the number of neurons, the neuron weights, and the neuron functions. The parameters include Testing the classification machine learning model includes testing the classification machine learning model. The method includes testing with data.

4. The method according to any one of claims 1 to 3, wherein the annotated segmentation is The annotation map and the annotated non-image data are used as a training set and a development set. and test sets, and to test the learning procedure and tune the parameters. using the development data to

5. A classification machine learning model trained according to any one of claims 1 to 4.

6. For training a classification machine learning model to classify structures or materials in an image of a subject.

1. A system comprising: a processor, the processor comprising: The annotated segmentation map and the annotated non-image data are , a training set and a test set, A mentation map is obtained by segmenting one or more images. There are steps, (a) implementing a classification machine learning model, the step comprising: a step including initializing the parameters of the learning model; (b) detecting the classification machine by executing a learning algorithm on the training data; updating parameters of the learning model; (c) testing the classification machine learning model on the test data; and (d) evaluating whether the classification machine learning model has sufficient performance; and repeating steps (a) through (d) if performance is insufficient in step (d); and Output the classification machine learning model as a trained classification machine learning model for deployment or flagging the classification machine learning model as a trained classification machine learning model. The system is configured to perform the steps of:

7. 1. A system for classifying structures or materials in an image of a subject, the system comprising: a segmenter for performing one or more segmentations of structures or materials in an image; and generating one or more segmentation results for the image from the segmentation. a map of the categories assigned from a set of one or more predefined categories; Segmentation map including categorization of pixels or voxels A segmenter that generates a segmentation map; A trained classification machine learning model, based on the segmentation map. The above classifications are generated, and the likelihood that the structure or substance or the subject falls into each of the classifications is calculated. a trained classification machine learning configured to assign each score indicative of a gender to the classification; A classifier that implements the model; and an output unit for outputting a result indicative of the classification and score.

8. 8. The system of claim 7, wherein the classifier is adapted to generate the segmentation for the subject. generating the one or more classifications based on a station map and non-image data; 。

9. 9. The system of claim 7 or 8, wherein the segmenter: i) assigning said pixels or voxels from a set of predefined structural categories; Configured to generate structural segmentation maps that include categorization of A structure segmenter ii) the pixel or voxel is assigned from a predefined set of material categories; The method is configured to generate a material segmentation map including a categorization of a material segmenter formed therein, and / or iii) the pixel or vector assigned from a predefined abnormal / normal category set; Generate an anomaly segmentation map that includes categorization for oxels The system comprises an anomaly segmenter configured to:

10. 10. The system of claim 9, wherein the structural segmenter is a structural segmentation machine. configured to generate the structural segmentation map using a machine learning model; The material segmenter uses a material segmentation machine learning model to generate the material segment. and generating an anomaly map, the anomaly segmenter configured to generate the anomaly segmentation map using a classification model. The system in question.

11. 11. The system according to claim 7, wherein the segmentation Downsample the segmentation map before the map is input to the classifier. and a segmentation map processor configured to process the segmentation map in a segmentation map pull or other manner. A system to prepare for such situations.

12. 12. The system of claim 7, wherein the classification machine learning model teeth, (a) Neural networks, support vector machines, and / or decision trees or (b) A convolutional neural network layer and a fully connected neural network layer. A system including a neural network.

13. 13. A system according to any one of claims 7 to 12, wherein the image is a medical image. The classification is as follows: (i) whether the structure or substance, or the subject, experiences a particular condition or symptom during each period of time; The probability of falling into (ii) whether the structure or substance or the subject has a particular condition or disease during the respective time periods; The probability of falling into a state in which the period is short-term, long-term, and the short-term and at least one medium-term period between said long-term period; (iii) causing the structure or substance or the subject to suffer from a respective condition or symptom. probability, (iv) the probability of each rate of progression of the disease or condition; (v) a probability of each rate of progression of a disease or condition, said classification being Probability, including one or more of: fixed, mild, and accelerated deterioration; (vi) the probability of efficacy of each treatment option; (vi) Probability of efficacy of each treatment option, said treatment options being relapsed. Probability, including antiresorptive and / or anabolic therapy; (viii) the respective medical condition; and / or (ix) each medical condition including one or more of osteomalacia, tumor, osteonecrosis, and infection; Target status Corresponding system.

14. 14. The system of claim 7, wherein the trained classification machine The learned model is a model trained by the method according to any one of claims 1 to 4. , system.

15. 1. A computer-implemented method for classifying structures or materials in an image of a subject, the method comprising: The law is forming one or more segmentations of structures or materials in the image; From the segmentation, one or more segmentation maps for the image are generated. The segmentation is performed by dividing the segment into groups of segments each of which is assigned from a set of predefined structure / material categories. Segmentation, including categorization of pixels or voxels in the segmentation map generating a map; Using a trained classification machine learning model, a The above classifications are generated, and the likelihood that the structure or substance or the subject falls into each of the classifications is calculated. assigning respective scores indicative of the suitability of the classification to the classification; and outputting a result indicative of the classification and score.

16. 16. The method of claim 15, wherein the trained classification machine learning model (i) based on the segmentation map and non-image data about the subject, and / or (ii) generating a classification for image data and non-image data associated with said subject. Based on the training subjects, each score trained with image data and non-image data A method for generating.

17. 17. The method of claim 15 or 16, wherein forming the one or more segmentations The steps of i) assigning said pixels or voxels from a set of predefined structural categories; generating a structural segmentation map including a categorization of the ii) the pixel or voxel is assigned from a predefined set of material categories; generating a material segmentation map including a categorization for and / or iii) the pixel or vector assigned from a predefined abnormal / normal category set; Generate an anomaly segmentation map that includes categorization for oxels A method comprising the steps of:

18. 18. The method of claim 17, further comprising: generating the structure segmentation map; generating said material segmentation map using a training model; and generating the anomaly segmentation map using an annotation model. Includes, a method.

19. 19. The method of any one of claims 15 to 18, wherein the trained classification machine learning The model is a model trained by a method according to any one of claims 1 to 4. ,method.

20. A method for determining whether a subject is a subject, the method comprising: A classification for the structure or material in the image.

21. When executed by one or more computing devices, the method of claims 1 to 4 and 15 20. A computer program adapted to carry out the method according to any one of claims 1 to 19. A computer readable medium comprising code.

Images