Method and system for image segmentation and identification

Abstract

To provide a method and system for managing training and retraining processes of a machine learning model for segmentation, efficiently.SOLUTION: In a segmentation and identification system 10, a segmentation and identification controller 32 includes a model evaluator. The model evaluator executes, for evaluation of a segmentation machine learning model, the steps of: using a trained segmentation machine learning model to generate a segmentation of an annotated evaluation image; forming a comparison of the generated segmentation of the evaluation image and an existing segmentation annotation; and deploying or releasing the trained segmentation machine learning model for use if the comparison indicates that the segmentation machine learning model is satisfactory.SELECTED DRAWING: Figure 1B

Description

[Technical Field]

[0001] The present invention relates to a method and system for image segmentation and identification. and especially medical images (e.g., bones and other anatomical structures, masses, tissues, landmarks, lesions, etc.). Segmentation and identification of pathological matters, etc., and computer Medical imaging modalities such as computed tomography (CT), magnetic resonance imaging (MR), ultrasound, and lesion scanner imaging The present invention relates to, but is not limited to, segmentation and Annotation of medical image data for training machine learning models for classification applications ) and methods and systems for evaluating and improving machine learning models This also applies to the

[0002] Related Applications This application is based on U.S. Patent Application No. 16 / 448,252 (filing date: June 21, 2019). No. 6,299,333, filed on Dec. 1, 2002, and claims priority thereto, the entire contents of which are incorporated by reference at the time of filing. It is inserted. [Background technology]

[0003] Accurate segmentation and identification of medical images is essential for quantitative analysis and disease diagnosis. Segmentation is the process of identifying objects (e.g., anatomical structures) in a medical image. Identification is the process of distinguishing an object (or tissue) from its background. Traditionally, segmentation and identification are done manually or The manual approach involves only outlining and extracting the target object. Experts with sufficient domain knowledge to label the objects requires.

[0004] There are also computer-assisted systems that provide semi-manual segmentation and identification. For example, such systems may use signal strength, edges, 2D / 3D curvature, shape or other Approximate object of interest based on selected parameters including 2D / 3D geometric features Contours can be detected, and then an expert can manually refine the segmentation or classification. Alternatively, an expert can provide such a system with information such as the approximate location of the target object. input data, and the computer-aided system performs segmentation and Identify.

[0005] Both manual and semi-manual methods are labor-intensive and time-consuming. The quality of the results depends heavily on the expertise of the expert. This makes a considerable difference in the resulting segmented and identified objects. It can be done.

[0006] In the past few years, machine learning, especially deep learning (e.g., deep neural networks), has become increasingly popular. Convolutional neural networks (DNNs and deep convolutional neural networks) are used in many visual recognition applications, including medical images. They have come to surpass humans in terms of work.

[0007] Patent document 1 discloses a method and system for medical image segmentation based on artificial intelligence. The method includes receiving a medical image of a patient, and determining a current state based on the medical image. Steps to automatically determine the segmentation context of the current segmentation Select at least one of several segmentation algorithms based on the context of the and automatically selecting one of the segmentation algorithms. At least one segmentation algorithm is used to identify target regions within a medical image. The target anatomical structure is segmented.

[0008] Patent document 2 discloses a method for segmenting an image of a target patient, The method includes the steps of: targeting 2D slices for a 3D anatomical region image; and providing nearest neighbor 2D slices, and Neural network (multi-slice FCN, multi-slice fully convolutional neural calculating a segmentation region by a neural network, A predefined intrabody solution that extends spatially across the target 2D slice and the nearest 2D slices The 2D slice and and each nearest 2D slice is the corresponding reduction of the iteratively reduced component of the multi-slice FCN. The processing is performed by small components, which are the target 2D slice and the nearest 2D slice. The order of the slices is based on the sequence of 2D slices extracted from the 3D anatomical image. The output of the iterative reduction component is a segmentation matrix for the target 2D slice. The computations are combined and processed by a single augmentation component that outputs a mask. Steps to do this.

[0009] Patent Document 3 describes a method for applying deep convolutional neural networks to medical images to realize real-time image recognition. and methods for generating real-time or near-real-time diagnoses or diagnostic recommendations - Patents.com The method includes the steps of: performing image segmentation on a plurality of medical images. wherein the image segmentation step comprises isolating a region of interest from each image. and a cascaded deep convolutional neural network detection structure. (cascaded deep convolutional neural network detection structure) has been segmented a step of applying the detection structure to the image, the detection structure being: i) a first stage, a first convolution; Within each 2D slice of a segmented medical image, a convolutional neural network is used to Screening is performed for all possible positions using a sliding window method. a first stage for identifying the candidate positions; and ii) a second stage for using a second convolutional neural network. Screening of 3D objects constructed from candidate positions using a neural network The screening is performed with random scale and random viewing angle. This is done by selecting at least one random position within the solid, and a second step of identifying one or more refined locations and classifying the refined locations. and automatically generating a report including a diagnosis or a recommended diagnosis. .

[0010] However, applying such techniques to medical image segmentation and classification is There is a problem with this: to train and validate machine learning algorithms, we need a ground truth data. Ground truth data is required, but this data is not annotated. This is brought about by human experts who perform the Improving a machine learning model ideally involves reducing false positives (e.g., the number of false positives) Dell has failed to segment or identify medical images, etc. Managing the training and retraining process effectively is challenging; For this method, it is difficult or impossible to obtain a large number of training images, and the training data is Limited data makes it difficult to efficiently train segmentation and discrimination models. . [Prior art documents] [Patent documents]

[0011] [Patent Document 1] International Publication No. 2018 / 015414 [Patent Document 2] U.S. Patent Application Publication No. 2018 / 0240235 [Patent Document 3] U.S. Patent No. 9,589,974 Summary of the Invention

[0012] It is an object of the present invention to provide a segmentation system that can integrate annotations. is included in the target.

[0013] According to a first aspect, the present invention provides an image segmentation system comprising: Each segmentation annotation is associated with an image (e.g., a medical image). Annotated training data with images is used to perform segmentation. Train a segmentation machine learning model to generate a trained segmentation machine learning model a training subsystem configured as follows: a model evaluator; and Uses a trained segmentation machine learning model to identify areas in an image (e.g., bone, muscle, fat) configured to segment structures or materials (including fat or other biological tissues) and a segmentation subsystem formed therein, The model evaluator evaluates the segmentation machine learning model. (i) Controlling the segmentation subsystem to implement the segmentation machine learning model At least one segmentation annotation associated with an existing segmentation annotation using Segment the evaluation image, thereby obtaining segmentation results for the annotated evaluation image. generating a mentation; (ii) Segmentation of the annotated evaluation image and the existing segmentation forming a comparison between the annotation and the If the comparison indicates that the segmentation machine learning model is successful, the trained segmentation model is deploying or releasing the segmentation machine learning model for use. It is configured to do this by:

[0014] Therefore, the model evaluator determines the operational segment used to make the segmentation. We evaluated a segmentation machine learning model using the annotation subsystem. and provides an integrated training and segmentation system.

[0015] In an embodiment, the system performs segmentation of the annotated evaluation image and If existing annotations match within a given threshold, the model is deployed or re-used for use. The device is configured to be leased.

[0016] In an embodiment, the system performs segmentation of the annotated evaluation image and If existing annotations do not match within a given threshold, the model will continue to train. For example, the system may modify the model algorithm and / or continues to train the model by adding additional annotated training data. This allows tuning of the threshold value to suit the desired application. Also, refinements can be made if the initial tuning is not satisfactory.

[0017] In an embodiment, the training subsystem comprises: (i) Image annotation (part of which is a segmentation machine learning model) (ii) a score associated with the annotation, the score being used to segment the image; It indicates the degree of success or failure of the segmentation machine learning model. A higher score indicates failure and a lower weighting indicates success. receiving a score indicative of success; Retrain or refine segmentation machine learning models using images and annotations a step of synthesizing the image annotations according to the score, including weighting the annotations of the images according to the score; It is configured to have steps.

[0018] Therefore, segmentation machine learning models must be continually retrained before and during deployment. Annotations for images can be trained or refined. Note that annotations can contain items, and optionally annotations can be partial. Specifically, the segmentation subsystem uses a segmentation machine learning model. Note that the .sigma..times ...

[0019] In an embodiment, the training subsystem refines an existing segmentation machine learning model. A segmentation machine learning model is generated by amplifying or modifying the segmentation data.

[0020] In an embodiment, the system includes an annotation subsystem, Annotated training images (i.e., each image with a segmentation annotation) This includes the step of: generating one or more candidate image annotations for the selected image; receiving an input identifying one or more portions of the candidate image annotation (wherein the portion A candidate image annotation may consist of the entire image (or a part of the image) and at least one or more parts of the image. and generating annotated training images. Specifically, the annotation subsystem can be implemented using either a) non-machine learning image processing methods, or b) a segmentation machine learning model, and is configured to generate

[0021] Therefore, annotations (which may have been generated by non-machine learning image processing methods) The annotation subsystem can be used to justify the , the valid part of the annotation can be kept and used.

[0022] In an embodiment, the system further comprises an identification subsystem, The data is further equipped with discriminative annotations, and the segmentation machine learning model It is a classification and discriminative machine learning model.

[0023] Combining the annotation and segmentation subsystems This will drive continuous improvement of segmentation systems and medical image analysis. This would be beneficial for applying artificial intelligence to

[0024] In an embodiment, the system further comprises an identification subsystem, The training data further provides discriminative annotations, and the training subsystem (i) After each image is segmented by the segmentation machine learning model, (ii) the annotated training data and (iii) the discriminative annotations are used to create a discriminative machine. It is further configured to train the learning model.

[0025] The training subsystem may include a model trainer, which utilizes machine learning to Train a segmentation machine learning model to determine the classification of each pixel / voxel in an image. The model trainer is configured to train the model, for example, a support vector machine or a random number generator. Rest trees and deep neural networks can be employed.

[0026] The structure or material may include bone, muscle, fat, or other biological tissue. The act of fusion is to separate bone from non-bone material (such as surrounding muscle and fat). Or it can be the separation of one bone from another.

[0027] According to a second aspect, the present invention provides a computer-implemented image segmentation method. The method comprises: Each image (e.g., a medical image) and the annotations are Train a segmentation machine learning model using the pre-trained training data to generating a segmentation machine learning model; evaluating the segmentation machine learning model, (i) Using a segmentation machine learning model to extract existing segmentation annotations and segmenting at least one evaluation image associated with the action, thereby generating a segmentation for the annotated evaluation image; (ii) Segmentation of the annotated evaluation image and comparison with existing annotations and an evaluating step, which does this by forming a comparison on If the comparison indicates that the segmentation machine learning model is successful, the trained segmentation model is and deploying or releasing the segmentation machine learning model for use. .

[0028] In an embodiment, the method comprises segmenting an annotated evaluation image and If the annotations match within a given threshold, the model is deployed or released for use. This involves using

[0029] In an embodiment, the method comprises segmenting an annotated evaluation image and If the annotations do not match within a given threshold, continuing to train the model involves For example, the method may involve modifying the model algorithm and / or adding additional annotations. This involves continuing to train the model by adding additional training data.

[0030] In an embodiment, the training comprises: (i) Annotations for use in retraining or refining segmentation machine learning models (ii) a score associated with the annotated image, The score is the segmentation score for segmenting the annotated image. A higher score indicates the success or failure of a machine learning model. The score is then calculated, with a weight indicating failure and a lower weight indicating success. Tep and Annotate according to scores when retraining or refining segmentation machine learning models and weighting the oriented image.

[0031] In an embodiment, the method refines or modifies an existing segmentation machine learning model. The method includes generating a segmentation machine learning model by:

[0032] In an embodiment, the method comprises: The method includes the step of obtaining at least one annotated training image from the image. This means that there is one for each unannotated or partially annotated image. The step of generating the above candidate image annotations and the step of generating one or more candidate image annotations receiving an input identifying one or more portions of the image; and b) generating annotated training images. using a) non-machine learning image processing methods, or b) segmentation machine learning models. generating at least one of the candidate image annotations using the image annotations.

[0033] In embodiments, the annotated training data further comprises identification data; The method also includes the step of converting a segmentation machine learning model into a segmentation and discrimination machine learning model. This includes training the model.

[0034] In an embodiment, the annotated training data further comprises discriminative annotations. The method also includes (i) determining whether each image is segmented by a segmentation machine learning model. (ii) the annotated training data after segmentation and (iii) the discriminative annotations. and training a discriminative machine learning model using the

[0035] The methods are (e.g., support vector machines, random forest trees, deep neural networks, etc.) We use machine learning (including neural networks) to train segmentation machine learning models. The method may include a step of refining the classification of each pixel / voxel of the image using the classification algorithm.

[0036] According to a third aspect, the present invention provides an image annotation system, comprising: The system: a) At least one unannotated or partially annotated an input for receiving an image (such as a medical image) being processed; b) At least one unannotated or partially annotated Annotated training images are generated from the images being processed. To do this, For unannotated or partially annotated images generating one or more candidate image annotations for the image; One or more parts of one or more candidate image annotations (wherein a part is a candidate image annotation) receiving an input identifying a plurality of the ... and providing annotated training images from at least one or more portions. , and steps.

[0037] In an embodiment, the annotator may use a) a non-machine learning image processing method, or b) a segmentation method. A machine learning model (e.g., of the type described above) is used to generate candidate image annotations. The method is configured to generate at least one of the following options:

[0038] According to a fourth aspect, the present invention provides an image annotation method, the method comprising: teeth: a) At least one unannotated or partially annotated receiving or accessing an image on which the image has been applied; b) At least one unannotated or partially annotated providing respective annotated training images from the images being processed, To do this, For unannotated images, one or more candidate image annotations are generated. and Receiving input identifying one or more portions of one or more candidate image annotations. and, and providing annotated training images from at least one or more portions. , and steps.

[0039] In an embodiment, the method is a) a non-machine learning image processing method, or b) a segmentation method. generating at least one candidate image annotation using the machine learning model, Includes flops.

[0040] According to a fifth aspect, the present invention provides a method for implementing a method according to the second aspect when executed by one or more processors. or the fourth aspect. This aspect includes computer program code as described above (e.g., non-volatile). A computer-readable medium may also be provided.

[0041] Therefore, certain aspects of the present invention provide an integrated annotation and segmentation system. Supports continuous training and evaluation of machine learning (e.g., deep learning) models built with the system do.

[0042] Annotation systems simply use images that have been annotated / identified by humans as well as combining segmented / identified results obtained by the following techniques: Image processing algorithms, annotation deep learning models, correction models, and segmentation Human input is used only when necessary, and annotations are It is only the final step to verify or correct the image. As more and more tasks are performed, the deep learning model improves and can perform tasks with less human intervention. In many cases, the data (e.g. annotation, correction, segmentation) segmented / identified results obtained from different models (e.g., segmentation / discrimination models) It was found that accurate annotations can be obtained by combining the results of It was.

[0043] Any of the various individual features of each of the above-described aspects of the invention, as well as the claims Any of the various individual features of the embodiments described herein, including those claimed can be combined in any suitable and desired manner. [Brief explanation of the drawings]

[0044] In order that the present invention may be more clearly defined, reference will now be made to the accompanying drawings, which illustrate, by way of example only, the principles of the invention. The form will be explained below.

[0045] [Figure 1A] 1 is a schematic diagram of the architecture of a segmentation and identification system with built-in annotation and training capabilities according to an embodiment of the present invention; [Figure 1B] 1B is a schematic diagram of the segmentation and identification system of FIG. 1A according to an embodiment of the present invention; [Figure 2] FIG. 1C is a flow diagram of the general workflow of the system of FIGS. 1A and 1B. [Figure 3] 1C is a pseudo flow diagram of the operation of the system of FIGS. 1A and 1B. [Figure 4] FIG. 1C is a schematic diagram of an annotation tool of the system of FIGS. 1A and 1B. [Figure 5] 1A and 1B is a flow diagram 70 for the annotation and training workflow of the system of FIGS. [Figure 6] FIG. 1C is a schematic diagram of a deep learning segmentation and discrimination model for the system of FIGS. 1A and 1B. [Figure 7A] 1A and 1B are schematic diagrams of three exemplary embodiments of segmentation and identification in the system of FIGS. [Figure 7B] 1A and 1B are schematic diagrams of three exemplary embodiments of segmentation and identification within the system of FIGS. [Figure 7C] 1C is a schematic diagram of three exemplary embodiments of segmentation and identification within the system of FIGS. 1A and 1B. FIG. [Figure 8A] 1A and 1B are schematic diagrams of three exemplary embodiments of the deployment of the system of FIGS. [Figure 8B] 1A and 1B are schematic diagrams of three exemplary embodiments of the system deployment of FIGS. [Figure 8C] 1A and 1B are schematic diagrams of three exemplary embodiments of the system deployment of FIGS. DETAILED DESCRIPTION OF THE INVENTION

[0046] Figure 1A shows a segmentation and classification system with built-in annotation and training capabilities. 1 is a schematic diagram of the architecture of the system 10. The system 10 is and training subsystem 12, segmentation and identification subsystem 14, and trained Segmentation and identification with a graphical user interface (GUI) and a user interface 18 including a user interface I) 20.

[0047] In general, the annotation and training subsystem 12 is Supervised "ground truth" (about the targeted anatomy or tissue) The system uses "(correct answer)" to annotate medical images selected as training data. The annotated training data (e.g., for deep learning algorithms) ) is fed into a machine learning algorithm to generate a trained segmentation and discrimination model1 6) to train the segmentation and discrimination model. The segmentation and discrimination model is used by the segmentation and discrimination subsystem 14. (26) have been proposed to segment and analyze objects of interest in new medical images. It is used to identify, and also includes classification and labeling of pixels / voxels of an image.

[0048] The GUI 20 can be implemented in several different ways, for example, by the annotator (i.e. (or an operator with the appropriate skills to recognize image features) display devices (e.g., personal computers, laptops, tablet computers) The GUI must be installed as software on a computer or mobile phone. In another example, the GUI 20 can be used to provide annotations to web pages. It can be provided and accessed via a web browser.

[0049] Using the annotated images, we train segmentation and classification models (22). In addition to providing the training data, the annotation and training subsystem 12 also provides the trained segmentation data. The performance of the algorithm and the discriminative model16 is evaluated (24) to determine whether the model is ready for deployment. This allows you to make decisions about how to improve your model and also allows you to retrain and improve your model.

[0050] However, the segmentation and discrimination models may fail for one or more particular images. If so, the images that the model failed are annotated as a new training dataset. The application and training subsystem 12 is added to the Retrain (28).

[0051] FIG. 1B is a schematic diagram of the system 10. The system 10 performs segmentation and and identification controller 32 and the aforementioned user interface 18. The application and identification controller 32 includes at least one processor 34 (or, in some embodiments, In some embodiments, the system 10 includes a processor (or processors) and a memory 36. as a combination of software and hardware on a computer (e.g., a personal computer or mobile computer) It may be implemented as a dedicated image segmentation system (e.g., as a computerized image processing device) or as a The system 10 may be described as optionally distributed. For example, all or some components of memory 36 may be accessible from processor 34. the user interface 18 can be located remotely from the memory 36 and and / or may be located remotely from the processor 34, and in fact it It may include a web browser and a mobile device application.

[0052] The memory 36 is in data communication with the processor 34 and is typically volatile. and non-volatile memory (and one or more of each memory type). (which may include memory), RAM (random access memory), ROM, and one or more This includes mass storage devices.

[0053] The processor 34 controls the annotation and training subsystem 12 and the segmentation subsystem 13. and identification subsystem 14. As will be described in more detail below, The application and training subsystem 12 processes the original images according to a non-machine learning image processing method. an initial segmenter and classifier 38, an image annotator 40, and a model trainer 42; and a model evaluator 44. The segmentation and identification subsystem 14 The processor includes a segmenter 46, a structure identifier 50, and a result evaluator 52. The server 34 also includes an I / O interface 54 and a result output 56 .

[0054] The memory 36 stores program code 58, image data 60, training data 62, and evaluation images 6 4. 66 ground truth images, 16 trained segmentation and discrimination models, and 16 trained annotations. The training data includes a training model 68, a trained correction model 69, and a training result model 69.

[0055] The segmentation and discrimination controller 32 is configured to receive, at least in part, the The program code 58 is implemented by the processor 34 executing the program code 58 from the processor 34 .

[0056] Generally speaking, the I / O interface 54 receives image data relating to a subject or patient. (e.g., DICOM format) into image data 60 or training data 62. This is for analysis and / or training (or both, as described below). Once the system 10 converts the structures into image data, Once segmented and identified from 60, the I / O interface 54 transmits the results of the analysis ( optionally in the form of a report), for example via the results output 56 and / or GUI 20. .

[0057] FIG. 2 is a flow diagram 20 of the general workflow of the system 10. In S72, A set of images (typically stored remotely or locally) (from a database about These images are input into the system 10 via interface 54 and used as training data. The training data 62 is stored in a database 62. The training data 62 is a set of images that are expected to be encountered in a clinical setting. The image should be representative of both base and edge cases (i.e., "positive" cases). The structure and organization in the base case should include both "normal" and "extreme" cases. , which can be more easily segmented or identified than those of edge cases. For example, For scanning, it is necessary to segment the bone from the surrounding muscle and fat. Scanning is easier in young, healthy subjects because In patients with fibroblasts, the bone boundaries were less porous and more clearly defined than in older, frailer patients, and in the latter The bone is very porous and the boundaries are less well defined, although the base case and edge It is desirable to collect examples of both the sizing and the sizing cases as training data.

[0058] In this step, we create a second set of images to evaluate the performance of the model being trained. These evaluation images are representative of clinical images. and stored in the evaluation image 64.

[0059] In S74, an operator with appropriate expertise is trained using the image annotator 40. Annotations are applied to both the data 62 and the evaluation image 64. The annotations required also change, e.g., if the model segments bone material from non-bone material, If the object should be trained to do this, then the annotation needs to be done on the bone pixels / voxels. The purpose of this study is to distinguish and identify bones from non-bone ones, and therefore segmentation data is included. On the other hand, more advanced models are required to identify each bone mass. In this case, annotation requires differentiation and labeling of each bone mass pixel / voxel. The purpose of the RFC is to tag the data and therefore additional identification data should be included.

[0060] In S76, the model trainer 42 uses the annotated training data 62 to generate a classifier model. In this embodiment, the model is a segmentation and discrimination model. This is a classifier that determines the classification of each pixel / voxel in the image. In training, the decision is made to reach the correct answer (annotation) from the input (training image). The purpose of the model is to determine a decision pattern. For example, machine learning algorithms such as support vector machines and random forest trees are used. It can be trained by using

[0061] In this embodiment, a deep neural network is used, as described below (see FIG. 6). This deep neural network consists of an input layer, an output layer, and layers in between. Each layer consists of artificial neurons, which receive one or more inputs. It is a mathematical function that takes a number of inputs and sums them to produce an output. The sum is then passed through a nonlinear function, with each weighted individually. As the network learns, the model weights are adjusted, and the adjustments are This is done according to the error (the difference between the network output and the annotation) , adjustments are made until the error cannot be reduced any further.

[0062] In S78, the model evaluator 44 controls the segmentation and identification subsystem 14. The trained model is evaluated by controlling the In S80, the model evaluator 44 evaluates whether the trained model is It is checked whether the test score has been reached or not, and the result of this process is used in the algorithm in S74. This is done by comparing the annotations with the trained model. The results generated by the model and the annotated image (ground truth image) used in S74. The difference between the annotation data associated with the 66 is less than a predetermined threshold. For example, the difference is determined by determining whether the model is Segmentation and segmentation suggested by annotation data The ratio of overlap to the segmentation suggested by the annotation data If they match perfectly, the overlap is obviously 100. %, which implicitly results in a pass. In some applications, this threshold is set to 90%.

[0063] If the model evaluator 44 determines in S80 that the model fails, processing continues to S82. Proceed and one or more new images are imported in the same way as the original training data was imported in S72 (original training data) and / or the learning algorithm is adjusted / modified. (e.g., tuning neural network parameters, Making changes to the layers of a neural network, or the neurons of a neural network This can be done by making changes to the startup function, etc.) Then, the process proceeds to S74. Return to.

[0064] In S80, the model evaluator 44 compares the results generated by the trained model with the results used in S74. If it is determined that the discrepancy between the input annotations is less than a predetermined threshold, i.e., training If it is determined that the trained model is acceptable, the process proceeds to S84, where the trained model is The trained segmentation and discrimination model 16 is stored and deployed. The phase is complete (at least for now), so S85 is a trained model. The code is used by the segmentation and identification subsystem 14 to Segment one or more structures or materials / tissues in the new medical image stored in image data 60. At S86, the result evaluator 52 evaluates the results to the target structure or material. Compared to one or more predefined conditions or parameters known to characterize quality The segmentation and classification results are verified by If the performance of the evaluation and classification model is shown to be unacceptable, the process proceeds to S88. The new images are added to the training set as new training data, and the process is These images can then be annotated, used to retrain models, etc. )Proceed to S74.

[0065] If the result evaluator 52 does not determine the result to be unsuccessful in S86, processing continues to S90. 10. The new medical image may be displayed by the user interface 18 of the system 10. The segmentation / identification results from the image are output. (Optionally, in S86, the result evaluator 52 If the result is not rejected, the segmentation and classification results are presented to the user. and (e.g.) The user may decide to reject the result (for example, by selecting a "fail" or "reject" button on the GUI 20). If the result is flagged as a failure, the result is considered a failure and processing proceeds to S88. The user accepts the results by selecting the "Pass" or "Accept" button on the GUI20. If the decision of the evaluator 52 is not to be appealed, the process proceeds to S90.

[0066] Then, after S90, the process ends. For example, the results may be used for diagnostic purposes or for further determination. It can be used as input for sexual analysis.

[0067] FIG. 3 is a pseudo-flowchart 70 of the operation of the system 10. As mentioned above, annotation The segmentation and training subsystem 12 performs segmentation and The system is configured to annotate medical images and their annotations. The annotators are used to train segmentation and discrimination models. The annotation is performed on a medical image set via a GUI20, which is In this specification, this may also be referred to as an annotation interface.

[0068] The annotator combines information from different resources to complete the annotation. In this embodiment, the annotator combines information from one or more candidate image annotations. and, if necessary, complete the annotation using the annotation tools in GUI20. In this embodiment, the candidate image annotations can be in the following form: Preliminary segmentation and identification results generated using existing non-machine learning image processing methods; or results produced by a partially trained segmentation and discrimination model. The complementary image annotations are displayed together (i.e., next to each other) and The data is arranged so that it is easy to compare which parts of each are best.

[0069] In addition, because the preliminary segmentation results are generated using existing image processing methods, You will find that the results typically require some improvement. Examples of image processing methods include contour detection, blob detection, and threshold-based object segmentation. These methods extract pixels corresponding to bones from the surrounding pixels in a CT scan. Although it can segment quickly, it often requires rough or coarse preliminary segmentation. This brings about

[0070] The GUI 20 includes a preliminary results window 102 and an annotated image window 104. The annotated image window 104 includes a plurality of annotation tools 106 The annotation tools 106 include both manual and semi-manual annotation tools. These can be displayed to and manipulated by the annotator. These tools are shown diagrammatically in Figure 4 and include manual tools. Included are a brush tool 130 and an eraser tool 132. is controlled by annotators to label pixels of different structures or materials with different values ​​or colors. The eraser tool 132 can be controlled by the annotator to remove over-segments. The semi-manual type allows the removal of missing or over-identified pixels from the target structure or material. The annotation tools include a fill tool134 that the annotator can control. You can control the contours to enclose the target structure or material and view the annotated image window. 104 can annotate all pixels / voxels within that contour. The semi-manual annotation tools also include a region expansion tool136, which After annotating a very small portion of the target structure or material, the annotator Control the Annotated Image window 104 to extend the annotation. The term "structure" can encompass the entire structure or material.

[0071] The system also includes machine learning-based tools that allow annotators to accurately segment For example, annotation tools1 06 includes machine learning-based annotation model control138, This calls the trained annotation model stored in annotation model 68. The annotation model 68 is controlled by the annotator to segment the object. Multiple points can be selected for the annotation model 68 to use when In this embodiment, the annotation model 68 provides the annotator with the four edges of the object. This encourages us to identify the four extreme points, i.e., the four extreme points. The points are the leftmost, rightmost, topmost, and bottommost pixels. The data can be displayed, for example, by touching the image with a stylus (if displayed on a touchscreen). Select each point in turn by pressing the The annotation model 68 uses these edge points to segment the object. In this way, the selected points (e.g., edge points) are added to the annotation 6 8. The annotation data 118 used to train the model 8 is constructed.

[0072] The annotation tool 106 includes an annotation motion capture tool 140. It is another mechanism for recording annotation data 118 and for annotators to The movements are recorded by the system 10 (in the trained correction model 69). train a corrected model (stored in You can move contours inwards or under-segmented contours outwards. The movements in and out and the positions at which they are made are recorded in the annotation motion capture. The trained modified model (stored in trained modified model 69) is trained by the trainer tool 140. It is also recorded as input for the annotation motion capture tool1. 40 records the amount of effort, such as the number of mouse movements and the amount of mouse scrolling, Training is used to give greater weight to more serious cases.

[0073] Returning to FIG. 3, in use, the preliminary segmentation and identification results 110 are The preliminary results will be presented to the annotator via the preliminary results window 102 of the UI 20. If the annotator is satisfied with the accuracy of the segmentation and classification results 110 The annotator can, for example, click the mouse on the preliminary results window 102. Move the preliminary results 110 to the annotated image window 104 of the GUI 20. If the annotator is satisfied with only some (but not all) of the preliminary results 110, the annotator only a sufficient portion of the preliminary result 110 into the annotated image window 104. Move and use the annotation tools 10 in the annotated image window 104. 6 to correct / complete the segmentation and identification.

[0074] If the annotator is satisfied with any part of the preliminary segmentation and classification results, If not, the annotator may use the annotation tool 106 to ab initio add the annotation to the image. Apply annotation.

[0075] The annotator has completed the annotation of the partially satisfactory preliminary results 110 or the image If you have either annotated the original image or annotated it yourself, The image 114 is in the annotated image window 104, and the correct image It is stored as 66.

[0076] After one or more correct images 66 are collected in this manner, the annotated original image That is, the ground truth image 66 is used to train the segmentation and discrimination model. The segmentation and discrimination model (in the trained segmentation and discrimination model 68) ) Once available, the segmentation and The classifications and identifications are also provided to the annotators as a reference. are satisfied with the results or parts of such results produced by the application and identification model. If so, the annotator moves a sufficient portion into the annotated image window 104. The annotator can select a sufficient portion of the preliminary results in the Annotated Image window 104. It is possible to combine insufficient annotations with sufficient parts from a pre-trained model. If there are still images or image parts that have been annotated, the annotator may The annotations can be corrected or completed using the tool 106.

[0077] The collected correct images in the correct image 66 and the original image 60 are used for the segmentation and identification model. To evaluate the trained model, we use a different ground truth image and In some embodiments, the same set of images is used for both training and evaluation. If the performance of the trained model is satisfactory during image evaluation, the model is considered to be trained. Any new image is fed to the segmentation and discrimination model 68. It is used by the segmentation and identification subsystem 14 for processing. If the number of images is insufficient, collect more correct images. by the system 10 (in determining whether the segmentation and discrimination model is acceptable). The criteria used in the system are adjustable. Such adjustments are typically made during the development of the system. This is done by the person who trained the model originally and is adjusted according to the requirements of the application. For example, vBMD (volumetric bone mineral density) is determined using To verify whether the entire bone is accurately segmented from the surrounding material, The criteria are set for the whole bone and the cortical bone segments when calculating cortical porosity. Set standards to verify station accuracy.

[0078] FIG. 5 is a flow diagram 150 for the annotation and training workflow. For reference, in S152, the original image 60 is selected or input, and in S154, the original image are processed by the initial segmenter and classifier 38, thereby forming a preliminary segmentation Processing continues to S156, where segmenter 48 generates the trained segmentation and classification results. The segmentation and discrimination model is stored in the trained segmentation and discrimination model 16. If so, the process proceeds to S158, and the trained The original image 60 is also processed using the segmentation and discrimination model 16. Processing continues at S160. At S156, the system 10 performs the trained segmentation and classification If it is determined that an alternative model is not available, the process proceeds to S160.

[0079] At S160, the annotator uses the system 10 to generate preliminary model results and trained models. Whether any part of the results pass or fail (usually these results are displayed in the user interface) If so, the process continues to S1 62, the annotator uses the annotation tool 106 to annotate the part that is deemed acceptable. The image is moved to the annotated image window 104, and the process proceeds to S164. If the annotator determines in S160 that none of the obtained results are acceptable, Then, processing proceeds to S164.

[0080] In S164, if none of the obtained results pass, the annotator If you are prompted to annotate ab initio or if you have no other options regarding annotation, The annotator continues to complete, correct, or supplement the annotation until the annotation is satisfactory. This is done by controlling the manual and semi-manual tools 106 with the The data may optionally be used to determine if a correction model 69 is available using a correction model control 142. If so, a modified model 69 can be invoked to refine the annotation.

[0081] In S166, segmentation and recognition are performed using the correct image and annotation data. In parallel, a separate model is trained, and in S168, the ground truth images and annotations are The data is used to train an annotation model 68 and a correction model 69 (at S164). If no corrected model is available, after the corrected model is generated in S168, The modified model is available in the modified model 69 in later passes up to 164. (Note that this is not possible.)

[0082] The process proceeds to S170, where the trained segmentation and discrimination model is used for annotation. The model is evaluated by the model evaluator 44 of the learning and training subsystem 12. If the result of the test is unsuccessful, the process proceeds to step S174, where the trained segmentation The segmentation and discrimination model is saved in the trained segmentation and discrimination model 68 and a new The image is expanded for use in processing image 60. Otherwise, processing returns to S152. , one or more additional ground truth images 66 are collected or selected and the process is repeated.

[0083] The new image is then processed by the segmentation and classification subsystem 14. This segments and identifies the target structure or material (e.g., tissue). As such, the segmentation and identification subsystem 14 comprises four modules: A pre-processor 46, a segmenter 48, a structure identifier 50, and a result evaluator 52. The sensor 46 verifies the validity of the input image (in this embodiment, a medical image). Verify the medical image information, including whether the image is complete and whether it has been corrupted. The pre-processor 46 also reads the medical images into image data 60 within the system 10. The image can be of different types and the pre-processor 46 is of primary interest. It is designed to read images in a format that has a correlation with the DICO standard. M is also included. Multiple pre-trained segmentation and discrimination models16 are available. Therefore, the pre-processor 46 may extract the Regarding segmentation and classification based on the information and configured to determine which of the application and identification models 16 should be used. For example, in one scenario, two classifier models are trained. The first classifier model can be used to segment and identify the radius from a CT scan of the wrist. The second classifier model is intended to segment the tibia from CT scans of the leg. The new CT format in DICOM format can be used for documenting and identifying images. If a scan is to be processed, the pre-processor 46 extracts the scanned body part from the DICOM header. Position information can be extracted to determine, for example, that a radial classifier model should be used.

[0084] The segmenter 48 and structure classifier 50 each identify target structures or materials in the image using training data. A model selected from the trained segmentation and discrimination models 68 is used to perform segmentation. The results of the segmentation and classification are then evaluated by a result evaluator 52. In this embodiment, the evaluation performed by the result evaluator 52 is It involves: changing one or more predefined conditions or parameters of the target structure or material. data (e.g., tolerance range of one or more structures or material dimensions, or structure or material Verify the results in relation to the volume of the object, etc., and check if the segmentation and identification results are clearly insufficient. The result evaluator 50 determines whether the segmentation and classification results are A result is output indicating whether it is indeed sufficient or not.

[0085] The automatic evaluation performed by the result evaluator 52 can be manually augmented, e.g. The results of the segmentation and identification and / or the evaluation of the result evaluator 52 are used for further evaluation. The results can be displayed to the user (e.g., a physician) to refine the automated assessment.

[0086] Images that fail the segmentation and classification results are The result evaluator 52 uses the segmentation images as additional training images for retraining another model. If the segmentation and identification results are passed, the segmentation and identification results are sent to the I / O interface. via an interface 54 (e.g., on the display of a computer or mobile device) ) results output 56 and / or user interface 18. In some embodiments, The segmentation and identification results are used as input for further quantitative analysis. For example, the radius is segmented and identified by the system 10 from a CT scan of the wrist. If so, attributes such as volume and density of the extracted radius can be calculated (locally or remotely). can be passed to another application (running on the system) for further analysis .

[0087] Convolutional Neural Networks for Segmentation and Classification According to Embodiments of the Present Invention The deep learning segmentation network is generally shown at 180 in FIG. Deep Learning Segmentation and Discriminative Models18 0 is a comparison of the contracted convolutional network (CNN)182 and the dilated convolutional network (CN The reduction or downsampling CNN 182 processes the input image and Dilation or upsampling reduces the resolution of the feature maps through various layers. Pulling CNN184 processes these features through layers of increasing resolution. This eventually generates a segmentation and classification mask.

[0088] In the example of Figure 6, the reduced CNN 182 has four layers 186, 188, 190, and 192. The extended CNN has four layers 194, 196, 198, and 192. Note that other values ​​for the number of layers may be used. , layer 192) is shared by the contraction and expansion networks 182, 184. Each layer 186,...,198 has an input and a feature map (at each layer The input of each layer 186,...,198 is a convolutional The image is then processed and converted into a feature map (represented by a hollow arrow in the figure). The convolution process or activation function can be, for example, the normalized linear unit (ReLU, rectified linear unit), sigmoid unit, or Tanh unit It is possible.

[0089] In the reduced CNN 182, the input of the first layer 186 is an input image 200. In 182, the feature maps on each layer are downsampled to another feature map. are input to each of the next lower layers 202, 204, 206. The downsampling process 208 may be implemented, for example, by a max pooling operation or a stripe This can be a code operation.

[0090] In the extended CNN184, feature maps208 on each layer192,198,196 ,201,212 are upsampled to separate feature maps 214, 216, 218, respectively. It is pulled and used in the next layers 198, 196, and 194. The sampling process may be, for example, an upsampled convolution operation, a transposed convolution, or a bilinear It can be upsampling.

[0091] To capture the localization patterns, each pair in the reduced CNN182 The corresponding high-resolution feature maps 220, 222, 223 in layers 190, 188, 186 are 24 is converted into upsampled feature maps 214, 216, and 218, respectively. The resulting The resulting high-resolution feature maps / feature map pairs 220 / 214,222 / 216, 224 / 218 are used as inputs for each layer 198, 196, 194. The final layer 194 of the extended CNN 184 uses the features resulting from the convolution process. The image map 226 is transformed and output as the final segmentation and identification map 228. This transformation is shown as a solid arrow in the figure. , can be implemented as a convolution operation or sampling, for example.

[0092] In this way, the deep learning segmentation and discrimination model 180 of FIG. Location information from the contraction path is transferred from the extension path to the Combined with contextual information in the expansion path Ultimately, it is localisation and context that xt) to get general information on combining

[0093] Illustrative Segmentation and Identification within System 10, According to an Embodiment of the Present Invention An example implementation is shown in Figures 7A to 7C. Referring to Figure 7A, The implementation 240 of the method and identification is characterized as a "one-step" embodiment. The first segmentation and identification 240 is performed after the annotation and training phase 24. 2 and a segmentation and identification phase 244. In the former, the original image 246 is The segmentation and classification are then annotated 248 and training is performed. After this, a segmentation and discrimination model 250 is generated. and in the identification phase 244, a new segmentation and identification model 250 is generated. The image 252 is subjected to a segmentation and identification act 254, after which the segmentation The pattern and the identification result 256 are output.

[0094] Thus, in the embodiment of FIG. 7A, model 250 is trained to perform segmentation and For example, segmentation of the radius and fibula in a CT scan of the wrist. When training the application and classification models, training images are annotated. In this case, different values ​​are assigned to the radius and fibula voxels. Once the model is ready, Any new wrist CT scan 252 is processed by the segmentation and discrimination model 250. and the result 256 is the radius and This is provided as a map for the fibula.

[0095] A second segmentation and identification implementation 260 is shown schematically in FIG. 7B. In implementation 260, segmentation and identification is implemented in two steps. Referring to FIG. 1, an example implementation 260 includes an annotation and training phase 262 and a segmentation phase 263. The annotation and training phase 262 also includes an annotation and identification phase 264. This involves training a segmentation model and training a discrimination model as separate processes. Therefore, the original image 266 is annotated with annotations 268 and segmented. A model 270 is trained. The segmented images 272 are annotated with annotations 274. After the two models 270 and 276 are trained, Any new image 264 is first segmented by the segmentation model 270. 80, and the resulting segmented image 282 is used to generate the discriminative model 276 The result of the classification is output as a result 286. By these two steps, and in these two steps, All bones can be first segmented from the surrounding material, and different types of bones can be It can be differentiated and identified.

[0096] A third segmentation and identification implementation 290 is shown schematically in FIG. 7C. In this implementation, the segmentation model achieves very accurate results, Also, the classification is based on human learning instead of machine learning based models. This is done using a heuristic-based algorithm. Referring to FIG. 7C, an example implementation 290 performs annotation and training phase 2. 92 and a segmentation and identification phase 294. The original image 266 is In the orientation and training phase292, annotation268 is used for segmentation. and used to train the segmentation model 270. In the segmentation and identification phase 264, new images 278 are used to generate segmentation models. 270 to produce a segmented image 282; and The classification is done using a heuristic-based algorithm. The identified results 286 are output. For example, the segmentation model 2 Once accurately segmented using 70, the wrist CT scan can be By volume calculation and differentiation This allows different bones to be distinguished.

[0097] For example, in a wrist HRpQCT scan, after bone segmentation, bone volume By calculating and comparing the products, one can easily distinguish and differentiate the radius from the ulna. This is because the radius is larger than the ulna in the same subject.

[0098] Three exemplary deployments for system 10 are shown generally in FIGS. 8A-8C. These are: on-premise deployment; cloud deployment; and hybrid deployment. As shown in FIG. 8A, in a first deployment example 300, the system 10 is deployed locally. (although it may be in a distributed manner) (e.g., model training and Any data processing (such as image segmentation and identification) may be performed by one or more of the systems 10. The data is stored in one or more data stores 304. The user can access the system via the user interface 18. 10 and interact with it (e.g., annotating images and inspecting results).

[0099] As shown in FIG. 8B, in a second development 310, the system 10 includes a user interface All but 18 are deployed within the cryptographic cloud 312. The user interface 18 and the encryption cloud 312 are located locally. The communication 314 between them is encrypted.

[0100] As shown in FIG. 8C, in a third development 320, the system 10 is partially cryptographically The data is stored in a cloud service 322, while the data storage unit 326 and a processing capability (e.g., processor) 328 and a user interface 18. The local portion 324 is locally located. In such a deployment, typically Most of the system 10 (except for the interface 18) is deployed within a cloud service 322. Although the data storage unit 326 and the processing capacity unit 328 are To support the service 18 and communication 330 between the local part 324 and the cloud service 322 The communication 330 between the local part 324 and the cloud service 322 is encrypted. are.

[0101] Those skilled in the art will recognize that many modifications can be made without departing from the scope of the invention. In particular, certain features of the embodiments of the present invention may be used to provide further embodiments. Note that it is possible.

[0102] If any references to prior art are made herein, such references are It should be noted that the disclosure of the prior art does not constitute an admission that the prior art forms part of the public knowledge in any country. I want to be.

[0103] In the appended claims and the foregoing detailed description of the invention, Therefore, unless the context requires otherwise, the terms "prepare" and "prepare" (third person singular) Terms such as "has" and "has" are used in an inclusive sense (i.e., the presence of the declared feature). Although the term "presence" is used to indicate the presence of additional features in various embodiments of the invention, This does not preclude the inclusion or addition of other elements. [Explanation of symbols]

[0104] 10 Systems 12 Annotation and Training Subsystem 14 Segmentation and Identification Subsystem 16 Pre-trained segmentation and discrimination models 18 User Interface 20 GUI 32 Segmentation and Identification Systems 34 processors 36 memory 66 Correct image 106 Annotation Tools 118 annotation data 182 Reduced Network 184 Extended Network 242 Annotation and Training 244 Segmentation and Identification 262 Annotation and Training 264 Segmentation and Identification 292 Annotation and Training 294 Segmentation and Identification 302 processor 304 Data storage unit 312 Encrypted Cloud Services 322 Encrypted Cloud Services 324 local part 326 Data Storage Unit 328 processors

Claims

[Claim 1] 1. An image segmentation system, comprising: a training subsystem configured to train a segmentation machine learning model using annotated training data comprising images associated with each segmentation annotation to generate a trained segmentation machine learning model; a model evaluator; and a segmentation subsystem configured to perform segmentation of structures or materials in an image using the trained segmentation machine learning model; The model evaluator performs an evaluation of the segmentation machine learning model by: (i) controlling the segmentation subsystem to segment at least one evaluation image associated with an existing segmentation annotation using the segmentation machine learning model, thereby generating a segmentation for the annotated evaluation image; (ii) forming a comparison of the segmentation of the annotated evaluation image with the existing segmentation annotations; If the comparison indicates that the segmentation machine learning model is acceptable, deploying or releasing the trained segmentation machine learning model for use; An image segmentation system configured to:

Images