Method and system for image segmentation and identification

Abstract

To provide a method and system for managing training and retraining processes of a machine learning model for segmentation, efficiently.SOLUTION: In a segmentation and identification system 10, a segmentation and identification controller 32 includes a model evaluator. The model evaluator executes, for evaluation of a segmentation machine learning model, the steps of: using a trained segmentation machine learning model to generate a segmentation of an annotated evaluation image; forming a comparison of the generated segmentation of the evaluation image and an existing segmentation annotation; and deploying or releasing the trained segmentation machine learning model for use if the comparison indicates that the segmentation machine learning model is satisfactory.SELECTED DRAWING: Figure 1B

Description

Technical Field

[0001] The present invention relates to methods and systems for image segmentation and identification, and more particularly to segmentation and identification of medical images (e.g., bones, other anatomical structures, masses, tissues, landmarks, lesions, pathological conditions, etc.), and also relates to medical imaging modalities such as computed tomography (CT), magnetic resonance (MR), ultrasound, lesion scanner imaging, etc., but is not limited to these applications. The present invention is for automating the annotation (annotation) of medical image data for training machine learning models for segmentation and identification purposes, and also relates to methods and systems for evaluating and improving machine learning models.

[0002] Related Applications This application claims priority to and the benefit of U.S. Patent Application No. 16 / 448,252, filed on June 21, 2019, the entire contents of which are incorporated herein by reference.

Background Art

[0003] Accurate segmentation and identification of medical images are required for quantitative analysis and disease diagnosis. Segmentation is the process of delineating an object (e.g., an anatomical structure or tissue) within a medical image from its background. Identification is the process of identifying an object and correctly labeling it. Traditionally, segmentation and identification are performed manually or semi-manually. Manual techniques require experts with sufficient knowledge in the field to be able to draw the contours of the target objects and label the extracted objects.

[0004] There also exist computer-assisted systems that provide semi-manual segmentation and identification. For example, such systems can detect an approximate contour of an object of interest based on selected parameters including signal strength, edges, 2D / 3D curvature, shape, or other 2D / 3D geometric features. Then, an expert manually refines the segmentation or identification. Alternatively, an expert can provide input data such as the approximate position of the target object to such a system, and the computer-assisted system performs segmentation and identification. Whether manual or semi-manual, both are labor-intensive and time-consuming methods. Also, the quality of the results strongly depends on the expertise and proficiency of the expert. Considerable differences can be introduced by the operator / expert with respect to the resulting segmented and identified objects. In the past few years, machine learning, particularly deep learning (e.g., deep neural networks and deep convolutional neural networks), has come to outperform humans in many visual recognition tasks, including medical imaging. Patent Document 1 discloses a method and system for medical image segmentation based on artificial intelligence. The method includes receiving a medical image of a patient, automatically determining a current segmentation context based on the medical image, and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context.

[0005]

[0006]

[0007] ​​​​​​​​​​​​​​automatically selecting one segmentation algorithm; and segmenting a target anatomical structure in a medical image using the selected at least one segmentation algorithm. Patent Document 2 discloses a method for segmenting an image of a target patient, and

[0008] the method includes the following steps: providing a target 2D slice and a neighboring 2D slice for a 3D anatomical site image, and calculating a segmentation region by a trained multi-slice fully convolutional neural network (multi-slice FCN), where the region includes defined intra-body anatomical features that spatially extend across the target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. Patent Document 3 discloses a system and method for applying a deep convolutional neural network to a medical image to generate a real-time or near-real-time diagnosis or a recommended diagnosis plan. al network), where the region spatially extends across the target 2D slice and the neighboring 2D slices and includes defined intra-body anatomical features, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. target 2D slice and the neighboring 2D slices, and each of the 2D slices and the neighboring 2D slices is processed by a corresponding shrinking component of the multi-slice FCN, and the processing is in accordance with the order of the target 2D slice and the neighboring 2D slices, which is based on a sequence of 2D slices extracted from the 3D anatomical image, and the output of the shrinking component is combined and processed by a single expanding component that outputs a segmentation mask for the target 2D slice. step of calculating.

[0009] Patent Document 3 discloses a system and method for applying a deep convolutional neural network to a medical image to generate a real-time or near-real-time diagnosis or a recommended diagnosis plan. in real time or near real time. Disclosed is a method, which includes the steps of: performing image segmentation on a plurality of medical images, where the image segmentation step separates a region of interest from each image; applying a cascaded deep convolutional neural network detection structure (cascaded deep convolutional neural network detection structure) to the segmented images, where the detection structure includes: i) a first stage that uses a first convolutional neural network to screen all possible positions within each 2D slice of the segmented medical images by means of a sliding window method to identify one or more candidate positions; and ii) a second stage that uses a second convolutional neural network to screen a 3D solid constructed from the candidate positions, where the screening is performed by selecting at least one random position within each solid with a random scale and a random viewing angle, thereby identifying one or more refined positions and classifying the refined positions; and automatically generating a report including a diagnosis or a recommended diagnosis plan. This is a step, and it includes the above steps. (cascaded deep convolutional neural network detection structure) to the segmented images, and this step is to apply the detection structure. The detection structure includes: i) a first stage wherein a first convolutional neural network is utilized to screen all possible positions within each 2D slice of the segmented medical images by means of a sliding window method to identify one or more candidate positions; and ii) a second stage wherein a second convolutional neural network is utilized to screen a 3D solid constructed from the candidate positions, and the screening is performed by selecting at least one random position within each solid with a random scale and a random viewing angle, thereby identifying one or more refined positions and classifying the refined positions. This step includes the above steps. .

[0010] However, there are problems with applying such a method to the segmentation and identification of medical images. That is, in order to train and validate a machine learning algorithm, ground truth data is required, but this data is provided by human experts who annotate the data, which requires a great deal of time and cost. which involves usage; the improvement of the machine learning model ideally requires the addition of misrecognition results (for example, when the trained model fails in medical image segmentation or identification, etc.), but efficiently managing the training and retraining processes is highly difficult; for some biomedical applications it is difficult or impossible to obtain a large number of training images, and also, when the training data is limited, it becomes difficult to efficiently train segmentation and identification models. .

Prior Art Documents

Patent Documents

[0011]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

[0012] An object of the present invention is to provide a segmentation system capable of integrating annotations. Specifically included.

[0013] According to a first aspect, the present invention is an image segmentation system comprising: A training subsystem configured to train a segmentation machine learning model using annotated training data (annotated training data) including images (such as medical images) associated with each segmentation annotation to generate a trained segmentation machine learning model, A model evaluator, Using the trained segmentation machine learning model to segment (for example, bones, muscles, fats, etc.) within the image configured to perform segmentation on a structure or material (including fat or other biological tissue), and includes a segmentation subsystem, wherein the model evaluator evaluates a segmentation machine learning model by: (i) controlling the segmentation subsystem to segment at least one evaluation image associated with existing segmentation annotations using the segmentation machine learning model, thereby generating a segmentation for the annotated evaluation image; (ii) forming a comparison between the segmentation of the annotated evaluation image and the existing segmentation annotation; and if the comparison indicates that the segmentation machine learning model is qualified, deploying or releasing the trained segmentation machine learning model for use.

[0014] Thus, the model evaluator evaluates a segmentation machine learning model that uses the operational segmentation subsystem for segmentation, providing an integrated training and segmentation system.

[0015] In an embodiment, the system is configured to deploy or release the model for use if the segmentation of the annotated evaluation image and the existing annotation match within a predetermined threshold.

[0016] In an embodiment, the system is configured to deploy or release the model for use if the segmentation of the annotated evaluation image and the existing annotation match within a predetermined threshold. ​​​​​​​​​​If the existing annotations do not match within a predetermined threshold, it is configured to continue training the model. For example, the system is configured to continue training the model by modifying the model algorithm and / or adding additional annotated training data. This allows tuning the predetermined threshold according to the desired application, and refinement can be made if the initial tuning is not satisfactory.

[0017] In an embodiment, the training subsystem (i) receives annotations for an image (some of which may optionally be generated by the segmentation subsystem using a segmentation machine learning model) and (ii) a score associated with the annotation, where the score indicates the degree of success or failure of the segmentation machine learning model for segmenting the image, with a higher score indicating failure and a lower weighting indicating success; and retrains or refines the segmentation machine learning model using the image and the annotation, including weighting the annotation of the image according to the score.

[0018] Accordingly, the segmentation machine learning model can be continuously retrained or refined before and during deployment. Note that the annotation for the image may include multiple items of annotation information, and optionally, the annotation may be partially generated by the segmentation subsystem using the segmentation machine learning model. Note that it can be generated as such.

[0019] In an embodiment, the training subsystem generates a segmentation machine learning model by refining or modifying an existing segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model.

[0020] In an embodiment, since the system includes an annotation subsystem, it is possible to use the annotation subsystem to show the validity of the annotation (which may be generated by a non-machine learning-based image processing method), and retain and use the valid part of the annotation. In an embodiment, the system further includes an identification subsystem, and the annotated training data In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model. In an embodiment, the system includes an annotation subsystem that provides at least one annotated training image (i.e., each image with a segmentation annotation) from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input for identifying one or more portions of the one or more candidate image annotations (where the portion can consist of the entire candidate image annotation), and generating the annotated training image with at least the one or more portions. For example, the annotation subsystem is configured to generate at least one candidate image annotation using a) a non-machine learning-based image processing method or b) a segmentation machine learning model.

[0021] Therefore, the annotation subsystem can be used to show the validity of the annotation (which may be generated by a non-machine learning-based image processing method), and retain and use the valid part of the annotation. Therefore, the annotation subsystem can be used to show the validity of the annotation (which may be generated by a non-machine learning-based image processing method), and retain and use the valid part of the annotation. Therefore, the annotation subsystem can be used to show the validity of the annotation (which may be generated by a non-machine learning-based image processing method), and retain and use the valid part of the annotation.

[0022] In an embodiment, the system further includes an identification subsystem, and the annotated training data The data further includes identification annotations, and the segmentation machine learning model is a segmentation and identification machine learning model.

[0023] Combining the annotation subsystem and the segmentation subsystem will promote the continuous improvement of the segmentation system, which can be said to be beneficial for applying artificial intelligence to medical image analysis.

[0024] In an embodiment, the system further includes an identification subsystem, the annotated training data further provides identification annotations, and the training subsystem is further configured to train the identification machine learning model using (i) the annotated training data after each image is segmented by the segmentation machine learning model and (ii) the identification annotations.

[0025] The training subsystem may include a model trainer, which is configured to utilize machine learning to train the segmentation machine learning model to determine the classification of each pixel / voxel of the image. The model trainer can adopt, for example, a support vector machine, a random forest tree, a deep neural network, etc.

[0026] The structure or material may include bone, muscle, fat, or other biological tissues. For example, the act of segmentation can be to separate bone from non-bone materials (such as surrounding muscle and fat, etc.), or to separate one bone from another bone.

[0027] According to a second aspect, the present invention provides a computer-implemented image segmentation method.​​​​​​​​​​​​ and the method is as follows: training a segmentation machine learning model using annotated training data each comprising an image (e.g., a medical image) and a segmentation annotation, to generate a trained segmentation machine learning model; evaluating the segmentation machine learning model, by: (i) segmenting at least one evaluation image associated with an existing segmentation annotation using the segmentation machine learning model, thereby generating a segmentation for the annotated evaluation image; (ii) forming a comparison between the segmentation of the annotated evaluation image and the existing annotation; and deploying or releasing the trained segmentation machine learning model for use if the comparison indicates that the segmentation machine learning model passes. In an embodiment, the method involves deploying or releasing the model for use if the segmentation of the annotated evaluation image and the existing annotation match within a predetermined threshold. In an embodiment, the method involves continuing to train the model if the segmentation of the annotated evaluation image and the existing annotation do not match within a predetermined threshold. For example, the method involves continuing to train the model by modifying the model algorithm and / or adding additional annotated training data. (ii) forming a comparison between the segmentation of the annotated evaluation image and the existing annotation; and deploying or releasing the trained segmentation machine learning model for use if the comparison indicates that the segmentation machine learning model passes. deploying or releasing the trained segmentation machine learning model for use if the comparison indicates that the segmentation machine learning model passes. (ii) forming a comparison between the segmentation of the annotated evaluation image and the existing annotation; and .

[0028] In an embodiment, the method involves deploying or releasing the model for use if the segmentation of the annotated evaluation image and the existing annotation match within a predetermined threshold. In an embodiment, the method involves continuing to train the model if the segmentation of the annotated evaluation image and the existing annotation do not match within a predetermined threshold. For example, the method involves continuing to train the model by modifying the model algorithm and / or adding additional annotated training data. .

[0029] In an embodiment, the method involves deploying or releasing the model for use if the segmentation of the annotated evaluation image and the existing annotation match within a predetermined threshold. In an embodiment, the method involves continuing to train the model if the segmentation of the annotated evaluation image and the existing annotation do not match within a predetermined threshold. For example, the method involves continuing to train the model by modifying the model algorithm and / or adding additional annotated training data. . By way of example, the method involves continuing to train the model by modifying the model algorithm and / or adding additional annotated training data. . By way of example, the method involves continuing to train the model by modifying the model algorithm and / or adding additional annotated training data.

[0030] In an embodiment, the training is (i) annotated images for use in retraining or refining a segmentation machine learning model, and (ii) scores associated with the annotated images, where the scores indicate the degree of success or failure of the segmentation machine learning model in segmenting the annotated images, with a higher score indicating failure and a lower weighting indicating success, and receiving steps, and weighting the annotated images according to the scores when retraining or refining the segmentation machine learning model. In an embodiment, the method includes generating a segmentation machine learning model by refining or modifying an existing segmentation machine learning model.

[0031] In an embodiment, the method includes providing at least one annotated training image from an unannotated image or a partially annotated image, which includes generating one or more candidate image annotations for the unannotated image or the partially annotated image, receiving an input identifying one or more portions of the one or more candidate image annotations, and providing the annotated training image with the at least one or more portions. By way of example, the method includes generating at least one of the candidate image annotations using a) a non-machine learning based image processing method, or b) a segmentation machine learning model.

[0032]

[0033] ​​​​​​​​​​​ In an embodiment, the annotated training data further comprises identification data, and the method includes training a segmentation machine learning model as a segmentation and identification machine learning model.

[0034] In an embodiment, the annotated training data further provides identification annotation, and the method includes training an identification machine learning model using (i) the annotated training data after each image has been segmented by the segmentation machine learning model and (ii) the identification annotation.

[0035] The method may include training a segmentation machine learning model using machine learning (including, for example, support vector machines, random forest trees, deep neural networks, etc.) to determine the classification of each pixel / voxel of the image.

[0036] According to a third aspect, the present invention provides an image annotation system, the system comprising: a) an input for receiving an image (such as a medical image) that has at least one unannotated or partially annotated area, b) an annotator configured to produce respective annotated training images from the image that has at least one unannotated or partially annotated area, and to do so by generating one or more candidate image annotations for the image that has at least one unannotated or partially annotated area, Receiving an input for identifying one or more portions of one or more candidate image annotations, where the portions can be composed of the entire candidate image annotation and generating annotated training images from at least one or more of the portions including steps In an embodiment, the annotator is configured to generate at least one of the candidate image annotations using a) a non-machine learning-based image processing method or b) a segmentation machine learning model (such as those of the type described above).

[0037]

[0038]

[0038] According to a fourth aspect, the present invention provides an image annotation method, the method comprising: a) receiving or accessing an image that has not been annotated or has been partially annotated and b) generating each annotated training image from the image that has not been annotated or has been partially annotated, wherein to do this generating one or more candidate image annotations for an image that has not been annotated receiving an input for identifying one or more portions of the one or more candidate image annotations and generating annotated training images from at least one or more of the portions including steps In an embodiment, the method includes the step of generating at least one of the candidate image annotations using a) a non-machine learning-based image processing method or b) a segmentation machine learning model including steps

[0039] including steps

[0040] According to a fifth aspect, the present invention provides computer program code configured to implement either the second aspect or the fourth aspect when executed by one or more processors. This aspect may also provide a computer-readable medium (which may be non-transitory) comprising the computer program code as described above.

[0041] Thus, certain aspects of the present invention assist in the continuous training and evaluation of machine learning (e.g., deep learning) models with an integrated annotation and segmentation system.

[0042] The annotation system not only simply uses images annotated / identified by humans, but also combines segmented / identified results obtained by the following methods: Image processing algorithms, annotation deep learning models, correction models, and segmentation models. Human input is only used as a final step to verify or correct annotations when required. As more images are annotated, the deep learning model is improved and requires less human intervention. In many cases, it has been found that accurate annotations can be obtained by combining segmented / identified results from different models (such as annotation, correction, and segmentation / identification models).

[0043] For each of the various aspects of the present invention described above, any of the various individual features, as well as the claims Any of the various individual features of the embodiments described in this specification that include the scope of the claim can be combined in suitable and desired manners

Brief Description of the Drawings

[0044] For the purpose of more clearly defining the present invention, exemplary implementations will be described below with reference to the accompanying drawings forms

[0045]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 7C

Figure 8A

Figure 8B

Figure 8C

[0046] FIG. 1A is a schematic diagram of the architecture of a segmentation and identification system 10 with built-in annotation and training functions. The system 10 includes an annotation and training subsystem 12, a segmentation and identification subsystem 14, a trained segmentation and identification module 16, and a user interface 18 including a graphical user interface (GUI) 20. Generally, the annotation and training subsystem 12 is configured to annotate medical images selected as training data with "ground truth" (correct answers) (for targeted anatomical structures and tissues) under the management of an operator with expertise. The annotated training data is input into a machine learning algorithm (such as a deep learning algorithm, etc.) to train a segmentation and identification model (stored in the trained segmentation and identification model 16). The trained segmentation and identification model is used by the segmentation and identification subsystem 14 to segment and identify objects of interest in new medical images. and a trained segmentation and identification module 16, and a user interface 18 including a graphical user interface (GUI) 20. I) 20. including.

[0047] Generally, the annotation and training subsystem 12 is under the management of an operator with expertise (for targeted anatomical structures and tissues) with "ground truth" (correct answer) to annotate medical images selected as training data. configured. The annotated training data is (for example, a deep learning algorithm, etc.) ) input into a machine learning algorithm, (in the trained segmentation and identification model 1 6) to train a segmentation and identification model. The trained segmentation mentation and identification model is used by the segmentation and identification subsystem 14 to segment and identify objects of interest in new medical images. It is used for identification and also includes the classification and labeling of pixels / voxels of an image.

[0048] GUI 20 can be implemented in several different ways. For example, it can be a GUI installed as software on a computing device (such as a personal computer, laptop, tablet computer, or mobile phone, etc.) used by an annotator (i.e., an operator having the appropriate skills to recognize the features of the image). In another example, GUI 20 can be provided as a web page for annotators and can be accessed by a web browser. It can be installed as software on a computing device (such as a personal computer, laptop, tablet computer, or mobile phone, etc.) used by an annotator (i.e., an operator having the appropriate skills to recognize the features of the image). In another example, GUI 20 can be provided as a web page for annotators and can be accessed by a web browser.

[0049] In addition to training (22) the segmentation and identification models using the annotated images, the annotation and training subsystem 12 can evaluate (24) the performance of the trained segmentation and identification model 16 to determine whether the model is ready to be deployed, and can also retrain or improve the model. If the segmentation and identification model fails for one or more specific images, the images for which the model fails are added as a new training dataset to the annotation and training subsystem 12, and the relevant segmentation and identification models are retrained (28).

[0050] However, if the segmentation and identification model fails for one or more specific images, the images for which the model fails are added as a new training dataset to the annotation and training subsystem 12, and the relevant segmentation and identification models are retrained (28). are added to the annotation and training subsystem 12 as a new training dataset, and the relevant segmentation and identification models are retrained (28).

[0051] Figure 1B is a schematic diagram of the system 10. The system 10 includes a segmentation and identification controller 32 and the aforementioned user interface 18. The segmentation and identification controller 32 includes at least one processor 34 (or, in some embodiments, In one state, it includes a plurality of processors) and a memory 36. The system 10 can be implemented, for example, as a combination of software and hardware on a computer such as a personal computer or a mobile computer as a computing device), or can be implemented as a dedicated image segmentation system Optionally, it can be described as being distributed with respect to the system 10; for example, all or some components of the memory 36 can be arranged at a location remote from the processor 34; the user interface 18 can be arranged at a location remote from the memory 36 and / or the processor 34, and in fact it can include a web browser and a mobile device application. For example, all or some components of the memory 36 can be arranged at a location remote from the processor 34; the user interface 18 can be arranged at a location remote from the memory 36 and / or the processor 34, and in fact it can include a web browser and a mobile device application. The memory 36 can communicate data with the processor 34 and typically includes both volatile

[0052] and non-volatile memory (and can include one or more of each memory type), including RAM (Random Access Memory), ROM, and one or more mass storage devices. The memory 36 can communicate data with the processor 34 and typically includes both volatile and non-volatile memory (and can include one or more of each memory type), including RAM (Random Access Memory), ROM, and one or more

[0053] The processor 34 includes an annotation and training subsystem 12 and a segmentation and identification subsystem 14. As will be described in more detail later, the annotation and training subsystem 12 includes an initial segmenter and identifier 38 (processing the original image according to a non-machine learning image processing method), an image annotator 40, a model trainer 42, and a model evaluator 44. The segmentation and identification subsystem 14 includes a pre-processor 46, a segmenter 48, a structure identifier 50, and a result evaluator 52. The processor The processor 34 also includes an I / O interface 54 and a result output 56 .

[0054] The memory 36 stores program code 58, image data 60, training data 62, evaluation images 6 4. 66 correct images, 16 pre-trained segmentation and classification models, 16 pre-trained annotations The model includes a trained refinement model 69, a training model 68, and a trained correction model 69.

[0055] The segmentation and discrimination controller 32 is configured to receive, at least in part, The program code 58 is implemented by the processor 34 executing the program code 58 from the processor 34 .

[0056] Generally speaking, the I / O interface 54 receives image data relating to a subject or patient. (e.g., DICOM format) into image data 60 or training data 62. This is configured to be used for analysis and / or training (or both, as described below). Once the system 10 converts the structures into image data, Once the segments are identified from 60, the I / O interface 54 outputs the results of the analysis ( Optionally in the form of a report, for example via the results output 56 and / or GUI 20. .

[0057] FIG. 2 is a flow diagram 20 of the general workflow of the system 10. In S72, A set of images (typically stored remotely or locally) The training images / data shall be imported from the database. These images are input into the system 10 via the interface 54 and used as training data. The training data 62 is stored in a database 62. The training data 62 is a set of images that are expected to be encountered in a clinical environment. should represent the image and include both the basic cases and the edge cases (i.e., each of the "normal" and "extreme" cases). The structure and organization in the basic cases are more easily segmentable or distinguishable than those in the edge cases. For example, for a wrist CT scan, it is necessary to segment the bone from the surrounding muscles and fat. This task is easier in the scans of young and healthy subjects. Because regarding these, the porosity of the bone boundary is lower and clearer than that of the elderly and frail patients, and regarding the latter, their bone is very porous and the clarity of the boundary is lower. However, it is desirable to collect examples for both the basic cases and the edge cases as training data.

[0058] In this step, a second image set for evaluating the performance of the trained model is also imported. These evaluation images for use in the evaluation should represent clinical images and are stored within the evaluation images 64.

[0059] In S74, an operator with appropriate expertise annotates both the training data 62 and the evaluation images 64 using the image annotator 40. If the use is different, the required annotation also changes. For example, if the model is to be trained to segment bone material from non-bone material, what is required in the annotation is to distinguish and identify the pixels / voxels of the bone from those of the non-bone and thus segmentation data should be included. On the other hand, if the model is further advanced and is required to identify each bone mass, what is required in the annotation is to distinguish and label regarding the pixels / voxels of each bone mass To attach a mark, additional identification data should be included accordingly.

[0060] In S76, the model trainer 42 trains the classifier model using the annotated training data 62. In this embodiment, it is a segmentation and identification model, which is a classifier that determines the classification of each pixel / voxel on the image. In the training of the model, the purpose is to determine the decision pattern from the input (training image) to the correct answer (annotation). The model can be trained by using machine learning algorithms such as support vector machines and random forest trees. For example, it can be trained by utilizing machine learning algorithms such as support vector machines and random forest trees.

[0061] In this embodiment, a deep neural network is used. As will be described later (see Fig. 6), this deep neural network consists of an input layer, an output layer, and various layers in between. Each layer consists of artificial neurons. An artificial neuron is a mathematical function that receives one or more inputs, sums them up, and produces an output. Usually, each input is individually weighted, and the sum passes through a non-linear function. As the neural network learns, the weight values of the model are adjusted, and the adjustment is made according to the resulting error (the difference between the network output and the annotation), and the adjustment is made until the error can no longer be reduced.

[0062] In S78, the model evaluator 44 controls the segmentation and identification subsystem 14 to evaluate the trained model, and uses the model to perform segmentation on the evaluation image. Perform augmentation and identification. In S80, the model evaluator 44 checks whether the trained model has reached the passing score, and does this by comparing the result of that process with the annotation used in S74. In this embodiment, this is done by determining whether the difference between the result generated by the trained model and the annotation data associated with the annotated image (ground truth image) 66 used in S74 is less than a predetermined threshold. For example, the difference can be calculated as the overlap between the segmentation generated by the model and the segmentation suggested by the annotation data, and the ratio to the segmentation suggested by the annotation data. If these match exactly, the overlap is clearly 100 %, and implicitly results in a passing result. In some applications, this threshold is

[0063] set to 90%. If in S80 the model evaluator 44 determines that the model has failed, the process proceeds to S82, and one or more new images are imported in the same way as the original training data was imported in S72 (to supplement the original training data), and / or the learning algorithm is adjusted / changed (for example, by tuning the parameters of the neural network, making changes to the layers of the neural network, or making changes to the activation function of the neurons of the neural network, etc.). Then the process returns to S74.

[0064] If in S80 the model evaluator 44 determines that the result generated by the If it is determined that the difference from the obtained annotation is less than a predetermined threshold, that is, training If it is determined that the trained model is qualified, the process proceeds to S84, and the trained model is (as one of the trained segmentation and identification models 16) stored and deployed. The training phase is (at least at this stage) completed, so in S85, the trained model is used by the segmentation and identification subsystem 14 to segment and identify one or more structures or materials / tissues in the new medical image input and stored in the image data 60. In S86, the result evaluator 52 verifies the result of this segmentation and identification by comparing the result with one or more predefined conditions or parameters known to characterize the targeted structure or material. If it is shown in this verification that the performance of the segmentation and identification model is unqualified, the process proceeds to S88 and the new image is added to the training set as new training data, and the process (since these images are used for annotation or retraining the model, etc.) proceeds to S74. If the result evaluator 52 does not determine in S86 that the result is unqualified, the process proceeds to S90 and the segmentation / identification result from the new medical image is output, such as being displayed by the user interface 18 of the system 10. (Optionally in S86, if the result evaluator 52 does not determine that the result is unqualified, the segmentation and identification results can be presented to the user so that the user can perform complementary manual verification. And (for example, by selecting the "unqualified" or "rejected" button on the GUI 20) the user can conclude

[0065] If the result evaluator 52 does not judge in S86 that the result is unqualified, the process proceeds to S90 and the segmentation / identification result from the new medical image is output, such as being displayed by the user interface 18 of the system 10. (Optionally in S86, if the result evaluator 52 does not determine that the result is unqualified, the segmentation and identification results can be presented to the user so that the user can perform complementary manual verification. And (for example, by selecting the "unqualified" or "rejected" button on the GUI 20) the user can conclude ​​If the fruit is flagged as non - compliant, the result is regarded as non - compliant and the process proceeds to S88. The user does not object to the decision of the user evaluator 52 by selecting the "Compliant" or "Accept" button on the GUI20, the process proceeds to S90.

[0066] After S90, the process ends. For example, the result can be used for diagnostic purposes or as input for further qualitative analysis.

[0067] Figure 3 is a pseudo - flowchart 70 of the operation of the system 10. As described above, the annotation and training subsystem 12 is configured to perform annotation on the segmentation and identification made on medical images. The medical images and their annotations are used to train the segmentation and identification models. The annotator performs annotation on the medical image set via the GUI20, which may also be referred to as the annotation interface in this specification.

[0068] The annotator combines information from different resources to complete the annotation. In this embodiment, the annotator combines information from one or more candidate image annotations and, if necessary, uses the annotation tools of the GUI20 to complete the annotation. In this embodiment, the candidate image annotations can be in the following form: Preliminary segmentation and identification results generated using existing non - machine - learning image - processing methods; or results generated by a partially trained segmentation and identification model. The candidate image annotations are displayed together (i.e., adjacent to each other), and the anno The data is made such that it is easy to compare which part of each is the best.

[0069] Also, since the preliminary segmentation results are generated using existing image processing methods, you, gentlemen, will realize that the preliminary results typically require improvement. Examples of existing image processing methods include contour detection, blob detection, and threshold-based object segmentation. Although these methods can segment pixels corresponding to bones in CT scans earlier than surrounding pixels, in many cases they result in a rough or approximate preliminary segmentation.

[0070] The GUI 20 includes a preliminary result window 102 and an annotated image window 104. The annotated image window 104 includes a plurality of annotation tools 106. The annotation tools 106 include both manual and semi-manual annotation tools, which can be displayed to and operated by an annotator. These tools are schematically shown in FIG. 4. These tools include manual tools, which include a pen tool 130 and an eraser tool 132. The pen tool 130 can be controlled by an annotator to label pixels of different structures or materials with different values or colors. The eraser tool 132 can be controlled by an annotator to remove over-segmented or over-identified pixels from the target structure or material. The semi-manual annotation tool includes a filling tool 134, which can be controlled by an annotator to draw a contour surrounding the target structure or material. Annotations can be applied to all pixels / voxels within the contour of W104 In addition, the semi-manual annotation tool includes a region expansion tool 136. After annotating a very small part of the target structure or material, the annotator can control this to expand the annotation by controlling the annotated image window 104 to include the entire structure or material.

[0071] System 10 also includes machine learning-based tools, which assist the annotator in performing accurate segmentation and identification quickly. For example, annotation tool 1 06 includes a machine learning-based annotation model control 138, which calls the trained annotation model stored in the annotation model 68. The annotation model 68 is controlled by the annotator to select a plurality of points for use by the annotation model 68 when segmenting an object. In this embodiment, the annotation model 68 prompts the annotator to identify four extreme points of the object, that is, the four extreme points are the leftmost, rightmost, uppermost, and lowermost pixels. Then, the annotator selects each point sequentially, for example, by touching the image with a stylus (when displayed on a touch screen) or by using a mouse. Then, the annotation model 68 segments the object using these extreme points. In this way, the selected points (e.g., extreme points) are used for annotation 6 ​Configure the annotation data 118 used to train 8.

[0072] The annotation tool 106 includes an annotation motion capture tool 140 and is another mechanism for recording the annotation data 118, and can also be activated by the annotator. Movements are recorded by the system 10 to train the modified model (stored in the trained modified model 69). For example, the annotator can move the over-segmentation contour inward or the under-segmentation contour outward. The inward or outward movement and the positions at which they are made are recorded as input for training the modified model (stored in the trained modified model 69) by the annotation motion capture tool 140. Also, the annotation motion capture tool 140 records the amount of labor, such as the number of mouse operations and the amount of mouse scrolling, and is used to

[0073] Returning to Figure 3, during use, the preliminary segmentation and identification results 110 will be presented to the annotator via the preliminary result window 102 of the GUI 20. If the annotator is satisfied with the accuracy of the preliminary segmentation and identification results 110, the annotator can move the preliminary results 110 to the annotated image window 104 of the GUI 20, for example, by clicking the mouse on the preliminary result window 102. If the annotator is satisfied with only (not all of) a part of the preliminary Move and use the annotation tool 106 of the annotated image window 104 to correct / complete segmentation and identification. 6.

[0074] If the annotator is not satisfied with any part of the preliminary segmentation and identification results, the annotator can perform initial annotation on the image using the annotation tool 106.

[0075] If the annotator has completed the annotation of the partially sufficient preliminary results 110 or has performed annotation on the image himself / herself, the annotated original image set 114 is within the annotated image window 104 and is stored as the correct image 66.

[0076] After one or more correct images 66 are collected in this way, the segmentation and identification model is trained using the annotated original image, i.e., the correct image 66. After the trained segmentation and identification model becomes available (within the trained segmentation and identification model 68), the segmentation and identification generated using the trained model are also provided to the annotator for reference. If the annotator is satisfied with the results generated by the trained segmentation and identification model or a part of such results, the annotator moves the sufficient part to the annotated image window 104. The annotator can combine the sufficient part of the preliminary results with the sufficient part from the trained model within the annotated image window 104. If there are still images or image parts with insufficient annotation, the annotator can use the annotation tool Annotations can be corrected or supplemented using Rule 106.

[0077] The collected ground truth images and original images 60 within the ground truth image 66 are used to train the segmentation and identification model. To evaluate the trained model, a different set of ground truth images and original images is used. In some embodiments, the same images are used for both training and evaluation. When the performance of the trained model is satisfied during image evaluation, the model is sent to the trained segmentation and identification model 68 and is used by the segmentation and identification subsystem 14 to process some new image. If the performance is insufficient, more ground truth images are collected. The criteria used by the system 10 when evaluating the model (i.e., when determining whether the segmentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. and are used by the segmentation and identification model for training. To evaluate the trained model, a different set of ground truth images and original images is used. In some embodiments, the same images are used for both training and evaluation. When the performance of the trained model is satisfied during image evaluation, the model is sent to the trained segmentation and identification model 68 and is used by the segmentation and identification subsystem 14 to process some new image. If the performance is insufficient, more ground truth images are collected. The criteria used by the system 10 when evaluating the model (i.e., when determining whether the segmentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. and original images are used. In some embodiments, the same images are used for both training and evaluation. When the performance of the trained model is satisfied during image evaluation, the model is sent to the trained segmentation and identification model 68 and is used by the segmentation and identification subsystem 14 to process some new image. If the performance is insufficient, more ground truth images are collected. The criteria used by the system 10 when evaluating the model (i.e., when determining whether the segmentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. is satisfied, the model is sent to the trained segmentation and identification model 68 and is used by the segmentation and identification subsystem 14 to process some new image. If the performance is insufficient, more ground truth images are collected. The criteria used by the system 10 when evaluating the model (i.e., when determining whether the segmentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. segmentation and identification model 68 and is used by the segmentation and identification subsystem 14 to process some new image. If the performance is insufficient, more ground truth images are collected. The criteria used by the system 10 when evaluating the model (i.e., when determining whether the segmentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. are used by the segmentation and identification subsystem 14 to process some new image. If the performance is insufficient, more ground truth images are collected. The criteria used by the system 10 when evaluating the model (i.e., when determining whether the segmentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. is insufficient, more ground truth images are collected. The criteria used by the system 10 when evaluating the model (i.e., when determining whether the segmentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. mentation and identification model passes) are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. are adjustable. Such adjustments are usually made by the developer of the system 10 or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. or the person who first trained the model and are adjusted according to the requirements of the application. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone. For example, when determining vBMD (volumetric bone mineral density), the criteria are set to verify whether the entire bone is accurately segmented from the surrounding material; when calculating cortical porosity, the criteria are set to verify the segmentation accuracy of the entire bone and cortical bone.

[0078] FIG. 5 is a flowchart 150 for annotation and training workflows. Referring to FIG. 5, at S152, the original image 60 is selected or input, and at S154, the original image is processed by the initial segmenter and identifier 38, thereby obtaining a preliminary segmentation. Referring to FIG. 5, at S152, the original image 60 is selected or input, and at S154, the original image is processed by the initial segmenter and identifier 38, thereby obtaining a preliminary segmentation. is obtained by processing the original image with the initial segmenter and identifier 38, thereby obtaining a preliminary segmentation. Generate Yon and identification results. The process proceeds to S156, and the segmenter 48 processes the original image 60 using the trained segmentation and identification model 16 if the trained segmentation and identification model is available within the trained segmentation and identification model 16. Then, the process proceeds to S160. If the system 10 determines at S156 that the trained segmentation and identification model is not available, the process proceeds to S160. Determine whether the trained segmentation and identification model is available within the trained segmentation and identification model 16. If so, the process proceeds to S158, and the original image 60 is also processed using the trained segmentation and identification model 16. Then, the process proceeds to S160. If the system 10 determines at S156 that the trained segmentation and identification model is not available, the process proceeds to S160. If the system 10 determines at S156 that the trained segmentation and identification model is not available, the process proceeds to S160. If the system 10 determines at S156 that the trained segmentation and identification model is not available, the process proceeds to S160. At S156, if the system 10 determines that the trained segmentation and identification model is not available, the process proceeds to S160. If the system 10 determines at S156 that the trained segmentation and identification model is not available, the process proceeds to S160.

[0079] At S160, the annotator verifies whether any part of the preliminary model result and the trained model generation result is qualified (usually by inspecting these results on the display of the user interface 18). If so, the process proceeds to S162, and the annotator moves the qualified part to the annotated image window 104 using the annotation tool 106, and the process proceeds to S164. If the annotator determines at S160 that none of the obtained results is qualified, the process proceeds to S164. At S160, the annotator verifies whether any part of the preliminary model result and the trained model generation result is qualified (usually by inspecting these results on the display of the user interface 18). If so, the process proceeds to S162, and the annotator moves the qualified part to the annotated image window 104 using the annotation tool 106, and the process proceeds to S164. If the annotator determines at S160 that none of the obtained results is qualified, the process proceeds to S164. At S160, the annotator verifies whether any part of the preliminary model result and the trained model generation result is qualified (usually by inspecting these results on the display of the user interface 18). If so, the process proceeds to S162, and the annotator moves the qualified part to the annotated image window 104 using the annotation tool 106, and the process proceeds to S164. If the annotator determines at S160 that none of the obtained results is qualified, the process proceeds to S164. At S160, if the annotator determines that any part of the obtained results is qualified, the process proceeds to S162, and the annotator moves the qualified part to the annotated image window 104 using the annotation tool 106, and the process proceeds to S164. If the annotator determines at S160 that none of the obtained results is qualified, the process proceeds to S164. At S160, if the annotator determines that any part of the obtained results is qualified, the process proceeds to S162, and the annotator moves the qualified part to the annotated image window 104 using the annotation tool 106, and the process proceeds to S164. If the annotator determines at S160 that none of the obtained results is qualified, the process proceeds to S164. At S160, if the annotator determines that none of the obtained results is qualified, the process proceeds to S164. At S160, if the annotator determines that none of the obtained results is qualified, the process proceeds to S164.

[0080] At S164, if none of the obtained results is qualified, the annotator is prompted to annotate the image ab initio or to complete, correct, or supplement the annotation in some way. The annotator does this by controlling the manual and semi-manual tools 106 until the annotation is satisfactory. Optionally, the annotator may also use the correction model control 142 (if the correction model 69 is available). At S164, if none of the obtained results is qualified, the annotator is prompted to annotate the image ab initio or to complete, correct, or supplement the annotation in some way. The annotator does this by controlling the manual and semi-manual tools 106 until the annotation is satisfactory. Optionally, the annotator may also use the correction model control 142 (if the correction model 69 is available). At S164, if none of the obtained results is qualified, the annotator is prompted to annotate the image ab initio or to complete, correct, or supplement the annotation in some way. The annotator does this by controlling the manual and semi-manual tools 106 until the annotation is satisfactory. Optionally, the annotator may also use the correction model control 142 (if the correction model 69 is available). At S164, if none of the obtained results is qualified, the annotator is prompted to annotate the image ab initio or to complete, correct, or supplement the annotation in some way. The annotator does this by controlling the manual and semi-manual tools 106 until the annotation is satisfactory. Optionally, the annotator may also use the correction model control 142 (if the correction model 69 is available). At S164, if none of the obtained results is qualified, the annotator is prompted to annotate the image ab initio or to complete, correct, or supplement the annotation in some way. The annotator does this by controlling the manual and semi-manual tools 106 until the annotation is satisfactory. Optionally, the annotator may also use the correction model control 142 (if the correction model 69 is available). If so, the correction model 69 can also be called to improve the annotation.

[0081] In S166, the segmentation and identification models are trained using the correct image and annotation data. In parallel, in S168, the annotation model 68 and the correction model 69 are trained using the correct image and annotation data. (If there is no available correction model in S164, note that after the correction model is generated in S168, the correction model will be available within the correction model 69 in a later pass up to S164.) In S166, the segmentation and identification models are trained using the correct image and annotation data. In parallel, in S168, the annotation model 68 and the correction model 69 are trained using the correct image and annotation data. (If there is no available correction model in S164, note that after the correction model is generated in S168, the correction model will be available within the correction model 69 in a later pass up to S164.) In S166, the segmentation and identification models are trained using the correct image and annotation data. In parallel, in S168, the annotation model 68 and the correction model 69 are trained using the correct image and annotation data. (If there is no available correction model in S164, note that after the correction model is generated in S168, the correction model will be available within the correction model 69 in a later pass up to S164.) If there is no available correction model in S164, note that after the correction model is generated in S168, the correction model will be available within the correction model 69 in a later pass up to S164. If there is no available correction model in S164, note that after the correction model is generated in S168, the correction model will be available within the correction model 69 in a later pass up to S164. If there is no available correction model in S164, note that after the correction model is generated in S168, the correction model will be available within the correction model 69 in a later pass up to S164.

[0082] The process proceeds to S170, where the trained segmentation and identification models are evaluated by the annotation and model evaluator 44 of the training subsystem 12. In S172, the result of the evaluation is verified. If it fails, the process proceeds to S174, where the trained segmentation and identification models are saved to the trained segmentation and identification model 68 and deployed for use in processing the new image 60. Otherwise, the process returns to S152, where one or more additional correct images 66 are collected or selected and the process is repeated. The process proceeds to S170, where the trained segmentation and identification models are evaluated by the annotation and model evaluator 44 of the training subsystem 12. In S172, the result of the evaluation is verified. If it fails, the process proceeds to S174, where the trained segmentation and identification models are saved to the trained segmentation and identification model 68 and deployed for use in processing the new image 60. Otherwise, the process returns to S152, where one or more additional correct images 66 are collected or selected and the process is repeated. The process proceeds to S170, where the trained segmentation and identification models are evaluated by the annotation and model evaluator 44 of the training subsystem 12. In S172, the result of the evaluation is verified. If it fails, the process proceeds to S174, where the trained segmentation and identification models are saved to the trained segmentation and identification model 68 and deployed for use in processing the new image 60. Otherwise, the process returns to S152, where one or more additional correct images 66 are collected or selected and the process is repeated. The process proceeds to S170, where the trained segmentation and identification models are evaluated by the annotation and model evaluator 44 of the training subsystem 12. In S172, the result of the evaluation is verified. If it fails, the process proceeds to S174, where the trained segmentation and identification models are saved to the trained segmentation and identification model 68 and deployed for use in processing the new image 60. Otherwise, the process returns to S152, where one or more additional correct images 66 are collected or selected and the process is repeated. The process proceeds to S170, where the trained segmentation and identification models are evaluated by the annotation and model evaluator 44 of the training subsystem 12. In S172, the result of the evaluation is verified. If it fails, the process proceeds to S174, where the trained segmentation and identification models are saved to the trained segmentation and identification model 68 and deployed for use in processing the new image 60. Otherwise, the process returns to S152, where one or more additional correct images 66 are collected or selected and the process is repeated. The process proceeds to S170, where the trained segmentation and identification models are evaluated by the annotation and model evaluator 44 of the training subsystem 12. In S172, the result of the evaluation is verified. If it fails, the process proceeds to S174, where the trained segmentation and identification models are saved to the trained segmentation and identification model 68 and deployed for use in processing the new image 60. Otherwise, the process returns to S152, where one or more additional correct images 66 are collected or selected and the process is repeated.

[0083] The new image is processed by the segmentation and identification subsystem 14, which segments and identifies the target structure or material (e.g., tissue). As described above, the segmentation and identification subsystem 14 comprises four modules: a pre-processor 46, a segmenter 48, a structure identifier 50, and a result evaluator 52. The pre-processor 46 verifies the validity of the input image (a medical image in this embodiment), The new image is processed by the segmentation and identification subsystem 14, which segments and identifies the target structure or material (e.g., tissue). As described above, the segmentation and identification subsystem 14 comprises four modules: a pre-processor 46, a segmenter 48, a structure identifier 50, and a result evaluator 52. The pre-processor 46 verifies the validity of the input image (a medical image in this embodiment), The new image is processed by the segmentation and identification subsystem 14, which segments and identifies the target structure or material (e.g., tissue). As described above, the segmentation and identification subsystem 14 comprises four modules: a pre-processor 46, a segmenter 48, a structure identifier 50, and a result evaluator 52. The pre-processor 46 verifies the validity of the input image (a medical image in this embodiment), The new image is processed by the segmentation and identification subsystem 14, which segments and identifies the target structure or material (e.g., tissue). As described above, the segmentation and identification subsystem 14 comprises four modules: a pre-processor 46, a segmenter 48, a structure identifier 50, and a result evaluator 52. The pre-processor 46 verifies the validity of the input image (a medical image in this embodiment), The new image is processed by the segmentation and identification subsystem 14, which segments and identifies the target structure or material (e.g., tissue). As described above, the segmentation and identification subsystem 14 comprises four modules: a pre-processor 46, a segmenter 48, a structure identifier 50, and a result evaluator 52. The pre-processor 46 verifies the validity of the input image (a medical image in this embodiment), Verify including whether the information of the medical image is complete and whether the image is damaged. Further, the pre-processor 46 even reads the medical image into the image data 60 within the system 10. Since the images can be of different formats, the pre-processor 46 is configured to be able to read images in a format having a main relevance, which includes DICOM. Also, since there may be a plurality of trained segmentation and identification models 16 available, the pre-processor 46 is configured to determine which of the trained segmentation and identification models 16 should be used for performing segmentation and identification based on the information extracted by the pre-processor 46 from the image. For example, in one scenario, two classifier models can be trained, the first classifier model is for segmenting and identifying the radius from a wrist CT scan, and the second classifier model can be for segmenting and identifying the tibia from a leg CT scan. When processing a new DICOM format CT scan, the pre-processor 46 extracts the scanned body part position information from the DICOM header and determines, for example, that the radius classifier model should be used.

[0084] The segmenter 48 and the structure identifier 50 each segment and identify the target structure or material in the image using the model selected from the trained segmentation and identification model 68. And the results of the segmentation and identification are automatically evaluated by the result evaluator 52. In this embodiment, the evaluation performed by the result evaluator 52 is ​​​​​​​​​accompanied by the following: one or more predefined conditions or parameters of the target structure or material (e.g., an acceptable range of one or more structures, or dimensions of the material, or volume of the structure or material etc.), verify the results in relation to the target, and determine whether the segmentation and identification results are clearly unsatisfactory or not. The result evaluator 50 outputs a result indicating whether the segmentation and identification results are actually sufficient.

[0085] The automatic evaluation performed by the result evaluator 52 can be enhanced manually. For example, the results of segmentation and identification and / or the evaluation of the result evaluator 52 can be displayed to a user (e.g., a doctor) for further evaluation, so that the automatic evaluation can be refined.

[0086] Images for which the segmentation and identification results are rejected are used as additional training images for retraining the segmentation and identification model. When the result evaluator 52 approves the segmentation and identification results, the segmentation and identification results are output to the result output 56 and / or the user interface 18 via the I / O interface 54 (e.g., on the display of a computer or mobile device). In some embodiments, the segmentation and identification results are used as input for further quantitative analysis. For example, if the radius is segmented and identified by the system 10 from a wrist CT scan, attributes such as the volume and density of the extracted radius can be passed to another application (executed locally or remotely) for further analysis.

[0087] Convolutional neural network for segmentation and identification according to an embodiment of the present invention The work is generally shown at reference numeral 180 in FIG. 6 and is in the form of a deep learning segmentation and identification model. The deep learning segmentation and identification model 18 0 includes a convolutional neural network (CNN) 182 and an extended convolutional neural network (CN N) 184. The reduced or downsampling CNN 182 processes the input image to generate a feature map that reduces their resolution through various layers. The extended or upsam pling CNN 184 processes these features through layers that increase the resolution and eventually generates a segmentation and identification mask.

[0088] In the example of FIG. 6, the reduced CNN 182 has four layers 186, 188, 190, 192 and the extended CNN has four layers 194, 196, 198, 192, but it should be noted that other numerical values can be used for the number of layers in each case. The lowest layer (i.e., layer 192) is shared by the reduced and extended networks 182, 184. Each layer 186,..., 198 has an input and a feature map (shown as a hollow box in each layer). The input of each layer 186,..., 198 is processed by a convolutional process and converted into a feature map (illustrated by a hollow arrow in the figure). The convolutional process or activation function can be, for example, a rectified linear unit (ReLU), a sigmoid unit, or a Tanh unit. (ReLU), a sigmoid unit, or a Tanh unit. (ReLU, rectified linear unit), a sigmoid unit, or a Tanh unit. and so on.

[0089] ​In the downsampling CNN 182, the input of the first layer 186 is the input image 200. The downsampling CNN 182, the feature maps on each layer are downsampled to another feature map, and are used as the inputs 202, 204, 206 of their respective next lower layers . The downsampling process 208 can be, for example, a max pooling operation or a stride operation .

[0090] In the upsampling CNN 184, the feature maps 208 , 201, 212 on each layer 192, 198, 196 are upsampled to their respective another feature maps 214, 216, 218 and are used within the next layers 198, 196, 194. The upsampling process can be, for example, an upsample convolution operation, a transposed convolution, or a bilinear upsampling .

[0091] To capture the localization pattern, the high-resolution feature maps 220, 222, 224 in the corresponding layers 190, 188, 186 within the downsampling CNN 182 are each cropped and concatenated to the upsampled feature maps 214, 216, 218 (illustrated as dashed arrows in the figure), and the resulting high-resolution feature map / feature map pairs 220 / 214, 222 / 216, 224 / 218 are used as the inputs of their respective layers 198, 196, 194. In the final layer 194 of the upsampling CNN 184, the feature map 226 resulting from the convolution process is transformed and output as the final segmentation and identification map 228, and this transformation is illustrated as a solid arrow in the figure. The transformation ​​​​​​​​​​​ For example, it can be implemented as a convolution operation or sampling.

[0092] In this way, the deep learning segmentation and identification model 180 of FIG. 6 combines the location information from the contraction path with the contextual information in the expansion path, ultimately obtaining general information that combines localisation and context. from the contraction path with the contextual information in the expansion path to ultimately obtain general information that combines localisation and context. context

[0093] Exemplary implementations related to segmentation and identification within the system 10 according to embodiments of the present invention are shown in FIGS. 7A to 7C. Referring to FIG. 7A, an implementation example 240 related to the first segmentation and identification is characterised as an example of "one step". The first segmentation and identification 240 includes an annotation and training phase 242 and a segmentation and identification phase 244. In the former, the original image 246 is subjected to annotation 248 for segmentation and identification, and after training is performed, a segmentation and identification model 250 is generated. In the segmentation and identification phase 244, a new image 252 is subjected to a segmentation and identification act 254 according to the segmentation and identification model 250, and then a segmentation and identification result 256 is output. Referring to FIG. 7A, an implementation example 240 related to the first segmentation and identification is characterised as an example of "one step". The first segmentation and identification 240 includes an annotation and training phase 242 and a segmentation and identification phase 244. In the former, the original image 246 is subjected to annotation 248 for segmentation and identification, and after training is performed, a segmentation and identification model 250 is generated. In the former, the original image 246 is subjected to annotation 248 for segmentation and identification, and after training is performed, a segmentation and identification model 250 is generated. In the former, the original image 246 is subjected to annotation 248 for segmentation and identification, and after training is performed, a segmentation and identification model 250 is generated. In the former, the original image 246 is subjected to annotation 248 for segmentation and identification, and after training is performed, a segmentation and identification model 250 is generated. In the segmentation and identification phase 244, a new image 252 is subjected to a segmentation and identification act 254 according to the segmentation and identification model 250, and then a segmentation and identification result 256 is output. In the segmentation and identification phase 244, a new image 252 is subjected to a segmentation and identification act 254 according to the segmentation and identification model 250, and then a segmentation and identification result 256 is output. In the segmentation and identification phase 244, a new image 252 is subjected to a segmentation and identification act 254 according to the segmentation and identification model 250, and then a segmentation and identification result 256 is output.

[0094] Therefore, in the embodiment of FIG. 7A, the model 250 is trained for segmentation and Perform identification simultaneously. For example, when segmenting and training the identification model for the radius and fibula of a wrist CT scan, annotations are applied to the training images, and different values are assigned to the voxels of the radius and fibula at that time. Once the model is ready, any arbitrary new wrist CT scan 252 is processed by the segmentation and identification model 250, and the result 256 is provided as a map for the radius and fibula in the form after segmentation and identification. A second segmentation and identification implementation example 260 is schematically shown in FIG. 7B. In implementation example 260, segmentation and identification are implemented in two steps. Referring to FIG. 7B,

[0095] implementation example 260 includes an annotation and training phase 262 and a segmentation and identification phase 264. However, the annotation and training phase 262 includes training of the segmentation model and training of the identification model as separate processes. Thus, annotations 268 are applied to the original image 266 and the segmentation model 270 is trained. Annotations 274 are applied to the segmented image 272 and the identification model 276 is trained. After the two models 270, 276 are trained, any arbitrary new image 264 is first segmented by the segmentation model 270, and the resulting segmented image 282 is then used for identification by the identification model 276, and the identified result 286 is output. For example, by these two steps and in these two steps, all the bones in the wrist CT scan can be first segmented from the surrounding material, and for different types of bones, the segmented image 272 is then used for identification by the identification model 276, and the identified result 286 is output. For example, by these two steps and in these two steps, all the bones in the wrist CT scan can be first segmented from the surrounding material, and for different types of bones, the segmented image 272 is then used for identification by the identification model 276, and the identified result 286 is output. For example, by these two steps and in these two steps, all the bones in the wrist CT scan can be first segmented from the surrounding material, and for different types of bones, any arbitrary new image 264 is first segmented by the segmentation model 270, and the resulting segmented image 282 is then used for identification by the identification model 276, and the identified result 286 is output. For example, by these two steps and in these two steps, all the bones in the wrist CT scan can be first segmented from the surrounding material, and for different types of bones, the segmented image 282 is then used for identification by the identification model 276, and the identified result 286 is output. For example, by these two steps and in these two steps, all the bones in the wrist CT scan can be first segmented from the surrounding material, and for different types of bones, by these two steps and in these two steps, all the bones in the wrist CT scan can be first segmented from the surrounding material, and for different types of bones, all the bones in the wrist CT scan can be first segmented from the surrounding material, and for different types of bones, can be differentiated and identified.

[0096] A third segmentation and identification implementation example 290 is schematically shown in FIG. 7C. In this implementation example, the segmentation model achieves very accurate results, and moreover, the identification can be performed using a heuristic-based algorithm instead of a machine learning based model. Referring to FIG. 7C, the implementation example 290 includes an annotation and training phase 2 92 and a segmentation and identification phase 294. The original image 266 is used in the annotation and training phase 292 to perform annotation 268 for segmentation and to train the segmentation model 270. In the segmentation and identification phase 264, a new image 278 is segmented 280 using the segmentation model 270 to obtain a segmented image 282, and then identification 284 is performed using a heuristic-based algorithm, and the identified result 286 is output. For example, once accurately segmented 280 using the segmentation model 2 70, a wrist CT scan can identify different bones, for example, by volume calculation and differentiation. For example, in a wrist HRpQCT scan, after bone segmentation, bone volume calculation and differentiation can be used to identify different bones. 270 to obtain a segmented image 282, and then identification 284 is performed using a heuristic-based algorithm, and the identified result 286 is output. For example, once accurately segmented 280 using the segmentation model 2 70, a wrist CT scan can identify different bones, for example, by volume calculation and differentiation. 70, once accurately segmented 280, a wrist CT scan can identify different bones, for example, by volume calculation and differentiation. by volume calculation and differentiation. can identify different bones.

[0097] For example, in a wrist HRpQCT scan, after bone segmentation, bone volume By calculating and comparing the products, the ulna can be easily distinguished and identified from the radius. Why? Because the radius is larger than the ulna of the same subject.

[0098] Three exemplary deployments of system 10 are schematically shown in FIGS. 8A-8C and each is as follows: on-premises deployment; cloud deployment; and hybrid deployment . As shown in FIG. 8A, in the first deployment example 300, system 10 is deployed locally (although it may be in a distributed manner). Any data processing (such as model training and image segmentation and identification, etc.) is performed by one or more processors 302 of system 10. The data is encrypted and stored in one or more data storage units 304 . The user interacts with system 10 (such as annotating an image or inspecting the results) via user interface 18 .

[0099] As shown in FIG. 8B, in the second deployment example 310, system 10 is deployed within an encrypted cloud 312, except for user interface 18. User interface 18 is located locally. The communication 314 between user interface 18 and encrypted cloud 312 is encrypted.

[0100] As shown in FIG. 8C, in the third deployment example 320, system 10 is partially deployed within an encrypted cloud service 322, while on the other hand, a local part 324 with data storage 326 and some processing capabilities (such as a processor) 328 and user interface 18 is located locally. In such a deployment example, usually, (the user ... ... Although most of the system 10 (excluding the interface 18) is deployed within the cloud service 322, the data storage unit 326 and the processing power holding unit 328 are sufficient to support the user interface 18 and the communication 330 between the local unit 324 and the cloud service 322. The communication 330 between the local unit 324 and the cloud service 322 is encrypted. Those skilled in the art should realize that many changes can be made without departing from the scope of the present invention, and in particular, it should be noted that additional embodiments can be brought about by using specific features of the embodiments of the present invention. If there were references to the prior art in this specification, it should be noted that such references do not form an admission that the prior art forms part of the well-known art in any country. In the appended claims and the foregoing detailed description of the invention, the terms "comprise", "comprises" (third-person singular present tense), "comprising", etc. are used in an inclusive sense (i.e., in the sense of specifying the presence of the recited features), but do not preclude the existence or addition of further features in various embodiments of the present invention, unless the context otherwise requires a different interpretation explicitly or by necessary implication.

[0101]

[0102]

[0103]

Explanation of Reference Numerals

[0104] 10 System 12 Annotation and Training Subsystem 14 Segmentation and Identification Subsystem 16 Trained Segmentation and Identification Model ​​​​​​​​​​​​18 User Interface 20 GUI 32 Segmentation and Identification System 34 Processor 36 Memory 66 Correct Image 106 Annotation Tool 118 Annotation Data 182 Reduction Network 184 Expansion Network 242 Annotation and Training 244 Segmentation and Identification 262 Annotation and Training 264 Segmentation and Identification 292 Annotation and Training 294 Segmentation and Identification 302 Processor 304 Data Storage Unit 312 Encryption Cloud Service 322 Encryption Cloud Service 324 Local Portion 326 Data Storage Unit 328 Processor

Claims

1. 1. An image segmentation system, comprising: Annotation database with an image associated with each segmentation annotation. Train a segmentation machine learning model using pre-trained training data to generate pre-trained segmentation data. a training subsystem configured to generate an annotation machine learning model; A model evaluator; The trained segmentation machine learning model is used to identify structures or materials in the image. and a segmentation subsystem configured to perform segmentation using the 、 The model evaluator performs an evaluation on the segmentation machine learning model by: (i) controlling the segmentation subsystem to generate the segmentation machine Using a learning model, we can generate at least one segmentation annotation that is associated with the existing segmentation annotations. Segment another evaluation image, thereby obtaining an annotated evaluation image. generating a segmentation for the (ii) Segmenting the annotated evaluation image and the existing segmentation forming a comparison between the annotation and the annotation; If the comparison indicates that the segmentation machine learning model is acceptable, The process of deploying or releasing the trained segmentation machine learning model for use. Top and The image segmentation system is configured to:

2. The system according to claim 1, wherein the system is configured to: If the image segmentation and the existing annotations match within a predefined threshold and configured to deploy or release the model for use.

3. 3. The system according to claim 1, wherein the system is The segmentation of the evaluation image and the existing annotations match within a predetermined threshold. If not, the system is configured to continue training the model.

4. 4. The system of claim 1, wherein the training subsystem comprises: A receiving step, (i) an image and annotations for the image; (ii) a score associated with the annotation, the score being associated with the image; The success of the segmentation machine learning model in segmenting It indicates the degree of failure, with a higher score indicating failure and a lower weight receiving a score (wherein lower weighting) indicates success; Reconstructing the segmentation machine learning model using the images and the annotations. A training or refining step is performed to refine the annotations of the images according to the scores. and weighting the plurality of patterns.

5. 5. The system of claim 1, wherein the training subsystem comprises: The segmentation is performed by refining or modifying an existing segmentation machine learning model. A system for generating an annotation machine learning model.

6. The system according to any one of claims 1 to 5, further comprising an annotation subsystem. The annotation subsystem includes an annotation system for generating an unannotated image or a partially annotated image. forming at least one of the annotated training images from the annotated images. The forming is configured to form the unannotated image or the partially annotated image. generating one or more candidate image annotations for the selected image; receiving an input identifying one or more portions of the candidate image annotation on the image; forming the annotated training image from at least the one or more portions. A system that is implemented by a group.

7. 7. The system according to claim 1, further comprising an identification subsystem. The annotated training data further comprises a discriminatory annotation system. Further equipped with a) the segmentation machine learning model is a segmentation and discrimination machine learning model Dell; or b) the training subsystem (i) training each of the images to the segmentation machine learning algorithm; the annotated training data after being segmented by a model; and (ii) ) further configured to train a discriminative machine learning model using the discriminative annotations. The system in question.

8. 8. The system according to claim 1, wherein the training subsystem comprises a a model trainer that uses machine learning to train the segmentation and training a machine learning model to determine a classification for each pixel / voxel of the image. A system that is structured as follows:

9. 1. A computer-implemented method for image segmentation, the method comprising: The annotated training data with each image and segmentation annotations is Train a segmentation machine learning model using the data and generate a trained segmentation generating a machine learning model; The segmentation machine learning model, (i) Using the segmentation machine learning model, Segmenting at least one evaluation image associated with the evaluation, thereby generating a segmentation for the annotated evaluation image using the (ii) Segmenting the annotated evaluation image and the existing annotations forming a comparison between the and evaluating If the comparison indicates that the segmentation machine learning model is acceptable, The process of deploying or releasing the trained segmentation machine learning model for use. and

10. 10. The method of claim 9, (a) Segmenting the annotated evaluation image and the existing annotations If the model matches within a predefined threshold, the model is deployed or released for use. and / or (b) Segmenting the annotated evaluation image and the existing annotations if the combinations do not match within a predetermined threshold, continuing to train the model. method.

11. 11. The method according to claim 9 or 10, wherein the training comprises: A receiving step, (i) using the segmentation machine learning model to retrain or refine it; Annotated images for classification, (ii) a score associated with the annotation, the score being associated with the image; The success of the segmentation machine learning model in segmenting It indicates the degree of failure, with a higher score indicating failure and a lower weight receiving a score (wherein lower weighting) indicates success; According to the scores, when retraining or refining the segmentation machine learning model, and weighting the annotated image according to the weighting factor.

12. The method according to any one of claims 9 to 11, further comprising: The segmentation machine learning model is refined or modified by the machine learning model. The method includes the step of generating:

13. The method according to any one of claims 9 to 12, further comprising the step of: At least one of the annotated training images from the partially annotated images. forming a partial annotated image based on the unannotated image or the partial annotated image; generating one or more candidate image annotations for the image annotated in the and an input identifying one or more portions of the one or more candidate image annotations. receiving the annotated training image from at least the one or more portions; and forming an image.

14. 14. The method according to any one of claims 9 to 13, (a) the annotated training data further comprises identification data; and The method further comprises: training the model as a learning model; or (b) the annotated training data further comprises discriminative annotations, The method includes (i) segmenting each of the images using the segmentation machine learning model. the annotated training data after being annotated; and (ii) the discriminative annotations. and training a discriminative machine learning model using the

15. 15. The method of claim 9, further comprising utilizing machine learning to determine the segmentation. Training a machine learning model to determine a classification for each pixel / voxel of the image. The method includes the step of:

16. 16. The method according to claim 9, wherein the structure or material is selected from the group consisting of bone, muscle, and the like. The method includes the step of:

17. 1. An image annotation system, comprising: a) At least one unannotated or partially annotated an input for receiving an image having been subjected to the processing; b) said at least one unannotated or partially annotated forming respective annotated training images from the images on which the annotation has been performed, The annotator, the formation comprising: The unannotated or partially annotated image generating one or more candidate image annotations for the image; receiving an input identifying one or more portions of the one or more candidate image annotations; and A step of producing the annotated training image from at least the one or more portions. and An image annotation system comprising:

18. 1. An image annotation method, comprising: a) At least one unannotated or partially annotated receiving or accessing an image having been subjected to the b) the annotation is not provided or is partially provided forming respective annotated training images from the images, the formation comprising: one or more candidate image annotations for the unannotated image; generating a receiving an input identifying one or more portions of the one or more candidate image annotations; To do, forming the annotated training image from at least the one or more portions. and A method comprising:

19. When executed by one or more processors, any one of claims 9 to 16 and 18 Computer program code configured to carry out one or more of the methods described herein.

20. 20. A computer readable medium comprising computer program code according to claim 19.

Images