Methods and systems for image segmentation and identification

Abstract

This invention provides an image segmentation system and method for integrating annotations. [Solution] In the system, the model evaluator performs the following steps to evaluate a segmentation machine learning model: controlling the segmentation and discrimination subsystem to segment at least one evaluation image associated with an existing segmentation annotation using the segmentation machine learning model, thereby generating a segmentation for the annotated evaluation image; forming a comparison between the segmentation for the annotated evaluation image and the existing segmentation annotation; and deploying or releasing the trained segmentation machine learning model for use if the comparison indicates that the segmentation machine learning model is satisfactory.

Description

Technical Field

[0001] The present invention relates to methods and systems for image segmentation and identification, and more particularly to segmentation and identification of medical images (e.g., bones, other anatomical structures, masses, tissues, landmarks, lesions, pathological conditions, etc.). The present invention also relates to medical imaging modalities such as computed tomography (CT), magnetic resonance (MR), ultrasound, lesion scanner imaging, etc., but is not limited to these applications. The present invention is for automating the annotation (annotation) of medical image data for training a machine learning model for segmentation and identification purposes, and also relates to methods and systems for evaluating and improving a machine

[0002] Related Applications This application claims the benefit of and priority to U.S. Patent Application No. 16 / 448,252, filed on June 21, 2019, the entire content of which is incorporated herein by reference.

Background Art

[0003] Accurate segmentation and identification of medical images are required for quantitative analysis and disease diagnosis. Segmentation is the process of delineating an object (e.g., an anatomical structure or tissue) within a medical image from its background. Identification is the process of identifying an object and correctly labeling it. Traditionally, segmentation and identification are performed manually or semi-manually. Manual techniques require an expert with sufficient knowledge in the field to be able to draw the contour of the target object and label the extracted objects.

[0004] Computer-assisted systems also exist that provide semi-manual segmentation and identification. For example, such a system could use signal strength, edge, 2D / 3D curvature, shape or other An approximate representation of the object of interest based on selected parameters, including 2D / 3D geometric features. It can detect contours. Then, experts can manually refine the segmentation or identification. Alternatively, an expert could use such a system to approximate the location of the target object, etc. It can provide input data, and the computer-aided system can perform segmentation and Perform identification.

[0005] Whether manual or semi-manual, both are labor-intensive and time-consuming methods. The quality of the results depends heavily on the expertise of the specialist. Therefore, there is a considerable difference in the resulting segmented and identified objects. It is possible.

[0006] In the past few years, machine learning, especially deep learning (for example, deep neural networks) has become popular. Twerks and deep convolutional neural networks are used for many visual recognition applications, including medical images. It has reached a level where it surpasses humans in terms of work performance.

[0007] Patent Document 1 describes a method and system for medical image segmentation based on artificial intelligence. The method is disclosed as follows: The method involves receiving a patient's medical image and, based on the medical image, currently Steps to automatically determine the segmentation context and the current segmentation Based on the context, select at least one segmentation algorithm from multiple segmentation algorithms. The process includes a step of automatically selecting one of the segmentation algorithms. Using at least one segmentation algorithm, within the medical image, The anatomical structure is segmented.

[0008] Patent Document 2 discloses a method for segmenting images of a target patient, The method includes the following steps: target 2D slices for 3D anatomical region images. and the step of providing the nearest neighbor 2D slice, and the trained multislice fully convolution Cosmic Network (Multi-slice FCN, multi-slice fully convolutional neur) A step of calculating a segmentation region using an al network, wherein the region is A predefined intra-body solution that extends spatially across get2D slices and nearest neighbor2D slices. The 2D slice includes an anatomical feature (defined intra-body anatomical feature) and Each nearest 2D slice corresponds to the corresponding shrinkage component of the multislice FCN's successive shrinkage component. The processing is performed by small components, and this processing involves target 2D slicing and nearest neighbor 2D slicing. It follows the order of the chair, and it is a sequence of 2D slices extracted from 3D anatomical images. Based on the S&P, the output of the sequential reduction component is segmented for target 2D slicing. Computational, which is combined and processed by a single extension component that outputs a punctuation mask. Steps to take.

[0009] Patent Document 3 describes applying a deep convolutional neural network to medical images to achieve rear System and method for generating diagnoses or recommended diagnoses in real time or near real time Disclosed is a method, which includes the steps of: performing image segmentation on a plurality of medical images, where the image segmentation step includes separating a region of interest from each image; applying a cascaded deep convolutional neural network detection structure to the segmented images, where the detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method ; ii) a second stage of using a second convolutional neural network to screen for a 3D solid constructed from the candidate positions, where the screening is performed by selecting at least one random position within each solid with random scales and random viewing angles, thereby identifying one or more refined positions and classifying the refined positions ; and automatically generating a report including a diagnosis or a recommended diagnosis plan . This is a step, and the image segmentation step includes separating a region of interest from each image. And a step including: applying a cascaded deep convolutional neural network detection structure (cascaded deep convolutional neural network detection structure) to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. This is a step of applying the detection structure to the segmented images. The detection structure includes: i) a first stage of using a first convolutional neural network to screen for one or more candidate positions at all possible positions within each 2D slice of the segmented medical image by means of a sliding window method. And a step of automatically generating a report including a diagnosis or a recommended diagnosis plan.

[0010] However, there are problems with applying such a method to the segmentation and identification of medical images. That is, in order to train and validate a machine learning algorithm, ground truth data is required, but this data is provided by human experts who annotate the data, which requires a great deal of time and cost. That is, in order to train and validate a machine learning algorithm, ground truth data is required, but this data is provided by human experts who annotate the data, which requires a great deal of time and cost. That is, in order to train and validate a machine learning algorithm, ground truth data is required, but this data is provided by human experts who annotate the data, which requires a great deal of time and cost. That is, in order to train and validate a machine learning algorithm, ground truth data is required, but this data is provided by human experts who annotate the data, which requires a great deal of time and cost. This involves use; ideally, improving machine learning models should correct misrecognition results (e.g., trained models). This requires additional steps, such as when Dell fails to segment or identify medical images. Efficiently managing the training and retraining process is highly difficult; some biomedical uses Regarding the process, it is difficult or impossible to obtain a large number of training images, and also, training days If the available data is limited, it becomes difficult to efficiently train segmentation and discrimination models. . [Prior art documents] [Patent Documents]

[0011] [Patent Document 1] International Public Gazette No. 2018 / 015414 [Patent Document 2] U.S. Patent Application Publication No. 2018 / 0240235 [Patent Document 3] U.S. Patent No. 9,589,974 [Overview of the Initiative]

[0012] The aim of this invention is to provide a segmentation system that can integrate annotations. It fits perfectly.

[0013] According to a first aspect, the present invention is an image segmentation system: Each segmentation annotation is associated with an image (e.g., a medical image). Using annotated training data with images (annotated training data), segmentation Train a segmentation machine learning model to generate a trained segmentation machine learning model. A training subsystem configured as follows, Model evaluator, Using a trained segmentation machine learning model, we can analyze the image (e.g., bone, muscle, fat). The structure or material (including fat or other biological tissues) is designed to perform segmentation. It is equipped with a segmentation subsystem, The model evaluator evaluates segmentation machine learning models. (i) Control the segmentation subsystem and the segmentation machine learning model Using a loop, at least one associated with an existing segmentation annotation Segment the evaluation images, thereby segmenting the annotated evaluation images. Steps to generate the mention, (ii) Segmentation of annotated evaluation images and existing segmentation Steps to form a comparison with the annotation, If the comparison shows that the segmentation machine learning model is passing, then the trained segmentation Steps to deploy or release a machine learning model for use It is structured in such a way that it is accomplished by [something].

[0014] Therefore, the model evaluator is used for operational segmentation. We evaluated a segmentation machine learning model that uses a maintenance subsystem. Yes, it provides an integrated training and segmentation system.

[0015] In this embodiment, the system performs segmentation of annotated evaluation images and If the existing annotations match within a predetermined threshold, the model will be expanded or rebuilt for use. It is configured to be leased.

[0016] In this embodiment, the system performs segmentation of annotated evaluation images and If the existing annotations do not match within a predetermined threshold, the system will continue training the model. It has been done. For example, the system modifies the model algorithm and / or We will continue training the model by adding additional annotated training data. It is configured in such a way that a predetermined threshold can be tuned to suit the desired application. Furthermore, if the initial tuning is unsatisfactory, it can be refined.

[0017] In one embodiment, the training subsystem is: (i) Annotation of images (part of which is a segmentation machine learning model) (i) (which can be arbitrarily generated by the segmentation subsystem using) A score associated with annotation, wherein the score segments the image. This indicates the degree of success or failure of the segmentation machine learning model for that purpose. Yes, a higher score indicates failure, and a lower weighting indicates success. Steps to indicate success and receive a score, Retrain or refine segmentation machine learning models using images and annotations. This is a step that involves weighting the image annotations according to the score. It is structured to form steps.

[0018] Therefore, the segmentation machine learning model continuously regenerates before and during deployment. It can be trained or refined. Annotation of images involves multiple annotation pieces. Please note that this may include items, and optionally, annotations are partial. In particular, a segmentation machine learning model is used in the segmentation subsystem. Please note that it is possible for it to be generated in this way.

[0019] In one embodiment, the training subsystem refines an existing segmentation machine learning model. A segmentation machine learning model is generated by modifying or refining the existing model.

[0020] In this embodiment, the system includes an annotation subsystem, and unannotated Annotated training images (immediately) from the original images or partially annotated images. This brings about at least one of each image with segmentation annotation. This includes steps, which are either unannotated or partially annotated images. The steps include generating one or more candidate image annotations for the given image, and one or more The step of receiving input that identifies one or more parts of candidate image annotations (Note: part (The entirety of the candidate image annotation may consist of at least one part) This is done by performing the step of obtaining annotated training images. To give an example, the annotation subsystem is a) a non-machine learning type image processing method, or b) Using a segmentation machine learning model, at least one candidate image annotation It is configured to generate.

[0021] Therefore, annotations (which may have been generated by non-machine learning image processing methods) An annotation subsystem can be used to demonstrate the validity of the decision. The valid portion of the annotation can be preserved and used.

[0022] In one embodiment, the system further comprises an identification subsystem and annotated training data The data further includes identification annotation, and the segmentation machine learning model segmentation This is a stationary and discriminative machine learning model.

[0023] Combining the annotation subsystem and the segmentation subsystem This will facilitate the continuous improvement of the segmentation system, and medical image analysis. It can be said that applying artificial intelligence to this is beneficial.

[0024] In the embodiment, the system further comprises an identification subsystem, and annotation The completed training data is further equipped with identification annotations, and the training subsystem (i) After each image has been segmented by a segmentation machine learning model (ii) using annotated training data and (ii) discrimination annotations to create a discrimination machine It is further configured to train a learning model.

[0025] The training subsystem may include a model trainer, which leverages machine learning. Train a segmentation machine learning model to determine the classification of each pixel / voxel in an image. It is configured in such a way. Model trainers include, for example, support vector machines and random number generators. Rest trees and deep neural networks can be employed.

[0026] The structure or material may include bone, muscle, fat, or other biological tissue. For example, segmented The process of bone separation is the separation of bone from non-bone material (such as surrounding muscle and fat). This can involve separating one bone from another.

[0027] According to a second aspect, the present invention provides a computer-implemented image segmentation method. The method is: Each image (e.g., medical image) and an annotation with segmentation annotation A segmentation machine learning model is trained using pre-trained training data. Steps to generate a segmentation machine learning model, A step in evaluating a segmentation machine learning model, (i) Using a segmentation machine learning model to analyze existing segmentation annotations Segment at least one evaluation image associated with the selection, thereby To generate segmentation for notated evaluation images, (ii) Segmentation of annotated evaluation images and existing annotations This is done by forming a comparison about the following, and the evaluation step is to If the comparison shows that the segmentation machine learning model is passing, then the trained segmentation This includes steps to deploy or release a structuring machine learning model for use. .

[0028] In this embodiment, the method involves segmentation of annotated evaluation images and existing If the annotations match within a predetermined threshold, the model will be expanded or released for use. It involves doing something.

[0029] In this embodiment, the method involves segmentation of annotated evaluation images and existing If the annotation does not match within a predetermined threshold, the model will continue to be trained. For example, the method involves modifying the model algorithm and / or adding annotations. This involves continuing to train the model by adding previously trained data.

[0030] In the embodiment, training is, (i) Annotations for use in retraining or refining segmentation machine learning models (ii) annotated image and the score associated with the annotated image, The score is a segmentation score for segmenting annotated images. This indicates the degree of success or failure of a machine learning model, with higher scores being the most accurate indicators. A low weight indicates failure, while a low weight indicates success, and the score and the received score are... Step and, When retraining or refining a segmentation machine learning model, annotations are made based on the score. This includes the step of weighting the processed images.

[0031] In one embodiment, the method refines or modifies an existing segmentation machine learning model. This includes the step of generating a segmentation machine learning model.

[0032] In the embodiment, the method involves an unannotated image or a partially annotated image. This includes the step of obtaining at least one annotated training image from the image. This means that one image is either unannotated or partially annotated. The above steps generate candidate image annotations, and one or more candidate image annotations A step of receiving an input that identifies one or more parts of n, and at least one or more parts This includes the step of obtaining annotated training images. For example, the method Using a) non-machine learning image processing method, or b) a segmentation machine learning model The process includes the step of generating at least one candidate image annotation.

[0033] In this embodiment, the annotated training data further comprises the identification data, Furthermore, the method involves segmentation machine learning models for segmentation and discrimination machine learning. This includes the step of training the model.

[0034] In this embodiment, the annotated training data is further provided with identification annotations. The method is (i) each image is segmented by a segmentation machine learning model. (ii) Annotated training data after annotation and discrimination annotation This includes the step of training a discriminative machine learning model using [the specified method].

[0035] The methods include (for example, support vector machines, random forest trees, and deep neural networks) We use machine learning (including multi-level networks, etc.) to train segmentation machine learning models. This may include the step of refining and determining the classification of each pixel / voxel in the image.

[0036] According to a third aspect, the present invention provides an image annotation system, The system is: a) At least one annotation is missing or partially annotated Input for receiving images (such as medical images) that have been applied, b) At least one annotation is missing or partially annotated. The system is configured to produce each annotated training image from the given images. It is a notator, and in order to do this, Images that are not annotated or are only partially annotated To generate one or more candidate image annotations, One or more parts of the annotation of one or more candidate images (Note that the part is a candidate image annotation) Receiving an input that identifies (which may consist of the entire sequence), This involves obtaining annotated training images from at least one or more parts. , including steps.

[0037] In the embodiment, the annotator is a) a non-machine learning type image processing method, or b) segmentation Annotation of candidate images using a machine learning model (for example, the type described above) It is configured to generate at least one of the following:

[0038] According to a fourth aspect, the present invention provides an image annotation method, the method teeth: a) At least one annotation is missing or partially annotated The steps include receiving or accessing the applied image, b) At least one annotation is missing or partially annotated. A step of obtaining each annotated training image from the given images, In order to do this, For images that have not been annotated, add one or more candidate image annotations. To generate, Receiving input that identifies one or more parts of one or more candidate image annotations. and, This involves obtaining annotated training images from at least one or more parts. , including steps.

[0039] The method in the embodiment is a) a non-machine learning type image processing method, or b) segmentation A machine learning model is used to generate at least one candidate image annotation. Includes pu.

[0040] According to the fifth aspect, the present invention is executed by one or more processors, as in the second aspect. Alternatively, a computer program code configured to implement either of the fourth aspects may be provided. Provided. The embodiment comprises the aforementioned computer program code (non-volatile It may also provide computer-readable media.

[0041] Therefore, certain aspects of the present invention relate to integrated annotation and segmentation systems. Supports the continuous training and evaluation of machine learning (e.g., deep learning) models performed using STEMs. do.

[0042] An annotation system simply uses images that have been annotated / identified by humans. In addition, combine the segmented / identified results obtained by the following methods: Image processing algorithms, annotation deep learning models, modification models, and segmentation An annotation model. Human input is used only when necessary, and annotations are checked. This is merely the final step for proof or correction. More images can be annotated. As the process continues, the deep learning model improves, requiring less human intervention. It will be completed. In many cases, (for example, annotation, correction, segmentation) Segmented / identified results obtained from different models (such as classification / identification models) It is understood that accurate annotation can be obtained by combining the results. It was.

[0043] Any of the various individual features of each of the above-described aspects of the present invention, and patent application Any of the various individual features of the embodiments described herein, including the scope of the request These can be combined in an appropriate and desired manner. [Brief explanation of the drawing]

[0044] To more clearly define the present invention, it is demonstrated illustratively with reference to the accompanying drawings. The form will be explained below.

[0045] [Figure 1A] This is a schematic diagram of the architecture of a segmentation and identification system with built-in annotation and training functions according to an embodiment of the present invention. [Figure 1B] This is a schematic diagram of the segmentation and identification system shown in Figure 1A, according to an embodiment of the present invention. [Figure 2] Figures 1A and 1B are flowcharts illustrating the general workflow of the system. [Figure 3] Figures 1A and 1B are simulated flowcharts illustrating the operation of the system. [Figure 4] Figures 1A and 1B are schematic diagrams of the annotation tools for the system. [Figure 5] Figure 1A and 1B show flowchart 70 of the annotation and training workflow for the system. [Figure 6] Figures 1A and 1B are schematic diagrams of the deep learning segmentation and classification models of the system. [Figure 7A] Figures 1A and 1B are schematic diagrams illustrating three exemplary embodiments of segmentation and identification in the system. [Figure 7B] Figures 1A and 1B are schematic diagrams illustrating three exemplary embodiments of segmentation and identification within the system. [Figure 7C] Figures 1A and 1B are schematic diagrams illustrating three exemplary embodiments of segmentation and identification within the system. [Figure 8A] These are schematic diagrams illustrating three exemplary embodiments of the system deployment shown in Figures 1A and 1B. [Figure 8B] These are schematic diagrams illustrating three exemplary embodiments of the system deployment shown in Figures 1A and 1B. [Figure 8C] These are schematic diagrams illustrating three exemplary embodiments of the system deployment shown in Figures 1A and 1B. [Modes for carrying out the invention]

[0046] Figure 1A shows a segmentation and discrimination system with built-in annotation and training capabilities. This is a schematic diagram of the architecture of Tem10. System 10 is an annotation and The training subsystem 12, the segmentation and discrimination subsystem 14, and the trained Segmentation and identification burning 16 and graphical user interface (GU I) Includes a user interface 18 including 20.

[0047] Generally, the annotation and training subsystem 12 is used by operators with specialized knowledge. Under control, "ground truth" (regarding the targeted anatomical structures and tissues) (Correct answer) Annotate the medical images selected as training data. It is composed of annotated training data (for example, deep learning algorithms, etc.). ) Input into machine learning algorithms, (trained segmentation and discrimination model 1 Train the segmentation and discrimination models (stored within 6). The segmentation and identification model is used by the segmentation and identification subsystem 14. (26) In order to segment the objects of interest within new medical images and Used for identification, it also includes the classification and labeling of image pixels / voxels.

[0048] GUI20 can be implemented in several different ways, for example, by an annotation (i.e. A computer used by an operator with the appropriate skills to recognize the features of an image. Handling devices (e.g., personal computers, laptops, tablet computers) The GUI should be installed as software on a computer or mobile phone, etc. This is possible. In another example, GUI20 allows those who annotate as web pages to It can be provided and accessed via a web browser.

[0049] Segmentation and classification models are trained using annotated images (22) In addition to this, the annotation and training subsystem 12 uses the trained segmentation The performance of the and identification model 16 was evaluated (24) to determine if the model was ready to be deployed. This allows for decisions to be made about how to proceed, and also enables the model to be retrained or improved.

[0050] However, the segmentation and discrimination models failed for one or more specific images. In that case, images that the model failed to process will be annotated and used as a new training dataset. The following are added to the training subsystem 12, along with the associated segmentation and discrimination models. Retrain (28).

[0051] Figure 1B is a schematic diagram of system 10. System 10 is segmentation and This includes the identification controller 32 and the aforementioned user interface 18. The and identification controller 32 has at least one processor 34 (or some implementations) The system includes multiple processors and memory 36. System 10 is, for example, a computer On a computer, as a combination of software and hardware (for example, a personal computer or mobile computer) It will be implemented (as a puteing device) or as a dedicated image segmentation system. It can be implemented in this way. Regarding system 10, it is optionally considered to be distributed. It is possible; for example, all or part of the memory 36 components of the processor 34 It can be placed in a remote location; the user interface 18 has memory 36 and It can be located remotely from processor 34, and in fact it can be located in a remote location. It may include a web browser and a mobile device application.

[0052] The memory 36 can communicate with the processor 34 for data, and is typically volatile. It also includes both non-volatile memory and one or more for each memory type. This may include,) RAM (Random Access Memory), ROM, and one or more This includes mass storage devices.

[0053] The processor 34 controls the annotation and training subsystem 12 and segmentation It includes an and identification subsystem 14. As described in more detail below, annotate The simulation and training subsystem 12 processes the original image according to a non-machine learning image processing method. The system includes an initial segmenter and classifier 38, an image annotator 40, and a model trainer 42. It includes a model evaluator 44. The segmentation and identification subsystem 14 is a preprocessor. It includes a 46, a segmenter 48, a structure discriminator 50, and a result evaluator 52. The 34 also includes the I / O interface 54 and the result output 56.

[0054] Memory 36 contains program code 58, image data 60, training data 62, and evaluation images 6 4. Ground truth images 66, trained segmentation and classification models 16, trained annotations This includes the simulation model 68 and the trained modified model 69.

[0055] The segmentation and identification controller 32 is at least partially connected to the memory 36. It is implemented by the processor 34 executing program code 58.

[0056] Broadly speaking, the I / O interface 54 provides image data related to the subject or patient. Read or receive (for example, DICOM format) into image data 60 or training data 62. It is configured in such a way that it is used for analysis and / or training (or both, as described below). (For the purpose of) each is intended to be used. System 10 once the various structures are image data Once segmented and identified from 60, the I / O interface 54 will analyze the results of the analysis ( (Optionally, in report format), for example, output via result output 56 and / or GUI 20. .

[0057] Figure 2 is a flowchart 20 of the general workflow of system 10. In S72, 1 image set (typically stored remotely or locally, such images (From the database) These will be imported and used as training images / data. These images are input into the system 10 via interface 54 for training data. It is stored in TA62. The training data 62 is a picture that is expected to be encountered in a clinical setting. It should be considered representative of the image, and both basic and edge examples (i.e., "correct It should include both "normal" and "extreme" cases. The structure and organization in the basic case should be These can be segmented or identified more easily than those of edge cases. For example, a CT scan of the wrist. The scan requires segmenting the bone from the surrounding muscle and fat. Scanning is easier in young, healthy subjects because, regarding these... In this case, the porosity of the bone boundary is lower and more pronounced than in elderly and frail patients, and in the latter case The bone is highly porous and its boundaries are less distinct. However, basic cases and et It is desirable to collect examples of both types of cases as training data.

[0058] In this step, a second set of images is used to evaluate the performance of the trained model. Also import them. These evaluation images used for evaluation are representative of clinical images. This should be done and is stored within evaluation image 64.

[0059] In S74, operators with appropriate expertise are trained using the image annotator 40. Annotation is applied to both data 62 and evaluation images 64. If the application is different, The required annotations also change. For example, the model will segment bone material from non-bone material. If training is required for this, then what is needed in annotation is bone pixels / voxes The purpose is to distinguish and identify those that are bone and non-bone, and therefore segmentation data is included. It should be done. On the other hand, the models have advanced further and require the identification of each bone mass. In this case, the annotation requires distinction and labeling for each bone mass's pixels / voxels. This involves adding a label, and therefore additional identification data should be included.

[0060] In S76, the model trainer 42 uses the annotated training data 62 to train the classifier model Dell is trained, and in this embodiment, it is used as a segmentation and discrimination model. This is a classifier that determines the classification of each pixel / voxel in the image. In training, decisions are made to arrive at the correct answer (annotation) from the input (training image). The objective is to determine the decision pattern. The model is For example, utilizing machine learning algorithms such as support vector machines and random forest trees. Training can be achieved through use.

[0061] In this embodiment, a deep neural network is used. As described later (see Figure 6), (I want to) This deep neural network has an input layer, an output layer, and layers in between them. It consists of artificial neurons. Each layer consists of artificial neurons. An artificial neuron is a neuron that receives one or more inputs. It is a mathematical function that sums them up to produce an output. Typically, for each input Then, by individually weighting the values, Sum will be passed through a nonlinear function. As the twerk learns, the model's weight values ​​are adjusted, but the adjustment is not It is done according to the error (difference between network output and annotation). The adjustments are made until the error can no longer be reduced.

[0062] In S78, the model evaluator 44 controls the segmentation and identification subsystem 14. By doing so, the trained model is evaluated, and that model is used to apply the evaluation image. It performs classification and identification. In S80, the model evaluator 44 determines that the trained model is The purpose is to confirm whether or not a passing grade has been reached, and the result of that process is used in S74. This is done by comparing with the notation. In this embodiment, this is done by the trained model The results generated by the program and the annotated image (ground truth image) used in S74. The difference between the annotation data associated with 66 is below a predetermined threshold. This is done by deciding whether or not to do so. For example, the difference is generated by the model. Segmentation and the segmentation suggested by the annotation data The ratio of overlap to the segmentation suggested by the annotation data. It can be calculated as a percentage. If these match perfectly, the overlap is clearly 100. This results in a percentage, which implicitly indicates a passing grade. In some applications, this threshold is set at 90%.

[0063] If the model evaluator 44 determines in S80 that the model is unacceptable, the process proceeds to S82. Proceed, and one or more new images are imported in S72, just as the original training data was imported (original Imported (to supplement the training data) and / or the learning algorithm is adjusted / modified. (For example, tuning the parameters of a neural network, neural network Making changes to the layers of a network, or the neurons of a neural network This can be done by making changes to the initiation function, etc. Then the process is S74 Return to the previous page.

[0064] In S80, the model evaluator 44 uses the results generated by the trained model in S74. If it is determined that the difference between the annotation and the given annotation is below a predetermined threshold, i.e., training If the trained model is determined to be acceptable, the process proceeds to S84, and the trained model is (trained It is stored and deployed (as one of the trained segmentation and discrimination models 16). The phase is complete (at least at this stage), and therefore the S85 is trained model The input and image are used by the segmentation and identification subsystem 14. Segment one or more structures or materials / tissues within the new medical image stored in the image data 60 To identify and distinguish. In S86, the result evaluator 52 evaluates the result to the targeted structure or material. Compare to one or more predefined conditions or parameters known to characterize quality. This allows us to verify the results of this segmentation and identification. If the performance of the station and identification models is deemed unsatisfactory, the process proceeds to S88. The new images are added to the training set as new training data, and the processing is (this These images are used for annotation, retraining models, and so on. Proceed to S74.

[0065] If the result evaluator 52 does not determine the result to be unacceptable in S86, the process proceeds to S90. The system then displays new medical images, such as those shown by the user interface 18 of the system 10. The segment / identification result from the image is output. (Optionally in S86, result evaluator 52) If the result is not deemed unsatisfactory, the segmentation and identification results will be provided to the user. It can be displayed to allow the user to perform supplementary manual verification. And (for example) For example, by selecting the "Fail" or "Reject" button on GUI20, the user can If the result is flagged as a failure, the result is considered a failure and the process proceeds to S88. The user can select the "Pass" or "Accept" button on GUI20 to determine the outcome. If you do not object to the decision of evaluator 52, the process proceeds to S90.

[0066] Then, after S90, the process ends. For example, the results can be used for diagnostic purposes or for further determination. It can be used as input for sexual analysis.

[0067] Figure 3 is a simulated flowchart 70 of the operation of system 10. As mentioned above, The training and instruction subsystem 12 performs segmentation and training on medical images. It is configured to apply annotation for identification. Medical images and their annotations The annotation is used to train segmentation and discrimination models. Annotation is applied to the medical image set via GUI20, and this is the process. In the specification, it may also be referred to as an annotation interface.

[0068] An annotator completes the annotation by combining information from different resources. In this embodiment, the annotator combines information from one or more candidate image annotations. Then, if necessary, complete the annotation using the GUI20 annotation tool. Finished. In this embodiment, candidate image annotation can be in the following format: Preliminary segmentation and classification results generated using existing non-machine learning image processing methods; Or results generated by a partially trained segmentation and discrimination model. Complementary image annotations are displayed together (i.e., next to each other), and The data is designed to allow for easy comparison of which part of each data set is the best.

[0069] Furthermore, since preliminary segmentation results are generated using existing image processing methods, As you will see, the results of the preparations typically require improvement. Examples of image processing methods include contour detection, blob detection, and threshold-based object segmentation. These methods include selecting pixels corresponding to bone in a CT scan from surrounding pixels. Although it allows for early segmentation, in many cases it is a rough or coarse preliminary segmentation. It brings about a sensation.

[0070] GUI20 consists of a preliminary results window 102 and an annotated image window 104. The annotated image window 104 includes multiple annotation tools 106 Includes. Annotation tool 106 is both manual and semi-manual annotation tool. These include elements that can be displayed in the annotator and manipulated by the annotator. These tools are schematically shown in Figure 4, and these tools include manual tools. This includes the brush tool 130 and the eraser tool 132. Brush tool 130 The annotator controls the labeling of pixels with different structures or materials using different values ​​or colors. The eraser tool 132 can erase excess segments controlled by the annotator. The pixel that is either incorrectly identified or over-identified can be removed from the target structure or material. Semi-manual type The annotation tools include fill tools 134, and the annotator controls them. It can draw the outline surrounding the target structure or material, and an annotated image window. U104 can be used to annotate all pixels / voxels within its contour. Furthermore, the semi-manual annotation tool includes the area expansion tool 136, After annotating a very small part of the structure or material, the annotator then... Control the annotated image window 104 to expand the annotation This can encompass the entire structure or material.

[0071] System 10 also includes machine learning-based tools, which allow the annotator to accurately segment It helps to perform maintenance and identification quickly. For example, annotation tool 1 06 includes a machine learning-based annotation model control 138, This calls the trained annotation model stored within annotation model 68. The annotation model 68 is controlled by an annotator to segment objects. In this process, multiple points can be selected for use by the annotation model 68. In this embodiment, the annotation model 68 provides the annotator with respect to the four edges of the object. The prompt is to identify the four extreme points, that is, the four extremes A point is the leftmost, rightmost, topmost, and bottommost pixel. And, annotate For example, touch the image with a stylus (if it's displayed on a touchscreen). Select each point sequentially by clicking or using the mouse. Then, Annotation model 68 uses these edge points to segment objects. In this way, the selected point (for example, the edge point) is annotated 6 The annotation data 118 used to train model 8 is constructed.

[0072] Annotation tool 106 includes annotation motion capture tool 140. Furthermore, it is another mechanism for recording annotation data 118, and also for the annotator Therefore, it can be started. The movements are recorded by system 10 (within the trained modified model 69). Train the modified model (stored in ). For example, the annotator oversegmentation You can move the contour inward or move the undersegmentation contour outward. Movements toward or outward, and the positions in which they occur, are annotated motion captures. The tool 140 trains the modified model (stored in the trained modified model 69). It is recorded as input for the purpose of [doing something]. Also, annotation motion capture tool 1 40 records, for example, the amount of effort required, such as the number of mouse movements or the amount of mouse scrolling. During training, it is used to give greater weight to more troublesome cases.

[0073] Returning to Figure 3, during use, the preliminary segmentation and identification result 110 is G The results will be presented to the annotator via the preliminary results window 102 of UI20. If the annotator is satisfied with the accuracy of the segmentation and identification results 110 The annotator can, for example, click the mouse on the preliminary results window 102. Move the preliminary result 110 to the annotated image window 104 of GUI20. If the notator is satisfied with only (but not all) of the preliminary result 110, then the notator This places only a sufficient portion of the preliminary result 110 into the annotated image window 104. Move and then use the annotation tool 10 on the annotated image window 104. Use step 6 to correct / complete segmentation and identification.

[0074] The annotator is satisfied with both the preliminary segmentation and the identification results. If not present, the annotator uses annotation tool 106 to add an annotation to the image via Abinitio. Annotate it.

[0075] The annotator has partially completed the annotation of sufficient preliminary results 110, or the image If you either manually annotate the original image, the annotated original image set To114 is located within the annotated image window 104, and is the correct image. It will be stored as 66.

[0076] After one or more ground truth images 66 are collected in this manner, the annotated original image In other words, the segmentation and classification models are trained using the 66 correct images. The segmentation and discrimination model (within the trained segmentation and discrimination model 68) ) Once available, the segmentation generated using the trained model and The identification is also provided to the annotator as a reference. The annotator is trained on segmentation. If you are satisfied with the results generated by the and discriminative models, or with respect to any part of such results If present, the annotator moves a sufficient portion to the annotated image window 104. The annotator will use the annotated image window 104 to determine the exact results. Minutes can be combined with sufficient portions from trained models. Insufficient annotations If an image or part of an image with annotations still exists, the annotator will... Annotations can be modified or supplemented using code 106.

[0077] The collected ground truth images and original image 60 within the ground truth image 66 are used for segmentation and discrimination. Used to train Dell. To evaluate the trained model, another ground truth image and A set of source images is used. In some embodiments, the same images are used for both training and evaluation. This is used. If the performance of the trained model is satisfactory during image evaluation, the model is trained The data is sent to the segmentation and discrimination model 68, and any new image is sent to it. It is used by the segmentation and identification subsystem 14 for processing. Performance If this is insufficient, collect more ground truth images. When evaluating the model (i.e., System 10 determines whether the classification and discrimination model is acceptable or not. The criteria used are adjustable. Such adjustments are typically made during the development of System 10. This is done by the person or model who initially trained it, and is adjusted according to the requirements of the application. For example, in determining vBMD (volumetric bone mineral density). In doing so, in order to verify whether the entire bone is accurately segmented from the surrounding material, The criteria are set accordingly; when calculating cortical porosity, the whole bone and the segments of the cortical bone are used. Establish criteria to verify station accuracy.

[0078] Figure 5 is a flowchart 150 of the annotation and training workflow. For reference, in S152, the original image 60 is selected or entered, and in S154, the original image This is processed by the initial segmenter and discriminator 38, thereby generating preliminary segmentation. The system generates the selection and identification results. The process proceeds to S156, where the segmenter 48 uses the trained segment. Segmentation and discrimination models are trained within segmentation and discrimination models 16 The system then determines whether it is available or not, and if so, proceeds to S158, and trained The original image 60 is also processed using the segmentation and discrimination model 16. The process proceeds to S160. In S156, system 10 performs training on segmentation and recognition. If it is determined that no other model is available, the process proceeds to S160.

[0079] In S160, the annotator uses system 10 to generate preliminary model results and trained models. Whether any part of the result is a pass or fail (usually these results are passed through the user interface) (By inspecting on the display of S18) it is verified, and if so, the process is S1 Moving on to 62, the annotator uses annotation tool 106 to identify the parts that pass. The image is moved to the notated image window 104, and the processing proceeds to S164. Proceed. If none of the results obtained by the annotator in S160 are deemed acceptable. The process then proceeds to S164.

[0080] In S164, if none of the obtained results pass, the annotator will reject the image. I was prompted to annotate in Abinitio, or I had no idea about annotation. Complete, correct, and supplement using that method. The annotator continues until the annotation is satisfactory. This is done by controlling the manual and semi-manual tools 106. Also, annotation Optionally, the modified model control 142 can be used (modified model 69 is available). If so, you can also call the modified model 69 to improve the annotations.

[0081] In S166, segmentation and recognition are performed using ground truth images and annotation data. A separate model is being trained, and in parallel, in S168, ground truth images and annotations are being processed. The annotation model 68 and the modified model 69 are trained using the data. (In S164) If no modified model is available, then after the modified model is generated in S168, S The modified model is available in subsequent passes up to 164 within modified model 69. Please note that this will become a Noh play.

[0082] The process proceeds to S170, where the trained segmentation and discriminant models are annotated. The model evaluation of the training subsystem 12 is performed by the model evaluator 44. In S172, the evaluation is performed. The results are verified, and if they fail, the process proceeds to S174, and the trained segmentation The segmentation and discrimination models are stored in the trained segmentation and discrimination model 68, and new The image 60 is expanded for processing. Otherwise, the process returns to S152. Then, one or more additional ground truth images 66 are collected or selected, and the process is repeated.

[0083] The new image is processed by the segmentation and identification subsystem 14. This involves segmenting and identifying the target structure or material (e.g., tissue). As shown above, the segmentation and identification subsystem 14 comprises four modules: The pre-processor 46, segmenter 48, structure discriminator 50, and result evaluator 52. The 46 verifies the effectiveness of the input image (medical image in this embodiment), and medical The medical image information will be verified, including whether the image is complete and whether it is damaged. The pre-processor 46 reads the medical image into the image data 60 within the system 10. It even gets mixed in. Images can be in different formats, and the previous processor 46 is the main factor. It is configured to read images in a format that has continuity, and this includes DICO M is also included. Additionally, multiple pre-trained segmentation and discrimination models 16 are available. Because this can happen, the previous processor 46 extracts from the image Regarding segmentation and identification based on the information, trained segmentation It is configured to determine which of the omission and identification model 16 should be used. This is possible. For example, in one scenario, two classifier models are trained. The first classifier model can segment and identify the radius from a CT scan of the wrist. The second classifier model is intended to segment the tibia from a CT scan of the leg. It can be used for identification and identification. A new CT in DICOM format. When processing a scan, the preprocessor 46 extracts the body to be scanned from the DICOM header. Positional information is extracted, and for example, it is decided that a radial classifier model should be used.

[0084] The segmenter 48 and the structure discriminator 50 each identify the target structure or material within the image. Using a model selected from the refined segmentation and discrimination models 68, segmentation The segmentation and identification are then performed by the result evaluator 52. This is automatically evaluated. In this embodiment, the evaluation performed by the result evaluator 52 is The following actions are involved: one or more predefined conditions or parameters of the target structure or material (For example, the tolerance range of one or more structures, or the dimensions of a material, or the structure or material) The results were examined in relation to the volume, etc., and it was found that the segmentation and identification results were clearly insufficient. To determine whether or not it is a minute. The result evaluator 50 determines whether the segmentation and identification results are Output a result indicating whether or not it is actually sufficient.

[0085] The automated evaluation performed by the result evaluator 52 can be manually enhanced, for example, The results of the gradation and identification, and / or the evaluation of the result evaluator 52, are to be further evaluated. This allows the results to be displayed to users (e.g., doctors), and enables the automated evaluation to be refined.

[0086] Images that failed segmentation and identification results were... It is used as additional training images to retrain a different model. The result evaluator 52 is segmented If the segmentation and identification results are deemed acceptable, the segmentation and identification results will be processed as I / O. via interface 54 (for example, on the display of a computer or mobile device) ) Output to result output 56 and / or user interface 18. In some embodiments, The segmentation and identification results are used as input for further quantitative analysis. For example, the radius is segmented and identified by system 10 from a CT scan of the wrist. If so, the attributes of the extracted radius, such as volume and density, can be recorded (locally or remotely). It can be passed to another application (that is currently running) for further analysis. .

[0087] Convolutional neural network for segmentation and identification according to an embodiment of the present invention The twerk is generally shown by reference numeral 180 in Figure 6, and deep learning segmentation This is the form of the deep learning segmentation and classification model. 0 represents a reduced convolutional network (CNN) 182 and an expanded convolutional network (CN) 182. The CNN 182 comprises N)184 and a reduction or downsampling CNN 182 which processes the input image. Create a feature map by reducing the resolution of those layers through various layers. (Enlarge or upscale) The Pring CNN184 processes these features through layers that increase the resolution. This eventually generates segmentation and identification masks.

[0088] In the example in Figure 6, the reduced CNN182 has four layers: 186, 188, 190, and 192. The extended CNN has four layers, 194, 196, 198, and 192, but Please note that other values ​​may be used for the number of layers. The lowest layer (i.e.) Layer 192 is shared by the reduced and expanded networks 182,184. Each layer 186, ..., 198 has an input and a feature map (each layer has (Illustrated as a hollow box). Each input of layers 186,...,198 is a convolution. The process is processed and converted into a feature map (indicated by the hollow arrows in the diagram) (Illustrated). The convolution process or activation function is, for example, a normalized linear unit. (ReLU, rectified linear unit), sigmoid unit, or Tanh unit It is possible.

[0089] In the reduced CNN182, the input to the first layer 186 is the input image 200. In version 182, the feature maps on each layer are downsampled to other feature maps. It is then processed as inputs 202, 204, and 206 of the next lower layer. It is used. Downsampling process 208 is used, for example, in maximum pooling operations and strike It can be operated as a D-type operation.

[0090] In the extended CNN184, feature maps 208 are placed on each layer 192, 198, and 196. 201 and 212 are upscaled to separate feature maps 214, 216, and 218, respectively. It is pulled and used in the following layers 198, 196, and 194. Sampling operations include, for example, upsampling convolution, transpose convolution, or bilinear convolution. Upsampling can be performed.

[0091] To capture localization patterns, each pair within the reduced CNN182 Corresponding layers 190, 188, 186 contain high-resolution feature maps 220, 222, 2 24 is converted to the upsampled feature maps 214, 216, and 218 respectively. P and are connected (shown as dashed arrows in the figure), and as a result The resulting high-resolution feature maps / feature map pairs are 220 / 214,222 / 216,224 / 218 are used as inputs to layers 198,196,194 respectively. In the final layer 194 of the extended CNN184, the feed resulting from the convolution process is The segment map 226 is converted and output as the final segmentation and identification map 228. This transformation is performed, and this transformation is illustrated as a solid arrow in the diagram. For example, this can be implemented as a convolution operation or sampling.

[0092] In this way, the deep learning segmentation and classification model 180 in Figure 6 is reduced in size. Location information from the contraction path is used for the extended route. Combined with contextual information in the expansion path Ultimately, it comes down to localization and context. Obtain general information by combining it with xt).

[0093] Examples of segmentation and identification within system 10 according to embodiments of the present invention Examples of implementations are shown in Figures 7A to 7C. Referring to Figure 7A, the first segmentation Implementation example 240 for identification and recognition is characterized as a "one-step" embodiment. The first segmentation and identification 240 is followed by annotation and training phase 24. This includes step 2 and a segmentation and identification phase 244. In the former, the original image 246 is Annotation 248 for segmentation and identification, and training is carried out. After that, a segmentation and identification model 250 is generated. And in the identification phase 244, new segments and identification models 250 are generated according to the segmentation and identification model 250. Image 252 is used for segmentation and identification activities 254, and then segmentation The output is the result 256.

[0094] Therefore, in the embodiment shown in Figure 7A, Model 250 is trained for segmentation and Simultaneous identification is performed. For example, segmentation of the radius and fibula in a CT scan of the wrist. When training the recognition and discrimination models, annotations are applied to the training images. At that time, different values ​​will be assigned to the voxels of the radius and fibula. Once the model is ready, Any new wrist CT scan 252 is used by the segmentation and discrimination model 250. The results were processed, and the result 256 was obtained in a segmented and identified form of the radius and This is presented as a map of the fibula.

[0095] A second example of segmentation and identification implementation, 260, is schematically shown in Figure 7B. In implementation example 260, segmentation and identification are implemented in two steps. Figure 7B Referring to the example implementation 260, the annotation and training phases 262 and segmentation This also includes the selection and identification phase 264, but the annotation and training phase 262 is This includes training the segmentation model and training the discriminative model as separate processes. Therefore, the original image 266 was annotated 268 and segmented. Model 270 is trained. Annotation 274 is applied to segmented images 272. The discrimination model 276 is trained after the process is completed. After the two models 270 and 276 have been trained Any new image 264 is first segmented by the segmentation model 270. 80 was performed, and the resulting segmented image 282 was identified by the discrimination model 276 It is used for identification, and the identified result 286 is output. For example, this Through these two steps, and in these two steps, within the wrist CT scan All bones can first be segmented based on the surrounding material, and different types of bones can be segmented. can be differentiated and identified.

[0096] A third segmentation and identification implementation example 290 is schematically shown in FIG. 7C. In this implementation example, the segmentation model achieves very accurate results, and moreover, the identification can be performed using a heuristic-based algorithm instead of a machine learning based model. Referring to FIG. 7C, the implementation example 290 includes an annotation and training phase 2 92 and a segmentation and identification phase 294. The original image 266 is annotated 268 for segmentation in the annotation and training phase 292 and is also used to train the segmentation model 270. In the segmentation and identification phase 264, a new image 278 is segmented 280 using the segmentation model 270 to yield a segmented image 282, and then identification 284 is performed using a heuristic-based algorithm, and the identified result 286 is output. For example, once accurately segmented 280 using the segmentation model 2 70, a wrist CT scan can identify different bones, for example, by volume calculation and differentiation. For example, in a wrist HRpQCT scan, after bone segmentation, bone volume

[0097] By calculating and comparing the product, the radius can be easily distinguished and identified from the ulna. Why? This is because the radius is larger than the ulna in the same subject.

[0098] Three illustrative deployment examples of System 10 are schematically shown in Figures 8A to 8C. These are as follows: on-premises deployment; cloud deployment; and hybrid deployment. As shown in Figure 8A, in the first deployment example 300, system 10 is deployed locally. (Although it may be considered a distributed form.) (For example, model training Furthermore, all data processing (such as image segmentation and identification) is performed by one or more systems 10. This is done by the processor 302. The data is stored in one or more data storage units 304. It is stored encrypted. The user accesses the system via the user interface 18. Interact with 10 (for example, by annotating images or inspecting the results).

[0099] As shown in Figure 8B, in the second deployment example 310, system 10 is a user interface Except for 18, it is deployed within the encrypted cloud 312. User interface 18 It is located locally. User interface 18 and encryption cloud 312 The communication between them (314) is encrypted.

[0100] As shown in Figure 8C, in the third deployment example 320, system 10 is partially cryptographic It is deployed within the cloud service 322, and on the other hand, the data storage unit 326 and The processing power unit (e.g., processor) 328 and the user interface 18 are associated with The local part 324 is located locally. In such deployment examples, typically, (user Most of the system 10 (excluding the interface 18) is deployed within the cloud service 322, but the data storage unit 326 and the processing power holding unit 328 support the user interface 18 and the communication 330 between the local unit 324 and the cloud service 322. The communication 330 between the local unit 324 and the cloud service 322 is encrypted.

[0101] Those skilled in the art should understand that many modifications can be made without departing from the scope of the present invention. In particular, it should be noted that additional embodiments can be achieved using specific features of the embodiments of the present invention.

[0102] Even if there are references to the prior art in this specification, it should be noted that such references do not form an admission that the prior art forms part of the well-known art in any country.

[0103] In the appended claims and the foregoing detailed description of the invention, the terms "comprising", "comprises" (third-person singular present tense), "comprising" and the like are used in an inclusive sense (i.e., in the sense of specifying the presence of the recited features), but do not preclude the existence or addition of further features in various embodiments of the present invention unless the context requires a different interpretation.

Explanation of Reference Signs

[0104] 10 System 12 Annotation and Training Subsystem 14 Segmentation and Identification Subsystem 16 Trained Segmentation and Identification Model ​​​​​​​18. User Interface 20 GUI 32 Segmentation and Identification Systems 34 processors 36 memory 66 Correct image 106 Annotation Tools 118 Annotation data 182 Shrinking Network 184 Extended Network 242 Annotation and Training 244 Segmentation and Identification 262 Annotation and Training 264 Segmentation and Identification 292 Annotation and Training 294 Segmentation and Identification 302 Processors 304 Data Storage Unit 312 Encryption Cloud Services 322 Encryption Cloud Services 324 Local portion 326 Data Storage Unit 328 processors

Claims

1. An image segmentation system, wherein the system is Annotation with images associated with each segmentation annotation Train a segmentation machine learning model using pre-trained training data. A training subsystem configured to generate a machine learning model, Model evaluator, Using the aforementioned trained segmentation machine learning model, the structure or material within the image is identified. It comprises a segmentation subsystem configured to perform segmentation. 、 The aforementioned model evaluator evaluates the segmentation machine learning model, (i) Control the segmentation subsystem to control the segmentation machine Using a learning model, at least the existing segmentation annotations are associated with Another evaluation image is segmented, thereby matching the annotated evaluation image. The steps include generating segmentation and (ii) Segmentation of the annotated evaluation image and the existing segmentation Steps to form a comparison with the annotation, If the comparison indicates that the segmentation machine learning model is satisfactory, The state to deploy or release a pre-trained segmentation machine learning model for use. Top, An image segmentation system configured to perform the following actions.

2. In the system according to claim 1, the system is the annotated evaluation drawing When the image segmentation and the existing annotations match within a predetermined threshold. A system configured to deploy or release the aforementioned model for use.

3. In the system according to claim 1 or 2, the system is the annotated The segmentation of the evaluation image and the existing annotations match within a predetermined threshold. If not, the system is configured to continue training the aforementioned model.

4. In the system according to any one of claims 1 to 3, the training subsystem is: The receiving step, (i) Images and annotations for said images, (ii) A score associated with the annotation, wherein the score is the same as the image The success of the aforementioned segmentation machine learning model in segmenting or This indicates the degree of failure; a higher score indicates failure, and a lower weight ( Lower weighting) indicates success, score, and steps to receive. Using the aforementioned images and annotations, the segmentation machine learning model is re-engineered. A training or refinement step, wherein the annotation of the image in accordance with the score A system configured to make steps, including weighting of n.

5. In the system according to any one of claims 1 to 4, the training subsystem is: By refining or modifying existing segmentation machine learning models, the segmentation A system that generates input machine learning models.

6. In the system according to any one of claims 1 to 5, the annotation subsystem The annotation subsystem includes an unannotated or partially annotated image. To form at least one of the annotated training images from the annotated images It is composed of the above, and the formation is the above unannotated image or partially annotated The steps of generating one or more candidate image annotations for the image and the one or more The steps include receiving an input that identifies one or more parts of the candidate image annotations mentioned above, and , steps to form the annotated training image from at least one of the above parts A system created by P.

7. In the system according to any one of claims 1 to 6, the system is an identification subsystem The system further includes the annotated training data, and the annotated training data is identified and annotated. Furthermore, a) The segmentation machine learning model is a segmentation and discrimination machine learning model Is it Dell; or b) The training subsystem (i) each of the images is the segmentation machine learning The annotated training data and (ii) after being segmented by the model ) Further configured to train a discriminative machine learning model using the aforementioned discriminative annotations. It is a system.

8. In the system according to any one of claims 1 to 7, the training subsystem is This includes a Dell trainer, which uses machine learning to train the segment A machine learning model is trained to determine the classification of each pixel / voxel in the image. The system is in place.

9. A computer-implemented image segmentation method, wherein the method is Annotated training data with each image and segmentation annotation We train a segmentation machine learning model using data, and the trained segmentation Steps to generate a machine learning model, The aforementioned segmentation machine learning model, (i) Using the segmentation machine learning model, the existing segmentation anomaly Segment at least one evaluation image associated with the station, and by doing so This involves generating segmentation for the annotated evaluation images, (ii) Segmentation of the annotated evaluation image and the existing annotation To form a comparison about the matter, The steps to evaluate by, If the comparison indicates that the segmentation machine learning model is satisfactory, The state to deploy or release a pre-trained segmentation machine learning model for use. Methods that include the top.

10. In the method according to claim 9, (a) Segmentation of the annotated evaluation images and the existing annotations If the condition is met within a predetermined threshold, the model will be deployed or released for use. The steps and / or (b) Segmentation of the annotated evaluation images and the existing annotations If the condition does not meet within a predetermined threshold, the process includes the step of continuing to train the model. method.

11. In the method according to claim 9 or 10, the training is The receiving step, (i) Used when retraining or refining the segmentation machine learning model Annotated images for this purpose, (ii) A score associated with the annotation, wherein the score is the same as the image The success of the aforementioned segmentation machine learning model in segmenting or This indicates the degree of failure; a higher score indicates failure, and a lower weight ( Lower weighting) indicates success, score, and steps to receive. When retraining or refining the aforementioned segmentation machine learning model, the score shall be used accordingly. A method comprising the step of weighting the annotated image.

12. A method according to any one of claims 9 to 11, wherein an existing segmentation machine By refining or modifying the machine learning model, the segmentation machine learning model A method including the step of generating.

13. A method according to any one of claims 9 to 12, wherein an unannotated image or part From the partially annotated images, at least one of the annotated training images The step of forming the unannotated image or partial For an annotated image, generate one or more candidate image annotations. A pp and an input that identifies one or more parts of the one or more candidate image annotations. The steps of receiving and the annotated training drawing from at least one of the parts A method that involves the steps of forming an image.

14. In the method according to any one of claims 9 to 13, (a) The annotated training data further comprises identification data, and The method involves using the segmentation machine learning model for segmentation and discrimination. Includes a step of training as a learning model; or (b) The annotated training data further comprises identification annotations, and the method The method is (i) each of the above images is segmented by the segmentation machine learning model (ii) the annotated training data after (ii) the identification annotation A method comprising the step of training a discriminative machine learning model using and .

15. The method according to any one of claims 9 to 14, wherein machine learning is used to determine the segment A maintenance machine learning model is trained to determine the classification of each pixel / voxel in the aforementioned image. A method that includes the steps of

16. In the method according to any one of claims 9 to 15, the structure or material is bone, muscle A method comprising meat, fat, or other biological tissue.

17. It is an image annotation system, a) At least one annotation is missing or partially annotated. An input unit for receiving the applied image, b) At least one of the annotations is not applied or is partially annotated. It is configured to form each annotated training image from an image that has been annotated. An annotator, wherein the formation is The aforementioned images are either not annotated or partially annotated. A step of generating one or more candidate image annotations for an image, The input identifies one or more parts of the one or more candidate image annotations. The steps to take, The stems from which the annotated training images are obtained from at least one of the aforementioned portions. An annotator, which is executed by the app, An image annotation system equipped with [specific features / features].

18. An image annotation method, a) At least one annotation is missing or partially annotated. The steps include receiving or accessing the applied image, b) The annotation is not applied or is partially applied. The step of forming each annotated training image from the image, wherein the formation is: For images that have not been annotated, one or more candidate image annotations To generate n, The input identifies one or more parts of the one or more candidate image annotations. To do, Forming the annotated training image from at least one of the aforementioned portions. The steps are performed by and Methods that include...

19. When executed by one or more processors, any one of claims 9 to 16 and 18 Computer program code configured to perform one of the following methods.

20. A computer-readable medium comprising the computer program code described in claim 19.

Images