Method and server for generating user-customized medical image segmentation models

Abstract

The present invention is a method performed by a processor of a medical image model generation device, and is configured to include the steps of: providing a model generation interface including a segmentation model selection list for predicting any one target region; determining any one of the segmentation models selected by a user through the model generation interface as a learning model; acquiring training data for a medical image to be input to the learning model and a target region specified in the medical image; and re-training the learning model based on the medical image to generate a segmentation model configured to predict a different target region using a 3D medical image as input.

Description

[Technical Field]

[0001] The present invention relates to a method and server for generating a user-customized medical image segmentation model. [Background technology]

[0002] Generally, to determine the presence or absence of a disease in a target area of ​​a subject, medical images of the target area of ​​the subject taken through imaging medical examinations (e.g., X-ray, ultrasound, CT (Computer Tomography), Angiography, Positron Emission Computed Tomography (PET-CT), Single Photon Emission Computed Tomography (SPECT-CT), Magnetic Resonance Imaging (MRI), etc.) are used. Medical staff have performed diagnoses by visually checking the medical images to identify the target area and determining the presence or absence of a disease in the target area (e.g., the presence or absence of a tumor).

[0003] However, noise may exist in the medical image itself due to the performance of the imaging device, the movement of the patient, etc. If the target region (e.g., an organ or a tumor in an organ) in the medical image is identified based solely on the medical staff's findings, differences in opinion may occur depending on the medical staff's skills and experience, even for the same medical image.

[0004] Therefore, a method of predicting a target region from a medical image using an artificial neural network model that has been trained to predict a target region based on the medical image has been widely used. However, the prediction results of an artificial neural network model can vary depending on the direction and the type of training data used. Therefore, a method is needed to generate an artificial neural network model that suits the intentions of medical staff who intend to use the artificial neural network model, but such a method does not currently exist.

[0005] The Background of the Invention has been prepared to facilitate a better understanding of the present invention and should not be construed as an admission that the matter described in the Background of the Invention exists as prior art. Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, a method is required for medical staff (hereinafter referred to as "user") to generate a customized medical image segmentation model by training an artificial neural network model using target region segmentation results for medical images that the user has.

[0007] As a result, the inventors of the present invention have developed a method for generating a user-customized artificial neural network model by learning training data held by the user based on the structure of an artificial neural network model capable of segmenting medical images.

[0008] The objects of the present invention are not limited to those mentioned above, and other objects not mentioned above will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, a method for generating a user-customized medical image segmentation model according to an embodiment of the present invention is provided. The method is performed by a processor of a medical image model generation device and includes the steps of: providing a model generation interface including a segmentation model selection list for predicting any one target region; determining any one of the segmentation models selected by a user through the model generation interface as a training model; acquiring training data for a medical image to be input to the training model and a target region area specified in the medical image; and re-training the training model based on the medical image to generate a segmentation model configured to predict a different target region using a 3D medical image as input.

[0010] According to a feature of the present invention, after the step of generating the segmentation model, the method may further include a step of inputting 3D medical images including at least two target regions into the generated segmentation model and a pre-stored segmentation model to predict multiple target regions.

[0011] According to another feature of the present invention, after the step of generating the segmentation model, the method may further include the steps of obtaining prediction data based on previously stored medical images using the generated segmentation model, and comparing the prediction data with the training data to verify the generated segmentation model.

[0012] According to another feature of the present invention, after the step of verifying the segmentation model, the method may further include a step of providing a model usage interface including a search area for specifying at least one of subject conditions, target area, and type of segmentation model capable of predicting the target area.

[0013] According to another feature of the present invention, the step of determining the learning model may further include a step of providing a learner adjustment area for adjusting at least one of hyperparameters, configuration, image coordinate system of input data, and trained weights of the learning model through the model generation interface.

[0014] According to another feature of the present invention, after the step of acquiring the medical image, the method may further include the steps of extracting a name for the target area matched to the learning data, and labeling the extracted name as a new name that can be learned based on a pre-stored tag list for each target area.

[0015] According to another aspect of the present invention, the step of generating the segmentation model may further include the step of inputting labeled medical images into the learning model.

[0016] According to another feature of the present invention, the method may further include, after the step of acquiring the training data, generating a plurality of sub-volume data using the medical image, and the segmentation model may be a model that is trained to determine a region corresponding to a target region using the plurality of sub-volume data as input.

[0017] According to another aspect of the present invention, the sub-volume data may include data on a movement direction of voxels constituting the sub-volume data with respect to any one of the axes.

[0018] In order to solve the above-mentioned problems, according to another embodiment of the present invention, there is provided a user-customized medical image segmentation model generation server, which is configured to provide a model generation interface including a segmentation model selection list for predicting any one target region, determine any one of the segmentation models selected by a user through the model generation interface as a training model, acquire training data for a medical image to be input to the training model and a target region area specified in the medical image, and train the training model based on the medical image to generate a segmentation model configured to predict a different target region using a 3D medical image as input.

[0019] Further details of the embodiments are included in the detailed description and drawings. [Effects of the Invention]

[0020] The present invention can generate an artificial neural network model for medical image segmentation that matches a user's diagnostic direction. For example, a user can select a target region to predict from various target regions according to user customization, and a medical image segmentation service system including a medical image segmentation model selectively trained by the user can be constructed.

[0021] The present invention automatically performs a process of verifying training data provided by a user through an interface for generating an artificial neural network model, thereby enabling a user to easily generate a customized artificial neural network model without the user having to standardize data one by one for training the artificial neural network model. For example, the present invention performs matching for using training data arbitrarily labeled by a user as input data, thereby improving the efficiency of generating a customized artificial neural network model and increasing the accessibility of generating an artificial neural network model.

[0022] The present invention allows hospitals that store a large number of diverse medical images to generate a user-customized medical image segmentation model, thereby enabling each medical staff device in the hospital to easily predict target areas such as organs and lesions in medical images through the unique medical image segmentation model.

[0023] The present invention can accurately predict the location of a target area in a medical image and visually display it to help medical staff diagnose the target area. The present invention goes beyond inputting only the 2D images constituting the medical image into a segmentation model that predicts the target area, and can improve the prediction accuracy of the model by providing information on before and after images of sequentially captured 2D images.

[0024] The effects of the present invention are not limited to the above-mentioned examples, and various other effects are included within the scope of the present invention. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a block diagram illustrating a configuration of a system to which a medical image segmentation model according to an embodiment of the present invention is applied. [Figure 2a] 1 is a block diagram illustrating a system for generating a user-customized medical image segmentation model according to an embodiment of the present invention. [Figure 2b] 1 is a schematic diagram illustrating a system for generating a user-customized medical image segmentation model according to an embodiment of the present invention; [Figure 3] FIG. 1 is a block diagram showing the configuration of a medical staff device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram illustrating the configuration of a medical image segmentation model generation server according to an embodiment of the present invention. [Figure 5] 1 is a schematic flowchart of a method for generating a medical image segmentation model according to an embodiment of the present invention. [Figure 6a] 10 is a diagram illustrating an example of a user interface screen for generating a medical image segmentation model according to an embodiment of the present invention. [Figure 6b] 10 is a diagram illustrating an example of a user interface screen for generating a medical image segmentation model according to an embodiment of the present invention. [Figure 7a] 10 is a diagram illustrating an example of a user interface screen for generating and using a medical image segmentation model according to an embodiment of the present invention. [Figure 7b] 10 is a diagram illustrating an example of a user interface screen for generating and using a medical image segmentation model according to an embodiment of the present invention. [Figure 7c] 10 is a diagram illustrating an example of a user interface screen for generating and using a medical image segmentation model according to an embodiment of the present invention. [Figure 8] 1 is a schematic diagram illustrating a method for training a medical image segmentation model according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0026] The advantages and features of the present invention, and methods for achieving them, will become more apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully convey the scope of the invention to those skilled in the art to which the present invention pertains. The present invention is defined only by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used to refer to like elements.

[0027] In this document, the terms "have," "can have," "include," or "can include" refer to the presence of a given feature (e.g., a value, function, operation, or component such as a part) and do not exclude the presence of additional features.

[0028] In this document, expressions such as "A or B," "at least one of A and / or B," or "one or more of A and / or B" include all possible combinations of the items listed together. For example, "A or B," "at least one of A and B," or "at least one of A or B" can refer to (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0029] Terms such as "first," "second," "first," or "second" used herein may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit the corresponding component. For example, a first user device and a second user device may refer to different user devices regardless of order or importance. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component, without departing from the scope of the rights described herein.

[0030] When a component (e.g., a first component) is referred to as being "operatively or communicatively coupled with" or "connected to" another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component or may be coupled through another component (e.g., a third component). In contrast, when a component (e.g., a first component) is referred to as being "directly coupled with" or "directly connected to" another component (e.g., a second component), it should be understood that there is no other component (e.g., a third component) between the component and the other component.

[0031] As used herein, the phrase "configured to" may be used in various ways, depending on the context, such as "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily refer to hardware that is "specifically designed to." Instead, in some contexts, the phrase "apparatus configured to" may mean that the apparatus, together with other devices or components, is "capable of." For example, the phrase "a processor configured to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the operations, or to a general-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing one or more software programs stored in a memory device.

[0032] The terms used in this document are merely used to describe particular embodiments and may not be intended to limit the scope of other embodiments. A singular expression may include a plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the art described in this document. Terms used in this document that are defined in a general dictionary may be interpreted to have the same or similar meaning as the meaning they have in the context of the relevant art, and unless explicitly defined in this document, they should not be interpreted in an idealized or overly formal sense. In some cases, even terms defined in this document may not be interpreted to exclude embodiments of this document.

[0033] The features of the various embodiments of the present invention may be partially or fully combined or combined with each other, and various technical interlocking and driving mechanisms are possible, as will be fully understood by those skilled in the art. Each embodiment may be implemented independently of the others, or may be implemented together in a related relationship.

[0034] For clarity of interpretation of this specification, the following defines terms used in this specification. The term "medical image" as used herein may refer to a two-dimensional image composed of multiple cuts (or slices) of a subject. Specifically, the medical image may be an enhanced or non-enhanced CT image generated in accordance with the DICOM standard. For example, the medical image may include a head and neck image including the entire area from the skull vertex to the lung apex, a chest image including the entire area from the thyroid to the liver dome, an abdominal image including the entire L1 spine at a position 3 cm away from the liver dome in the cranial direction, and a pelvic image including the entire ischium at a position 3 cm away from the L1 spine in the cranial direction.

[0035] The term "segmentation model" as used herein may refer to a model trained to predict a region corresponding to a target site using a 3D medical image as input. Specifically, the segmentation model in this specification may refer to a model trained to predict whether each 3D slab or cube corresponds to a target site. For example, the segmentation model may be a model trained to predict whether each of a plurality of subvolume data corresponds to a target site such as the head, neck, chest, abdomen, or pelvis. As another example, the segmentation model may be a model trained to predict whether each of a plurality of subvolume data corresponds to one of a plurality of organs arranged sequentially in a part of a subject's body. As another example, the segmentation model may be a model trained to predict whether each of a plurality of volume data corresponds to a region of an organ, bone, or muscle that is suspected of being diseased. Here, the suspected disease region may be a region suspected of being a tumor, and the medical image may be a medical image of the head and neck, chest, abdomen, or pelvis.

[0036] In various embodiments, the segmentation model may be composed of at least one model or two or more ensemble models of CNN (Convolutional Neural Network)-based VGG net, R, DenseNet, FCN (Fully Convolutional Network) with an encoder-decoder structure, DNN (deep neural network) such as SegNet, DeconvNet, DeepLAB V3+, U-net, SqueezeNet, Alexnet, ResNet18, MobileNet-v2, GoogLeNet, Resnet50, Resnet101, and Inception-v3.

[0037] In various embodiments, the segmentation model may be a model trained using training data having different slice thicknesses depending on the type of medical image. In the present invention, the slice thickness can be unified to a reference value by resampling the slices and performing training without performing a preprocessing process of increasing or decreasing the slice thickness. Specifically, the segmentation model uses 256x256x16 slab data or 96x96x96 cubic data as subvolume data. The segmentation model can train without image loss of the medical image by performing different numbers of bootstrappings depending on the slice thicknesses input to the training data. For example, if slices having a thickness of 3 mm are input in the initial training, the segmentation model can perform training by stacking the slices to generate subvolume data in 48 mm units. If slices having a thickness of 5 mm are input in the next initial training, the segmentation model can perform training by stacking the slices to generate subvolume data in 90 mm units. This training can be performed a specified number of times (epochs).

[0038] The present invention will now be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. FIG. 1 is a block diagram showing the configuration of a system to which a medical image segmentation model according to an embodiment of the present invention is applied.

[0039] 1, the medical image segmentation model generation system 1000 may be a system for generating a segmentation model trained to segment only a region of interest in a medical image. Here, segmenting a region of interest may be understood as displaying a region corresponding to the region of interest in a medical image. To this end, the medical image segmentation model generation system 1000 may include an image capturing device 100 for capturing an image of a part of a subject's body, and a medical staff device 200 for identifying a region corresponding to the region of interest, generating a medical image segmentation model for segmenting the region corresponding to the region of interest in a medical image, and diagnosing the subject.

[0040] The imaging device 100 is a device capable of obtaining medical images of a target region of a subject 10, such as a human or animal, and may include a cylindrical bore 110 into which the subject 10 is carried and a transport device 130 on which the subject 10 is seated and which carries the subject 10 inside. Here, the target region may include various organs, bones, muscles, etc. of the subject 10, such as the brain, neck, chest, abdomen, pelvis, etc. The imaging device 100 can obtain medical images of the subject 10 by irradiating the subject with X-rays capable of projecting the subject 10.

[0041] In various embodiments, the imaging device 100 can acquire a gray scale or RGB two-dimensional image, a single still image, a video consisting of multiple cuts, etc. as a medical image for predicting the area corresponding to the target area.

[0042] The medical staff device 200 may be a device that generates a medical image segmentation model based on medical images captured using the image capturing device 100 and displays the medical image segmentation results. For example, the medical staff device 200 may include a smartphone, a tablet PC (Personal Computer), a notebook computer, a PC, etc. The medical staff device 200 may newly learn a medical image segmentation model based on medical images stored in the device and regions of interest segmented according to the diagnosis of a user (medical staff). The medical staff device 200 may determine a region of interest based on a certain plane using the learned model.

[0043] In various embodiments, the medical staff device 200 may be configured to install or execute a web or mobile application or program provided by the medical image segmentation model generation server 300. The medical staff device 200 can display only a portion of the medical image corresponding to a target region through the web or mobile application or program, and can independently perform a series of medical image segmentation model generation and medical image segmentation model calculations performed by the medical image segmentation model generation server 300.

[0044] FIG. 2a is a block diagram illustrating a system for generating a user-customized medical image segmentation model according to an embodiment of the present invention. 2a, the medical image segmentation model generation system 1000 may further include a medical image segmentation model generation server 300 that can generate a user-customized medical image segmentation model at the request of the medical staff device 200. The medical image segmentation model generation server 300 may be a server that acquires the medical image and learning conditions from the medical staff device 200 and predicts a region corresponding to a target region in the medical image using the medical image segmentation model. For example, the medical image segmentation model generation server 300 may include a general-purpose computer, a laptop, a data server, etc. Here, the learning conditions may include hyperparameters of the learning model, a layer configuration of the medical image segmentation model, an image coordinate system of input data corresponding to the medical image, trained weights, etc.

[0045] The medical image segmentation model generation server 300 may perform deep learning on medical images through interaction with the medical staff device 200 to generate a user-customized medical image segmentation model. To this end, the medical image segmentation model generation server 300 may provide various interfaces to the medical staff device 200, and the medical staff device 200 may generate a medical image segmentation model based on its own medical images. For example, the medical staff device 200 may display user interface screens such as a model generation interface for generating a medical image segmentation model, a data load interface for loading medical images captured by the image capture device 100 or already stored in the medical staff device 200, and a data management interface for managing learning data. As another example, the medical staff device 200 may display a model usage interface screen for displaying medical image segmentation results segmented using the medical image segmentation model.

[0046] In relation to this, FIG. 2b is a schematic diagram illustrating a user-customized medical image segmentation model generation system according to one embodiment of the present invention. 2b, the medical staff device 200 can select a segmentation model to be generated from target region segmentation models through a model generation interface provided by the medical image segmentation model generation server 300. For example, the medical staff device 200 can select whether to learn a segmentation model based on the user's medical image for one of the target regions, namely, the head, neck, chest, abdomen, pelvis, eyeball, heart, breast, and bowels. As another example, the medical staff device 200 may select whether to learn a segmentation model based on the user's medical image for one of the target regions in the neck, namely, the thyroid gland, oral cavity, mandible, left submandibular gland, right submandibular gland, pharynx, larynx, left parotid gland, right parotid gland, left temporomandibular joint, right temporomandibular joint, left brachial plexus, and right brachial plexus. The segmentation model selected by the user may be defined as a learning model, and the medical image segmentation model generation server 300 may acquire learning data for the medical image from the medical staff device 200 and re-learn the learning model based on the acquired data to generate a new segmentation model. Here, the learning data may include medical images, types of medical images, target region areas specified in the medical images, labels or tags according to the types of target regions, etc., and the new segmentation model may be a model whose structure, weighting, etc. are modified from an existing learning model according to learning conditions specified by the user.

[0047] The medical image segmentation model generation server 300 can input a medical image acquired from the medical staff device 200 into the newly generated segmentation model and the previously stored segmentation model for each target region, and output the region corresponding to the target region in the medical image. However, the medical staff device 200 can input the medical image only into the newly generated segmentation model and output only the region corresponding to the target region in the medical image, as selected by the medical staff device 200. In this way, the medical staff device 200 can acquire a segmentation model based on the learning data it provides, and by using this, can acquire a segmentation result that matches its diagnostic intent when a new medical image is input.

[0048] In various embodiments, the medical image segmentation model generation server 300 may provide a medical image segmentation result that displays a region corresponding to a target region in a medical image through a segmentation model usage interface screen. Here, the medical image segmentation result may be provided in a manner that highlights a region corresponding to a target region in each of a plurality of slices based on one of an axial plane, a coronal plane, and a sagittal plane in different colors.

[0049] Meanwhile, in the present invention, the medical image segmentation model generated by the medical image segmentation model generation server 300 can predict a region corresponding to the target region in 3D volume data, rather than predicting a region corresponding to the target region in each of multiple 2D slices constituting the medical image. Specifically, the medical image segmentation model generation server 300 can generate multiple subvolume data using multiple slices and predict a region corresponding to the target region based on the subvolume data. Here, the subvolume data can be generated by stacking multiple slices constituting the medical image to match the pre-stored height and dividing the slices in a direction perpendicular to the slice plane. For example, the subvolume data can be slab data of 256×256×16 size, or as another example, cubic data of 96×96×96 size. In addition, voxels constituting the subvolume data each have a size of 2×2×3 mm. 3 , 0.7×0.7×1.0mm 3 The size of the voxel can be adjusted to 0.7 x 0.7 x 1.0 mm. 3 From 2 x 2 x 3 mm 3 The size can vary within the range of

[0050] In the present invention, the plurality of subvolume data includes data on the movement direction of voxels constituting the subvolume data based on any one axis, thereby improving the segmentation accuracy of the segmentation model compared to predicting a target region in slice units. For example, if the subvolume data is slab data, each subvolume data may include data on the movement direction of voxels in a direction perpendicular to an axial plane. As another example, if the subvolume data is cubic data, each subvolume data may include data on the movement direction of voxels in at least one of an axial plane, a coronal plane, and a sagittal plane.

[0051] As a result, the medical image segmentation model generation server 300 can input multiple subvolume data into a segmentation model trained to predict one target region using a 3D medical image as input, and determine a region corresponding to the target region in the multiple subvolume data. In other words, the medical image segmentation model generation server 300 can predict whether each of the multiple subvolume data is a region corresponding to the target region. For example, the medical image segmentation model generation server 300 can predict whether each of the multiple subvolume data is a region corresponding to the target region, such as the head, neck, chest, abdomen, or pelvis.

[0052] In various embodiments, the medical image segmentation model generation server 300 may include multiple segmentation models that can predict specific regions of the head, neck, chest, abdomen, and pelvis, and may group medical images by target region to use any one of the multiple segmentation models.

[0053] So far, a description has been given of the medical image segmentation model generation system 1000 according to one embodiment of the present invention. According to the present invention, the medical image segmentation model generation system 1000 generates a medical image segmentation model based on the medical image stored in the medical staff device 200 and segments the medical image using the model, thereby increasing the convenience and accessibility of the system for medical staff and assisting medical staff in diagnosing target areas.

[0054] In the following, with reference to FIG. 3, a medical staff device 200 for generating a user-customized medical video segmentation model will be described. FIG. 3 is a block diagram showing the configuration of a medical staff device according to an embodiment of the present invention.

[0055] 3, the medical staff device 200 may include a memory interface 210, one or more processors 220, and a peripherals interface 230. The various components within the medical staff device 200 may be coupled by one or more communication buses or signal lines.

[0056] The memory interface 210 is connected to the memory 250 and can transmit various data to the processor 220. Here, the memory 250 can include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., SD or XD memory), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and a blockchain database.

[0057] In various embodiments, the memory 250 may store a web / app application or program for segmenting only the region corresponding to the target region in the medical image using a user-customized medical image segmentation model. The memory 250 may also store identification information of the subject (e.g., age, gender, presence or absence of disease), the medical image of the subject, the region corresponding to the target region in the medical image, labels or tags for the medical image, etc.

[0058] In various embodiments, the memory 250 may store at least one of an operating system 251, a communications module 252, a graphic user interface module (GUI) 253, a sensor processing module 254, a telephony module 255, and an application module 256. Specifically, the operating system 251 may include instructions for processing basic system services and instructions for performing hardware operations. The communications module 252 may communicate with at least one of another device, computer, and server. The graphic user interface module (GUI) 253 may process a graphic user interface. The sensor processing module 254 may process sensor-related functions (e.g., processing audio input received through one or more microphones 292). The telephony module 255 may process telephony-related functions. The application module 256 may perform various functions of user applications, such as electronic messaging, web browsing, media processing, searching, imaging, and other processing functions. Additionally, the medical staff device 200 may store in memory 250 one or more software applications 256-1, 256-2 associated with one type of service (eg, a medical image segmentation model generation application).

[0059] In various embodiments, memory 250 can store a digital assistant client module 257 (hereinafter, DA client module), which can store commands for performing client-side functions of the digital assistant and various user data 258 (e.g., user-customized vocabulary data, preference data, user electronic address book, etc.).

[0060] Meanwhile, the DA client module 257 can obtain user voice input, text input, touch input, and / or gesture input through various user interfaces (e.g., the I / O subsystem 240) provided on the medical staff device 200.

[0061] The DA client module 257 can also output audiovisual and tactile data. For example, the DA client module 257 can output data consisting of a combination of at least two or more of voice, sound, notification, text message, menu, graphic, video, animation, and vibration. In addition, the DA client module 257 can communicate with a digital assistant server (not shown) using the communication subsystem 280.

[0062] In various embodiments, the DA client module 257 can collect additional information about the surrounding environment of the medical staff device 200 from various sensors, subsystems, and peripheral devices to form a context associated with the user input. For example, the DA client module 257 can provide context information along with the user input to the digital assistant server to infer the user's intent. Here, context information that may accompany the user input can include sensor information such as light, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, the context information can include the physical state of the medical staff device 200 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc.). As yet another example, the context information can include information related to the software state of the medical staff device 200 (e.g., processes running on the medical staff device 200, installed programs, past and present network activity, background services, error logs, resource usage, etc.).

[0063] In various embodiments, memory 250 may include additional or deleted instructions, and medical staff device 200 may also include additional components or omit some components from those shown in FIG.

[0064] The processor 220 can control the overall operation of the medical staff device 200, run applications or programs stored in the memory 250 to generate a medical image segmentation model, and execute various commands to implement a user interface that allows the medical staff to view an area corresponding to a target site in the medical image or to check the predicted results provided through the segmentation model.

[0065] The processor 220 may correspond to a computing device such as a central processing unit (CPU) or an application processor (AP), and may be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) that integrates various computing devices that perform machine learning, such as a neural processing unit (NPU).

[0066] In various embodiments, the processor 220 can generate a medical image segmentation model through an application or program provided by the medical image segmentation model generation server 300, and output a result displaying an area corresponding to the target area in a newly acquired medical image using the generated medical image segmentation model.

[0067] In various embodiments, the processor 220 may provide a user with a model generation interface including a segmentation model selection list for predicting any one target region, determine any one of the segmentation models selected by the user through the model generation interface as a training model, acquire training data for the medical image input to the training model and the target region matched to the medical image, and train the training model again based on the medical image to generate a segmentation model configured to predict different target regions using a 3D medical image as input. Specifically, the processor 220 may generate multiple sub-volume data using the medical image and use the sub-volume data as input data.

[0068] The peripheral interface 230 may be connected to various sensors, subsystems, and peripheral devices to provide data to enable the medical staff device 200 to perform various functions. Here, when the medical staff device 200 performs a certain function, it may be understood that the function is performed by the processor 220.

[0069] The peripheral interface 230 can receive data from a motion sensor 260, an illumination sensor (light sensor) 261, and a proximity sensor 262, through which the medical staff device 200 can perform orientation, light, and proximity sensing functions, etc. As another example, the peripheral interface 230 can receive data from other sensors 263 (positioning systems—GPS receivers, temperature sensors, biometric sensors), through which the medical staff device 200 can perform functions associated with the other sensors 263.

[0070] In various embodiments, the medical staff device 200 may include a camera subsystem 270 coupled to the peripheral interface 230 and an optical sensor 271 coupled thereto, through which the medical staff device 200 can perform various imaging functions such as taking photographs and recording video clips.

[0071] In various embodiments, medical staff device 200 can include a communications subsystem 280 coupled to peripheral interface 230. Communications subsystem 280 can be configured with one or more wired / wireless networks and can include various communications ports, radio frequency transceivers, and optical transceivers.

[0072] In various embodiments, the medical staff device 200 includes an audio subsystem 290 coupled to the peripheral interface 230, which may include one or more speakers 291 and one or more microphones 292, enabling the medical staff device 200 to perform voice-activated functions, such as voice recognition, voice duplication, digital recording, and telephone functions.

[0073] In various embodiments, the medical staff device 200 can include an I / O subsystem 240 coupled to the peripheral interface 230. For example, the I / O subsystem 240 can control a touchscreen 243 included in the medical staff device 200 through a touchscreen controller 241.

[0074] For example, the touchscreen controller 241 may detect a user's touch and movement or the cessation of touch and movement using any one of a number of touch sensing technologies, such as capacitive, resistive, infrared, surface acoustic wave technology, a proximity sensor array, etc. As another example, the I / O subsystem 240 may control other input / control devices 244 included in the medical staff device 200 through an other input controller 242. As one example, the other input controller 242 may control one or more buttons, rocker switches, thumbwheels, an infrared port, a USB port, and a pointer device such as a stylus.

[0075] So far, the medical staff device 200 according to one embodiment of the present invention has been described. According to the present invention, by generating a medical staff-specific medical image segmentation model using the medical staff device 200, it is possible to identify the location and size of a target region to be diagnosed in a medical image, such as an organ, bone, or tumor, in accordance with the intention of the medical staff, thereby enabling an accurate diagnosis of the subject's health condition.

[0076] Hereinafter, with reference to FIG. 4, a medical video segmentation model generation server 300 will be described, which generates a medical video segmentation model, verifies the model, and then provides a service. FIG. 4 is a block diagram showing the configuration of a medical image segmentation model generation server according to an embodiment of the present invention.

[0077] Referring to FIG. 4, the medical image segmentation model generation server 300 may include a communication interface 310, a memory 320, an I / O interface 330, and a processor 340, and each component may communicate with each other through one or more communication buses or signal lines.

[0078] The communication interface 310 can be connected to the image capture device 100 and the medical staff device 200 through a wired / wireless communication network to exchange data. For example, the communication interface 310 can receive medical images from the image capture device 100 or the medical staff device 200, and can receive a request to generate a medical image segmentation model based on the medical images from the medical staff device 200. As another example, the communication interface 310 can transmit an interface for generating a medical image segmentation model to the medical staff device 200, and can provide a medical image segmentation model trained based on the medical images and training data including the medical images.

[0079] Meanwhile, the communication interface 310 that enables the transmission and reception of such data includes a wired communication port 311 and wireless circuitry 312, where the wired communication port 311 can include one or more wired interfaces, such as Ethernet, Universal Serial Bus (USB), Firewire, etc. The wireless circuitry 312 can transmit and receive data to and from external devices through RF signals or optical signals. In addition, the wireless communication can use at least one of a number of communication standards, protocols, and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

[0080] The memory 320 may store various data used by the medical image segmentation model generation server 300. For example, the memory 320 may store identification information of the image capture device 100 and the medical staff device 200, medical images provided by the medical staff device 200, a tag list for each target region, a segmentation model trained to predict a region corresponding to any one target region using a 3D medical image as input, a newly trained segmentation model, a classification model trained to classify types of target regions using a medical image as input, and the configuration and training data thereof.

[0081] In various embodiments, the memory 320 may include a volatile or non-volatile storage medium capable of storing various data, instructions, and information. For example, the memory 320 may include at least one of the following types of storage media: flash memory, hard disk, multimedia card micro, card-type memory (e.g., SD or XD memory), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0082] In various embodiments, the memory 320 may store configurations for at least one of an operating system 321 , a communications module 322 , a user interface module 323 , and one or more applications 324 .

[0083] The operating system 321 (e.g., an embedded operating system such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers for controlling and managing general system operations (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.

[0084] The communications module 322 can facilitate communication with other devices through the communications interface 310. The communications module 322 can include various software components for processing data received by the wired communications port 311 or wireless circuitry 312 of the communications interface 310.

[0085] The user interface module 323 can receive user requests or inputs from a keyboard, touch screen, keyboard, mouse, microphone, etc. through the I / O interface 330 and provide a user interface on a display.

[0086] The application 324 may include programs or modules configured to be executed by one or more processors 340. Here, applications for generating, training, and computing medical image segmentation models may be implemented on a server farm.

[0087] The I / O interface 330 can connect an input / output device (not shown) of the medical image segmentation model generation server 300, such as at least one of a display, a keyboard, a touch screen, and a microphone, to the user interface module 323. The I / O interface 330 can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module 323 and process commands according to the received input.

[0088] The processor 340 is connected to the communication interface 310, the memory 320, and the I / O interface 330 to control the overall operation of the medical image segmentation model generation server 300, and can learn a segmentation model through an application or program stored in the memory 320 and execute various commands to segment a target area in a medical image.

[0089] The processor 340 may correspond to a computing device such as a central processing unit (CPU) or an application processor (AP). The processor 340 may also be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) in which various computing devices are integrated. Alternatively, the processor 340 may include a module for calculating an artificial neural network model, such as a neural processing unit (NPU).

[0090] Hereinafter, a method for generating a medical image segmentation model by the processor 340 of the medical image segmentation model generation server 300 will be described with reference to FIGS. 5 to 7c. FIG. 5 is a schematic flowchart of a method for generating a medical image segmentation model according to an embodiment of the present invention.

[0091] Referring to FIG. 5, the processor 340 may provide a model generation interface including a segmentation model selection list for predicting any one target region (S110). Specifically, the processor 340 may provide the medical staff device 200 with a model generation interface that allows the medical staff device 200 to select a target region to be newly learned and a corresponding segmentation model. The segmentation model selection list includes medical image segmentation models previously learned by an administrator or user (medical staff) and may include data on the name of the segmentation model, the target region that can be predicted (e.g., neck, abdomen, brain, breast), the input data type (dimension) (e.g., 3D slab, cubic data), and the size (model shape) (e.g., 384*384*16*4, 96*96*96*17). In addition, the segmentation model may be a model that predicts the target region using a 3D medical image rather than predicting the target region in a 2D slice-unit medical image.

[0092] After step S110, the processor 340 may determine any one of the division models selected by the user through the model generation interface as the learning model (S120). Specifically, when any one of the division models is selected from the division model selection list, the processor 340 may provide a learner adjustment area through the model generation interface where the structure and input data of the corresponding division model can be changed.

[0093] In this regard, FIGS. 6a and 6b are exemplary illustrations of user interface screens for generating a medical image segmentation model according to one embodiment of the present invention. 6a, the processor 340 may provide a model generation interface including a medical image segmentation model list 11 learned by an administrator or a user to the medical staff device 200. When a user selects one of the segmentation models in the medical image segmentation model list 11 as a learning model, the model generation interface may provide a learner adjustment area 12 in which the structure of the corresponding learning model and specific information of the input data can be changed. For example, the learner adjustment area 12 of the model generation interface may include an area 12-1 for setting hyperparameters such as Contrast, Batch size, Background Ratio, Boostlearn (e.g., Lung_R, Lung_L, Heart, Humerus_R, Humerus_L, Trachea), and Final Shaping (e.g., Normalize, Randomized shaping), an area 12-2 for setting the configuration of the learning model such as Model shape, Growth Rate, Model depth, Number of Layers, and Number of Channels, an area 12-3 for setting the image coordinate system of the medical image, which is the input data, and an area 12-4 for setting trained weights.

[0094] 5, the processor 340 may acquire learning data for the medical image to be input to the learning model and the target region matched to the medical image (S130). Here, the learning data may include labels or tags according to the medical image (e.g., enhanced and non-enhanced CT images), the type of medical image (e.g., 3D slab, cubic data), the target region specified in the medical image (e.g., image displayed in the medical image), the type of target region (e.g., brain, neck, chest, abdomen, pelvis), etc.

[0095] 6A, the processor 340 may provide the medical staff device 200 with a model generation interface that further includes an area 13 displaying a list of the medical images acquired in step S130. The model generation interface may also include a graphic object 14 for performing preprocessing of the medical image, a graphic object 15 for displaying the medical image, a graphic object 16 for selecting a medical image to be used for learning, a graphic object 16 for verifying the medical image to be used for learning, a graphic object 17 for selecting a medical image to be used as a reference for verifying a segmentation model newly generated after learning, and a graphic object 18 for selecting a medical image to be used as a reference for evaluating a segmentation model newly generated after learning.

[0096] Meanwhile, medical images acquired from the medical staff device 200 may be matched to different terms by medical staff even for the same target region. Therefore, when acquiring training data, the processor 340 may extract names for target regions matched to the training data and label the extracted names as new names that can be learned based on a pre-stored tag list for each target region. In this manner, the processor 340 may label medical images included in the training data with tags pre-stored in the memory 320 and input the labeled medical images into the training model, thereby improving the efficiency of classification model training. If the names do not match, the processor 340 may group multiple slices constituting the medical image by target region using the classification model. Specifically, the processor 340 may input the acquired medical image into a classification model trained to classify target regions by type using the medical image, and classify the acquired medical image into at least one target region of the head, neck, chest, abdomen, and pelvis.

[0097] Furthermore, while the division model generated in the present invention uses three-dimensional volume data as input, the processor 340 can generate multiple subvolume data using medical images. Subvolume data can be generated by stacking multiple slices that make up the medical image to match the pre-stored height and dividing them in a direction perpendicular to the plane of the slices. For example, the subvolume data can be slab data of 256 x 256 x 16 size, or as another example, the subvolume data can be cubic data of 96 x 96 x 96 size. In addition, the voxels that make up the subvolume data each have a size of 2 x 2 x 3 mm. 3 , 0.7×0.7×1.0mm 3 The size of the voxel can be adjusted to 0.7 x 0.7 x 1.0 mm. 3 From 2 x 2 x 3 mm 3 The processor 340 can generate a plurality of sub-volume data based on an axis including any one of an axial plane, a coronal plane, and a sagittal plane.

[0098] In various embodiments, the plurality of subvolume data may include data on the movement direction of voxels constituting the subvolume data based on any one axis, thereby improving the accuracy of prediction results compared to predicting the target region in slice units. For example, if the subvolume data is slab data, each subvolume data may include data on the movement direction of voxels in a direction perpendicular to a horizontal plane (axial). As another example, if the subvolume data is cubic data, each subvolume data may include data on the movement direction of voxels in at least one of a horizontal plane (axial), a coronal plane, and a sagittal plane.

[0099] After step S130, processor 340 can retrain the learning model based on the medical image to generate a segmentation model configured to predict a different target region using a 3D medical image as input (S140). Processor 340 can retrain the segmentation model, which has been trained to input a medical image and output region A at the target region, to input a medical image at the user's request and output region A' at the target region. Here, region A and region A' have the same overall prediction results, but may differ in areas where organs and bones are adjacent. As a result, even if the same target region is segmented, the newly trained segmentation model may have a different region for the target region.

[0100] Referring to FIG. 6b, the processor 340 may generate a segmentation model using the learning model and provide the medical staff device 200 with a model generation interface capable of displaying prediction results obtained through the segmentation model. The model generation interface may include an area 21 for inputting and adjusting parameters such as batch size, iteration, and epoch, an area 22 for displaying the learning progress, and an area 23 for displaying the training data used for training. The model generation interface may also include a graphic object 24 for training the segmentation model, an area 25 for displaying a list of segmentation models undergoing training when the corresponding graphic object 24 is selected, which may display the loss function and measure used for training, and a graphic object for selecting whether to visualize and display the prediction results. The train value is updated each time a batch is completed, and the processor 340 may update the valid value once training of the entire training data is completed, i.e., each time an epoch is completed. The model generation interface may also include an area 26 for graphically displaying the train value and valid value for each segmentation model through segmentation model training. For example, the train value may be displayed as a solid line, and the valid value may be displayed as a dotted line. The model generation interface may also include an area 30 for displaying a prediction result of the target region using the segmentation model within the medical image. Here, the model generation interface may include a graphic object 27 for selecting which cross section to use as a reference for displaying the region corresponding to the target region, a graphic object 28 for selecting a display format for the region corresponding to the target region, and a graphic object 29 for setting the opacity of the region corresponding to the target region within the medical image. Here, the model generation interface may highlight and display only the region corresponding to the target region within the medical image, while also displaying prediction results for each of a plurality of sub-volume data.

[0101] In various embodiments, the processor 340 may predict multiple target regions by inputting a 3D medical image including at least two target regions into the segmentation model generated in step S140 and the stored segmentation model. For example, the medical image may be an image in which multiple bones and organs are arranged on the same plane, such as a head and neck image including the entire area from the skull vertex to the lung apex, a chest image including the entire area from the thyroid to the liver dome, an abdominal image including the entire area of ​​the lumbar spine 3 cm away from the liver dome in the head direction, or a pelvic image including the entire area of ​​the ischium 3 cm away from the lumbar spine in the head direction. In this way, the processor 340 can predict multiple target regions by inputting a 3D medical image including two or more target regions into a segmentation model generated through user interaction and a pre-stored segmentation model, respectively. However, the processor 340 can also input the medical image only into a newly generated segmentation model selected by the medical staff device 200 and output only the region of the medical image corresponding to the target region.

[0102] In various embodiments, the processor 340 may verify a previously generated segmentation model before using it. Specifically, the processor 340 may obtain prediction data based on previously stored medical images using the generated segmentation model. The processor 340 may verify the generated segmentation model by comparing the prediction data with training data. Here, the training data to be compared may be data specified by a user through a model generation interface. The processor 340 may determine whether the region of interest displayed in the prediction data and the region of interest included in the training data are identical by a predetermined ratio or more. If the two data are identical by a predetermined ratio or more, the processor 340 may determine the generated segmentation model as an available segmentation model. If the two data are not identical, the processor 340 may provide an alarm to the medical staff device 200 to retrain the learning model.

[0103] In various embodiments, the processor 340 may provide a model usage interface that allows the medical staff device 200 to use pre-trained segmentation models along with validated segmentation models. Specifically, the processor 340 may provide a model usage interface that includes a search area for specifying at least one of the subject condition, the target region, and the type of segmentation model that can predict the target region.

[0104] In this regard, FIGS. 7a to 7c are exemplary diagrams of user interface screens for generating and using a medical image segmentation model according to one embodiment of the present invention. Referring to FIG. 7a, the processor 340 can provide the medical staff device 200 with a model usage interface including an area 31 displaying a list of division models currently being calculated, an area 32 displaying the number of division model uses by date, and an area 33 displaying the division model usage history.

[0105] 7b, when a user intends to use a segmentation model, the processor 340 may provide the medical staff device 200 with a model usage interface including an area 34 displaying usage rules for the segmentation model created by the user, an area 35 for setting the subject's gender, the body area to be segmented, and the segmentation model, and an area 36 for setting any one of the channels included in the segmentation model. Here, the usage rules for the segmentation model may be specified by the user when creating the segmentation model, and the user may thereby segment and predict a target region in a medical image using their desired segmentation model.

[0106] 7c, in response to a user request, the processor 340 may provide the medical staff device 200 with a model usage interface including an area 37 showing detailed information about the channels included in the segmentation model. Specifically, the detailed information about each channel may include the name of the target region, the color displayed in the medical image, the type of the target region (e.g., organ, bone), etc.

[0107] So far, the medical image segmentation model generation server 300 according to one embodiment of the present invention has been described. According to the present invention, the medical image segmentation model generation server 300 can generate and provide a user-customized medical image segmentation model for each hospital that stores a large number of diverse medical images.

[0108] The general flow of the segmentation model learning process performed by the medical image segmentation model generation server 300 will be described below. FIG. 8 is a schematic diagram illustrating a method for learning a segmentation model according to an embodiment of the present invention.

[0109] 8, the medical image segmentation model generation server 300 may determine one of a plurality of segmentation models as a learning model, preprocess a plurality of slices (learning data sets) to be used for learning, and then generate a plurality of subvolume data based on the preprocessing. Here, the preprocessing may correspond to, for example, adjusting the size or resolution of the medical image. The medical image segmentation model generation server 300 may input a plurality of subvolume data to perform learning, and may provide an operator that can correct a difference between the learning result and the actual result.

[0110] As described above, the medical image segmentation model generation server 300 can also learn the segmentation model, and when a prediction request for a medical image is received at the request of an administrator or medical staff, it can load the segmentation model and segment the area corresponding to the target part in the medical image.

[0111] Although one embodiment of the present invention has been described in detail above with reference to the accompanying drawings, the present invention is not necessarily limited to such an embodiment and may be variously modified within the scope of the technical concept of the present invention. Therefore, the disclosed embodiments of the present invention are intended to be illustrative rather than limiting the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. Therefore, the above-described embodiments should be understood to be illustrative in all respects and not limiting. The scope of protection of the present invention should be interpreted by the scope of the following claims, and all technical concepts within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

Claims

1. 1. A method performed by a processor of a medical image model generation device, comprising: providing a model generation interface including a list of segmentation models for predicting any one of the target regions; determining any one of the division models selected by the user through the model generation interface as a training model; acquiring training data for a medical image to be input to the learning model and a target region designated in the medical image; and A method for generating a user-customized medical image segmentation model, comprising: a step of further training the learning model based on the medical image to generate a segmentation model configured to predict different target regions using a 3D medical image as input.

2. After the step of generating the segmentation model, 2. The method of claim 1, further comprising: inputting a 3D medical image including at least two target regions into the generated segmentation model and a pre-stored segmentation model to predict multiple target regions.

3. After the step of generating the segmentation model, Obtaining prediction data based on pre-stored medical images using the generated segmentation model; and The method of claim 1 , further comprising: comparing the predicted data with the training data to validate the generated segmentation model.

4. After the step of verifying the division model, The method for generating a user-customized medical image segmentation model according to claim 3, further comprising a step of providing a model usage interface including a search area for specifying at least one of subject conditions, target area, and type of segmentation model capable of predicting the target area.

5. The step of determining the learning model includes:

2. The method of claim 1, further comprising providing a learner adjustment area for adjusting at least one of hyper parameters, a configuration, a video coordinate system of input data, and trained weights of the learning model through the model generation interface.

6. After the step of acquiring the medical image, Extracting names for target regions matched to the training data; and The method of claim 1 , further comprising: labeling the extracted name based on a pre-stored tag list for each target region into a new learnable name.

7. The step of generating the segmentation model includes: The method for generating a user-customized medical image segmentation model according to claim 6 , further comprising the step of: inputting labeled medical images into the learning model.

8. After the step of acquiring the learning data, generating a plurality of sub-volume data using the medical image; The division model is The method for generating a user-customized medical image segmentation model according to claim 1 , wherein the model is trained to determine a region corresponding to a target region using the plurality of sub-volume data as input.

9. The sub-volume data is The method of claim 8 , further comprising data on a movement direction of voxels constituting the sub-volume data with respect to any one of the axes.

10. communication interface; memory; and a processor operatively coupled to the communication interface and the memory; The processor: A user-customized medical image segmentation model generation server configured to provide a model generation interface including a segmentation model selection list for predicting any one target region, determine any one of the segmentation models selected by a user through the model generation interface as a learning model, acquire learning data for a medical image input to the learning model and a target region specified in the medical image, and further train the learning model based on the medical image to generate a segmentation model configured to predict a different target region using a 3D medical image as input.

11. The processor:

11. The user-customized medical image segmentation model generation server of claim 10, further configured to, after generating the segmentation model, input 3D medical images including at least two or more target regions into the generated segmentation model and a pre-stored segmentation model to predict multiple target regions.

12. The processor:

11. The user-customized medical image segmentation model generation server of claim 10, further configured to, after generating the segmentation model, obtain prediction data based on previously stored medical images using the generated segmentation model, and compare the prediction data with the training data to verify the generated segmentation model.

13. The processor: The user-customized medical image segmentation model generation server of claim 12, further configured to provide a model usage interface including a search area for specifying at least one of subject conditions, target region, and type of segmentation model capable of predicting the target region after verifying the segmentation model.

14. The processor:

11. The user-customized medical image segmentation model generation server of claim 10, further configured to provide a learner adjustment area for adjusting at least one of hyper parameters, configuration, an image coordinate system of input data, and trained weights of the learning model through the model generation interface.

15. The processor:

11. The user-customized medical image segmentation model generation server of claim 10, further configured to: extract names for target regions matched to the learning data after acquiring the medical image; and label the extracted names as new names that can be learned based on a pre-stored tag list for each target region.

16. The processor: The user-customized medical video segmentation model generation server of claim 15 , further configured to input labeled medical videos into the learning model.

17. The processor: After acquiring the training data, the method is further configured to generate a plurality of sub-volume data using the medical image, The division model is The user-customized medical image segmentation model generation server according to claim 10 , wherein the model is trained to determine a region corresponding to a target region using the plurality of sub-volume data as input.

18. The sub-volume data is The user-customized medical image segmentation model generation server of claim 17 , further comprising data on a movement direction of voxels constituting the sub-volume data based on any one of the axes.

Images