Method for segmenting medical image and device for performing same

Abstract

The present invention relates to a method performed by a processor of a device for segmenting a medical image, the method comprising the steps of: acquiring a medical image of a subject; generating a plurality of pieces of sub-volume data by using the medical image; determining a region, corresponding to a target part, in the plurality of pieces of sub-volume data by inputting the plurality of pieces of sub-volume data to a prediction model trained to predict the target part by using a three-dimensional medical image as an input; and providing the region, corresponding to the target part, in the medical image.

Description

Medical image segmentation method and apparatus for performing the same

[0001] The present invention relates to a medical image segmentation method and a device for performing the same.

[0002] Typically, medical images of the target area of ​​a subject are taken through imaging tests (e.g., X-ray, ultrasound, computed tomography (CT), angiography, positron emission tomography (PET-CT), single-photon emission computed tomography (SPECT-CT), magnetic resonance imaging (MRI), etc.) to determine whether there is a disease in the target area of ​​the subject. Medical staff have been making a diagnosis by visually checking the medical images to identify the target area area and determine whether there is a disease in the target area area (e.g., whether there is a tumor).

[0003] However, medical images themselves can contain noise due to factors such as the performance of the imaging device and the subject's movements. If the target area of ​​a medical image (e.g., an organ or tumor within an organ) is identified solely based on the medical team's opinion, even within the same medical image, differences in opinion may arise depending on the team's skill and experience.

[0004] Accordingly, a method of predicting target areas from medical images using an artificial neural network model trained to predict target areas based on medical images is widely used.

[0005] However, the artificial neural network model is configured to predict the area where the three-dimensional target area exists using two-dimensional slice-unit medical images, so there is a problem that the accuracy and reliability of the prediction results are low for some slices.

[0006] The background technology of the invention has been prepared to facilitate a better understanding of the present invention. It should not be construed as an admission that the matters described in the background technology of the invention constitute prior art.

[0007] A model has been developed to segment only the target area using a conventional 3D medical image, but the size of the data for training the model is too large, limiting learning.

[0008] Accordingly, a method is required to improve the accuracy and reliability of prediction results while minimizing the amount of computation in the artificial neural network model so as to resolve the uncertainty of the artificial neural network model.

[0009] As a result, the inventors of the present invention have constructed a method capable of accurately segmenting a target area while minimizing the amount of computation by inputting medical image data capable of three-dimensional configuration into an artificial neural network model that predicts the target area.

[0010] The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0011] In order to solve the above-described problem, a medical image segmentation method according to an embodiment of the present invention is provided. The method is a method performed by a processor of a medical image segmentation device, and is configured to include the steps of: acquiring a medical image of a subject; generating a plurality of sub-volume data using the medical image; inputting the plurality of sub-volume data into a prediction model trained to predict a target region using a 3D medical image as input, thereby determining an area corresponding to the target region from the plurality of sub-volume data; and providing an area corresponding to the target region within the medical image.

[0012] According to a feature of the present invention, the sub-volume data may include data on the movement direction of voxels constituting the sub-volume data with respect to one axis.

[0013] According to another feature of the present invention, the step of generating the sub-volume data may be a step of stacking a plurality of slices constituting the medical image to a pre-stored height and dividing the slices in a direction perpendicular to the plane of the slices to generate a plurality of sub-volume data.

[0014] According to another feature of the present invention, before generating the sub-volume data, the method may further include a step of inputting the medical image into a classification model trained to classify the type of target region using the medical image as input, a step of determining the type of target region included in each of a plurality of slices constituting the medical image, and a step of grouping the plurality of slices by target region based on the classification result.

[0015] According to another feature of the present invention, the step of providing an area corresponding to the target area may be a step of providing a user interface screen including a segmentation model selection area for selecting a target area to be segmented in a medical image and a segmentation result display area for displaying an area corresponding to the target area.

[0016] According to another feature of the present invention, the step of providing an area corresponding to the target area may further include a step of combining volume data having a probability value corresponding to the target area greater than a preset value from each of the plurality of sub-volume data using the prediction model and displaying the combined volume data on the user interface screen.

[0017] According to another feature of the present invention, the step of providing an area corresponding to the target area may further include the step of displaying each of the plurality of sub-volume data on the user interface screen with different transparency according to a probability value corresponding to the target area using the prediction model.

[0018] According to another feature of the present invention, the step of generating the sub-volume data may be generated based on an axis including any one of an axial plane, a coronal plane, and a sagittal plane, and the step of providing an area corresponding to the target part may further include a step of obtaining any one of a axial plane, a coronal plane, and a sagittal plane for displaying the target part through the user interface screen, and a step of rendering an area corresponding to the target part based on the reference plane.

[0019] According to another feature of the present invention, prior to the step of acquiring the medical image, the method may further include the step of acquiring learning data having different slice thicknesses according to the type of medical image, the step of generating a learning data set by performing different number of bootstrappings according to the thickness of the learning data, and the step of generating the prediction model configured to predict one target region based on the learning data set.

[0020] In order to solve the above-described problem, a medical image segmentation device according to another embodiment of the present invention is provided. The device is configured to acquire a medical image of a subject, generate a plurality of sub-volume data using the medical image, input the plurality of sub-volume data into a prediction model trained to predict a target region using a 3D medical image as input, determine an area corresponding to the target region from the plurality of sub-volume data, and provide an area corresponding to the target region within the medical image of the subject.

[0021] Specific details of other embodiments are included in the detailed description and drawings.

[0022] The present invention goes beyond inputting only two-dimensional images constituting a medical image into a prediction model that predicts a target region, and by providing information on the front and back images of sequentially captured two-dimensional images, the prediction accuracy of the model can be improved. In particular, the present invention can improve the prediction accuracy across the entire two-dimensional image by providing data on changes in pixels (e.g., the direction of movement of a specific pixel) based on a plurality of pixels constituting a two-dimensional image constituting a medical image.

[0023] The present invention can provide meaningful results that are not biased in any one direction by linearly classifying medical images using a learned classification model before predicting a target area, thereby using a prediction model.

[0024] The present invention can help medical staff diagnose a target area by accurately predicting the location of the target area in a medical image and visually displaying it.

[0025] The effects according to the present invention are not limited to those exemplified above, and more diverse effects are included within the present invention.

[0026] FIG. 1 is a block diagram showing the configuration of a medical image segmentation system according to one embodiment of the present invention.

[0027] Figure 2 is a block diagram showing the configuration of a medical device according to one embodiment of the present invention.

[0028] FIG. 3 is a block diagram showing the configuration of a medical image segmentation device according to one embodiment of the present invention.

[0029] Figure 4 is a schematic flowchart of a medical image segmentation method according to one embodiment of the present invention.

[0030] FIG. 5 is a schematic diagram illustrating a medical image segmentation method according to one embodiment of the present invention.

[0031] FIG. 6a and FIG. 6b are exemplary diagrams of user interface screens showing medical image segmentation results according to one embodiment of the present invention.

[0032] FIG. 7 is an example diagram of a user interface screen that uses a prediction model for medical image segmentation according to one embodiment of the present invention.

[0033] Figure 8 is a schematic diagram illustrating a method for learning a prediction model according to one embodiment of the present invention.

[0034] FIG. 9 is a schematic diagram illustrating the operation of a prediction model for segmenting a medical image according to one embodiment of the present invention.

[0035] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0036] In this document, the expressions "has," "may have," "includes," or "may include" indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), but do not exclude the presence of additional features.

[0037] In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A or / and B" can include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" can all refer to cases where (1) at least one A is included, (2) at least one B is included, or (3) at least one A and at least one B are included.

[0038] The terms "first," "second," "first," or "second," as used herein, may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, without limiting the components. For example, a first user device and a second user device may represent different user devices, regardless of order or importance. For example, without departing from the scope of the rights set forth in this document, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0039] When it is said that a component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the component is directly coupled to the other component, or can be connected via another component (e.g., a third component). Conversely, when it is said that a component (e.g., a first component) is "directly coupled to" or "directly connected to" another component (e.g., a second component), it should be understood that no other component (e.g., a third component) exists between the first component and the other component.

[0040] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something that is "specifically designed to" in hardware. Instead, in some contexts, the expression "a device configured to" can mean that the device, together with other devices or components, is "capable of." For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0041] The terms used in this document are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this document. Terms defined in general dictionaries among the terms used in this document may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this document. In some cases, even if a term is defined in this document, it cannot be interpreted to exclude the embodiments of this document.

[0042] The individual features of the various embodiments of the present invention can be partially or wholly combined or combined with each other, and as can be fully understood by those skilled in the art, various technical connections and operations are possible, and each embodiment can be implemented independently of each other or can be implemented together in a related relationship.

[0043] For clarity in the interpretation of this specification, the terms used in this specification are defined below.

[0044] As used herein, the term 'medical image' may be a two-dimensional image composed of multiple cuts (or slices) of a subject. Specifically, the medical image in the present specification may be an enhanced or non-enhanced computed tomography (CT) image generated according to the DICOM standard. For example, the medical image may include a head and neck image including the entire area from the skull vertex to the lung apex, a chest image including the entire area from the thyroid to the liver dome, an abdomen image including the entire lumbar vertebrae (L1 spine) at a location 3 cm away from the liver dome in a cranial direction, and a pelvic image including the entire ischium at a location 3 cm away from the L1 spine in a cranial direction.

[0045] The term 'predictive model' used herein may be a model trained to predict a region corresponding to a target region by inputting a 3D medical image. Specifically, the predictive model in this specification may be a model trained to predict whether each 3D slab or cube corresponds to a target region. For example, the predictive model may be a model trained to predict whether each of a plurality of sub-volume data corresponds to a region corresponding to a target region such as the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), or pelvis (Pelvis). As another example, the predictive model may be a model trained to predict whether each of a plurality of sub-volume data corresponds to a region among a plurality of sequentially arranged organs in a body part of a subject. As yet another example, the predictive model may be a model trained to predict whether each of a plurality of volume data corresponds to a region suspected of being a disease in an organ, bone, or muscle. Here, the area suspected of being diseased may be an area suspected of being a tumor, and the medical images may be medical images taken of the head and neck, chest, abdomen, and pelvis.

[0046] In various embodiments, the prediction model may be composed of at least one model or two or more ensemble models of VGG net, R, DenseNet based on CNN (Convolutional Neural Network), FCN (Fully Convolutional Network) with encoder-decoder structure, SegNet, DeconvNet, DeepLAB V3+, DNN (deep neural network) such as U-net, SqueezeNet, Alexnet, ResNet18, MobileNet-v2, GoogLeNet, Resnet50, Resnet101, Inception-v3.

[0047] In various embodiments, the prediction model may be a model trained with training data having different slice thicknesses depending on the type of medical image. In the present invention, in order to unify the slice thickness according to a reference value, training can be performed by resampling the slices without performing a preprocessing step of increasing or decreasing the slice thickness. Specifically, the prediction model uses slab data of a size of 256x256x16 or cubic data of a size of 96x96x96 as sub-volume data, and by performing different number of bootstrappings depending on the thickness of the slice input as training data, the prediction model can be trained without image loss of the medical image. For example, if a slice with a thickness of 3 mm is input at the beginning of learning, the prediction model can perform learning by stacking slices to create sub-volume data in units of 48 mm, and if a slice with a thickness of 5 mm is input at the beginning of learning, the prediction model can perform learning by stacking slices to create sub-volume data in units of 90 mm, and such learning can be performed a specified number of times (epoch).

[0048] The term "classification model" as used herein may be a classification model trained to segment the type of target region using a medical image as input. Specifically, the classification model may be a model trained to classify at least one target region among the head, neck, chest, abdomen, and pelvis, thereby grouping sequentially captured slices by type of target region (e.g., head, neck, chest, abdomen, and pelvis).

[0049] In various embodiments, by using a classification model before predicting whether a volume data is a target region using a prediction model, computational efficiency can be improved compared to using the prediction model alone.

[0050] Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the attached drawings.

[0051] FIG. 1 is a block diagram showing the configuration of a medical image segmentation system according to one embodiment of the present invention.

[0052] Referring to FIG. 1, a medical image segmentation system (1000) may be a system configured to segment only a target region from a medical image. Here, segmenting the target region may be understood as indicating a region corresponding to the target region within the medical image. To this end, the medical image segmentation system (1000) may include an imaging device (100) for photographing a part of a subject's body, a medical staff device (200) for identifying a region corresponding to the target region and diagnosing the subject, and a medical image segmentation device (300) for segmenting a region corresponding to the target region within the medical image.

[0053] The imaging device (100) is a device that can obtain a medical image of a target part of a subject (10), such as a human or an animal, and may include a cylindrical bore (110) into which the subject (10) is introduced, a transport device (130) in which the subject (110) is seated, and the subject (110) is introduced inside. Here, the target part may include various organs, bones, muscles, etc. of the subject (10), such as the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), and pelvis (Pelvis). The imaging device (100) can obtain a medical image of the subject (10) by irradiating an X-ray capable of projecting the subject (10) toward the subject.

[0054] In various embodiments, the imaging device (100) can obtain a two-dimensional image in gray scale or RGB, a single still image, a video composed of multiple cuts, etc. as a medical image for predicting an area corresponding to a target area.

[0055] The medical device (200) may be a device capable of transmitting a request for segmentation of a medical image to a medical image segmentation device (300) and displaying the medical image segmentation result. For example, the medical device (200) may include a smartphone, a tablet PC (personal computer), a laptop, a PC, etc.

[0056] In various embodiments, the medical device (200) may display the medical image segmentation results provided by the medical image segmentation device (300) through a user interface screen. For example, the medical image segmentation results may be provided by highlighting the area corresponding to the target region in each of a plurality of slices based on a single plane with different colors.

[0057] In various embodiments, the medical device (200) is configured to be able to install or execute a web or mobile application or program provided by the medical image segmentation device (300), so as to perform a series of medical image segmentation methods performed by the medical image segmentation device (300), which will be described later.

[0058] A medical image segmentation device (300) acquires a medical image of a subject from an imaging device (100) and, based on the acquired image, predicts an area corresponding to a target area. To this end, the medical image segmentation device (100) may include a general-purpose computer, laptop, data server, etc. capable of performing deep learning on medical images and analyzing them.

[0059] In the present invention, the medical image segmentation device (300) does not predict an area corresponding to a target area in each of a plurality of slices constituting a medical image, but generates a plurality of sub-volume data using a plurality of slices, and can predict an area corresponding to a target area based on the sub-volume data. Here, the sub-volume data can be generated by stacking a plurality of slices constituting a medical image according to a pre-stored height, and dividing in a direction perpendicular to the plane of the slices. For example, the sub-volume data can be slab data having a size of 256x256x16, and for another example, the sub-volume data can be cubic data having a size of 96x96x96. In addition, each voxel constituting the sub-volume data is 2x2x3mm. 3 , 0.7x0.7x1.0mm 3 It can be made in size.

[0060] In the present invention, the plurality of sub-volume data include data on the movement direction of voxels constituting the sub-volume data with respect to a single axis, thereby increasing the accuracy of the prediction result compared to predicting the target region on a slice-by-slice basis. For example, when the sub-volume data are slab data, each sub-volume data may include data on the movement direction of voxels in a direction perpendicular to the horizontal plane (Axial). As another example, when the sub-volume data are cubic data, each sub-volume data may include data on the movement direction of voxels in at least one of the horizontal plane (Axial), coronal plane (Coronal), and sagittal plane (Sagittal) directions.

[0061] In various embodiments, the medical image segmentation device (300) may input a plurality of sub-volume data into a prediction model trained to predict a target region using a 3D medical image as input, and determine an area corresponding to the target region from the plurality of sub-volume data. In other words, the medical image segmentation device (300) may predict whether each of the plurality of sub-volume data corresponds to a region corresponding to the target region. For example, the medical image segmentation device (300) may predict whether each of the plurality of sub-volume data corresponds to a target region such as the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), and pelvis (Pelvis).

[0062] To this end, the medical image segmentation device (300) may include a plurality of prediction models capable of predicting specific regions of each of the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), and pelvis (Pelvis), and the medical images may be grouped by target region to use any one of the plurality of prediction models.

[0063] In various embodiments, the medical image segmentation device (300) inputs a medical image of a subject into a classification model trained to classify the type of target region using the medical image as input, and determines the type of target region included in each of a plurality of slices constituting the medical image. For example, the classification model can determine whether the slice includes any one of the target regions of the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), pelvis (Pelvis), eye (Optic), heart (Heart), breast (Breast), and intestines (Bowels). The medical image segmentation device (300) can group the plurality of slices by target region based on the classification result.

[0064] In various embodiments, the medical image segmentation device (300) may combine predicted results based on a plurality of sub-volume data to display an area corresponding to a target region in a medical image and provide the display to the medical staff device (200). For example, the medical image segmentation device (300) may provide a user interface screen including a segmentation result display area displaying an area corresponding to a target region to the medical staff device (200). Here, the user interface screen may include a segmentation model selection area for selecting a target region to be segmented in the medical image, thereby highlighting and displaying only a specific target region that the medical staff wishes to confirm in the medical image. In addition, since the medical image segmentation device (300) predicts an area corresponding to a target region based on a plurality of sub-volume data, the medical image segmentation device (300) may display and provide a target region captured in any one of the axial, coronal, and sagittal planes according to the selection of the medical staff.

[0065] So far, a medical image segmentation system (1000) according to one embodiment of the present invention has been described. According to the present invention, the medical image segmentation system (1000) can accurately predict the location of a target area in a medical image and visually display it, thereby assisting medical staff in diagnosing the target area.

[0066] Hereinafter, with reference to FIG. 2, a medical device (200) that utilizes the medical image segmentation results will be described.

[0067] Figure 2 is a block diagram showing the configuration of a medical device according to one embodiment of the present invention.

[0068] Referring to FIG. 2, the medical device (200) may include a memory interface (210), one or more processors (220), and a peripheral interface (230). Various components within the medical device (200) may be connected by one or more communication buses or signal lines.

[0069] The memory interface (210) is connected to the memory (250) and can transmit various data to the processor (220). Here, the memory (250) can include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0070] In various embodiments, the memory (250) may store a web / app application or program for segmenting only the region corresponding to the target region in a medical image. Furthermore, the memory (250) may store identification information of the subject (e.g., age, gender, presence of disease), the subject's medical image, and the region corresponding to the target region within the medical image.

[0071] In various embodiments, the memory (250) may store at least one of an operating system (251), a communication module (252), a graphical user interface module (GUI) (253), a sensor processing module (254), a telephone module (255), and an application module (256). Specifically, the operating system (251) may include instructions for processing basic system services and instructions for performing hardware operations. The communication module (252) may communicate with at least one of one or more other devices, computers, and servers. The graphical user interface module (GUI) (253) may process a graphical user interface. The sensor processing module (254) may process sensor-related functions (e.g., processing voice input received through one or more microphones (292). The telephone module (255) may process telephone-related functions. The application module (256) may perform various functions of a user application, such as electronic messaging, web browsing, media processing, navigation, imaging, and other processing functions. Additionally, the identifier device (100) can store one or more software applications (256-1, 256-2) (e.g., a medical image segmentation application) associated with one type of service in the memory (250).

[0072] In various embodiments, the memory (250) may store a digital assistant client module (257) (hereinafter, DA client module), and accordingly, may store commands for performing client-side functions of the digital assistant and various user data (258) (e.g., user-customized vocabulary data, preference data, other data such as the user's electronic address book, etc.).

[0073] Meanwhile, the DA client module (257) can obtain the user's voice input, text input, touch input, and / or gesture input through various user interfaces (e.g., I / O subsystem (240)) provided in the medical device (200).

[0074] Additionally, the DA client module (257) can output data in audiovisual and tactile forms. For example, the DA client module (257) can output data consisting of a combination of at least two or more of voice, sound, notification, text message, menu, graphic, video, animation, and vibration. In addition, the DA client module (257) can communicate with a digital assistant server (not shown) using a communication subsystem (280).

[0075] In various embodiments, the DA client module (257) may collect additional information about the surroundings of the medical device (200) from various sensors, subsystems, and peripheral devices to construct a context associated with the user input. For example, the DA client module (257) may provide context information along with the user input to a digital assistant server to infer the user's intent. Here, the context information that may accompany the user input may include sensor information, such as lighting, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, the context information may include the physical state of the medical device (200) (e.g., device orientation, device position, device temperature, power level, speed, acceleration, motion patterns, cellular signal strength, etc.). As yet another example, the context information may include information related to the software state of the medical device (200) (e.g., processes running on the medical device (200), installed programs, past and present network activity, background services, error logs, resource usage, etc.).

[0076] In various embodiments, the memory (250) may include additional or deleted instructions. Furthermore, the medical device (200) may also include additional configurations other than those illustrated in FIG. 2, or may exclude some configurations.

[0077] The processor (220) can control the overall operation of the medical device (200), and can execute various commands to display an area corresponding to a target area in a medical image to a medical professional by running an application or program stored in the memory (250), or to implement a user interface that can confirm a prediction result provided through a prediction model.

[0078] The processor (220) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). In addition, the processor (220) may be implemented in the form of an integrated chip (IC), such as a SoC (System on Chip) that integrates various computing devices that perform machine learning, such as an NPU (Neural Processing Unit).

[0079] In various embodiments, the processor (220) may display a target region in a medical image through a user interface screen via a medical image segmentation application or program provided by the medical image segmentation device (300), or may request segmentation of the target region and display the corresponding result.

[0080] In various embodiments, the processor (220) may acquire a medical image from the imaging device (100) and generate a plurality of sub-volume data using the medical image. In addition, the processor (220) may input a plurality of sub-volume data into a prediction model trained to predict a target region using a 3D medical image as input, determine an area corresponding to the target region from the plurality of sub-volume data, and provide an area corresponding to the target region within the medical image through a user interface screen.

[0081] The peripheral interface (230) can be connected to various sensors, subsystems, and peripheral devices to provide data so that the medical device (200) can perform various functions. Here, it can be understood that the function performed by the medical device (200) is performed by the processor (220).

[0082] The peripheral interface (230) can receive data from a motion sensor (260), a light sensor (optical sensor) (261), and a proximity sensor (262), through which the medical device (200) can perform orientation, light, and proximity detection functions, etc. For another example, the peripheral interface (230) can receive data from other sensors (263) (positioning system - GPS receiver, temperature sensor, biometric sensor), through which the medical device (200) can perform functions related to the other sensors (263).

[0083] In various embodiments, the medical device (200) may include a camera subsystem (270) connected to a peripheral interface (230) and an optical sensor (271) connected thereto, which enables the medical device (200) to perform various photographing functions, such as taking pictures and recording video clips.

[0084] In various embodiments, the medical device (200) may include a communication subsystem (280) connected to a peripheral interface (230). The communication subsystem (280) may be comprised of one or more wired / wireless networks and may include various communication ports, radio frequency transceivers, and optical transceivers.

[0085] In various embodiments, the medical device (200) includes an audio subsystem (290) coupled to the peripheral interface (230), the audio subsystem (290) including one or more speakers (291) and one or more microphones (292), thereby enabling the medical device (200) to perform voice-activated functions, such as voice recognition, voice replication, digital recording, and telephony.

[0086] In various embodiments, the medical device (200) may include an I / O subsystem (240) connected to a peripheral interface (230). For example, the I / O subsystem (240) may control a touch screen (243) included in the identifier device (100) via a touch screen controller (241).

[0087] For example, the touch screen controller (241) may detect user contact and movement or cessation of contact and movement using any one of a plurality of touch sensing technologies, such as capacitive, resistive, infrared, surface acoustic wave technology, proximity sensor array, etc. In another example, the I / O subsystem (240) may control other input / control devices (244) included in the medical device (200) via other input controller(s) (242). As an example, the other input controller(s) (242) may control one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and pointer devices, such as a stylus.

[0088] So far, a medical device (200) according to one embodiment of the present invention has been described. According to the present invention, the medical device (200) can be used to accurately identify the location and size of a target area to be diagnosed in a medical image, such as an organ, bone, or tumor, thereby accurately diagnosing the health status of a subject.

[0089] Hereinafter, a medical image segmentation device (300) that provides a result of segmenting a target area in a medical image will be described with reference to FIG. 3.

[0090] FIG. 3 is a block diagram showing the configuration of a medical image segmentation device according to one embodiment of the present invention.

[0091] Referring to FIG. 3, the medical image segmentation device (300) may include a communication interface (310), a memory (320), an I / O interface (330), and a processor (340), and each component may communicate with each other through one or more communication buses or signal lines.

[0092] The communication interface (310) can be connected to the imaging device (100) and the medical device (200) via a wired / wireless communication network to exchange data. For example, the communication interface (310) can receive a medical image of a subject from the imaging device (100) or the medical device (200). As another example, the communication interface (310) can transmit the result of predicting an area corresponding to a target area in a medical image to the medical device (200) and provide a user interface screen for visually displaying the result.

[0093] Meanwhile, the communication interface (310) that enables transmission and reception of such data includes a wired communication port (311) and a wireless circuit (312), wherein the wired communication port (311) may include one or more wired interfaces, for example, Ethernet, Universal Serial Bus (USB), FireWire, etc. In addition, the wireless circuit (312) may transmit and receive data with an external device via an RF signal or an optical signal. In addition, the wireless communication may use at least one of a plurality of communication standards, protocols, and technologies, for example, GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

[0094] The memory (320) can store various data used in the medical image segmentation device (300). For example, the memory (320) can store identification information of the image capturing device (100) and the medical staff device (200), a prediction model learned to predict an area corresponding to a target area by inputting a 3D medical image, and a configuration and learning data of a classification model learned to segment the type of target area by inputting a medical image.

[0095] In various embodiments, the memory (320) may include a volatile or non-volatile storage medium capable of storing various data, commands, and information. For example, the memory (320) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0096] In various embodiments, the memory (320) may store configurations of at least one of an operating system (321), a communication module (322), a user interface module (323), and one or more applications (324).

[0097] An operating system (321) (e.g., embedded operating systems such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers to control and manage general system operations (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.

[0098] The communication module (323) can support communication with other devices through the communication interface (310). The communication module (320) can include various software components for processing data received by the wired communication port (311) or wireless circuit (312) of the communication interface (310).

[0099] The user interface module (323) can receive a user's request or input from a keyboard, touch screen, keyboard, mouse, microphone, etc. through an I / O interface (330) and provide a user interface on the display.

[0100] The application (324) may include a program or module configured to be executed by one or more processors (340). Here, the application for medical image classification and segmentation may be implemented on a server farm.

[0101] The I / O interface (330) can connect at least one of an input / output device (not shown) of the medical image segmentation device (300), such as a display, a keyboard, a touch screen, and a microphone, to the user interface module (323). The I / O interface (330) can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module (323) and process a command according to the received input.

[0102] The processor (340) is connected to a communication interface (310), a memory (320), and an I / O interface (330) to control the overall operation of the medical image segmentation device (300), learn a prediction model and a classification model through an application or program stored in the memory (320), and perform various commands to segment a target area in a medical image when a new medical image is input.

[0103] The processor (340) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Furthermore, the processor (340) may be implemented in the form of an integrated chip (IC), such as a SoC (System on Chip) in which various computing devices are integrated. Alternatively, the processor (340) may include a module for calculating an artificial neural network model, such as an NPU (Neural Processing Unit).

[0104] Hereinafter, with reference to FIGS. 4 to 6, a method for segmenting a medical image by a processor (340) of a medical image segmentation device (300) will be described.

[0105] Figure 4 is a schematic flowchart of a medical image segmentation method according to one embodiment of the present invention.

[0106] Referring to FIG. 4, the processor (340) can acquire a medical image of a subject (S110). Here, the medical image is a two-dimensional image composed of a plurality of cuts (or slices), and may be an enhanced or non-enhanced computed tomography (CT) image. For example, the medical image may include a head and neck image including the entire area from the skull vertex to the lung apex, a chest image including the entire area from the thyroid to the liver dome, an abdomen image including the L1 spine at a location 3 cm away from the liver dome in a cranial direction, and a pelvic image including the entire ischium at a location 3 cm away from the L1 spine in a cranial direction.

[0107] In various embodiments, the processor (340) may group a plurality of slices constituting a medical image by classifying them according to a target region, thereby increasing the computational efficiency of a prediction model. Specifically, the processor (340) may input a medical image of a subject into a classification model trained to classify the type of target region using the medical image as an input, thereby determining the type of target region included in each of the plurality of slices constituting the medical image. For example, the classification model may determine whether a slice includes one of the target regions of the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), and pelvis (Pelvis). The medical image segmentation device (300) may group the plurality of slices according to the target region based on the classification result.

[0108] After step S110, the processor (340) can generate a plurality of sub-volume data using the medical image (S120). Specifically, the sub-volume data can be generated by stacking a plurality of slices constituting the medical image according to a pre-stored height and dividing them in a direction perpendicular to the plane of the slices. For example, the sub-volume data can be slab data having a size of 256x256x16, and for another example, the sub-volume data can be cubic data having a size of 96x96x96. In addition, each voxel constituting the sub-volume data has a size of 2x2x3mm. 3 , 0.7x0.7x1.0mm 3 It can be made in size. The processor (340) can generate a plurality of sub-volume data based on an axis including any one of the axial, coronal, and sagittal planes.

[0109] In various embodiments, the plurality of sub-volume data may include data on the movement direction of voxels constituting the sub-volume data with respect to one axis, thereby increasing the accuracy of the prediction result compared to predicting the target region on a slice-by-slice basis. For example, when the sub-volume data is slab data, each sub-volume data may include data on the movement direction of voxels in a direction perpendicular to the horizontal plane (Axial). As another example, when the sub-volume data is cubic data, each sub-volume data may include data on the movement direction of voxels in at least one of the horizontal plane (Axial), coronal plane (Coronal), and sagittal plane (Sagittal) directions.

[0110] After step S120, the processor (340) inputs a plurality of sub-volume data into a prediction model trained to predict a target area using a 3D medical image as input, and can determine an area corresponding to the target area from the plurality of sub-volume data (S130).

[0111] In relation to this, FIG. 5 is a schematic diagram for explaining a medical image segmentation method according to one embodiment of the present invention.

[0112] Referring to FIG. 5, the processor (340) can generate a plurality of sub-volume data by stacking and dividing a medical image (11) composed of a plurality of slices. Here, the medical image (11) can be a medical image classified as including any one of the target regions of the head, neck, chest, abdomen, and pelvis. The processor (340) can generate a plurality of sub-volume data defined as slab data (12a) having a size of 256x256x16 or cubic data (12b) having a size of 96x96x96 by using the medical image (11).

[0113] In various embodiments, the processor (340) may input slab data (12a) into a first target region prediction model (13a) or input cubic data (12b) into a second target region prediction model (13b) depending on the type of generated sub-volume data, and output a result (14) of predicting whether each sub-volume data corresponds to a region corresponding to a target region. Here, the target region may include organs, bones, or muscles such as the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), and pelvis (Pelvis), or may include tumors suspected of being diseases formed around organs, bones, or muscles.

[0114] Meanwhile, in order to obtain such prediction results, the prediction model can be trained with training data having different slice thicknesses depending on the type of medical image. Specifically, the prediction model uses slab data having a size of 256x256x16 or cubic data having a size of 96x96x96 as sub-volume data, and the processor (340) can train the prediction model without image loss of the medical image by performing different number of bootstrappings depending on the thickness of the slice input as training data. For example, when a slice having a thickness of 3 mm is input at the beginning of training, the processor (340) can stack slices to generate sub-volume data in units of 48 mm to perform training of the prediction model, and when a slice having a thickness of 5 mm is input at the beginning of the next training, the processor (340) can stack slices to generate sub-volume data in units of 90 mm to perform training of the prediction model. The processor (340) can perform this learning as many times (epoch) as specified in the memory (320).

[0115] However, in addition to this, the processor (340) may also preprocess multiple slices to be input into the prediction model and use them as training data. For example, the processor (340) may copy one slice and stack them to match the reference thickness of the training data, or perform linear interpolation to resample the thickness of the slice to match the reference thickness.

[0116] Referring again to FIG. 4, the processor (340) may provide an area corresponding to a target region within a medical image (S140). Specifically, the processor (340) may determine whether each of the plurality of sub-volume data corresponds to the target region through a predictive model, and may segment each of the plurality of sub-volume data based on the result. The processor (340) may combine the segmentation results to display an area corresponding to the target region in a plurality of slices constituting the medical image.

[0117] In relation to this, FIGS. 6A and 6B are exemplary diagrams of user interface screens showing medical image segmentation results according to one embodiment of the present invention.

[0118] Referring to FIGS. 6A and 6B , the processor (340) may display an area corresponding to a target area within a medical image and provide the display to the medical device (200). Specifically, the processor (340) may use a prediction model to combine volume data having a probability value corresponding to the target area greater than or equal to a preset value from each of a plurality of sub-volume data and display the combined volume data on a user interface screen. For example, as illustrated in FIG. 6A , the processor (340) may highlight and display only the edge area (15) of the target area having a probability value corresponding to the target area greater than or equal to the preset value. In addition, the processor (340) may display an area predicted to be the target area in the medical image in different colors depending on its type.

[0119] In addition, the processor (340) can display each of the plurality of sub-volume data on the user interface screen with different transparency according to the probability value corresponding to the target region using the prediction model. For example, as illustrated in FIG. 6b, the processor (340) can highlight an area (15') predicted to be the target region in the medical image with different transparency according to the probability value. The processor (340) can control the target region to be highlighted in the medical image by lowering the transparency as the probability value increases.

[0120] In various embodiments, the processor (340) may provide a user interface screen including a segmentation model selection area for selecting a target area to be segmented within a medical image and a segmentation result display area (15) for displaying an area corresponding to the target area.

[0121] In relation to this, FIG. 7 is an example diagram of a user interface screen that uses a prediction model for medical image segmentation according to one embodiment of the present invention.

[0122] Referring to FIG. 7, the processor (340) may train a prediction model and provide a user interface screen capable of displaying prediction results obtained through the prediction model to the medical device (200). The user interface screen may include an area (21) in which learning parameters in the prediction model can be input and adjusted. For example, batch size, iteration, and epoch parameters can be set through the user interface screen. The processor (340) may train a prediction model based on the parameters input through the user interface screen, and the user interface screen may include an area (22) in which learning progress is displayed and an area (23) in which a learning data set to be used for learning is displayed. In addition, the user interface screen may include a graphic object (24) for training a prediction model, and an area (25) in which a list of prediction models to be trained is displayed when the graphic object (24) is selected. In addition, the area (25) in which the list of prediction models to be trained can display a loss function and a scale used for training, and may include a graphic object in which it is possible to select whether to display the prediction results in a visualized manner. The train value is updated each time a batch ends, and the processor (340) can update the valid value once the entire training data is trained, i.e., each time an epoch ends. In addition, the user interface screen may include an area (26) that displays the train value and the valid value in a graph for each prediction model according to the prediction model training. For example, the train value may be displayed as a solid line, and the valid value may be displayed as a dotted line. In addition, the user interface screen may include an area (30) for displaying the prediction result of the target area using the prediction model within the medical image.Here, the user interface screen may include a graphic object (27) for selecting a cross-section based on which area corresponding to the target area will be displayed, a graphic object (28) for selecting a method for displaying the area corresponding to the target area, and a graphic object (29) for setting the opacity for displaying the area corresponding to the target area within the medical image. Here, the user interface screen may display only the area corresponding to the target area within the medical image by highlighting it, and at the same time display the prediction results for each of the plurality of sub-volume data.

[0123] The processor (340) can obtain any one of the reference planes for displaying the target area among the axial, coronal, and sagittal planes through a user interface screen according to the selection of the medical staff, and can render an area corresponding to the target area based on the reference plane to provide an area corresponding to the target area.

[0124] So far, a medical image segmentation method performed by a processor (340) of a medical image segmentation device (300) according to an embodiment of the present invention has been described. According to the present invention, the medical image segmentation device (300) can improve the prediction accuracy of the model by providing information on front and back images of sequentially captured two-dimensional images through volume data, rather than just inputting two-dimensional images constituting a medical image into a prediction model that predicts a target region.

[0125] Below, the overall flow of the learning process and prediction process of the prediction model performed by the medical image segmentation device (300) will be described.

[0126] Figure 8 is a schematic diagram illustrating a method for learning a prediction model according to one embodiment of the present invention.

[0127] Referring to FIG. 8, the medical image segmentation device (300) can preprocess a plurality of slices (learning data sets) to be used for learning a prediction model, and then generate a plurality of sub-volume data based on the preprocessed data. The medical image segmentation device (300) can input a plurality of sub-volume data to perform learning, and can provide an operator that can correct the difference between the learning result and the actual result.

[0128] As described above, the medical image segmentation device (300) can learn a prediction model, and when a prediction request for a medical image is received at the request of an administrator or medical staff, the device can load the prediction model and segment an area corresponding to a target area within the medical image.

[0129] In relation to this, FIG. 9 is a schematic diagram for explaining the operation method of a prediction model for segmenting a medical image according to one embodiment of the present invention.

[0130] Referring to FIG. 9, the medical image segmentation device (300) can acquire enhanced and non-enhanced computed tomography (CT) images generated according to the DICOM standard and determine whether to perform auto-segmentation. For example, the medical staff device (200) can directly perform segmentation of a target area in a medical image or request auto-segmentation through a web or app application provided by the medical image segmentation device (300).

[0131] If automatic segmentation is requested, the medical image segmentation device (300) can detect which target region was captured by the plurality of slices constituting the medical image using the classification model. That is, the medical image segmentation device (300) can classify (or detect) the head (Brain), neck (Neck), chest (Chest), abdomen (Abdomen), pelvis (Pelvis) and other organs from the plurality of slices using the classification model, and can apply an additional prediction model according to the type of target region or the type of subject. For example, the medical image segmentation device (300) can additionally use a prediction model to predict the ocular (Optic) region for a slice classified as the head (Brain). As another example, the medical image segmentation device (300) can additionally use a prediction model to predict the breast (Breast) region for a slice classified as the chest (Chest) region in a medical image of a female subject. As another example, the medical image segmentation device (300) may additionally use a prediction model to predict the heart region in a slice classified as the chest in a medical image of a subject with cardiac disease. As another example, the medical image segmentation device (300) may additionally use a prediction model to predict the bowel region in a slice classified as the abdomen and pelvis.

[0132] As described above, the medical image segmentation device (300) can classify and group slices using a classification model, divide the grouped slices into a plurality of sub-volume data, and then use a prediction model based on this to predict whether each of the plurality of sub-volume data is an area corresponding to a target area.

[0133] Although the embodiments of the present invention have been described in more detail with reference to the attached drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be implemented without departing from the technical spirit of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain it, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all aspects and not restrictive. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

[0134] [National Research and Development Project Supporting This Invention]

[0135] [Project ID] 1425160412

[0136] [Assignment Number] S3033240

[0137] [Ministry Name] Ministry of SMEs and Startups

[0138] [Name of Project Management (Specialist) Agency] Small and Medium Business Technology Information Promotion Agency

[0139] [Research Project Name] Startup Growth Technology Development

[0140] [Research Project Name] Artificial Intelligence for Tumor Treatment / Management Using Medical Image-Based Radiation Therapy

[0141] Platform Development

[0142] [Contribution rate] 1 / 1

[0143] [Name of the project performing organization] Oncosoft Co., Ltd.

[0144] Research Period: January 1, 2022 - November 30, 2022

Claims

1. A method performed by a processor of a medical image segmentation device, A step of acquiring a medical image of a subject; A step of generating a plurality of sub-volume data using the above medical image; A step of inputting a plurality of sub-volume data into a prediction model trained to predict a target region using a 3D medical image as input, and determining an area corresponding to the target region from the plurality of sub-volume data; and A medical image segmentation method comprising: a step of providing an area corresponding to the target area within the medical image of the subject; 2. In paragraph 1, The above sub-volume data is, A medical image segmentation method, comprising data on the movement direction of voxels constituting the sub-volume data based on one axis.

3. In paragraph 1, The step of generating the above sub-volume data is: A medical image segmentation method, comprising the step of stacking a plurality of slices constituting the medical image according to a pre-stored height and dividing the slices in a direction perpendicular to the plane of the slices to generate a plurality of sub-volume data.

4. In paragraph 1, Before generating the above sub-volume data, A step of inputting a medical image into a classification model trained to classify the type of target area by inputting the medical image; A step of determining the type of target region included in each of a plurality of slices constituting the medical image; and A medical image segmentation method further comprising a step of grouping the plurality of slices by target region according to the classification result.

5. In paragraph 1, The step of providing an area corresponding to the above target area is: A method for segmenting a medical image, comprising a step of providing a user interface screen including a segmentation model selection area for selecting a target area to be segmented within a medical image and a segmentation result display area for displaying an area corresponding to the target area.

6. In paragraph 5, The step of providing an area corresponding to the above target area is: A medical image segmentation method further comprising: using the above prediction model, combining volume data having a probability value greater than a preset value corresponding to a target region from each of the plurality of sub-volume data and displaying the combined volume data on the user interface screen; 7. In paragraph 5, The step of providing an area corresponding to the above target area is: A medical image segmentation method further comprising: a step of displaying each of the plurality of sub-volume data on the user interface screen with different transparency according to a probability value corresponding to a target area using the above prediction model; 8. In paragraph 5, The step of generating the above sub-volume data is: It is generated based on an axis that includes one of the axial, coronal, and sagittal planes. The step of providing an area corresponding to the above target area is: A step of obtaining one of the reference planes for displaying the target area among the axial, coronal and sagittal planes through the user interface screen; and A medical image segmentation method further comprising: a step of rendering an area corresponding to the target area based on the reference plane; 9. In paragraph 1, Before the step of acquiring the above medical images, A step of acquiring learning data having different slice thicknesses depending on the type of medical image; A step of generating a learning data set by performing bootstrapping a different number of times according to the thickness of the learning data; and A medical image segmentation method further comprising: a step of generating the prediction model configured to predict one target region based on the learning data set; 10. Communication interface; memory; and a processor operably connected to the communication interface and the memory; The above processor, A medical image segmentation device configured to acquire a medical image of a subject, generate a plurality of sub-volume data using the medical image, input the plurality of sub-volume data into a prediction model trained to predict a target region using a three-dimensional medical image as input, determine an area corresponding to the target region from the plurality of sub-volume data, and provide an area corresponding to the target region within the medical image of the subject.

11. In paragraph 10, The above sub-volume data is, A medical image segmentation device comprising data on the movement direction of voxels constituting the sub-volume data based on one axis.

12. In paragraph 10, The above processor, A medical image segmentation device configured to stack a plurality of slices constituting the medical image according to a pre-stored height and to generate a plurality of sub-volume data by dividing the slices in a direction perpendicular to the plane of the slices.

13. In paragraph 10, The above processor, A medical image segmentation device further configured to input a medical image, a classification model trained to classify the type of target region by inputting the medical image before generating the above sub-volume data, determine the type of target region included in each of a plurality of slices constituting the medical image, and group the plurality of slices by target region based on the classification result.

14. In paragraph 10, The above processor, A medical image segmentation device configured to provide a user interface screen including a segmentation model selection area for selecting a target area to be segmented within a medical image and a segmentation result display area for displaying an area corresponding to the target area.

15. In paragraph 14, The above processor, A medical image segmentation device further configured to display on the user interface screen a combination of volume data having a probability value corresponding to a target area greater than a preset value from each of the plurality of sub-volume data using the above prediction model.

16. In paragraph 14, The above processor, A medical image segmentation device further configured to display each of the plurality of sub-volume data on the user interface screen with different transparency according to a probability value corresponding to the target area using the above prediction model.

17. In paragraph 14, The above processor, The above sub-volume data is generated based on an axis that includes one of the axial, coronal, and sagittal planes, A medical image segmentation method further configured to obtain one of the reference planes for displaying the target region among the axial plane, coronal plane, and sagittal plane through the user interface screen to provide a region corresponding to the target region, and to render the region corresponding to the target region based on the reference plane.

18. In paragraph 10, The above processor, A medical image segmentation method, further configured to obtain learning data having different slice thicknesses according to the type of medical image before acquiring the medical image, perform bootstrapping a different number of times for each thickness of the learning data to generate a learning data set, and generate the prediction model configured to predict one target region based on the learning data set.

Images