Artificial neural network-based medical image analysis apparatus and method, and training method therefor

Abstract

In the present invention, disclosed are an artificial neural network-based medical image analysis apparatus and method, and a training method therefor, the analysis apparatus and method receiving a first medical image and a second medical image obtained by capturing, at different points in time, the same subject to be diagnosed, and determining a change between the first medical image and the second medical image by using a pre-trained medical image analysis model.

Description

Medical image analysis device, method, and learning method based on artificial neural network

[0001] The present invention relates to a medical image analysis device, method, and learning method thereof.

[0002] X-ray photography is a technique in the field of radiology that can obtain images of the inside of the human body and has been used in the diagnosis and treatment of various diseases. Depending on the diagnostic purpose, such as simple chest radiography, simple abdomen radiography, simple skeletal radiography, simple paranasal sinus radiography, simple neck soft tissue radiography, and mammography, it can be obtained as anterior-posterior, lateral, or diagonal radiography.

[0003] These X-rays are much cheaper than tests such as computed tomography (CT) and magnetic resonance imaging (MRI), and the examination method is simple, yet the number of diseases that can be diagnosed through them is large.

[0004] However, distinguishing changes through X-ray images is difficult to solve using existing CNN networks because it requires distinguishing only changes due to disease based on criteria judged by radiologists while ignoring changes in the patient's breathing, posture, age, etc.

[0005] A problem to be solved according to one embodiment of the present invention includes determining changes between medical images acquired for the same diagnosis subject by using a model learned using images read by an arbitrary subject.

[0006] Additionally, it includes making the feature maps of each read image similar to each other.

[0007] Additional, unspecified, purposes of the present invention can be additionally considered within the scope that can be easily inferred from the detailed description and effects thereof below.

[0008] In order to solve the above problem, an artificial neural network-based medical image analysis device according to an embodiment of the present invention includes a memory in which a pre-learned medical image analysis model is stored, and a processor that inputs a first medical image and a second medical image acquired by photographing the same diagnosis subject at different times, and determines a change between the first medical image and the second medical image using the medical image analysis model, wherein the medical image analysis model is learned by using the first and second read images of any subject that have been read as learning input images and using information that reads a change between the first and second read images as a read label.

[0009] Here, the processor can determine a region of change according to a disease among changes between the first medical image and the second medical image.

[0010] Here, the determined change area may be minimally affected by changes in breathing, posture, and age of the subject of the diagnosis.

[0011] A method for learning a medical image analysis model based on an artificial neural network according to another embodiment of the present invention includes a step in which a processor inputs into a first network a first read image and a second read image of a subject that have been read as learning input images and information obtained by reading a change between the first read image and the second read image as a read label, and a step in which a first parameter of the first network is adjusted based on an output result of a similarity value of the first read image and the second read image.

[0012] A method for learning a medical image analysis model based on an artificial neural network according to another embodiment of the present invention includes a step of receiving a first medical image and a second medical image acquired by capturing images at different times for the same diagnosis subject, and a step of determining a change between the first medical image and the second medical image using a pre-learned medical image analysis model, wherein the medical image analysis model is learned using the first and second read images of any subject that have been read as learning input images, and using information obtained by reading the change between the first and second read images as read labels.

[0013] In addition, a computer program including instructions for performing a medical image analysis model learning method including a step of inputting information on a change between the first and second read images of a previously read subject as learning input images and a step of adjusting a first parameter of the first network based on an output result of a similarity value of the first and second read images can be provided.

[0014] In addition, a computer program stored on a computer-readable storage medium can be provided that includes instructions for performing a medical image analysis model learning method, including a step of inputting information on a change between the first and second read images of a previously read subject as learning input images and a step of adjusting a first parameter of the first network based on an output result of a similarity value of the first and second read images.

[0015] As described above, according to the embodiments of the present invention, changes according to disease can be determined in medical images acquired for the same diagnosis subject using a pre-learned model.

[0016] Additionally, the error in reading can be reduced by ensuring that the feature maps of each reading image are similar to each other.

[0017] Even if the effect is not explicitly mentioned herein, the effect and its provisional effect described in the following specification expected by the technical features of the present invention are treated as described in the specification of the present invention.

[0018] FIG. 1 is a block diagram illustrating an artificial neural network-based medical image analysis device according to one embodiment of the present invention.

[0019] Figure 2 is a flowchart illustrating a medical image analysis method based on an artificial neural network according to one embodiment of the present invention.

[0020] Figure 3 is a flowchart illustrating a method for learning a medical image analysis model according to one embodiment of the present invention.

[0021] FIG. 4 illustrates an example of a learning input image used for learning a medical image analysis model according to one embodiment of the present invention.

[0022] FIG. 5 is a drawing for explaining an exemplary structure of a medical image analysis model according to one embodiment of the present invention.

[0023] FIG. 6 is a diagram for explaining a first network and a second network of a medical image analysis model according to one embodiment of the present invention.

[0024] FIG. 7 is a diagram for explaining the effect according to the second network of the medical image analysis model according to one embodiment of the present invention.

[0025] Hereinafter, the medical image analysis device, method, and learning method based on an artificial neural network related to the present invention will be described in more detail with reference to the drawings. However, the present invention can be implemented in various different forms and is not limited to the described embodiments. In addition, for the purpose of clearly explaining the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same elements.

[0026] The suffixes "module" and "part" used for components in the following description are given or used interchangeably only for the convenience of writing specifications, and do not have distinct meanings or roles in themselves.

[0027] The present invention relates to a medical image analysis device and method based on an artificial neural network and a learning method thereof.

[0028] FIG. 1 is a block diagram illustrating an artificial neural network-based medical image analysis device according to one embodiment of the present invention.

[0029] Referring to FIG. 1, an artificial neural network-based medical image analysis device (10) according to one embodiment of the present invention includes a processor (11) and a memory (12).

[0030] An artificial neural network-based medical image analysis device (10) according to one embodiment of the present invention is a device for reading changes due to a disease from an X-ray image of a subject.

[0031] Specifically, it operates like the procedure actually performed by radiologists by using a module that allows for robust learning of changes in the subject's natural variations (breathing, age, shooting posture, etc.).

[0032] To this end, it is designed with a CNN structure to distinguish changes and utilizes multiple labels. The medical image analysis model of the medical image analysis device (10) based on an artificial neural network includes a first network, a second network, and a disease information providing neural network, and the multiple labels may include an interpretation label which is information on interpreting changes in an X-ray image, an adjustment label which is a target pixel value of the X-ray image, and a disease label which is information on interpreting changes according to a disease.

[0033] The memory (12) can store programs (one or more instructions) for processing and controlling the processor (11), store a pre-learned medical image analysis model, and include at least one type of computer-readable storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0034] The programs stored in the memory (12) can be divided into multiple modules according to function, and the pre-learned medical image analysis model described below can be configured as a software module.

[0035] The processor (11) executes one or more instructions stored in the memory (12), and specifically, receives first and second medical images acquired by capturing images at different times for the same diagnosis subject, and determines changes between the first and second medical images using a medical image analysis model.

[0036] Specifically, the areas of change according to disease are determined among the changes between the first medical image and the second medical image, and the determined areas of change are minimally affected by changes in breathing, posture, and age of the subject of diagnosis.

[0037] Here, the processor (11) may be divided into multiple modules according to function, or may perform functions in a single processor. The processor may include one or more of a central processing unit (CPU), an application processor (AP), a microcontroller unit (MCU), or a communication processor (CP).

[0038] The medical image analysis model of the medical image analysis device (10) based on an artificial neural network according to one embodiment of the present invention is learned by using the first and second read images of any subject that have been read as learning input images and using information obtained by reading the change between the first and second read images as read labels.

[0039] The medical image analysis model is described in detail in Figure 5 below.

[0040] Figure 2 is a flowchart illustrating a medical image analysis method based on an artificial neural network according to one embodiment of the present invention.

[0041] Referring to FIG. 2, an artificial neural network-based medical image analysis method according to one embodiment of the present invention is performed by a medical image analysis device, and in step S10, a first medical image and a second medical image acquired by capturing images at different points in time for the same diagnosis subject are input.

[0042] In step S20, a pre-learned medical image analysis model is used to determine changes between the first medical image and the second medical image.

[0043] Specifically, the areas of change according to disease are determined among the changes between the first medical image and the second medical image, and the determined areas of change are minimally affected by changes in breathing, posture, and age of the subject of diagnosis.

[0044] Here, the medical image analysis model is learned using the first and second read images of a random subject that have been read as learning input images, and the information obtained by reading the change between the first and second read images as read labels.

[0045] Figure 3 is a flowchart illustrating a method for learning a medical image analysis model according to one embodiment of the present invention.

[0046] Referring to FIG. 3, a method for learning a medical image analysis model according to an embodiment of the present invention is performed by a processor, and in step S110, a first reading image and a second reading image of an arbitrary subject read are used as learning input images, and information obtained by reading a change between the first reading image and the second reading image is input to a first network as a reading label.

[0047] In step S120, the first parameter of the first network is adjusted based on the output result of the similarity value of the first read image and the second read image.

[0048] Here, the step (S120) of adjusting the first parameter of the first network extracts a first feature vector from a first read image, extracts a second feature vector from a second read image, integrates the first feature map and the second feature map generated as the first feature vector and the second feature vector pass through at least one layer using an integration layer, and then adjusts the first parameter based on the output result of the similarity value that has passed through a plurality of layer structures.

[0049] In step S130, the first read image and the second read image that have passed through at least one layer of the first network and the target pixel value are input to the second network as adjustment labels.

[0050] In step S140, the second parameter of the second network is adjusted so that the feature map similarity value of each of the first read image and the second read image that passed through at least one layer of the first network becomes the maximum value.

[0051] In step S150, the first read image and the second read image that have passed through at least one layer of the first network and the second network and the information on the change according to the disease are input into the disease information providing neural network as disease labels.

[0052] In step S160, based on the output result according to the presence or absence of a disease, the third parameter of the disease information providing neural network is adjusted.

[0053] Here, the first parameter, the second parameter, and the third parameter are updated based on the output finally returned through the first network, the second network, and the disease information providing neural network.

[0054] FIG. 4 illustrates an example of a learning input image used for learning a medical image analysis model according to one embodiment of the present invention.

[0055] The learning input images used for learning the medical image analysis model according to one embodiment of the present invention use the first reading image and the second reading image of any subject that have been previously read.

[0056] As shown in Fig. 4, the learning of the medical image analysis model according to one embodiment of the present invention is performed entirely using approximately 400,000 pairs of Chest X-ray follow-up data.

[0057] The learning input images are composed of images taken by the patient at different times. The first reading image, which is the image taken previously (a), is called the baseline, and the second reading image, which is the image taken after (b), is called the follow-up.

[0058] The training input image, which is X-ray data, is converted into an 8-bit PNG format and used as network input. During the training process, data augmentation techniques such as random cropping, random brightness conversion, and random rotation are used to train the medical image analysis model. The input data is normalized to values ​​between 0 and 1 and input to train the medical image analysis model.

[0059] FIG. 5 is a drawing for explaining an exemplary structure of a medical image analysis model according to one embodiment of the present invention.

[0060] Referring to FIG. 5, a medical image analysis model according to one embodiment of the present invention includes a first network (110, 120, 130, 140), a second network (200), and a disease information providing neural network (310, 320).

[0061] Specifically, the first network (110, 120, 130, 140) includes at least one layer that returns a similarity value between a first read image (image A) and a second read image (image B) based on a first parameter.

[0062] Specifically, the first network is trained to extract a first feature vector from a first read image using a first module (110), extract a second feature vector from a second read image using a second module (120), integrate the first feature map and the second feature map generated as the first feature vector and the second feature vector pass through at least one layer using an integration layer (130), and then return a feature map similarity value by passing through a plurality of layer structures (140).

[0063] Here, the first reading image (image A) is a baseline (image A), the second reading image (image B) is a follow-up (image B), and the first module (110) and the second module (120) are implemented as ResNet.

[0064] For example, the first module (110) and the second module (120) of the first network constitute a Siamese network. A Siamese network is a network configuration that independently executes two convolutional neural networks on two inputs and then compares them.

[0065] The training of the Siamese network is done by inputting two photos into the two networks and encoding them with a convolutional neural network. If the two photos are similar, the distance between the encodings should be small, and if the two photos are different, the distance between the encodings should be large.

[0066] Through this, it becomes possible to determine whether there is a change using the similarity value between the first reading image and the second reading image.

[0067] Specifically, baseline (video A) and follow-up (video B) are input and passed through ResNet (110, 120), and the latent vectors generated here are used to distinguish normal / abnormal and change / no-change.

[0068] Additionally, the integration layer (130) is implemented as a concat layer, and the multiple layer structure (140) uses a structure that passes through MLP.

[0069] That is, in order to distinguish between change / no-change, a structure is used that concatenates the latent vector from the baseline and the latent vector from the follow-up and passes it through the MLP.

[0070] However, since the configuration of the first network alone cannot exclude changes due to the patient's breathing, age, and shooting posture, the second network and the disease information-providing neural network are added to complement each other and designed.

[0071] The second network (200) is trained so that the feature map similarity value of each of the first and second read images that have passed through at least one layer of the first network is the maximum value, and the second parameter is adjusted using the target pixel value as the adjustment label.

[0072] Here, the target pixel value refers to a pixel value based on the first reading image to correct the difference that occurs when the first reading image and the second reading image are partially misaligned with the direction of the X-ray depending on the subject's posture.

[0073] The second network (200) is implemented as an Attention Module and has the effect of making each feature map from the baseline and follow-up photos have the same shape.

[0074] Here, making them have the same shape is to correct the difference that occurs when the first and second reading images are slightly misaligned with the direction of the X-ray depending on the posture of the subject, by using the first reading image as a reference and correcting the shape of the second reading image as if the subject was photographed in a state where the direction of the X-ray was the same.

[0075] To this end, correction is performed based on the pixel values ​​of the two images so that the pixel values ​​are similar.

[0076] The disease information providing neural network (310, 320) is trained to adjust the third parameter using the information obtained by reading the changes according to the disease in the first and second reading images as the disease label so as to output the presence or absence of the disease.

[0077] Here, the disease information providing neural network (310, 320) uses the Cross-Entropy Softmax algorithm.

[0078] Information on changes in disease is information that records the type of disease by analyzing the results of the readings in the actual records.

[0079] For example, the types of conditions that can be detected on a chest X-ray may include lung cancer, tuberculosis, pneumonia, and pneumothorax.

[0080] By using this, even if changes in the image are determined, only changes due to disease can be determined, excluding changes due to breathing.

[0081] Here, the first parameter, the second parameter, and the third parameter are updated based on the output finally returned through the first network, the second network, and the disease information providing neural network.

[0082] The loss of the entire learning can be designed as shown in Equation 1.

[0083]

[0084] Here, L disease , L change Cross-Entropy or Pairwise Loss (Triplet, Contrastive, etc.) can be used. λ is used as a factor representing the weight of each loss.

[0085] L diseaseIn one embodiment of the present invention, each X-ray has a function of providing information for distinguishing between normal and abnormal by performing a judgment as to whether there is a disease or not, and is used in a disease information providing neural network.

[0086] L change In one embodiment of the present invention, it receives two follow-up x-rays as input and distinguishes whether they are change, change, or nochange, and is used in the first network.

[0087] L matching It allows the baseline and follow-up images to look at the same place and is used in the second network.

[0088] L disease Wow L matching By using it, after focusing on the location of a lesion in one x-ray image, you can check if the disease is in the same location in another x-ray image.

[0089] FIG. 6 is a diagram for explaining a first network and a second network of a medical image analysis model according to one embodiment of the present invention.

[0090] In the above FIG. 5, one ResNet of the first network and the Attention Module of the second network are illustrated as being connected, but according to various embodiments of the invention, the Attention Modules (210, 220, 230, 240) of the second network are each connected to a plurality of modules (111, 112, 113, 114, 115) of the first network, so that the parameters of the Attention Modules can be sequentially adjusted so that the feature map similarity values ​​of the first and second read images passing through each layer of the first network become the maximum values.

[0091] Accordingly, the different pixel values ​​of the first read image and the second read image, which have passed through each layer of different dimensions in the first network, can become similar as they pass through the Attention Module of the second network.

[0092] Specifically, we design a Region Matching Loss that makes the two feature maps that passed through the Attention Module look at the same place.

[0093] Here, to make them look at the same place, the first reading image is used as a reference to correct the difference that occurs when the first reading image and the second reading image are slightly misaligned with the direction of the X-ray depending on the posture of the subject, and the shape of the second reading image is corrected as if the subject was photographed in a state where the direction of the X-ray is the same, and a Region Matching Loss is designed to match the attention of the two images so that the cosine similarity is maximized.

[0094] Region Matching Loss is defined by the following mathematical expression 2.

[0095]

[0096] Here, F b Wow F f represent feature maps from baseline and follow-up, respectively.

[0097] Region Matching Loss has the effect of making the feature maps similar to each other by calculating the similarity value of two different feature maps.

[0098] Since the structure is such that the loss is minimized when the two feature maps are similar to each other, adding this loss function and learning can have the effect of making the feature maps of the baseline and follow-up similar to each other.

[0099] FIG. 7 is a diagram for explaining the effect according to the second network of the medical image analysis model according to one embodiment of the present invention.

[0100] Figure 7 (a) shows the feature map of baseline and basline, and Figure 7 (b) shows the feature map of follow-up and follow-up.

[0101] According to one embodiment of the present invention, a feature map obtained by using the Region matching module of the second network can confirm that the same location in the feature map is activated despite the different shooting positions.

[0102] In addition, a computer program including commands for performing a medical image analysis model learning method including a step of inputting information on a change between the first and second read images of a previously read subject as learning input images and a step of adjusting a first parameter of the first network based on an output result of a similarity value of the first and second read images; can be provided on a recording medium storing a computer program including commands for performing a medical image analysis model learning method including a step of inputting information on a change between the first and second read images as a learning label into a first network.

[0103] In addition, a computer program stored on a computer-readable storage medium can be provided that includes instructions for performing a medical image analysis model learning method, including a step of inputting information on a change between the first and second read images of a previously read subject as learning input images and a step of adjusting a first parameter of the first network based on an output result of a similarity value of the first and second read images.

[0104] Such computer-readable media may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the recording medium may be those specially designed and configured for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The above-mentioned hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0105] The above description merely exemplifies one embodiment of the present invention. Those skilled in the art will appreciate that modifications and variations can be implemented without departing from the essential characteristics of the present invention. Therefore, the scope of the present invention is not limited to the aforementioned embodiments, but should be interpreted to encompass various embodiments within the scope equivalent to the claims.

Claims

1. In a medical image analysis device, Memory storing pre-trained medical image analysis models; and A processor that receives first and second medical images acquired at different times for the same diagnosis subject and determines a change between the first and second medical images using the medical image analysis model; The above medical image analysis model is, The first and second read images of a random subject that have been read are used as learning input images, and the information obtained by reading the change between the first and second read images is used as a read label. A medical image analysis device comprising a first network including at least one layer that returns a similarity value between the first read image and the second read image based on a first parameter.

2. In paragraph 1, The above processor is a medical image analysis device that determines a change area according to a disease among changes between the first medical image and the second medical image.

3. In paragraph 2, The above-determined change area is a medical image analysis device in which the influence of changes in breathing, posture, and age of the subject of the diagnosis is minimized.

4. In paragraph 1, A medical image analysis device in which the first network is trained to extract a first feature vector from the first read image, extract a second feature vector from the second read image, integrate a first feature map and a second feature map generated as the first feature vector and the second feature vector pass through at least one layer using an integration layer, and then return a feature map similarity value of the first feature map and the second feature map based on passing through a plurality of layer structures.

5. In paragraph 4, The medical image analysis model is a medical image analysis device including a second network that is trained to adjust a second parameter using a target pixel value as an adjustment label so that the feature map similarity value becomes the maximum value.

6. In paragraph 5, The medical image analysis model is a medical image analysis device including a disease information providing neural network that is trained to adjust a third parameter using information obtained by reading changes according to disease in the first and second reading images as a disease label so as to output the presence or absence of a disease.

7. In paragraph 6, A medical image analysis device in which the first parameter, the second parameter, and the third parameter are updated based on the output finally returned through the first network, the second network, and the disease information providing neural network.

8. In the method of learning a medical image analysis model, A step in which a processor inputs information on a change area between the first and second read images of a previously read subject as learning input images and inputs the information into a first network as a read label; and A method for learning a medical image analysis model, comprising: a step of adjusting a first parameter of the first network based on the output result of the similarity value of the first read image and the second read image.

9. In paragraph 8, The step of adjusting the first parameter of the first network comprises: A medical image analysis model learning method comprising: extracting a first feature vector from the first read image, extracting a second feature vector from the second read image, integrating a first feature map and a second feature map generated by the first feature vector and the second feature vector passing through at least one layer using an integration layer, and then adjusting the first parameter based on an output result of a feature map similarity value of the first feature map and the second feature map based on passing through a plurality of layer structures.

10. In paragraph 9, A step of inputting the first read image and the second read image that have passed through at least one layer of the first network and the target pixel value as an adjustment label into the second network; and A method for learning a medical image analysis model, comprising: a step of adjusting a second parameter of the second network so that the feature map similarity value becomes the maximum value.

11. In paragraph 10, A step of inputting the first and second read images that have passed through at least one layer of the first network and the second network and the information on the changes according to the disease as disease labels into a disease information providing neural network; and A method for learning a medical image analysis model, comprising: a step of adjusting a third parameter of the disease information providing neural network based on an output result according to the presence or absence of the disease.

12. In paragraph 11, A method for learning a medical image analysis model, wherein the first parameter, the second parameter, and the third parameter are updated based on the output finally returned through the first network, the second network, and the disease information providing neural network.

13. In a medical image analysis method performed by a medical image analysis device, A step of receiving first and second medical images acquired by shooting at different times for the same diagnosis subject; and A step of determining a change area between the first medical image and the second medical image using a pre-learned medical image analysis model; The above medical image analysis model is, The first and second read images of a random subject that have been read are used as learning input images, and the information obtained by reading the change between the first and second read images is used as a read label. A medical image analysis method comprising a first network including at least one layer that returns a similarity value between the first read image and the second read image based on a first parameter.

14. In paragraph 13, A method for analyzing medical images, wherein the step of determining a change area between the first medical image and the second medical image determines a change area according to a disease among the changes between the first medical image and the second medical image.

15. In paragraph 14, A medical image analysis method in which the above-determined change area is minimally affected by at least one of changes in breathing, posture, and age of the subject of the diagnosis.

Images