Disease diagnosis method using neural network trained with multi-phase biological images, and disease diagnosis system for carrying out said method

Abstract

A disease diagnosis method using a neural network trained with multi-phase biometric images and a disease diagnosis system for performing the method are disclosed. According to one aspect of the present invention, the method includes the steps of: generating a diagnostic neural network for predicting a diagnosis result for a predetermined disease using biometric images; acquiring a plurality of training biometric images (each of the plurality of training biometric images is labeled with a diagnosis result for the corresponding disease); and training the diagnostic neural network using the plurality of training biometric images, wherein for each of the plurality of training biometric images, the method includes the steps of: (a) inserting noise having different characteristics into the training biometric image to generate K noise-inserted images corresponding to the training biometric image (where K is an integer greater than or equal to 2); and (b) inputting the K noise-inserted images corresponding to the training biometric image into the diagnostic neural network to train the diagnostic neural network.

Description

[Technical field]

[0001] The present invention relates to a disease diagnosis system and method using a neural network, and more particularly to a disease diagnosis method using a neural network trained using multi-phase biological images, and a disease diagnosis system that performs the same. [Background technology]

[0002] One of the main tasks performed by a pathology or pathology department is to diagnose a condition or symptom for a particular disease by reading a patient's biological slides (e.g., tissue image slides from a biopsy, etc.). Such a diagnosis relies on the experience and knowledge of experienced medical personnel over a long period of time.

[0003] In recent years, advances in machine learning have led to attempts to automate tasks such as image recognition and classification by computer systems. In particular, attempts have been made to automate diagnoses that were previously performed by skilled medical professionals by using neural networks, a type of machine learning (e.g., deep learning methods using convolutional neural networks (CNNs)).

[0004] In particular, diagnosis through deep learning using neural networks (e.g., CNN, deep learning networks) does not simply automate the experience and knowledge of traditionally experienced medical professionals, but rather finds characteristic elements through self-learning and derives the desired answer, and in some cases can even find characteristics of disease factors from images that even experienced medical professionals are unaware of.

[0005] In general, disease diagnosis by neural networks using biological images, which are scanned images of biological slides, uses biological images or patches (also called tiles), which are fragments of biological images. That is, a medical professional familiar with the biological images or patches annotates the state of a particular disease (e.g., whether cancer is present or not) and uses multiple annotation tiles as training data for learning the neural network. In this case, a convolutional neural network may be used as the neural network.

[0006] Meanwhile, most of the current neural network-based diagnostic methods use already scanned data for learning and predict the diagnosis result from the scanned data. In other words, the conventional neural network-based diagnostic method works relatively well for fixed data. However, it has a disadvantage that it is vulnerable when noise is present. For example, in the case of image data captured in real time by a microscope, the image contains a lot of noise generated by the microscope camera, and the diagnosis result keeps changing due to the noise of the camera itself.

[0007] On the other hand, the diagnostic speed using neural networks is generally slower than the frame rate of the camera, so if the part being diagnosed with the microscope continues to move, there is a possibility that the camera image will be interrupted. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Republic of Korea Patent Publication No. 10-2016-0034814 "Client device including neural network and system including same" Summary of the Invention [Problem to be solved by the invention]

[0009] The technical problem that the present invention aims to achieve is to provide a diagnostic system and method using a neural network that can perform stable diagnosis regardless of noise caused by the camera.

[0010] Another object of the present invention is to provide a diagnostic system and method that detects camera movement and does not perform a diagnosis when the camera is moving or the screen is blank, but only when a specific part is fixed and being viewed, thereby enabling smooth screen movement.

[0011] Furthermore, the present invention provides a neural network structure that can distinguish not only the presence or absence of a disease through a specific biological image, but also the diseased area in the biological image. In particular, the present invention provides a neural network structure that can efficiently perform division for dividing the diseased area by adding a sub-architecture for division to a classification neural network architecture that can determine whether a patch has a disease. [Means for solving the problem]

[0012] According to one aspect of the present invention, there is provided a method including: a step of a diagnostic system using a neural network generating a diagnostic neural network for predicting a diagnosis result for a predetermined disease using a biometric image; a step of the diagnostic system using the neural network acquiring a plurality of training biometric images (wherein each of the plurality of training biometric images is labeled with a diagnosis result for the corresponding disease) by the neural network; and a step of the diagnostic system using the plurality of training biometric images training the diagnostic neural network, wherein the step of training the diagnostic neural network includes a step of, for each of the plurality of training biometric images, (a) inserting noise having different characteristics into the training biometric image to generate K noise-inserted images (wherein K is an integer equal to or greater than 2) corresponding to the training biometric image; and (b) inputting the K noise-inserted images corresponding to the training biometric images into the diagnostic neural network to train the diagnostic neural network.

[0013] In one embodiment, the step (b) may include a step of concatenating all K noise-inserted images corresponding to the training biometric image to generate one training data corresponding to the training biometric image (wherein the training data corresponding to the training biometric image is labeled with a diagnosis result for the training biometric image), and a step of inputting the training data corresponding to the training biometric image to the diagnostic neural network to train the diagnostic neural network.

[0014] In one embodiment, the above method further includes a diagnosis step, which may include a step of acquiring K biometric images of a diagnostic subject continuously photographed by an image sensor, and a step of inputting the K biometric images of a diagnostic subject into the diagnostic neural network to predict a diagnostic result for the disease.

[0015] In one embodiment, the diagnosis step further includes a step of determining whether the image sensor is moving based on the biological images of the K diagnostic subjects, and the step of inputting the biological images of the K diagnostic subjects into the diagnostic neural network to predict a diagnostic result for the disease may include a step of inputting the biological images of the K diagnostic subjects into the diagnostic neural network to predict a diagnostic result for the disease when it is determined that the image sensor is not moving.

[0016] In one embodiment, the above method further includes a diagnosis step, which may include a step of acquiring K biometric images of the diagnostic target continuously captured by an image sensor, a step of linking all of the K biometric images of the diagnostic target to generate one diagnostic target data, and a step of inputting the diagnostic target data into the diagnostic neural network to predict a diagnostic result for the disease.

[0017] In one embodiment, the diagnostic neural network is a partitioned neural network that inputs the diagnostic object data to an input layer and identifies an area where a disease exists in the biological image of the diagnostic object, and the partitioned neural network may include a classification neural network that inputs the diagnostic object data to an input layer and outputs a classification result regarding whether the disease exists in the biological image, and a partitioned architecture that inputs feature maps generated from each of two or more feature map extraction layers among hidden layers included in the classification neural network and identifies an area where a disease exists in the biological image.

[0018] In one embodiment, the segmentation architecture may include a convolution sub-architecture including a convolution node corresponding to each of the two or more feature extraction layers, where each of the convolution nodes performs a convolution or two or more different convolutions on a feature map input from the corresponding feature extraction layer, and a segmentation sub-architecture for identifying a region of the patch where a disease is present based on a convolution result generated by the convolution sub-architecture.

[0019] In one embodiment, the disease may be characterized as prostate cancer.

[0020] According to another aspect of the invention there is provided a computer program recorded on a computer readable medium for performing the method as set out above when installed on a data processing device.

[0021] According to another aspect of the present invention, there is provided a neural network-based diagnostic system comprising a processor and a memory storing a computer program which, when executed by the processor, causes the neural network-based diagnostic system to perform the method described above.

[0022] According to another aspect of the present invention, there is provided a diagnostic system using a neural network, the diagnostic system including: a storage module for storing a diagnostic neural network for predicting a diagnosis result for a predetermined disease using a biometric image; an acquisition module for acquiring a plurality of training biometric images (wherein each of the plurality of training biometric images is labeled with a diagnosis result for the corresponding disease); and a learning module for training the diagnostic neural network using the plurality of training biometric images, the learning module performing, for each of the plurality of biometric images, the steps of: (a) inserting noise having different characteristics into the training biometric image to generate K noise-inserted images (wherein K is an integer equal to or greater than 2) corresponding to the training biometric image; and (b) inputting the K noise-inserted images corresponding to the training biometric images into the diagnostic neural network to train the diagnostic neural network.

[0023] In one embodiment, the step (b) may include a step of concatenating all K noise-inserted images corresponding to the training biometric image to generate one training data corresponding to the training biometric image (wherein the training data corresponding to the training biometric image is labeled with a diagnosis result for the training biometric image), and a step of inputting the training data corresponding to the training biometric image to the diagnostic neural network to train the diagnostic neural network.

[0024] In one embodiment, the diagnostic system using the neural network may further include a diagnostic module that acquires K biological images of a diagnostic subject continuously photographed by an image sensor, and inputs the K biological images of the diagnostic subject into the diagnostic neural network to predict a diagnostic result for the disease.

[0025] In one embodiment, the diagnostic module may determine whether the image sensor is moving based on the biological images of the K diagnostic subjects, and if it is determined that the image sensor is not moving, input the biological images of the K diagnostic subjects into the diagnostic neural network to predict a diagnosis result for the disease.

[0026] In one embodiment, the diagnostic system using the neural network may further include a diagnostic module that acquires K biological images of a diagnostic subject continuously captured by an image sensor, combines all of the K biological images of the diagnostic subject to generate one diagnostic subject data, and inputs the diagnostic subject data into the diagnostic neural network to predict a diagnostic result for the disease. Effect of the Invention

[0027] According to the technical concept of the present invention, it is possible to provide a diagnostic system and method using a neural network that can perform stable diagnosis without being affected by noise from the camera.

[0028] It is also possible to provide a diagnostic system and method that detects camera movement and performs diagnosis only when a specific area is fixed and viewed, without performing diagnosis when the camera is moving or the screen is blank.

[0029] In addition, an efficient neural network structure can be provided that can not only classify each biological image to determine whether or not a disease has developed, but also segment the biological image to distinguish areas where a disease has developed. [Brief description of the drawings]

[0030] In order to provide a more complete understanding of the drawings referred to in the detailed description of the present invention, a brief description of each of the drawings is provided.

[0031] [Figure 1]FIG. 1 is a diagram showing an operating environment of a method for learning a neural network using multi-phase biological images and diagnosing a disease using the neural network according to the technical concept of the present invention. [Diagram 2] 1 is a diagram showing a schematic configuration of a disease diagnosis system according to one embodiment of the present invention. [Diagram 3] 1 is a flow chart illustrating a process by which a diagnostic system trains a diagnostic neural network using a plurality of training anatomical images, in accordance with one embodiment of the present invention. [Figure 4] FIG. 13 is a diagram showing an example in which two noise-added images are combined and used for learning. [Diagram 5] 1 is a flowchart showing a process in which a diagnostic system diagnoses a biological image of a diagnostic target according to an embodiment of the present invention. [Figure 6] FIG. 13 is a diagram for explaining a process of performing a diagnosis using two consecutively captured biological images of a diagnostic subject. [Figure 7] FIG. 2 illustrates the overall structure of a diagnostic neural network for segmentation according to one embodiment of the present invention. [Figure 8] FIG. 2 is a diagram for explaining the overall structure of a partitioned architecture according to an embodiment of the present invention. [Figure 9] FIG. 2 illustrates an example of a diagnostic neural network for segmentation according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Since the present invention can be modified in various ways and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, it is not intended to limit the present invention to a specific embodiment, and it should be understood that the present invention includes all modifications, equivalents, and alternatives that fall within the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted.

[0033] Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.

[0034] The terms used in this application are used only to describe specific embodiments and are not intended to limit the present invention. The singular forms include the plural forms unless the context clearly dictates otherwise.

[0035] As used herein, the terms "comprising" or "having" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described herein, and are to be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0036] In addition, as used herein, when a component "transmits" data to another component, the component may transmit the data directly to the other component, or may transmit the data to the other component via at least one other component. Conversely, when a component "transmits" data to another component, it means that the data is transmitted from the component to the other component without passing through the other component.

[0037] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, in which the same reference numerals denote the same elements.

[0038] Since the present invention can be modified in various ways and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, it is not intended to limit the present invention to a specific embodiment, and it should be understood that the present invention includes all modifications, equivalents, and alternatives that fall within the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted.

[0039] Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.

[0040] The terms used in this application are used only to describe specific embodiments and are not intended to limit the present invention. The singular forms include the plural forms unless the context clearly dictates otherwise.

[0041] As used herein, the terms "comprising" or "having" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described herein, and are to be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0042] In addition, as used herein, when a component "transmits" data to another component, the component may transmit the data directly to the other component, or may transmit the data to the other component via at least one other component. Conversely, when a component "transmits" data to another component, it means that the data is transmitted from the component to the other component without passing through the other component.

[0043] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, in which the same reference numerals denote the same elements.

[0044] 1 is a diagram showing an operating environment of a method for diagnosing a disease by training a neural network using multi-phase biological images according to the technical concept of the present invention. Referring to FIG. 1, the method according to the technical concept of the present invention may be performed by a diagnostic system 100.

[0045] In the technical concept of the present invention, the diagnostic system 100 may be a computing system. Alternatively, the technical concept of the present invention may be realized by being installed in a predetermined server 10. The server 10 means a data processing device having a computing capability for realizing the technical concept of the present invention, and an average expert in the technical field of the present invention can easily deduce that not only a data processing device to which a client (terminal 20) can be connected via a network, but also any device capable of performing a specific service, such as a personal computer or a mobile terminal, can be defined as a server.

[0046] The server 10 may include a processor and a storage device. The processor may mean a computing device capable of running a program for realizing the technical idea of ​​the present invention. The storage device may mean a data storage means capable of storing a program and various data required for realizing the technical idea of ​​the present invention, and may be realized as a plurality of storage means depending on the embodiment. In addition, the storage device may mean not only a main storage device included in the server 10, but also a temporary storage device or memory that may be included in the processor.

[0047] When the diagnostic system 100 is implemented as included in a specific server 10, the diagnostic system 100 may communicate with at least one client (e.g., 20) that can connect to the server 10. In this case, the client (e.g., 20) may transmit a biometric image to the diagnostic system 100, and the diagnostic system 100 may perform a diagnosis on the transmitted biometric image according to the technical idea of ​​the present invention. Also, the diagnostic result may be transmitted to the client (e.g., 20).

[0048] The diagnostic system 100 may train a neural network for diagnosing biological images using a plurality of training biological images, and after training the neural network, may use the trained neural network to predict a diagnostic result for a diagnostic target image provided by the client 20.

[0049] The client 20 may include an image sensor 21. The image sensor 21 may be a sensor for capturing or scanning a biometric image. The image sensor 21 may capture the biometric images continuously. For example, the image sensor 21 may capture the biometric images at 60 frames per second.

[0050] In one embodiment, the image sensor 21 may be a camera.

[0051] In another embodiment, the image sensor 21 may be a microscope. In this case, the image sensor 21 can magnify and capture a specific area of ​​the biological slide, and a diagnostician who is trying to diagnose the biological image using the client 20 can diagnose the specific area of ​​the biological slide while moving around the capture area on the slide.

[0052] Although the disease diagnosis system 100 is illustrated in FIG. 1 as being realized as any one of the physical devices, it will be easily understood by an average expert in the technical field of the present invention that multiple physical devices can be organically combined as necessary to realize the diagnostic system 100 according to the technical idea of ​​the present invention.

[0053] Depending on the embodiment, the disease diagnosis system 100 may be implemented in a form in which it is installed and operated directly on the client 20, rather than on the server 10.

[0054] FIG. 2 is a diagram showing a schematic configuration of a disease diagnostic system 100 according to an embodiment of the present invention.

[0055] Referring to FIG. 2, the system 100 may include a storage module 110, a generation module 130, an acquisition module 140, a learning module 150, and a diagnosis module 160. In some embodiments of the present invention, some of the above-mentioned components may not necessarily correspond to components essential for realizing the present invention, and the diagnosis system 100 may include more components in some embodiments. For example, the system 100 may further include a control module (not shown) for controlling the functions and / or resources of other components of the system 100 (e.g., the storage module 110, the generation module 130, the acquisition module 140, the learning module 150, the diagnosis module 160, etc.). In addition, in some embodiments, the system 100 may further include a database (Database; DB; 300) for storing various information and / or data necessary to realize the technical idea of ​​the present invention.

[0056] The system 100 may refer to a logical configuration having hardware resources and / or software necessary to realize the technical idea of ​​the present invention, and does not necessarily refer to one physical component or device. In other words, the system 100 may refer to a logical combination of hardware and / or software provided to realize the technical idea of ​​the present invention, and may be realized as a set of logical configurations for realizing the technical idea of ​​the present invention by installing each function in a device separated from each other, as necessary. The system 100 may also refer to a set of configurations realized separately for each function or role for realizing the technical idea of ​​the present invention. For example, each of the storage module 110, the generation module 130, the acquisition module 140, the learning module 150, and the diagnosis module 160 may be located in different physical devices, or may be located in the same physical device. In addition, depending on the embodiment, the combinations of software and / or hardware constituting each of the storage module 110, the generation module 130, the acquisition module 140, the learning module 150, and the diagnosis module 160 may also be located in different physical devices, and the configurations located in the different physical devices may be organically combined with each other to realize each module.

[0057] In this specification, the term "module" may refer to a functional and structural combination of hardware for implementing the technical idea of ​​the present invention and software for driving the hardware. For example, a module may refer to a logical unit of a certain code and a hardware resource for executing the certain code, and it is easy for an average expert in the technical field of the present invention to deduce that a module does not necessarily refer to a physically connected code or a type of hardware.

[0058] The DB 200 may store a plurality of training biological images, which may be of various types, including tissue images and biopsy images.

[0059] Moreover, the training biological images may be biological slides, or may be patches that are portions of divided biological slides.

[0060] Each of the plurality of training biological images may be labeled with a disease diagnosis result. The labeled diagnosis result may be the same type as the diagnosis result diagnosed by the disease diagnosis system 100. Note that the diagnosis result labeled on each training biological image may be a disease judgment result previously made by a medical professional based on the biological image.

[0061] For example, when the disease diagnosis system 100 is a system for predicting a classification result for the presence or absence of disease, the diagnosis result labeled for each of the plurality of training biological images may also be for the presence or absence of disease. Alternatively, in some embodiments, the diagnosis result for the biological image may include not only the presence or absence (negative / positive) of a specific disease, but also the progression of the specific disease (or the probability of being at the progression). For example, when the technical idea of ​​the present invention is used for diagnosing prostate cancer, the diagnosis result may include a Gleason pattern or Gleason score, which is an index of the progression of prostate cancer. For example, the Gleason score value is 2 to 10, and a value of 6 to 10 can usually be regarded as cancer, and a higher value indicates a more severe degree of prostate cancer. The Gleason pattern is classified into levels from 1 to 5.

[0062] If the disease diagnosis system 100 is a system that predicts disease onset areas or pixel-by-pixel diagnostic results (i.e., a segmentation system), the labeled diagnostic results in each of the multiple training biological images may also be related to the disease onset areas or pixel-by-pixel diagnostic results.

[0063] On the other hand, the disease may be prostate cancer, and the following description will focus on prostate cancer, but it will be easily understood by a person of ordinary skill in the art to which the present invention pertains that the technical concept of the present invention is not limited to prostate cancer.

[0064] The diagnostic neural network 120 may be stored in the storage module 110, and the diagnostic neural network 120 may be generated by the generation module 130. The diagnostic neural network 120 may be a neural network that predicts a diagnosis outcome of a certain disease using a biological image.

[0065] The storage module 110 may refer to a data storage means capable of storing the diagnostic neural network 120, and may be realized as a plurality of storage means depending on the embodiment. In addition, the storage module 110 may refer to not only the main storage device included in the server 10, but also a temporary storage device or memory.

[0066] The diagnostic neural network 120 may be a neural network for diagnosing a patch-by-patch biological image. In this case, the patch may be a partial image obtained by dividing a given slide into a given size. That is, one slide may be divided into a plurality of patches, and the diagnostic neural network 120 may input the patches for performing patch-level diagnosis. Before performing the diagnosis, the diagnostic neural network 120 may be trained in advance by the learning module 150.

[0067] Meanwhile, the information output by the diagnostic neural network 120 may be information that enables a determination as to whether or not a specific disease (e.g., a specific type of cancer) has developed in the tissue corresponding to the patch, or a region in which a specific disease has developed. For example, the information output by the diagnostic neural network 160 may be information indicating the probability of whether or not a specific disease (e.g., a specific type of cancer) has developed in the tissue corresponding to the patch. In this case, if the probability value output by the diagnostic neural network 120 is equal to or greater than a specific reference value (threshold value), it may be determined that a disease (e.g., prostate cancer) has developed in the patch. Of course, depending on the embodiment, the diagnostic neural network 120 may output information indicating not only whether or not a specific disease has developed, but also the progression level (or the probability of reaching the progression level) of the specific disease. For example, when the technical idea of ​​the present invention is used for diagnosing prostate cancer, the information output by the diagnostic neural network 120 may include a Gleason pattern or a Gleason score, which is an index of the progression level of prostate cancer. On the other hand, the threshold value used by the diagnostic neural network 120 may be set in various ways, and depending on the threshold value, a particular patch may be judged as a patch in which a disease is manifested, i.e., a disease patch, or may be judged as a normal patch.

[0068] Alternatively, the diagnostic neural network 120 may be a neural network for inputting a patch and predicting a disease classification result for each pixel constituting the patch. The diagnostic neural network 120 that outputs the classification result for each pixel may perform division to distinguish disease onset areas. In this case, the diagnostic neural network 120 may output the presence or absence of disease onset or the disease onset probability for each pixel constituting the patch, and may determine an area constituted by pixels determined to have developed a disease or pixels whose onset probability exceeds a certain threshold as an onset area.

[0069] Depending on the embodiment, the diagnostic neural network 120 may predict and output the presence or absence of a disease (or the probability of developing a disease) and / or the stage of a disease (or the probability of being at the stage of a disease) for each pixel.

[0070] On the other hand, before performing the segmentation (ie, predicting the diagnosis result for each pixel), the diagnostic neural network 120 may be pre-trained by the learning module 150 .

[0071] As used herein, a neural network may refer to a collection of information representing a set of design items that define the neural network. In one embodiment, the diagnostic neural network 120 may be a convolutional neural network.

[0072] As is well known, a convolutional neural network may include an input layer, multiple hidden layers, and an output layer. Each of the multiple hidden layers may include a convolutional layer and a pooling layer (or a subsampling layer).

[0073] The convolutional neural network may be defined by the functions, filters, strides, weight coefficients, etc., for defining each of these layers. Furthermore, the output layer may be defined as a fully connected FeedForward layer.

[0074] The details of the design of each layer constituting a convolutional neural network are well known. For example, the number of layers included in the multiple layers, the convolution function for defining the multiple layers, the pooling function, and the activation function may be publicly known functions, or functions defined separately to realize the technical idea of ​​the present invention may be used.

[0075] An example of the convolution function is a discrete convolution sum. An example of the pooling function may be max pooling, average pooling, etc. An example of the activation function may be sigmoid, tanh, rectified linear unit (ReLU), etc.

[0076] When such a convolutional neural network design is defined, the convolutional neural network with the defined design may be stored in a storage device, and when the convolutional neural network is trained, the weight coefficients corresponding to each layer may be identified.

[0077] That is, training of a convolutional neural network may refer to a process in which weight coefficients of each layer are determined. Also, when a convolutional neural network is trained, the trained convolutional neural network may input input data to an input layer and output output data through a predefined output layer.

[0078] A neural network according to an embodiment of the present invention may be defined by selecting any one or more of the commonly known design considerations described above, or a unique design may be defined for the neural network.

[0079] 2, the acquisition module 140 may acquire training data for machine learning for the diagnostic neural network 120. That is, the acquisition module 140 may acquire a plurality of training biometric images, and each of the plurality of training biometric images may be labeled with a corresponding diagnosis result. For example, the acquisition module 140 may acquire the training biometric images from the DB 200.

[0080] Furthermore, although the present specification has mainly described an example in which the technical concept of the present invention has been applied to prostate cancer, an average expert in the technical field of the present invention can easily deduce that accurate diagnosis is possible when the technical concept of the present invention is applied to other diseases in which diagnosis of a specific tissue requires consideration of not only the condition of the specific tissue but also the condition of the tissue surrounding the specific tissue.

[0081] Alternatively, the learning module 150 may train the diagnostic neural network 120 using a number of training biometric images.

[0082] More specifically, the learning module 150 may insert noise having different characteristics into each of a plurality of training biometric images to generate K (where K is an integer greater than or equal to 2) noise-inserted images corresponding to the training biometric images, and input the K noise-inserted images corresponding to the training biometric images to the diagnostic neural network to train the diagnostic neural network.

[0083] FIG. 3 is a flow diagram illustrating a process by which diagnostic system 100 trains diagnostic neural network 120 using a number of training biometric images, in accordance with one embodiment of the present invention.

[0084] Referring to FIG. 3, the acquisition module 140 of the diagnostic system 100 may acquire N (N is an integer equal to or greater than 2) training biological images labeled with disease diagnoses (S100).

[0085] Thereafter, diagnostic module 150 of diagnostic system 100 may perform processes S120 to S150 for each of the N training body images (S110).

[0086] The Jth training biometric image S j In the case of the learning process through the above, a training biometric image S j Then, noises with different characteristics are inserted into the training biometric images S j K (where K is an integer equal to or greater than 2) noise-inserted images T(J,1) Or T (J,K) may be generated (S120).

[0087] For example, the noise-inserted image T (J,1) Or T (J,K) is the training biometric image S j The image may be an image in which Gaussian noise with different means and / or variances is added to the biometric image. When Gaussian noise is added, various parameters may be changed in addition to the mean and / or variance, and various forms of noise may be added to the biometric image in addition to the Gaussian noise.

[0088] In step S130, the training biometric image S j The K noise-injected images T (J,1) Or T (J,K) The training biometric image S j One training data U corresponding to j In this case, the training data U j is the training biometric image S j The label may be labeled with a label (S130).

[0089] In step S140, the training biometric image S j The training data U j may be input to the diagnostic neural network 120 to train the diagnostic neural network 120.

[0090] FIG. 4 is a diagram showing an example in which two noise-added images generated by adding two different noises to one training biometric image are combined and used for learning (that is, an example in which K is 2).

[0091] As shown in FIG. 4, noise 1 and noise 2 may be added to the original image to be trained to generate two different noise-inserted images, which may then be combined and input to the diagnostic neural network 120 for training.

[0092] 2 again, the diagnosis module 160 can obtain K consecutively captured biological images of the diagnostic subject, and input the K biological images of the diagnostic subject to the diagnostic neural network 120 trained by the learning module 150 to predict a diagnosis result for a disease. In one embodiment, the K biological images of the diagnostic subject may be biological images captured by the image sensor 21.

[0093] FIG. 5 is a flowchart showing a process in which the diagnostic system 100 diagnoses a biological image of a diagnostic target according to an embodiment of the present invention.

[0094] Referring to FIG. 5, the diagnosis module 160 receives K consecutive images O1 to O2 of a diagnostic target. K may be acquired (S200).

[0095] The diagnosis module 160 receives K diagnostic subject biometric images O1 to O K It may be determined whether a predetermined diagnostic condition is satisfied based on the result (S210).

[0096] In one embodiment, the diagnostic module (160) may determine whether the image sensor 21 capturing the diagnostic target images is moving based on the K biometric images of the diagnostic target, and if it is determined that the image sensor 21 is not moving, it can determine that the diagnostic condition is satisfied. That is, the diagnostic module 160 can determine that the diagnostic condition is satisfied when the image sensor 21 is not moving but is stationary and continues to capture a specific portion of the biometric image.

[0097] For example, the diagnostic module 160 may calculate a number or value indicating the relationship between two consecutive images, and determine that no movement is occurring if the calculated number or value is within a predetermined limit range.

[0098] Alternatively, in some embodiments, the diagnostic module 160 may determine that the diagnostic condition is satisfied when the image sensor 21 is not moving and the K diagnostic target biological images capture biological tissues. Also, various diagnostic conditions may be preset.

[0099] If the diagnostic condition is not satisfied, the K biometric images taken thereafter can be acquired without diagnosing the disease (see S210 and S200).

[0100] If the diagnosis condition is met, the diagnosis module 160 outputs K biometric images O1 to O K may be concatenated to generate diagnostic object data P (S220), and the diagnostic object data P may be input to a diagnostic neural network 120 to predict a diagnosis result for a disease.

[0101] FIG. 6 is a diagram for explaining a process of making a diagnosis using two consecutively captured images of a living body of a diagnostic subject.

[0102] 6, two diagnostic target images (i.e., Image 1, Image 2) acquired from an input device (e.g., image sensor 21) may be combined together and input to an algorithm to determine whether a diagnostic condition is met. If the diagnostic condition is met, the combined data (i.e., diagnostic target data) may be input to a trained diagnostic neural network 120, which can predict a diagnostic outcome.

[0103] As described above, in some embodiments, the diagnostic neural network 12 may be a segmentation neural network that performs segmentation to identify disease-affected areas of a biological image (e.g., a patch), and for this purpose the segmentation neural network may have a highly unique structure.

[0104] The diagnostic neural network 120 for segmentation according to an embodiment of the present invention can be realized by combining a sub-architecture for segmentation with a neural network that performs classification to determine whether or not a disease exists in a biological image (a "classification neural network" to be described later). The structure of such a diagnostic neural network 120 is shown in FIG.

[0105] FIG. 7 illustrates the overall structure of a diagnostic neural network 120 for segmentation according to one embodiment of the present invention.

[0106] As shown in FIG. 7, a diagnostic neural network 120 according to one embodiment of the present invention may include a classification neural network 200 and a partitioning architecture 260.

[0107] The classification neural network 200 may input diagnostic target data to an input layer and output a classification result (e.g., a score as shown in FIG. 7) regarding whether or not a disease exists in a diagnostic target image corresponding to the diagnostic target data. This is called classification, and in the classification process, the classification neural network 200 can generate features for the input (i.e., a patch) as intermediate products in some hidden layers included therein. In particular, when a matrix of two or more dimensions such as an image is received as an input, the generated features are in the form of a two-dimensional matrix, and therefore the term feature map may be used. Meanwhile, hereinafter, a layer that generates a feature map among the hidden layers included in the classification neural network 200 will be called a feature map extraction layer.

[0108] On the other hand, the segmentation architecture 260 may input feature maps (e.g., f1, f2, and f3 shown in FIG. 4) generated in each of two or more feature map extraction layers among the hidden layers included in the classification neural network 200, and identify and output areas of the patch where a disease exists.

[0109] FIG. 7 shows an example in which the classification neural network 200 generates three feature maps (f1, f2, f3) during the classification process, but it is understood that more or less feature maps may be generated depending on the embodiment.

[0110] Meanwhile, according to an embodiment of the present invention, the classification neural network 200 for performing the classification uses a known dense net, which may be designed to take into account not only the specific patch to be diagnosed but also the surrounding patches as disclosed in the previous application. In addition, various neural networks may be utilized, and in any case, the classification neural network 200 may be defined to receive a specific patch as an input and output a feature value corresponding to the disease occurrence probability of the specific patch.

[0111] On the other hand, FIG. 8 is a diagram for explaining the overall structure of the division architecture 260. As shown in FIG.

[0112] Referring to FIG. 8, the partitioning architecture 260 may include a convolution sub-architecture 261, a partitioning sub-architecture 263, and may further include a trimming sub-architecture 264 depending on the embodiment.

[0113] As described above, in the classification process performed in the classification neural network 200, feature maps F1, F2, and F3 may be generated by each feature extraction layer, and each feature map may be input to convolution nodes 262-1 to 262-3 included in the convolution sub-architecture 261.

[0114] Each of the convolution nodes 262-1 to 262-3 corresponds to two or more feature extraction layers included in the classification neural network 200, and may perform convolution or two or more different convolutions on the feature maps F1 to f3 input from the corresponding feature extraction layer. Meanwhile, depending on the embodiment, each of the convolution nodes 262-1 to 263-3 may perform convolution after upscaling (e.g., upsampling) or downscaling (e.g., downsampling).

[0115] Each of the convolution nodes 262-1 to 262-3 may perform one or more convolutions to generate one or more results. Depending on the embodiment, the convolutions performed by the convolution nodes 262-1 to 262-3 may be dilated convolutions (also called atrous convolutions). Unlike conventional convolutions, dilated convolutions are a method of performing convolutions at a predetermined interval (rate) instead of extracting features from adjacent pixels. For example, any one of the convolution nodes (e.g., 511-2) may perform a 1x1 dilated convolution, a 3x3 dilated convolution with a rate of 6, a 3x3 dilated convolution with a rate of 12, and a 3x3 dilated convolution with a rate of 18 to generate four convolution results (features).

[0116] Meanwhile, the segmentation sub-architecture 263 may identify areas of the patch where disease exists based on the convolution results generated by the convolution sub-architecture 261.

[0117] The partitioning sub-architecture 263 may perform certain operations on the convolution results generated by the convolution sub-architecture 261. The operations performed by the partitioning sub-architecture 263 may be defined as combinations of concatenation and / or convolution. Depending on the embodiment, concatenation and convolution may be combined in various ways.

[0118] On the other hand, in some embodiments, the cropping sub-architecture 264 may generate the final result for the partitioning by performing center cropping on the result output from the partitioning sub-architecture 263. This is because the relatively central parts of the result of the convolution sub-architecture 261 and partitioning sub-architecture 263 tend to more accurately reflect the results.

[0119] FIG. 9 illustrates an example of a diagnostic neural network 120 for segmentation according to one embodiment of the present invention.

[0120] Referring to FIG. 9, the classification neural network 200 included in the diagnostic neural network may perform classification. For classification, the classification neural network 200 may input a patch to an input layer and generate a 1 / 4-sized raw feature map f1 through convolution and pooling operations in a first feature extraction layer L1. Then, a 1 / 16-sized middle feature map f2 may be generated in a second feature extraction layer L2 through a first dense block and first transition operation (denseblock1, transition1) and a second dense block and second transition operation (denseblock2, transition2). Then, a 1 / 16-sized end feature map f3 may be generated in a third feature extraction layer L3 through a third dense block and third transition operation (denseblock3, transition3), a fourth dense block (denseblock4), and 1x1 convolution in sequence. Then, a score for the presence or absence of a disease may be output through average pooling.

[0121] Alternatively, segmentation architecture 260 may use the features generated in classification neural network 200 to perform segmentation.

[0122] More specifically, each convolution node 510 included in the convolution sub-architecture 261 may perform at least one predefined convolution on a feature map input from a corresponding feature extraction layer. In the example of FIG. 9, the first convolution node 262-1 may perform 1x1 convolution on a feature map f1 input from a corresponding first feature extraction layer L1. The second convolution node 262-2 may perform 1x1 convolution, 3x3 dilated convolution of rate 6, 3x3 dilated convolution of rate 12, and 3x3 dilated convolution of rate 18 on a feature map F2 input from a corresponding second feature extraction layer L2 to generate four convolution results (features). The third convolution node 262-3 may perform 2x upsampling and then 1x1 convolution on a feature map f3 input from a corresponding third feature extraction layer L3.

[0123] Meanwhile, the partitioning sub-architecture 263 may receive the results generated by the convolution sub-architecture 261 and perform predefined operations. In the example of Fig. 9, the partitioning sub-architecture 263 may combine all of the convolution results (features) generated by the second convolution node 511-2 and the third convolution node 511-3 to perform 1x1 convolution, combine this with the convolution result (feature) generated by the first convolution node 511-1, and then perform 3x3 convolution.

[0124] Center trimming may then be performed in the trimming sub-architecture 264.

[0125] By using a neural network having such technical features of the present invention, the accuracy of segmentation can be significantly improved. It is empirically well known that in order to improve the segmentation accuracy of a conventional neural network, it is necessary to extract features well from the input data and properly set the initial weights of the neural network before training and then perform training. However, as described above, according to the technical idea of ​​the present invention, segmentation is performed by a neural network of a type that combines a segmentation architecture having a specific structure based on a classification neural network, and the features extracted in the patch-level classification process reflect the features of the input data very well. Therefore, by using this as it is in the segmentation process, it has the effect of improving the accuracy of segmentation.

[0126] Meanwhile, in this specification, an example in which the technical concept of the present invention has been applied to prostate cancer has been mainly described, but an average expert in the technical field of the present invention can easily deduce that accurate diagnosis is possible when the technical concept of the present invention is applied to other diseases in which diagnosis of a specific tissue requires consideration of not only the state of the specific tissue but also the state of the tissue surrounding the specific tissue.

[0127] Meanwhile, depending on the embodiment, the diagnostic system 100 may include a processor and a memory that stores a program executed by the processor. The processor may include a single-core CPU or a multi-core CPU. The memory may include a high-speed random access memory and may include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory by the processor and other components may be controlled by a memory controller.

[0128] Meanwhile, the diagnosis method using the multi-color model and the neural network according to the embodiment of the present invention may be realized in the form of computer readable program instructions and may be stored in a computer readable recording medium, and the control program and the target program according to the embodiment of the present invention may also be stored in a computer readable recording medium. The computer readable recording medium includes any kind of recording device in which data readable by a computer system is stored.

[0129] The program instructions contained in the recording media may be those specially designed and constructed for the present invention, or they may be of the well-known and available to those skilled in the software arts.

[0130] Examples of computer-readable recording media include hardware devices specifically configured to store and execute program instructions, such as magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, ROMs, RAMs, flash memories, etc. Furthermore, computer-readable recording media can be distributed among computer systems connected via a network, allowing computer-readable codes to be stored and executed in a distributed manner.

[0131] Examples of program instructions include machine code, such as produced by a compiler, as well as higher level language codes that may be executed by a device for processing information electronically, such as a computer, using an interpreter or the like.

[0132] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of this invention, and vice versa.

[0133] The above description of the present invention is for illustrative purposes only, and those skilled in the art will understand that the present invention can be easily converted into other specific forms without changing the technical idea or essential characteristics of the present invention. Therefore, the above-described embodiment should be understood to be illustrative in all respects and not restrictive. For example, each component described in a single form may be distributed, or the components described as being similarly distributed may be implemented in a combined form.

[0134] The scope of the present invention is indicated by the claims set forth below rather than by the above detailed description, and all modifications and variations that come within the meaning and scope of the claims and their equivalent concepts should be construed as being included within the scope of the present invention. [Industrial Applicability]

[0135] The present invention relates to a method for diagnosing a disease using a neural network trained using multi-phase biological images, and a disease diagnosis system for carrying out the same.

Claims

1. A diagnostic system using a neural network generates a diagnostic neural network for predicting a diagnosis result regarding a predetermined disease using a biological image; acquiring a plurality of training body images by the neural network-based diagnostic system, where each of the plurality of training body images is labeled with a corresponding diagnosis for the disease; training the diagnostic neural network using the plurality of training body images; Including, The step of training the diagnostic neural network comprises: inserting noises having different characteristics into the training biometric images to generate K noise-inserted images (where K is an integer equal to or greater than 2) corresponding to the training biometric images; Concatenating all the K noise-inserted images corresponding to the training biometric images to generate one training data corresponding to the training biometric images, where the training data corresponding to the training biometric images is labeled with a diagnosis result for the training biometric images; and training the diagnostic neural network by inputting training data corresponding to the training biometric images into the diagnostic neural network. method.

2. The method further comprises a diagnostic step, The diagnosis step includes: acquiring K biological images of a diagnostic subject successively captured by an image sensor; The method of claim 1 , further comprising: inputting the K biological images of diagnostic subjects into the diagnostic neural network to predict a diagnosis outcome for the disease.

3. The diagnosis step includes: determining whether the image sensor is moving based on the K biological images of the diagnostic subject; The step of inputting the K biological images of a diagnostic subject into the diagnostic neural network to predict a diagnosis result for the disease includes: The method of claim 2 , further comprising the step of inputting the K biological images of the diagnostic subject into the diagnostic neural network to predict a diagnosis outcome for the disease if it is determined that the image sensor is not moving.

4. The method further comprises a diagnostic step, The diagnosis step includes: acquiring K biological images of a diagnostic subject successively captured by an image sensor; generating one diagnostic subject data by linking all of the K diagnostic subject biometric images; and inputting the diagnostic subject data into the diagnostic neural network to predict a diagnosis outcome for the disease.

5. The diagnostic neural network comprises: a segmented neural network that inputs the diagnostic object data into an input layer and identifies an area in the biological image of the diagnostic object where the disease exists; The split neural network comprises: a classification neural network that inputs the diagnostic object data to an input layer and outputs a classification result regarding whether the disease is present in the biological image; The method of claim 4, further comprising: a segmentation architecture that inputs feature maps generated in two or more feature map extraction layers among hidden layers included in the classification neural network, and identifies an area in the biological image where the disease exists.

6. The partitioned architecture comprises: a convolution sub-architecture including a convolution node corresponding to each of the two or more feature extraction layers (wherein each of the convolution nodes performs convolution or two or more mutually different convolutions on a feature map input from the corresponding feature extraction layer); and a segmentation sub-architecture that identifies disease-present areas of the patch based on the convolution results generated by the convolution sub-architecture.

7. The disease is The method of claim 1, characterized in that the cancer is prostate cancer.

8. A computer program recorded on a computer readable medium for performing the method according to any one of claims 1 to 7 when installed on a data processing device.

9. A diagnostic system using a neural network, a processor and a memory for storing a computer program; The computer program, when executed by the processor, causes the neural network-based diagnostic system to perform the method of any one of claims 1 to 7.

10. a memory module for storing a diagnostic neural network for predicting a diagnosis result for a predetermined disease using a biological image; an acquisition module for acquiring a plurality of training images of a living body, each of the plurality of training images being labeled with a corresponding diagnosis for the disease; a learning module for training the diagnostic neural network using the plurality of training biometric images; Including, The learning module, for each of the plurality of training biometric images, inserting noises having different characteristics into the training biometric images to generate K noise-inserted images (where K is an integer equal to or greater than 2) corresponding to the training biometric images; Concatenating all the K noise-inserted images corresponding to the training biometric images to generate one training data corresponding to the training biometric images, where the training data corresponding to the training biometric images is labeled with a diagnosis result for the training biometric images; training the diagnostic neural network by inputting training data corresponding to the training biometric images into the diagnostic neural network; A diagnostic system that uses neural networks to perform the following:

11. Acquire K biological images of a diagnostic subject successively captured by an image sensor; The neural network-based diagnostic system according to claim 10 , further comprising a diagnostic module that inputs the K biological images of diagnostic subjects to the diagnostic neural network to predict a diagnosis result of a disease.

12. The diagnostic module includes: determining whether the image sensor is moving based on the K biological images of the diagnostic subject; The neural network-based diagnostic system according to claim 11 , wherein, when it is determined that the image sensor is not moving, the K biological images of the diagnostic subject are input to the diagnostic neural network to predict a diagnosis result for the disease.

13. Acquire K biological images of a diagnostic subject successively captured by an image sensor; Concatenating all of the K diagnostic subject biometric images to generate one diagnostic subject data set; The neural network-based diagnostic system of claim 10, further comprising a diagnostic module for inputting the diagnostic object data into the diagnostic neural network to predict a diagnosis result for the disease.

Images