Neural network output analysis method and system therefor

Abstract

A method and system for analyzing the output of a neural network are disclosed. According to one aspect of the present invention, a method for analyzing the output of a neural network that has been trained to output a probability of disease onset for each pixel of a biological image includes the steps of: an output analysis system determining an optimal reference value for each pixel as a reference for detecting the presence or absence of disease onset in the corresponding pixel depending on whether the output result value of the neural network is equal to or greater than an optimal reference value; a step of determining an optimal cutoff value for determining whether a detected lesion area on the biological image is valid for a detected lesion area on the biological image that is identified when the presence or absence of disease on each pixel of the biological image is detected using the determined optimal reference value; and a step of performing an output analysis on the output result using the determined optimal reference value and optimal cutoff value when the output analysis system receives an output result corresponding to a diagnostic biological image to be diagnosed from the neural network.

Description

[Technical field]

[0001] The present invention relates to a method and system for analyzing the output of a neural network, and more particularly to a method and system for effectively analyzing the output results output by a neural network trained for diagnosing a disease, and for performing a more accurate diagnosis of the disease. [Background technology]

[0002] One of the main tasks performed in pathology or pathology department is to read the images of a patient's body and make a diagnosis to determine the state or symptoms of a particular disease. Such a diagnosis is a method that relies on the experience and knowledge of experienced medical personnel over many years.

[0003] In recent years, with the development of machine learning, there have been active attempts to automate tasks such as image recognition and classification using computer systems. In particular, attempts have been made to automate diagnoses that were previously performed by skilled medical professionals using neural networks, a type of machine learning (e.g., deep learning methods using convolutional neural networks (CNN)).

[0004] In particular, diagnosis using deep learning with neural networks (e.g., convolution neural network, CNN) does not simply automate the experience and knowledge of experienced medical professionals, but rather finds characteristic elements through its own learning and derives the desired answer, and may even find characteristics of disease factors in images that even experienced medical professionals would not have been able to identify.

[0005] In general, disease diagnosis using neural networks using biological images uses fragments of biological images, i.e., patches (also called tiles). That is, for each tile, a skilled medical professional annotates the state of a particular disease (e.g., whether cancer has developed or not), and the neural network is trained using a number of such annotated tiles as training data.

[0006] The neural network trained in this way may be implemented to output, for each pixel, the likelihood that the pixel is included in a region where a disease has occurred, i.e., a lesion region.

[0007] This is useful as information to estimate the possibility that each pixel is included in a lesion area, but it does not definitively determine whether or not it is included in a lesion area. Therefore, a further output analysis step may be required to determine that the pixel-by-pixel possibility information output by the neural network is information regarding whether or not it is actually included in a lesion area.

[0008] FIG. 1 is a diagram for explaining a typical process of diagnosing a disease using a conventional neural network.

[0009] Referring to FIG. 1, a biological image (also called a whole slide image) may be divided into partial images (also called patch images) of a certain size, and a neural network processing process may be performed in which a trained neural network for each divided partial image outputs the possibility that each pixel in the corresponding patch is included in a lesion area.

[0010] Then, through a process of analyzing the output results output by the neural network for each patch, a lesion area in the corresponding biological image is finally detected.

[0011] It goes without saying that in the process of analyzing the output results, a process of combining the output results for each patch with a slide range may also be performed.

[0012] Typically, in the process of analyzing the output results, if the degree of possibility for each pixel is 0.5 or more, it is determined that it is included in the lesion area, and if it is less than 0.5, it is determined that it is not included in the lesion area. When training the neural network, training may be performed taking into account that analysis will be performed based on such assumptions.

[0013] This method may be optimal when there is no error in the annotation information used to train the neural network, and the overall size of the lesion area in the biological image used for training is similar to the overall size of the tissue area not included in the lesion. However, since the annotation information is constructed based on the subjective judgment of a pathologist, the constructed annotation information may have some inconsistency and error, and therefore, a neural network trained to learn this may determine that some lesion areas are not lesion areas or may misidentify some non-lesion areas as lesion areas. Therefore, with only this conventional method, there is a problem that the accuracy of the lesion area detected through the output analysis process is reduced.

[0014] Therefore, a technical idea is required that can more effectively and accurately analyze the output of the neural network to determine whether each pixel is included in a lesion area. [Prior art documents] [Patent documents]

[0015] [Patent Document 1] Korean Patent Application Publication No. 10-2019-0143510 "Two-face disease diagnosis system and method" Summary of the Invention [Problem to be solved by the invention]

[0016] The technical problem that the present invention aims to achieve is to provide a method and system capable of effectively and accurately analyzing the output results based on the output results of a neural network trained to output the possibility of disease manifestation for each pixel, i.e., the possibility of it being included in a lesion area. [Means for solving the problem]

[0017] According to one aspect of the present invention, a neural network output analysis method for analyzing the output result of a neural network trained to output a probability of disease onset for each pixel of a biological image includes the steps of: an output analysis system determining an optimal reference value serving as a standard for detecting the presence or absence of disease onset in a corresponding pixel based on whether or not the output result value of the neural network for each pixel is equal to or greater than an optimal reference value; a step of determining an optimal cut-off value for determining whether or not a detected lesion area on a biological image identified when the output analysis system detects the presence or absence of disease onset for each pixel of the biological image using the determined optimal reference value, the detected lesion area being valid; and a step of performing an output analysis on the output result using the determined optimal reference value and the optimal cut-off value when the output analysis system receives an output result corresponding to a diagnostic biological image to be diagnosed from the neural network.

[0018] The step of determining the optimal reference value may include a step in which the output analysis system acquires an output value for each pixel of each of a plurality of biological images using the neural network; a step in which the output analysis system sets a plurality of candidate reference values ​​and calculates a degree of agreement when each candidate reference value is applied to detect disease onset for each of the set candidate reference values ​​(here, the degree of agreement is determined according to the degree of agreement between the detection result detected by applying the corresponding candidate reference value to each pixel and the annotated result); and a step in which the output analysis system determines the optimal reference value based on the degree of agreement for each of the candidate reference values ​​calculated by the output analysis system.

[0019] The step of calculating the degree of match when the presence or absence of a disease is determined by applying each of the candidate reference values ​​includes calculating the accuracy, IoU (Intersection over Union), or DSC (Dice Similarity Coefficient) of the entire plurality of biological images, and the accuracy, IoU (Intersection over Union), or DSC (Dice Similarity Coefficient) of each of the plurality of biological images. and an average of the mean correlation coefficient (FP+TN) and the mean of the mean correlation coefficient (FP+TN), wherein the accuracy is defined as (TP+TN) / (TP+TN+FP+FN), the IoU is defined as TP / (TP+FP+FN), and the DSC is defined as 2×TP / (2×TP+FP+FN), where TP is the number of pixels contained in both a first lesion area detected by the output analysis system and annotated second lesion area, FN is the number of pixels contained only in the second lesion area, FP is the number of pixels contained only in the first lesion area, and TN is the number of pixels not contained in both the first lesion area and the second lesion area.

[0020] The step of determining an optimal cutoff value for determining whether the detected lesion area is valid may include a step in which the output analysis system acquires an output value for each pixel of each of a plurality of biological images using the neural network; a step in which the output analysis system detects at least one first lesion area contained in each of the plurality of biological images using the output value for each pixel of each of the acquired plurality of biological images and the optimal reference value, thereby acquiring the number of pixels contained in the first lesion area; and a step of determining an optimal cutoff value that serves as a standard for judging the validity of the detected first lesion area based on the number of pixels for the at least one detected first lesion area and whether a disease has actually occurred.

[0021] In the step of determining an optimal cutoff value serving as a criterion for judging the validity of the detected first lesion area based on a second lesion area, which is an annotated lesion area in each of the multiple biological images, and the number of pixels contained in the first lesion area, the output analysis system may determine the optimal cutoff value using any one of linear regression, logistic regression, or a support vector machine.

[0022] The step of determining an optimal cutoff value serving as a criterion for judging the validity of the detected first lesion area based on a second lesion area, which is an annotated lesion area of ​​each of the multiple biological images, and the number of pixels contained in the first lesion area, may include a step of the output analysis system determining an initial optimal cutoff value, a step of the output analysis system judging whether the predetermined initial optimal cutoff value satisfies a predetermined minimum sensitivity or a predetermined minimum specificity, and a step of determining the initial optimal cutoff value as the optimal cutoff value if the initial optimal cutoff value satisfies a predetermined minimum sensitivity or a predetermined minimum specificity as a result of the judgment, and searching for the optimal cutoff value by sequentially changing the initial optimal cutoff value in certain units until the minimum sensitivity or the minimum specificity is satisfied if the initial optimal cutoff value does not satisfy the minimum sensitivity or the minimum specificity.

[0023] The neural network output analysis method may further include a step in which the output analysis system determines a pixel-by-pixel reference value for each pixel using the determined optimal reference value, and the output analysis system uses the determined pixel-by-pixel reference value to detect the presence or absence of a disease for each pixel in the diagnostic biological image to be diagnosed, and the pixel-by-pixel reference value may be determined by correcting the optimal reference value using an output value of the neural network for at least one predetermined surrounding pixel, with a target pixel as a reference.

[0024] Another example of a neural network output analysis method includes a step in which an output analysis system acquires an output result value of the neural network for each pixel of a diagnostic biological image to be diagnosed, and a step in which the output analysis system compares the acquired output result value with a predetermined reference value to detect the presence or absence of a disease for each pixel, wherein the reference value can be determined to be different for each pixel, and is determined by correcting an overall reference value determined by a predetermined method for the entire diagnostic biological image using an output result value of the neural network for at least one predetermined surrounding pixel based on a target pixel.

[0025] The above method may be implemented by a computer program.

[0026] According to another embodiment, an output analysis system for a neural network (200) includes a memory having a processor and a program stored therein, and the processor drives the program to determine an optimal reference value serving as a standard for detecting the presence or absence of a disease in a pixel depending on whether an output result value of the neural network for each pixel is equal to or greater than an optimal reference value, and determines an optimal cutoff value for determining whether a detected lesion area on a detected lesion area on a biological image identified when the presence or absence of a disease in each pixel of the biological image is detected using the determined optimal reference value, and determines an optimal cutoff value for determining whether the detected lesion area is valid, and when an output result corresponding to a diagnostic biological image to be diagnosed is input from the neural network, an output analysis is performed on the output result using the determined optimal reference value and the optimal cutoff value.

[0027] The processor may run the program to use the neural network to obtain output values ​​for each pixel of a plurality of biological images, set a plurality of candidate reference values, calculate a degree of match when each of the set candidate reference values ​​is applied to detect disease onset (here, the degree of match is determined according to the degree of match between the detection result detected by applying the corresponding candidate reference value to each pixel and the annotated result), and determine the optimal reference value based on the degree of match of each of the calculated candidate reference values.

[0028] The processor may run the program to use the neural network to obtain an output value for each pixel of a plurality of biological images, and detect at least one first lesion area contained in each of the plurality of biological images using the output value for each pixel of the obtained plurality of biological images and the optimal reference value to obtain the number of pixels contained in the first lesion area, and determine the optimal cutoff value, which serves as a standard for judging the validity of the detected first lesion area, based on the number of pixels for the at least one detected first lesion area and whether or not a disease has actually occurred.

[0029] The processor may run the program to determine an initial optimal cutoff value, determine whether the determined initial optimal cutoff value satisfies a predetermined minimum sensitivity or a predetermined minimum specificity, and if the determination result indicates that the initial optimal cutoff value satisfies the minimum sensitivity or the minimum specificity, determine the initial optimal cutoff value as the optimal cutoff value, and if the initial optimal cutoff value does not satisfy the minimum sensitivity or the minimum specificity, search for the optimal cutoff value by sequentially changing the initial optimal cutoff value in certain units until the initial optimal cutoff value satisfies the minimum sensitivity or the minimum specificity.

[0030] The processor runs the program to determine a pixel-by-pixel reference value for each pixel using the determined optimal reference value, and uses the determined pixel-by-pixel reference value to detect the presence or absence of a disease at each pixel in the diagnostic biological image to be diagnosed. The pixel-by-pixel reference value may be determined by correcting the optimal reference value using an output value of the neural network for at least one predetermined surrounding pixel, with the target pixel as a reference.

[0031] According to another aspect, a neural network output analysis system according to the technical idea of ​​the present invention includes a processor and a memory having a program stored therein, and the processor drives the program to obtain an output result value of the neural network for each pixel of a diagnostic biological image to be diagnosed, and compares the output result value with a predetermined reference value to detect the presence or absence of a disease for each pixel, wherein the reference value can be determined to be different for each pixel, and is determined by correcting an overall reference value determined in a predetermined manner for the entire diagnostic biological image using an output result value of the neural network for at least one predetermined surrounding pixel based on a target pixel.

[0032] According to another aspect of the invention there is provided a computer program recorded on a medium for being installed in a data processing device and for carrying out the method as described above. Effect of the Invention

[0033] According to the technical idea of ​​the present invention, a reference value for determining whether a pixel is included in a lesion area is effectively determined for each pixel so that the lesion area detected based on the analysis results of the output of a neural network coincides as closely as possible with the actual lesion area (annotated lesion area), thereby achieving the effect of improving the accuracy of disease diagnosis based on the output analysis results compared to the case where a reference value determined arbitrarily in advance is used.

[0034] In addition, by defining and utilizing a cutoff value for the lesion area analyzed using the reference value determined in this manner, it is possible to analyze the output of the neural network in such a way that the presence or absence of a lesion area detected on a sample-by-sample (biological image-by-biological image) basis coincides as closely as possible with the presence or absence of a disease in the actual sample. [Brief description of the drawings]

[0035] In order to more fully understand the drawings referred to in the detailed description of the present invention, a brief description of each of the drawings is provided. [Figure 1] FIG. 1 is a diagram for explaining a schematic process of detecting a lesion area using a conventional neural network. [Diagram 2] FIG. 1 is a diagram for explaining a schematic system configuration for realizing a neural network output analysis method according to an embodiment of the present invention. [Diagram 3] 1 is a diagram for explaining a schematic configuration of a neural network output analysis system according to an embodiment of the present invention; [Figure 4] 1 is a flow chart for illustrating a concept of analyzing an output result of a neural network according to an embodiment of the present invention. [Diagram 5] 4 is a flowchart illustrating a method for determining an optimal reference value according to an embodiment of the present invention. [Figure 6] 4 is a flowchart illustrating a method for determining an optimal cutoff value according to an embodiment of the present invention. [Figure 7] 4 is a flowchart illustrating a method for determining a reference value for each pixel according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The present invention can be modified in various ways and can have various embodiments, and a specific embodiment will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, and it should be understood that the present invention includes all modifications, equivalents, or alternatives within the spirit and technical scope of the present invention. In describing the present invention, if it is determined that a detailed description of related publicly known technology may obscure the gist of the present invention, the detailed description will be omitted.

[0037] Terms such as "first" and "second" may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.

[0038] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless otherwise clearly indicated in the context.

[0039] In this specification, terms such as "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof set forth in the specification, but should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0040] In addition, in this specification, when a component "transmits" data to another component, it means that the component can transmit the data directly to the other component, or can transmit the data to the other component via at least one other component. Conversely, when a component "directly transmits" data to another component, it means that the data is transmitted from the component to the other component without passing through any other component.

[0041] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, in which the same reference numerals in each drawing denote the same elements.

[0042] FIG. 2 is a diagram for explaining a schematic system configuration for implementing the neural network output analysis method according to the embodiment of the present invention.

[0043] 2, an output analysis system 100 according to the technical idea of ​​the present invention can be installed in a predetermined server 10 to realize the technical idea of ​​the present invention. The server 10 means a data processing device having a computing capability for realizing the technical idea of ​​the present invention, and an average expert in the technical field of the present invention can easily deduce that not only a data processing device accessible by a client via a network but also any device capable of performing a specific service, such as a personal computer, a mobile terminal, etc., can be defined as a server.

[0044] The output analysis system 100 may be implemented to receive output results output by the neural network 200 from the pre-trained neural network 200 .

[0045] The output analysis system 100 may communicate with the neural network 200 via wired or wireless communication, or may receive the output results via a predetermined protocol such as a function call.

[0046] The neural network 200 may be trained to output a probability value for the presence or absence of a given disease. The neural network 200 can output a numerical value indicating the possibility of the occurrence of the disease, i.e., a probability value, for each pixel in the biological image.

[0047] The technical ideas for realizing the neural network 200 to perform such functions are disclosed in detail in the prior art Korean patent registration numbers (102172213, Two-face disease diagnosis system and method therefor) and (101944536, Disease diagnosis system and method using neural network) filed by the applicant, and therefore a detailed description thereof will be omitted in this specification.

[0048] The server 10 including the neural network 200 and the output analysis system 100 can be a system for diagnosing a disease, i.e., a diagnostic system. The neural network 200 and the output analysis system 100 may be provided in one physical device, i.e., the server 10, but depending on the embodiment, they may be provided in different physical devices, and various modifications are possible as needed, as can be easily deduced by an average expert in the technical field of the present invention.

[0049] The diagnostic system 10 can communicate with a predetermined terminal 20, 20-1 in order to train the neural network 200.

[0050] The terminal 20, 20-1 may transmit a plurality of training data to the neural network 200 in order to train the neural network 200, and the neural network 200 may be trained using the received training data.

[0051] The neural network 200 may include, but is not limited to, a convolutional neural network (CNN), which is widely known in the field of deep learning.

[0052] Furthermore, the terminals 20 and 20-1 can output a biological image to be diagnosed to the diagnostic system 10 in order to diagnose a disease using the trained neural network 200.

[0053] As a result, the neural network 200 can transmit output results including a probability value for the presence or absence of a disease for each pixel of the biological image to the output analysis system 100, and the output analysis system 100 can analyze the received output results according to the technical idea of ​​the present invention and output the analysis results. The output analysis results can be transmitted to the terminals 20 and 20-1.

[0054] The analysis result may be a result of dividing a lesion area in a biological image, and may include information on whether each pixel is included in the lesion area.

[0055] The output analysis system 100 according to the technical idea of ​​the present invention receives an output result from the neural network 200, analyzes the received output result, and outputs an analysis result according to a predetermined criterion so as to match as closely as possible with the actual disease state (presence or absence of disease and / or pixels where the disease is manifested, i.e., the lesion area).

[0056] Needless to say, the actual disease state can use annotation information annotated by experts.

[0057] For this reason, the output analysis system 100 must determine which reference value to use for each pixel, and if the reference value is exceeded, it must be determined that a disease has developed in that pixel.

[0058] That is, according to the technical idea of ​​the present invention, instead of determining the presence or absence of a disease for each pixel using a reference value that is arbitrarily determined in advance by the user's will, a process can be performed in which a reference value is determined so as to match the actual disease state as much as possible using multiple output results of the trained neural network 200. The reference value determined through such a process can be defined as an optimal reference value in this specification.

[0059] After going through the process of determining the optimal reference value, the output analysis system 100 receives the output result of the neural network 200 corresponding to the biological image to be actually diagnosed, and can determine for each pixel whether the corresponding pixel is included in the lesion area using the determined optimal reference value.

[0060] Furthermore, according to the technical idea of ​​the present invention, the output analysis system 100 does not only use the optimal reference value, but can also set a criterion in advance for determining whether the lesion area detected by the optimal reference value is actually a valid lesion area. Such a criterion can be defined as a cutoff value in this specification.

[0061] That is, according to the technical idea of ​​the present invention, output analysis system 100 can detect a lesion area by analyzing the output result of neural network 200 of a biological image to be diagnosed using an optimal reference value, and can finally determine the lesion area in the biological image by re-determining whether the detected lesion area is actually a valid lesion area.

[0062] The neural network 200 may be trained to minimize the sum of errors in the entire biological image through the probability value for the presence or absence of disease for each pixel, and in this case, if the output result of the neural network 200 is used as is, the error for each pixel may be small. However, if the output result of the neural network 200 is used as is (i.e., using the judgment for the presence or absence of disease for each pixel) to detect a lesion area, there is a high possibility that the result will not match the presence or absence of disease for each actual specimen (each biological image).

[0063] The reason for this is that even if the likelihood of a disease manifesting in a very localized area (e.g., an area corresponding to a specific pixel) is high, this may not mean that the disease will actually manifest, and the actual presence or absence of a disease is more likely to be determined by taking into account the conditions of the localized area corresponding to a specific pixel and the surrounding areas, rather than being determined solely by the individual condition of the localized area corresponding to the specific pixel.

[0064] Therefore, when a judgment is made as to whether or not a disease is present for each pixel using the probability value of the presence or absence of a disease for each pixel, and the lesion area is determined using only that result, no matter how optimized the optimal standard value is, there is always the possibility that the determined lesion area is actually an area where the disease is not present,

[0065] Therefore, according to the technical idea of ​​the present invention, not only is an optimal reference value determined so that the presence or absence of a disease for each pixel detected as the analysis result of the output analysis system 100 coincides as closely as possible with the actual condition (annotated information), but an optimal cut-off value can also be determined so that the lesion area detected using the determined optimal reference value coincides as closely as possible with the actual condition (annotated information).

[0066] The optimal cutoff value can be used as a criterion for determining whether a lesion area detected using the optimal reference value is actually a valid lesion area, and the optimal cutoff value may be a criterion for the number of pixels contained in the detected lesion area.

[0067] The output analysis system 100 primarily detects the lesion area using the optimal reference value, and can determine the lesion area as a valid lesion area only if the number of pixels contained in the detected lesion area is equal to or greater than the optimal cutoff value.

[0068] Ultimately, according to the technical idea of ​​the present invention, the output analysis system 100 can undergo a process of determining a criterion for analyzing the output results of the trained neural network 200 so as to match the actual state as much as possible, separate from the training of the neural network 200.

[0069] In addition, such criteria may include an optimal reference value that causes the analysis result for each pixel (the judgment result for the presence or absence of disease for each pixel) to match the actual condition as closely as possible, and / or an optimal cut-off value that causes the lesion area detected using the optimal reference value to match the actual condition as closely as possible.

[0070] An output analysis system 100 for realizing such a technical idea can have a configuration as shown in FIG.

[0071] FIG. 3 is a diagram for explaining a schematic configuration of a neural network output analysis system according to an embodiment of the present invention.

[0072] The output analysis system 100 may include a memory 120 in which a program for implementing the technical concept of the present invention is stored, and a processor 110 for executing the program stored in the memory 120 .

[0073] An average expert in the technical field of the present invention can easily deduce that the processor 110 can be named by various names such as a CPU, a mobile processor, etc., depending on the implementation of the output analysis system 100. In addition, the output analysis system 100 may be realized by organically combining a plurality of physical devices, and in such a case, an average expert in the technical field of the present invention can easily deduce that the output analysis system 100 of the present invention can be realized by providing at least one processor 110 for each physical device.

[0074] The memory 120 may be realized by any form of storage device in which a program is stored and which the processor can access to run the program. Also, depending on the hardware implementation, the memory 120 may be realized by multiple storage devices rather than one storage device. Also, the memory 120 may include not only a main storage device but also a temporary storage device. Also, the memory 120 may be realized by a volatile memory or a non-volatile memory, and can be defined to include any form of information storage means in which a program is stored and which can be run by a processor.

[0075] The program may be a set of codes for determining optimal reference values ​​and / or optimal cut-off values ​​according to the technical concept of the present invention, as described below. The program may also determine different reference values ​​for each pixel, i.e., pixel-by-pixel reference values, as described below.

[0076] In addition, when a biometric image to be diagnosed is input into the diagnostic system 10, the program receives an output result corresponding to the biometric image from the neural network 200, and analyzes the received output result using the determined optimal reference value, optimal cutoff value, and / or reference value for each pixel, thereby outputting the analysis result.

[0077] Depending on the embodiment, the output analysis system 100 may be a system operated and implemented by an entity (e.g., a medical institution) that wishes to diagnose a disease using the neural network 200 according to the technical ideas of the present invention, and may be implemented in various ways, such as a web server, a computer, etc., and may be defined as including any type of data processing device capable of performing the functions defined in this specification.

[0078] Furthermore, the output analysis system 100 may be provided in the same physical device as the neural network 200, or they may be provided in different physical devices. The output analysis system 100 and the neural network 200 may be combined to implement a diagnosis system 10 that performs diagnosis.

[0079] In addition, various peripheral devices 130 may be further provided in accordance with the embodiment of the output analysis system 100. For example, a person skilled in the art of the present invention can easily deduce that a keyboard, a monitor, a graphic card, a communication device, a scanner, etc. may be further included in the output analysis system 100 as peripheral devices.

[0080] Hereinafter, in this specification, when the output analysis system 100 performs a specified function, it will be easily understood by an average expert in the technical field of the present invention that the processor 110 drives a program stored in the memory 120 to perform the corresponding function.

[0081] FIG. 4 is a flow chart for explaining a concept of analyzing the output result of a neural network according to an embodiment of the present invention.

[0082] Referring to FIG. 4, to realize the neural network output analysis method according to the technical idea of ​​the present invention, the output analysis system 100 may determine an optimal reference value for analyzing the output result of the neural network 200 (S100).

[0083] In addition, the lesion area can be simply detected using the optimal reference value and the detected detected lesion area can be determined as the final lesion area, but as described above, the output analysis system 100 may further determine an optimal cutoff value for determining whether or not the detected lesion area detected using the determined optimal reference value is valid (S200).

[0084] After determining the optimal reference value and / or optimal cut-off value in this manner, the output analysis system 100 may then, upon receiving an output result corresponding to the biological image to be diagnosed from the neural network 200, analyze the received output result using the previously determined optimal reference value and / or optimal cut-off value, and output the analysis result.

[0085] The analysis result may be the final determination of whether or not a pixel has a disease and / or information on the lesion area (e.g., position, boundary, number of pixels, etc.) Of course, even if a specific pixel is initially determined to be included in the lesion area by using the optimal reference value, if the corresponding lesion area is determined to be invalid through the optimal cutoff value, the initial determination for the specific pixel becomes invalid, and it can be finally determined that the pixel is not included in the lesion area (no disease is present).

[0086] The process of determining the optimal reference value is as shown in FIG.

[0087] FIG. 5 is a flow chart illustrating a method for determining an optimal reference value according to an embodiment of the present invention.

[0088] Referring to FIG. 5, first, the output analysis system 100 can obtain output results including output values ​​for each pixel of each of a plurality of biometric images from the neural network 200 (S110).

[0089] The plurality of biometric images may be biometric images that have not been used in learning the neural network 200. In addition, the plurality of biometric images may all be annotated.

[0090] This allows the output analysis system 100 to set a plurality of candidate reference values ​​(S120). The plurality of candidate reference values ​​may be set to have a difference of, for example, a preset unit interval (for example, 0.01, 0.001, etc.).

[0091] This allows the output analysis system 100 to calculate the degree of match for each set candidate reference value. The degree of match may be information indicating how well the determination result of the output analysis system 100 for each pixel matches the actual state (i.e., the annotated result) for multiple biological images.

[0092] For this purpose, the output analysis system 100 can analyze the output result of the biological image using each candidate reference value and generate the presence or absence of disease for each pixel, i.e., the analysis result. For example, if the probability value included in the output result for a specific pixel is equal to or greater than the candidate reference value, it can be determined that the pixel is included in the lesion area, and if not, it can be determined that the pixel is not included in the lesion area.

[0093] In addition, the degree of match can be calculated using the analysis results generated for each pixel, and the degree of match can be determined according to the degree to which the detection result detected by applying the corresponding candidate reference value for each pixel matches the annotated result.

[0094] The degree of agreement can be defined in a variety of ways.

[0095] In one example, the degree of match can be defined using any one or a combination of accuracy, IoU (Intersection over Union), or DSC (Dice Similarity Coefficient), which is calculated by comparing the judgment results for each pixel with the annotation results for multiple biometric images.

[0096] Alternatively, the degree of match can be defined using any one or a combination of the average accuracy, the average Intersection over Union (IoU), or the average Dice Similarity Coefficient (DSC) calculated for each of multiple biometric images.

[0097] If output analysis system 100 defines the detected lesion area as the first lesion area as a result of analysis by applying each candidate reference value, and defines the lesion area included in the pre-annotated information as the second lesion area, TP can indicate the number of pixels included in both the first lesion area and the second lesion area, FN can indicate the number of pixels included only in the second lesion area, FP can indicate the number of pixels included only in the first lesion area, and TN can indicate the number of pixels not included in both the first lesion area and the second lesion area.

[0098] The output analysis system 100 can calculate TP, TN, FP, and FN from the analysis results obtained by using each candidate reference value, and can calculate accuracy, IoU, and DSC based on the calculated values. Accuracy, IoU, and DSC are widely used as indices for measuring the performance of a trained neural network, and accuracy can be defined as (TP+TN) / (TP+TN+FP+FN), IoU as TP / (TP+FP+FN), and DSC as 2×TP / (2×TP+FP+FN).

[0099] Needless to say, the output analysis system 100 can determine the candidate reference value with the highest degree of match as the optimal reference value based on the degree of match of each of the candidate reference values.

[0100] Meanwhile, the process of determining the optimal cutoff value is as shown in FIG.

[0101] FIG. 6 is a flow chart illustrating a method for determining an optimal cutoff value according to an embodiment of the present invention.

[0102] 6, the output analysis system 100 can obtain an output result including an output value for each pixel of a plurality of biometric images from the neural network 200 (S210). The plurality of biometric images used when determining the optimal cutoff value may also be biometric images that have not been used in the learning of the neural network 200. Of course, they may be different from the biometric images used when determining the optimal reference value, or may be the same. All of the biometric images at this time may be annotated.

[0103] In this way, output analysis system 100 can detect a lesion area using a predetermined optimum reference value (S220). That is, if the probability value determined by neural network 200 for each pixel is equal to or greater than the optimum reference value, the pixel is detected as being included in the lesion area, and if not, the pixel is detected as not being included in the lesion area, thereby making it possible to detect a lesion area from each biological image.

[0104] The lesion area may be a collection of pixels that are determined to have a disease manifestation, i.e., pixels whose probability value is equal to or exceeds an optimal threshold value, and the output analysis system 100 can calculate the number of pixels contained in each lesion area (S230).

[0105] Also, an optimal cutoff value can be determined based on the calculated number of pixels (S240).

[0106] In this case, the calculated number of pixels may be the total for each biological image or the total for each lesion area. For example, a single biological image may have multiple lesion areas that are independent of each other (separate from each other).

[0107] In this case, according to one embodiment, even if multiple lesion areas are present in one biometric image, the number of pixels contained in the lesion area for each biometric image can be calculated by simply adding up all the numbers of pixels contained in the lesion area for each biometric image, or the number of pixels can be calculated for each independent lesion area, rather than for each biometric image.

[0108] In the former case, the optimal cutoff value can be determined so that the presence or absence of a disease in the information annotated based on a single biological image matches as closely as possible to the results of the lesion area detected by the output analysis system 100; in the latter case, the optimal cutoff value can be determined so that the presence or absence of a disease in the information annotated based on each lesion area matches as closely as possible to the results of the lesion area detected by the output analysis system 100.

[0109] That is, even when no disease is actually present, output analysis system 100 may detect a lesion area, and in such a case, it is preferable to be able to cut off the lesion area.

[0110] Therefore, the output analysis system 100 can determine an optimal cutoff value (for each biological image or for each lesion area) that allows the actual disease state to match the detected result as closely as possible when lesion areas having a pixel count below the optimal cutoff value are invalidated after initially detecting lesion areas in multiple biological images.

[0111] For example, when the optimal cutoff value is determined by adding up the number of pixels based on a biological image, even if there are multiple lesion areas detected in a specific biological image, the number of pixels contained in each lesion area can be simply added up, and the optimal cutoff value can be determined so that the total number of pixels matches the presence or absence of a disease in the specific biological image. For example, the output analysis system 100 can determine the optimal cutoff value using a predetermined algorithm, and at this time, if the specific biological image does not show a disease, the algorithm can be designed so that a value larger than the total number of pixels is more likely to be determined as the optimal cutoff value, and if the specific biological image shows a disease, the algorithm can be designed so that a value smaller than the total number of pixels is more likely to be determined as the optimal cutoff value.

[0112] For example, when the optimal cutoff value is determined by adding up the number of pixels based on each lesion area, the optimal cutoff value can be determined so that the number of pixels in each detected lesion area matches the presence or absence of disease in the lesion area. For example, the output analysis system 100 can determine the optimal cutoff value using a predetermined algorithm, and in this case, if a specific lesion area detected by the output analysis system 100 does not actually have a disease, the algorithm can be designed so that a value larger than the number of pixels in the specific lesion area is more likely to be determined as the optimal cutoff value, and if a disease actually has appeared in the specific lesion area, the algorithm can be designed so that a value smaller than the number of pixels in the specific lesion area is more likely to be determined as the optimal cutoff value.

[0113] Determination of such an optimal cutoff value can be defined as a process of searching for an optimal value that makes the detection result of the output analysis system 100 when the optimal cutoff value is applied to each biological image (each lesion area) match the actual disease state as closely as possible, and such a process can be defined as a one-dimensional (linear) optimization problem.

[0114] Such an optimization problem can be defined as a linear regression, logistic regression, or support vector machine problem, as is widely known, and an average expert in the technical field of the present invention can easily deduce that the optimal cut-off value can be determined using at least one of such algorithms.

[0115] Meanwhile, in a system for diagnosing a disease, there may be cases where the system must be designed to satisfy at least a satisfactory sensitivity or specificity. Sensitivity is the ratio of an area where the disease is actually present to an area where the disease is detected as being present, and specificity can mean the ratio of an area where the disease is not actually present to an area where the disease is not detected as being present, and there are cases where sensitivity or specificity is prioritized depending on the type of disease.

[0116] Therefore, the output analysis system 100 may also determine an optimal cut-off value such that a preset minimum sensitivity or specificity is met.

[0117] To this end, the output analysis system 100 first determines the optimal cutoff value in the above-described manner, and the thus determined optimal cutoff value may become the initial optimal cutoff value (S240).

[0118] This allows the output analysis system 100 to determine whether the current optimal cutoff value satisfies the minimum sensitivity or minimum specificity (S250), and if the determination results in yes, the current optimal cutoff value can be identified as the final optimal cutoff value (S270).

[0119] If the result of the judgment (S250) is that the current optimal cutoff value does not satisfy the minimum sensitivity or minimum specificity, the current optimal cutoff value is changed by a predetermined unit interval (e.g., 1, 5, etc.) (S260), and the process of determining whether the changed optimal cutoff value satisfies the minimum sensitivity or minimum specificity is repeated, so that an optimal cutoff value that finally satisfies the minimum sensitivity or minimum specificity can be identified (S270).

[0120] Of course, to meet the minimum sensitivity, the number of cutoffs can be decreased by a predetermined unit interval at the current optimal cutoff value, and to meet the minimum specificity, the number of cutoffs can be increased by a predetermined unit interval at the current optimal cutoff value, and then it can be determined whether the changed optimal cutoff value meets the minimum sensitivity or minimum specificity.

[0121] On the other hand, the above-mentioned optimal reference value can be uniformly applied to all pixels contained in a biological image and can serve as a standard for determining whether each pixel is included in a lesion area, and according to the technical idea of ​​the present invention, such a reference value may be set differently for each pixel.

[0122] The reason for this is that if the same reference value (optimum reference value) is applied to all pixels, the detection result may be completely different from that of the surrounding pixels, and since the detection result of a specific pixel is usually closely related to the detection result of the surrounding pixels, it may be more preferable to correct this. For example, there may be cases where a specific pixel is determined to be disease-free, but all of the surrounding pixels of the specific pixel are determined to be disease-affected, and in such cases, there is a high possibility that the results are different from the actual disease manifestation state.

[0123] Therefore, according to the technical idea of ​​the present invention, it is necessary to apply a different reference value to each pixel to determine the presence or absence of a disease for each pixel. To this end, the output analysis system 100 can determine a reference value for each pixel by first determining an optimal reference value, as described in FIG. 5, and then correcting the optimal reference value taking into account the probability values ​​of the surrounding pixels.

[0124] One such example is described with reference to FIG.

[0125] FIG. 7 is a flowchart illustrating a method for determining a reference value for each pixel according to an embodiment of the present invention.

[0126] 7, the output analysis system 100 can first determine an optimal reference value commonly used in the analysis of a biological image, as shown in FIG. 5 (S140).

[0127] Also, the optimum reference value can be corrected for each pixel to specify the reference value for each pixel (S170).

[0128] At this time, the correction value for correcting the optimal reference value for each pixel can use the probability value of the surrounding pixels of the pixel (that is, the output value of the neural network 200).

[0129] For example, the output analysis system 100 may set a window including a specific pixel (target pixel) to determine a pixel-by-pixel reference value of the specific pixel. The window may have a preset size (e.g., 3 by 3, 5 by 5, etc.).

[0130] Thus, the output analysis system 100 can identify other pixels included in the window that are neighboring pixels of the specific pixel, and check the output values ​​of the neural network 200 of the other pixels (S150).

[0131] Also, a correction value to be applied to a particular pixel may be calculated based on the output values ​​of the neural network 200 for other pixels (S160). The correction value may be determined in various ways.

[0132] For example, the standard deviation of the output values ​​of a particular pixel and other pixels, or a value obtained by dividing the standard deviation by a certain value, can be the correction value.

[0133] This allows output analysis system 100 to specify the reference value for each pixel at a particular pixel by adding or subtracting a correction value from the optimal reference value (S170).

[0134] When the reference value for each pixel is determined in this manner, the probability values ​​of disease occurrence in the surrounding pixels influence the judgment as to whether or not a disease is present in a particular pixel, so that the judgment result of a particular pixel is less likely to be completely different from the judgment of the surrounding pixels.

[0135] Meanwhile, the output analysis method according to the embodiment of the present invention may be realized in the form of computer readable program instructions and stored in a computer readable recording medium, and the control program and the target program according to the embodiment of the present invention may also be stored in a computer readable recording medium. The computer readable recording medium includes any kind of recording device in which data that can be read by a computer system is stored.

[0136] The program instructions contained on the recording media may be those specially designed and constructed for the present invention, or they may be of the type commonly known and available to those having skill in the software arts.

[0137] Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, etc. Also, computer readable recording media can be distributed among computer systems connected via a network to store and execute computer readable code in a distributed manner.

[0138] Examples of program instructions include not only machine code, such as produced by a compiler, but also high level language code executable by a device for processing information electronically, such as a computer, using an interpreter or the like.

[0139] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0140] The above description of the present invention is for illustrative purposes only, and a person having ordinary skill in the art to which the present invention pertains can easily understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described in a single form may be implemented in a distributed form, and similarly, components described as distributed may be implemented in a combined form.

[0141] The scope of the present invention is indicated by the following claims rather than the above detailed description, and all modifications and variations derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. [Industrial Applicability]

[0142] The present invention can be used for a neural network output analysis method and a system therefor.

Claims

1. A method for analyzing an output of a neural network, the method comprising: analyzing an output result of a neural network trained to output a probability of disease occurrence for each pixel of a biological image, the method comprising: an output analysis system for determining, for each pixel, the optimal reference value serving as a criterion for detecting the presence or absence of a disease in the corresponding pixel according to whether the output result value of the neural network is equal to or greater than the optimal reference value; determining an optimal cutoff value for determining whether a detected lesion area on a biological image identified when the output analysis system detects the presence or absence of a disease on a pixel-by-pixel basis in the biological image using the determined optimal reference value, for determining whether the detected lesion area is valid; and when the output analysis system receives an output result corresponding to a diagnostic biological image to be diagnosed from the neural network, the output analysis system performs an output analysis on the output result using the determined optimal reference value and the determined optimal cutoff value, The step of determining the optimal reference value comprises: The output analysis system obtains an output value for each pixel of each of a plurality of biometric images using the neural network; The output analysis system sets a plurality of candidate reference values, and for each of the set candidate reference values, calculates a degree of agreement between the detection result obtained by applying the corresponding candidate reference value to each pixel when the disease expression is detected by applying the corresponding candidate reference value and the annotated result; determining the optimal reference value based on the degree of agreement of each of the candidate reference values ​​calculated by the output analysis system; The step of calculating a degree of agreement when determining the presence or absence of a disease by applying each of the candidate reference values ​​includes: The accuracy, Intersection over Union (IoU), or Dice Similarity Coefficient (DSC) of the entire plurality of biometric images; the average of accuracy, Intersection over Union (IoU), or Dice Similarity Coefficient (DSC) of each of the plurality of biometric images; Calculating the degree of agreement using at least one of the following: The accuracy is defined as (TP+TN) / (TP+TN+FP+FN), the IoU is defined as TP / (TP+FP+FN), and the DSC is defined as 2×TP / (2×TP+FP+FN), wherein the TP is the number of pixels included in both the first lesion area detected by the output analysis system and the annotated second lesion area, the FN is the number of pixels included only in the second lesion area, the FP is the number of pixels included only in the first lesion area, and the TN is the number of pixels not included in either the first lesion area or the second lesion area, The step of determining an optimal cutoff value for determining whether the detected lesion area is valid comprises: The output analysis system obtains an output value for each pixel of each of a plurality of biometric images using the neural network; the output analysis system detects at least one first lesion area included in each of the plurality of acquired biological images using the output value for each pixel of each of the plurality of acquired biological images and the optimal reference value, thereby acquiring the number of pixels included in the first lesion area; determining an optimal cutoff value as a criterion for judging the validity of the detected first lesion area based on the number of pixels for each of the at least one detected first lesion area and whether or not a disease has actually occurred; A method for analyzing the output of a neural network, comprising:

2. The step of determining an optimal cutoff value serving as a criterion for judging the validity of the detected first lesion area based on a second lesion area which is an annotated lesion area of ​​each of the plurality of biological images and the number of pixels included in the first lesion area, includes:

2. The method of claim 1, wherein the output analysis system determines the optimal cutoff value using one of linear regression, logistic regression, or a support vector machine.

3. The step of determining an optimal cutoff value serving as a criterion for judging the validity of the detected first lesion area based on a second lesion area which is an annotated lesion area of ​​each of the plurality of biological images and the number of pixels included in the first lesion area, includes: said power analysis system determining an initial optimal cutoff value; The output analysis system determines whether the predetermined initial optimal cutoff value satisfies a minimum sensitivity at which a predetermined ratio of actually diseased regions detected as diseased or a minimum specificity at which a predetermined ratio of actually diseased regions detected as diseased is minimized; If the result of the judgment indicates that the condition is satisfied, the initial optimal cutoff value is determined as the optimal cutoff value, and if the condition is not satisfied, the initial optimal cutoff value is sequentially changed in a certain unit to search for the optimal cutoff value until the minimum sensitivity or the minimum specificity is satisfied; 2. The method of claim 1, further comprising:

4. The method for analyzing the output of a neural network includes the steps of: the output analysis system further comprising determining a pixel-by-pixel criterion value for each pixel using the determined optimal criterion value; The output analysis system detects the presence or absence of a disease for each pixel in the diagnostic biological image to be diagnosed using the determined reference value for each pixel, 2. The neural network output analysis method according to claim 1, wherein the reference value for each pixel is determined by correcting the optimal reference value using an output value of the neural network for at least one predetermined peripheral pixel of a target pixel.

5. A computer program recorded on a medium for being installed in a data processing device and for carrying out the method according to any one of claims 1 to 4.

6. A processor; A memory in which a neural network and a program trained to output a probability of disease occurrence for each pixel of a biological image are stored; Including, The processor runs the program, determining, for each pixel, the optimal reference value serving as a criterion for detecting the presence or absence of a disease in the corresponding pixel depending on whether the output result value of the neural network is equal to or greater than the optimal reference value; determining an optimal cutoff value for determining whether a detected lesion area on a biological image specified when detecting the presence or absence of a disease on a pixel-by-pixel basis in the biological image using the determined optimal reference value, the detected lesion area being valid; When an output result corresponding to a diagnostic biological image to be diagnosed is input from the neural network, an output analysis is performed on the output result using the determined optimal reference value and the determined optimal cutoff value; The determination of the optimal reference value is Using the neural network, an output value for each pixel of each of a plurality of biometric images is obtained; A plurality of candidate reference values ​​are set, and for each of the set candidate reference values, a degree of match is calculated according to the degree to which a detection result obtained by applying the corresponding candidate reference value to each pixel when a disease expression is detected by applying the corresponding candidate reference value matches an annotated result; determining the optimal reference value based on the degree of coincidence of each of the candidate reference values ​​calculated; The calculation of the degree of agreement is The accuracy, Intersection over Union (IoU), or Dice Similarity Coefficient (DSC) of the entire plurality of biometric images; the average of accuracy, Intersection over Union (IoU), or Dice Similarity Coefficient (DSC) of each of the plurality of biometric images; Calculating the degree of match using at least one of the following: The accuracy is defined as (TP+TN) / (TP+TN+FP+FN), the IoU is defined as TP / (TP+FP+FN), and the DSC is defined as 2×TP / (2×TP+FP+FN), the TP is the number of pixels contained in both the first lesion area detected by the output analysis system and the annotated second lesion area, the FN is the number of pixels contained only in the second lesion area, the FP is the number of pixels contained only in the first lesion area, and the TN is the number of pixels not contained in either the first lesion area or the second lesion area; Using the neural network, an output value for each pixel of each of a plurality of biometric images is obtained; detecting at least one first lesion area included in each of the plurality of acquired biological images using the output value for each pixel of each of the plurality of acquired biological images and the optimal reference value, thereby obtaining the number of pixels included in the first lesion area; A neural network output analysis system that determines the optimal cutoff value, which serves as a criterion for judging the validity of the detected first lesion area, based on the number of pixels for each of the at least one detected first lesion area and whether or not a disease has actually occurred.

7. The processor runs the program, 7. The neural network output analysis system of claim 6, further comprising: determining an initial optimal cutoff value; determining whether the determined initial optimal cutoff value satisfies a minimum sensitivity at which a predetermined region where a disease is actually manifested is detected as having a disease, or a minimum specificity at which a predetermined region where a disease is not actually manifested is detected as having no disease, and if the initial optimal cutoff value satisfies the minimum sensitivity or the minimum specificity, the initial optimal cutoff value is determined to be the optimal cutoff value if it does not satisfy the minimum sensitivity or the minimum specificity, and searching for the optimal cutoff value by sequentially changing the initial optimal cutoff value in a certain unit until the initial optimal cutoff value satisfies the minimum sensitivity or the minimum specificity.

8. The processor runs the program, determining a pixel-by-pixel reference value for each pixel using the determined optimal reference value; Using the determined reference value for each pixel, the presence or absence of a disease is detected for each pixel in the diagnostic biological image to be diagnosed; 7. The neural network output analysis system according to claim 6, wherein the reference value for each pixel is determined by correcting the optimal reference value using an output value of the neural network for at least one predetermined peripheral pixel of a target pixel.

Images