Method and apparatus for visualizing decision basis of neural network via unfold and conquer attribution guidance

Abstract

A visualization device according to one embodiment may perform the following operations: acquiring a target image; generating a plurality of patch images divided to include a portion of the target image; generating a plurality of heatmaps that visualize the decision of a model when the plurality of patch images are input into a predetermined neural network model; calculating a confidence score for each of the plurality of heatmaps; and generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is placed in the divided area of ​​the target image.

Description

Technology Field

[0001] The present invention relates to a method for visualizing a neural network decision using division and aggregation attribution guidelines and an apparatus for the same. More specifically, it relates to a method for visualizing a neural network decision through the division of a target image and a method of aggregating visualization information, and an apparatus for the same. Background Technology

[0002] An image classification neural network model using an artificial intelligence neural network is trained to classify the correct class from an image using a set of data in which specific images are mapped to class labels.

[0003] In this context, the relationship between the input and output of a neural network model acts as a "black box," making it difficult to directly understand the process or method by which the model handles decisions from input to output. Therefore, since identifying the reasons behind specific decisions opens up the possibility of better understanding and improving artificial intelligence, research on "explainable AI"—which visualizes the model's processing to make it understandable to humans—is actively underway.

[0004] Meanwhile, in the case of image classification neural network models, there have been attempts to visualize the parts most frequently referenced by the model when selecting the correct answer in order to understand why the neural network model determines a specific class for an input image.

[0005] For example, Class Activation Mapping and Layer-wise Relevance Propagation techniques are used to visualize which regions of an input image a neural network model primarily references to derive results.

[0006] Meanwhile, in the case of class activation mapping technology, the range that can be expressed by the explanation map is limited due to the difference in resolution between the input image and the visualization explanation map. For example, in the case of CNN's ResNet50, a square image of 224 pixels is input, but the explanation map is expressed as a square image of 7 pixels, resulting in a 32-fold difference in resolution and thus limiting expressive power.

[0007] Furthermore, hierarchical relevance propagation technology has limitations, such as the phenomenon where image noise occurs and representation becomes unclear when expressed at the pixel level, and the reliability of explanation maps is low due to a lack of ability to provide explanations specialized for specific classes.

[0008] Therefore, if a decision visualization methodology for neural network models is developed that can overcome the aforementioned limitations, it is expected to be of great help to the understanding and advancement of artificial intelligence neural networks. Prior art literature

[0009] Korean Patent Publication No. 10-2446792 (September 20, 2022) The problem to be solved

[0010] The technical problem that the present invention aims to solve is to provide a technology that enhances the granularity of explanation maps and explicit accuracy by class, regardless of the structure of the neural network model. To this end, the invention seeks to provide an improved 'Explainable Artificial Intelligence' technology by raising the level of the explanation map through an approach that segments a target image to generate patch images containing partial information of the target image, and aggregates visualization information of heatmaps for the patch images.

[0011] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0012] A visualization method performed by a visualization device including a processor and memory according to an embodiment of the present invention for achieving the above technical problem may include: an operation of acquiring a target image; an operation of generating a plurality of patch images divided to include a portion of the target image; an operation of generating a plurality of heatmaps that visualize the decision of a model when the plurality of patch images are input to a predetermined neural network model; an operation of calculating a confidence score for each of the plurality of heatmaps; and an operation of generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is accumulated and arranged so as to be in the area divided in the target image.

[0013] According to one embodiment, the operation of generating the plurality of patch images may include: generating a window of a preset size smaller than the target image; moving the window at preset intervals so that the window passes through all areas of the target image; and generating the plurality of patch images by capturing an area of ​​the target image included in the window at each interval.

[0014] According to one embodiment, the operation of generating the plurality of patch images may include the operation of storing location information in which each patch image is divided from the target image.

[0015] According to one embodiment, the operation of generating the plurality of patch images includes the operation of resizing the plurality of patch images to the size of the target image.

[0016] The operation of generating the above description map may include resizing each of the plurality of heatmaps to the size of the partial area and arranging each of the plurality of heatmaps resized to the size of the partial area so as to be located in the divided area of ​​the target image.

[0017] According to one embodiment, the operation of generating the plurality of heatmaps may include the operation of generating a heatmap that visualizes information about the decision of the neural network model by applying a Class Activation Mapping technique or a Layer-wise Relevance Propagation technique to the neural network model when a patch image is input to the neural network model.

[0018] According to one embodiment, the operation of calculating the confidence score may include the operation of calculating the confidence score of a heatmap for a patch image based on the ratio of the logit value output by the neural network model to the class to be observed in the patch image for all classes of the neural network model.

[0019] According to one embodiment, the operation of determining the reliability score can calculate the reliability score for each patch image by applying the following mathematical formula 1 to the logit value output when a patch image is input to the neural network model.

[0020] [Mathematical Formula 1]

[0021]

[0022] ( is the reliability score, is the logit value for the patch image, k is the class to observe in the patch image, Is A function that calculates a confidence score using k as input, where N is the total number of classes, i is the index specifying the class, and j is the index specifying the final node that outputs the logit)

[0023] According to one embodiment, the operation of generating the explanation map may include generating sum information by applying the reliability score of each heatmap as a weight to the pixel information of each heatmap accumulated at a first location, and generating the explanation map by normalizing the sum information by the number of heatmaps accumulated at the first location.

[0024] A visualization device according to an embodiment of the present invention for achieving the above technical problem includes a memory containing instructions; and a processor that executes the instructions, wherein the operation of the processor may include: an operation of acquiring a target image; an operation of generating a plurality of patch images divided to include a portion of the target image; an operation of generating a plurality of heatmaps that visualize the decision of the model when the plurality of patch images are input to a predetermined neural network model; an operation of calculating a confidence score for each of the plurality of heatmaps; and an operation of generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is accumulated and arranged to be in the area divided in the target image.

[0025] A computer program stored in a computer-readable recording medium according to an embodiment of the present invention for achieving the above technical problem, wherein the computer program, when executed by a processor, may include instructions for the processor to perform a method comprising: acquiring a target image; generating a plurality of patch images divided to include a portion of the target image; generating a plurality of heatmaps visualizing the decision of a model when the plurality of patch images are input to a predetermined neural network model; calculating a confidence score for each of the plurality of heatmaps; and generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is accumulated and arranged to fall within the divided area of ​​the target image. Effects of the invention

[0026] According to the present invention as described above, by dividing a target image to generate a patch image containing partial information of the target image and combining visualization information of a heatmap for the patch image to derive an explanatory method with enhanced granularity and accuracy, it is possible to visualize which region of the target image a neural network model primarily refers to when making a specific decision.

[0027] Accordingly, the present invention enables intuitive verification of which features in a target image have a major influence on the decision-making of a neural network model. In particular, in fields with a heavy decision-making burden, such as medicine, it can improve the accuracy and efficiency of the diagnostic process by presenting the basis for diagnosis; and in fields directly related to human casualties, such as autonomous driving, it allows for pre-identification and correction of whether the neural network model's decision-making is derived from incorrect features.

[0028] As such, when the present invention is applied in various fields, users can receive guidance for improving neural network models and datasets, thereby utilizing neural network models more effectively and contributing to the understanding and development of the field of artificial intelligence neural networks.

[0029] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0030] FIG. 1 is a drawing illustrating the overall configuration of a visualization device according to one embodiment. FIG. 2 is a flowchart showing representative steps of a visualization method performed by a visualization device according to one embodiment. Figure 3 is an example diagram to help understand each operation of the visualization method according to the embodiment of Figure 2. FIG. 4 is a flowchart exemplarily illustrating the operation of generating a patch image of step S1020 according to one embodiment. FIG. 5 is an example diagram to help understand each operation of the patch image generation step according to the embodiment of FIG. 4. FIG. 6 is an example diagram to help understand the operation of generating an explanation map according to one embodiment. Figure 7 is a diagram comparing the result of generating an explanation map by applying a conventional visualization method with the result of generating an explanation map by applying an embodiment of the present invention to the conventional visualization method. FIG. 8 is a diagram comparing the result of generating an explanation map by applying a conventional visualization method to the neural network models of ResNet50 and VGG19 with the result of generating an explanation map by applying an embodiment of the present invention to the conventional visualization method. FIG. 9 is a performance evaluation table comparing the performance of a deletion / insertion game test when conventional visualization methods (GCAM, GCAM++, WGCAM) are applied to neural network models of ResNet50, DenseNet121, and InceptionV3, and the performance of a deletion / insertion game test when an embodiment of the present invention is applied to conventional visualization methods (GCAM*, GCAM++*, WGCAM*). FIG. 10 is a performance evaluation table comparing scores for the degree to which each description map specifies an object when various visualization description techniques and embodiments of the present invention are applied. Specific details for implementing the invention

[0031] Detailed information regarding the purpose, technical configuration, and resulting effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. An embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0032] The embodiments disclosed herein should not be interpreted or used to limit the scope of the invention. It is obvious to those skilled in the art that the description including the embodiments herein has various applications. Accordingly, any embodiments described in the detailed description of the invention are illustrative for better explaining the invention and are not intended to limit the scope of the invention to the embodiments.

[0033] The functional blocks shown in the drawings and described below are merely examples of possible implementations. In other implementations, other functional blocks may be used without departing from the spirit and scope of the detailed description. Additionally, while one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0034] Furthermore, the expression that it includes certain components is an "open-ended" expression that merely refers to the existence of such components and should not be understood as excluding additional components.

[0035] Furthermore, when it is stated that one component is "connected" or "joined" to another component, it should be understood that while it may be directly connected or joined to that other component, there may also be other components present in between.

[0036] Below, detailed embodiments of the present invention will be examined with reference to the drawings.

[0037] FIG. 1 is a drawing that exemplarily illustrates the overall configuration of a visualization device (100) (hereinafter referred to as 'device (100)') according to one embodiment.

[0038] However, this is merely a preferred embodiment for achieving the purpose of the present invention, and it is understood that some components may be added or deleted as needed, and that other components may perform the role of one component together.

[0039] A device (100) according to one embodiment may include a processor (10), a network interface (20), a memory (30), a storage (40), and a data bus (50) connecting them, and may further include additional components required to achieve the purpose of the present invention.

[0040] The processor (10) controls the overall operation of each component. The processor (10) may be any one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), or an artificial intelligence processor of a form widely known in the technical field to which the present invention belongs. In addition, the processor (10) may perform operations for at least one application or program to perform an operation or operation performed by the device (100) according to one embodiment.

[0041] The network interface (20) supports wired and wireless internet communication of the device (100) according to one embodiment, and may also support other known communication methods. Accordingly, the network interface (20) may be configured to include a communication module accordingly.

[0042] The memory (30) stores various information, commands and / or information, and can load one or more computer programs (41) from storage (40) to perform operations or actions performed by the device (100) according to one embodiment. In FIG. 1, RAM is shown as one of the memory (30), but it is obvious that various storage media can also be used as memory (30).

[0043] Storage (40) may store one or more computer programs (41) non-temporarily. Such storage (40) may be any one of non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, hard disk (HDD), secondary storage match (SSD), removable disk, or any form of computer-readable recording medium widely known in the art to which the present invention belongs.

[0044] A computer program (41) can be loaded into memory (30) and, by one or more processors (10), perform the operation of acquiring a target image; the operation of generating a plurality of patch images divided to include a portion of the target image; the operation of generating a plurality of heatmaps that visualize the decision of the model when the plurality of patch images are input into a predetermined neural network model; the operation of calculating a confidence score for each of the plurality of heatmaps; and the operation of generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is placed in the divided area of ​​the target image.

[0045] The actions or operations performed by the computer program (41) briefly mentioned above can be viewed as one function of the computer program (41), and a more detailed explanation will be provided later in the description of the method performed by the device (100) according to one embodiment.

[0046] The data bus (50) serves as a path for the movement of commands and / or information between the processor (10), network interface (20), memory (30), and storage (40) described above.

[0047] The device (100) according to the embodiment described above may be in the form of an independent device, for example, an electronic device or a server (including a cloud). Here, the electronic device may be not only a device such as a desktop PC or server device that is fixedly installed and used in one place, but also a portable device that is easy to carry, such as a smartphone, tablet PC, laptop PC, PDA, PMP, etc. Any electronic device that is equipped with a CPU corresponding to the processor (10) and possesses only network functions may be used. For example, the device (100) according to the embodiment may be implemented in the form of a "server" among the forms of electronic devices that are independent devices. Additionally, the device (100) according to the embodiment may be implemented in the form of a "terminal" that a user possesses to use an application or service provided by the server among the forms of electronic devices that are independent devices.

[0048] Hereinafter, a method performed by a device (100) according to one embodiment will be described with reference to FIGS. 2 to FIGS. 10.

[0049] FIG. 2 is a flowchart showing representative steps of a visualization method performed by a device (100) according to one embodiment, and FIG. 3 is a conceptual diagram illustrating each step of the visualization method according to the embodiment of FIG. 2 to aid in understanding.

[0050] Each step disclosed in FIGS. 2 and FIGS. 3 can be performed through a device (100) according to one embodiment, and specifically, the operation of each step can be understood as an operation performed by a processor (10).

[0051] Each step disclosed in FIGS. 2 and FIGS. 3 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and any one step may be included in another step.

[0052] The order of each operation disclosed in FIGS. 2 and FIGS. 3 is arranged for convenience of understanding only, and such order is not limited to a chronological order, and the order may be changed and operated differently depending on the designer's choice.

[0053] Referring to FIG. 2 and FIG. 3 together, in step S1010, the device (100) can acquire a target image. Here, the target image refers to an image input to a neural network model that the user wishes to observe.

[0054] Additionally, at step S1010, the device (100) may acquire a neural network model that is the subject of analysis of 'explainable artificial intelligence'. The neural network model may include a neural network model that receives a target image as input and determines the class of the target image. For example, the neural network model may include a neural network model trained to classify the class of an image based on various learning algorithms.

[0055] The neural network model according to the embodiment of this document refers to a model used in machine learning that has problem-solving capabilities and is composed of artificial neurons (nodes) forming a network through the combination of synapses.

[0056] A neural network model may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the neural network model may include synapses connecting the neurons. In the neural network model, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0057] The form of a neural network model can be defined by the connection patterns between neurons in each layer, the learning process that updates model parameters, and the activation function that generates output values. Each layer of a neural network model may contain one or more neurons and may include synapses connecting the neurons. Each neuron in a neural network model can output an activation value of the activation function in response to input signals, weights, and biases input through the synapses.

[0058] Model parameters of a neural network refer to parameters determined through learning; for example, model parameters may include the weights of synaptic connections and the biases of neurons. Additionally, hyperparameters refer to parameters that must be set prior to training in a machine learning algorithm, and hyperparameters may include the learning rate, epochs, batch size, and initialization functions.

[0059] Accordingly, an embodiment of the present invention presents a method for enhancing the level of an explanation map output from a specific visualized explanation technique by additionally adding a segmentation and aggregation method to the preceding and succeeding stages of the visualized explanation technique when using a visualized explanation technique for a predetermined neural network model. That is, regardless of which visualized explanation technique is used, the present invention derives an enhanced level of explanation map by incorporating the segmentation and aggregation method described below into the preceding and succeeding stages of the visualized explanation technique. Through the description of the subsequent steps, the operation of the device (100) generating a final explanation map by aggregating segmentation and visualization information for a target image in the preceding and succeeding stages of the visualized explanation technique is explained.

[0060] Accordingly, in step S1020, the device (100) may generate a plurality of patch images divided to include a portion of the target image. At this time, the device (100) may store location information where each patch image is divided from the target image.

[0061] A specific example of step S1020 is explained with the following Figures 4 and 5.

[0062] FIG. 4 is a flowchart exemplarily illustrating the operation of generating a patch image in step S1020 according to one embodiment, and FIG. 5 is an example diagram to help understand each operation of the patch image generation step according to the embodiment of FIG. 4.

[0063] Referring to FIGS. 4 and 5, the device (100) creates a window of a preset size smaller than the target image (S1021), moves the window at preset intervals so that the window passes through all areas of the target image (S1022), and can create multiple patch images by capturing areas of the target image included in the window at each window movement interval (S1023). Here, since the captured patch images are smaller than the target image and have the same size as the window, the device (100) can perform an operation of resizing each patch image to have the same size as the target image to unify the resolution (S1024). For example, the device (100) can resize the patch images to the size of the target image through linear interpolation.

[0064] Next, in step S1030, the device (100) can generate multiple heatmaps that visualize the decisions of the neural network model when each of the multiple patch images is input to the neural network model to be observed. The device (100) can generate heatmaps that visualize the decisions of the neural network model using various visualization explanation techniques. The heatmaps may include pixel information that visualizes information about which area of ​​the input patch image the neural network model mainly referenced to derive a class (e.g., multiple heatmaps of FIG. 3 generated using GradCAM).

[0065] Visualization explanation techniques are technologies that generate a heatmap visualizing information about which regions of an input image a neural network model primarily referenced to derive a class. For example, visualization explanation techniques may include Class Activation Mapping (hereinafter referred to as 'CAP') and Layer-wise Relevance Propagation (hereinafter referred to as 'LRP').

[0066] CAP includes gradient-based class mapping technology (GradCAM) and score-based class mapping technology (ScoreCAM). GradCAM provides visual explanations by linearly combining the features of the intermediate layer of an artificial neural network with the gradients for a specific class, while ScoreCAM can provide visual explanations by directly confounding the input image with the features of the intermediate layer and linearly combining the model's prediction rate.

[0067] LRP is a method that reconstructs the backpropagation method of a neural network model to calculate contributions at the pixel level, and can approximate the decisions of the neural network model using a Taylor series.

[0068] The types of visualization explanation techniques described above are merely examples, and the applicable scope of the embodiments is not limited to the types described above; various visualization explanation techniques other than those described above may be incorporated into the embodiments of the present invention.

[0069] Next, in step S1040, the device (100) can calculate a confidence score for each of the multiple heatmaps based on the logit of the neural network model for the patch image. For example, the confidence score of the heatmap for the patch image can be calculated based on the ratio of the logit value output by the neural network model to the logit value of all classes of the neural network model of the device (100) based on the class to be observed in the patch image. Here, since there is a possibility that among the multiple heatmaps there may be a heatmap that does not fully contain the object of the target image, each confidence score is used when collecting information for each heatmap as described later.

[0070] An example of calculating a confidence score can be implemented more specifically by the following mathematical formula 1. For example, when a patch image is input to a neural network model, the device (100) can calculate a confidence score per patch image by applying the following mathematical formula 1 to the output logit value.

[0071] [Mathematical Formula 1]

[0072]

[0073] ( is the reliability score, is the logit value for the patch image, k is the class to observe in the patch image, Is A function that calculates a confidence score using k as input, where N is the total number of classes, i is the index specifying the class, and j is the index specifying the final node that outputs the logit)

[0074] Next, in step S1050, when the device (100) accumulates and arranges a plurality of heatmaps such that each heatmap falls within a segmented area of ​​the target image, it can generate an explanation map by reflecting the reliability score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each accumulated heatmap at a predetermined location (e.g., by pixel, by area of ​​a preset size) among the areas occupied by the accumulation of the plurality of heatmaps. A specific example of generating an explanation map is described with reference to FIG. 6.

[0075] FIG. 6 is an example diagram to help understand the operation of generating an explanation map according to one embodiment.

[0076] Referring to FIG. 6, the device (100) comprises a plurality of heatmaps, each of which (left C1, C2, C3, ..., C in FIG. 6). n Resize ) to the size of the window, and the resized multiple heatmaps (C1, C2, C3, ..., C on the right side of Fig. 6) n Each can be stacked and placed so as to be located in a segmented area of ​​the target image (bottom right of FIG. 6). At this time, the device (100) can generate sum information by adding the reliability score of each heatmap as a weight to the pixel information of each heatmap accumulated at a first location (e.g., a specific pixel, a specific size area) among the areas occupied by the stacked heatmaps, and can normalize the sum information based on the number of heatmaps accumulated at the first location.

[0077] Here, since there is a possibility that some of the multiple heatmaps may not fully contain the objects in the target image, the previously calculated confidence score is applied as a weight to the pixel information of each heatmap. Additionally, when stacking multiple heatmaps so that each corresponds to a segmented area in the target image, the number of accumulated heatmaps may vary by location; therefore, normalization of the summation information is performed based on the number of accumulated heatmaps at each location.

[0078] Accordingly, in step S1050, the device (100) can generate a final explanation map by performing the operation performed at the first location described above at all locations occupied by a plurality of heat maps arranged cumulatively. At this time, the left side of the explanation map exemplified in FIG. 3 is an explanation map derived from a target image using GradCAM itself, and the right side of the explanation map exemplified in FIG. 3 is an explanation map derived from a target image by applying an embodiment of the present invention to the preceding and succeeding steps of GradCAM. Referring to the example of the explanation map shown in FIG. 3, it can be confirmed that the explanation map through the embodiment of the present invention more completely specifies the area of ​​the object referenced by the neural network model.

[0079] These comparison examples are further examined through the following Figures 7 and 8.

[0080] Figure 7 is a diagram comparing the result of generating an explanation map by applying a conventional visualization method with the result of generating an explanation map by applying an embodiment of the present invention to the conventional visualization method.

[0081] In the comparison of Fig. 7, visualization of a neural network model for a target image (Class: Gold Fish) was performed using GradCAM, FullGrad, RAP, and AGF among the visualization explanation techniques. Original in Fig. 7 is an explanation map derived from the target image using GradCAM, FullGrad, RAP, and AGF itself, and Ours in Fig. 7 is an explanation map derived from the target image by applying an embodiment of the present invention to the preceding and succeeding steps of GradCAM, FullGrad, RAP, and AGF. Referring to Fig. 7, it can be seen that the explanation map of the embodiment of the present invention more completely identifies the region of the object referenced by the neural network model compared to an explanation map using only the existing techniques.

[0082] FIG. 8 is a diagram comparing the result of generating an explanation map by applying a conventional visualization method to the neural network models of ResNet50 and VGG19 with the result of generating an explanation map by applying an embodiment of the present invention to the conventional visualization method.

[0083] In the comparison of Fig. 8, visualization of a neural network model for target images (Class: Red Fox, Cheetah, Magpie, Green Snake) was performed using GradCAM, RSP, and AGF among the visualization explanation techniques. The top right row of the target image in Fig. 8 is an explanation map derived from the target image using GradCAM, RSP, and AGF itself, and the bottom right row of the target image in Fig. 8 is an explanation map derived from the target image by applying an embodiment of the present invention to the preceding and succeeding steps of GradCAM, RSP, and AGF. Referring to Fig. 8, it can be seen that the explanation map of the embodiment of the present invention more completely identifies the area of ​​the object referenced by the neural network model compared to an explanation map using only the existing techniques.

[0084] FIG. 9 is a performance evaluation table comparing the performance of a deletion / insertion game test when conventional visualization methods (GCAM, GCAM++, WGCAM) are applied to neural network models of ResNet50, DenseNet121, and InceptionV3, and the performance of a deletion / insertion game test when an embodiment of the present invention is applied to conventional visualization methods (GCAM*, GCAM++*, WGCAM*).

[0085] Referring to FIG. 9, it can be seen that in all cases where the embodiment of the present invention is applied, the AUC score of the deletion game test is lowered and performance is improved, and the AUC score of the insertion game test is higher and performance is improved.

[0086] FIG. 10 is a performance evaluation table comparing scores for the degree to which each description map specifies an object when various visualization description techniques and embodiments of the present invention are applied.

[0087] The embodiment of the present invention used in the comparison of FIG. 10 is an embodiment in which the division and aggregation process of the present invention is incorporated into the pre- and post-processes of GradCAM. Referring to FIG. 10, it can be seen that the embodiment of the present invention shows a significantly improved performance compared to the comparison method.

[0088] According to the present invention as described above, by dividing a target image to generate a patch image containing partial information of the target image and combining visualization information of a heatmap for the patch image to derive an explanatory method with enhanced granularity and accuracy, it is possible to visualize which region of the target image a neural network model primarily refers to when making a specific decision.

[0089] Accordingly, the present invention enables intuitive verification of which features of a target image significantly influence the decision-making of a neural network model. In particular, in fields with a heavy decision-making burden, such as medicine, it can improve the accuracy and efficiency of the diagnostic process by presenting the basis for diagnosis; and in fields directly related to human casualties, such as autonomous driving, it allows for pre-identification and correction of whether the neural network model's decision-making is derived from incorrect features.

[0090] As such, when the present invention is applied in various fields, users can receive guidance for improving neural network models and datasets, thereby utilizing neural network models more effectively and contributing to the understanding and development of the field of artificial intelligence neural networks.

[0091] Additionally, the method according to the above-described embodiment may also be implemented in the form of a computer program stored on a computer-readable medium or a computer-readable recording medium. A computer program stored on a computer-readable recording medium according to one embodiment may include instructions for the processor to perform a method comprising: acquiring a target image when executed by a processor; generating a plurality of patch images divided to include a portion of the target image; generating a plurality of heatmaps that visualize the decision of a model when the plurality of patch images are input into a predetermined neural network model; calculating a confidence score for each of the plurality of heatmaps; and generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is accumulated and arranged so as to be in the area divided in the target image.

[0092] Additionally, the processor of the embodiments of this document may call at least one instruction among one or more instructions stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to at least one called instruction. Such one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0093] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0094] 10: Processor 20: Network Interface 30: Memory 40: Storage 41: Computer program 50: Data bus 100: Visualization device

Claims

Claim 1 A visualization method performed by a visualization device including a processor and memory, comprising: an operation of acquiring a target image; an operation of generating a plurality of patch images divided to include a portion of the target image; an operation of generating a plurality of heatmaps that visualize the decision of the model when the plurality of patch images are input into a predetermined neural network model; an operation of calculating a confidence score for each of the plurality of heatmaps; and an operation of generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is placed in the area divided in the target image. The method wherein the operation of calculating the confidence score includes an operation of calculating the confidence score of the heatmap for the specific patch image based on a relative attribution ratio, which is the ratio of the logit value output by the neural network model to the class to be observed in the specific patch image with respect to the logit value of all classes of the neural network model. Claim 2 A method according to claim 1, wherein the operation of generating the plurality of patch images comprises: generating a window of a preset size smaller than the target image; moving the window at preset intervals so that the window passes through all areas of the target image; and capturing an area of ​​the target image included in the window at each interval to generate the plurality of patch images. Claim 3 A method according to claim 1, wherein the operation of generating a plurality of patch images includes the operation of storing location information in which each patch image is divided from the target image. Claim 4 A method according to claim 1, wherein the operation of generating the plurality of patch images includes the operation of resizing the plurality of patch images to the size of the target image, and the operation of generating the description map includes the operation of resizing each of the plurality of heatmaps to the size of the partial area and arranging each of the plurality of heatmaps resized to the size of the partial area so as to be located in the divided area of ​​the target image. Claim 5 A method according to claim 1, wherein the operation of generating the plurality of heatmaps includes the operation of generating a heatmap that visualizes information about the decision of the neural network model by applying a Class Activation Mapping technique or a Layer-wise Relevance Propagation technique to the neural network model when a patch image is input to the neural network model. Claim 6 delete Claim 7 In claim 1, the operation of calculating the confidence score calculates the confidence score for each patch image by applying the following mathematical formula 1 to the logit value output when a patch image is input to the neural network model, [Mathematical Formula 1] ( is the reliability score, is the logit value for the patch image, k is the class to observe in the patch image, Is A method for a function that calculates a confidence score using k as input, where N is the total number of classes, i is an index specifying a class, and j is an index specifying the final node that outputs the logit. Claim 8 A method according to claim 1, wherein the operation of generating the explanation map comprises generating sum information by adding the reliability score of each heatmap as a weight to the pixel information of each heatmap accumulated at a first location, and generating the explanation map by normalizing the sum information by the number of heatmaps accumulated at the first location. Claim 9 An apparatus comprising: a memory containing instructions; and a processor executing said instructions, wherein the operation of the processor comprises: an operation of acquiring a target image; an operation of generating a plurality of patch images divided to include a portion of the target image; an operation of generating a plurality of heatmaps visualizing the decisions of said model when the plurality of patch images are input to a predetermined neural network model; an operation of calculating a confidence score for each of said heatmaps; and an operation of generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of said heatmaps is accumulated and arranged to fall within the area divided in said target image; wherein the operation of calculating the confidence score comprises an operation of calculating the confidence score of a heatmap for said specific patch image based on a relative attribution ratio, which is the ratio of the logit value output by said neural network model to the class to be observed in said specific patch image with respect to the logit value of all classes of said neural network model. Claim 10 A computer program stored on a computer-readable recording medium, wherein the computer program, when executed by a processor, comprises instructions for the processor to perform a method including: acquiring a target image; generating a plurality of patch images divided to include a portion of the target image; generating a plurality of heatmaps that visualize the decision of a model when the plurality of patch images are input to a predetermined neural network model; calculating a confidence score for each of the plurality of heatmaps; and generating an explanation map by reflecting the confidence score of each accumulated heatmap and the number of accumulated heatmaps in the pixel information of each heatmap accumulated by position when each of the plurality of heatmaps is accumulated and arranged to fall within the area divided in the target image; wherein the operation of calculating the confidence score includes calculating the confidence score of the heatmap for the specific patch image based on a relative attribution ratio, which is the ratio of the logit value output by the neural network model to the class to be observed in the specific patch image with respect to the logit value of all classes of the neural network model.

Images