Neural network interpretability evaluation method, device, storage medium and equipment
By using semantic segmentation graphs in neural networks to segment pixels and sub-regions, and calculating target ratios, the problem that neural networks are difficult to explain their decision-making process is solved, the accuracy and stability of the model prediction level is improved, and the prediction results are achieved fine-grained explanation.
Patent Information
- Application Number
- CN202210101318.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-27
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-01-27
AI Technical Summary
Existing deep learning neural networks are difficult to explain their decision-making process in image classification tasks, making it difficult to ensure model performance and difficult to correct or innovate.
By obtaining test datasets, including image and semantic segmentation graphs, pixel area and sub-region segmentation, target ratio values are calculated to evaluate the prediction level of the neural network, and to explain the prediction results through local interpretation models.
The accuracy of sub-region segmentation is improved, the accuracy and stability of the neural network prediction level is ensured, and the fine-grained interpretation of the neural network prediction results is achieved.
Smart Images

Figure CN114529720B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium and equipment for evaluating the interpretability of a neural network. Background Art
[0002] With the continuous improvement of deep learning methods, the performance of artificial intelligence systems in more and more complex tasks has reached or even exceeded the level of humans. At present, systems based on deep learning have been widely used in image classification, sentiment analysis, speech understanding and other fields, realizing the process of replacing manual decision-making.
[0003] In the image classification scenario, there are many deep learning-based network models that can classify images more accurately. For example, commonly used standard network models include LeNet, VGGNet, GoogLeNet, ResNet, etc., and commonly used lightweight network models include MobileNet, ShuffleNet, etc. The model structure of these network models generally includes input layer, convolution layer, fully connected layer, maximum pooling layer, residual structure, output layer, etc., including the superposition of a series of nonlinear functions. When these network models are used to classify images, the network model can return the image label after analyzing the input image. The label generally comes from a predefined category set.
[0004] However, although deep learning can achieve image classification and prediction of image labels, it is difficult for users to know the actual analysis process of the network model on the input image, and cannot understand the basis on which the network model makes decisions. As a result, it is difficult to guarantee the performance of the network model, and it is impossible to ensure whether the network model can make accurate judgments on the input image next time. It is also difficult to modify and innovate the network model. Summary of the invention
[0005] In view of the fact that related technologies cannot provide an explanation of image prediction by a neural network, the present invention provides a method, device, storage medium and equipment for evaluating the interpretability of a neural network.
[0006] The technical solution of the present invention includes:
[0007] A method for evaluating the interpretability of a neural network, the method comprising:
[0008] Obtain a test data set, where the test data set includes multiple images, a label for each image, and a semantic segmentation map corresponding to each image;
[0009] For the image x in the test dataset i , based on the image x i The corresponding semantic segmentation map for the image x iSegmentation is performed to obtain the image x i A set of pixel regions;
[0010] For the image x i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x i The sub-region segmentation result;
[0011] Based on the image x i The sub-region segmentation result determines the image x i The target ratio value, the image x i The target ratio value is used to indicate the image x i The sub-region of interest is related to the image x i The degree of overlap between the main areas of the image x i The sub-region of interest is the neural network under test for the image x i The sub-region of interest, the image x i The main area refers to the image x i The sub-region corresponding to the label;
[0012] An evaluation result of the neural network under test is determined by using target ratio values of all images in the test data set, and the evaluation result is used to indicate the prediction level of the neural network under test.
[0013] Optionally, the image x i The sub-region segmentation result determines the image x i The target ratios include:
[0014] Based on the image x i The sub-region segmentation result constructs the image x i The local explanation model of the image x i The local explanation model is used to indicate the image x i Each sub-region in the sub-region segmentation result has a certain effect on the image x i The contribution of the prediction results;
[0015] By the image x i The local explanation model of the image x i Sub-areas of interest;
[0016] Based on the image x i The sub-region of interest and the image x i The main area of the target is calculated;
[0017] When the target ratio value converges, the target ratio value is used as the image x i The target ratio value.
[0018] Optionally, the image x i The sub-region segmentation result constructs the image x i The local explanation model includes:
[0019] Take the image x i In the sub-region segmentation result, each sub-region is in the image x i Whether the local explanation model appears in the image x i The vector representation of ;
[0020] Based on the image x i The vector represents the image x i Processed, and the processed image x′ i As input, take the image x i The label is output and the classifier is trained;
[0021] Based on the classifier, the image x i The prediction results and the image x i , construct the image x i Local explanation model.
[0022] Optionally, the image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The sub-region that contributes the most to the prediction result; or, the image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The contribution of the prediction results to the sub-regions meets the set threshold.
[0023] Optionally, the image x i The sub-region of interest and the image x i The main area of the calculation target ratio includes:
[0024] By the image x i The sub-region of interest is related to the image x i The target ratio is calculated by the intersection-and-union ratio between the main regions of
[0025] or,
[0026] By the image x i The sub-region of interest is located in the image x iThe part of the main area of the image x i The target ratio is calculated by the inside-outside ratio of the parts outside the main body area.
[0027] Optionally, the image x i The sub-region of interest and the image x i The main area of the target ratio is calculated, and it also includes:
[0028] When the target ratio value does not converge, adjusting the segmentation parameters used in performing the sub-region segmentation; wherein the adjusted segmentation parameters make the sub-region area obtained after performing the sub-region segmentation smaller;
[0029] The sub-region segmentation is performed using the adjusted segmentation parameters, from the image x i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x i The sub-region segmentation results are started again.
[0030] Optionally, determining the evaluation result of the tested neural network by using target ratio values of all images in the test data set includes:
[0031] Performing image classification on all images in the test data set to obtain at least one category of images;
[0032] Performing statistical processing on the target ratio values corresponding to the at least one type of image respectively to obtain statistical results of the at least one type of image;
[0033] Based on the statistical results of the at least one type of image, the prediction level of the tested neural network for the at least one type of image is compared to obtain an evaluation result of the tested neural network.
[0034] A neural network interpretability evaluation device, comprising:
[0035] An acquisition module is used to acquire a test data set, wherein the test data set includes multiple images, a label of each image, and a semantic segmentation map corresponding to each image;
[0036] The first segmentation module is used for the image x in the test data set i , based on the image x i The corresponding semantic segmentation map for the image x i Segmentation is performed to obtain the image x i A set of pixel regions;
[0037] The second segmentation module is used to segment the image x iEach pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x i The sub-region segmentation result;
[0038] A calculation module for calculating a i The sub-region segmentation result determines the image x i The target ratio value, the image x i The target ratio value is used to indicate the image x i The sub-region of interest is related to the image x i The degree of overlap between the main areas of the image x i The sub-region of interest is the neural network under test for the image x i The sub-region of interest, the image x i The main area refers to the image x i The sub-region corresponding to the label;
[0039] An evaluation module is used to determine an evaluation result of the tested neural network through target ratio values of all images in the test data set, and the evaluation result is used to indicate the prediction level of the tested neural network.
[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the above-mentioned neural network interpretability evaluation method.
[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the interpretability evaluation method of the neural network as described above.
[0042] A computer program product, when the computer program product is run on a computer device, causes the computer device to execute the above-mentioned neural network interpretability evaluation method.
[0043] Compared with the prior art, the present invention has at least the following advantages:
[0044] 1. By introducing the semantic segmentation map for sub-region division, different objects or elements indicated by the semantic segmentation map are avoided from being segmented into the same sub-region, thereby improving the accuracy of sub-region segmentation;
[0045] 2. Through the iterative process of the target ratio, the influence of some factors (such as too rough sub-region segmentation, random failure of equipment, etc.) on the target ratio value is eliminated, and more accurate and stable calculation results can be obtained, ensuring accurate evaluation of the prediction level of the neural network;
[0046] 3. It realizes the category-by-category evaluation of the prediction level of the neural network for each type of image, and provides a fine-grained explanation of the prediction results given by the neural network from a category perspective, which more fully explains the prediction effect of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 Flowchart of the method for evaluating the interpretability of neural networks.
[0048] Figure 2 Block diagram of the setup for evaluating the interpretability of neural networks. DETAILED DESCRIPTION
[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only specific embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] The present invention proposes a neural network interpretability evaluation method, such as Figure 1 As shown, the following steps are included (steps 110 to 150).
[0051] Step 110: Obtain a test data set.
[0052] The test data set of the present invention includes multiple images, a label for each image, and a semantic segmentation map corresponding to each image.
[0053] In the application scenario of image classification, each image has only one label. For example, if there is a cat in the image, the label of the image is cat.
[0054] The semantic segmentation map corresponding to the image can be obtained in the following two ways: (1) manual annotation; (2) a combination of automatic assisted annotation (such as with the help of tools such as VIA-VGG Image Annotator, OpenCV / CVAT, RectLabel, etc.) and manual correction.
[0055] Step 120: For the image x in the test dataset i , based on image x i The corresponding semantic segmentation map for image x i Segmentation is performed to obtain image x i A collection of pixel regions.
[0056] The pixel region set obtained by segmentation in this step includes n (n is a positive integer) pixel regions. These n pixel regions are consistent with the image x. i (i is a positive integer) has a corresponding relationship between the objects and / or elements in the image x.i The corresponding semantic segmentation map indicates the image x i Different objects and / or elements in the image x i The corresponding semantic segmentation map for image x i By segmenting, we can obtain the pixel areas corresponding to the objects and / or elements indicated by the semantic segmentation map. The set of these pixel areas is the image x i A collection of pixel regions.
[0057] In one example, in order to simplify the segmentation process and reduce the amount of processing calculation, the image x i When segmenting, you can also use image x i The corresponding semantic segmentation map converts the image x i Divided into two pixel regions, respectively: image x i The pixel area corresponding to the label (main area), image x i The pixel area corresponding to the background, the set of these two pixel areas is the image x i A set of pixel regions. i The background is image x i Except for image x i Objects and elements other than tags.
[0058] For example, image x i There are people, dogs, trees, ground, sky, image x i The label is human, then the image x i The background is dog, tree, ground and sky; in image x i In the corresponding semantic segmentation map, different colors are used to indicate the image x i objects and / or elements in the image (i.e., person, dog, tree, ground, sky), based on the image x i The corresponding semantic segmentation map for image x i After segmentation, we can get image x i A set of pixel regions may include two pixel regions, namely: a subject region (ie, a pixel region corresponding to a person) and a pixel region corresponding to a background (ie, a dog, a tree, the ground, and the sky).
[0059] In another example, in order to obtain a better segmentation result, not only the image x i The background can be cut, and the image x i The pixel area corresponding to the label is cut.
[0060] For example, image x i The label is dog, image x iThe background of the video is a tree, the ground and the sky. Not only can the background be cut into tree crown, tree trunk, sky, ground 1 and ground 2, but the dog can also be cut into head, body, tail and legs to obtain multiple main areas.
[0061] Step 130: For image x i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation results are applied to the image x i On, we get the image x i The sub-region segmentation result.
[0062] In the image x i When each pixel region in the pixel region set is segmented into sub-regions, an initial segmentation parameter is set, and the segmentation parameter is used to perform random sub-region segmentation. The present invention does not limit the shape of the segmented sub-regions. Optionally, the shape of the segmented sub-regions can be uniform, such as uniformly square, uniformly triangle, etc.; or, the shape of the segmented sub-regions is random, such as randomly square, triangle, parallelogram, pentagon, etc.; or, the shape of the segmented sub-regions in a pixel region is uniform, and so on.
[0063] In the present invention, the segmentation parameters include but are not limited to: the area of the segmented sub-region (which may be the area of each segmented sub-region or the maximum area of the segmented sub-region), the number of segmented sub-regions, etc. In the application scenario of image classification, the segmentation parameters used for each pixel region in the pixel region set may be the same, but the present invention does not exclude the implementation of using different segmentation parameters for each pixel region in practical applications. For example, when the segmentation parameter is the number of segmented sub-regions, a larger segmentation parameter is used for pixel regions with a larger area, and a smaller segmentation parameter is used for pixel regions with a smaller area; or, a larger segmentation parameter is used for pixel regions with complex texture information, and a smaller segmentation parameter is used for pixel regions with simple texture information, and so on. It should be understood that these should all fall within the scope of protection of the present invention.
[0064] In addition, when performing sub-region segmentation, each pixel region in the pixel region set can be randomly segmented into sub-regions in turn, that is, the sub-region segmentation of each pixel region is executed serially, such as performing sub-region segmentation in order from large to small according to the area of the pixel region, or first performing sub-region segmentation on the main region, and then performing sub-region segmentation in order from near to far according to the distance between the main region, etc.; or, each pixel region in the pixel region set can be randomly segmented into sub-regions at the same time, that is, the sub-region segmentation of each pixel region is executed in parallel.
[0065] Step 140: Based on image x i The sub-region segmentation result of image x is determinedi The target ratio value.
[0066] Imagex i The target ratio value is used to indicate the image x i The sub-region of interest and the image x i The degree of overlap between the main areas of the image x i The sub-region of interest is the neural network under test for the image x i The sub-region of interest, image x i The main area of the image x i The sub-region corresponding to the label.
[0067] In one example, based on the image x i The sub-region segmentation result of image x is determined i The target ratio value of the ratio comprises the following sub-steps (steps 142 to 148):
[0068] Step 142: Based on image x i The sub-region segmentation result is used to construct the image x i Local explanation model.
[0069] Imagex i The local explanation model is used to indicate the image x i The sub-region segmentation results of each sub-region are for image x i Optionally, the construction process of the local explanation model includes the following sub-steps (steps 1422 to 1426):
[0070] Step 1422: Take image x i In the sub-region segmentation result, each sub-region is in the image x i Whether it appears in the local explanation model of image x i The vector representation of .
[0071] When constructing the local interpretation model of the present invention, it is necessary to randomly select some sub-regions from the sub-region segmentation results. i The sub-region segmentation result of the image x appears in the local explanation model to determine whether i The vector representation of .
[0072] Among them, the image x i The vector representation of can be a 0-1 vector representation, where the subregion appears in the local interpretation model and the subregion does not appear in the local interpretation model. For example, the image x i The sub-area division results include p1, p2, ...p n Total n i sub-regions, then the image x iThe vector representation of can be 0-1 vector representation Vector representation r i Include i elements, each of which indicates whether the corresponding sub-region appears in the local explanation model.
[0073] Step 1424: Based on image x i The vector representation of the image x i After processing, the processed image x′ i As input, take image x i The label is output and the classifier is trained.
[0074] Based on image x i The vector representation of r i , we can use the image x i For processing, such as for image x i The sub-region segmentation result is to retain the sub-regions corresponding to the elements with a value of 1, and replace the sub-regions corresponding to the elements with a value of 0 with gray; then, the processed image x′ i As input, take image x i The label (the known quantity in the test data set in step 110) is output, and the classifier f is trained to obtain the trained classifier f.
[0075] Step 1426: Classify image x based on the classifier i The prediction results and image x i , construct the image x i Local explanation model.
[0076] Assume that the trained classifier f is trained on image x i The prediction result is y i =f(x i ), then the local explanation model can be expressed as:
[0077]
[0078] Among them, x i,j For image x i The jth sub-region in (j is a positive integer less than or equal to n), w i,j For x i,j The weight of w i,j Used to indicate x i,j For image x i The effect size of the final prediction result, w i,j The larger the absolute value of is, the more obvious its effect on the final prediction result is. A positive value indicates a positive effect on the final prediction result, while a negative value indicates a negative effect on the final prediction result.
[0079] Step 144: Pass Image x i The local explanation model of image x i sub-area of interest.
[0080] Imagex i The sub-region of interest is the neural network under test for the image x i The sub-region of interest. It should be noted that the sub-region of interest can be one sub-region or multiple sub-regions, which is not limited in the present invention. i,j Can indicate image x i The effect of the j-th sub-region in the final prediction result (here, the trained classifier f has the same properties as the tested neural network by default), so the computer device uses the image x i The local explanation model of image x i sub-area of interest.
[0081] Optionally, the image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The sub-region with the greatest contribution to the prediction result (it can be a sub-region with the greatest contribution, or multiple sub-regions with the greatest contribution), that is, the sub-region corresponding to the maximum weight indicated by the local explanation model (it can be the largest weight, or multiple largest weights, and the maximum weight can be the largest value or the largest absolute value); or, image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The sub-region whose contribution to the prediction result meets the set threshold, that is, the sub-region corresponding to the weight reaching the threshold indicated by the local explanation model.
[0082] Among them, w in the local explanation model i,j The sub-regions of interest can be displayed visually, for example, through analysis diagrams, approval / disapproval diagrams, weighted heat maps, etc. Among them, the analysis diagram selects several sub-regions that have the greatest impact on the final prediction result according to the absolute value of the weight, displays the relevant content in the original diagram and hides other regions; in the approval / disapproval diagram, green blocks are used to represent positively correlated sub-regions, and red blocks are used to represent negatively correlated sub-regions; in the weighted heat map, blue blocks are used to represent positively correlated sub-regions, and red blocks are used to represent negatively correlated sub-regions. The darker the color, the stronger the correlation.
[0083] Step 146: Based on image x i The sub-region of interest and image x i The main area of the target is calculated.
[0084] As described in step 120, the subject area of the present invention refers to the image x i The pixel region corresponding to the label can be the image x i The pixel area corresponding to the label can also be the entire area containing the image x i At least one sub-region is obtained after the pixel region corresponding to the label is segmented into sub-regions.
[0085] The target ratio may indicate the degree of overlap between the focus sub-region and the main region. The present invention does not limit the implementation method of the target ratio.
[0086] In one example, by using image x i The sub-region of interest and the image x i The target ratio is calculated by the intersection-and-union ratio between the main regions of . For example, if the focus sub-region is C and the main region is G, the target ratio can be expressed as:
[0087]
[0088] In another example, by image x i The sub-region of interest is located in the image x i The part in the main area of the image x i The target ratio is calculated by the inside-outside ratio of the part outside the main area of the focus sub-area. For example, the sub-area in the main area of the focus sub-area is U, and the sub-area outside the main area of the focus sub-area is V, then the target ratio can be expressed as: U / V.
[0089] It should be noted that, usually when segmenting an image, there is no limitation on the area and shape of the segmented sub-region. Therefore, in order to more accurately describe the overlap between the sub-region of interest and the main region, the target ratio is in the form of an area ratio, that is, the above C, G, U, V, etc. are all expressed as areas. Of course, in the case where the area of the sub-region after segmentation is a uniform value, the target ratio can also be implemented in the form of a number ratio. For example, the target ratio is implemented as: the ratio of the number of sub-regions in the sub-region of interest that are located in the main region to the number of sub-regions in the sub-region of interest that are located outside the main region.
[0090] Based on the above, we can know that the sub-region of interest includes the neural network under test for the image x i When making predictions, the sub-regions that contribute to the final prediction results meet the target conditions. Therefore, based on the focus sub-regions, it can be clear how the tested neural network obtains the prediction results, that is, the basis on which the tested neural network obtains the prediction results; while the main region reflects the image x iThe origin of the label, in the application scenario of image classification, the degree of overlap between the focus sub-region and the main region (target ratio) can reflect whether the basis for the prediction result of the tested neural network is correct.
[0091] Step 148: When the target ratio value converges, the target ratio value is used as the image x i The target ratio value.
[0092] In order to obtain more accurate and stable calculation results and ensure accurate evaluation of the prediction level of the neural network, the computer device needs to determine whether the target ratio value converges. The target ratio value convergence means that the target ratio value tends to be stable. When the target ratio value converges, the segmentation parameters for sub-region segmentation in step 130 are optimal segmentation parameters, and the target ratio value at this time can be recorded as the image x i The target ratio value.
[0093] However, the target ratio value obtained based on step 146 does not necessarily converge. When the target ratio value does not converge, it is necessary to adjust the segmentation parameters for sub-region segmentation in step 130, wherein the adjusted segmentation parameters make the sub-region area obtained after the sub-region segmentation smaller. Afterwards, the sub-region segmentation is performed with the adjusted segmentation parameters, and the process is executed again from step 130. The adjustment method of the segmentation parameters is different based on different expressions of the segmentation parameters. For example, if the segmentation parameter is the area of the sub-region after segmentation, the segmentation parameter needs to be gradually reduced; if the segmentation parameter is the number of sub-regions after segmentation, the segmentation parameter needs to be gradually increased. The increase and decrease described here include but are not limited to: increasing or decreasing in proportion, increasing or decreasing a certain value on the original value, and the like.
[0094] It should be understood that the image x i is any image in the test data set. The computer device traverses all images in the test data set, and executes steps 120 to 140 for each image, so as to obtain the target ratio value of each image in the test data set.
[0095] Step 150: Determine the evaluation result of the neural network under test by testing the target ratio values of all images in the data set.
[0096] The evaluation results of the present invention are used to indicate the prediction level of the neural network under test. The computer equipment can average and perform statistical processing on the target ratio values of all images in the test data set to determine the evaluation results of the neural network under test. For example, the expectation of the target ratio values of all images is calculated. The larger the expectation, the better the prediction level of the neural network under test.
[0097] In one example, in order to evaluate the prediction level of the neural network for each type of image on a category-by-category basis, the above step 150 includes the following sub-steps (steps 152 to 156 ).
[0098] Step 152: Perform image classification on all images in the test data set to obtain at least one category of images.
[0099] In one example, images in the test data set are classified according to their labels, and images with the same label are classified into the same category. For example, if there are images with labels of people, cats, and dogs in the test data set, images with the label of people are classified into one category, images with the label of cats are classified into one category, and images with the label of dogs are classified into one category.
[0100] Step 154: Perform statistical processing on the target ratio values corresponding to at least one type of image to obtain statistical results of at least one type of image.
[0101] In one example, the statistical processing includes finding the expectation and / or the variance. Based on this, the present invention obtains the statistical results of each type of image by calculating the expectation and / or the variance of the target ratio value corresponding to each type of image. For example, if there are 10 images labeled as people in the test data set, the expectation and variance of the target ratio value are calculated for the images of the type of people using the target ratio values corresponding to these 10 images.
[0102] Step 156: Based on the statistical results of at least one type of image, compare the prediction level of the tested neural network for at least one type of image to obtain an evaluation result of the tested neural network.
[0103] In one example, the statistical result is obtained based on the expectation and variance of the target ratio values corresponding to each type of image. The larger the expectation, the larger the target ratio value corresponding to this type of image; the smaller the variance, the more convergent the target ratio value corresponding to this type of image. That is, the larger the expectation and the smaller the variance: the target ratio value converges when it is larger, and the degree of convergence is higher; the smaller the expectation and the smaller the variance: the target ratio value converges when it is smaller, and the degree of convergence is higher; the smaller the expectation and the larger the variance: the target ratio value converges when it is smaller, and the degree of convergence is lower; the larger the expectation and the larger the variance: the target ratio value converges when it is larger, and the degree of convergence is lower. Based on this, the larger the expectation and the smaller the variance of the target ratio value corresponding to a certain type of image, the better the prediction level of the tested neural network for this type of image.
[0104] Based on the comparison of the above prediction levels, the computer device can generate and output the evaluation results of the tested neural network. The evaluation results include but are not limited to: the image category with the best prediction level of the tested neural network, the expectation and variance of the target ratio value corresponding to each category of images, and the prediction accuracy of the tested neural network for each category of images.
[0105] To summarize, the present invention introduces a semantic segmentation map to first divide the entire image into a set of pixel regions, and then performs sub-region segmentation on the pixel region sets respectively, thereby avoiding the situation where different objects or elements are divided into the same sub-region when sub-region segmentation is performed directly based on the entire image, thereby improving the accuracy of sub-region segmentation.
[0106] In addition, after the sub-region is segmented and the focus sub-region is determined, the present invention calculates the target ratio based on the focus sub-region and the main region. If the target ratio value does not converge, the segmentation parameters are adjusted and the execution starts again from the sub-region segmentation until the target ratio value converges. Through the iterative process of the target ratio, some factors (such as overly rough sub-region segmentation, random failure of equipment, etc.) that affect the target ratio value are eliminated. Compared with only performing a sub-region segmentation once and performing subsequent calculations based on the sub-region segmentation results, the present invention can obtain more accurate and more stable calculation results, ensuring accurate evaluation of the prediction level of the neural network.
[0107] In addition, the present invention classifies the images in the test data set and performs statistical processing on the target ratio values of each type of image, thereby evaluating the prediction level of the neural network for each type of image on a category by category basis, and provides a fine-grained interpretation of the prediction results given by the neural network from a category perspective, which more fully explains the prediction effect of the neural network.
[0108] refer to Figure 2 , which shows a block diagram of a neural network interpretability evaluation device provided by an embodiment of the present invention. The device can be a computer device, or can be set in a computer device. Figure 2 As shown, the device 200 includes: an acquisition module 210 , a first segmentation module 220 , a second segmentation module 230 , a calculation module 240 and an evaluation module 250 .
[0109] An acquisition module 210 is used to acquire a test data set, where the test data set includes multiple images, a label for each image, and a semantic segmentation map corresponding to each image;
[0110] The first segmentation module 220 is used to segment the image x in the test data set. i , based on the image x i The corresponding semantic segmentation map for the image x i Segmentation is performed to obtain the image x i A set of pixel regions;
[0111] The second segmentation module 230 is used to segment the image x i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x iThe sub-region segmentation result;
[0112] The calculation module 240 is used to calculate the image x based on the image x. i The sub-region segmentation result determines the image x i The target ratio value, the image x i The target ratio value is used to indicate the image x i The sub-region of interest is related to the image x i The degree of overlap between the main areas of the image x i The sub-region of interest is the neural network under test for the image x i The sub-region of interest, the image x i The main area refers to the image x i The sub-region corresponding to the label;
[0113] The evaluation module 250 is used to determine the evaluation result of the neural network under test through the target ratio values of all images in the test data set, and the evaluation result is used to indicate the prediction level of the neural network under test.
[0114] Optionally, the calculation module 240 includes: a model building unit, a focus sub-region acquisition unit, a target ratio calculation unit and a target ratio value confirmation unit, wherein the model building unit is used to calculate the target ratio value based on the image x i The sub-region segmentation result constructs the image x i The local explanation model of the image x i The local explanation model is used to indicate the image x i Each sub-region in the sub-region segmentation result has a certain effect on the image x i The contribution degree of the prediction result; focus on the sub-region acquisition unit, which is used to obtain the image x i The local explanation model obtains the image x i A target ratio calculation unit for calculating a target ratio based on the image x i The sub-region of interest and the image x i the main area of the image x, and calculating the target ratio; a target ratio value confirmation unit, for, when the target ratio value converges, using the target ratio value as the image x i The target ratio value.
[0115] Optionally, the model building unit is used to: i In the sub-region segmentation result, each sub-region is in the image x i Whether the local explanation model appears in the image x i Vector representation of the image x i The vector represents the image xi Processed, and the processed image x′ i As input, take the image x i The label is output, and the classifier is trained; based on the classifier, the image x i The prediction results and the image x i , construct the image x i Local explanation model.
[0116] Optionally, the image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The sub-region that contributes the most to the prediction result; or, the image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The contribution of the prediction results to the sub-regions meets the set threshold.
[0117] Optionally, the target ratio calculation unit is used to: i The sub-region of interest is related to the image x i The target ratio is calculated by the intersection-and-union ratio between the main regions of the image x i The sub-region of interest is located in the image x i The part of the main area of the image x i The target ratio is calculated by the inside-outside ratio of the parts outside the main body area.
[0118] Optionally, the device is further used to: when the target ratio value does not converge, adjust the segmentation parameters used when performing the sub-region segmentation; wherein the adjusted segmentation parameters make the sub-region area obtained after the sub-region segmentation smaller; perform the sub-region segmentation with the adjusted segmentation parameters, and i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x i The sub-region segmentation results are started again.
[0119] Optionally, the evaluation module 250 is used to: perform image classification on all images in the test data set to obtain at least one category of images; perform statistical processing on the target ratio values corresponding to the at least one category of images to obtain statistical results of the at least one category of images; based on the statistical results of the at least one category of images, compare the prediction level of the tested neural network for the at least one category of images to obtain an evaluation result of the tested neural network.
[0120] For the detailed description of the implementation process, beneficial effects, etc. of the device module, please refer to the introduction of the above method embodiment, which will not be elaborated here.
[0121] In an exemplary embodiment, a computer device is also provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned neural network interpretability evaluation method.
[0122] In an exemplary embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the method for evaluating the interpretability of a neural network as described above is implemented.
[0123] In an exemplary embodiment, a computer program product is also provided. When the computer program product is run on a computer device, the computer device executes the interpretability evaluation method of the neural network as described above.
[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for evaluating the interpretability of a neural network, characterized in that: The method comprises: Obtain a test data set, where the test data set includes multiple images, a label for each image, and a semantic segmentation map corresponding to each image; For the image x in the test dataset i , based on the image x i The corresponding semantic segmentation map for the image x i Segmentation is performed to obtain the image x i A set of pixel regions; For the image x i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x i The sub-region segmentation result; Based on the image x i The sub-region segmentation result determines the image x i The target ratio value, the image x i The target ratio value is used to indicate the image x i The sub-region of interest is related to the image x i The degree of overlap between the main areas of the image x i The sub-region of interest is the neural network under test for the image x i The sub-region of interest, the image x i The main area refers to the image x i The sub-region corresponding to the label; wherein the sub-region based on the image x i The sub-region segmentation result determines the image x i The target ratios include: Based on the image x i The sub-region segmentation result constructs the image x i The local explanation model of the image x i The local explanation model is used to indicate the image x i Each sub-region in the sub-region segmentation result has a certain effect on the image x i The contribution of the prediction result based on the image x i The sub-region segmentation result constructs the image x i The local explanation model includes: Take the image x i In the sub-region segmentation result, each sub-region is in the image x i Whether the local explanation model appears in the image x i The vector representation of ; Based on the image x i The vector represents the image x i Process it and use the processed image x i ′ As input, the image x i The label is output and the classifier is trained; Based on the classifier, the image x i The prediction result and the image x i , construct the image x i Local interpretation model of By the image x i The local explanation model of the image x i Sub-areas of interest; Based on the image x i The sub-region of interest and the image x i The main area of the target is calculated; When the target ratio value converges, the target ratio value is used as the image x i ; determining an evaluation result of the tested neural network through the target ratio values of all images in the test data set, wherein the evaluation result is used to indicate the prediction level of the tested neural network.
2. The method according to claim 1, characterized in that The image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The sub-region that contributes most to the prediction result; or, The image x i The sub-region of interest is the image x i The local explanation model of the image x indicates i The contribution of the prediction results to the sub-regions meets the set threshold.
3. The method according to claim 1, characterized in that The image x i The sub-region of interest and the image x i The main area of the calculation target ratio includes: By the image x i The sub-region of interest is related to the image x i The target ratio is calculated by the intersection-and-union ratio between the main regions of By the image x i The sub-region of interest is located in the image x i The part of the main area of the image x i The target ratio is calculated by the inside-outside ratio of the parts outside the main body area.
4. The method according to claim 1, characterized in that The image x i The sub-region of interest and the image x i The main area of the target ratio is calculated, and it also includes: When the target ratio value does not converge, adjusting the segmentation parameters used in performing the sub-region segmentation; wherein the adjusted segmentation parameters make the sub-region area obtained after performing the sub-region segmentation smaller; The sub-region segmentation is performed using the adjusted segmentation parameters, from the image x i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x i The sub-region segmentation results are started again.
5. The method according to any one of claims 1 to 4, characterized in that: Determining the evaluation result of the tested neural network by using the target ratio values of all images in the test data set includes: Performing image classification on all images in the test data set to obtain at least one category of images; Performing statistical processing on the target ratio values corresponding to the at least one type of image respectively to obtain statistical results of the at least one type of image; Based on the statistical results of the at least one type of image, the prediction level of the tested neural network for the at least one type of image is compared to obtain an evaluation result of the tested neural network.
6. A neural network interpretability evaluation device, characterized in that: include: An acquisition module is used to acquire a test data set, wherein the test data set includes multiple images, a label of each image, and a semantic segmentation map corresponding to each image; The first segmentation module is used for the image x in the test data set i , based on the image x i The corresponding semantic segmentation map for the image x i Segmentation is performed to obtain the image x i A set of pixel regions; The second segmentation module is used to segment the image x i Each pixel region in the pixel region set is segmented into sub-regions, and the segmentation result is applied to the image x i On the image x i The sub-region segmentation result; A calculation module for calculating the image x based on the image i The sub-region segmentation result determines the image x i The target ratio value, the image x i The target ratio value is used to indicate the image x i The sub-region of interest is related to the image x i The degree of overlap between the main areas of the image x i The sub-region of interest is the neural network under test for the image x i The sub-region of interest, the image x i The main area refers to the image x i The sub-region corresponding to the label; wherein the sub-region based on the image x i The sub-region segmentation result determines the image x i The target ratios include: Based on the image x i The sub-region segmentation result constructs the image x i The local explanation model of the image x i The local explanation model is used to indicate the image x i Each sub-region in the sub-region segmentation result has a certain effect on the image x i The contribution of the prediction result based on the image x i The sub-region segmentation result constructs the image x i The local explanation model includes: Take the image x i In the sub-region segmentation result, each sub-region is in the image x i Whether the local explanation model appears in the image x i The vector representation of ; Based on the image x i The vector represents the image x i Process it and use the processed image x i ′ As input, the image x i The label is output and the classifier is trained; Based on the classifier, the image x i The prediction result and the image x i , construct the image x i Local interpretation model of By the image x i The local explanation model of the image x i Sub-areas of interest; Based on the image x i The sub-region of interest and the image x i The main area of the target is calculated; When the target ratio value converges, the target ratio value is used as the image x i An evaluation module is used to determine an evaluation result of the tested neural network through the target ratio values of all images in the test data set, and the evaluation result is used to indicate the prediction level of the tested neural network.
7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method according to any one of claims 1 to 5.
8. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 5 when executed.
