Method and apparatus for visualizing interpretation and robustness verification

Abstract

The application relates to a method and device for visualizing interpretation and robustness verification, the method comprising a processing step, an inference step, an analysis step and an evaluation step. The processing step comprises performing mask processing on a plurality of regions of an original image to obtain a plurality of image samples. The inference step comprises performing inference on the plurality of image samples to obtain a score result, the score result comprising a quantitative score or a confidence probability. The analysis step comprises analyzing the influence degree of the plurality of regions on the score result to obtain a significant difference. The evaluation step comprises evaluating a key feature region and confirming the robustness of an inference process according to at least the significant difference. The scheme provided by the application realizes visualized interpretation and robustness verification of inference results of a machine vision system in the form of a black box / white box model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision, in particular to visualized explanation and robustness verification for inference process of machine vision system. BACKGROUND

[0002] At present, many commonly used machine vision systems in industrial scenarios are black box models, that is, their internal principles are completely encapsulated and unknown. Typical examples include deep learning image classification models, template matching algorithms owned by manufacturers and encapsulated, etc. The inference result scores of these black box models are inconsistent in probability and have poor explainability. Users cannot understand what image features the inference process depends on, nor can they accurately estimate the degree of mastery of reliable features of the measured target by the model or algorithm judgment process according to the size of the score. In addition, if the image features relied on by the inference process are too small, do not match the actual experience, are easily disturbed by the environment, etc., the model may not be robust enough, and a small amount of reasonable input interference may cause a large abnormal result change. Existing machine learning explainability solutions often need to disassemble the black box model to understand the internal structure design, workflow, parameter details, etc., but it is difficult to meet the requirements in industrial scenarios, and the workload for complex black box models is extremely large. SUMMARY

[0003] The present application provides a method and device for visualized explanation and robustness verification of judgment process of machine vision system, which can be applied to both machine vision systems in the form of black box model and machine vision systems in the form of white box model. The method and device for visualized explanation and robustness verification of the present application can highlight and display the saliency regions concerned by deep learning neural network models or template matching algorithms in various industrial machine vision selection classification and mistake-proofing applications, and can provide quantitative indicators for calculating visualized region data characteristics to verify the inference robustness of the model or algorithm.

[0004] According to a first aspect of the present application, a method for visualized explanation and robustness verification is provided, which comprises a processing step, an inference step, an analysis step and an evaluation step. The processing step comprises performing mask processing on a plurality of regions of an original image to obtain a plurality of image samples. The inference step comprises inferring the plurality of image samples to obtain a score result, the score result comprising a quantitative score or a confidence probability. The analysis step comprises analyzing the influence degree of the plurality of regions on the score result to obtain a saliency difference. The evaluation step comprises evaluating a key feature region and confirming the robustness of the inference process according to at least the saliency difference.

[0005] According to one or more embodiments of the first aspect of the present application, optionally, the analyzing step comprises fusing the score results using algorithmic operations, calculating feature saliency regions and saliency values, and visualizing the feature saliency regions and the saliency values as a saliency difference map, the algorithmic operations including data dimension correlation analysis and matrix element weighting operations.

[0006] According to one or more embodiments of the first aspect of the present application, optionally, the evaluating step comprises performing data distribution analysis on the saliency values, binarizing and segmenting the saliency difference map, and weighting and superimposing the saliency difference map to obtain a dominant saliency map that can reflect the size and boundary of the most critical feature region.

[0007] According to one or more embodiments of the first aspect of the present application, optionally, the evaluating step comprises calculating the proportion of image pixels occupied by the most critical feature region according to the proportion of binarized segmentation, and determining that the inference process is robust and reliable if the proportion is appropriate, or determining that the inference process is unreasonable if the proportion is inappropriate.

[0008] According to the second aspect of the present application, a device for visualizing explanation and robustness verification is provided, the device comprising a processing unit, an inference unit, an analyzing unit, and an evaluating unit. The processing unit is configured to mask a plurality of regions of an original image to obtain a plurality of image samples. The inference unit is configured to perform inference on the plurality of image samples to obtain score results, the score results including quantitative scores or confidence probabilities. The analyzing unit is configured to analyze the influence of the plurality of regions on the score results to obtain saliency differences. The evaluating unit is configured to evaluate a critical feature region and confirm the robustness of an inference process according to at least the saliency differences.

[0009] According to one or more embodiments of the second aspect of the present application, optionally, the analyzing step unit is further configured to fuse the score results using algorithmic operations, calculate feature saliency regions and saliency values, and visualize the feature saliency regions and the saliency values as a saliency difference map, the algorithmic operations including data dimension correlation analysis and matrix element weighting operations.

[0010] According to one or more embodiments of the second aspect of the present application, optionally, the evaluating unit is further configured to perform data distribution analysis on the saliency values, binarize and segment the saliency difference map, and weight and superimpose the saliency difference map to obtain a dominant saliency map that can reflect the size and boundary of the most critical feature region.

[0011] According to one or more embodiments of the second aspect of the application, optionally, the evaluation unit is further configured to: calculate a proportion of pixels of the image occupied by the most critical feature region according to the proportion of the binaryzation segmentation, and determine that the inference process is robust and reliable if the proportion is appropriate in size, and determine that the inference process is not reasonable if the proportion is inappropriate in size.

[0012] According to a third aspect of the application, a computer storage medium is provided, the computer storage medium comprising instructions which, when executed, perform the method according to the first aspect of the application.

[0013] According to a fourth aspect of the application, a machine vision system is provided, the machine vision system comprising the apparatus according to the first aspect of the application.

[0014] The method and apparatus for visualized interpretation and robustness verification according to the application further provide quantitative calculation indexes on the basis of visualized interpretation, which can be used as a basis for accurately evaluating the robustness of the vision system. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above and other objects and advantages of the present application will become more apparent by describing in detail the following non-restrictive embodiments thereof with reference to the attached drawings, in which like reference numerals refer to like elements throughout.

[0016] Figure 1 A schematic diagram of the method for visualized interpretation and robustness verification according to one embodiment of the application is shown;

[0017] Figure 2 A schematic diagram of the apparatus for visualized interpretation and robustness verification according to one embodiment of the application is shown. DETAILED DESCRIPTION

[0018] The method and apparatus for visualized interpretation and robustness verification according to the application will be described in further detail below with reference to the accompanying drawings. It should be noted that the following detailed description is illustrative only and is not intended to limit the present application, and is not intended to identify key or critical elements or limit the scope of protection.

[0019] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0020] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0021] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0022] Below, a method 100 for visualizing explanation and robustness verification of an inference process of a machine vision system according to an embodiment of the present application will be explained with reference to a block diagram illustration, a block diagram and / or a flowchart of the method 100. Figure 1

[0023] Figure 1 is a schematic diagram of the method 100 for visualizing explanation and robustness verification according to an embodiment of the present application. In an embodiment of the present application, the method 100 for visualizing explanation and robustness verification is applicable to an inference process of a machine vision system, and assesses the robustness of the inference process of the machine vision system by highlighting the salient regions of interest of the algorithm and quantifying the proportion of image pixels occupied by the most critical feature regions. As shown in Figure 1 The method 100 for visualizing explanation and robustness verification comprises a processing step 110, an inference step 120, an analyzing step 130 and an assessing step 140.

[0024] ​The processing step 110 can respectively perform mask processing on multiple regions of the original image acquired by the machine vision system to obtain multiple image samples. In the embodiment, the mask processing is random mask interference for the original image. The processing step 110 can obtain sufficient image samples by adding multiple interference mask processing to different regions randomly selected from a small amount of original images by using multiple specific algorithms. The obtained image samples are generally similar but different due to the different regions of the interference mask.

[0025] The inference step 120 includes recording information of the interference mask region, and performing batch inference or matching on the multiple image samples obtained by the interference mask processing by combining the deep learning neural network model or the traditional template matching algorithm to be evaluated, to obtain a large number of different score results. The obtained score results can include quantitative scores or confidence probability results.

[0026] The analysis step 130 includes using data dimension correlation analysis and matrix element weighting operation algorithms to analyze the different influence degrees of each region interfered by the mask on the score results, and fusing the information about the different influence degrees to obtain the significant difference of the model or algorithm to the unmasked feature region of the original input image. The significant difference includes a significant region and a significant value. Optionally, the analysis step 130 further includes visualizing the obtained feature significant region and significant value as a significant difference map. The significant region and significant value can be visualized as a significant difference map in the form of a heat map, contour map, isogram, etc. by superimposing the visualized and quantified feature significant region and significant value on the original image acquired by the machine vision system.

[0027] The evaluation step 140 includes performing data distribution analysis on the scalar data of the significant difference, calculating a significant binary threshold according to a specific formula, performing binary segmentation on the significant difference map using the set binary threshold, and performing weighted superposition on the significant difference map visualized in the form of a heat map, etc. and weighted operation on the significant value, to obtain a dominant significant map. The dominant significant map can visualize and quantify the dominance of the key significant region over the secondary and irrelevant features in the form of a heat map, contour map, etc., thereby reflecting the size and boundary of the most key significant region. The evaluation step 140 can evaluate the key feature region that the understanding algorithm is really concerned about according to the dominant significant map, and confirm the correctness, rationality and robustness of the model or algorithm inference process by combining the quantitative indicators of the feature region and the original score results.

[0028] Optionally, the evaluating step 140 further comprises: quantifying the proportion of the image pixels occupied by the most critical feature region according to the proportion of the binary segmentation, and if the proportion is appropriate, judging that the inference process of the model on the image is robust and reliable, and if the proportion is inappropriate, judging that the inference process of the model on the image is susceptible to interference, and thus the inference result is unreasonable.

[0029] In some embodiments, the proportion of the image pixels occupied by the most critical feature region corresponding to judging that the inference process is robust and reliable ranges from 0.1% to 1%. For example, when the proportion of the image pixels occupied by the most critical feature region ranges from 0.59%, it can be judged that the inference process of the machine vision system is robust and reliable. In some embodiments, the proportion of the image pixels occupied by the most critical feature region corresponding to judging that the inference process is unreasonable ranges from less than 0.1%. For example, when the proportion of the image pixels occupied by the most critical feature region ranges from 0.049%, it can be judged that the inference process of the machine vision system is unreasonable. In other embodiments, other ranges can be selected as the interval range corresponding to judging that the inference process is robust and reliable or the interval range corresponding to judging that the inference process is unreasonable.

[0030] Next, the device 200 for visualizing explanation and robustness verification according to an embodiment of the present application will be described in conjunction with Figure 2

[0031] Figure 2 is a schematic diagram of the device 200 for visualizing explanation and robustness verification according to an embodiment of the present application. The device 200 for visualizing explanation and robustness verification can be arranged in a machine vision system for visualizing explanation and evaluating the correctness, reasonableness and robustness of the inference process of the machine vision system. As shown in Figure 2 , the device 200 for visualizing explanation and robustness verification comprises a processing unit 210, an inference unit 220, an analysis unit 230 and an evaluation unit 240.

[0032] The processing unit 210 is configured to perform the processing step 110 as described above, including respectively performing mask processing on a plurality of regions of the original image acquired by the machine vision system to obtain a plurality of image samples.

[0033] The inference unit 220 is configured to perform the inference step 120 as described above, including batch inference or matching on the plurality of image samples obtained by mask interference to obtain the result of quantified score or confidence probability.

[0034] ​The analysis unit 230 is configured to perform the analysis step 130 as described above, including employing algorithmic operations to analyze the varying degrees of influence of each region masked by the mask on the score result, and fusing information about the varying degrees of influence to derive the salient region and saliency value. The salient region and saliency value can be visualized as a saliency difference map.

[0035] The evaluation unit 240 is configured to perform the evaluation step 140 as described above, including binarizing the saliency difference map using a set binarization threshold and performing a weighted overlay of the saliency difference map, thereby visualizing and quantitatively embodying the advantage of the salient region over secondary, irrelevant features, to evaluate the region of the most critical feature at the time, and utilize the advantage of the salient region to assess the robustness of the inference process.

[0036] According to another aspect of the present application, there is provided a computer readable storage medium having stored therein instructions which, when executed by a processor, cause the processor to perform any one of the methods as described above. The computer readable medium as used in the present application includes all types of computer storage media readable by a general or special purpose computer, including volatile and non-volatile computer storage media, removable and non-removable computer storage media. By way of example, and not limitation, computer readable media can comprise RAM, ROM, EPROM, E2PROM, registers, hard disks, removable disks, CD-ROMs or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other storage medium(s) which can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Disk typically carry data using magnetic media as an example, while disks carry data using optical media as an example. The foregoing also provides an exemplary non-exhaustive list of non-transitory computer readable media that can be used. A "disk" as used herein includes compact disks (CDs), laser disks, optical disks, DVD, etc. A "disk" as used herein also includes a computer hard drive or other storage device that stores data as examples. Combinations of the above should also be included within the scope of computer readable media. The exemplary storage media are coupled to the processor such that the processor can read information from, and write information to, the storage media. In the alternative, the storage media can be integral to the processor. The processor and the storage media can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage media can reside as discrete components in a user terminal.

[0037] The present application realizes visualized explanation and robustness verification of inference results of machine vision systems in the form of black / white box model (i.e. algorithm encapsulation / adjustable). And on the basis of visualized explanation, the present application further provides quantitative calculation indexes, which can be used as a basis for accurate evaluation of the robustness of the vision system. The overall implementation cost of the scheme is low, fast, easy to understand and promote implementation.

[0038] While only certain embodiments of this application have been described, any person skilled in the art will appreciate that many other modifications can be made that still fall within one of more aspects of the application. Accordingly, the examples and embodiments described and pictured herein are to be considered as illustrative only and no limitation is placed on the scope of the application as defined in the claims that follow.

Claims

1. A method for visualizing interpretation and robustness verification, characterized in that, The method comprises: a processing step comprising mask processing a plurality of regions of an original image to obtain a plurality of image samples; an inference step comprising inferring the plurality of image samples to obtain a score result comprising a quantitative score or a confidence probability; an analysis step comprising analyzing an influence degree of the plurality of regions on the score result to derive a saliency difference, and the analysis step further comprises fusing the score result using an algorithmic operation to calculate a feature saliency region and a saliency value, and visualizing the feature saliency region and the saliency value as a saliency difference map, the algorithmic operation comprising a data dimension correlation analysis and a matrix element weighting operation; and an evaluation step comprising evaluating a key feature region according to at least the saliency difference and confirming a robustness of an inference process.

2. The method of claim 1, wherein, The evaluation step comprises performing a data distribution analysis on the saliency value, binarizing and weighting superimposing the saliency difference map to obtain a dominant saliency map capable of embodying a size and a boundary of a most key feature region.

3. The method of claim 2, wherein, The evaluation step comprises calculating a proportion of image pixels occupied by the most key feature region according to a proportion of binarization, and if a size of the proportion is appropriate, determining that the inference process is robust and reliable, and if the size of the proportion is inappropriate, determining that the inference process is unreasonable.

4. An apparatus for visualizing interpretation and robustness verification, characterized in that, The apparatus comprises: a processing unit configured to mask process a plurality of regions of an original image to obtain a plurality of image samples; an inference unit configured to infer the plurality of image samples to obtain a score result comprising a quantitative score or a confidence probability; an analysis unit configured to analyze an influence degree of the plurality of regions on the score result to derive a saliency difference, and the analysis unit is further configured to fuse the score result using an algorithmic operation to calculate a feature saliency region and a saliency value, and visualize the feature saliency region and the saliency value as a saliency difference map, the algorithmic operation comprising a data dimension correlation analysis and a matrix element weighting operation; and an evaluation unit configured to evaluate a key feature region according to at least the saliency difference and confirm a robustness of an inference process.

5. The apparatus of claim 4, wherein, The evaluation unit is further configured to perform a data distribution analysis on the saliency value, binarize and weight superimpose the saliency difference map to obtain a dominant saliency map capable of embodying a size and a boundary of a most key feature region.

6. The apparatus of claim 5, wherein, The evaluation unit is further configured to calculate a proportion of image pixels occupied by the most key feature region according to a proportion of binarization, and if a size of the proportion is appropriate, determine that the inference process is robust and reliable, and if the size of the proportion is inappropriate, determine that the inference process is unreasonable.

7. A computer storage medium, characterized in that The computer storage medium comprises instructions which, when executed, perform the method of any one of claims 1 to 3.

8. A machine vision system characterized by, The machine vision system comprises the apparatus of any of claims 4 to 6.

Images