Network model interpretation method and device and related product

Abstract

The invention provides a network model interpretation method and device and a related product, and relates to the technical field of computers and Internet. The method comprises the following steps: acquiring an original image and a first adversarial interpretation image; determining a first disturbance of the first adversarial interpretation image relative to the original image; determining that the maximum disturbance absolute value of the first disturbance is smaller than or equal to the local minimum disturbance value; determining a first disturbance variable quantity along a gradient direction of classification change towards the target category; updating the first disturbance according to the first disturbance variable quantity; wherein the updated first disturbance is used for controlling the prediction category of the original image to approach the target category; and adding the first disturbance to the original image to generate a second adversarial interpretation image so as to interpret the process of classifying the original image into the target category by the target model through the second adversarial interpretation image. Through the second adversarial interpretation image provided by the invention, the basis of decision making of the target model can be clearly displayed, and the interpretability of the model is enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer and Internet technology, and in particular to a network model interpretation method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] With the development of technology, deep learning models have been applied to various fields for classification and prediction. For example, in the field of medical image analysis, deep learning models have been widely used in disease classification and diagnosis.

[0004] At present, the relevant technology proposes Gradient-weighted Class Activation Mapping (Grad-CAM) to explain the prediction process of deep models. This method generates heat maps to highlight the areas that the model focuses on during the decision-making process. However, although these heat maps can show the size and importance of the areas that the model focuses on, their interpretability is very limited. Especially in medical image classification tasks, the subtle features between different categories are often difficult to clearly distinguish through Grad-CAM, resulting in medical professionals lacking sufficient trust in the basis for their judgment of the model. Summary of the invention

[0005] The purpose of the present disclosure is to provide a network model interpretation method, device, electronic device, computer-readable storage medium and computer program product, which can improve the interpretability of the network model.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] The disclosed embodiment provides a network model interpretation method, comprising: acquiring an original image and a first adversarial interpretation image; determining a first perturbation of the first adversarial interpretation image relative to the original image; determining that a maximum perturbation absolute value of the first perturbation is less than or equal to a local minimum perturbation value; determining a first perturbation change amount along a gradient direction that changes toward the target category classification; updating the first perturbation according to the first perturbation change amount; wherein the updated first perturbation is used to control the predicted category of the original image to approach the target category; adding the first perturbation to the original image to generate a second adversarial interpretation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial interpretation image.

[0008] In some embodiments, determining a first disturbance change amount along a gradient direction that changes toward the target category classification includes: obtaining a preset disturbance step size; classifying the first adversarial interpretation image through the target model to determine a predicted category corresponding to the first adversarial interpretation image; determining a target loss between the predicted category and the target category; determining a gradient direction of the target loss relative to the adversarial interpretation image; and determining the first disturbance change amount based on the gradient direction and the preset perturbation step size.

[0009] In some embodiments, updating the first disturbance according to the first disturbance change includes: adding the first disturbance change to the first disturbance to determine the second disturbance; and setting the first disturbance equal to the second disturbance to update the first disturbance.

[0010] In some embodiments, the method also includes: determining that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; multiplying the first disturbance by a target coefficient to determine a second disturbance change; wherein the target coefficient is greater than 1; updating the first disturbance according to the second disturbance change; adding the second disturbance to the original image to generate a third adversarial explanation image, so as to explain the process of the target model classifying the original image into a target category through the third adversarial explanation image.

[0011] In some embodiments, the method also includes: if it is determined that the second norm of the first perturbation is less than the global perturbation value, setting the global perturbation value equal to the second norm of the first perturbation; if it is determined that the second norm of the first perturbation is greater than or equal to the global perturbation value, initializing the first perturbation; adding the initialized first perturbation to the original image to generate a fourth adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the fourth adversarial explanation image.

[0012] In some embodiments, the method also includes: determining the infinite norm of the first disturbance; if the infinite norm of the first disturbance is greater than the local minimum disturbance value, determining that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; if the infinite norm of the first disturbance is less than or equal to the local minimum disturbance value, determining that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value.

[0013] The embodiment of the present disclosure provides a network model interpretation device, including: an image acquisition module, a first disturbance determination module, a first comparison module, a first disturbance change amount determination module, a disturbance update module and a second adversarial interpretation image generation module.

[0014] Among them, the image acquisition module is used to acquire an original image and a first adversarial interpretation image; the first disturbance determination module can be used to determine a first disturbance of the first adversarial interpretation image relative to the original image; the first comparison module can be used to determine that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value; the first disturbance change amount determination module can be used to determine the first disturbance change amount along the gradient direction of the change toward the target category classification; the disturbance update module can be used to update the first disturbance according to the first disturbance change amount; wherein the updated first disturbance is used to control the predicted category of the original image to be close to the target category; the second adversarial interpretation image generation module can be used to add the first disturbance to the original image to generate a second adversarial interpretation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial interpretation image.

[0015] An embodiment of the present disclosure proposes an electronic device, which includes: a memory and a processor; the memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement any of the network model interpretation methods described above.

[0016] The embodiment of the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon to implement the network model interpretation method as described in any one of the above items.

[0017] The embodiment of the present disclosure provides a computer program product or a computer program, which includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the above network model interpretation method.

[0018] The network model interpretation method, device, electronic device, computer-readable storage medium and computer program product provided by the embodiments of the present disclosure can perform a small range search along the gradient direction toward the target category classification, so as to find the first disturbance change amount, and update the first disturbance by the first disturbance change amount. According to the first disturbance, the original image can be superimposed to shield the information in the original image that is irrelevant to the target category classification result, and the information in the original image that contributes more to the target category classification result can be highlighted. Through this method, on the one hand, the area of ​​interest of the target model classification can be displayed, and the attention information of different category features can also be provided, especially when the category difference is small; on the other hand, the vulnerability of the model interpretation method in the face of adversarial samples can be reduced, the network model interpretation method can be prevented from being affected by small disturbances, the classification accuracy can be improved, and the credibility of the network model interpretation method can be improved; in short, the above method can make each update more effectively approach the optimal interpretation image through adversarial disturbance, can avoid invalid disturbances, and improve the efficiency and accuracy of the adversarial interpretation image process, thereby improving the transparency of the model output and enhancing the interpretability.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0021] Figure 1 A scenario schematic diagram showing a network model interpretation method or a network model interpretation device that can be applied to an embodiment of the present disclosure is shown.

[0022] Figure 2 The figure is a flowchart of a network model interpretation method according to an exemplary embodiment.

[0023] Figure 3 The figure is a flow chart of a method for determining a first disturbance variation according to an exemplary embodiment.

[0024] Figure 4 The figure is a flowchart of a network model interpretation method according to an exemplary embodiment.

[0025] Figure 5 The figure is a flowchart of a network model interpretation method according to an exemplary embodiment.

[0026] Figure 6 It is a flowchart of a network model interpretation method according to an exemplary embodiment.

[0027] Figure 7 The figure is a block diagram of a network model interpretation device according to an exemplary embodiment.

[0028] Figure 8 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0030] Those skilled in the art will appreciate that the embodiments of the present disclosure may be a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0031] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0032] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0033] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0035] In the description of the present disclosure, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a kind of association relationship describing the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different; the terms "comprising", "including" and "having" are used to express the meaning of open inclusion and mean that in addition to the listed elements / components / etc., there may be other elements / components / etc.

[0036] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0037] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0038] First, some terms involved in the embodiments of the present disclosure are explained below to facilitate understanding by those skilled in the art.

[0039] Adversarial Sample: refers to a sample generated by applying small perturbations to the original input data. These perturbations are almost imperceptible to human observers, but may cause machine learning models to make incorrect classifications or judgments.

[0040] Medical Classification Model: A machine learning model used to analyze and classify medical images or data to assist doctors in making diagnostic decisions. The model is trained based on a large amount of training data to identify specific diseases or conditions.

[0041] Explainability: refers to the transparency and understandability of model outputs, that is, the ability of users to understand how the model makes a specific decision or prediction. In the medical field, model explainability is crucial to enhancing doctors’ trust and improving diagnostic accuracy.

[0042] Stochastic Gradient Descent (SGD): An optimization algorithm used to update model parameters in machine learning and deep learning. It randomly selects a small portion of data for each iteration to reduce computational cost and speed up the convergence process.

[0043] Output Class: refers to the predictions generated by the model after processing the input data, usually expressed as class labels. For example, in medical image classification, the output class may represent different types of diseases.

[0044] Perturbation Amount: The amount of change applied during adversarial sample generation, used to evaluate the difference between adversarial samples and original samples. An appropriate amount of perturbation can effectively change the model output without affecting human recognition of images.

[0045] Image Mapping: The generated adversarial examples are adjusted to a specific image output space to ensure that the generated images meet actual visualization standards, such as pixel values ​​ranging between [0, 1].

[0046] Gradient Direction: During the optimization process, the partial derivative of the model output category is calculated according to the loss function to determine the most effective update direction to improve the interpretability of the model.

[0047] The foregoing introduces some terms and concepts involved in the embodiments of the present disclosure. The following introduces the technical features involved in the embodiments of the present disclosure.

[0048] Among related technologies, although methods such as Grad-CAM can provide local interpretability, they cannot effectively display the decision boundaries and feature differences between different categories, resulting in medical professionals lacking sufficient trust in the basis for their judgment of the model.

[0049] There are at least the following technical problems in the related art.

[0050] 1. Lack of interpretability: As mentioned earlier, although methods such as Grad-CAM show the area that the model focuses on, they cannot provide specific information about category characteristics, especially when the category differences are small.

[0051] 2. Poor robustness: Deep learning models are vulnerable to adversarial samples and are easily affected by small perturbations, resulting in misclassification. This not only affects the accuracy of the model, but also weakens its application credibility in the medical field to a certain extent.

[0052] 3. Lack of transparency: Due to the black box nature of deep learning models, medical practitioners often find it difficult to understand the model’s decision-making process, which leads to doubts about its output results, limiting the model’s widespread application.

[0053] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0054] Figure 1 A scenario schematic diagram showing a network model interpretation method or a network model interpretation device that can be applied to an embodiment of the present disclosure is shown.

[0055] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present disclosure.

[0056] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0057] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual reality devices, smart homes, etc.

[0058] The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device.

[0059] The server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms, etc. The present disclosure does not impose any restrictions on this.

[0060] The server 105 may, for example, obtain an original image and a first adversarial interpretation image; the server 105 may, for example, determine a first perturbation of the first adversarial interpretation image relative to the original image; the server 105 may, for example, determine that the maximum perturbation absolute value of the first perturbation is less than or equal to the local minimum perturbation value; the server 105 may, for example, determine a first perturbation change amount along a gradient direction that changes toward a target category classification; the server 105 may, for example, update the first perturbation according to the first perturbation change amount; wherein the updated first perturbation is used to control the predicted category of the original image to approach the target category; the server 105 may, for example, add the first perturbation to the original image to generate a second adversarial interpretation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial interpretation image.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the figure is only for illustration. The server 105 may be a physical server or may be composed of multiple servers. According to actual needs, it may have any number of terminal devices, networks and servers.

[0062] Figure 2 is a flowchart of a network model interpretation method according to an exemplary embodiment. The method provided by the embodiment of the present disclosure can be executed by any electronic device with computing processing capabilities. For example, the method can be executed by the above Figure 1 The server or terminal device in the embodiment may be executed, or the server and the terminal device may be executed together. In the following embodiments, the server is used as the execution subject for example, but the present disclosure is not limited to this.

[0063] Reference Figure 2 The network model interpretation method provided by the embodiment of the present disclosure may include the following steps.

[0064] Step S202, obtaining an original image and a first adversarial interpretation image.

[0065] In some embodiments, the original image may refer to an image without adding disturbances. The original image may be a medical image or other image that needs to be classified, and the present application does not impose any limitation on this.

[0066] The first adversarial interpretation image may refer to an image after adding disturbance to the original image, and the first adversarial interpretation image may be used to explain the process of the target model classifying the original image into the target category. For example, the key information that the target model classifies the original image into the target category may be highlighted, and the information that the target model classifies the original image into the target category may be weakened, and the present application does not limit this.

[0067] During training, the first adversarial interpretation image can be initialized as the original image, or it can be set to the second adversarial interpretation image, the third adversarial interpretation image, the fourth adversarial interpretation image, etc. generated by subsequent training. For example, during the first training, the first adversarial interpretation image can be set as the initial image, during the second training, the result of the first training - the second adversarial interpretation image can be set as the first adversarial interpretation image, and during the third training, the result of the second training - the third adversarial interpretation image can be set as the first adversarial interpretation image. In short, the adversarial interpretation image obtained in each training can be used as the first adversarial interpretation image for the next training.

[0068] Among them, the above-mentioned target model can be any network model that can perform classification processing, such as a deep learning model, a machine learning model, etc. This application does not limit the network structure of the target model.

[0069] Step S204: determining a first disturbance of the first adversarial interpretation image relative to the original image.

[0070] In some embodiments, a first perturbation of the first adversarially interpreted image relative to the original image may be determined.

[0071] For example, assuming that the first adversarial interpretation image is the adversarial interpretation image result obtained in the last training, then the first perturbation updated in the last training process can be directly obtained as the first perturbation of this training.

[0072] For another example, the pixel values ​​of the first adversarial interpretation image may be subtracted from the pixel values ​​of the original image by position to obtain the first disturbance.

[0073] In some embodiments, the first perturbation may be a matrix having the same size as the original image (or the first adversarial interpretation image).

[0074] Step S206: determine whether the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value.

[0075] In some embodiments, the absolute value of the first disturbance at each position may be obtained, and then the maximum absolute value of the disturbance in the first disturbance may be obtained, and finally the maximum absolute value of the disturbance in the first disturbance may be compared with a preset local minimum disturbance value (preset value).

[0076] For example, the infinite norm of the first disturbance can be determined as the maximum disturbance absolute value of the first disturbance. If the infinite norm of the first disturbance is greater than the local minimum disturbance value, it is determined that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; if the infinite norm of the first disturbance is less than or equal to the local minimum disturbance value, it is determined that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value.

[0077] In some embodiments, if the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value, it means that the disturbance in the first adversarial interpretation image may be small, and it is necessary to slowly search for the optimal second adversarial interpretation image in a small amplitude.

[0078] Step S208, determining a first disturbance change amount along a gradient direction of change toward the target category classification.

[0079] In some embodiments, the first disturbance variation may be searched along a gradient direction toward the target category classification so as to determine a second adversarial interpretation image according to the first disturbance variation.

[0080] For example, the predicted classification result after classifying the original image through the target model can be determined, and then the loss between the predicted classification result and the target category can be calculated. The calculated loss is then used to find the gradient along the target category direction to determine the change direction of the first disturbance amount, and finally the change direction determines the first disturbance amount.

[0081] Step S210, updating the first disturbance according to the first disturbance variation; wherein the updated first disturbance is used to control the predicted category of the original image to be close to the target category.

[0082] In some embodiments, the first disturbance may be added to the first disturbance variation to determine the second disturbance; and then the first disturbance may be set equal to the second disturbance to update the first disturbance.

[0083] It can be understood that since the first disturbance after the update is based on

[0084] Step S212, adding the first perturbation to the original image to generate a second adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial explanation image.

[0085] In some embodiments, the first perturbation may be added to the original image to generate a second adversarially interpreted image.

[0086] The adversarial interpretation image may be an adversarial image generated by perturbation, and the adversarial image generated by perturbation may highlight the information classified as the target category in the original image, and may also weaken the information classified as the target category in the original image. In addition, by interpreting the classification process of the target model through the adversarial image, the influence of the adversarial perturbation on the classification process may be enhanced, and the influence of different categories on the interpretation process may be reduced.

[0087] The target model may be a trained network model capable of classifying images. The present application does not impose any restrictions on the network structure of the target model.

[0088] The above method can find the first disturbance variation by searching in a small range along the gradient direction toward the target category classification when the local maximum disturbance of the first disturbance is relatively small, and update the first disturbance by the first disturbance variation. According to the first disturbance, the original image can be superimposed to shield the information in the original image that is irrelevant to the target category classification result, and the information in the original image that contributes more to the target category classification can be amplified and displayed. Through this method, on the one hand, the area of ​​interest of the target model classification can be displayed, and the attention information of different category features can also be provided, especially when the category difference is small; on the other hand, the vulnerability of the model interpretation method in the face of adversarial samples can be reduced, the network model interpretation method can be prevented from being affected by small disturbances, the classification accuracy can be improved, and the credibility of the network model interpretation method can be improved; in short, the above method can make each update more effectively approach the optimal explanation image through adversarial disturbance, avoid invalid disturbances, and improve the efficiency and accuracy of the adversarial explanation image process, thereby improving the transparency of the model output and enhancing the interpretability.

[0089] Figure 3 The figure is a flow chart of a method for determining a first disturbance variation according to an exemplary embodiment.

[0090] refer to Figure 3 , the above-mentioned method for determining the first disturbance change amount may include the following steps.

[0091] Step S302, obtaining a preset disturbance step size.

[0092] In some embodiments, the preset disturbance step size may be a pre-set step size value, which may be a few pixels or dozens of pixels, and the present application does not impose any limitation on this.

[0093] Generally speaking, the preset disturbance step size is smaller than or much smaller than the first disturbance.

[0094] The preset step size may be a matrix having the same size as the first disturbance.

[0095] Step S304: classify the first adversarial interpretation image through the target model to determine the predicted category corresponding to the first adversarial interpretation image.

[0096] In some embodiments, the first adversarial interpretation image may be processed by a target model to determine a predicted category s corresponding to the first adversarial interpretation image.

[0097] Step S306, determining the target loss between the predicted category and the target category.

[0098] In some embodiments, a loss L(s, t) between a predicted category and a target category may be determined, where s represents the predicted category of the first adversarial interpretation image, and t represents a preset target category. The predicted category and the target category may be the same or different, and this application does not limit this.

[0099] Step S308, determining the gradient direction of the target loss relative to the adversarial interpretation image.

[0100] In some embodiments, the gradient direction of the target loss with respect to the adversarial explanation image can be determined, for example, by Determine the gradient direction of the target loss relative to the adversarial explanation image. represents the loss between the predicted category and the target category, and (x+δ) represents the adversarial explanation image.

[0101] Step S310: determining a first disturbance variation according to a gradient direction and a preset disturbance step size.

[0102] In some embodiments, a preset disturbance step length α may be extended along the gradient direction to determine the first disturbance change amount, for example, by Where α represents the preset perturbation step size.

[0103] In some embodiments, after determining the first disturbance variable, the first disturbance can be added to the first disturbance variation to determine the second disturbance; and then the first disturbance can be set equal to the second disturbance to update the first disturbance.

[0104] The first perturbation variable determined by the above method is along the gradient direction of the transformation to the target category classification. The second adversarial interpretation image is determined by the first perturbation variable, and the second adversarial interpretation image that meets the conditions can be searched along the gradient direction of the transformation to the target category classification.

[0105] Figure 4The figure is a flowchart of a network model interpretation method according to an exemplary embodiment.

[0106] refer to Figure 4 , the above network model interpretation method may include the following steps.

[0107] Step S402, obtaining an original image and a first adversarial interpretation image.

[0108] Step S404: determine a first disturbance of the first adversarial interpretation image relative to the original image.

[0109] Step S406, determining whether the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value.

[0110] In some embodiments, the infinite norm of the first disturbance can be determined; if the infinite norm of the first disturbance is greater than the local minimum disturbance value, it is determined that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; if the infinite norm of the first disturbance is less than or equal to the local minimum disturbance value, it is determined that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value.

[0111] If the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value, step S408 to step S412 are executed.

[0112] If the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value, step S414 to step S418 are executed.

[0113] Step S408, determining a first disturbance change amount along a gradient direction that changes toward the target category classification.

[0114] Step S410, updating the first disturbance according to the first disturbance variation; wherein the updated first disturbance is used to control the predicted category of the original image to be close to the target category.

[0115] Step S412, adding the first perturbation to the original image to generate a second adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial explanation image.

[0116] In some embodiments, if the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value, it can be determined that the disturbance in the first adversarial interpretation image is small, and the optimal adversarial interpretation image can be searched slowly and in small increments through steps S408 to S412.

[0117] Step S414, multiplying the first disturbance by the target coefficient to determine a change in the second disturbance; wherein the target coefficient is greater than 1.

[0118] In some embodiments, the target coefficient may be 1.5, 2, etc., and this application does not impose any limitation on this.

[0119] By determining the second disturbance variation amount according to the above target coefficient, the search step size can be increased so as to search for the second adversarial interpretation image in as large a range as possible.

[0120] Step S416: update the first disturbance according to the second disturbance change.

[0121] In some embodiments, if the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value, it can be determined that there is a large local disturbance in the first adversarial interpretation image, and the optimal third adversarial interpretation image can be searched extensively through steps S414 to S416.

[0122] Step S418, adding the second perturbation to the original image to generate a third adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the third adversarial explanation image.

[0123] Through the above method, the optimal adversarial interpretation image can be searched for in different situations. For example, when the local perturbation in the first adversarial interpretation image is large, the optimal adversarial interpretation image can be searched for in a large scale; when the local perturbation in the first adversarial interpretation image is small, a small search can be performed in the gradient direction of the target category classification.

[0124] Figure 5 The figure is a flowchart of a network model interpretation method according to an exemplary embodiment.

[0125] refer to Figure 5 , the above network model interpretation method may include the following steps.

[0126] Step S502, obtaining an original image and a first adversarial interpretation image.

[0127] Step S504: determine a first disturbance of the first adversarial interpreted image relative to the original image.

[0128] Step S506, determining whether the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value.

[0129] In some embodiments, the infinity norm of the first disturbance can be determined; if the infinity norm of the first disturbance is greater than the local minimum disturbance value, it is determined that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; if the infinity norm of the first disturbance is less than or equal to the local minimum disturbance value, it is determined that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value.

[0130] If the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value, step S508 to step S512 are executed.

[0131] If the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value, steps S514 to S526 are executed.

[0132] Step S508, determining a first disturbance change amount along a gradient direction of change toward the target category classification.

[0133] Step S510, updating the first disturbance according to the first disturbance variation; wherein the updated first disturbance is used to control the predicted category of the original image to be close to the target category.

[0134] Step S512, adding the first perturbation to the original image to generate a second adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial explanation image.

[0135] Step S514: multiply the first disturbance by the target coefficient to determine the second disturbance variation, wherein the target coefficient is greater than 1.

[0136] Step S516: update the first disturbance according to the second disturbance change.

[0137] Step S518, adding the second perturbation to the original image to generate a third adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the third adversarial explanation image.

[0138] Step S520, determining whether the second norm of the first disturbance is smaller than the global disturbance value.

[0139] In some embodiments, if the second norm of the first disturbance is less than the global disturbance value, it means that the disturbance average value of the first disturbance (or the global disturbance or the overall disturbance) is smaller; if the second norm of the first disturbance is greater than or equal to the global disturbance value, it means that the disturbance average value of the first disturbance (or the global disturbance or the overall disturbance) is larger.

[0140] If it is determined that the second norm of the first disturbance is smaller than the global disturbance value, step S522 is executed.

[0141] Step S522, setting the global disturbance value equal to the second norm of the first disturbance.

[0142] In some embodiments, if the second norm of the first disturbance is smaller than the global disturbance value, it means that the disturbance average value of the first disturbance (or the global disturbance or the overall disturbance) is smaller, and the global disturbance value can be set equal to the second norm of the first disturbance.

[0143] If it is determined that the second norm of the first disturbance is greater than or equal to the global disturbance value, steps S524 to S526 are executed.

[0144] If the second norm of the first perturbation is greater than or equal to the global perturbation value, it indicates that the perturbation average value (or global perturbation or overall perturbation) of the first perturbation is large, and the first perturbation may affect the expression of effective information in the original image (for example, it may blur all effective information), and the first perturbation is no longer suitable as a perturbation of the adversarial interpretation image. In this case, the first perturbation can be initialized to re-search the optimal adversarial interpretation image globally.

[0145] Step S524, initializing the first disturbance.

[0146] Step S526, adding the initialized first perturbation to the original image to generate a fourth adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the fourth adversarial explanation image.

[0147] The present disclosure also proposes a network model interpretation method, in which the target network model can be a medical classification model. The following embodiment will illustrate the application of the network model interpretation method to the field of medical image classification, but the embodiment can also be applied to other classification fields, and the present application does not impose any restrictions on this.

[0148] The technical solution provided in this embodiment can solve the technical problem of how to improve the interpretability of classification models (such as medical image classification models), so that the model can not only focus on key features when making decisions, but also clearly display the impact of these features on model decisions in a visual way. In addition, the adversarial sample method is used to enhance the robustness of the model, thereby improving the credibility of the medical image classification model in practical applications. By strengthening the features of specific categories, the decision differences between categories can be effectively demonstrated, thereby promoting the development and application of medical image analysis technology.

[0149] The model interpretation method provided in this embodiment may include the following steps.

[0150] 1. Get the original image and the first adversarial interpretation image.

[0151] In some embodiments, an original image x may be obtained.

[0152] In some embodiments, a first adversarial interpretation image may be obtained

[0153] In the first training process, the first adversarial interpretation image can be made equal to the original image, such as Alternatively, a small perturbation is randomly overlaid in the original image to generate the first adversarial interpretation image.

[0154] In some embodiments, the first disturbance may also be initialized to 0, such as δ0←0.

[0155] In some embodiments, a local minimum disturbance value ρ may also be set, where ρ may be a smaller value.

[0156] Among them, the local minimum disturbance value ρ can be fixed.

[0157] In some embodiments, the global perturbation value can also be initialized to infinity, for example, M best ←inf.

[0158] Among them, the global perturbation value can be updated during the training process.

[0159] 2. Determine a first perturbation of the first adversarial interpretation image relative to the original image.

[0160] In some embodiments, the first adversarial interpreted image may be subtracted from the original image to obtain the first perturbation.

[0161] For example, during the first training, the first perturbation δ0=0 of the first adversarial interpretation image relative to the original image.

[0162] 3. Determine whether the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value.

[0163] In some embodiments, the infinite norm of the first disturbance |||δ|||∞ can be determined; if the infinite norm of the first disturbance is greater than the local minimum disturbance value ρ (such as |||δ|||∞>ρ), then it is determined that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; if the infinite norm of the first disturbance is less than or equal to the local minimum disturbance value (such as |||δ|||≤ρ), then it is determined that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value.

[0164] If the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value, execute 4 to 6.

[0165] If the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value, execute 7 to 13.

[0166] 4. Determine the first perturbation change along the gradient direction toward the target category classification.

[0167] In some embodiments, Determine the gradient direction of the target loss with respect to the adversarial explanation image, and then pass Determine the first disturbance change. represents the loss between the predicted category and the target category, (x+δ) represents the adversarial explanation image, and α represents the preset perturbation step size.

[0168] In some embodiments, the direction of the adversarial explanation image update can also be the gradient direction of the target category t explained in the output category s Among them, s t It can represent the target loss. Then multiply the gradient direction by the set adversarial perturbation step size α to calculate the change in the first perturbation after each update. This gradient represents the sensitivity of the model to the explained category (such as the target category) t when predicting the output category s. Based on the calculated gradient, an update scheme can be determined, which aims to change the sensitivity of the model to the explained category t when predicting the output category s. This may mean increasing or decreasing the model's attention to the explained category t.

[0169] This approach can be used to improve the interpretability of the model. By adjusting the model's focus on specific categories, we can better control the model's decision-making process. In the case of adversarial attacks, this approach can also be used to analyze the model's behavior when facing adversarial examples, thereby discovering potential weaknesses in the model.

[0170] This method can be used to improve the interpretability of the model. By adjusting the model's focus on specific categories, we can better understand the model's decision-making process. In the case of adversarial attacks, this method can also be used to analyze the model's behavior when facing adversarial samples, thereby discovering potential weaknesses in the model.

[0171] 5. Update the first disturbance according to the change of the first disturbance; wherein the updated first disturbance is used to control the predicted category of the original image to be close to the target category.

[0172] 6. Add the first perturbation to the original image to generate a second adversarial explanation image, so that the process of the target model classifying the original image into the target category is explained by the second adversarial explanation image.

[0173] In some embodiments, after the second adversarial interpretation image is obtained, the second adversarial interpretation image may be used as the first adversarial interpretation image, and steps 1 to 13 may be executed in a loop until a preset number of loops is reached or other preset loop conditions are met.

[0174] 7. Multiply the first disturbance by the target coefficient to determine the change of the second disturbance, where the target coefficient is greater than 1.

[0175] In some embodiments, the first disturbance may be multiplied by a target coefficient to determine a second disturbance change, such as Δδ←2δ.

[0176] 8. Update the first disturbance according to the change of the second disturbance.

[0177] 9. Add the second perturbation to the original image to generate a third adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the third adversarial explanation image.

[0178] 10. Determine whether the second norm of the first perturbation is less than the global perturbation value.

[0179] In some embodiments, if the second norm of the first disturbance is less than the global disturbance value, it means that the disturbance average value of the first disturbance (or the global disturbance or the overall disturbance) is smaller; if the second norm of the first disturbance is greater than or equal to the global disturbance value, it means that the disturbance average value of the first disturbance (or the global disturbance or the overall disturbance) is larger.

[0180] If it is determined that the second norm of the first disturbance is smaller than the global disturbance value, and the maximum disturbance absolute value of the first disturbance is smaller than or equal to the local minimum disturbance value, it means that the current disturbance is the current optimal solution, and step S522 is executed.

[0181] 11. Let the global perturbation value be equal to the second norm of the first perturbation.

[0182] In some embodiments, if the second norm of the first disturbance is smaller than the global disturbance value, it means that the disturbance average value of the first disturbance (or the global disturbance or the overall disturbance) is smaller, then the global disturbance value can be made equal to the second norm of the first disturbance, thereby narrowing the disturbance search range.

[0183] If the absolute value of the maximum disturbance of the first disturbance is less than or equal to the local minimum disturbance value, but the second norm of the first disturbance is greater than or equal to the global disturbance value, it is determined that the local disturbance of the first disturbance exceeds the disturbance amount ρ but is not better than the previous optimal solution, then steps 12 to 13 are executed.

[0184] If the second norm of the first perturbation is greater than or equal to the global perturbation value, it indicates that the perturbation average value (or global perturbation or overall perturbation) of the first perturbation is large, and the first perturbation may affect the expression of effective information in the original image (for example, it may blur all effective information), and the first perturbation is no longer suitable as a perturbation of the adversarial interpretation image. In this case, the first perturbation can be initialized to re-search the optimal adversarial interpretation image globally.

[0185] 12. Initialize the first perturbation.

[0186] 13. Add the initialized first perturbation to the original image to generate a fourth adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the fourth adversarial explanation image.

[0187] In some embodiments, after obtaining the third adversarial interpretation image or the fourth adversarial interpretation image, the third adversarial interpretation image or the fourth adversarial interpretation image can be used as the first adversarial interpretation image, and steps 1 to 13 are executed in a loop until a preset number of loops is reached or other preset loop conditions are met.

[0188] In some embodiments, the updated adversarial explanation sample can be finally (such as the second adversarial interpretation image, the third adversarial interpretation image, the fourth adversarial interpretation image, etc.) are mapped to the effective image output space [0,1] (clip can be used to clip elements in an array or data set Clipping is performed so that its value is limited to between 0 and 1. Specifically, for each element in the array, if the value of the element is less than 0, it will be set to 0; if the value of the element is 1, it will be set to 1; if the value of the element is between 0 and 1, it will remain unchanged).

[0189] Figure 6 It is a flowchart of a network model interpretation method according to an exemplary embodiment.

[0190] like Figure 6 As shown, the original image can be an ophthalmoscopy image, and the target model can be a classification model F(w). The classification model F(w) can be used to classify the ophthalmoscopy image to determine the disease type A of the ophthalmoscopy image, and then the first perturbation determined by the above method is used as an adversarial sample. Finally, the adversarial sample is superimposed on the ophthalmoscopy image to generate an adversarial explanation map that can be used to explain why the classification model determines it as disease A.

[0191] The above embodiments can solve the problem of insufficient interpretability of classification models (such as medical classification models) and provide an effective method to enhance the interpretability of models through adversarial samples. Through iterative updates of optimal perturbations, the observability of adversarial interpretation images is significantly improved, thereby improving the trust and transparency of the model to users and doctors.

[0192] In the medical field, the above method can significantly improve the interpretability of medical image classification models through the generation and iterative updating of adversarial samples, making it easier for doctors to understand the decision-making process of the model. This increased transparency enhances the trust of doctors and patients in the model, provides a clearer basis for medical decision-making, and thus promotes the application and promotion of the model in clinical practice and improves the quality of medical services.

[0193] The above method uses the generation characteristics of adversarial samples to selectively enhance the features of specific categories in the image, and the generated adversarial samples are adversarial explanations. Compared with the commonly used Grad-CAM heat map local interpretation method that only focuses on the weight and size of the region, through adversarial interpretation, a visual local interpretation method, we can better understand how the changes in the focus area will affect the judgment of the image model decision, which can help the model owner better understand the decision boundaries of various categories in medical tasks. There is a clear difference between adversarial explanations and adversarial samples, that is, adversarial samples control the range of added disturbances to make their disturbances small enough to be imperceptible to the human eye. On the contrary, adversarial interpretation imposes a minimum value constraint on the range of disturbance generation, so that the disturbance generated by the adversarial interpretation must be greater than a certain artificially set value to reach the range that can be observed by the human eye. In this way, the superimposed adversarial samples will enhance the features of the specified category. Through the adversarial interpretation generated in this way, the feature information of the differences between the image model categories can be better understood.

[0194] The adversarial interpretation method is applied to the medical image classification model to perform interpretable visualization and analysis of the model category classification. This is because the boundary distance between medical image categories is small, and there may be only some subtle differences between different diseases. Therefore, compared with natural images, the adversarial interpretation method is more suitable for medical imaging tasks. Adversarial interpretation only needs to strengthen the key details of the image input classification. This strengthening is often less than the strength required for adversarial interpretation of natural images. Therefore, it is only necessary to strengthen the features of specific categories in the original medical image input. The generated adversarial interpretation can provide a local visualization explanation of the classification of the deep medical image model.

[0195] Compared with the prior art, the above method has significant advantages in at least the following aspects.

[0196] 1. Enhanced explainability: By generating adversarial samples, the basis for model decisions can be clearly demonstrated, making the interpretation of medical images more intuitive. This is crucial for medical professionals to understand model outputs and helps improve the transparency of clinical decision-making.

[0197] 2. Accurately control disturbances: Adopt a dynamic evaluation mechanism to adjust the adversarial disturbances in real time according to the model output, so that each update can more effectively approach the optimal interpretation image. Compared with traditional methods, this invention can avoid invalid disturbances and improve the efficiency and accuracy of the generation process.

[0198] 3. Improve visualization quality: The final interpretable image is mapped to a valid image output space to ensure compliance with actual visualization standards. This allows doctors to directly observe how the model analyzes and processes medical images, thereby improving the reliability of diagnosis.

[0199] 4. Potential for clinical application: By enhancing the interpretability of medical image classification models, this invention helps to improve doctors’ trust in the models and paves the way for wider clinical applications, which is crucial for patient safety and treatment outcomes.

[0200] In summary, the present invention generates visual interpretable images of medical classification models through adversarial samples, which not only improves the transparency of model output, but also provides more intuitive and reliable decision support for the medical field.

[0201] It should be particularly pointed out that the various steps in the various embodiments of the above network model interpretation method can be mutually crossed, replaced, added, or deleted. Therefore, these reasonable permutations, combinations, and transformations of the network model interpretation method should also fall within the scope of protection of the present disclosure, and the scope of protection of the present disclosure should not be limited to the embodiments.

[0202] Based on the same inventive concept, the present disclosure also provides a network model interpretation device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0203] Figure 7 is a block diagram of a network model interpretation device according to an exemplary embodiment. Figure 7 The network model interpretation device 700 provided in the embodiment of the present disclosure may include: an image acquisition module 701, a first disturbance determination module 702, a first comparison module 703, a first disturbance change amount determination module 704, a disturbance update module 705 and a second adversarial interpretation image generation module 706.

[0204] Among them, the image acquisition module 701 can be used to acquire the original image and the first adversarial interpretation image; the first perturbation determination module 702 can be used to determine the first perturbation of the first adversarial interpretation image relative to the original image; the first comparison module 703 can be used to determine that the maximum perturbation absolute value of the first perturbation is less than or equal to the local minimum perturbation value; the first perturbation change amount determination module 704 can be used to determine the first perturbation change amount along the gradient direction of the change toward the target category classification; the perturbation update module 705 can be used to update the first perturbation according to the first perturbation change amount; wherein the updated first perturbation is used to control the predicted category of the original image to be close to the target category; the second adversarial interpretation image generation module 706 can be used to add the first perturbation to the original image to generate a second adversarial interpretation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial interpretation image.

[0205] It should be noted here that the above-mentioned image acquisition module 701, the first disturbance determination module 702, the first comparison module 703, the first disturbance change amount determination module 704, the disturbance update module 705 and the second adversarial interpretation image generation module 706. 710 corresponds to S202 to S212 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0206] In some embodiments, the first disturbance change amount determination module 704 may include: a disturbance step size acquisition submodule, a classification processing submodule, a target loss determination submodule, a gradient direction determination submodule and a first disturbance change amount determination submodule.

[0207] Among them, the perturbation step acquisition submodule can be used to obtain a preset perturbation step size; the classification processing submodule can be used to classify the first adversarial interpretation image through the target model to determine the predicted category corresponding to the first adversarial interpretation image; the target loss determination submodule can be used to determine the target loss between the predicted category and the target category; the gradient direction determination submodule can be used to determine the gradient direction of the target loss relative to the adversarial interpretation image; the first perturbation change amount determination submodule can be used to determine the first perturbation change amount according to the gradient direction and the preset perturbation step size.

[0208] In some embodiments, the disturbance updating module 705 may include: a second disturbance determining submodule and a first disturbance updating submodule.

[0209] The second disturbance determination submodule can be used to add the first disturbance change to the first disturbance to determine the second disturbance; the first disturbance update submodule can be used to make the first disturbance equal to the second disturbance to update the first disturbance.

[0210] In some embodiments, the network model interpretation device 700 may further include: a second comparison module, a multiplication module, a first disturbance update module and a third adversarial interpretation image generation module.

[0211] Among them, the second comparison module can be used to determine whether the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; the multiplication module can be used to multiply the first disturbance by the target coefficient to determine the second disturbance change; wherein the target coefficient is greater than 1; the first disturbance update module can be used to update the first disturbance according to the second disturbance change; the third adversarial explanation image generation module can be used to add the second disturbance to the original image to generate a third adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the third adversarial explanation image.

[0212] In some embodiments, the network model interpretation device 700 may also include: a global disturbance value updating module, a first disturbance initialization module and a fourth adversarial interpretation image generating module.

[0213] Among them, the global disturbance value update module can be used to set the global disturbance value equal to the second norm of the first disturbance if it is determined that the second norm of the first disturbance is less than the global disturbance value; the first disturbance initialization module can be used to initialize the first disturbance if it is determined that the second norm of the first disturbance is greater than or equal to the global disturbance value; the fourth adversarial explanation image generation module can be used to add the initialized first disturbance to the original image to generate a fourth adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the fourth adversarial explanation image.

[0214] In some embodiments, the network model interpretation device 700 may further include: an infinite norm determination module, a first judgment module, and a second judgment module.

[0215] The infinite norm determination module can be used to determine the infinite norm of the first disturbance; the first judgment module can be used to determine that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value if the infinite norm of the first disturbance is greater than the local minimum disturbance value; the second judgment module can be used to determine that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value if the infinite norm of the first disturbance is less than or equal to the local minimum disturbance value.

[0216] Since the functions of the device 700 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0217] The modules and / or submodules and / or units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The modules and / or submodules and / or units described may also be arranged in a processor. The names of these modules and / or submodules and / or units do not, in certain circumstances, constitute limitations on the modules and / or submodules and / or units themselves.

[0218] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a part of a module or program segment, and a part of the above-mentioned module or program segment contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer program instructions.

[0219] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0220] Figure 8 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Figure 8 The electronic device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0221] like Figure 8 As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0222] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0223] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains computer program instructions for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present disclosure are executed.

[0224] It should be noted that the computer-readable storage medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable computer program instructions. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The computer program instructions contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0225] As another aspect, the present disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by a device, the device can realize functions including: obtaining an original image and a first adversarial interpretation image; determining a first disturbance of the first adversarial interpretation image relative to the original image; determining that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value; determining a first disturbance change along the gradient direction of the change toward the target category classification; updating the first disturbance according to the first disturbance change; wherein the updated first disturbance is used to control the predicted category of the original image to be close to the target category; adding the first disturbance to the original image to generate a second adversarial interpretation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial interpretation image.

[0226] According to one aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer program instructions, the computer program instructions being stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0227] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of computer program instructions for enabling an electronic device (which can be a server or a terminal device, etc.) to execute the method according to the embodiment of the present disclosure.

[0228] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0229] It should be understood that the present disclosure is not limited to the detailed structures, drawings or implementations shown herein, but rather the present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A network model interpretation method, characterized in that: include: Obtaining an original image and a first adversarial interpretation image; determining a first perturbation of the first adversarially interpreted image relative to the original image; Determining that a maximum disturbance absolute value of the first disturbance is less than or equal to a local minimum disturbance value; Determining a first disturbance change amount along a gradient direction of a change toward the target category classification; updating the first disturbance according to the first disturbance change; wherein the updated first disturbance is used to control the predicted category of the original image to be closer to the target category; The first perturbation is added to the original image to generate a second adversarial explanation image, so that the process of the target model classifying the original image into the target category is explained through the second adversarial explanation image.

2. The method according to claim 1 or 2, characterized in that: Determining a first disturbance change amount along a gradient direction of a change toward the target category classification includes: Get the preset perturbation step size; Classify the first adversarial interpretation image using the target model to determine a predicted category corresponding to the first adversarial interpretation image; determining a target loss between the predicted class and the target class; Determining a gradient direction of the target loss relative to the adversarial explanation image; The first disturbance change amount is determined according to the gradient direction and the preset disturbance step size.

3. The method according to claim 2, characterized in that: Updating the first disturbance according to the first disturbance change amount includes: Adding the first disturbance to the first disturbance variation to determine a second disturbance; The first disturbance is set equal to the second disturbance to update the first disturbance.

4. The method according to claim 1, characterized in that: The method further comprises: Determining that a maximum disturbance absolute value of the first disturbance is greater than a local minimum disturbance value; Multiplying the first disturbance by a target coefficient to determine a second disturbance change; wherein the target coefficient is greater than 1; updating the first disturbance according to the second disturbance change; The second perturbation is added to the original image to generate a third adversarial explanation image, so that the process of the target model classifying the original image into the target category is explained by the third adversarial explanation image.

5. The method according to claim 4, characterized in that: The method further comprises: If it is determined that the second norm of the first disturbance is less than the global disturbance value, setting the global disturbance value equal to the second norm of the first disturbance; If it is determined that the second norm of the first disturbance is greater than or equal to the global disturbance value, initializing the first disturbance; The initialized first perturbation is added to the original image to generate a fourth adversarial explanation image, so as to explain the process in which the target model classifies the original image into the target category through the fourth adversarial explanation image.

6. The method according to claim 1 or 4, characterized in that: The method further comprises: determining an infinity norm of the first disturbance; If the infinite norm of the first disturbance is greater than the local minimum disturbance value, determining that the maximum disturbance absolute value of the first disturbance is greater than the local minimum disturbance value; If the infinite norm of the first disturbance is less than or equal to the local minimum disturbance value, it is determined that the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value.

7. A network model interpretation device, characterized in that: include: An image acquisition module, used for acquiring an original image and a first adversarial interpretation image; A first disturbance determination module, configured to determine a first disturbance of the first adversarial interpretation image relative to the original image; A first comparison module, used to determine whether the maximum disturbance absolute value of the first disturbance is less than or equal to the local minimum disturbance value; A first disturbance change amount determination module is used to determine a first disturbance change amount along a gradient direction of the change toward the target category classification; A disturbance updating module, used for updating the first disturbance according to the first disturbance change; wherein the updated first disturbance is used for controlling the predicted category of the original image to be close to the target category; A second adversarial explanation image generating module is used to add the first perturbation to the original image to generate a second adversarial explanation image, so as to explain the process of the target model classifying the original image into the target category through the second adversarial explanation image.

8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the network model interpretation method as described in any one of claims 1-6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the network model interpretation method as described in any one of claims 1-6 is implemented.

10. A computer program product, comprising computer program instructions, wherein the computer program instructions are stored in a computer-readable storage medium, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images