A method for visualizing analysis of adversarial robustness of a deep neural network
By constructing a generative low-dimensional latent space and forward computation of a neural network model, and combining generative adversarial networks and coordinate mapping networks, the efficiency and accuracy issues of adversarial robustness assessment of deep neural network models are solved, achieving fast and accurate robustness assessment and visualization analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2022-12-02
- Publication Date
- 2026-06-02
Smart Images
Figure CN116151357B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visualization and visual analytics, and in particular to an interactive visual analytics method for robustness against deep neural network models. Background Technology
[0002] In recent years, next-generation artificial intelligence technologies, represented by deep learning, have been increasingly widely applied to all aspects of human production and life, including safety-critical areas such as facial recognition, autonomous driving, and malware detection. For example, autonomous driving systems in automobiles use deep neural networks (DNNs) for path planning and environmental perception. While deep neural networks have demonstrated their powerful capabilities in these complex tasks, recent research indicates that even state-of-the-art deep neural network models can exhibit significant vulnerability to subtle input perturbations. This vulnerability manifests when a carefully designed, imperceptible, minute noise perturbation is added to a sample that the model could otherwise classify correctly, causing the model to make an incorrect prediction with high confidence. Such carefully designed samples intended to deliberately confuse and mislead the model's predictions are called adversarial examples. Correspondingly, the academic community typically refers to the ability of deep learning models to resist adversarial examples and make correct predictions as adversarial robustness. Because adversarial examples possess strong attack concealment and harmfulness, they can seriously threaten safety-critical areas such as autonomous driving where deep neural network models are deployed. Therefore, comprehensive and accurate model robustness assessment becomes crucial before deep neural network models are put into practical application. Traditional robustness calculation methods are time-consuming and therefore cannot be directly applied to interactive analysis systems that involve calculating the robustness of large numbers of samples.
[0003] Currently, in practice, the adversarial robustness of deep learning models is mainly evaluated by calculating metrics. There are two main evaluation metrics: adversarial accuracy and mean minimum adversarial perturbation. A higher metric value indicates greater adversarial robustness. However, metric-based robustness evaluation methods have two limitations.
[0004] First, these metrics only return a single statistic reflecting the overall robustness of the model. They measure the model's average performance on the sample test set and fail to reveal potential vulnerabilities at the partial or individual sample level, potentially leading to biased evaluation results. To better understand, diagnose, and improve the adversarial robustness of deep learning models, analysts need not only information on the model's global performance across the entire test set but also on its local performance at the sample level. This includes understanding which samples exhibit higher (or lower) robustness and whether samples demonstrating high (or low) robustness share common semantic features—providing more comprehensive information.
[0005] Secondly, robustness evaluation metrics are often calculated on existing test sets. However, such test results may not accurately reflect the model's robustness in real-world applications because the real world contains a vast number of samples with diverse characteristics, which are almost impossible to cover by a single existing test set. Research on deep learning model testing indicates that using only a limited, fixed test set to test model robustness is often insufficient. If the test set fails to reflect the model's performance in some possible edge cases, it can lead to unexpected and incorrect predictions in real-world use. Therefore, to identify potential security vulnerabilities in deep neural network models before formal deployment, a more ideal approach is to test model robustness on more new test samples beyond the original test set to obtain more accurate evaluation results. Summary of the Invention
[0006] To address the limitations of the existing technologies, this invention proposes a visual analysis method for the adversarial robustness of deep neural networks, enabling interactive visual analysis of the adversarial robustness of deep neural network models in a generative low-dimensional latent space based on Generative Adversarial Networks (GANs).
[0007] This invention is achieved using the following technical solution:
[0008] A visual analysis method for adversarial robustness of deep neural networks, comprising the following steps:
[0009] Step 1: Combine the trained coordinate mapping network with the generative adversarial network model to construct a generative low-dimensional latent space. The process of constructing the coordinate mapping network training set includes: using the coordinate mapping network to map two-dimensional coordinates into a high-dimensional latent vector z that meets the input dimension requirements of the generative adversarial network model. i The high-dimensional latent vectors are then input into the generative adversarial network model to generate test samples. An isometric feature mapping algorithm is used to reduce the dimensionality of the generated samples to a two-dimensional latent space, represented as two-dimensional coordinates (x, y). i ,y i ), and the high-dimensional latent vector z will be used as the label. i With the two-dimensional coordinates (x) as input features i ,y i Combine ) a batch of input and label data into <(x) i ,y i ),z i >Construct a coordinate mapping network training set, and use the coordinate mapping network training set to train the coordinate mapping network model;
[0010] Step 2: Construct a training set for a robust prediction network model based on a neural network. Use this training set to train the robust prediction network model and then use it to predict the robustness of image samples. The training set for the robust prediction network model consists of a batch of training data on <Π(p i ),r i >Composition, where p i For the i-th image sample in the DNN model training set, Π(p i Let r be the output distribution of the neurons in the penultimate layer of the DNN model for the image sample. i The robustness value is calculated for the image sample using an adversarial attack algorithm.
[0011] Step 3: Construct a robust real-time rendering system for visualizing the distribution and perform robust visual analysis of image samples.
[0012] Compared to traditional robustness calculation methods, this invention achieves the following beneficial effects:
[0013] 1) The designed neural network model for fast prediction of sample robustness can significantly improve the measurement efficiency of sample robustness, thereby helping to realize the real-time rendering of the robustness distribution of a large number of generated samples on a two-dimensional plane and achieve fast robustness prediction.
[0014] 2) Since it only involves the forward computation of the neural network model, the present invention can significantly reduce the robust computation time for large batches of samples, thereby meeting the real-time response requirements of interactive analysis systems;
[0015] 3) The evaluation results can more accurately reflect the model's adversarial robustness in practical applications.
[0016] 4) Visualization makes it easy to manipulate and display robust prediction results. Attached Figure Description
[0017] Figure 1 This is an overall flowchart of a visual analysis method for the adversarial robustness of deep neural networks according to the present invention;
[0018] Figure 2 This is a schematic diagram illustrating the process of generating the training set for a coordinate mapping network.
[0019] Figure 3 This is a schematic diagram illustrating the generation process of new samples in a generative low-dimensional latent space.
[0020] Figure 4 This is a schematic diagram illustrating the process of generating the training set for a robust prediction network model.
[0021] Figure 5A schematic diagram illustrating the process of constructing a visually robust distribution real-time rendering;
[0022] Figure 6 This is a schematic diagram of the system interface for visualization. Detailed Implementation
[0023] The technical solution will now be clearly described in conjunction with the accompanying drawings and embodiments. All other embodiments and technical substitutions of the embodiments obtained by those skilled in the art based on the embodiments of this invention without departing from the spirit of the invention and without creative effort will fall within the scope of protection of this invention.
[0024] This invention comprises three steps: constructing a generative low-dimensional latent space, predicting the robustness of a DNN model, and implementing visual analysis of the adversarial robustness of a DNN model. It achieves the exploration of the robustness of a DNN model in a two-dimensional plane.
[0025] Step 1: Combine the trained coordinate mapping network with the generative adversarial network (GAN) model to construct a generative low-dimensional latent space; where the generative low-dimensional latent space refers to a two-dimensional latent space that can generate high-dimensional data samples through arbitrary latent vector mapping, and each two-dimensional coordinate in the latent space is mapped to a sample.
[0026] The key to constructing a generative low-dimensional latent space is establishing a mapping relationship between the two-dimensional plane and the high-dimensional latent space of the GAN. The specific process includes: using a coordinate mapping network to map two-dimensional coordinates into a high-dimensional latent vector z that meets the input dimension requirements of the generative adversarial network model. i The high-dimensional latent vectors are then input into the generative adversarial network model to generate high-quality test samples. The mapping from low-dimensional latent space to high-dimensional latent space should satisfy the condition that image samples corresponding to coordinate points that are close together in the two-dimensional plane have similar visual features; that is, coordinate points that are closer together in the two-dimensional plane correspond to more similar samples. For example, by providing a training dataset that reflects this mapping relationship to the neural network model, the model can automatically learn the mapping relationship from it. A neural network designed in this way can learn the mapping relationship.
[0027] like Figure 2 The diagram shown illustrates the process of generating the training set for the coordinate mapping network.
[0028] First, a batch of high-dimensional noise following a certain prior distribution (the choice of the prior distribution and the dimension of the latent vectors are determined by a specific GAN model) is randomly generated and represented as high-dimensional latent vectors z. iThe input is a Generative Adversarial Network (GAN) model to generate samples. The generated samples are then dimensionality-reduced to a two-dimensional latent space using the Isometric Mapping (Isomap) algorithm, represented as two-dimensional coordinates (x, y). i ,y i ), and the high-dimensional latent vector z will be used as the label. i With the two-dimensional coordinates (x) as input features i ,y i Combine ) a batch of input and label data into <(x) i ,y i ),z i >Construct a coordinate mapping network training set, collecting 20,000 <(x i ,y i ),z i The data pairs serve as the training set for the coordinate mapping network;
[0029] Model Design and Training: The coordinate mapping network is defined as a fully connected neural network. Its input features are two-dimensional coordinates, and its output is a high-dimensional latent vector of the GAN's latent space, the dimension of which is determined by the dimension of the GAN's latent space. The coordinate mapping network typically has 3-5 hidden layers, each containing 1024-2048 neurons, all activated using the ReLU activation function. The model is trained using a training set.
[0030] like Figure 3 The diagram illustrates the process of generating new samples in a generative low-dimensional latent space. The generative low-dimensional latent space is constructed by combining ① the trained coordinate mapping network with ② the generative adversarial network (GAN) model.
[0031] The above process ensures that a large number of generated samples on the two-dimensional plane have sufficiently high image quality and feature diversity. Model evaluation on these generated samples can more fully and accurately reflect the adversarial robustness of the model in practical applications.
[0032] Step 2: Construct a training set for a robust prediction network model based on neural networks, train the robust prediction network model using the training set, and use the robust prediction network model based on neural networks to make fast and robust predictions on samples.
[0033] Since this method only involves the forward computation of the neural network model, it can significantly reduce the robust computation time for large batches of samples, thus meeting the real-time response requirements of interactive analysis systems.
[0034] The training set for robust prediction network models consists of a batch of training data on <Π(p) i ),r i >Composition, where pi For the i-th image sample in the DNN model training set, П(p) i ) represents the output distribution of the neurons in the penultimate layer of the DNN model for this sample, r i This is the robustness value calculated for this sample using an adversarial attack algorithm. Collecting training data requires collecting the neuron output distribution П(p) of a batch of samples. i ) and robustness value r i .
[0035] like Figure 4 The diagram illustrates the process of generating the training set for a robust prediction network model. A batch of image sample datasets is randomly selected from the training set of the deep neural network (DNN) model. These image sample datasets are then input into the DNN model under test. Forward operations are performed to obtain the output distribution П(p) of the neurons in the penultimate layer of the DNN model for each batch of image samples. i The robustness value r of the same batch of image samples was calculated and saved using adversarial attack algorithms (such as PGD algorithm). i The neuron output distribution П(p) i ) and robustness value r i Combined into a batch of training data pairs <∏(p i ),r i The training set for the robustness prediction network model was constructed using 10,000 training data pairs to ensure training effectiveness. The robustness of the predicted samples was calculated iteratively using an adversarial attack algorithm.
[0036] Model Structure and Training: The robust prediction model is defined as a fully connected neural network, whose input features are the output distribution Π(p) of the penultimate layer neurons in the DNN model. i The label is the robustness value calculated by an adversarial attack algorithm for the same batch of samples. A shallow, simple, fully connected neural network is sufficient. Specifically, 3-5 hidden layers with 128-512 neurons each (each hidden layer followed by a ReLU activation function) can achieve good prediction performance, and the model can be trained using a training set.
[0037] Step 3: Construct a real-time rendering visualization of robust distribution, and perform visual analysis of image samples and their robustness, such as... Figure 5The diagram illustrates the process of constructing a robust distribution visualization and rendering it in real time. It includes: ① The user selects the display area of the latent space using view transformation operations (e.g., panning and zooming) integrated into the visualization system. ② The currently selected display area is divided into a set of grids of equal size, and a two-dimensional coordinate is selected from each grid as the representative coordinate point of that grid. ③ A coordinate mapping network model maps each two-dimensional coordinate selected in the previous step into a high-dimensional latent vector. ④ A generative adversarial network (GAN) model is used to generate a set of high-quality image samples using the high-dimensional latent vector as input. ⑤ A robustness prediction network model is used to predict the robustness values of the samples generated by the GAN model. This yields the robustness prediction values of the representative generated samples corresponding to each grid in the generative low-dimensional latent space display area. ⑥ The robust distribution of the image samples within the generative low-dimensional latent space display area is presented by drawing contour maps filled with gradient colors. During the exploration task, as the user continuously performs view transformation operations, the above six-step workflow is repeated, rendering the robust distribution in the new latent space display area selected by the user in real time.
[0038] like Figure 6 The diagram shown illustrates the visualization system interface. It primarily includes two views: a robustness distribution view and a sample distribution view. These views collaboratively display the generated samples and their robustness distributions corresponding to different regions in the latent space. Furthermore, various view transformations and sample selection operations support a range of flexible exploration tasks related to the robustness of DNN models in generative low-dimensional latent spaces.
[0039] As shown in (6a), this is a robustness distribution view. Each coordinate in the view corresponds to a generated sample, and its color code reflects the robustness of the corresponding generated sample. The legend in the lower right corner of the view shows the robustness range corresponding to different colors. In addition, users can adjust the resolution slider at the top of the view at any time to dynamically adjust the grid sampling granularity, thereby changing the fineness of the robustness distribution view.
[0040] As shown in (6b), the sample distribution view. This view is to the right of the robustness distribution view, showing representative samples corresponding to the center point of each grid within the current latent space display range, totaling 400. The sample distribution view and the robustness distribution view are locked together; they always display the robustness distribution and sample distribution within the same latent space display range. If the user changes the display range of either view through a view transformation operation, the other view will also respond in real time and synchronously switch to display the robustness distribution or sample distribution within the new display range.
[0041] The visualization system primarily supports robust interactive analysis of the model through integrated view manipulation and data selection functions.
[0042] View operations include: the system supports various view operations, including zooming in, zooming out, panning, and bounding box zooming. Users can not only drag and pan the robust distribution view with the mouse, but also perform the same operation on the sample distribution view. For bounding box zooming, users can click the corresponding button to drag a box in either the left or right view and zoom directly to the area of interest.
[0043] Data selection functionality. Users can use point selection to click on any coordinate in the robustness distribution view or any sample in the sample distribution view to view its corresponding information. The system interface can be designed with an information panel to display detailed information about the selected sample, including the sample image, robustness prediction value, and the predicted probability distribution output by the tested DNN model. Users can also use bounding box selection to select a region of interest in the view; a rectangle of the exact same shape and size will then appear simultaneously in the same location in another view. Users can arbitrarily change the shape and size of the rectangle or drag and pan it to any region in the view; the rectangle in the other view will respond synchronously and display the same change. This bounding box selection function allows users to focus on specific local areas in the latent space and accurately correlate the robustness distribution and sample distribution within those areas, avoiding potential visual interference during exploration tasks.
[0044] In summary, the core idea of visualizing DNN model robustness in a generative low-dimensional latent space is to construct an interactive exploration framework based on generative adversarial networks (GANs). This framework transforms each low-dimensional (2D) coordinate in a 2D projection plane into a generated sample with high-quality and diverse features. Then, based on the robustness values of the corresponding generated samples, the 2D plane is visually encoded using gradient-varying colors to represent the model's robustness distribution in the latent space. Simultaneously, various basic view operations, such as zooming, panning, and bounding selection, are configured on the 2D plane, allowing analysts to explore any region of the plane like operating an online map. The color distribution on the 2D plane provides intuitive and comprehensive information on sample robustness. Furthermore, the large number of newly generated samples with diverse features in the 2D plane can approximate the sample situations that the DNN model might encounter in the application domain, thus more accurately revealing the model's potential vulnerabilities in real-world applications.
[0045] This invention allows analysts to intuitively and flexibly explore the adversarial robustness of models at both the global and local levels in a two-dimensional plane, uncovering more comprehensive robustness information that statistical indicators cannot reveal. This enables more effective assessment, diagnosis, and improvement of model adversarial robustness. Furthermore, the generative nature of the low-dimensional latent space allows for the generation of a large number of new samples that conform to the distribution of the training set data in the two-dimensional plane, approximating various sample situations the model might encounter in real-world applications. This results in evaluation results that more accurately reflect the model's adversarial robustness in practical applications.
Claims
1. A visual analysis method for the adversarial robustness of deep neural networks, characterized in that, The method includes the following steps: Step 1: Combine the trained coordinate mapping network with the generative adversarial network model to construct a generative low-dimensional latent space. The process of constructing the coordinate mapping network training set includes: dividing the currently selected display area into a set of grids of equal size, selecting a two-dimensional coordinate from each grid as the representative coordinate point of that grid; and using the coordinate mapping network to map the two-dimensional coordinates into a high-dimensional latent vector z that meets the input dimension requirements of the generative adversarial network model. i The high-dimensional latent vectors are then input into the generative adversarial network model to generate test samples. An isometric feature mapping algorithm is used to reduce the dimensionality of the generated samples to a two-dimensional latent space, represented as two-dimensional coordinates (x, y). i ,y i ), and the high-dimensional latent vector z will be used as the label. i With the two-dimensional coordinates (x) as input features i ,y i Combine a batch of input and label data. A coordinate mapping network training set is constructed, and the coordinate mapping network model is trained using the coordinate mapping network training set; wherein the input features of the coordinate mapping network model are two-dimensional coordinates, and the output is a high-dimensional latent vector of the latent space of the generative adversarial network. Step 2: Construct a training set for a robust prediction network model based on a neural network. Use this training set to train the robust prediction network model and then use it to predict the robustness of image samples. The training set for the robust prediction network model consists of a batch of training data... Composition, in which For the DNN model training set Image samples, This represents the output distribution of the neurons in the penultimate layer of the DNN model for image samples. The robustness value is calculated for the image sample using an adversarial attack algorithm; the robust prediction model's input feature is the output distribution of the penultimate layer neurons in the DNN model. The label is the robustness value calculated by the adversarial attack algorithm for the same batch of samples; Step 3: Construct a real-time rendering of a robust visual distribution and perform visual analysis of image samples and their robustness.
2. The visual analysis method for adversarial robustness of deep neural networks as described in claim 1, characterized in that, The coordinate mapping network uses a fully connected neural network with 3-5 hidden layers. Each hidden layer contains 1024-2048 neurons, and all neurons are activated using the ReLU activation function.
3. The visual analysis method for adversarial robustness of deep neural networks as described in claim 1, characterized in that, The robust prediction model employs a fully connected neural network with 3-5 hidden layers, each containing 128-512 neurons, followed by a ReLU activation function.
4. The visual analysis method for adversarial robustness of deep neural networks as described in claim 1, characterized in that, The real-time rendering and generation process of the visualization robustness distribution in step three includes: ① The user selects the display area of the latent space using the view transformation operation integrated in the visualization system; ② The coordinate mapping network model maps each two-dimensional coordinate selected in the previous step into a high-dimensional latent vector; ③ A set of high-quality image samples is generated using a generative adversarial network model with the high-dimensional latent vector as input; ④ The robustness value of the samples generated by the tested deep neural network model is predicted using a robust prediction network model; the robustness prediction value of the representative generated sample corresponding to each grid in the generative low-dimensional latent space display area is obtained; ⑤ The robustness distribution of the image samples in the generative low-dimensional latent space display area is presented by drawing a contour map filled with gradient colors.