Image Detection Method Based on Embedded Manifold Representation and Information Bottleneck Constraints

Abstract

An image detection method based on embedded manifold representation and information bottleneck constraints is proposed. In the offline stage, a discriminator network and a generative flow model network (acting as an encoder or decoder) are initialized, and reconstruction loss function, information bottleneck loss function, and Jacobian matrix isometry loss function are calculated sequentially to obtain the total loss function. This allows for backpropagation training of both the generative flow model network and the discriminator network. In the online stage, the trained generative flow model network obtains discriminative and common features of the image to be tested. Image reconstruction and submanifold image reconstruction are performed only on images whose common features belong to the common feature space. Image classification is achieved by calculating the distance between the reconstructed image and the reconstructed images of each category's submanifold. This invention avoids meaningless calculations on invalid samples by pre-analyzing the images to be classified, saving computational resources. Furthermore, by utilizing the guidance of embedded manifold representation algorithm in image sample projection and information bottleneck constraints in feature extraction, it demonstrates excellent classification robustness.

Description

Technical Field

[0001] This invention relates to a technology in the field of neural network applications, specifically an image detection method based on embedded manifold representation and information bottleneck constraints. Background Technology

[0002] The primary task of image classification is to identify an input image into a given category. As one of the most fundamental tasks in computer vision and the foundation for other high-level vision tasks such as image detection and segmentation, image classification has wide applications in many fields, such as face recognition, object detection, and video analysis. These applications often require the model algorithm to have a certain degree of robustness, which refers to the model algorithm's ability to resist attack samples. High robustness means that the model algorithm can still exhibit sufficiently good classification accuracy when faced with attack samples. Attack samples are obtained through methods such as adding random noise to the original image, and are intended to cause the model algorithm to give incorrect classification results. However, most existing classification algorithms based on deep neural networks are black-box models, and their classification mechanisms are opaque. This means that they cannot guarantee their ability to resist attack samples, and therefore cannot guarantee the robustness of their classification results. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes an image detection method based on embedded manifold representation and information bottleneck constraints. By pre-analyzing the images to be classified, images that do not belong to any category are rejected for classification, thus avoiding meaningless calculations on invalid samples and saving computational resources. At the same time, by utilizing the guidance of embedded manifold representation algorithm in image sample projection and information bottleneck constraints in feature extraction, its classification robustness is excellent.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to an image detection method based on embedded manifold representation and information bottleneck constraints. In the offline stage, a discriminator network and a generative flow model network acting as an encoder or decoder are initialized, and the reconstruction loss function, information bottleneck loss function, and Jacobian matrix equidistant loss function are calculated sequentially to obtain the total loss function. This allows backpropagation training of the generative flow model network and the discriminator network. In the online stage, the discriminative features and common features of the image to be tested are obtained through the trained generative flow model network. Image reconstruction and submanifold image reconstruction are performed only on images whose common features belong to the common feature space. Image classification is achieved by calculating the distance between the reconstructed image and the reconstructed images of each category of submanifold.

[0006] The term "encoder" or "decoder" refers to the following: when the generative stream model network acts as an encoder, it obtains the common features and discriminative features corresponding to the input image by mapping the input image to the feature encoding space and performing a Cartesian product of the discriminative feature space and the common feature space; when the generative stream model network acts as a decoder, it obtains the corresponding reconstructed image by jointly mapping the common features and discriminative features of the input to the image space.

[0007] The generative flow model network includes a dimension lifting layer and a basic flow module layer. The dimension lifting layer increases the dimension of the input common features and discriminative features to the dimension corresponding to the image space by padding the end of the feature with a specified number of zeros, thus serving as the intermediate feature of the network. The basic flow module layer performs arbitrarily complex nonlinear transformations on the intermediate features to generate the corresponding reconstructed image, while ensuring that the Jacobian matrix volume element of the model network has sufficient complexity.

[0008] The dimension-boosting layer does not contain any neural network parameters.

[0009] The basic flow module layer includes: a reversible convolutional layer and an affine coupling layer stacked sequentially. The reversible convolutional layer mixes the channel features of the input features while ensuring reversibility. The affine coupling layer performs arbitrarily complex nonlinear transformations on the input features while ensuring the same reversibility. Specifically, the first part of the output obtained by segmenting along the image channel dimension is kept the same as the first part of the input features. The second part of the output features is equal to the second part of the input features superimposed on the nonlinear transformation of the first part of the input features, after translation and scaling.

[0010] The reversible convolutional layer refers to a convolutional layer with a kernel size of 1×1 and a weight matrix that is a randomly initialized and non-trainable rotation matrix. It does not contain trainable parameters and its transpose convolution operation is an inverse mapping.

[0011] The affine coupling layer includes stacked convolutional layers and linear rectifier units.

[0012] The discriminator network obtains predicted category labels based on common features, specifically including: fully connected layers and linear rectifier units stacked sequentially, wherein: the fully connected layers perform linear mapping on the input features to obtain high-level features, and the linear rectifier units perform linear rectification processing on the features, introducing nonlinear operations to enable the neural network to fit nonlinear functions.

[0013] The reconstruction loss function Where: g is the generative flow model network. It is the set of discriminative feature cluster center parameters for all trainable categories. The cluster centers are the discriminative features corresponding to category y, and (x, y) is the distribution of self-trained samples p. data The image x obtained from mid-sample and the class label y are paired. For mathematical expectation, It is the inverse g of the generative flow model network. -1 As an encoder, it performs an operational mapping to obtain common features of samples; ||·|| is the L2 norm. Used to determine whether the encoded features exceed the feature space The metric is set to 1 when the input encoded features exceed the feature space, and 0 otherwise. The reconstruction loss function ensures that the generative flow model network can decode and reconstruct the training samples from the joint features of the corresponding discriminative and common features, while the encoded joint features fall within the feature space. This induces the learning of an implicit peripheral generative manifold and provides a basis for accepting or rejecting further classification based on the range of common features.

[0014] The information bottleneck loss function Where: d represents the discriminator network. Let be the classification loss function based on cross-entropy, where: It is the predicted class probability distribution obtained by the softmax operation, where: And d j Let (s) be the j-th component of d(s). Theoretically, it can be proven that this information bottleneck loss function is an implementation of the following Lagrangian function. In form, it is actually the original information bottleneck Lagrangian function under condition T, where X, Y, S, and T are random variables of the input image, input image label, common features, and discriminant features, respectively. Therefore, the information bottleneck loss function is a specific implementation of the information bottleneck Lagrangian function. It adopts an adversarial approach between the generator flow model network and the discriminator network to ensure that the common features encoded by the generator flow model network do not contain any category-related information.

[0015] The Jacobian matrix isometric loss function in: Applying this to the generating manifold, it constrains the local coordinate functions to be equidistant everywhere, thus satisfying the condition that the geodesic distance between two points on the manifold is equal to the Euclidean distance of the corresponding feature encoding in the feature space, which facilitates the estimation of geodesics on the manifold. Acting on the outer space of the generating manifold, the Jacobian matrix of the basic flow model layer of the generating flow model network, while orthogonal, has constant singular values ​​corresponding to the outer space direction. A generating flow model network satisfying this property can theoretically guarantee that input samples located on the outer space of the generating manifold can be projected onto the generating manifold in a geometrically rigorous manner, i.e., the vector between the input image and its outer projection onto the generating manifold is perpendicular to the tangent space of the generating manifold at the outer projection location. This rigorous projection property has a significant impact on the robustness of the classification method; that is, the isometric loss function of the Jacobian matrix on the generating manifold is... Among them: J g (z) is the Jacobian matrix of the generative flow model network g at z. It is J g The transpose of (z), It is the feature space, which is the Cartesian product of common features and discriminative features, ||·|| F Let Frobenius norm be the matrix; the isometry loss function for generating the Jacobian matrix on the outer space of the manifold is: Where: f is the basic flow module layer in the generative flow model network g, J f (u;v) is the Jacobian matrix of the basic flow module layer at [u;v]. It is J f The transpose of (u; v) and These correspond to the coordinate function components on the generating manifold and the spaces corresponding to the coordinate function components on the outer space of the generating manifold, respectively. This refers to element-wise multiplication of matrices. yes The index set corresponding to the coordinate components, s j (u;v) is J f The modulus of the j-th column of (u;v), yes The average value.

[0016] This invention relates to a system for implementing the above method, comprising: a feature extraction unit, a peripheral projection unit, a category projection unit, and a distance calculation unit, wherein: the feature extraction unit extracts the corresponding common features and discriminant features from the input image, and determines whether to perform subsequent classification steps based on whether the common features fall within a preset spatial range; the peripheral projection unit reconstructs the input image on the peripheral manifold based on the extracted common features and discriminant features; the category projection unit reconstructs the input image on each category submanifold based on the peripheral projection and the cluster centers of the learned corresponding category discriminant features; and the distance calculation unit calculates the geodesic distance from the peripheral projection to the projection of each category submanifold on the manifold based on the peripheral projection and the projection of each category submanifold, and selects the category corresponding to the submanifold with the smallest distance as the final classification result.

[0017] Technical effect

[0018] This invention utilizes a generative model that satisfies the isometric property of the Jacobian matrix to project input image samples onto the outer manifold and onto each sample submanifold. Compared with existing technologies, this invention can guarantee the stability of attack samples during the two-stage projection process, thereby ensuring the stability of the classification results. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention;

[0020] Figure 2 This invention generates a flow model network structure diagram;

[0021] Figure 3 This is a network structure diagram of the discriminator model of the present invention;

[0022] Figure 4 This is a schematic diagram illustrating the effect of an example. Detailed Implementation

[0023] like Figure 1 As shown in the figure, this embodiment involves a robust classification method based on embedded manifold representation and information bottleneck algorithm. In the offline stage, the discriminator network and the generator flow model network are initialized, and the reconstruction loss function, information bottleneck loss function, and Jacobian matrix equidistant loss function are calculated in sequence to obtain the total loss function. This allows backpropagation to train the generator flow model network and the discriminator network. In the online stage, the discriminative features and common features of the image to be tested are obtained through the trained generator flow model network. Only images whose common features belong to the common feature space are reconstructed into images and submanifold images. Image classification is achieved by calculating the distance between the reconstructed image and the reconstructed images of each category of submanifold.

[0024] The offline phase specifically includes:

[0025] Step 1: Collect raw data and construct a training dataset Where: N is the number of samples, x (i) and y (i) Let K be the input image and its corresponding category label; the dimensions of the initial discriminative feature space and the common feature space are both K and range from [-γ, γ]. K The joint feature space of discriminative features and common features Initialize the set of discriminant feature cluster centers for all categories. Where: the number of categories is V, It is the discriminant feature cluster center corresponding to the y-th class. Initialize and generate the manifold's external spatial constraint range parameters. Jacobian parameters are estimated ∈ R. Initialize the generative flow model network. Where D > 2K is the dimension of the image space. Initialize the discriminator network.

[0026] Step 2, as follows Figure 2 As shown, for the image Affine coupling layer The reverse It can calculate the input y = [y1; y2] corresponding to y′, where Furthermore, α and β are obtained based on the known y1; dimensionality enhancement layer The reverse The operation involves removing the last D-2K elements of the input vector. Therefore, the inverse of the generative flow model network g is g_inverse g_out. -1 It can be represented as For each training step, a batch of training samples is randomly sampled. Calculate the corresponding reconstruction loss respectively and information bottleneck losses in: This is a common feature encoder based on the inverse of the generative flow model network g, that is, the common feature encoding corresponding to the input image x. d j (s(x)) is the j-th component of d(s(x)).

[0027] The common feature encoding mentioned above is calculated by first using the inverse g of the generative flow model network. -1 The joint encoding of the discriminative features and common features corresponding to x is obtained. Take again As The results show that t(x) and s(x) are the discriminant feature and common feature of x, respectively.

[0028] Step 3: For each training step, from the self-feature space Randomly sample a batch of feature codes And sample a batch based on the constraint range parameter ξ of the generated manifold's outer space. Based on this, calculate the isometry loss of the Jacobian matrix. Where: loss on the generating manifold Loss on the outer space of the generating manifold The Jacobian matrix J of the generated flow model network at u g The i-th column of (u) is All components except the j-th component, which has the value ∈, are equal to zero. For α i The transpose of (u), δ ij For Kronecker notation, δ when i = j ij =1 otherwise δ ij =0; β i (u; v) represents the basic flow module layer of the generated flow model network. The Jacobian matrix J at [u; v] f The i-th column of (u; v) is estimated using the same method as α. i (u), φ i (u; v) is β i The modulus of (u; υ) For J f (u; υ) The mean of the modulus length of the D-2K+1th column and thereafter.

[0029] Step 4: Calculate the overall loss function Backpropagation yields the gradients of the generating flow model network g. and the gradient of the discriminator network d Then update the two networks separately: Until the training converges.

[0030] The online phase specifically includes:

[0031] Step a, such as Figure 3 As shown, extract features from the input image: given an input image First, the discriminative features are obtained by inverse mapping of the generative flow model network g. and common characteristics The Union

[0032] Step b, Input image pre-classification judgment: When the common features s(x) of the input image x fall within the preset common feature space range [-γ, γ] K When s(x) exceeds the common feature space, its projection does not fall on the outer generating manifold, so the image is considered not to belong to any class and is rejected for classification; otherwise, the subsequent classification steps are performed.

[0033] Step c: Project the input image onto the projection of the outer manifold: Obtain the reconstructed image of x by jointly reconstructing the discriminative features t(x) and common features s(x) of the input image x. The input image x generates a manifold on the periphery induced by the generative flow model network g. Projection on;

[0034] Step d: Project the input image onto the manifold on its periphery. Further projection onto each category of submanifolds: the generative flow model network g induces C submanifolds. in: Let be the submanifold induced by the generative flow model network g corresponding to category i. For category i, take the common features s(x) of the input image x calculated in step a, and take the cluster centers of the discriminative features of category i. Then, the generative flow model network g is used to analyze the joint features. Decode and reconstruct to obtain As an external projection to submanifold of class i Projection on;

[0035] Step e: Calculate the peripheral projection To each category of submanifold distance in: for arrive The distance, i.e. arrive The geodesic distance on a manifold is determined by the constraint of the Jacobian matrix equidistant loss function and the guarantee of the embedded manifold representation theory. equal and The Euclidean distance between corresponding discriminative features, i.e. Where: t(x) is Discriminant features for The discriminative features.

[0036] Step f: Take the distance from the outer perimeter projection to each category of sub-manifold. The category c corresponding to the minimum value * The final classification result output by the method is...

[0037] Through specific practical experiments, a neural network model was trained using a single NVIDIA RTX-2080Ti GPU under the Ubuntu 18.04 operating system. The network model was written using the PyTorch framework. The hyperparameters were set as follows: common feature and discriminant feature space dimension K = 10, feature space range parameter γ = 1, manifold bounding space constraint range parameter ξ = 0.5, and Jacobian estimation parameter ∈ = 0.1. The learning rate was 0.0001, and the Adam optimizer was used for optimization, with 200,000 training steps. The experimental results on the MNIST handwritten digit recognition dataset are shown in Table 1 and... Figure 4 As shown in Table 1, this embodiment compares the classification accuracy with existing technologies on FGSM attack samples.

[0038] Table 1

[0039] FGSM attack sample classification accuracy This invention 88％ Feedforward Neural Network 57％

[0040] FGSM, short for Fast Gradient Sign Method, is a gradient-based algorithm for generating adversarial examples. Specifically, this algorithm adds noise to the original input image samples in a direction that causes the model to fail to classify them. Compared to existing techniques, Table 1 illustrates that this method is better able to resist attack samples.

[0041] like Figure 4 As shown, the generative flow model network projects the input image onto the outer generative manifold and further projects the outer projection onto each category's submanifold. It visually demonstrates how the model resists attack samples. Different rows correspond to the intensity of noise added to the original input image sample; the first row has zero intensity, corresponding to the original input image. For columns, the first column shows the image input to the generative flow model network, the second column shows the projection of this input image onto the outer generative manifold, and the third and subsequent columns show the projections of this outer projection onto each category's submanifold. It can be seen that as the intensity of noise added to the original image sample increases, the outer projection of the input image and its projection onto each category's submanifold remain stable. Therefore, the method can provide classification results with strong resistance to attack samples and is relatively robust.

[0042] Compared with existing technologies, this invention uses a generative model with a Jacobian matrix satisfying the equidistant property to project input image samples onto the outer manifold and each sample submanifold, thereby ensuring the stability of attack samples during the two-stage projection process and thus ensuring the stability of the classification results. Specifically, image features are decomposed into class-related discriminative features and class-independent common features, and the rigor of the model's projection onto input samples and the accuracy of estimating geodesic distances on the manifold based on feature encoding Euclidean distance are guaranteed. This enables the model not only to reject or accept further classification based on common features, but also to better resist attack samples based on embedded manifold representations. The classification mechanism is transparent and the results are robust, thus having significant practical application value in the field of image classification where robustness is highly required.

[0043] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. An image detection method based on embedding manifold representation and information bottleneck constraint, characterized in that, The discriminator network and the generative flow model network are initialized in the offline stage, and the reconstruction loss function, the information bottleneck loss function and the Jacobian matrix equidistance loss function are calculated in sequence to obtain the total loss function, so that the generative flow model network and the discriminator network are trained by back propagation; in the online stage, the discriminative features and the common features of the image to be tested are obtained through the trained generative flow model network, and only the common features of the image to be tested are reconstructed and submanifold image reconstructed, and the distance between the reconstructed image and the submanifold reconstructed image of each category is calculated to realize image classification. The generative flow model network comprises a dimension lifting layer and a basic flow module layer, wherein: the dimension lifting layer lifts the dimension of the joint input common feature and discriminative feature to the dimension corresponding to the image space by padding a specified number of zeros at the end of the feature, as the intermediate feature of the network; the basic flow module layer generates the corresponding reconstructed image by performing arbitrary complex nonlinear transformation on the intermediate feature on the basis of ensuring that the volume element of the Jacobian matrix of the model network has sufficient complexity; The dimension lifting layer does not contain any neural network parameters; The basic flow module layer comprises reversibly stacked reversible convolution layers and affine coupling layers, wherein: the reversible convolution layer mixes the channel features of the input features on the basis of ensuring reversibility, and the affine coupling layer performs arbitrary complex nonlinear transformation on the input features on the basis of ensuring reversibility, specifically: the first part of the output obtained by dividing along the image channel dimension is the same as the first part of the input feature, and the second part of the output feature is equal to the second part of the input feature superimposed on the feature result of the nonlinear transformation of the first part of the input feature after translation and scale expansion.

2. The image detection method based on embedding manifold representation and information bottleneck constraint according to claim 1, characterized in that, The offline stage specifically comprises: Step 1, initialization, specifically comprising: 1.1 Collect raw data and construct training dataset wherein: is the number of samples, and are the input image and its corresponding class label, respectively; 1.2 initialize the dimension of discriminative feature space and common feature space to and the range of , the joint feature space of discriminative feature and common feature ; 1.3 initialize the set of cluster centers of discriminative features for all classes where: the number of classes is , is the cluster center of discriminative features corresponding to the th class; 1.4 initialize generating manifold ambient space constraint range parameters , jacobian column estimate parameters ; 1.5 Initialize the generative flow model network where: is the dimension of the image space; 1.6 Initializing the discriminator network ; Step 2, for the image of input base stream module layer , inverse of affine coupling layer can be calculated corresponding to the input , wherein: and based on the known get; dimension lifting layer inverse of the operation of the last elements of the input vector, so that the generated stream model network inverse is ;​ For each step of training, a batch of training samples is randomly sampled , the corresponding reconstruction loss and information bottleneck loss are calculated respectively, where: is the common feature encoder based on the inverse of the generative flow model network , that is, the input image corresponds to the common feature encoding, , is the th component of ; Step 3, for each step of training, a batch of feature encodings is randomly sampled from the self-feature space and the generated manifold peripheral space constraint range parameters are sampled On this basis, the Jacobian matrix isoparametric loss is calculated Wherein: the loss on the generated manifold The loss on the generated manifold peripheral space The Jacobian matrix of the generated flow model network at The first column of the Jacobian matrix of the generated flow model network at , All components are equal to zero except the first component value , The transpose of The Kronecker symbol, when Otherwise ; , , The basic flow module layer of the generated flow model network The first column of the Jacobian matrix of the generated flow model network at The estimation method is the same as , The length of The first and the average of the column length after it;​​​​​ Step 4, compute the overall loss function , backpropagate to get the gradient of the generator flow model network and the gradient of the discriminator network , then update the two networks respectively: , ; until the training converges.

3. The image detection method based on embedding manifold representation and information bottleneck constraint according to claim 2, characterized in that, The common feature code refers to: first using the inverse of the generation flow model network Obtained The joint of the corresponding discriminative feature and common feature code , and taking As The result is obtained, wherein: And The discriminative feature and common feature of , respectively.

4. The image detection method based on embedding manifold representation and information bottleneck constraint according to claim 1, characterized in that, The online stage specifically comprises: Step a, extracting features of the input image: given an input image , first use the inverse mapping of the generative flow model network to obtain the joint of its discriminative features and common features ;​ Step b, input image pre-classification judgment: when input image falls in the preset common feature space range of common features , when it exceeds the common feature space range, its projection does not fall on the peripheral generated manifold, so it is considered that the image does not belong to any class, and the classification is rejected; otherwise, subsequent classification steps are performed; Step c, Project the input image onto the projection of the outer manifold: based on the input image Discriminant features and common characteristics The joint reconstruction was obtained Reconstructed image as input image In the generative flow model network Induced peripheral generating manifold Projection on; Step d, projection of input image on the peripheral generated manifold Further projection to each category sub-manifold: generated flow model network Induced sub-manifold wherein: is the category corresponding to the sub-manifold induced by the generated flow model network for the category , taking the common features of the input image calculated in step a and the cluster center of the discriminative features of the category , then using the generated flow model network to decode and reconstruct the joint features , obtaining as the projection of the peripheral projection to the category sub-manifold ​​ Step e, calculating the peripheral projection to the category sub-manifold of distance wherein: to distance, i.e. to the geodesic distance on the manifold, due to the constraint of the Jacobian matrix isometric loss function and the guarantee of the embedding manifold representation theory, equals and the Euclidean distance between the discriminant features corresponding to, i.e. wherein: the discriminant feature of the discriminant feature of Step f, taking the peripheral projection to the distance of each category submanifold the category corresponding to the minimum value in the middle the classification result as the final output of the method, that is .

Images