An adversarial robust model fingerprint identification method based on deep representation invariants

Abstract

This invention discloses an adversarial robust model fingerprinting method based on deep representation invariants, belonging to the field of model security technology. Addressing the problems of strong adversarial vulnerability, shallow feature dependence, single-modality limitations, insufficient theoretical guarantees, and limited architecture generalization ability in existing model fingerprinting methods, this invention constructs a multimodal fusion fingerprint by extracting geometric structural features from the intermediate representation space of the neural network based on the CKA similarity matrix, activation topological features based on persistent homology, and decision boundary manifold features based on boundary geometry. It then employs cross-architecture layer alignment and adaptive weight fusion to calculate model similarity distance and determines model theft or independence based on normalized distance and a threshold. This method achieves high-precision and high-robustness identification of model theft and plagiarism, supports comparison of models with different architectures, and provides technical support for the protection of model intellectual property rights.

Description

Technical Field

[0001] This invention belongs to the field of model security technology, and in particular relates to an adversarial robust model fingerprinting method based on deep representation invariants. Background Technology

[0002] Training deep learning models requires substantial computational resources and expertise. Model theft, which replicates the functionality of target models through techniques such as knowledge distillation and model extraction without incurring the corresponding training costs, has become a serious threat to the security of model intellectual property. Developing effective model fingerprinting technology is urgently needed to protect model intellectual property and trace model origins. Traditional model comparison methods primarily rely on distance metrics in parameter space, directly comparing model weight parameters to determine similarity. However, this method has significant limitations: models with different architectures have different parameter dimensions and cannot be directly compared; even with the same architecture, due to the high-dimensional non-convex nature of the parameter space, functionally similar models may have completely different parameter configurations.

[0003] In recent years, comparison methods based on model behavior have gained attention. For example, comparing different architecture models using LIME signatures has addressed the architecture dependency problem to some extent. However, these methods suffer from drawbacks such as vulnerability to shallow features, fragility under the local linearity assumption, and a lack of deep structural information. While existing model watermarking techniques can verify model ownership, they require embedding the watermark during the training phase and cannot be applied to already deployed models. Backdoor detection methods focus on malicious model behavior, which differs from the goal of model attribution. None of these techniques solve the problem of adversarial model fingerprinting.

[0004] Recent advances in representation learning theory have provided new perspectives for model comparison. Studies have shown that the representation space of intermediate layers in neural networks contains the core semantic information of the model and possesses certain invariance. The CKA method can measure the similarity of representation spaces and is invariant to orthogonal transformations and scaling, while the SVCCA method can analyze the evolution of neural network representations. The application of topological data analysis in machine learning opens up new directions for model fingerprinting. Persistent homology can capture the topological structure features of high-dimensional data and has theoretical robustness guarantees against local perturbations. The geometric properties of decision boundaries are also an important manifestation of the essential characteristics of the model, and their shape encodes the model's classification logic. However, these studies have not yet been systematically applied to solve the model fingerprinting problem. Existing technologies suffer from drawbacks such as strong adversarial vulnerability, shallow feature dependence, single-modality limitations, insufficient theoretical guarantees, and limited architecture generalization ability, making it difficult to reliably identify model theft and plagiarism in real-world adversarial environments. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes an adversarial robust model fingerprinting method based on deep representation invariants, thereby resolving the issues present in the prior art.

[0006] Firstly, to achieve the above objectives, this invention provides an adversarial robust model fingerprinting method based on deep representation invariants, comprising the following steps:

[0007] Extract the representation space geometric fingerprint of the model to be detected, which is based on the similarity matrix of the intermediate layer representation;

[0008] Extract the activation topological fingerprint of the model to be detected, which is based on the persistent cohomological features of neuron activation;

[0009] Extract the decision boundary manifold fingerprint of the model to be detected, the manifold fingerprint being based on the geometry of the decision boundary;

[0010] The geometric fingerprint, topological fingerprint, and manifold fingerprint are fused to form a multimodal fused fingerprint;

[0011] Based on the multimodal fused fingerprint, the similarity distance between the models is calculated;

[0012] Based on the similarity distance, the theft or independence relationship between the models is determined.

[0013] Optionally, the process of extracting a fingerprint representing spatial geometry includes:

[0014] By registering a forward propagation hook function, the activation representations of all convolutional and fully connected layers of the model on the reference dataset are extracted;

[0015] The activation representation of each layer is subjected to mean-neutralization to eliminate translation components;

[0016] For any two layers of the model, calculate the CKA similarity between the mean-centered activation representations;

[0017] Based on the CKA similarity of all layer pairs, a similarity matrix of layer number multiplied by layer number is constructed, where the diagonal elements represent self-similarity and the off-diagonal elements represent the inter-layer information flow pattern.

[0018] Optionally, the process of extracting the activation topological fingerprint includes:

[0019] For the activation representation of each layer, the sample activation is regarded as a point cloud in a high-dimensional space;

[0020] When the point cloud dimension exceeds a preset threshold, random Gaussian projection is used for dimensionality reduction to maintain the stability of topological features.

[0021] On the dimensionality-reduced point cloud, a Vietoris-Rips complex family is constructed based on a distance threshold;

[0022] The birth and death radii of topological features of different dimensions in the complex family are tracked to calculate the persistence interval;

[0023] Based on all persistent intervals, a persistent graph is constructed as the activation topological fingerprint.

[0024] Optionally, the process of extracting the fingerprint of the decision boundary manifold includes:

[0025] For each reference sample, an optimization problem is solved to generate the minimum perturbation that makes the model's prediction scores for the two classes equal, thereby obtaining points near the decision boundary;

[0026] At the boundary point, the normal vector of the decision boundary is extracted. The normal vector is defined as the gradient difference between the two closest class prediction scores at the boundary point.

[0027] At the boundary point, the local curvature of the decision boundary is calculated, and the curvature is obtained through the Hessian matrix of the model prediction score;

[0028] Collect the normal vectors and curvatures of all boundary points to form the fingerprint of the decision boundary manifold.

[0029] Optionally, the process of fusing the geometric fingerprint, topological fingerprint, and manifold fingerprint includes:

[0030] Calculate the CKA geometric distance between the geometric fingerprints;

[0031] Calculate the Wasserstein topological distance between the topological fingerprints;

[0032] Calculate the Fréchet distance between the manifold fingerprints;

[0033] The geometric distance, topological distance, and manifold distance are standardized and normalized to eliminate dimensional differences;

[0034] Meta-learning is used to optimize adaptive weights on known model pairs, and the normalized distances are weighted and fused into a multimodal fusion distance.

[0035] Optionally, the process of calculating the similarity distance between models includes:

[0036] When the number of model layers is different, a dynamic programming algorithm is used for layer alignment, with the goal of minimizing the CKA distance between layers to find the optimal layer correspondence;

[0037] The multimodal fusion distance is calculated based on the layer alignment.

[0038] The process for determining the theft or independence relationship between models includes:

[0039] The average distance between independent models on the reference model set is used as the reference distance.

[0040] Calculate the normalized distance of the model pair to be tested, wherein the normalized distance is the ratio of the multimodal fusion distance to the reference distance;

[0041] A threshold is set. If the normalized distance is less than the threshold, it is determined to be theft; otherwise, it is determined to be independent.

[0042] Secondly, the present invention also provides an adversarial robust model fingerprinting system based on deep representation invariants, used to implement an adversarial robust model fingerprinting method based on deep representation invariants, the system comprising:

[0043] A geometric fingerprint extraction module is used to extract the representation space geometric fingerprint of the model to be detected, wherein the geometric fingerprint is based on the similarity matrix of the intermediate layer representation;

[0044] The topological fingerprint extraction module is used to extract the activation topological fingerprint of the model to be detected, wherein the topological fingerprint is based on the persistent cohomological features of neuron activation;

[0045] A manifold fingerprint extraction module is used to extract the decision boundary manifold fingerprint of the model to be detected, wherein the manifold fingerprint is based on the geometry of the decision boundary;

[0046] The fingerprint fusion module is used to fuse the geometric fingerprint, topological fingerprint, and manifold fingerprint to form a multimodal fused fingerprint;

[0047] The similarity calculation module is used to calculate the similarity distance between models based on the multimodal fused fingerprint;

[0048] The relationship determination module is used to determine the theft or independence relationship between models based on the similarity distance.

[0049] Thirdly, the present invention also provides a computer terminal device, comprising:

[0050] One or more processors;

[0051] A memory, coupled to the processor, for storing one or more programs;

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the adversarial robust model fingerprinting method based on deep representation invariants in the first aspect described above.

[0053] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the adversarial robust model fingerprint recognition method based on deep representation invariants in the first aspect described above.

[0054] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the adversarial robust model fingerprint recognition method based on deep representation invariants in the first aspect described above.

[0055] Compared with the prior art, the present invention has the following advantages and technical effects:

[0056] This invention provides an adversarial robust model fingerprinting method based on deep representation invariants. This method constructs a multimodal fusion fingerprint by extracting geometric structural features from the intermediate representation space of the neural network based on the CKA similarity matrix, activation topological features based on persistent homology, and decision boundary manifold features based on boundary geometry. This achieves high-precision and high-robustness identification of model theft and plagiarism. This method effectively overcomes the shortcomings of existing technologies, such as strong adversarial vulnerability, dependence on shallow features, and limitations of single modality. It utilizes deep representation invariants to ensure fingerprint stability under adversarial attacks and supports comparison of different neural network architectures through cross-architecture layer alignment and adaptive weight fusion, expanding its practical application scope. Furthermore, this method does not require pre-embedding information during the training phase and can directly trace the origin of deployed models, providing reliable technical support for model intellectual property protection and model traceability. Attached Figure Description

[0057] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0058] Figure 1 This is an overall flowchart of the adversarial robust model fingerprint recognition method based on deep representation invariants according to an embodiment of the present invention;

[0059] Figure 2 This is a detailed flowchart of multimodal fingerprint extraction according to an embodiment of the present invention, including the extraction process of CKA geometric fingerprint, persistent homology topological fingerprint and decision boundary manifold fingerprint;

[0060] Figure 3 This is an architecture diagram of the cross-architecture model comparison and source tracing reasoning system according to an embodiment of the present invention. Detailed Implementation

[0061] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0062] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0063] Example 1

[0064] This embodiment provides an adversarial robust model fingerprinting method based on deep representation invariants, including:

[0065] Extract the representation space geometric fingerprint of the model to be detected, which is based on the similarity matrix of the intermediate layer representation;

[0066] Extract the activation topological fingerprint of the model to be detected, which is based on the persistent cohomological features of neuron activation;

[0067] Extract the decision boundary manifold fingerprint of the model to be detected, the manifold fingerprint being based on the geometry of the decision boundary;

[0068] The geometric fingerprint, topological fingerprint, and manifold fingerprint are fused to form a multimodal fused fingerprint;

[0069] Based on the multimodal fused fingerprint, the similarity distance between the models is calculated;

[0070] Based on the similarity distance, the theft or independence relationship between the models is determined.

[0071] Specifically, the steps include the following:

[0072] S1: Construct a representation space geometric fingerprint extraction module based on Centered Kernel Alignment (CKA), calculate the CKA similarity matrix of the multi-layer intermediate representation of the model to be detected, and obtain the geometric fingerprint of the information processing topology of the encoded model by analyzing the inter-layer information flow pattern.

[0073] S2: Design an activation topological fingerprint extraction module based on persistent homology. By constructing a Vietoris-Rips complex and calculating the persistence graph, the topological invariant features of neuron activation patterns are extracted, and the topological stability theorem is used to ensure adversarial robustness.

[0074] S3: Construct a decision boundary manifold fingerprint extraction module based on Fréchet distance. Through adversarial example-guided boundary sampling and local geometric feature extraction, obtain the manifold representation and curvature features of the decision boundary.

[0075] S4: Design a multimodal fingerprint fusion framework, adopt an adaptive weighting mechanism to integrate fingerprint features from three modalities: geometry, topology, and boundary. Through meta-learning, optimize the fusion weights on known stolen model pairs to achieve robust model similarity measurement.

[0076] S5: Construct layer alignment algorithms for different architecture models, and use dynamic programming to find the optimal layer correspondence under the constraint of minimizing CKA distance, thus solving the cross-architecture model comparison problem;

[0077] S6: Design a normalized distance calculation and threshold determination mechanism. Determine the determination threshold by statistically analyzing the distance distribution on the reference model set to achieve automated identification of stolen models.

[0078] S7: Construct an end-to-end model source tracing and inference system that supports single-source detection, multi-source hybrid recognition, and training method inference, achieving comprehensive protection of model intellectual property rights.

[0079] As one implementation method in this embodiment, the process of extracting a spatial geometric fingerprint includes:

[0080] By registering a forward propagation hook function, the activation representations of all convolutional and fully connected layers of the model on the reference dataset are extracted;

[0081] The activation representation of each layer is subjected to mean-neutralization to eliminate translation components;

[0082] For any two layers of the model, calculate the CKA similarity between the mean-centered activation representations;

[0083] Based on the CKA similarity of all layer pairs, a similarity matrix of layer number multiplied by layer number is constructed, where the diagonal elements represent self-similarity and the off-diagonal elements represent the inter-layer information flow pattern.

[0084] Step S1 above specifically includes the following:

[0085] S1.1: Intermediate representation extraction. For the input model f, a hook mechanism is used to register the forward propagation hook functions of all convolutional and fully connected layers, in the reference dataset. Perform forward propagation to extract the first... Layer activation representation Where n is the number of samples, Indicates the feature dimension of the corresponding layer;

[0086] S1.2: Centering process, the extracted activation matrix is ​​mean-removed:

[0087] (1)

[0088] in Since it is an all-1 vector, the centering operation eliminates the translation components of the activation, making the CKA metric more stable;

[0089] S1.3: Cross-layer CKA similarity calculation: For any two layers of the model, calculate the CKA similarity of their representation spaces.

[0090] (2)

[0091] in, For the first The activation matrix of the layer, For the first The activation matrix of the layer, The Frobenius norm is represented by the CKA metric, which is orthogonally invariant and scale invariant, and can capture the essential geometric relations of the representation space.

[0092] S1.4: Construct the CKA fingerprint matrix. For a model with L layers, construct an L×L CKA fingerprint matrix:

[0093] (3)

[0094] in This represents the output of the i-th layer (i.e., the representation of that layer). It represents the CKA similarity between the representations of the i-th and j-th layers. The diagonal elements of this matrix encode the self-similarity of each layer, while the off-diagonal elements encode the information flow patterns between layers, thus fully characterizing the topological structure of the model's representation space.

[0095] As one implementation method in this embodiment, the process of extracting the activation topological fingerprint includes:

[0096] For the activation representation of each layer, the sample activation is regarded as a point cloud in a high-dimensional space;

[0097] When the point cloud dimension exceeds a preset threshold, random Gaussian projection is used for dimensionality reduction to maintain the stability of topological features.

[0098] On the dimensionality-reduced point cloud, a Vietoris-Rips complex family is constructed based on a distance threshold;

[0099] The birth and death radii of topological features of different dimensions in the complex family are tracked to calculate the persistence interval;

[0100] Based on all persistent intervals, a persistent graph is constructed as the activation topological fingerprint.

[0101] Step S2 above specifically includes the following:

[0102] S2.1: Activate point cloud construction, for the first... Layer, representing the activation of n samples Considered A point cloud in space, where each point Representative sample The representation of this layer;

[0103] S2.2: Dimensionality Reduction. Due to the extremely high computational complexity of topological calculations in high-dimensional spaces, the Johnson-Lindenstrauss random projection theorem is used for dimensionality reduction. Dimension represents projection to Dimensional space:

[0104] (4)

[0105] in, For the sample In the The original activation vector of the layer; The projection vector after dimensionality reduction; The projection matrix is ​​a random Gaussian projection, which guarantees the distance. This ensures the stability of topological features by maintaining the original value.

[0106] S2.3: Vietoris-Rips Complex Construction. A family of Vietoris-Rips complexes is constructed on the projected point cloud, for a distance threshold. Define an r-complex:

[0107] (5)

[0108] For the Vietoris-Rips complex at the distance threshold r; The subset of sample point indices (simulacra); n is the number of samples; and Samples and In the The dimension-reduced projection vector of the layer is calculated by formula (4); r is the distance threshold parameter; This represents the Euclidean distance. As r gradually increases from 0, the topology of the complex changes, with connected components merging, and loop structures appearing and disappearing.

[0109] S2.4: Persistent cohomology computation tracks the birth and death of topological features of different dimensions. For k-dimensional topological features (k=0 represents connected components, k=1 represents cycles, and k=2 represents holes), its persistence interval is recorded:

[0110] (6)

[0111] in, This refers to Persistent Homology. For the first The activation matrix of the layer; Let be the radius of the birth of the i-th feature; Let be the radius of death for the i-th feature; The number of topological features in the k-th dimension; persistence The larger the value, the more stable and essential the topological feature.

[0112] S2.5: Persistence graph construction, which includes all persistence intervals. In a two-dimensional plane The above is represented as a set of points, forming a persistent graph, which serves as the first... Topological fingerprint of a layer:

[0113] (7)

[0114] in Indicates the first Persistent homology fingerprint of the layer; K is the highest topological dimension considered; This represents the total number of topological features across all dimensions. This fingerprint fully characterizes the [missing information - likely a specific feature or feature]. The topological structure features of layer-activated manifolds. According to the topological stability theorem, if the Hausdorff distance between two point clouds is... Then the bottleneck distance of its persistence graph does not exceed This ensures robustness to small disturbances.

[0115] As one implementation method in this embodiment, the process of extracting the decision boundary manifold fingerprint includes:

[0116] For each reference sample, an optimization problem is solved to generate the minimum perturbation that makes the model's prediction scores for the two classes equal, thereby obtaining points near the decision boundary;

[0117] At the boundary point, the normal vector of the decision boundary is extracted. The normal vector is defined as the gradient difference between the two closest class prediction scores at the boundary point.

[0118] At the boundary point, the local curvature of the decision boundary is calculated, and the curvature is obtained through the Hessian matrix of the model prediction score;

[0119] Collect the normal vectors and curvatures of all boundary points to form the fingerprint of the decision boundary manifold.

[0120] Step S3 above specifically includes the following:

[0121] S3.1: Generation of decision boundary sampling points. An adversarial example generation method is used to find points close to the decision boundary, for each reference sample x and its predicted category. Solve the optimization problem:

[0122] (8)

[0123] in This is the final boundary sampling point, located near the decision boundary; To represent the search for the perturbation that minimizes the above score difference; This represents the minimum score difference between the predicted category and the second-highest scoring category; For the model to classify perturbed samples The predicted score; For the model to classify perturbed samples The predicted score.

[0124] This optimization problem seeks to minimize the perturbation such that the model's prediction scores for the two classes are equal. Located near the decision boundary;

[0125] S3.2: Boundary normal vector extraction, at boundary points At the point where the normal vector of the decision boundary is located. Defined as:

[0126] (9)

[0127] in, To calculate the gradient (partial derivative vector) with respect to input x; i, j are the values ​​at x, y, j. The two classes with the closest scores are identified, and the direction of their normal vectors is perpendicular to the decision hyperplane. Their magnitude reflects the gradient strength at the boundary.

[0128] S3.3: Boundary curvature calculation, calculating the local curvature of the decision boundary using the Hessian matrix:

[0129] (10)

[0130] in, Boundary point Curvature at that point; The curvature value is a Hessian matrix, and it describes the degree of curvature of the decision boundary. Smooth boundaries have small curvature, while complex boundaries have large curvature.

[0131] S3.4: Boundary manifold feature aggregation, collecting the normal vector and curvature information of all boundary sampling points to construct the boundary fingerprint:

[0132] (11)

[0133] in, Fingerprint for decision boundary; This is the i-th boundary sampling point; For the i-th boundary point, there is a feature pair containing the normal vector and curvature; The fingerprint, representing the number of boundary sampling points, encodes the geometry and local curvature distribution of the decision boundary.

[0134] S3.5: Boundary distance metric, using Fréchet distance to measure the similarity between two boundary manifolds for the model. and Boundary manifold and :

[0135] (12)

[0136] in, For Fréchet distance; and These are the parameterization tables for the two boundary manifolds, For curve parameters; Fréchet distance measures the shortest "rope length" required to traverse both curves simultaneously.

[0137] As one implementation method in this embodiment, the process of fusing the geometric fingerprint, topological fingerprint, and manifold fingerprint includes:

[0138] Calculate the CKA geometric distance between the geometric fingerprints;

[0139] Calculate the Wasserstein topological distance between the topological fingerprints;

[0140] Calculate the Fréchet distance between the manifold fingerprints;

[0141] The geometric distance, topological distance, and manifold distance are standardized and normalized to eliminate dimensional differences;

[0142] Meta-learning is used to optimize adaptive weights on known model pairs, and the normalized distances are weighted and fused into a multimodal fusion distance.

[0143] Step S4 above specifically includes the following:

[0144] S4.1: Multimodal distance calculation, for the models to be compared. and Calculate the distances for the three modes respectively:

[0145] CKA geometric distance:

[0146] (13)

[0147] in, For the model and The geometric distance between them is CKA. For the model CKA fingerprint matrix; For the model CKA fingerprint matrix; This represents the Frobenius norm (the square root of the sum of the squares of all elements of a matrix).

[0148] Topological distance (Wasserstein distance):

[0149] (14)

[0150] in, Representation Model and Topological distance between them; and The models are respectively and The number of floors; This indicates that the smaller layer number should be used for alignment; and The two models are respectively the first The persistence graph of the layers; where For 2-Wasserstein distance:

[0151] (15)

[0152] in, and There are two persistence graphs (which can be viewed as probability distributions); From arrive The transmission scheme (coupling); For the set of all possible transmission schemes; This indicates that the definitive boundary is taken out in all transmission schemes; This represents the expected value of the transmission cost; For point-to-point The square of the Euclidean distance between them.

[0153] Boundary flow distance:

[0154] (16)

[0155] in, Represents the boundary manifold distance between models; The aforementioned Fréchet distance; Let be the decision boundary manifold of the model.

[0156] S4.2: Distance normalization. To eliminate the dimensional differences between different modal distances, each distance is normalized:

[0157] (17)

[0158] in, This represents the normalized distance; Represents the original distance; Modal type; This represents the mean distance of mode m to the reference model set; This represents the standard deviation of the distance of mode m on the reference model set; normalization ensures that the distances of different modes have the same dimensions and scale.

[0159] S4.3: Adaptive weight learning, using meta-learning on a known model with a dataset. The optimal fusion weights are learned. Used as the reference source model; The model to be verified; Indicate the relationship between the two: express Stealing (Positive sample pair) This indicates independent training (negative sample pairs), and the optimization objective is to maximize AUC:

[0160] (18)

[0161] in, The optimal fusion weights; These are the weighting coefficients for CKA distance, topological distance, and boundary distance, respectively. This indicates the search for the weight combination that maximizes the AUC; AUC is the Area Under Curve of the Receiver Operating Feature, which measures classification performance. This represents the fusion distance between all model pairs and label pairs in the training set; The training set is a model pair with known relationships; where the fusion distance is defined as:

[0162] (19)

[0163] in, This represents the final distance after multimodal fusion. The normalized CKA (Centered Kernel Alignment) distance is used to measure the similarity of activation patterns between model layers; The normalized persistent homology distance is used to measure the topological similarity of the activation space. The normalized decision boundary distance is used to measure the geometric similarity of the model's decision boundaries; constraints. Ensure that the fusion weights are normalized and non-negative. For fusion weighting coefficients.

[0164] As one implementation method in this embodiment, the process of calculating the similarity distance between models includes:

[0165] When the number of model layers is different, a dynamic programming algorithm is used for layer alignment, with the goal of minimizing the CKA distance between layers to find the optimal layer correspondence;

[0166] The multimodal fusion distance is calculated based on the layer alignment.

[0167] The process for determining the theft or independence relationship between models includes:

[0168] The average distance between independent models on the reference model set is used as the reference distance.

[0169] Calculate the normalized distance of the model pair to be tested, wherein the normalized distance is the ratio of the multimodal fusion distance to the reference distance;

[0170] A threshold is set. If the normalized distance is less than the threshold, it is determined to be theft; otherwise, it is determined to be independent.

[0171] Step S5 above specifically includes the following:

[0172] S5.1: Handling differences in the number of layers, for models with different numbers of layers. The Frobenius distance of the CKA matrix cannot be calculated directly; layer alignment is required first.

[0173] S5.2: Alignment at the dynamic programming layer, defining alignment as a set of mapping relationships. The optimization objective is to minimize the alignment cost:

[0174] (20)

[0175] in, A set of mapping relationships representing layer alignment; Indicates optimal alignment; Represents the Cartesian product of all possible combinations of layer pairs; Represents a layer pair, i.e., a model The Layers and Models The Layer alignment; This indicates that the layer pair is contained in the alignment set; The CKA distance between each pair of alignment layers is defined as follows: the closer the CKA distance, the higher the similarity and the lower the cost. The sparsity penalty coefficient, The number of layer pairs to be aligned.

[0176] S5.3: Distance calculation after alignment, based on optimal alignment Calculate the alignment distance:

[0177] (twenty one)

[0178] in, This indicates the aligned CKA distance; This represents the number of layer pairs contained in the optimal alignment. This indicates taking the average value; The distance between each pair of aligned layers is the CKA distance; this distance reflects the layer correspondence and representation space similarity, solving the problem of comparing different architecture models.

[0179] Step S6 includes the following:

[0180] S6.1: Reference distance statistics: Train multiple pairs of independent models (same architecture, same dataset, different random seeds) on a reference model set, and calculate the average distance between them as the reference distance. :

[0181] (twenty two)

[0182] Where K is the number of independent model pairs; and These are two independent trained models in the k-th pair; The fusion distance between the k-th pair of models is calculated according to formula (19).

[0183] S6.2: Definition of normalized distance, for the model pair to be detected Its normalized distance is:

[0184] (twenty three)

[0185] in, Represents the normalized distance; The fusion distance of the model pair to be detected is calculated according to formula (19); The aforementioned reference distance; if This indicates that the two models are more similar than the independently trained models, suggesting theft.

[0186] S6.3: Threshold determination, set the determination threshold as τ, and the determination rule is as follows:

[0187] like If it is, then it is judged as theft; if If so, it is determined to be independent; If the model is determined to be independent (a conservative strategy is used in boundary cases to prevent false positives), the false positive rate and false negative rate are balanced by adjusting τ on the validation set. The above process of determining whether a model is stolen is equivalent to determining whether it is a stolen model, and the above process of determining whether a model is independent is equivalent to determining whether it is an independent model. After determining whether a model is stolen, proceed to step S7. If the model is determined to be independent, exit the loop and end the process.

[0188] Furthermore, step S7 includes the following:

[0189] S7.1: Single-source theft detection, given a target model and candidate source model set Calculate the normalized distance and sort them. If so, it is determined to be theft and the most likely source model is returned;

[0190] S7.2: Multi-source hybrid identification. For a target model obtained by fusing multiple source models, the source models and their weights are identified by solving a linear combinatorial optimization problem.

[0191] (twenty four)

[0192] Constraints:

[0193] (25)

[0194] in, This represents the optimal mixed weight vector. This represents the contribution ratio of the k-th source model to the target model; The fingerprint features of the target model (which can be a CKA matrix, a persistence graph, or boundary features); The fingerprint features of the k-th candidate source model; Represents a linear combination of the source model fingerprints; The Frobenius norm distance between the target model fingerprint and the linear combination of the source models; constraints. This indicates that the sum of the weights is 1; constraint conditions This indicates that all weights are non-negative.

[0195] S7.3: Training method inference, based on fingerprint feature patterns, infer derived methods (distillation, fine-tuning, pruning, etc.). For example, knowledge distillation usually keeps high-level representations similar while low-level representations are significantly different, while fine-tuning does the opposite. Method recognition is achieved through feature pattern matching.

[0196] S7.4: Source tracing report generation, outputting a complete source tracing report containing information such as source model identification, similarity score, confidence estimation, and derivation method inference, providing technical support for intellectual property protection.

[0197] The present invention will now be described in further detail with reference to the accompanying drawings.

[0198] This invention relates to an adversarial robust model fingerprinting method based on deep representation invariants, see attached figure. Figure 1 The specific implementation process is as follows:

[0199] Step 1: Data preparation and model loading;

[0200] Step 2: CKA geometric fingerprint extraction;

[0201] Step 3: Persistent homology topological fingerprint extraction;

[0202] Step 4: Extraction of decision boundary manifold fingerprint;

[0203] Step 5: Align across architectural layers;

[0204] Step 6: Multimodal distance calculation and fusion;

[0205] Step 7: Normalized distance calculation and determination;

[0206] Step 8: Model source tracing reasoning.

[0207] In step 1, the specific operation process is as follows: Select a reference dataset. For the image classification task, randomly select 128 images from the test set as reference data. Load the two models to be compared. and These can be different architectures (such as ResNet20 versus ResNet50).

[0208] In step 2, refer to the appendix. Figure 2 The specific operation process is as follows:

[0209] CKA geometric fingerprint extraction: for the model Perform forward propagation on the reference data to extract the activations of all convolutional and fully connected layers. The calculation is centered for each layer, as shown in formula (1). Then, the CKA between any two layers is calculated, as shown in formula (2), to construct the CKA fingerprint matrix. Similarly, the fingerprints are extracted. The CKA fingerprint matrix.

[0210] In step 3, refer to the appendix Figure 2 The specific operation process is as follows:

[0211] For the model Activation per layer ,if Dimensionality reduction is performed using a random Gaussian projection matrix. A Vietoris-Rips complex is constructed on the projected point cloud, and a distance matrix is ​​generated. The persistent homology is calculated using the Ripser library to obtain the set of persistent intervals, as shown in formula (6). Similarly, extraction is performed... The persistence graph of each layer.

[0212] In step 4, refer to the appendix Figure 2 The specific operation process is as follows:

[0213] For the model Each sample And its predicted category, using the PGD method to solve the boundary points, as shown in Equation 8, to calculate the normal vector (Equation 9) and curvature (Equation 10) at the boundary points. Collect all The normal vectors and curvatures of each boundary point constitute the boundary fingerprint. Similarly, extract the model. Boundary fingerprint .

[0214] In step 5, refer to the appendix. Figure 3 The specific operation process is as follows:

[0215] Due to the model and model The number of layers may differ, making it impossible to directly calculate the Frobenius distance of the CKA matrix; layer alignment is required first. A dynamic programming algorithm is used, defining the state... For the model The former Layers and The minimum alignment cost for the first j is given by the state transition equation:

[0216] (26)

[0217] in, Representation Model The first i layers and the model The minimum alignment cost of the first j layers; To skip the penalty coefficient; and These are the activation matrices for the i-th and j-th layers of the two models, respectively; initial conditions. Boundary conditions , ,calculate Then, the optimal alignment set is obtained by backtracking the path. .

[0218] In step 6, the specific operation process is as shown in formula (19), which calculates three distances based on optimal alignment. , , The three distances are standardized and normalized, and then fused after normalization.

[0219] After determining that the model has been stolen, proceed to step 7, in which the specific operation process is as follows:

[0220] On the reference model set, N pairs of independent models are trained using the same architecture, the same dataset, but different random seeds, and the average fusion distance between them is calculated as the reference distance. .

[0221] Calculate the normalized distance between the pairs of models to be tested, as shown in formula (23), and set the judgment threshold as follows. The determination is made based on the normalized distance; if the normalized distance is less than... Then judge the model Stealing from the model If the normalized distance is greater than or equal to If so, then the two models are determined to be trained independently.

[0222] In step 8, refer to the appendix. Figure 3 The specific operation process is as follows:

[0223] Given target model and K candidate source models Calculate the normalized distance separately:

[0224] (27)

[0225] in, For the kth candidate source model (k=1,2,...,K); The target model. If there exists a source model whose normalized distance is less than a threshold. If so, then the source model is determined to be the most likely source of theft.

[0226] Further analysis of the inter-layer similarity pattern of the CKA fingerprint matrix: Calculate the average CKA similarity between the high-level (last 25%) and low-level (first 25%) layers. If the similarity of the high-level layers is significantly higher than that of the low-level layers (ratio > 1.5), it is inferred to be knowledge distillation; if the similarity of each layer is uniform, it is inferred to be parameter replication or complete fine-tuning.

[0227] For the multi-source mixed case, solve the linear combination optimization as shown in Equation (24) and Equation (25);

[0228] Solving using the nonnegative least squares method, we obtain... Based on the weight distribution, the source models that contribute to the target model and their mixing ratios are identified.

[0229] The above describes an embodiment of the present invention for achieving adversarial robust model fingerprinting. By applying deep invariant features such as representation space geometry, activation topology, and decision boundary manifolds to the field of model fingerprinting, the present invention achieves robust protection against adversarial attacks for the first time, providing strong technical support for applications such as model intellectual property protection and model tracing.

[0230] Example 2

[0231] In this embodiment, a computer terminal device is provided, including:

[0232] One or more processors;

[0233] A memory, coupled to the processor, for storing one or more programs;

[0234] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-described adversarial robust model fingerprinting method based on deep representation invariants.

[0235] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described adversarial robust model fingerprint recognition method based on deep representation invariants.

[0236] In this embodiment, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the above-described adversarial robust model fingerprint recognition method based on deep representation invariants.

[0237] In this embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the above-described adversarial robust model fingerprint recognition method based on deep representation invariants.

[0238] The aforementioned program can run on a processor or be stored in memory (or a computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0239] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented by different modules for different steps.

[0240] This embodiment provides such a device or system. The system, referred to as an adversarial robust model fingerprint recognition system based on deep representation invariants, includes:

[0241] A geometric fingerprint extraction module is used to extract the representation space geometric fingerprint of the model to be detected, wherein the geometric fingerprint is based on the similarity matrix of the intermediate layer representation;

[0242] The topological fingerprint extraction module is used to extract the activation topological fingerprint of the model to be detected, wherein the topological fingerprint is based on the persistent cohomological features of neuron activation;

[0243] A manifold fingerprint extraction module is used to extract the decision boundary manifold fingerprint of the model to be detected, wherein the manifold fingerprint is based on the geometry of the decision boundary;

[0244] The fingerprint fusion module is used to fuse the geometric fingerprint, topological fingerprint, and manifold fingerprint to form a multimodal fused fingerprint;

[0245] The similarity calculation module is used to calculate the similarity distance between models based on the multimodal fused fingerprint;

[0246] The relationship determination module is used to determine the theft or independence relationship between models based on the similarity distance.

[0247] As one implementation method in this embodiment, the geometric fingerprint extraction module includes:

[0248] The activation representation extraction unit is used to extract the activation representations of all convolutional and fully connected layers of the model on the reference dataset by registering a forward propagation hook function.

[0249] A centralization processing unit is used to perform mean-reduction centralization processing on the activation representation of each layer to eliminate translation components;

[0250] The similarity calculation unit is used to calculate the CKA similarity between the centered activation representations for any two layers of the model. The CKA similarity has orthogonality invariance and scale invariance.

[0251] The matrix construction unit is used to construct a layer-by-layer similarity matrix based on the CKA similarity of all layer pairs, where the diagonal elements represent self-similarity and the off-diagonal elements represent the inter-layer information flow pattern.

[0252] As one implementation method in this embodiment, the topological fingerprint extraction module includes:

[0253] Point cloud building unit, used for activation representation of each layer, treats sample activation as point cloud in high-dimensional space;

[0254] The dimensionality reduction processing unit is used to reduce the dimensionality of the point cloud by random Gaussian projection when the dimensionality exceeds a preset threshold, so as to maintain the stability of the topological features.

[0255] Complex building units are used to construct Vietoris-Rips complex families on the dimensionality-reduced point cloud based on a distance threshold;

[0256] The persistent cohomology computation unit is used to track the birth and death radii of topological features of different dimensions in the complex family and calculate the persistence interval;

[0257] The topology graph construction unit is used to construct a persistent graph as an activation topology fingerprint based on all persistent intervals.

[0258] As one implementation method in this embodiment, the manifold fingerprint extraction module includes:

[0259] Boundary point generation unit is used to solve an optimization problem for each reference sample to generate the minimum perturbation so that the model's prediction scores for the two classes are equal, thereby obtaining points near the decision boundary;

[0260] A normal vector extraction unit is used to extract the normal vector of the decision boundary at the boundary point, wherein the normal vector is defined as the gradient difference between the two closest class prediction scores at the boundary point.

[0261] A curvature calculation unit is used to calculate the local curvature of the decision boundary at the boundary point, wherein the curvature is obtained through the Hessian matrix of the model prediction score;

[0262] The fingerprint aggregation unit is used to collect the normal vectors and curvatures of all boundary points to form the fingerprint of the decision boundary manifold.

[0263] As one implementation method in this embodiment, the fingerprint fusion module includes:

[0264] A geometric distance calculation unit is used to calculate the CKA geometric distance between the geometric fingerprints;

[0265] A topological distance calculation unit is used to calculate the Wasserstein topological distance between the topological fingerprints;

[0266] A manifold distance calculation unit is used to calculate the Fréchet manifold distance between the manifold fingerprints;

[0267] The normalization processing unit is used to normalize the geometric distance, topological distance, and manifold distance to eliminate dimensional differences.

[0268] The weight fusion unit is used to optimize adaptive weights on known model pairs through meta-learning, and to fuse the normalized distances into a multimodal fusion distance.

[0269] As one implementation method in this embodiment, the similarity calculation module includes:

[0270] The layer alignment unit is used to perform layer alignment using a dynamic programming algorithm when the number of model layers is different, with the goal of finding the optimal layer correspondence by minimizing the inter-layer CKA distance;

[0271] A fusion distance calculation unit is used to calculate the multimodal fusion distance based on the layer alignment;

[0272] The relationship determination module includes:

[0273] The reference distance statistics unit is used to calculate the average distance between independent models on the reference model set as the reference distance;

[0274] A normalized distance calculation unit is used to calculate the normalized distance of the model pair to be tested, wherein the normalized distance is the ratio of the multimodal fusion distance to the reference distance;

[0275] A threshold determination unit is used to set a determination threshold. When the normalized distance is less than the threshold, it is determined to be theft; otherwise, it is determined to be independent.

[0276] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.

[0277] The above implementation method solves the problem of adversarial robust model fingerprint recognition based on deep representation invariants in related technologies, thereby ensuring that the problems existing in the prior art are resolved.

[0278] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A fingerprint recognition method based on deep representation invariants and adversarial robustness, characterized in that, Includes the following steps: Extract the representation space geometric fingerprint of the model to be detected, which is based on the similarity matrix of the intermediate layer representation; The process of extracting a spatial geometric fingerprint includes: By registering a forward propagation hook function, the activation representations of all convolutional and fully connected layers of the model on the reference dataset are extracted; The activation representation of each layer is subjected to mean-neutralization to eliminate translation components; For any two layers of the model, calculate the CKA similarity between the mean-centered activation representations; Based on the CKA similarity of all layer pairs, a similarity matrix of layer number multiplied by layer number is constructed, where the diagonal elements represent self-similarity and the off-diagonal elements represent the inter-layer information flow pattern. Extract the activation topological fingerprint of the model to be detected, which is based on the persistent cohomological features of neuron activation; The process of extracting the activation topological fingerprint includes: For the activation representation of each layer, the sample activation is regarded as a point cloud in a high-dimensional space; When the point cloud dimension exceeds a preset threshold, random Gaussian projection is used for dimensionality reduction to maintain the stability of topological features. On the dimensionality-reduced point cloud, a Vietoris-Rips complex family is constructed based on a distance threshold; The birth and death radii of topological features of different dimensions in the complex family are tracked to calculate the persistence interval; Based on all persistent intervals, a persistent graph is constructed as the activation topological fingerprint; Extract the decision boundary manifold fingerprint of the model to be detected, the manifold fingerprint being based on the geometry of the decision boundary; The process of extracting the fingerprint of the decision boundary manifold includes: For each reference sample, an optimization problem is solved to generate the minimum perturbation that makes the model's prediction scores for the two classes equal, thereby obtaining points near the decision boundary; At the boundary point, the normal vector of the decision boundary is extracted. The normal vector is defined as the gradient difference between the two closest class prediction scores at the boundary point. At the boundary point, the local curvature of the decision boundary is calculated, and the curvature is obtained through the Hessian matrix of the model prediction score; Collect the normal vectors and curvatures of all boundary points to form the fingerprint of the decision boundary manifold; The geometric fingerprint, topological fingerprint, and manifold fingerprint are fused to form a multimodal fused fingerprint; Based on the multimodal fused fingerprint, the similarity distance between the models is calculated; Based on the similarity distance, the theft or independence relationship between the models is determined.

2. The method according to claim 1, characterized in that, The process of fusing the geometric fingerprint, topological fingerprint, and manifold fingerprint includes: Calculate the CKA geometric distance between the geometric fingerprints; Calculate the Wasserstein topological distance between the topological fingerprints; Calculate the Fréchet distance between the manifold fingerprints; The geometric distance, topological distance, and manifold distance are standardized and normalized to eliminate dimensional differences; Meta-learning is used to optimize adaptive weights on known model pairs, and the normalized distances are weighted and fused into a multimodal fusion distance.

3. The method according to claim 2, characterized in that, The process of calculating the similarity distance between models includes: When the number of model layers is different, a dynamic programming algorithm is used for layer alignment, with the goal of minimizing the CKA distance between layers to find the optimal layer correspondence; The multimodal fusion distance is calculated based on the layer alignment. The process for determining the theft or independence relationship between models includes: The average distance between independent models on the reference model set is used as the reference distance. Calculate the normalized distance of the model pair to be tested, wherein the normalized distance is the ratio of the multimodal fusion distance to the reference distance; A threshold is set. If the normalized distance is less than the threshold, it is determined to be theft; otherwise, it is determined to be independent.

4. A fingerprint recognition system based on a deep representation invariant adversarial robust model, characterized in that, The system for implementing the method according to any one of claims 1-3 comprises: A geometric fingerprint extraction module is used to extract the representation space geometric fingerprint of the model to be detected, wherein the geometric fingerprint is based on the similarity matrix of the intermediate layer representation; The topological fingerprint extraction module is used to extract the activation topological fingerprint of the model to be detected, wherein the topological fingerprint is based on the persistent cohomological features of neuron activation; A manifold fingerprint extraction module is used to extract the decision boundary manifold fingerprint of the model to be detected, wherein the manifold fingerprint is based on the geometry of the decision boundary; The fingerprint fusion module is used to fuse the geometric fingerprint, topological fingerprint, and manifold fingerprint to form a multimodal fused fingerprint; The similarity calculation module is used to calculate the similarity distance between models based on the multimodal fused fingerprint; The relationship determination module is used to determine the theft or independence relationship between models based on the similarity distance.

5. A computer terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the steps of the method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-3.

Images