A method for extracting parameters from deep neural networks with constrained architectural knowledge conditions

Abstract

This invention discloses a method for extracting parameters from deep neural networks under constrained architectural knowledge conditions. Its key feature is that before extracting parameters from the current hidden layer, it analyzes the algebraic consistency between the perturbation basis matrix and the output difference vector in the input space to accurately identify whether there are any missing neurons in the previous layer. When a missing neuron is detected, a re-extraction mechanism is triggered, increasing the sampling density of key points and re-performing parameter recovery of the missing layer. Compared with existing technologies, this invention achieves efficient and accurate extraction of DNN parameters under constrained architectural knowledge conditions through an adaptive extraction workflow and neuron omission detection. Its error detection recall rate reaches 100%, and its error detection specificity remains stable between 1.91% and 2.03%. This provides a robust and efficient technical solution for model vulnerability assessment, black-box model parameter extraction attack prevention, and neural network intellectual property protection in the field of artificial intelligence security, and has broad application prospects and commercial value.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security and cryptographic analysis technology, and in particular to a method for extracting parameters from deep neural networks under constrained architectural knowledge conditions. Background Technology

[0002] Deep neural networks (DNNs) have become the core engine of modern artificial intelligence systems, achieving breakthroughs in fields such as image recognition, natural language processing, and autonomous driving. DNN parameters are typically obtained through large-scale training, a process that takes months and is extremely costly, thus giving these parameters significant intellectual property value. Meanwhile, DNNs often provide services via black-box interfaces (oracles), allowing users to submit inputs and obtain outputs, which provides opportunities for model parameter extraction attacks. In recent years, researchers have proposed various black-box model extraction attack methods based on cryptanalysis. Carlini et al. first revealed the structural similarity between DNNs and block ciphers in 2020 and proposed a neuronal signature recovery method based on differential cryptanalysis. Subsequently, Canales-Martinez et al. proposed the Neuron Wiggle technique, solving the exponential time problem of symbol recovery. Liu et al. further addressed the rank deficiency and keypoint interference problems in deep networks, achieving end-to-end multinomial time extraction of an eight-layer ReLU network.

[0003] However, the aforementioned existing methods generally assume that the attacker has complete knowledge of the target network's architecture, including the number of hidden layers and the number of neurons per layer. This assumption often fails in real-world attack scenarios, severely limiting the applicability of existing methods. Under limited architectural knowledge, existing attack methods suffer from the following technical problems:

[0004] 1) Neuron omission problem: Due to the lack of prior information on the layer dimension, the attack algorithm cannot determine whether all neurons in the current layer have been fully recovered, and it is easy to miss neurons due to insufficient sampling density.

[0005] 2) Error propagation and cascade failure: Missing neurons can cause algebraic inconsistencies in the parameter recovery of subsequent layers, resulting in dimensional defects in the calculated transformation matrix. This leads to the complete failure of signature recovery and symbol recovery, and there is no early warning mechanism for this failure mode in the existing extraction process.

[0006] 3) Lack of self-checking and recovery capabilities: Existing methods cannot detect whether there are missing neurons, nor can they locate the specific layer where the missing neurons occur, and they do not have the ability to automatically re-extract and recover from errors.

[0007] In summary, existing deep neural network parameter extraction methods fail completely under limited architectural knowledge due to the lack of prior information on layer widths, leading to missed neurons, error propagation, and a lack of self-checking capabilities. Therefore, achieving complete, accurate, and efficient extraction of DNN parameters under limited architectural knowledge is a pressing technical problem in the field of artificial intelligence security. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for deep neural network parameter extraction under constrained architectural knowledge conditions. It employs an adaptive extraction workflow, integrating a neuron omission detection algorithm based on algebraic consistency verification at the front end of layer-by-layer parameter extraction, and optimizing the termination judgment mechanism in the algorithm. This constructs a post-quantum-safe model extraction method that can completely and accurately recover the target network parameters even when the number and dimension of the hidden layers are unknown. While maintaining polynomial time and polynomial query complexity, it significantly improves the robustness and completeness of the extraction process under constrained architectural knowledge conditions. This method performs exceptionally well in scenarios such as black-box model parameter extraction attacks, neural network intellectual property protection, and model security auditing, significantly improving the robustness and practicality of deep neural network model extraction attacks under constrained architectural knowledge conditions. It is particularly suitable for situations where the number and dimension of the hidden layers of the target deep neural network (DNNs) are unknown, and it improves upon previous deep neural network parameter extraction methods. Under the condition of constrained architectural knowledge, an efficient and accurate extraction of DNN parameters is achieved through an adaptive extraction workflow and a neuron omission detection algorithm. This provides a robust and efficient technical solution for model vulnerability assessment, black-box model parameter extraction attack prevention, and neural network intellectual property protection in the field of artificial intelligence security, and has broad application prospects and commercial value.

[0009] The objective of this invention is achieved as follows: a method for extracting parameters from deep neural networks under constrained architectural knowledge conditions, characterized in that the method specifically includes:

[0010] (a) The mechanism for detecting and restoring the integrity of the previous layer of neurons, including the following steps:

[0011] a. Before extracting parameters for the current hidden layer, omission detection is first performed on the neuron recovery results of the previous hidden layer. The omission detection determines whether there are any unrecovered omission neurons in the previous hidden layer by analyzing the algebraic consistency between the perturbation basis matrix and the output difference vector in the input space.

[0012] b. If an omission is detected in step a above, the re-recovery process is triggered, the key point sampling density is increased, and the signature recovery and symbol recovery of the previous hidden layer are re-executed.

[0013] (ii) Layer-by-layer termination judgment mechanism

[0014] a. After the parameters of each hidden layer are extracted, determine whether to continue extracting the next layer by judging whether all collected key points have been successfully clustered into the recovered neurons.

[0015] b. If there are unclustered key points, it is determined that the current hidden layer has not been fully recovered, and signature recovery is continued to discover new neurons;

[0016] c. If all key points have been clustered and no new key points can be clustered, then the current layer is determined to be the output layer, and the parameter restoration of the output layer is performed instead.

[0017] The integrity detection of the previous layer of neurons further includes the following steps:

[0018] Step 1: Let the current layer to be extracted be the th layer. Layer, in which The previous layer The parameters have been partially recovered using signature recovery and symbol recovery algorithms;

[0019] Step 2: Select several key input points as input samples. These input samples are then processed by the previous... Propagating layer by layer forward, for any The first in the layer one neuron If its output value before ReLU activation satisfies If the input sample is identified as a key point of a recovered neuron, it is excluded to ensure that the selected input sample does not belong to the previous layer. Key points of neurons that have been restored;

[0020] Step 3: For any selected input sample Perform the following detection sub-steps:

[0021] Step 3.1: Input sample Several small perturbation vectors are randomly selected from the input space. Each of them The modulus length satisfies , Let be a pre-defined, extremely small positive number, such that and Located within the same linear neighborhood; number of perturbation directions The value must be greater than the input sample. Propagation forward to the 1st The number of neurons in an activated state at each layer, i.e. The number of elements is used to ensure that the constructed perturbation basis matrix has sufficient rank for neuron parameter recovery;

[0022] Step 3.2: Input Sample and its perturbation samples Submit to the black-box interface respectively Obtain the corresponding output difference vector. The output difference vector The calculation formula is as follows:

[0023] .

[0024] in The output function of the target deep neural network; For the first One perturbation vector;

[0025] Step 3.3: Utilize the restored previous layer Parameters, for each perturbation direction Spread to the Layer, let the output after propagation be The direction of the disturbance In the The perturbation basis matrix in the layer input space is: Calculate the rank of the perturbation basis matrix. And calculate the augmented matrix. rank ;

[0026] Step 3.4: If a certain disturbance direction exists Make Then determine the current input sample The corresponding detection result is the previous layer. There are missed neurons, and this result is included in the statistics. ;

[0027] Step 4: Select a threshold If the statistic Then determine the previous layer. Restore complete; if the statistic Then determine the previous layer. There are missing neurons, the threshold The value range is 5%.

[0028] ~ The specific value is determined based on the complexity and quality of the target network.

[0029] Step 5: When it is determined that there are missing neurons in the previous layer, the re-extraction process is triggered.

[0030] The re-extraction process further includes the following steps:

[0031] Step 1: Increase the keypoint sampling density to collect an additional number of keypoints;

[0032] Step 2: Using the additional collected key points, re-execute the previous layer. Neuron parameter recovery;

[0033] Step 3: Repeat the integrity detection step of the previous layer of neurons until the previous layer is determined. layer Only after restoring to full can the current step be continued. Layer parameter extraction.

[0034] The step-by-step termination determination further includes the following steps:

[0035] Step 1: After extracting the parameters of the current hidden layer, collect the set of all acquired key points. ;

[0036] Step 2: Determine the set of key points Have all key points been successfully located to the recovered neurons?

[0037] Step 3: If there are unlocated key points, it is determined that there are still unknown hidden layers that have not been fully recovered, and parameter recovery is continued to discover new neurons; if all key points have been located, the current layer is determined to be the output layer.

[0038] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0039] 1) By introducing a front-layer neuron omission detection mechanism based on algebraic consistency verification, an adaptive re-extraction process, and a dynamic termination judgment mechanism based on critical point clustering in the deep neural network parameter extraction process, the optimized parameter extraction method can achieve complete functional equivalence extraction for the first time under the condition of constrained architectural knowledge, while maintaining the polynomial time and polynomial query complexity of the original cryptanalysis method. The neuron omission detection recall rate reaches 100%, and the detection specificity is stable between 1.91% and 2.03%, effectively avoiding the error propagation and cascade failure problem caused by neuron omission.

[0040] 2) It supports flexible adjustment of the detection threshold (5% to 10%) based on the complexity and quality of the target network. It successfully extracted a ReLU network with four hidden layers and a total of about 1.1 million parameters on the CIFAR-10 dataset. It provides a robust and efficient technical solution for model vulnerability assessment, black box model parameter extraction attack prevention, and neural network intellectual property protection in the field of artificial intelligence security. It has broad application prospects and commercial value. Attached Figure Description

[0041] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0042] This invention proposes an adaptive parameter extraction workflow. Under conditions of unknown network depth and layer dimensions, it recovers neuron parameters layer by layer and designs a novel neuron omission detection method. By analyzing the algebraic consistency between the perturbation basis matrix and the output difference vector in the input space, it accurately identifies whether there are missing neurons in the previous layer. When an omission is detected, a re-extraction mechanism is triggered, increasing the sampling density of key points and re-executing the parameter recovery of the missing layer. This invention was experimentally validated on the CIFAR-10 dataset, successfully extracting a ReLU network with four hidden layers and a total of approximately 1.1 million parameters. The false detection recall rate reached 100%, and the false detection specificity remained stable between 1.91% and 2.03%. The overall extraction process maintained polynomial time and polynomial query complexity, significantly improving the robustness and practicality of deep neural network model extraction attacks under conditions of limited architectural knowledge.

[0043] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of methods consistent with some aspects of this specification as detailed in the appended claims, and will fully describe the implementation process of the technical solution of the present invention. It should be noted that the improved method described above will be specifically implemented in the neuron omission detection, re-extraction process, and layer termination determination in the layer-by-layer parameter recovery step of this section. The described embodiments are used to fully illustrate the technical details of the present invention, but the scope of protection of the present invention is not limited thereto.

[0044] Example 1

[0045] This embodiment extracts parameters from a four-hidden-layer fully connected ReLU network trained on the CIFAR-10 dataset. The target network architecture is 3072-512×3-64-10.

[0046] See Figure 1 The figure illustrates the overall process of neuron omission detection, re-extraction, and layer termination judgment as described in this invention, which optimizes the extraction of parameters from existing neural networks.

[0047] The system parameters used in this embodiment are defined as follows:

[0048] 1) Select the perturbation step size for Secondary perturbation step size for ;

[0049] 2) Set the detection threshold It is 10%;

[0050] 3) Set the initial sampling number of key points to: ,in The total number of neurons in the target network is 1600.

[0051] 4) The parameters for the first hidden layer have been restored. The restored parameters are as follows: weight matrix. The dimension is 511×3072, and the bias vector is... The dimension is 512; a total of 511 neuron signatures were recovered. The first layer should actually have 512 neurons, but one neuron's weight parameter is missing. Due to the lack of prior architectural information, existing methods cannot detect this missing neuron and will directly proceed to parameter extraction in the second layer.

[0052] In this embodiment, taking the recovery of neuron parameters in the second hidden layer as an example, the specific steps for neuron omission detection are as follows:

[0053] Step 1: Set the current layer to be extracted as the first layer. Layer, in which The previous layer The parameters have been partially recovered using signature recovery and symbol recovery algorithms, but there is actually a missing neuron.

[0054] Step 2: Select 1000 input keypoints as input samples, and propagate forward layer by layer through the first layer. For any... The first in the layer one neuron If its output value before ReLU activation If the input sample is identified as a key point of a recovered neuron, it is excluded. After screening, the valid input samples are retained.

[0055] Step 3: For any selected input sample Perform the following detection sub-steps:

[0056] Step 3.1: Input sample In the input space, pass Propagating forward layer by layer, the calculation yields the [number]th [layer]. Number of active neurons in the layer Therefore, m is randomly selected. A small perturbation vector matrix Each of them The modulus length satisfies , .

[0057] Step 3.2: Input Sample and its perturbation samples Submit to the black-box interface respectively The corresponding output difference vector is obtained using the formula. .

[0058] Step 3.3: Using the recovered first-layer parameters, adjust each perturbation direction Propagate to the first layer, and denote the output after propagation as... Then the perturbation basis matrix is Calculate the rank of the perturbation basis matrix. And calculate the augmented matrix. rank .

[0059] Step 3.4: After calculation, if Determine the current input sample The corresponding detection result is that there are missing neurons in the first layer, otherwise there are no missing neurons, and this result is included in the statistics. .

[0060] Repeat the above test for all valid input samples and statistically determine the proportion of samples that were missed. .

[0061] Step 4: Select the detection threshold 10%. If the statistic... If a neuron is found to be missing in the first layer, the re-extraction process is triggered; otherwise, if no neuron is found to be missing in the first layer, the neuron parameter extraction continues.

[0062] In this example, when it is determined that there are missing neurons in the previous layer, the specific steps of the re-extraction process are as follows:

[0063] Step 1: Increase the keypoint sampling density by increasing the number of keypoint samples. A key point for an additional number.

[0064] Step 2: Using the additional collected key points, re-perform the neuron parameter recovery (signature recovery and symbol recovery) of the first layer, skipping the extracted neurons.

[0065] Step 3: Repeat the detection steps described in claim 2 and recalculate the statistic. .like If the first layer is determined to be completely restored, the parameter extraction of the second layer can continue; otherwise, the re-extraction process will be executed again.

[0066] In this example, after completing the first... After the layer parameters are restored, the specific steps for terminating the judgment layer by layer are as follows:

[0067] Step 1: In the first After the parameters of the hidden layer are extracted, collect the set of all acquired key points. .

[0068] Step 2: Determine the set of key points by propagating the key points layer by layer forward. Have all key points been successfully located to the recovered neurons?

[0069] Step 3: If there are no unlocalized keypoints, determine that the current layer is the output layer; otherwise, determine that the current layer is the hidden layer and execute the neuron parameter recovery process.

[0070] When the current layer is determined to be an output layer, the existing output layer parameter recovery algorithm is executed to recover the output layer weight matrix and bias vector, thus completing the recovery of the entire model parameters.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, various modifications, equivalent substitutions, or improvements can be made to the above embodiments without departing from the technical concept of the present invention, and all such modifications, substitutions, or improvements should fall within the scope of protection of the present invention.

Claims

1. A method for extracting parameters from deep neural networks under constrained architectural knowledge conditions, characterized in that, The method specifically includes: (a) Detection and recovery of the integrity of the previous layer of neurons a. Before extracting parameters for the current hidden layer, omission detection is performed on the neuron recovery results of the previous hidden layer. The omission detection determines whether there are any unrecovered omission neurons in the previous hidden layer by analyzing the algebraic consistency between the perturbation basis matrix and the output difference vector in the input space. b. If an omission is detected, the re-recovery process is triggered, the key point sampling density is increased, and the signature recovery and symbol recovery of the previous hidden layer are re-executed; (ii) Termination judgment at each level a. After the parameters of each hidden layer are extracted, determine whether to continue extracting the next layer by judging whether all collected key points have been successfully clustered into the recovered neurons. b. If there are unclustered key points, it is determined that the current hidden layer has not been fully recovered, and signature recovery is continued to discover new neurons; c. If all key points have been clustered and no new key points can be clustered, then the current layer is determined to be the output layer, and the parameter restoration of the output layer is performed instead.

2. The method for extracting parameters of a deep neural network under constrained architectural knowledge conditions according to claim 1, characterized in that, The integrity detection of the previous layer of neurons includes the following steps: Step 1: Let the current layer to be extracted be the th layer. Layer, in which The previous layer The parameters have been partially recovered using signature recovery and symbol recovery algorithms; Step 2: Select several key input points as input samples The input sample Before Propagating layer by layer forward, for any The first in the layer one neuron If its output value before ReLU activation satisfy Then the input sample is determined. Key points of recovered neurons are excluded to ensure the selected input samples are correct. Not belonging to the previous layer Key points of neurons that have been restored; Step 3: For any selected input sample Perform the following detection sub-steps: Step 3.1: Input sample Several small perturbation vectors are randomly selected from the input space. Each of them The modulus length satisfies , Let be a pre-defined, extremely small positive number, such that and Located within the same linear neighborhood; Step 3.2: Input Sample and its perturbation samples Submit to the black-box interface respectively Obtain the corresponding output difference vector. The output difference vector Calculated by the following formula: ； in The output function of the target deep neural network; For the first One perturbation vector; Step 3.3: Utilize the restored previous layer Parameters, for each perturbation direction Spread to the Layer, let the output after propagation be The direction of the disturbance In the The perturbation basis matrix in the layer input space is Calculate the rank of the perturbation basis matrix. And calculate the augmented matrix. rank ; Step 3.4: If a certain disturbance direction exists Make Then determine the current input sample The corresponding detection result is the previous layer. There are missed neurons, and this result is included in the statistics. ; Step 4: Select a threshold If the statistic Then determine the previous layer. Restore complete; if the statistic Then determine the previous layer. There are missing neurons, the threshold The value range is 5%~ The specific value is determined based on the complexity and quality of the target network; Step 5: When it is determined that there are missing neurons in the previous layer, the re-extraction process is triggered.

3. The method for extracting parameters from deep neural networks under constrained architectural knowledge conditions according to claim 1, characterized in that, The step-by-step termination judgment specifically includes: Step 1: After extracting the parameters of the current hidden layer, collect the set of all acquired key points. ; Step 2: Determine the set of key points Have all key points been successfully located to the recovered neurons? Step 3: If there are unlocated key points, it is determined that there are still unknown hidden layers that have not been fully recovered, and parameter recovery is continued to discover new neurons; if all key points have been located, the current layer is determined to be the output layer.

4. The method for extracting parameters of a deep neural network under constrained architectural knowledge conditions according to claim 2, characterized in that, The re-extraction process in step 5 specifically includes: Step 1: Increase the keypoint sampling density to collect an additional number of keypoints; Step 2: Using the additional collected key points, re-execute the previous layer. Neuron parameter recovery; Step 3: Repeat the integrity detection step of the previous layer of neurons until the previous layer is determined. Only after restoring to full can the current step be continued. Layer parameter extraction.

Images