Hard label black-box model stealing method based on semantic saliency

Abstract

The application discloses a hard label black box model stealing method based on semantic saliency, and through a three-stage cooperative mechanism of "static semantic division + saliency guided detection + information gain weighted refinement", blind pixel-level disturbance is improved to high-explanation semantic block-level detection within a limited query budget, and decision boundary information is efficiently triggered. Meanwhile, a consistency gating mechanism is introduced to realize high-value sample reinforcement learning and unreliable sample robust error correction, and the cloning accuracy and behavior consistency of a substitute model to a victim model are significantly improved.

Description

Technical Field

[0001] This invention relates to the technical field of artificial intelligence and model security assessment, and in particular to a method for stealing black-box models based on semantic saliency hard-labeled models. Background Technology

[0002] The rapid iteration of deep learning technology has driven breakthrough applications of artificial intelligence models in fields such as computer vision and natural language processing. However, training high-precision, highly generalizable deep neural network models requires massive amounts of high-quality labeled data, significant computing resources, and a long training period. These mature deep learning models have become core digital assets.

[0003] To achieve commercial monetization and lower the barrier to entry for users, model holders generally adopt the "Machine Learning as a Service" (MLaaS) cloud deployment model, encapsulating the model on a cloud server and providing inference services to the outside world through APIs. This black-box deployment model not only improves service convenience but also physically isolates users from access to the model's internal structure and parameters.

[0004] Existing security research confirms that even with strict access control, black-box APIs still face a serious threat of model theft attacks. Attackers construct targeted query samples and collect API output prediction results to build proxy datasets, and then train structurally similar and functionally equivalent alternative models locally. These high-fidelity alternative models not only directly cause economic losses to the model holders, but can also serve as springboards for white-box attacks.

[0005] To address this threat, current mainstream MLaaS platforms employ a hard-label defense strategy, where the API only returns the most probable discrete category index or name, hiding the specific confidence value. This strategy significantly increases the difficulty of model theft, rendering traditional theft algorithms based on gradient approximation and fine-grained knowledge distillation largely ineffective in hard-label scenarios.

[0006] Existing hard-label model stealing methods are mainly divided into three categories: data-driven, data-free generation, and hybrid active detection. Data-driven methods rely on public proxy datasets. If the semantic distribution of the data deviates significantly from the target task, the query efficiency is extremely low. Data-free generation methods rely on generative models to synthesize samples. Under hard label constraints, they are prone to pattern collapse problems. Hybrid active detection methods combine a small amount of prior data with active perturbation strategies. However, random pixel-level perturbations can destroy the global semantic structure of the image, causing the detected samples to deviate from the natural image manifold and failing to perceive semantically salient regions. A large amount of query budget is wasted on background regions with no decision-making value.

[0007] In addition, there is a training coupling noise problem in the early stage of hard label theft: the fitting ability of the substitute model is weak in the initial stage, and the sample feature extraction and salient region estimation have large deviations. If all the probe samples and corresponding hard labels are directly included in the training, the erroneous samples will continue to interfere with the training process, resulting in slow model convergence and low cloning accuracy.

[0008] In summary, developing a hard-labeled black-box model stealing method that combines semantic structure awareness and training bias calibration capabilities has significant practical application value and technical necessity. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a hard-label black-box model theft method based on semantic saliency. Through a three-stage collaborative mechanism of "static semantic partitioning + saliency-guided detection + information gain weighted refinement", it improves blind pixel-level perturbation into highly interpretable semantic block-level detection within a limited query budget, and efficiently triggers decision boundary information. At the same time, it introduces a consistency gating mechanism to realize reinforcement learning of high-value samples and robust error correction of unreliable samples, which significantly improves the cloning accuracy and behavioral consistency of the substitute model to the victim model.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] A method for stealing hard-labeled black-box models based on semantic saliency, including the following steps:

[0012] S1. Obtain the original image dataset to be queried for the target visual classification task and complete the preprocessing. Initialize the network structure and parameters of the alternative model that approximates the behavior of the victim model and set the maximum query budget for interacting with the victim model.

[0013] S2. Perform static semantic space partitioning on the preprocessed original image to be queried, and use the superpixel segmentation algorithm to divide the pixel space of the image into multiple semantic superpixel blocks with local consistency to obtain a semantic superpixel set.

[0014] S3. Calculate the class activation saliency map of the preprocessed original image to be queried on the alternative model of the current iteration to obtain pixel-level saliency weights, and spatially aggregate the pixel-level saliency weights within each semantic superpixel block of the semantic superpixel set to obtain the saliency score of each semantic superpixel block. Select the semantic superpixel block with the highest saliency score as the target semantic block.

[0015] S4. Perform information erasure operation on the image spatial region corresponding to the target semantic block, and use the normalized channel mean of the preprocessed original image to fill the erased region with pixels, thereby generating the corresponding probe sample.

[0016] S5. Send the preprocessed original image to be queried and the probe sample as query input in pairs to the victim model, and receive the hard label output results returned by the victim model for the original image and the probe sample;

[0017] S6. Based on the consistency between the predicted category of the alternative model on the preprocessed original image to be queried and the hard label of the original image returned by the victim model, a consistency gating indicator is constructed. This indicator is used to divide the samples into reliable samples and unreliable samples. For reliable samples, a two-sample weighted loss function is constructed based on the output distribution divergence and label jump. For unreliable samples, a single-sample error correction loss function is constructed.

[0018] S7. Under the constraint of query budget, combine the above loss functions to iteratively update the parameters of the alternative model until the preset stopping condition is met, and obtain an alternative model that approximates the behavior of the victim model.

[0019] Furthermore, step S1 specifically includes the following steps:

[0020] S1.1 Obtain the original image dataset to be queried for the target visual classification task;

[0021] S1.2. Perform standardization preprocessing on each image in the dataset, including scaling the images to a fixed resolution size required by the alternative model, and performing Z-score normalization using pre-statistical mean and standard deviation to map discrete pixel values ​​into a continuous standardized vector space, which is beneficial for feature extraction by the alternative model.

[0022] S1.3 Initialize the alternative model, using a deep neural network architecture pre-trained on a large-scale dataset as the backbone network, loading pre-trained weights and adjusting the output dimension of the classification head to match the number of victim model categories, while setting the maximum query budget. .

[0023] 3. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 2, wherein the original image dataset to be queried includes publicly available natural image datasets, historical business image data related to the semantics of the target task, or semantic prior image data generated by a generative model.

[0024] Furthermore, step S2 specifically includes the following steps:

[0025] S2.1 Perform static semantic space partitioning on the preprocessed original image to be queried;

[0026] S2.2. The Quickshift superpixel segmentation algorithm is used to convert the image from the RGB color space to the LAB color space, and the kernel density of each pixel is estimated within its set neighborhood window in the joint feature space containing color features and spatial coordinates.

[0027] S2.3 Connect each pixel to its nearest neighbor pixel with higher density along the direction of the increasing density gradient to construct a spanning tree;

[0028] S2.4 When the feature space distance between two pixels exceeds the preset maximum distance threshold, the connection edge between them is cut off, and multiple independent connected components are formed. Each connected component constitutes a semantic superpixel block, and all semantic superpixel blocks together form a semantic superpixel set covering the entire image.

[0029] Furthermore, step S3 specifically includes the following steps:

[0030] S3.1. The Grad-CAM algorithm is used to input the original image into the current iteration of the alternative model for forward propagation, and the predicted probability scores of each category output by the alternative model are obtained.

[0031] S3.2 Select the category with the highest predicted probability score as the target category, and calculate the gradient of the target category score relative to the output feature map of the last target convolutional layer of the alternative model; perform global average pooling on the obtained gradient in the spatial dimension to obtain the neuron importance weight of each feature map channel;

[0032] S3.3. Perform a linear weighted sum of the neuron importance weights and the corresponding target convolutional layer output feature maps, and filter out negative values ​​using the ReLU activation function;

[0033] S3.4 The bilinear interpolation algorithm is used to upsample the weighted summed feature map to the same resolution size as the original image, resulting in a pixel-level class activation saliency map;

[0034] S3.5. Traverse each semantic superpixel block in the semantic superpixel set, extract the saliency values ​​of all pixels in the block on the class activation saliency map, and calculate the average value as the saliency score. Sort the blocks in descending order of the scores and select the semantic superpixel block with the highest score as the target semantic block that best represents the core semantics of the image.

[0035] Furthermore, step S4 specifically includes the following steps:

[0036] S4.1 Perform information erasure operation on the image spatial region corresponding to the target semantic block; replace simple pixel zeroing with a dataset normalized mean filling strategy to keep the probe sample within the natural image data distribution range; specifically, in the preprocessed original image dataset to be queried, calculate the global mean of all image pixels according to the three color channels of RGB in advance to obtain a three-dimensional pixel mean vector; replace the values ​​of all pixels in the original image that belong to the target semantic block range with the three-dimensional pixel mean vector;

[0037] S4.2. When performing erasure and fill operations, a boundary smoothing transition mechanism is introduced. The proportion of the area of ​​the target semantic block to the total area of ​​the original image is calculated. When this proportion exceeds the preset area proportion threshold, it indicates that the coverage of the target semantic block is too large. Directly erasing the entire block will cause serious image semantic damage and distribution mutation. At this time, partition filling is performed on the pixels within the target semantic block. A transition area of ​​a certain pixel width is retained at the edge of the target semantic block. Only the core area inside the transition area is filled with a three-dimensional pixel mean vector. The pixel values ​​in the transition area are smoothly transitioned by linear interpolation of the original pixel values ​​and the mean vector, thereby generating a detection sample with high concealment and effective detection capability.

[0038] Furthermore, step S5 specifically includes the following steps:

[0039] S5.1 Combine the preprocessed original image to be queried and the generated probe sample into a query pair, and send it to the victim model through API interface or network communication protocol;

[0040] S5.2 The victim model performs inference on the input original image and probe sample respectively, and only returns the category label with the highest predicted probability, i.e., the hard label output; at the same time, a query log recording module is established to record the detailed information of the current query each time a query is initiated to the victim model, including the original image identifier, probe sample identifier, query timestamp, the hard label returned by the victim model, and the current remaining query budget. The query log is used for subsequent sample weight calculation and model training process monitoring.

[0041] Furthermore, step S6 specifically includes the following steps:

[0042] S6.1, Define the original image as The detection sample is The victim model is The alternative model is The hard labels of the original images returned by the victim model are denoted as The hard label of the probe sample is recorded as The output probability distribution of the alternative model for the original image is as follows: Its prediction category is ;

[0043] S6.2. Based on the consistency between the predicted category of the alternative model on the preprocessed original image to be queried and the hard label of the original image returned by the victim model, a binarized consistency gating indicator is constructed. :

[0044] When predicting category This indicates that the alternative model has already fitted the basic decision-making logic of the victim model on the current sample, and the consistency gating indicator is then applied. Set it to 1, and classify the sample as a reliable sample;

[0045] When predicting category When this occurs, it indicates that the alternative model's predictions on the current samples are severely biased, and the consistency gating indicator is adjusted. Set it to 0, and classify the sample as unreliable;

[0046] S6.3 For reliable samples, calculate dynamic sample weights based on the output distribution divergence of the alternative model on the original image and the probe sample, as well as the victim model label jump. :

[0047] First, calculate the Kullback-Leibler divergence between the output probability distributions of the surrogate model on the original image and the probe samples, denoted as . :

[0048] ;

[0049] in, The total number of categories for the target classification task. and These represent the alternative model's predictions for the original image and the probe sample, respectively. The probability of a class Indexed by category;

[0050] Subsequently, dynamic sample weights are constructed. The calculation formula is as follows:

[0051] ;

[0052] in, The exponential function is used to amplify divergence differences; This is an indicator function that takes a value of 1 when the victim model's hard labels for the original image and the probe sample are inconsistent, and 0 otherwise; This is the preset label transition amplification factor;

[0053] Based on dynamic sample weights Construct a two-sample weighted loss function for reliable samples. :

[0054] ;

[0055] in, The standard cross-entropy loss function;

[0056] 6.4 For unreliable samples, since the alternative model has not yet correctly fitted its underlying semantics, the perturbations introduced by the probe samples may cause the optimization direction to deviate. Therefore, dynamically adjustable fixed error correction weights are used. Constructing a single-sample error correction loss function Basic behavior correction is performed using only the supervision signal from the original image:

[0057] ;

[0058] Among them, fixed error correction weights The value of is dynamically adjusted according to the training stage of the alternative model. A larger value is taken in the early stage of training to accelerate the fitting of basic features, and a smaller value is taken in the middle and later stages to avoid interfering with the fine characterization of the decision boundary.

[0059] Furthermore, step S7 specifically includes the following steps:

[0060] S7.1 Under the query budget constraint, combine the two-sample weighted loss function with the single-sample error correction loss function to construct the overall objective function for all samples in the current training batch. :

[0061] ;

[0062] in, This represents the total number of samples in the current training batch. For the first in the batch One original image; This is a sample reliability indicator function, when the sample... When judged as a reliable sample =1, otherwise =0; It is a two-sample weighted loss function; For single-sample error correction loss function;

[0063] Using the overall objective function Gradient calculation is performed on the network parameters of the alternative model;

[0064] S7.2 The optimizer iteratively updates the replacement model parameters; the optimizer selected is either a stochastic gradient descent optimizer with momentum or an adaptive moment estimation optimizer; throughout the iterative training process, a learning rate decay strategy based on the proportion of query budget consumption is introduced:

[0065] Set the initial learning rate to The maximum query budget is The current number of queries consumed is The learning rate of the current iteration step Dynamically adjust according to the cosine annealing strategy;

[0066] S7.3 After each parameter update, check whether the preset stopping conditions are met;

[0067] When the preset stopping condition is met, the iterative stealing process is terminated, the finally trained alternative model is output, and the alternative model is deployed to the local computing device to perform the same visual classification task as the victim model, thereby realizing the localized transfer of knowledge from the victim model.

[0068] Furthermore, the stopping conditions include any of the following:

[0069] Current number of queries consumed Reaching maximum query budget ;

[0070] The alternative model achieves a preset performance threshold in terms of prediction accuracy or behavioral consistency with the victim model on a pre-defined independent validation set.

[0071] Alternative models in continuously set Within each training epoch, the performance improvement on the validation set was consistently below the preset minimum increment threshold. .

[0072] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0073] 1. Significantly improved the efficiency of information utilization in hard-label scenarios.

[0074] This invention abandons the traditional random pixel-level perturbation strategy and innovatively introduces a static semantic space partitioning mechanism. By using the Quickshift algorithm to map a high-dimensional pixel grid into a set of locally consistent semantic superpixels, the probe space is elevated from the disordered pixel level to the structured semantic block level. This mechanism significantly filters out high-frequency noise and background regions that do not contribute to model decisions, ensuring that each erasing operation accurately targets the core semantic components of the image. This maximizes the acquisition of effective information about the victim model's decision boundaries within a limited query budget.

[0075] 2. Enhanced the stability and sample concealment of decision boundary detection.

[0076] This invention combines the internal feature responses of alternative models, utilizes the Grad-CAM algorithm to extract class activation saliency maps, and spatially aggregates them with static semantic superpixels to achieve semantically aware saliency detection. By precisely locating and erasing the highest-scoring semantic blocks, it can trigger a jump in the victim model's output label with a very high probability, thereby exposing vulnerable boundary regions. Simultaneously, by employing dataset normalized mean padding and boundary smoothing transition mechanisms, it ensures that the generated detection samples visually conform to the data distribution of natural images, effectively circumventing the cloud platform's defense mechanisms based on abnormal input detection.

[0077] 3. Effectively suppresses the accumulation of early training noise, ensuring the stability of model convergence.

[0078] To address the error amplification problem that easily occurs during hard label theft, this invention designs an information gain weighted refinement mechanism based on consistency gating. By comparing the synergy between the substitution model's predictions and the victim's hard labels, samples are strictly divided into reliable and unreliable categories. For reliable samples, dynamic weights are constructed using KL divergence and label jump terms to enhance the substitution model's learning of high-value boundary samples; for unreliable samples, misleading gradients of the probe samples are cut off, retaining only the supervision signal from the original image for basic error correction. This asymmetric optimization strategy completely breaks the vicious cycle of erroneous features in iteration, enabling the substitution model to converge smoothly and quickly to the parameter space of the victim model in the highly challenging hard label environment. Attached Figure Description

[0079] Figure 1 This is a flowchart of the method of the present invention.

[0080] Figure 2 This is a schematic diagram illustrating the principle of static semantic space partitioning and semantic superpixel set generation in this invention.

[0081] Figure 3 This is a schematic diagram of the semantic saliency detection and detection sample generation process in this invention.

[0082] Figure 4 This is a schematic diagram of the information gain weighted refining mechanism based on consistency gating in this invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0084] The experimental environment configuration for this embodiment is as follows: The software environment uses Python 3.8 or above and PyTorch 1.10.0 or above deep learning framework. Relying on the framework's automatic differentiation and modular construction capabilities, it enables rapid iterative training of alternative models; the hardware configuration is an Intel(R) Xeon(R) Gold 6240 @ 2.60GHz CPU, 128GB DDR4 memory, dual NVIDIA GeForce RTX 4090 (24GB VRAM) GPUs, and the operating system is Ubuntu 20.04 LTS, ensuring stable and efficient model computation and cloud interaction.

[0085] like Figures 1 to 4 As shown, this embodiment provides a method for stealing hard-labeled black-box models based on semantic saliency. The entire stealing process can be roughly divided into four core stages: The first stage is data preparation and model initialization, which is mainly responsible for acquiring unlabeled query data and establishing a standardized input stream; the second stage is static semantic segmentation and saliency detection, which is mainly responsible for performing structured segmentation on the input image and locating core semantic regions to generate probe samples; the third stage is black-box query and consistency gating, which is mainly responsible for interacting with the victim model in the cloud to obtain hard labels and classifying samples as reliable or unreliable based on consistency gating; the fourth stage is dynamic weighted refinement and parameter update, which is mainly responsible for constructing differentiated loss functions based on the classification and optimizing alternative models. The specific steps are as follows:

[0086] S1. Obtain the original image dataset to be queried for the target visual classification task and complete preprocessing; initialize the network structure and parameters of the alternative model that approximates the victim model's behavior; and set the maximum query budget for interacting with the victim model. Specifically, this includes the following steps:

[0087] S1.1 Obtain the original image dataset to be queried for the target visual classification task, construct a multi-source unlabeled query pool, and adapt to the attack conditions of no original training data in the real black box scenario; the original image dataset to be queried includes publicly available natural image datasets, historical business image data related to the semantics of the target task, or semantic prior image data generated by generative models.

[0088] S1.2. Perform standardization preprocessing on each image in the dataset, including scaling the images to a fixed resolution size required by the alternative model, and performing Z-score normalization using pre-statistical mean and standard deviation to map discrete pixel values ​​into a continuous standardized vector space, which is beneficial for feature extraction by the alternative model.

[0089] S1.3 Initialize the alternative model, using a deep neural network architecture pre-trained on a large-scale dataset as the backbone network to improve knowledge transfer efficiency. Load the pre-trained weights and adjust the output dimension of the classification head to match the number of victim model categories, while setting the maximum query budget. This simulates the cost and frequency limitations of API calls in real-world scenarios.

[0090] S2. Perform static semantic space partitioning on the preprocessed original image to be queried. Use the Quickshift superpixel segmentation algorithm to divide the pixel space of the image into multiple semantic superpixel blocks with local consistency, obtaining a semantic superpixel set; specifically including the following steps:

[0091] S2.1 Perform static semantic space partitioning on the preprocessed original image to be queried;

[0092] S2.2. The Quickshift superpixel segmentation algorithm is used to convert the image from the RGB color space to the LAB color space that conforms to visual perception. In the five-dimensional joint feature space containing L, A, B color features and spatial coordinates, the kernel density of each pixel is estimated within its set neighborhood window using a Gaussian kernel.

[0093] S2.3 Connect each pixel to its nearest neighbor pixel with higher density along the direction of the increasing density gradient to build a spanning tree;

[0094] S2.4 When the feature space distance between two pixels exceeds a preset maximum distance threshold, the connection edge between them is severed, and the image is segmented into connected components with consistent color and texture. This results in multiple independent connected components, each constituting a semantic superpixel block. All semantic superpixel blocks together form a semantic superpixel set covering the entire image, denoted as [S2.4]. ; where, set Each element in Indicates the first A semantic superpixel block, This is the index number of the superpixel block.

[0095] S3. Calculate the class activation saliency map (Grad-CAM) of the preprocessed original image to be queried on the alternative model of the current iteration to obtain pixel-level saliency weights. Then, spatially aggregate these pixel-level saliency weights within each semantic superpixel block of the semantic superpixel set to obtain the saliency score of each semantic superpixel block. Select the semantic superpixel block with the highest saliency score as the target semantic block. Specifically, this includes the following steps:

[0096] S3.1. The Grad-CAM algorithm is used to input the original image into the current iteration of the alternative model for forward propagation, and the predicted probability scores of each category output by the alternative model are obtained.

[0097] S3.2 Select the category with the highest predicted probability score as the target category, and calculate the gradient of the target category score relative to the output feature map of the last target convolutional layer of the alternative model through automatic differentiation; perform global average pooling operation on the obtained gradient in the spatial dimension to obtain the neuron importance weight of each feature map channel;

[0098] S3.3. Perform a linear weighted sum of the neuron importance weights and the corresponding target convolutional layer output feature maps, and filter out negative values ​​using the ReLU activation function;

[0099] S3.4 uses bilinear interpolation to upsample the weighted summed feature map to the same resolution as the original image, obtaining a pixel-level class activation saliency map, denoted as... ,in The value at each pixel position represents the degree of contribution of that pixel to the prediction result of the surrogate model;

[0100] S3.5. Traverse each semantic superpixel block in the semantic superpixel set, extract the saliency values ​​of all pixels within the block on the class activation saliency map, and calculate the average as the saliency score. Indicates the first semantic superpixel blocks The saliency score is used to quantify the semantic superpixel block. The blocks are sorted in descending order of score, and the top-ranked semantic superpixel block is selected as the target semantic block representing the core semantics of the image and influencing the model's decision-making. This block is denoted as [missing information]. .

[0101] S4. Perform information erasure operation on the image spatial region corresponding to the target semantic block, and fill the erased region with pixels using the normalized channel mean of the preprocessed original image to be queried, thereby generating the corresponding probe sample; specifically including the following steps:

[0102] S4.1 Perform information erasure operation on the image spatial region corresponding to the target semantic block; to avoid the detection sample deviating from the natural image distribution and being intercepted by the victim model's anomaly detection mechanism due to pixel zeroing, a dataset normalized mean filling strategy is adopted instead of simple pixel zeroing to keep the detection sample within the natural image data distribution range; specifically, in the preprocessed original image dataset to be queried, the global mean of all image pixels is pre-calculated according to the three color channels of RGB to obtain a three-dimensional pixel mean vector; the values ​​of all pixels in the original image that belong to the target semantic block range are replaced with the three-dimensional pixel mean vector to avoid abnormal color blocks caused by simple zeroing and reduce the risk of being intercepted by the victim model's anomaly detection.

[0103] S4.2. When performing erasure and fill operations, a boundary smoothing transition mechanism is introduced. The proportion of the area of ​​the target semantic block to the total area of ​​the original image is calculated. When this proportion exceeds the preset area proportion threshold, it indicates that the coverage of the target semantic block is too large. Directly erasing the entire block will cause serious image semantic damage and distribution mutation. At this time, partition filling is performed on the pixels within the target semantic block. A transition area of ​​a certain pixel width is retained at the edge of the target semantic block. Only the core area inside the transition area is filled with a three-dimensional pixel mean vector. The pixel values ​​in the transition area are smoothly transitioned by linear interpolation of the original pixel values ​​and the mean vector. This generates a detection sample with high concealment and effective detection capability, taking into account both core semantic masking and sample naturalness.

[0104] S5. Send the preprocessed original image to be queried and the probe sample as query input to the victim model in pairs. It also receives hard-labeled output results from the victim model for the original image and the probe sample; specifically, it includes the following steps:

[0105] S5.1 Combine the preprocessed original image to be queried and the generated probe sample into a query pair, and send it to the victim model through the API interface or HTTP / HTTPS network communication protocol;

[0106] S5.2 The victim model performs inference on the input original image and probe sample respectively, and only returns the category label with the highest predicted probability, i.e., the hard label output; at the same time, a query log recording module is established to record the detailed information of the current query each time a query is initiated to the victim model, including the original image identifier, probe sample identifier, query timestamp, the hard label returned by the victim model, and the current remaining query budget. The query log is used for subsequent sample weight calculation and model training process monitoring.

[0107] S6. Based on the consistency between the predicted category of the substitution model on the preprocessed original image to be queried and the hard label of the original image returned by the victim model, a consistency gating indicator is constructed. This indicator is used to divide the samples into reliable samples and unreliable samples. For reliable samples, a two-sample weighted loss function is constructed based on the output distribution divergence and label jump. For unreliable samples, a single-sample error correction loss function is constructed. The specific steps include:

[0108] S6.1, Define the original image as The detection sample is The victim model is The alternative model is The hard labels of the original images returned by the victim model are denoted as The hard label of the probe sample is recorded as To simplify the following description, the victim model's hard labels for the original images will be abbreviated as... ,Right now The hard labels of the victim model on the probe samples are abbreviated as ,Right now The output probability distribution of the alternative model for the original image is as follows: Its predicted category is the index of the item with the highest probability, denoted as . ;

[0109] S6.2. Based on the consistency between the predicted category of the alternative model on the preprocessed original image to be queried and the hard label of the original image returned by the victim model, a binarized consistency gating indicator is constructed. :

[0110] When predicting category This indicates that the alternative model has already well fitted the basic decision-making logic of the victim model on the current sample, and the consistency gating indicator is then applied. Set it to 1, and classify the sample as a reliable sample;

[0111] When predicting category When this occurs, it indicates that the alternative model's predictions on the current samples are severely biased, and the consistency gating indicator is adjusted. Set it to 0, and classify the sample as unreliable;

[0112] S6.3 For reliable samples, their probe samples can provide high-quality decision boundary detection information. Therefore, dynamic sample weights are calculated based on the output distribution divergence of the alternative model on the original image and the probe samples, as well as the victim model label jump situation. :

[0113] First, calculate the Kullback-Leibler divergence (KL divergence) between the output probability distributions of the surrogate model on the original image and the probe samples, denoted as... The larger the KL divergence, the more drastic the impact of erasing the target semantic block on the output distribution of the replacement model, and the closer the sample is to the decision boundary.

[0114] ;

[0115] in, The total number of categories for the target classification task. and These represent the alternative model's predictions for the original image and the probe sample, respectively. The probability of a class Indexed by category;

[0116] Subsequently, dynamic sample weights are constructed. The calculation formula is as follows:

[0117] ;

[0118] in, The exponential function is used to amplify divergence differences; This is an indicator function that takes a value of 1 when the victim model's hard labels for the original image and the probe sample are inconsistent, and 0 otherwise; This is the preset label transition amplification factor;

[0119] Based on dynamic sample weights Construct a two-sample weighted loss function for reliable samples. To enhance the learning of high-value detection information:

[0120] ;

[0121] in, The standard cross-entropy loss function; The distribution shift of the surrogate model before and after the detection perturbation was measured, capturing the extent to which the erasure operation affected the internal representation of the surrogate model. This explicitly amplifies and rewards high-value samples that are located near the victim's decision boundary and successfully trigger label transitions;

[0122] 6.4 For unreliable samples, since the alternative model has not yet correctly fitted its basic semantics, forcibly introducing perturbation information from the probe samples may cause a serious deviation in the gradient optimization direction, exacerbating noise accumulation in the early stages of training. Therefore, for unreliable samples, the feedback from the probe samples is directly discarded, and dynamically adjustable fixed error correction weights are used. Constructing a single-sample error correction loss function Basic behavior correction is performed using only the supervision signal from the original image:

[0123] ;

[0124] Among them, fixed error correction weights The value is dynamically adjusted based on the training phase of the alternative model; in the early stages of training... Taking a larger value accelerates the fitting of basic features, forcing the model to quickly fit the basic features of the victim model; in the mid-to-late stages. The error correction signal is gradually reduced to a smaller value to avoid coarse error correction signals interfering with the model's fine characterization of the decision boundary.

[0125] S7. Under the constraint of query budget, iteratively update the parameters of the alternative model by combining the above loss functions until the preset stopping condition is met, thereby obtaining an alternative model that approximates the behavior of the victim model; specifically including the following steps:

[0126] S7.1 Under the query budget constraint, combine the two-sample weighted loss function with the single-sample error correction loss function to construct the overall objective function for all samples in the current training batch. :

[0127] ;

[0128] in, This represents the total number of samples in the current training batch. For the first in the batch One original image; This is a sample reliability indicator function, when the sample... When judged as a reliable sample =1, otherwise =0; It is a two-sample weighted loss function; For single-sample error correction loss function;

[0129] Using the overall objective function Gradient calculation is performed on the network parameters of the alternative model;

[0130] S7.2 The optimizer iteratively updates the replacement model parameters; this embodiment selects a stochastic gradient descent optimizer with momentum or an adaptive moment estimation optimizer; throughout the iterative training process, in order to ensure that the model converges smoothly within a finite budget, a learning rate decay strategy based on the proportion of query budget consumption is introduced:

[0131] Set the initial learning rate to The maximum query budget is The current number of queries consumed is The learning rate of the current iteration step Dynamically adjust according to the cosine annealing strategy;

[0132] S7.3 After each parameter update, the system will check whether the preset stopping conditions are met; the stopping conditions include any one of the following:

[0133] Current number of queries consumed Reaching maximum query budget ;

[0134] The alternative model achieves a preset performance threshold in terms of prediction accuracy or behavioral consistency with the victim model on a pre-defined independent validation set.

[0135] Alternative models in continuously set Within each training round ( (The preset number of early stopping patience rounds is used to determine whether the model has converged.) The performance improvement on the validation set was consistently below the preset minimum increment threshold. ( (A very small positive number, used to measure whether the performance improvement is statistically significant), indicating that the model has hit a convergence bottleneck;

[0136] When any stopping condition is met, the system automatically terminates the iterative theft process, outputs the finally trained alternative model, and deploys the alternative model to the local computing device for performing security quantification assessment or functional equivalence verification on the target model, thereby realizing the localized transfer and replication of the victim's model knowledge and capabilities.

[0137] Extensive experimental evaluations were conducted on multiple standard test sets using the trained alternative models. This embodiment selected the general CIFAR-10 dataset and the fine-grained CUB-200-2011 bird dataset for comparative experiments. The victim model used a ResNet-50 architecture, while the alternative model used a ResNet-34 architecture, with the query budget uniformly limited to 100,000 queries. Baseline methods used for comparison included representative hard-label stealing methods such as Random Selection, ActiveThief, Knockoff Nets, BBD, and AOT.

[0138] Experimental results show that the proposed semantic saliency-based hard-label black-box model stealing method significantly outperforms existing methods on all datasets. Particularly on the CUB-200 dataset, a fine-grained dataset with extremely complex class boundaries, the proposed method significantly improves cloning accuracy and behavioral consistency compared to suboptimal methods. This verifies that the "static semantic partitioning and saliency detection" mechanism in this invention can more effectively capture fine-grained features, while the "consistency gating and weighted refinement" mechanism effectively suppresses training noise under complex boundaries, ensuring efficient model convergence.

[0139] Further ablation experiments verified the effectiveness of each core module of the invention. On the CIFAR-10 dataset, the introduction of static semantic partitioning alone significantly improved the cloning accuracy, indicating that the structured probe space is superior to the unordered pixel space; the accuracy was further improved after superimposing saliency-guided probe, proving the importance of locating core semantic regions; and the consistency gating and weighted refinement mechanism introduced in the complete scheme again brought significant performance improvements, demonstrating the importance of suppressing early training noise in hard label theft.

[0140] Furthermore, this invention exhibits strong robustness against cloud-based defenses. Traditional theft methods suffer significant performance degradation when faced with perturbations in the output probability of the victim model and misleading label defenses. However, this invention, through global semantic detection and gating error correction mechanisms, controls the performance degradation of the alternative model to within 3%, ensuring stable performance. It can serve as both an efficient theft method and a standardized tool, quantifying the usability of defense strategies and the balance between theft prevention and control.

[0141] Even in scenarios with extremely scarce data, this invention remains feasible in engineering: when there are no real query samples, semantically relevant samples can be generated using a text-generated graph model as a supplement, relying on semantic block detection and weighted refinement to amplify effective boundary information, and noise can be generated through consistency gating. Experiments have shown that under pure generated sample conditions, the consistency of model behavior still reaches more than 78%, which is better than traditional data-free methods.

[0142] In summary, this invention achieves high-precision model cloning within a limited query budget through semantic awareness detection and information gain weighted refinement. It has the advantages of clear architecture, module decoupling, and strong robustness, and can be applied to AI model security assessment, risk quantification, and defense scheme design.

[0143] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A method for stealing hard-labeled black-box models based on semantic saliency, characterized in that, Including steps, S1. Obtain the original image dataset to be queried for the target visual classification task and complete the preprocessing. Initialize the network structure and parameters of the alternative model that approximates the behavior of the victim model and set the maximum query budget for interacting with the victim model. S2. Perform static semantic space partitioning on the preprocessed original image to be queried, and use the superpixel segmentation algorithm to divide the pixel space of the image into multiple semantic superpixel blocks with local consistency to obtain a semantic superpixel set. S3. Calculate the class activation saliency map of the preprocessed original image to be queried on the alternative model of the current iteration to obtain pixel-level saliency weights, and spatially aggregate the pixel-level saliency weights within each semantic superpixel block of the semantic superpixel set to obtain the saliency score of each semantic superpixel block. Select the semantic superpixel block with the highest saliency score as the target semantic block. S4. Perform information erasure operation on the image spatial region corresponding to the target semantic block, and use the normalized channel mean of the preprocessed original image to fill the erased region with pixels, thereby generating the corresponding probe sample. S5. Send the preprocessed original image to be queried and the probe sample as query input in pairs to the victim model, and receive the hard label output results returned by the victim model for the original image and the probe sample; S6. Based on the consistency between the predicted category of the alternative model on the preprocessed original image to be queried and the hard label of the original image returned by the victim model, a consistency gating indicator is constructed. This indicator is used to divide the samples into reliable samples and unreliable samples. For reliable samples, a two-sample weighted loss function is constructed based on the output distribution divergence and label jump. For unreliable samples, a single-sample error correction loss function is constructed. S7. Under the constraint of query budget, combine the above loss functions to iteratively update the parameters of the alternative model until the preset stopping condition is met, and obtain an alternative model that approximates the behavior of the victim model.

2. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 1, characterized in that, Step S1 specifically includes the following steps: S1.1 Obtain the original image dataset to be queried for the target visual classification task; S1.

2. Perform standardization preprocessing on each image in the dataset, including scaling the images to a fixed resolution size required by the alternative model, and performing Z-score normalization using pre-statistical mean and standard deviation to map discrete pixel values ​​into a continuous standardized vector space, which is beneficial for feature extraction by the alternative model. S1.3 Initialize the alternative model, using a deep neural network architecture pre-trained on a large-scale dataset as the backbone network, loading pre-trained weights and adjusting the output dimension of the classification head to match the number of victim model categories, while setting the maximum query budget. .

3. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 2, characterized in that, The original image dataset to be queried includes publicly available natural image datasets, historical business image data related to the semantics of the target task, or semantic prior image data generated by generative models.

4. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 1, characterized in that, Step S2 specifically includes the following steps. S2.1 Perform static semantic space partitioning on the preprocessed original image to be queried; S2.

2. The Quickshift superpixel segmentation algorithm is used to convert the image from the RGB color space to the LAB color space, and the kernel density of each pixel is estimated within its set neighborhood window in the joint feature space containing color features and spatial coordinates. S2.3 Connect each pixel to its nearest neighbor pixel with higher density along the direction of the increasing density gradient to build a spanning tree; S2.4 When the feature space distance between two pixels exceeds the preset maximum distance threshold, the connection edge between them is cut off, and multiple independent connected components are formed. Each connected component constitutes a semantic superpixel block, and all semantic superpixel blocks together form a semantic superpixel set covering the entire image.

5. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 1, characterized in that, Step S3 specifically includes the following steps. S3.

1. The Grad-CAM algorithm is used to input the original image into the current iteration of the alternative model for forward propagation, and the predicted probability scores of each category output by the alternative model are obtained. S3.2 Select the category with the highest predicted probability score as the target category, and calculate the gradient of the target category score relative to the output feature map of the last target convolutional layer of the alternative model; perform global average pooling on the obtained gradient in the spatial dimension to obtain the neuron importance weight of each feature map channel; S3.

3. Perform a linear weighted sum of the neuron importance weights and the corresponding target convolutional layer output feature maps, and filter out negative values ​​using the ReLU activation function; S3.4 The bilinear interpolation algorithm is used to upsample the weighted summed feature map to the same resolution size as the original image, resulting in a pixel-level class activation saliency map; S3.

5. Traverse each semantic superpixel block in the semantic superpixel set, extract the saliency values ​​of all pixels in the block on the class activation saliency map, and calculate the average value as the saliency score. Sort the blocks in descending order of the scores and select the semantic superpixel block with the highest score as the target semantic block that best represents the core semantics of the image.

6. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 1, characterized in that, Step S4 specifically includes the following steps. S4.1 Perform information erasure operation on the image spatial region corresponding to the target semantic block; replace simple pixel zeroing with a dataset normalized mean filling strategy to keep the probe sample within the natural image data distribution range; specifically, in the preprocessed original image dataset to be queried, calculate the global mean of all image pixels according to the three color channels of RGB in advance to obtain a three-dimensional pixel mean vector; replace the values ​​of all pixels in the original image that belong to the target semantic block range with the three-dimensional pixel mean vector; S4.

2. When performing erasure and fill operations, a boundary smoothing transition mechanism is introduced. The proportion of the area of ​​the target semantic block to the total area of ​​the original image is calculated. When this proportion exceeds the preset area proportion threshold, it indicates that the coverage of the target semantic block is too large. Directly erasing the entire block will cause serious image semantic damage and distribution mutation. At this time, partition filling is performed on the pixels within the target semantic block. A transition area of ​​a certain pixel width is retained at the edge of the target semantic block. Only the core area inside the transition area is filled with a three-dimensional pixel mean vector. The pixel values ​​in the transition area are smoothly transitioned by linear interpolation of the original pixel values ​​and the mean vector, thereby generating a detection sample with high concealment and effective detection capability.

7. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 1, characterized in that, Step S5 specifically includes the following steps. S5.1 Combine the preprocessed original image to be queried and the generated probe sample into a query pair, and send it to the victim model through API interface or network communication protocol; S5.2 The victim model performs inference on the input original image and probe sample respectively, and only returns the category label with the highest predicted probability, i.e., the hard label output; at the same time, a query log recording module is established to record the detailed information of the current query each time a query is initiated to the victim model, including the original image identifier, probe sample identifier, query timestamp, the hard label returned by the victim model, and the current remaining query budget. The query log is used for subsequent sample weight calculation and model training process monitoring.

8. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 1, characterized in that, Step S6 specifically includes the following steps. S6.1, Define the original image as The detection sample is The victim model is The alternative model is The hard labels of the original images returned by the victim model are denoted as The hard label of the probe sample is recorded as The output probability distribution of the alternative model for the original image is as follows: Its prediction category is ; S6.

2. Based on the consistency between the predicted category of the alternative model on the preprocessed original image to be queried and the hard label of the original image returned by the victim model, a binarized consistency gating indicator is constructed. : When predicting category This indicates that the alternative model has already fitted the basic decision-making logic of the victim model on the current sample, and the consistency gating indicator is then applied. Set it to 1, and classify the sample as a reliable sample; When predicting category When this occurs, it indicates that the alternative model's predictions on the current samples are severely biased, and the consistency gating indicator is adjusted. Set it to 0, and classify the sample as unreliable; S6.3 For reliable samples, calculate dynamic sample weights based on the output distribution divergence of the alternative model on the original image and the probe sample, as well as the victim model label jump. : First, calculate the Kullback-Leibler divergence between the output probability distributions of the surrogate model on the original image and the probe samples, denoted as . : ； in, The total number of categories for the target classification task. and These represent the alternative model's predictions for the original image and the probe sample, respectively. The probability of a class Indexed by category; Subsequently, dynamic sample weights are constructed. The calculation formula is as follows: ； in, The exponential function is used to amplify divergence differences; This is an indicator function that takes a value of 1 when the victim model's hard labels for the original image and the probe sample are inconsistent, and 0 otherwise; This is the preset label transition amplification factor; Based on dynamic sample weights Construct a two-sample weighted loss function for reliable samples. : ； in, The standard cross-entropy loss function; 6.4 For unreliable samples, since the alternative model has not yet correctly fitted its underlying semantics, the perturbations introduced by the probe samples may cause the optimization direction to deviate. Therefore, dynamically adjustable fixed error correction weights are used. Constructing a single-sample error correction loss function Basic behavior correction is performed using only the supervision signal from the original image: ； Among them, fixed error correction weights The value of is dynamically adjusted according to the training stage of the alternative model. A larger value is taken in the early stage of training to accelerate the fitting of basic features, and a smaller value is taken in the middle and later stages to avoid interfering with the fine characterization of the decision boundary.

9. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 1, characterized in that, Step S7 specifically includes the following steps. S7.1 Under the query budget constraint, combine the two-sample weighted loss function with the single-sample error correction loss function to construct the overall objective function for all samples in the current training batch. : ； in, This represents the total number of samples in the current training batch. For the first in the batch One original image; This is a sample reliability indicator function, when the sample... When judged as a reliable sample =1, otherwise =0; It is a two-sample weighted loss function; For single-sample error correction loss function; Using the overall objective function Gradient calculation is performed on the network parameters of the alternative model; S7.2 The optimizer iteratively updates the replacement model parameters; the optimizer selected is either a stochastic gradient descent optimizer with momentum or an adaptive moment estimation optimizer; throughout the iterative training process, a learning rate decay strategy based on the proportion of query budget consumption is introduced: Set the initial learning rate to The maximum query budget is The current number of queries consumed is The learning rate of the current iteration step Dynamically adjust according to the cosine annealing strategy; S7.3 After each parameter update, check whether the preset stopping conditions are met; When the preset stopping condition is met, the iterative stealing process is terminated, the finally trained alternative model is output, and the alternative model is deployed to the local computing device to perform the same visual classification task as the victim model, thereby realizing the localized transfer of knowledge from the victim model.

10. The method for stealing hard-labeled black-box models based on semantic saliency according to claim 9, characterized in that, The stopping conditions include any of the following: Current number of queries consumed Reaching maximum query budget ; The alternative model achieves a preset performance threshold in terms of prediction accuracy or behavioral consistency with the victim model on a pre-defined independent validation set. Alternative models in continuously set Within each training epoch, the performance improvement on the validation set was consistently below the preset minimum increment threshold. .

Images