Cloud model security protection method and system based on query traceability analysis

Abstract

The invention discloses a cloud model security protection method and system based on query traceability analysis, and belongs to the field of machine learning system security. The method comprises the following steps: respectively constructing query traceability graphs for a normal query sequence and an attack query sequence, and training a graph classifier based on the query traceability graphs; initializing the to-be-detected query traceability graph based on a plurality of normal queries; wherein the initial nodes are not connected; adding queries in the to-be-detected query sequence into the to-be-detected query traceability graph, and based on the similarity between the queries and nodes in the to-be-detected query traceability graph, constructing corresponding edges and setting edge weights; and after all queries in the to-be-detected query sequence are added into the to-be-detected query traceability graph, obtaining a detection result of the to-be-detected query sequence in combination with the graph classifier. According to the method, the accuracy and robustness of detection can be improved, and the false alarm rate and the system delay are remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of machine learning system security, and specifically to a cloud model security protection method and system based on query provenance analysis (QPA). Background Art

[0002] With the widespread application of machine learning (ML) and deep neural networks (DNN) in various fields, their security issues have received increasing attention. Studies have shown that DNN models are vulnerable to adversarial samples, which are malicious inputs generated by adding small perturbations to normal inputs. Such inputs can mislead DNNs to make incorrect classification decisions. Existing adversarial attack strategies can be divided into two categories: white-box attacks and black-box attacks. White-box attacks require a comprehensive understanding of the architecture and parameters of the target model, while black-box attacks only require query access to the model to understand the confidence of its output or the predicted label. In comparison, black-box attacks are more threatening in practical applications.

[0003] In the prior art, query-based black-box attacks have been widely studied and focused on in machine learning systems. These attacks allow attackers to generate adversarial samples by manipulating input queries, thereby causing system misclassification. Practical application scenarios of these attacks include Machine Learning as a Service (MLaaS), such as Amazon Rekognition. These services provide machine learning models for public access, but only allow users to query the models and obtain output results, so black-box attacks are particularly practical.

[0004] In order to defend against query-based black-box attacks, the prior art proposes stateful defense models (SDMs), such as Blacklight: Scalable defense for neural networks against query-based black-box attacks, in 31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 2117–2134. and PIHA: Detection method using perceptual image hashing against query-based adversarial attacks, Future Gener. Comput. Syst., vol. 145, no. C, p. 563–577, 2023. These models detect adversarial samples and reject them by comparing the similarity between new queries and historical queries. However, the existing technology has significant weaknesses against advanced adaptive attacks (such as Oracle-guided Adaptive Rejection Sampling, OARS) proposed in the literature (Stateful defenses for machine learning models are not yet secure against black-box attacks, in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2023, p. 786–800.). OARS uses the decision boundary information leaked by the defense model to fine-tune the direction and step size of the attack query, thereby successfully evading the detection of existing SDMs. Specifically, OARS can combine existing attack algorithms, adjust the direction and step size of the perturbation, and use an adaptive method to generate queries, thereby bypassing the detection mechanism of SDMs.

[0005] In summary, the state defense model in the prior art has the following technical problems when defending against query-based black-box attacks:

[0006] 1. Vulnerability to adaptive attacks: Existing SDMs such as BlackLight and PIHA can reject queries similar to historical queries, but they are easily bypassed by adaptive attacks such as Oracle-guided Adaptive Rejection Sampling (OARS). OARS uses the feedback information of the defense model to adjust the direction and step size of the attack sample so that it can evade the detection of SDMs.

[0007] 2. Insufficient detection accuracy: Existing SDMs mainly rely on the features of a single query for detection, ignoring the relationship between query sequences. This detection method often has difficulty accurately distinguishing malicious queries from normal queries when dealing with complex adaptive attacks, resulting in insufficient detection accuracy.

[0008] 3. High false positive rate: In order to improve the robustness of defense, lowering the threshold of single query similarity will cause a large number of normal queries to be falsely reported as malicious queries, thereby increasing the false positive rate (FPR) of the system.

[0009] 4. Low system efficiency: When processing a large number of queries in real time, the existing SDMs have high computational overhead and response time, making it difficult to effectively apply them in large-scale systems. Especially when processing high-frequency queries, the performance bottleneck of the existing model is obvious and it is difficult to meet the actual application needs. Summary of the invention

[0010] In response to the above technical problems, the present invention proposes a cloud-based model security protection method and system based on query tracing analysis, which improves the accuracy and robustness of detection and significantly reduces the false alarm rate and system delay by utilizing query sequence features, thereby solving the shortcomings of the prior art.

[0011] To achieve the above-mentioned object of the invention, the technical solution of the present invention includes the following contents.

[0012] A cloud model security protection method based on query tracing analysis, the method comprising:

[0013] Constructing query traceability graphs for normal query sequences and attack query sequences respectively, and training a graph classifier based on the query traceability graphs; wherein the nodes in the query traceability graphs are queries, and the edges in the query traceability graphs are generated based on the similarity between the nodes;

[0014] Initialize the query traceability graph to be detected based on several normal queries; wherein there is no connection between the initial nodes;

[0015] Add the query in the query sequence to be detected to the query traceability graph to be detected, and based on the similarity between the query and the node in the query traceability graph to be detected, construct the corresponding edge and set the edge weight;

[0016] After all queries in the query sequence to be detected are added to the query traceability graph to be detected, the detection result of the query sequence to be detected is obtained in combination with the graph classifier.

[0017] Furthermore, the process of obtaining the attack query sequence includes:

[0018] A black-box attack algorithm is used to randomly select several samples to perform a black-box attack on the cloud model to generate an attack query sequence; wherein the black-box attack algorithm includes: NES, Boundary, Square, HSJA, QEBA and Surfree.

[0019] Furthermore, the edges in the query traceability graph are generated based on the similarity between nodes, including:

[0020] Extract node features using locality sensitive hashing and local binary patterns;

[0021] For each node v i , calculate node v based on the node features j With the node v i The feature similarity between them, and get the node v with the highest similarity value j ; Wherein, i and j are natural numbers, and i≠j;

[0022] Compare the highest similarity value with a set threshold;

[0023] When the highest similarity value is higher than the set threshold, node v is established. i and the corresponding node v j The edge between them, and the highest similarity value is used as the weight of the edge;

[0024] When the highest similarity value is lower than the set threshold, the node v i and query any node v in the traceability graph j No connection.

[0025] Furthermore, the process of obtaining the set threshold includes:

[0026] Randomly select several normal queries;

[0027] Calculate the similarity between normal queries pairwise and calculate the average of all similarities;

[0028] The average value of all similarities is multiplied by a set value t to obtain a set threshold value; wherein 0<t≤1.

[0029] Furthermore, training a graph classifier based on the query tracing graph includes:

[0030] The query traceability graph is converted into a line graph representation, and the edge features are input into a graph convolutional neural network as node features to extract graph embedding;

[0031] Embedding the graph into a binary classifier and calculating the cross entropy loss based on the classification results;

[0032] The parameters of the graph convolutional neural network and the binary classifier are adjusted based on the cross entropy loss.

[0033] Furthermore, obtaining the detection result of the query sequence to be detected in combination with the graph classifier includes:

[0034] Based on the sum of the edge weights in the query traceability graph to be detected, the anomaly score of the query traceability graph to be detected is obtained;

[0035] According to the anomaly score, determining whether the query traceability graph to be detected is a suspicious query traceability graph;

[0036] In the case that the query tracing graph to be detected is a suspicious query tracing graph, the query tracing graph to be detected is detected based on the graph classifier to obtain a detection result of the query sequence to be detected.

[0037] Further, after the query source graph to be detected is detected based on the graph classifier to obtain the detection result of the query sequence to be detected, the method further includes:

[0038] Get new query;

[0039] Calculate the similarity between the new query and all queries in the current query traceability graph, and obtain the query traceability graph where the query with the highest similarity to the new query is located;

[0040] If the query traceability graph of the query with the highest similarity to the new query is a normal query traceability graph, then the new query is a normal query;

[0041] If the query traceability graph of the query with the highest similarity to the new query is an abnormal query traceability graph, the new query is an illegal query.

[0042] Furthermore, after obtaining the anomaly score of the query traceability graph to be detected based on the sum of the edge weights in the query traceability graph to be detected, the method further includes:

[0043] The K query traceability graphs with the highest anomaly scores are saved in memory, and the remaining query traceability graphs are moved to the disk database.

[0044] Furthermore, after the K query tracing graphs with the highest anomaly scores are stored in the memory and the remaining query tracing graphs are moved to the disk database, the method further includes:

[0045] Periodically clear the query provenance graph in the memory and disk database, and reinitialize the graph structure of the query provenance graph.

[0046] A cloud model security protection system based on query tracing analysis, the system comprising:

[0047] A training module is used to construct query traceability graphs for normal query sequences and attack query sequences respectively, and train a graph classifier based on the query traceability graphs; wherein the nodes in the query traceability graphs are queries, and the edges in the query traceability graphs are generated based on the similarity between the nodes;

[0048] A testing module is used to obtain a query sequence to be detected, and based on some normal queries in the query sequence to be detected, generate initial nodes of a traceability graph of the queries to be detected, wherein there is no connection between the initial nodes; add other queries in the query sequence to be detected to the traceability graph of the queries to be detected, and based on the similarity between the other queries and the nodes in the traceability graph of the queries to be detected, construct corresponding edges and set edge weights; after all queries in the query sequence to be detected are added to the traceability graph of the queries to be detected, obtain the detection results of the query sequence to be detected in combination with the graph classifier.

[0049] Compared with the prior art, the present invention has at least the following technical effects.

[0050] 1. More robust defense against query-based black-box attacks.

[0051] QPA can effectively defend against powerful adaptive attacks (such as OARS) by using query sequence features to detect malicious queries. We conducted preliminary experiments on this, randomly selecting 100 images as target images on four datasets: MNIST, CIFAR10, ImageNet, and CelebaHQ, to attack OARS. If OARS successfully generates corresponding adversarial samples, the attack is considered successful. If the attack process is terminated early or fails after more than 100,000 queries, the attack is considered failed. The experimental results show that when QPA defends against OARS attacks, the attack success rate (ASR) of OARS is only 4.08%, which is about 20 times lower than the existing defense methods. Compared with traditional methods that rely on single query features, QPA uses the overall characteristics of the query sequence to avoid the evasion of individual query features by adaptive attacks and improve the robustness of the defense.

[0052] 2. It has higher detection and defense accuracy.

[0053] QPA combines statistical analysis and graph neural network classifiers for anomaly detection, which can more accurately distinguish normal queries from malicious queries. Experimental results show that QPA has a detection coverage of 96% and a detection accuracy of 98% for six query-type attacks on four datasets: MNIST, CIFAR10, ImageNet, and CelebaHQ, which is significantly better than existing state defense models (SDMs). By leveraging the structural features of the query provenance graph, QPA can capture the unique patterns of malicious query sequences and maintain high detection accuracy even in noisy data environments.

[0054] 3. The efficiency of the defense system has been significantly improved.

[0055] QPA adopts dynamic management strategies, including graph eviction and reset, which effectively reduces memory and computational overhead. QPA can maintain high system efficiency when processing a large number of queries. Experimental results show that QPA's throughput is increased by 7.67 times and its response latency is reduced by 11.09 times. By updating the query traceability graph in real time and using an efficient anomaly detection algorithm, QPA can quickly process high-frequency queries in large-scale systems to meet practical application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of the offline training phase.

[0057] Figure 2 Flowchart of the online detection phase. DETAILED DESCRIPTION

[0058] The present invention is further described in detail below in conjunction with specific embodiments. The given examples are only for illustrating the present invention, but not for limiting the scope of the present invention.

[0059] The cloud model security protection method based on query tracing analysis of the present invention includes an offline training phase and an online detection phase.

[0060] 1. Offline training phase.

[0061] like Figure 1 As shown in the figure, the offline training phase mainly includes the construction of the query traceability graph and the training of the graph classifier.

[0062] A. Construction of query traceability graph.

[0063] Step A1: Use the existing black-box attack algorithm to randomly select several samples to perform black-box attacks on the model, and record the attack query sequence on the model; randomly select a batch of samples from the data set as normal query sequence data.

[0064] Specifically, this embodiment uses six existing black-box attack algorithms, namely NES, Boundary, Square, HSJA, QEBA and Surfree, randomly selects several samples to perform black-box attacks on the model, and records the attack query sequence on the model.

[0065] Step A2: Construct query traceability graphs for normal query sequences and attack query sequences respectively.

[0066] For each normal query sequence and attack query sequence, this embodiment constructs a query traceability graph by calculating the similarity between each query in the sequence and other queries.

[0067] Specifically, this embodiment first uses Locality Sensitive Hashing (LSH) and Local Binary Pattern (LBP) to extract features from query data, and then calculates the similarity between a query and the historical query before the query in the sequence. Finally, the similarity threshold T is set to determine whether the query is connected to the most similar query: if the similarity between a query and the historical query is lower than the similarity threshold T, the query is only added to the graph as a node and is not connected to other nodes. Conversely, if the similarity between a query and some historical queries is higher than the similarity threshold T, the query is connected to the most similar query and the similarity is used as the edge weight.

[0068] B. Training of graph classifiers.

[0069] Graph Convolutional Networks (GCN) are used to extract graph embeddings and a binary classifier is trained to classify the output of statistical analysis. The original query traceability graph is converted into a line graph representation, and the edge features are input into GCN as node features. The graph classifier consists of three convolutional layers and one linear layer. The loss function is cross entropy loss. The best classification model is obtained through offline training.

[0070] 2. Online detection stage.

[0071] like Figure 2 As shown, the online detection phase mainly includes feature extraction and similarity calculation, dynamic update of query traceability graph, anomaly detection, dynamic management of query traceability graph and graph reset.

[0072] C. Feature extraction and similarity calculation.

[0073] For the query sequence to be detected, this embodiment constructs the query traceability graph to be detected in a manner basically the same as the offline training phase. That is to say, the online detection phase also uses Locality Sensitive Hashing (LSH) and Local Binary Pattern (LBP) to extract features from historical query data. For each query x, the hash value feature F(x) of the query is extracted, and its similarity with the historical query is calculated. The similarity is characterized by the number of identical hash value features of the two queries.

[0074] D. Dynamic update of query traceability graph.

[0075] As new queries arrive, the query traceability graph is incrementally updated, connecting the new query with the most similar historical query, and using the similarity as the edge weight. If the similarity between the new query and the historical query is lower than the threshold T, the new query is only added to the graph as a node without connecting to other nodes. The threshold is set to the 90th percentile of the similarity calculated between 1,000 randomly selected normal data.

[0076] Unlike the training phase, when constructing the query traceability graph for the queries to be detected in the online detection phase, the query traceability graph for the queries to be detected will be initialized first, that is, a small number of normal queries will be selected as initial nodes, and it is ensured that there is no connection between these initial nodes. After that, the query sequences to be detected are added to the query traceability graph one by one. The purpose of initializing the query traceability graph based on a small number of normal queries is to reduce the aggregation of normal queries when constructing the query traceability graph. Because the sequence to be detected in the online monitoring process is a mixture of normal and abnormal, and the abnormality rate is very low, it is hoped that when constructing the query traceability graph, the abnormal queries can be more effectively clustered together, while the normal queries are less clustered.

[0077] E. Anomaly detection.

[0078] The statistical analysis module calculates the anomaly score of each query traceability graph and uses statistical methods (such as Grubbs's Test) to detect anomaly scores that deviate from normal levels and preliminarily screen out suspicious query graphs. The anomaly score of a query traceability graph is the sum of all edge weights of the graph.

[0079] The filtered query graph is input into the pre-trained graph classifier, graph embedding is extracted, and binary classification detection is performed to finally determine whether the query is malicious. If a malicious query is detected, the query is rejected and the attack behavior is recorded; if the query is normal, the processing continues and the model output is returned.

[0080] In addition, the online detection process will return a detection result for each query to determine whether the query is normal or abnormal. The specific process is that when a new query arrives, the similarity with all queries in the current query traceability graph is calculated, and the closest one is selected. If the query traceability graph where the closest query is located is abnormal, then the new query is judged to be an abnormal query; if the closest query belongs to the normal query traceability graph, the query is a normal query.

[0081] F. Dynamically manage the query traceability graph.

[0082] A graph eviction strategy is adopted to keep the K graphs with the highest anomaly scores in memory and move the rest of the graphs to the disk database. To ensure that the system remains efficient when processing a large number of queries, K is set to 25.

[0083] G. Resetting of the graph.

[0084] Reset the query provenance graph storage regularly (e.g. every 24 hours), clear the memory and database, and reinitialize the graph structure to ensure the stability and efficiency of the system during long-term operation.

[0085] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology in this field to understand the content of the present invention and implement it, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud model security protection method based on query tracing analysis, characterized in that: The method comprises: Constructing query traceability graphs for normal query sequences and attack query sequences respectively, and training a graph classifier based on the query traceability graphs; wherein the nodes in the query traceability graphs are queries, and the edges in the query traceability graphs are generated based on the similarity between the nodes; Initialize the query traceability graph to be detected based on several normal queries; wherein there is no connection between the initial nodes; Add the query in the query sequence to be detected to the query traceability graph to be detected, and based on the similarity between the query and the node in the query traceability graph to be detected, construct the corresponding edge and set the edge weight; After all queries in the query sequence to be detected are added to the query traceability graph to be detected, the detection result of the query sequence to be detected is obtained in combination with the graph classifier.

2. The method according to claim 1, characterized in that: The process of obtaining the attack query sequence includes: A black-box attack algorithm is used to randomly select several samples to perform a black-box attack on the cloud model to generate an attack query sequence; wherein the black-box attack algorithm includes: NES, Boundary, Square, HSJA, QEBA and Surfree.

3. The method according to claim 1, characterized in that The edges in the query traceability graph are generated based on the similarity between nodes, including: Extract node features using locality sensitive hashing and local binary patterns; For each node v i , calculate node v based on the node features j With the node v i The feature similarity between them, and get the node v with the highest similarity value j ; Wherein, i and j are natural numbers, and i≠j; Compare the highest similarity value with a set threshold; When the highest similarity value is higher than the set threshold, node v is established. i and the corresponding node v j The edge between them, and the highest similarity value is used as the weight of the edge; When the highest similarity value is lower than the set threshold, the node v i and query any node v in the traceability graph j No connection.

4. The method according to claim 3, characterized in that The process of obtaining the set threshold value includes: Randomly select several normal queries; Calculate the similarity between normal queries pairwise and calculate the average of all similarities; Multiply the average value of all similarities by a set value t to obtain the set threshold; <t≤1。 5. The method according to claim 1, characterized in that Training a graph classifier based on the query tracing graph includes: The query traceability graph is converted into a line graph representation, and the edge features are input into a graph convolutional neural network as node features to extract graph embedding; Embedding the graph into a binary classifier and calculating the cross entropy loss based on the classification results; The parameters of the graph convolutional neural network and the binary classifier are adjusted based on the cross entropy loss.

6. The method according to claim 1, characterized in that Combining the graph classifier to obtain the detection result of the query sequence to be detected includes: Based on the sum of the edge weights in the query traceability graph to be detected, the anomaly score of the query traceability graph to be detected is obtained; According to the anomaly score, determining whether the query traceability graph to be detected is a suspicious query traceability graph; In the case that the query tracing graph to be detected is a suspicious query tracing graph, the query tracing graph to be detected is detected based on the graph classifier to obtain a detection result of the query sequence to be detected.

7. The method according to claim 6, characterized in that After detecting the query source graph to be detected based on the graph classifier and obtaining the detection result of the query sequence to be detected, the method further includes: Get new query; Calculate the similarity between the new query and all queries in the current query traceability graph, and obtain the query traceability graph where the query with the highest similarity to the new query is located; If the query traceability graph of the query with the highest similarity to the new query is a normal query traceability graph, then the new query is a normal query; If the query traceability graph of the query with the highest similarity to the new query is an abnormal query traceability graph, the new query is an illegal query.

8. The method according to claim 6, characterized in that After obtaining the anomaly score of the query traceability graph to be detected based on the sum of the edge weights in the query traceability graph to be detected, the following further includes: The K query traceability graphs with the highest anomaly scores are saved in memory, and the remaining query traceability graphs are moved to the disk database.

9. The method according to claim 8, characterized in that After the K query tracing graphs with the highest anomaly scores are stored in the memory and the remaining query tracing graphs are moved to the disk database, the method further includes: Periodically clear the query provenance graph in the memory and disk database, and reinitialize the graph structure of the query provenance graph.

10. A cloud-based model security protection system based on query tracing analysis, characterized in that: The system comprises: A training module is used to construct query traceability graphs for normal query sequences and attack query sequences respectively, and train a graph classifier based on the query traceability graph; wherein the nodes in the query traceability graph are queries, and the edges in the query traceability graph are generated based on the similarity between the nodes; A testing module is used to obtain a query sequence to be detected, and based on some normal queries in the query sequence to be detected, generate initial nodes of a traceability graph of the queries to be detected, wherein there is no connection between the initial nodes; add other queries in the query sequence to be detected to the traceability graph of the queries to be detected, and based on the similarity between the other queries and the nodes in the traceability graph of the queries to be detected, construct corresponding edges and set edge weights; after all queries in the query sequence to be detected are added to the traceability graph of the queries to be detected, obtain the detection results of the query sequence to be detected in combination with the graph classifier.

Images