Defense model training method and device

Abstract

The invention provides a defense model training method and device, is applied to the technical field of network security, and is used for effectively defending attack and stealing behaviors. The method comprises the following steps: inputting a plurality of query requests into a first model, and obtaining a first prediction result and a second prediction result output by the first model; the second prediction result is obtained after the first prediction result is interfered by a preset noise transfer matrix in the first model; inputting the plurality of query requests into a second model, and obtaining a third prediction result output by the second model; a first loss value between the first prediction result and the second prediction result and a second loss value between the first prediction result and the third prediction result are calculated, and the purpose is that the first loss value is smaller than a first threshold value and the second loss value is larger than a second threshold value. And adjusting parameters of the preset noise transfer matrix to obtain an optimized first model.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a defense model training method and device. Background Art

[0002] As more and more companies deploy models in the cloud or provide services to the outside world through application programming interfaces (APIs), attackers can query the model interface and analyze the output results, gradually infer the structure and parameters of the model, and thus steal the model. Training the model requires a lot of computing resources and data, and the model has high commercial value. Once stolen, the attacker can reproduce the model's functions without incurring high training costs, thereby causing unfair competition.

[0003] Therefore, how to effectively defend against attacks and thefts is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a defense model training method and device for achieving effective defense against attack and theft behaviors.

[0005] In a first aspect, an embodiment of the present application provides a defense model training method, which can be applied to any electronic device with processing capabilities, and the method includes: Input multiple query requests into the first model, and obtain a first prediction result and a second prediction result output by the first model; the first prediction result is the probability distribution of the category to which the query data in each query request belongs predicted by the first model; the second prediction result is obtained after the first prediction result is interfered by the noise transfer matrix preset in the first model; Input multiple query requests into the second model, and obtain a third prediction result output by the second model; the second model is a model trained based on the multiple query requests and the second prediction results, and the third prediction result is a probability distribution of the category to which the query data in each query request belongs predicted by the second model; Calculate the first loss value between the first prediction result and the second prediction result, and the second loss value between the first prediction result and the third prediction result, and adjust the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than the first threshold and the second loss value greater than the second threshold to obtain the optimized first model.

[0006] In this scheme, the first prediction result output by the first model is interfered with by a preset noise transfer matrix to obtain the second prediction result, so that the second prediction result has a certain degree of disruption, and the attacker cannot directly obtain the true probability distribution (i.e., the first prediction result) output by the first model; the second model is trained based on the second prediction result and multiple query requests, that is, a model cloned by an attacker who simulates stealing the first model, and then the first loss value is calculated according to the first prediction result and the second prediction result, and the second loss value is calculated according to the third prediction result output by the second model and the first prediction result. The parameters of the noise transfer matrix are adjusted with the purpose of making the first loss value less than the first threshold and the second loss value greater than the second threshold, so as to ensure that the noise transfer matrix will not significantly affect the use of the first model by legitimate users, and can effectively ensure that the attacker cannot accurately steal the first model, thereby achieving effective defense against attack and theft behaviors.

[0007] Optionally, before inputting multiple query requests into the second model, the method also includes: inputting the multiple query requests into the initial second model, obtaining a fourth prediction result of the initial second model for each of the multiple query requests; calculating a third loss value between the second prediction result and the fourth prediction result, and adjusting the initial second model with the purpose of making the third loss value less than a third threshold, to obtain a second model.

[0008] Optionally, adjusting the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than a first threshold and the second loss value greater than a second threshold includes: calculating a comprehensive loss value of the first loss value and the second loss value according to a first formula, and the first formula is as follows: ; in, represents the comprehensive loss value, represents the first loss value, represents the second loss value, T represents the preset noise transfer matrix, θ g represents the weight of the second model; λ is a constant; With the goal of making the comprehensive loss value smaller than a fourth threshold, the parameters of the preset noise transfer matrix are adjusted.

[0009] Optionally, before inputting multiple query requests into the first model, the method also includes: constructing a behavior recognition model based on features of normal query requests in historical records; the behavior recognition model is used to identify and classify the input features, and output the category to which the input features belong; the category includes in-distribution data and out-of-distribution data, the in-distribution data is used to indicate that the query request to which the input features belong is a normal query request, and the out-of-distribution data is used to indicate that the query request to which the input features belong is an abnormal query request; inputting features of each query request among several query requests into the behavior recognition model, and obtaining the category of each query request output by the behavior recognition model; obtaining query requests whose category is out-of-distribution data, and obtaining multiple query requests.

[0010] Optionally, the in-distribution data includes multiple feature sets, each of the multiple feature sets is used to indicate a type; the features of each query request among a number of query requests are input into a behavior recognition model to obtain the category of each query request output by the behavior recognition model, including: calculating the Mahalanobis distance between the features of each query request and the multiple feature sets; setting the category of the query request whose Mahalanobis distance is greater than a distance threshold as out-of-distribution data; otherwise, setting it to in-distribution data.

[0011] Optionally, query requests whose category is out-of-distribution data are obtained to obtain multiple query requests, including: generating a random number for each query request whose category is out-of-distribution data; obtaining query requests corresponding to all random numbers whose values ​​are not greater than a preset probability threshold to obtain multiple query requests.

[0012] In a second aspect, an embodiment of the present application provides a defense model training device, including: A first processing module is used to: input multiple query requests into the first model, and obtain a first prediction result and a second prediction result output by the first model; the first prediction result is the probability distribution of the category to which the query data in each query request belongs predicted by the first model; the second prediction result is obtained after the first prediction result is interfered by the noise transfer matrix preset in the first model; A second processing module is used to: input multiple query requests into the second model, and obtain a third prediction result output by the second model; the second model is a model trained based on the multiple query requests and the second prediction results, and the third prediction result is a probability distribution of the category to which the query data in each query request belongs predicted by the second model; The optimization module is used to calculate a first loss value between a first prediction result and a second prediction result, and a second loss value between a first prediction result and a third prediction result, and adjust the parameters of a preset noise transfer matrix with the purpose of making the first loss value less than a first threshold and the second loss value greater than a second threshold, so as to obtain an optimized first model.

[0013] Optionally, before the first processing module inputs multiple query requests into the second model, the second processing module is also used to: input the multiple query requests into the initial second model, obtain the fourth prediction result of the initial second model for each of the multiple query requests; calculate the third loss value between the second prediction result and the fourth prediction result, and adjust the initial second model with the purpose of making the third loss value less than a third threshold, so as to obtain the second model.

[0014] Optionally, when the optimization module adjusts the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than the first threshold and the second loss value greater than the second threshold, it is specifically used to: calculate the comprehensive loss value of the first loss value and the second loss value according to a first formula, and the first formula is as follows: ; in, represents the comprehensive loss value, represents the first loss value, represents the second loss value, T represents the preset noise transfer matrix, θ g represents the weight of the second model; λ is a constant; With the goal of making the comprehensive loss value smaller than a fourth threshold, the parameters of the preset noise transfer matrix are adjusted.

[0015] Optionally, before inputting multiple query requests into the first model, the first processing module is also used to: build a behavior recognition model based on features of normal query requests in historical records; the behavior recognition model is used to identify and classify the input features, and output the category to which the input features belong; the category includes in-distribution data and out-of-distribution data, the in-distribution data is used to indicate that the query request to which the input features belong is a normal query request, and the out-of-distribution data is used to indicate that the query request to which the input features belong is an abnormal query request; input the features of each query request among several query requests into the behavior recognition model to obtain the category of each query request output by the behavior recognition model; obtain query requests whose category is out-of-distribution data to obtain multiple query requests.

[0016] Optionally, the in-distribution data includes multiple feature sets, each of the multiple feature sets is used to indicate a type; when the first processing module inputs the features of each query request among a number of query requests into the behavior recognition model to obtain the category of each query request output by the behavior recognition model, it is specifically used to: calculate the Mahalanobis distance between the features of each query request and the multiple feature sets; set the category of the query request whose Mahalanobis distance is greater than the distance threshold as out-of-distribution data; otherwise, set it to in-distribution data.

[0017] Optionally, when the first processing module obtains a query request whose category is out-of-distribution data and obtains multiple query requests, it is specifically used to: generate a random number for each query request whose category is out-of-distribution data; obtain query requests corresponding to all random numbers whose values ​​are not greater than a preset probability threshold, and obtain multiple query requests.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising at least one processor, wherein the at least one processor is used to execute a computer program stored in a memory so that a method as in the first aspect or any optional implementation of the first aspect is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the method in the first aspect or any optional implementation of the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program code, which, when executed on a computer, enables the method in the first aspect or any optional implementation of the first aspect to be implemented.

[0021] The technical effects or advantages of one or more technical solutions provided in the second, third, fourth and fifth aspects of the embodiments of the present application can be correspondingly explained by the technical effects or advantages of one or more corresponding technical solutions provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a defense model training method provided in an embodiment of the present application; Figure 2 An example diagram of an attack and theft behavior provided in an embodiment of the present application; Figure 3 An example diagram of an interactive training model provided in an embodiment of the present application; Figure 4 A flowchart of a defense method provided in an embodiment of the present application; Figure 5 A structural diagram of a defense model training device provided in an embodiment of the present application; Figure 6 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In the technical solution of this application, the collection, dissemination, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0024] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0025] The technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0026] It should be understood that "multiple" in the description of the embodiment of the present application refers to two or more. "First", "second" etc. in the embodiment of the present application are used to distinguish different objects, rather than to describe a specific order. The term "and / or" in the embodiment of the present application is merely a kind of association relationship describing associated objects, indicating that three relationships may exist, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "includes" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. The module in the embodiment of the present application refers to a part with independent functions in a software system.

[0027] For ease of understanding, the following is an introduction to some professional terms involved in the embodiments of this application: 1. Model Stealing Attack (MSA) is when an attacker accesses a machine learning model's API (usually black-box access), sends queries and obtains model outputs, and then uses these query-output pairs to train a model of their own, thereby cloning a model with similar functions to the original model. The attacker does not need to know the specific architecture or training data of the model, but only repeatedly sends queries to gradually replicate the model's behavior.

[0028] 2. Out-Of-Distribution (OOD) means that some input data is distributed differently from the model’s training data and is data that the model has never seen or is unfamiliar with.

[0029] 3. Mahalanobis distance (MD) is a metric used to measure the distance between a data point and the center of the data distribution. It can take into account the correlation between various features and is widely used in high-dimensional space. Unlike the common Euclidean distance (ED), Mahalanobis distance can be adaptively adjusted according to the shape of the data distribution, which is suitable for situations where the data has different scales and correlations.

[0030] 4. Embedding Vector is a method of mapping high-dimensional data to low-dimensional space, which is mainly used in machine learning and deep learning to process text, images, user behavior and other tasks. Embedding vector maps complex, high-dimensional or sparse data to low-dimensional, dense vector space through embedding functions, retaining the semantic or structural information of the original data, thus facilitating similarity calculation, classification and other operations.

[0031] With the widespread application of deep learning and large-scale models, the demand for model theft defense is increasing. Training AI models usually requires a lot of computing resources and data, so trained AI models have high commercial value. Once the model is stolen, attackers can directly reproduce the functions of the model without incurring high training costs, which may lead to unfair competition. Many models are provided to users through APIs or cloud services, which allows attackers to gradually infer their structure and parameters through limited interaction with the model and steal the model. This not only threatens the commercial interests of the model, but may also lead to privacy leaks, especially when the model is used in fields such as medical diagnosis or financial risk control, where thieves may infer sensitive data contained in the model summary.

[0032] Therefore, how to effectively defend against model stealing has become an important research topic. Model stealing defense can be divided into three main strategies: detection, perturbation, and post-verification. The current mainstream methods include: stealing attack detection based on decision trees, which trains decision trees by constructing benign sample sets, verification sample sets, and sample sets to be tested selected from the query data set of the user to be tested as data sets, and then calculates the difference in information gain and the difference in the proportion of OOD images in the benign sample set and the sample set to be tested to determine the detection result. Although it can guarantee a high detection accuracy for the sample data set, it takes a lot of time to collect sample data and train decision trees, and it is impossible to defend against stealing attacks in real time. Moreover, since detection usually occurs after the stealing attack, there is a defense gap. The delay of this method cannot meet the actual operational needs; training with known attack samples has weak resistance to new attack methods or multi-layer attacks. Alternatively, a stealing attack defense method based on gradient calculation is adopted. By calculating the directional similarity between the gradient of the proxy stealing model and the true gradient as the gradient angle loss, the output difference between the proxy stealing model and the target model on the out-of-distribution data is calculated as the information acquisition loss until the parameters of the target model are updated, and then the stealing attack defense is achieved by gradient projection. However, the gradient projection may not be based on the effective update of the target model, especially when the natural direction of the gradient and the projection direction are very different, the model may find it difficult to learn important task-related features, reducing the effectiveness of the model. Alternatively, a defense method based on adversarial training is adopted. By applying adversarial training in the training phase to obtain a more robust model, the effect of the model stealing attack is weakened, resulting in the attacker needing to submit more malicious queries to achieve the expected attack target. Malicious query detection is then used to detect and identify malicious queries and mark malicious users. For malicious users, the defender adopts an adaptive reply strategy to reply to malicious users with perturbed results, and finally verifies the model ownership through the perturbed result set. However, this method mainly achieves targeted defense, has weak resistance to new attack methods, and if the attack method changes, the model may not have sufficient defense.

[0033] In view of this, an embodiment of the present application is proposed, which inputs multiple query requests into a first model to obtain a first prediction result output by the first model and a second prediction result obtained after the first prediction result is interfered with by a noise transfer matrix preset in the first model; inputs multiple query requests into a second model trained based on the second prediction result and the multiple query requests to obtain a third prediction result output by the second model, the second model can simulate the model cloned by the attacker, and the obtained third prediction result can be regarded as the prediction result output by the model cloned by the attacker; and then calculates a first loss value between the first prediction result and the second prediction result (i.e., a loss value between a true prediction result output by the first model and a prediction result interfered with by a preset noise transfer matrix), and calculates a second loss value between the first prediction result and the third prediction result (i.e., a loss value between a true prediction result and a prediction result of the model cloned by the attacker), and adjusts the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than a first threshold value and the second loss value greater than a second threshold value to obtain an optimized first model, so that the obtained first model can ensure that legitimate users can obtain relatively accurate prediction results, and can prevent attackers from accurately cloning the first model, thereby achieving effective defense against attack and theft behaviors.

[0034] It can be understood that the embodiments of the present application can be applied to any defense scenario of attacking and stealing models, and the embodiments of the present application do not limit the application scenarios.

[0035] See also Figure 1 , a defense model training method provided in an embodiment of the present application, can be applied to any electronic device with processing capabilities, the method comprising steps S101 to S104: S101. Input multiple query requests into a first model, and obtain a first prediction result and a second prediction result output by the first model.

[0036] The first prediction result is the probability distribution of the category to which the query data in each query request belongs predicted by the first model; the second prediction result is obtained after the first prediction result is interfered by the noise transfer matrix preset in the first model.

[0037] The first model is a model that has been trained or has been put into use in practical applications. The embodiment of the present application does not limit the category of the first model, which can be a visual classification model, an image recognition model, etc.

[0038] See also Figure 2, which is an example diagram of an attacker stealing a model provided in an embodiment of the present application. Usually, the stealing attacker does not know the data distribution D(x) of the victim model (i.e., the first model in the embodiment of the present application) (data distribution is used to indicate the expansion or dispersion mode of the data set of the model in a certain space, provides a statistical description of the data characteristics, reveals the internal structure of the data, and helps understand the characteristics and trends of the data), so, Figure 2 As shown, the attacker first sends a query request to the victim model (i.e. Figure 2 The query input image in ), according to the prediction results fed back by the victim model, train the clone model (i.e. Figure 2 By reducing the loss, the clone model gradually adapts to the data distribution D(x) of the victim model to steal the victim model.

[0039] Therefore, in view of this situation, the embodiment of the present application sets a preset noise transfer matrix (such as Figure 2 T n ), which is used to interfere with the first prediction result (i.e., the real prediction result) output by the first model to obtain the second prediction result. The output fed back to the attacker is the second prediction result, so that the attacker cannot accurately obtain the real prediction result.

[0040] The preset noise transfer matrix may be a randomly generated probability matrix at the initial stage, and is continuously optimized and adjusted during the subsequent training process.

[0041] S102: Input multiple query requests into the second model, and obtain a third prediction result output by the second model.

[0042] The second model is a model trained based on multiple query requests and second prediction results, and the third prediction result is the probability distribution of the category to which the query data in each query request belongs predicted by the second model.

[0043] In a possible embodiment, before executing step S102, it is necessary to pre-train a second model. The specific implementation method of training the second model can be as follows: Inputting multiple query requests into the initial second model, and obtaining a fourth prediction result of the initial second model for each query request in the multiple query requests; A third loss value between the second prediction result and the fourth prediction result is calculated, and the initial second model is adjusted with the purpose of making the third loss value less than a third threshold value to obtain a second model.

[0044] It can be understood that the second model is a model cloned by an attacker in the embodiment of the present application. By calculating the loss value between the fourth prediction result and the second prediction result for each query request output by the second model, and adjusting the parameters of the second model with the purpose of making the loss value less than the third threshold, the scenario of cloning the model based on the output of the first model is simulated.

[0045] Exemplarily, the method for calculating the third loss value between the second prediction result and the fourth prediction result can be selected according to actual needs. The embodiment of the present application provides an example of a method for calculating the third loss value: ; in, represents the third loss value; g(x; θ g ) is used to indicate the fourth prediction result output by the second model, f(x; θ f ) is used to indicate the first prediction result output by the first model, T is the preset noise transfer matrix, f(x;θ f )·T is used to indicate the second prediction result output by the first model; x is used to indicate the query data in the input query request, θ g and θ f They are used to indicate the weights of the second model and the first model respectively; Dpair is used to indicate a training data pair, that is, the query data x in the input query request and the second prediction result corresponding to the query data x.

[0046] The third loss value is calculated based on the above formula, and θ is updated by gradient descent. g ,Right now: ; Among them, t is used to represent the number of iterations, α represents the learning rate of the second model, and ∇ represents gradient descent. Iterations are repeated until the third loss value is less than the third threshold or the third loss value converges.

[0047] S103: Calculate a first loss value between the first prediction result and the second prediction result, and a second loss value between the first prediction result and the third prediction result.

[0048] It can be understood that by calculating the first loss value between the first prediction result and the second prediction result, that is, calculating the loss value between the true prediction result output by the first model and the prediction result after interference by the preset noise transfer matrix, the degree of disturbance of the preset noise transfer matrix to the true prediction result can be determined according to the first loss value; by calculating the second loss value between the first prediction result and the third prediction result, that is, calculating the loss value between the true prediction result output by the first model and the prediction result output by the model cloned by the attacker (that is, the second model), the accuracy of the model cloned by the attacker can be determined according to the second loss value.

[0049] In practical applications, the method for calculating the first loss value and the second loss value can be selected according to actual needs. The embodiment of the present application provides an example of calculating the first loss value and the second loss value, which is introduced in detail below through Example 1 and Example 2 respectively.

[0050] Example 1: Calculate a first loss value between a first prediction result and a second prediction result.

[0051] The first loss value is calculated using the following formula:

[0052] ; in, represents the first loss value; k represents the category with the largest probability in the probability distribution of the category to which the query data belongs in the query request output by the first model.

[0053] In this way, the first loss value calculated based on the above formula can reflect the gap between the category with the highest probability (i.e., the correct category) in the probability distribution obtained after the preset noise transfer matrix and the category with the highest probability in the true probability distribution, and then determine the degree of disturbance of the preset noise transfer matrix on the true prediction result.

[0054] Example 2: Calculate the second loss value between the first prediction result and the third prediction result.

[0055] The second loss value is calculated using the following formula: ; ; in, represents the second loss value; k represents the category with the largest probability in the probability distribution of the category to which the query data belongs in the query request output by the first model.

[0056] In this way, the second loss value calculated based on the above formula can reflect the gap between the category with the highest probability in the true probability distribution output by the first model and the category with the highest probability in the probability distribution output by the second model, thereby determining the accuracy of cloning the first model by the second model.

[0057] It can be understood that the above example 1 and example 2 are only one possible method for calculating the first loss value and the second loss value, and are not limited thereto.

[0058] S104, with the goal of making the first loss value smaller than the first threshold and the second loss value larger than the second threshold, adjusting the parameters of the preset noise transfer matrix to obtain an optimized first model.

[0059] It can be understood that the first loss value can be used to indicate the degree of disturbance of the preset noise transfer matrix on the actual prediction result. In order to ensure that the prediction result obtained by the legitimate user will not be significantly modified by the preset noise transfer matrix, for example, the first model predicts that the query data x in the query request belongs to category c1 and the confidence (i.e., probability) is very high, then the probability of c1 in the prediction result after the interference of the noise transfer matrix should also be high, rather than the probability being significantly shifted to other categories. Therefore, it is necessary to adjust the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than the first threshold (usually the first loss value converges, i.e., the first loss value continues to decrease during the iteration process until it reaches the critical value).

[0060] Similarly, the second loss value can be used to indicate the accuracy of the model cloned by the attacker. In order to reduce the accuracy of the cloned model, that is, to reduce the confidence (that is, probability) of the cloned model for the correct category (that is, the category with the highest probability in the output probability distribution), the attacker cannot accurately learn the true probability distribution when cloning the model (that is, the second model). For example, the query data x in the query request output by the first model belongs to category c1, and the confidence is very high (for example, 90%). However, after being interfered by the preset noise transfer matrix, the output of the model cloned by the attacker may be on category c1, but the confidence (that is, probability) is greatly reduced (for example, from 90% to 50%), thereby making the learning of the cloned model unreliable. Therefore, the parameters of the preset noise transfer matrix are adjusted with the purpose of the second loss value being greater than the second threshold (usually also the convergence of the second loss value, that is, the second loss value continues to increase during the iteration process until it reaches the critical value).

[0061] In a possible embodiment, the embodiment of the present application combines the first loss value and the second loss value to strike a balance between disrupting the attacker's theft behavior and maintaining the accuracy of the prediction results of the legitimate user. The specific implementation of step S104 is as follows: The combined loss value of the first loss value and the second loss value is calculated according to the first formula. The first formula is as follows: ; in, represents the comprehensive loss value, represents the first loss value, represents the second loss value, T represents the preset noise transfer matrix, θ g represents the weight of the second model; λ is a constant; With the goal of making the comprehensive loss value smaller than a fourth threshold, the parameters of the preset noise transfer matrix are adjusted.

[0062] Exemplarily, λ is a parameter used to control the trade-off between disrupting the attacker's stealing behavior and maintaining the accuracy of the prediction results of legitimate users, and can be set according to actual needs. For example, if the number of abnormal queries occurring in the current time period is large, the value of λ can be appropriately reduced so that the preset noise transfer matrix is ​​more inclined to disrupt the attacker's stealing behavior, that is, introduce more noise and reduce the accuracy of the model cloned by the attacker; if the number of abnormal queries occurring in the current time period is small, the value of λ can be appropriately increased so that the preset noise transfer matrix is ​​more inclined to ensure that the accuracy of the prediction results of legitimate users is high, that is, reduce the introduced noise to ensure the normal use of the first model by legitimate users.

[0063] Optionally, in practical applications, only the first loss value or only the second loss value may be used to adjust the preset noise transfer matrix, that is, the value of λ may be 0 or 1.

[0064] The method for adjusting the parameters of the preset noise transfer matrix can also adopt the gradient descent method, that is, ; Where t represents the number of iterations and β represents the learning rate of the noise transfer matrix.

[0065] In a possible embodiment, in order to further improve the defense effect of the first model, after obtaining the optimized first model, the second model can be trained again based on the optimized first model, that is, the second model is trained based on the second prediction result output by the optimized first model (that is, the prediction result after interference based on the optimized noise transfer matrix) and multiple request queries, so that the optimized second model can further optimize the parameters of the noise transfer matrix by using the method of the above steps S101~S104, and so on. The first model and the second model can be trained with each other until the expected requirements are met.

[0066] For details, see Figure 3 , which is an example diagram of interactive training provided in an embodiment of the present application.

[0067] Among them, T t represents the noise transfer matrix after t iterations, θ t Represents the weight (i.e., parameter) of the second model after t iterations; First, the true probability distribution predicted by the training samples (i.e., multiple query requests) is disturbed based on the noise transfer matrix contained in the first model to obtain the second prediction result. The second model is trained based on the second prediction result and the training samples, that is, the training samples are input into the second model to obtain the third prediction result output by the second model, and the third loss value (i.e., Figure 3 In ), based on the third loss value, the weight of the second model (i.e., θ) is adjusted using the gradient descent method t →θ t+1 ).

[0068] Then, the first model (mainly the noise transfer matrix in the first model) is trained based on the updated second model, that is, the loss value is calculated using the method in steps S101 to S104 above. , based on the loss value, the parameters of the noise transfer matrix are adjusted using the gradient descent method.

[0069] It can be understood that after updating the noise transfer matrix, it is necessary to ensure that the updated noise transfer matrix is ​​a legal probability transfer matrix, that is, the elements of each row of the noise transfer matrix are non-negative, and the sum of the elements of each row is 1. Therefore, after adjusting the parameters of the noise transfer matrix, we can get Figure 3 The matrix w shown t+1 After that, the noise transfer matrix needs to be non-negatively adjusted and row normalized.

[0070] Exemplary non-negative adjustment methods may be: ; That is, all elements in the noise transfer matrix that are less than 0 are set to 0 to ensure that each element in the noise transfer matrix is ​​non-negative.

[0071] The row normalization method can be: ; That is, each row in the noise transfer matrix is ​​normalized so that the sum of the elements in each row is 1, ensuring that each row represents a valid probability distribution.

[0072] In the matrix w t+1 After non-negativity adjustment and row normalization, the optimized noise transfer matrix T can be obtained. t +1 Then repeat the above steps until the noise transfer matrix T that meets the expected requirements is obtained. n .

[0073] The above is merely an example of interactive training of the first model and the second model provided in an embodiment of the present application, and is not limited thereto.

[0074] In this embodiment, the first prediction result output by the first model is interfered with by a preset noise transfer matrix to obtain the second prediction result, so that the second prediction result has a certain degree of disruption, and the attacker cannot directly obtain the true probability distribution (i.e., the first prediction result) output by the first model; the second model is trained based on the second prediction result and multiple query requests, that is, a model cloned by an attacker who simulates stealing the first model, and then the first loss value is calculated according to the first prediction result and the second prediction result, and the second loss value is calculated according to the third prediction result output by the second model and the first prediction result. The parameters of the noise transfer matrix are adjusted with the purpose of making the first loss value less than the first threshold and the second loss value greater than the second threshold, which can not only ensure that the noise transfer matrix will not significantly affect the use of the first model by legitimate users, but also effectively ensure that the attacker cannot accurately steal the first model, thereby achieving effective defense against attack and theft behaviors.

[0075] In one possible design, since not all query requests need to trigger defense, some query requests are normal query requests from legitimate users and do not need to trigger the defense mechanism. That is, the first model mainly defends against abnormal query requests that need to trigger the defense mechanism.

[0076] Therefore, in order to improve the effectiveness and accuracy of the first model trained to defend against theft attacks, the embodiment of the present application further provides a method for screening the training samples (i.e., multiple query requests) input into the first model and the second model before executing step S101. The method specifically includes steps S201 to S203: S201. Construct a behavior recognition model based on the features of normal query requests in historical records.

[0077] Among them, the behavior recognition model is used to identify and classify the input features, and output the category to which the input features belong; the categories include in-distribution data and out-of-distribution data, the in-distribution data is used to indicate that the query request to which the input features belong is a normal query request, and the out-of-distribution data is used to indicate that the query request to which the input features belong is an abnormal query request.

[0078] For example, in order to identify abnormal query requests (ie, out-of-distribution data), it is necessary to first establish a baseline model (ie, behavior recognition model) of normal query requests (ie, in-distribution data).

[0079] Specifically, common statistical features are extracted from normal query requests in historical records, such as: query data contained in the query request, such as images, text, etc.; output of the query request, such as classification results, confidence (i.e. probability) distribution, prediction score, etc.; timestamp (i.e. travel time information, used to monitor query frequency and time interval); user identity, i.e. the unique identifier of the querying user, such as API key, IP address, etc.; request source, client information, etc.

[0080] Since different users or scenarios may have different query habits, different feature sets may be created to indicate different scenarios or different types of features.

[0081] If the query data in the output query request is an image, the visual classification model (VisionTransformer, ViT) can be used for feature extraction to convert the input query data x into an embedding vector x embed ∈R e , where e represents the dimension of the embedding space. In this way, the high-dimensional features of the image can be obtained and converted into an embedding representation that is more suitable for distance calculation. It can be understood that the above feature extraction method is only a possible example. In actual applications, other appropriate methods can be selected according to scene requirements.

[0082] The extracted features are preprocessed to ensure that the features input into the behavior recognition model meet the preset requirements, and the behavior recognition model is trained based on the preprocessed features, and the parameters of the behavior recognition model are adjusted to ensure that the output of the behavior recognition model also meets the expected requirements. The method of training the behavior recognition model is a conventional method and will not be described in detail in the embodiments of this application.

[0083] S202: Input the feature of each query request among the plurality of query requests into the behavior recognition model to obtain the category of each query request output by the behavior recognition model.

[0084] Exemplarily, the feature extraction method used in the above-mentioned training of the behavior recognition model (such as the above-mentioned feature extraction of images based on the visual classification model) is used to extract features from each of the several query requests.

[0085] In a possible embodiment, the data in the distribution includes multiple feature sets, and each feature set in the multiple feature sets is used to indicate a type; the specific implementation of step S202 is: The Mahalanobis distance between the feature of each query request and multiple feature sets is calculated; the category of the query request with a Mahalanobis distance greater than a distance threshold is set as out-of-distribution data; otherwise, it is set as in-distribution data.

[0086] Exemplarily, the following formula is used to calculate the Mahalanobis distance between the feature of each query request and multiple feature sets: ; in, represents the Mahalanobis distance between the features of each query request and multiple feature sets; c represents the category, that is, different feature sets; x embed Embedding vectors (i.e., features) representing the input query request; The mean vector representing the feature set for each category; The covariance matrix representing the set of features for each class.

[0087] By calculating the Mahalanobis distance between the embedding vector (i.e., feature) of each query request and the mean vector of the feature set of multiple categories, we can determine how far each query request is from the in-distribution data based on the Mahalanobis distance, and then determine whether each query request is out-of-distribution data based on the Mahalanobis distance, that is, whether each query request is an abnormal query request.

[0088] Specifically, if the Mahalanobis distance between the query request feature and multiple feature sets is greater than the distance threshold, it means that the query request feature does not belong to any feature set, which means that the query request belongs to out-of-distribution data, and the query request may be an abnormal query request. If the Mahalanobis distance between the query request feature and multiple feature sets is less than the distance threshold, it means that the query request feature belongs to one of the multiple feature sets, the query request belongs to in-distribution data, and the query request is a normal query request.

[0089] S203: Obtain query requests of the category of out-of-distribution data, and obtain multiple query requests.

[0090] It can be understood that the categories of the multiple query requests obtained are all out-of-distribution data, that is, abnormal query requests, and the first model trained based on the abnormal query requests can more effectively and accurately defend against the theft behavior initiated by attackers.

[0091] In a possible embodiment, in order to further avoid making erroneous feedback to legitimate users' erroneous operations, a random number can be generated for each query request of the category of out-of-distribution data; query requests corresponding to all random numbers whose values ​​are not greater than a preset probability threshold are obtained to obtain multiple query requests. In this way, the query requests are further screened, and the obtained multiple query requests are of higher quality, thereby making the subsequent training of the first model more accurate and efficient.

[0092] For one possible design, see Figure 4 , a defense method is also provided for the embodiment of the present application, which is applied to the above-mentioned optimized first model and behavior recognition model, and the specific implementation of the method is as follows: Receive the query request from the user through the API, obtain the query data x included in the query request, and input the query data x into the visual classification model for feature extraction to obtain the output embedding vector x embed ; Based on the embedding vector x embed Input into the behavior recognition model for user behavior analysis, and calculate the embedding vector x embed The Mahalanobis distance between all feature sets is less than the distance threshold (i.e. Figure 4 t in distance ), directly input the query data x into the first model for prediction, and output the first prediction result (i.e. the real prediction result); If the Mahalanobis distance is greater than the distance threshold, a random number r is randomly generated (the value range of r is between 0-1). If the random number r is not greater than the preset threshold p, the query data x is directly input into the first model for prediction, and the first prediction result (that is, the true prediction result) is output; if the random number r is greater than the preset threshold p, the noise transfer matrix needs to be activated for defense, that is, the query data x is input into the first model, and the second prediction result (that is, the prediction result after interference by the noise transfer matrix) is output.

[0093] In this embodiment, by combining user behavior recognition and internal and external distribution detection methods, it is possible to accurately identify whether the user's input is an abnormal query, and has a strong ability to cope with a wide range of unknown attack scenarios, and then accurately determine whether it is necessary to activate the noise transfer matrix for defense, thereby achieving effective defense against attack theft behaviors.

[0094] The above defense method is only a possible example provided by the embodiment of the present application based on the optimized first model, and is not limited to this.

[0095] The method provided by the embodiment of the present application is introduced above, and the device provided by the embodiment of the present application is introduced below.

[0096] Based on the same technical concept, the embodiment of the present application provides a defense model training device, which includes a module / unit / means for executing the method executed by the electronic device in the above method embodiment. The module / unit / means can be implemented by software, or by hardware, or the corresponding software can be implemented by hardware.

[0097] For example, Figure 5 As shown, the device 500 includes: The first processing module 501 is used to: input multiple query requests into the first model, and obtain a first prediction result and a second prediction result output by the first model; the first prediction result is the probability distribution of the category to which the query data in each query request belongs predicted by the first model; the second prediction result is obtained after the first prediction result is interfered by the noise transfer matrix preset in the first model; The second processing module 502 is used to: input multiple query requests into the second model, and obtain a third prediction result output by the second model; the second model is a model trained based on the multiple query requests and the second prediction results, and the third prediction result is a probability distribution of the category to which the query data in each query request belongs predicted by the second model; The optimization module 503 is used to calculate a first loss value between the first prediction result and the second prediction result, and a second loss value between the first prediction result and the third prediction result, and adjust the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than the first threshold and the second loss value greater than the second threshold, so as to obtain the optimized first model.

[0098] Optionally, before the first processing module 501 inputs multiple query requests into the second model, the second processing module 502 is also used to: input the multiple query requests into the initial second model, obtain the fourth prediction result of the initial second model for each of the multiple query requests; calculate the third loss value between the second prediction result and the fourth prediction result, and adjust the initial second model with the purpose of making the third loss value less than a third threshold, so as to obtain the second model.

[0099] Optionally, when the optimization module 503 adjusts the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than the first threshold and the second loss value greater than the second threshold, it is specifically used to: calculate the comprehensive loss value of the first loss value and the second loss value according to a first formula, and the first formula is as follows: ; in, represents the comprehensive loss value, represents the first loss value, represents the second loss value, T represents the preset noise transfer matrix, θ g represents the weight of the second model; λ is a constant; With the goal of making the comprehensive loss value smaller than a fourth threshold, the parameters of the preset noise transfer matrix are adjusted.

[0100] Optionally, before inputting multiple query requests into the first model, the first processing module 501 is also used to: construct a behavior recognition model based on features of normal query requests in historical records; the behavior recognition model is used to identify and classify the input features, and output the category to which the input features belong; the category includes in-distribution data and out-of-distribution data, the in-distribution data is used to indicate that the query request to which the input features belong is a normal query request, and the out-of-distribution data is used to indicate that the query request to which the input features belong is an abnormal query request; the features of each query request among several query requests are input into the behavior recognition model to obtain the category of each query request output by the behavior recognition model; and query requests whose category is out-of-distribution data are obtained to obtain multiple query requests.

[0101] Optionally, the in-distribution data includes multiple feature sets, each of the multiple feature sets is used to indicate a type; when the first processing module 501 inputs the features of each query request among a number of query requests into the behavior recognition model to obtain the category of each query request output by the behavior recognition model, it is specifically used to: calculate the Mahalanobis distance between the features of each query request and the multiple feature sets; set the category of the query request whose Mahalanobis distance is greater than the distance threshold as out-of-distribution data; otherwise, set it to in-distribution data.

[0102] Optionally, when the first processing module 501 obtains a query request whose category is out-of-distribution data and obtains multiple query requests, it is specifically used to: generate a random number for each query request whose category is out-of-distribution data; obtain query requests corresponding to all random numbers whose values ​​are not greater than a preset probability threshold, and obtain multiple query requests.

[0103] It should be understood that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0104] Based on the same technical concept, see Figure 6 The embodiment of the present application further provides an electronic device 600, including: At least one processor 601; and a communication interface 603 that is communicatively connected to the at least one processor 601; the at least one processor 601 executes instructions stored in the memory 602, so that the electronic device 600 executes the method steps performed by the billboard in the above method embodiment through the communication interface 603.

[0105] Optionally, the memory 602 is located outside the electronic device 600 .

[0106] Optionally, the electronic device 600 includes the memory 602, the memory 602 is connected to the at least one processor 601, and the memory 602 has instructions that can be executed by the at least one processor 601. Figure 6The dashed lines indicate that the memory 602 is optional for the electronic device 600 .

[0107] The at least one processor 601 and the memory 602 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0108] The specific connection medium between the at least one processor 601, the memory 602 and the communication interface 603 is not limited in the embodiment of the present application. Figure 6 In the embodiment, at least one processor 601, a memory 602 and a communication interface 603 are connected via a bus 604. Figure 6 The connection between other components is only for schematic illustration and is not intended to be limiting. The bus part may be an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 It is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.

[0109] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.

[0110] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0111] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which acts as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DirectRambus RAM, DR RAM).

[0112] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0113] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0114] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the computer executes the method steps executed by any device in the above method embodiments.

[0115] Based on the same technical concept, an embodiment of the present application also provides a computer program product, including computer program code. When the computer program code is executed on a computer, the method steps executed by any device in the above method embodiment are implemented.

[0116] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0117] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0120] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A defense model training method, characterized in that: include: Inputting multiple query requests into a first model, and obtaining a first prediction result and a second prediction result output by the first model; The first prediction result is a probability distribution of the category to which the query data in each query request belongs predicted by the first model; The second prediction result is obtained after the first prediction result is interfered by the noise transfer matrix preset in the first model; Inputting the multiple query requests into a second model, and obtaining a third prediction result output by the second model; The second model is a model trained based on the multiple query requests and the second prediction results, and the third prediction result is a probability distribution of the category to which the query data in each query request belongs predicted by the second model; Calculate a first loss value between the first prediction result and the second prediction result, and a second loss value between the first prediction result and the third prediction result, and adjust the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than a first threshold and the second loss value greater than a second threshold to obtain an optimized first model.

2. The method according to claim 1, characterized in that Before inputting the plurality of query requests into the second model, the method further includes: Inputting the multiple query requests into the initial second model, and obtaining a fourth prediction result of the initial second model for each query request in the multiple query requests; A third loss value between the second prediction result and the fourth prediction result is calculated, and with the purpose of making the third loss value less than a third threshold, the initial second model is adjusted to obtain the second model.

3. The method according to claim 1, characterized in that The step of adjusting the parameters of the preset noise transfer matrix with the purpose of making the first loss value smaller than a first threshold and the second loss value larger than a second threshold includes: The combined loss value of the first loss value and the second loss value is calculated according to a first formula, wherein the first formula is as follows: ； Among them, the represents the comprehensive loss value, the represents the first loss value, the represents the second loss value, T represents the preset noise transfer matrix, and θ g represents the weight of the second model; the λ is a constant; With the goal of making the comprehensive loss value smaller than a fourth threshold, the parameters of the preset noise transfer matrix are adjusted.

4. The method according to claim 1, characterized in that Before inputting the plurality of query requests into the first model, the method further includes: A behavior recognition model is constructed based on the features of normal query requests in the historical records; the behavior recognition model is used to recognize and classify the input features, and output the category to which the input features belong; the category includes in-distribution data and out-of-distribution data, the in-distribution data is used to indicate that the query request to which the input features belong is a normal query request, and the out-of-distribution data is used to indicate that the query request to which the input features belong is an abnormal query request; Inputting a feature of each query request among the plurality of query requests into the behavior recognition model to obtain a category of each query request output by the behavior recognition model; The query request whose category is out-of-distribution data is obtained to obtain the multiple query requests.

5. The method according to claim 4, characterized in that The distributed data includes multiple feature sets, each of the multiple feature sets is used to indicate a type; the inputting the feature of each query request in the plurality of query requests into the behavior recognition model to obtain the category of each query request output by the behavior recognition model includes: Calculating the Mahalanobis distance between the feature of each query request and the multiple feature sets; The category of the query request whose Mahalanobis distance is greater than the distance threshold is set as out-of-distribution data; otherwise, it is set as in-distribution data.

6. The method according to claim 4 or 5, characterized in that The obtaining of the query request for out-of-distribution data, obtaining the multiple query requests, includes: For each query request of the category of out-of-distribution data, generate a random number; Query requests corresponding to all random numbers whose values ​​are not greater than a preset probability threshold are obtained to obtain the multiple query requests.

7. A defense model training device, characterized in that: include: A first processing module is used to: input multiple query requests into a first model, and obtain a first prediction result and a second prediction result output by the first model; the first prediction result is a probability distribution of the category to which the query data in each query request belongs predicted by the first model; the second prediction result is obtained after the first prediction result is interfered by a noise transfer matrix preset in the first model; A second processing module, used to: input the multiple query requests into a second model, and obtain a third prediction result output by the second model; The second model is a model trained based on the multiple query requests and the second prediction results, and the third prediction result is a probability distribution of the category to which the query data in each query request belongs predicted by the second model; The optimization module is used to calculate a first loss value between the first prediction result and the second prediction result, and a second loss value between the first prediction result and the third prediction result, and adjust the parameters of the preset noise transfer matrix with the purpose of making the first loss value less than a first threshold and the second loss value greater than a second threshold, so as to obtain an optimized first model.

8. The device according to claim 7, characterized in that When the optimization module adjusts the parameters of the preset noise transfer matrix with the purpose of making the first loss value smaller than a first threshold and the second loss value larger than a second threshold, the optimization module is specifically used to: The combined loss value of the first loss value and the second loss value is calculated according to a first formula, wherein the first formula is as follows: ； Among them, the represents the comprehensive loss value, the represents the first loss value, the represents the second loss value, T represents the preset noise transfer matrix, and θ g represents the weight of the second model; the λ is a constant; With the purpose of making the comprehensive loss value smaller than the fourth loss value, the parameters of the preset noise transfer matrix are adjusted.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method according to any one of claims 1 to 6 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to have a computer program, and the computer program includes program instructions. When the program instructions are executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images