Method and apparatus for protecting against operator attacks

Abstract

The application provides a method and device for protecting against operator attacks. The method for protecting against operator attacks comprises the following steps: obtaining an operator list of a first model, wherein the operator list comprises information of at least one operator used by the first model; in a case where the behavior of a first operator is determined to be malicious attack during the running of the first model, determining that the first operator is an attack operator, wherein the first operator is one of the operators in the operator list; and blocking the attack operator. The application can avoid the risk of model parameter theft and reasoning result tampering.

Description

Technical Field

[0001] This application relates to artificial intelligence (AI) technology, and more particularly to a method and apparatus for protecting against operator attacks. Background Technology

[0002] In technologies related to Trusted Execution Environments (TEEs), if an attacker tampers with the operator code of the parent instance or customizes malicious operators, thereby bypassing access to protected assets, they may steal model parameters during the training / inference process of an AI model.

[0003] Therefore, how to eliminate malicious attacks on operators in the AI ​​model training / inference framework, thereby avoiding model parameter theft, inference result tampering, etc., has become a problem to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for protecting against operator attacks, so as to avoid the risk of model parameter theft and inference result tampering.

[0005] In a first aspect, this application provides a method for protecting against operator attacks, comprising: obtaining an operator list of a first model, the operator list including information on at least one operator used by the first model; during the operation of the first model, when the behavior of the first operator is determined to be a malicious attack, identifying the first operator as an attack operator, the first operator being one of the operators in the operator list; and blocking the attack operator.

[0006] In this embodiment, by obtaining the list of operators of the model and then blocking operators in the list that are determined to be malicious attacks during the model's operation, malicious attacks on operators in the AI ​​model training / inference framework can be eliminated, avoiding the risk of model parameter theft and inference result tampering.

[0007] The first model can include models trained using any AI technology, such as open-source models like the Pangu open-source series, or models trained using frameworks like PyTorch and MindSpore. This application does not impose any specific limitations on this. The operator list of the first model includes information about one or more operators used by the first model, such as operator names and descriptions of operator behavior.

[0008] Optionally, during the deployment phase of the first model, the operators used by the first model can be analyzed to obtain a list of operators for the first model (also known as a whitelist of operators for the first model). In one possible implementation, data acquisition statements are inserted at the beginning and end of the startup script of the first model, which is trained based on multiple runtime environments of the first model. Then, operator data of the first model is obtained based on the acquisition statements, which includes multiple operator names. Duplicate operator names are removed from the multiple operator names to obtain the list of operators for the first model. The aforementioned operation of inserting acquisition statements is a high-level operation that does not require awareness of the underlying layer, i.e., it does not depend on the hardware environment, thus enabling automated extraction of model operators.

[0009] It should be noted that the above describes a method for obtaining the operator list of the first model, but this is not a limitation on the method. Other methods can also be used to obtain the operator list of the first model, and no specific limitation is made in this application.

[0010] In one possible implementation, if the behavior of the first operator is determined to be a malicious attack, the following steps are taken: If the initial address of the first operator in memory and its runtime address in memory are different, the behavior of the first operator is determined to be a malicious attack. The initial address of the first operator in memory can be obtained. The first operator is any one of at least one operators in the operator list of the first model. During the operation of the first model, the runtime address of the first operator in memory is obtained. When the runtime address differs from the initial address, the first operator is considered an attack operator. By comparing the initial address and the runtime address of the first operator, malicious attack behavior of the first operator can be detected in real time during the operation of the first model, and the first operator can be blocked in a timely manner, thereby quickly preventing malicious attacks by the first operator and achieving dynamic protection of the first model.

[0011] For example, first, the original memory addresses of the operators in the operator list of the first model are recorded. Then, during the operation of the first model, the runtime addresses of the operators are compared. If the runtime address of an operator changes compared to its original address, it indicates that there is a malicious operator tampering problem implemented by dynamic replacement, and the operator is marked as an attack operator.

[0012] In one possible implementation, if the behavior of the first operator is determined to be a malicious attack, the following steps are taken: obtaining an operator attack behavior knowledge base, which includes description information of dangerous functions; during the operation of the first model, if the first operator calls any dangerous function, the behavior of the first operator is determined to be a malicious attack.

[0013] In one possible implementation, dynamic substitution can be used to instrument dangerous functions, which can be any one or more from the operator attack behavior knowledge base. The invocation of dangerous functions is detected based on instrumentation. When a dangerous function is invoked, its call stack is retrieved. If the first operator appears in the call stack of a dangerous function, then the first operator is determined to have invoked the dangerous function. That is, when the runtime environment of the first model contains a first function with descriptive information corresponding to at least one operator attack behavior, this first function is a dangerous function. The call stack of the dangerous function is retrieved; if the call stack of the dangerous function determines that the first operator invoked the dangerous function, then the first operator is identified as an attack operator. For example, using Python's dynamic substitution capability (monkey patching), dangerous functions involved in the operator attack behavior knowledge base can be instrumented. Then, based on instrumentation, the invocation of dangerous functions is detected. When a dangerous function is invoked, its operation call stack is examined to determine which operator invoked it, and that operator is marked as an attack operator. By instrumenting the corresponding dangerous functions according to the description information of dangerous functions in the operator attack behavior knowledge base, when a dangerous function is called, the first operator appearing in its call stack is identified as a dangerous operator and then blocked. This can detect existing first operator attack behavior in the first model operation and block the first operator, thereby preventing malicious attacks by the first operator and protecting the first model.

[0014] It should be understood that the structure of the operator attack behavior knowledge base has been described for example above, but this is not intended to limit the operator attack behavior knowledge base. This application does not specifically limit the structure and implementation of the operator attack behavior knowledge base.

[0015] In one possible implementation, the method for obtaining the operator attack behavior knowledge base may include: obtaining a common list of operators, which includes information on multiple operators corresponding to multiple models; constructing a function call chain for a second operator in the common list of operators, where the second operator is any one of the multiple operators; identifying the attack behavior of the second operator based on the function call chain of the second operator and the functional information of the second operator (the functional information may include the functional definition information, functional description information, functional feature knowledge of the operator, etc.); and adding the attack behavior of the second operator to the operator attack behavior knowledge base.

[0016] The process of obtaining the common operator list includes: inserting acquisition statements at the beginning and end of the startup scripts of multiple models, the startup scripts of multiple models being trained based on multiple running environments of multiple models; acquiring operator data of multiple models based on the acquisition statements, the operator data including multiple operator names; removing duplicate operator names from the multiple operator names to obtain the operator list of multiple models; and extracting the common operator list by intersecting the operator lists of multiple models.

[0017] The process of identifying the attack behavior of the second operator based on the function call chain and functional information of the second operator includes: obtaining the functions called by the second operator based on the function call chain of the second operator; identifying the functions called by the second operator based on the functional information of the second operator to determine the functions that cannot be called by the second operator; and obtaining the attack behavior of the second operator based on the functions that cannot be called by the second operator.

[0018] Once identified as an attack operator, the attack operator can be blocked, thereby preventing malicious attacks by the attack operator and protecting the first model.

[0019] Based on the protection methods described above, and exemplarily, based on malicious behavior analysis of operator behavior, a runtime malicious operator protection capability is constructed. The specific steps are as follows:

[0020] (1) Record the original memory address of the operator in the operator list of the first model.

[0021] (2) During the operation of the first model, the runtime addresses of operators are compared. If the runtime address of an operator changes compared to its original address, it indicates a malicious operator tampering problem implemented through dynamic replacement. This operator is marked as an attack operator and blocked. If the runtime address of an operator does not change compared to its original address, subsequent steps can be executed. Based on address comparison, malicious attack behavior of operators can be detected in real time during the operation of the first model, and such operators can be blocked in a timely manner, thereby quickly preventing malicious attacks by such operators and achieving dynamic protection of the first model.

[0022] (3) By leveraging the dynamic substitution capability (monkey patching) of the Python language, dangerous functions involved in the operator attack behavior knowledge base are instrumented and subsequent detection is performed. Since the instrumentation and detection of dangerous functions do not affect the original training and inference performance, they do not incur additional overhead for model computation.

[0023] (4) Examine the call stack of dangerous functions to determine which operator calls them. Mark the operator as an attack operator and block it. A knowledge base of operator attack behavior is used to implement protection against operator attacks. It only instrumentes dangerous functions, thus only monitoring the call operations of dangerous functions. When a dangerous function is not called, there is zero overhead on the original training / inference process of the model. However, when a dangerous function is called, the operator calling the dangerous function is directly blocked, thereby eliminating malicious operator attacks in the AI ​​model training / inference framework and avoiding the risks of model parameter theft and inference result tampering.

[0024] It should be noted that the above steps are merely an example of one implementation method of this application and do not constitute a limitation on the scope of this application.

[0025] Secondly, this application provides a protection device against operator attacks, comprising: an acquisition module for acquiring an operator list of a first model, the operator list including information on at least one operator used by the first model; an analysis module for determining that the first operator is an attack operator when the behavior of the first operator is determined to be a malicious attack during the operation of the first model, the first operator being one of the operators in the operator list; and a blocking module for blocking the attack operator.

[0026] In one possible implementation, the acquisition module is specifically used to insert acquisition statements at the beginning and end of the startup script of the first model, the startup script of the first model being trained based on multiple runtime environments of the first model; acquire operator data of the first model based on the acquisition statements, the operator data including multiple operator names; and remove duplicate operator names from the multiple operator names to obtain the operator list of the first model.

[0027] In one possible implementation, the analysis module is specifically used to determine that the behavior of the first operator is a malicious attack when the initial address of the first operator in memory and the running address of the first operator in memory are different.

[0028] In one possible implementation, the analysis module is specifically used to acquire an operator attack behavior knowledge base, which includes description information of dangerous functions; during the operation of the first model, if the first operator calls any of the dangerous functions, the behavior of the first operator is determined to be a malicious attack.

[0029] In one possible implementation, the analysis module is specifically used to instrument the dangerous function using dynamic replacement, wherein the dangerous function is any one or more of the operator attack behavior knowledge base; detect the calling status of the dangerous function based on the instrumentation; when the dangerous function is called, obtain the call stack of the dangerous function; if the first operator appears in the call stack of the dangerous function, then the first operator calls the dangerous function.

[0030] In one possible implementation, the acquisition module is further configured to acquire a common operator list, the common operator list including information of multiple operators corresponding to multiple models; construct a function call chain of a second operator, the second operator being any one of the multiple operators; identify the attack behavior of the second operator based on the function call chain of the second operator and the functional information of the second operator; and include the attack behavior of the second operator in the operator attack behavior knowledge base.

[0031] In one possible implementation, the acquisition module is specifically used to insert acquisition statements at the beginning and end of the startup scripts of the multiple models, the startup scripts of the multiple models being trained based on multiple runtime environments of the multiple models; acquire operator data of the multiple models based on the acquisition statements, the operator data including multiple operator names; remove duplicate operator names from the multiple operator names to obtain an operator list of the multiple models; and extract the common operator list by intersecting the operator lists of the multiple models.

[0032] In one possible implementation, the acquisition module is specifically used to acquire functions called by the second operator based on the function call chain of the second operator; identify the functions called by the second operator based on the functional information of the second operator, and determine functions that cannot be called by the second operator; and acquire the attack behavior of the second operator based on the functions that cannot be called by the second operator.

[0033] Thirdly, this application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method as described in any one of the first aspects above.

[0034] Fourthly, this application provides a computer-readable storage medium including a computer program that, when executed on a computer, causes the computer to perform the method described in any one of the first aspects above.

[0035] Fifthly, this application provides a computer program product comprising computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects above. Attached Figure Description

[0036] Figure 1 This is a schematic block diagram of a device 100 according to this application;

[0037] Figure 2 This is a schematic diagram of the model's operating environment;

[0038] Figure 3 A flowchart of process 300 for the operator attack protection method provided in the embodiments of this application;

[0039] Figure 4 Flowchart for extracting the operator list for the first model;

[0040] Figure 5 A diagram illustrating a knowledge base for operator attack behaviors;

[0041] Figure 6 A flowchart for constructing a knowledge base of operator attack behaviors;

[0042] Figure 7 A flowchart illustrating real-time protection against malicious operators during the runtime of the first model;

[0043] Figure 8 This is an architecture diagram of protection technologies against operator attacks.

[0044] Figure 9 This is a schematic diagram of the structure of the operator attack protection device 900 of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0047] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0048] Figure 1 This is a schematic block diagram of a device 100 according to this application, such as Figure 1As shown, the device 100 may include a processor 101 and a transceiver / transceiver pin 102, and optionally, a memory 103.

[0049] The various components of device 100 are coupled together via bus 104, which includes a data bus, a power bus, a control bus, and a status signal bus. For clarity, all buses are referred to as bus 104 in the figure.

[0050] Optionally, the memory 103 can be used to store instructions in the embodiments of this application. The processor 101 can be used to execute the instructions in the memory 103, control the receive pin to receive signals, and control the transmit pin to transmit signals.

[0051] Device 100 may be an electronic device or a chip of an electronic device in the embodiments of this application.

[0052] In this application, all relevant content of each step involved in the embodiments can be referenced to the functional description of the corresponding functional module, and can be referred to the embodiments below.

[0053] Figure 2 This is a schematic diagram of the model's operating environment, such as... Figure 2 As shown, in this operating environment, under typical training / inference scenarios, due to the requirement that user data not leave the domain, it is necessary to build L1 / L2 large models based on the L0 large model in a private cloud environment. The L0 model is located in an independent bucket, and Enclave sub-instances are used to protect the runtime large model assets. Existing confidential computing capabilities are used to protect the data from the embedded neural network processing unit (NPU) to the NPU computation process.

[0054] 1. Training:

[0055] 1.1 Create an instance using the L0 large model in the bucket;

[0056] 1.2 Training starts the trusted user application, loads user data from the user bucket, and accesses confidential assets through a secure channel;

[0057] 1.3 Execute the script to start the training operation and load the model through the NPU driver of the confidential instance;

[0058] 1.4 The model is loaded onto the computing resources of the internally deployed private cloud for training. The confidential computing capabilities of the NPU ensure the confidentiality of data during the NPU driver-to-NPU computing process.

[0059] 1.5 Return the trained L1 / L2 large model.

[0060] 2. Reasoning:

[0061] 2.1 Load the pre-trained L1 / L2 large model from the user bucket and perform computation on the internal private cloud computing resources.

[0062] If a malicious user attempts to steal model assets or personal data, the following may occur:

[0063] 2.2 Creating malicious operators.

[0064] 2.3 Tampering operators.

[0065] 2.4 Loading malicious operators during training and push.

[0066] The operator attack protection method of this application can load an operator protection engine in step 1.3, and combine the operator attack behavior knowledge base and the operator list constructed by static analysis of the model to block maliciously tampered model operators in real time, thereby preventing the risk of theft of model assets in the running state.

[0067] Since this application involves artificial intelligence (AI) technology, for ease of understanding, some AI-related terms or concepts used in this application are explained below, and these terms or concepts are also part of the content of the invention.

[0068] (1) Compute Architecture for Neural Networks (CANN) is a heterogeneous computing architecture specifically designed and optimized for the high-performance computing needs of neural networks. As a heterogeneous computing architecture for AI processors, CANN supports a variety of mainstream AI frameworks, including MindSpore and PyTorch, and provides more than 1,200 basic operators.

[0069] The main functions and features of CANN:

[0070] 1. Supports multiple AI frameworks: CANN supports multiple mainstream AI frameworks in the industry, including MindSpore, PyTorch, etc.

[0071] 2. Programming Interface: CANN provides an open and easy-to-use Ascend Computing Language (ACL) programming interface, which supports graph-level and operator-level compilation optimization and automatic tuning of network models.

[0072] 3. Performance optimization: CANN has a multi-level architecture, including the Da Vinci architecture at the hardware level and full-stack support at the software level. It is designed to provide a powerful hardware foundation and a software stack for managing network models, computational flow and data flow to support the efficient execution of neural networks on heterogeneous processors.

[0073] Application scenarios and development process of CANN:

[0074] 1. Application Development: CANN supports multiple AI frameworks, allowing developers to utilize its provided programming interfaces and tools for efficient application development. For example, the CANN community offers numerous online demo examples, which developers can use for learning and further development.

[0075] 2. Performance Optimization: CANN provides functions such as graph optimization, graph compilation, and graph execution for network models, making it easier for developers to optimize overall network performance. In addition, CANN includes tools such as the Huawei Collective Communication Library (HCCL) and the Ascend Operator Library (AOL), which are used to improve network congestion and enhance the performance of large models, respectively.

[0076] (2) Compute Unified Device Architecture (CUDA)

[0077] CUDA is a general-purpose parallel computing architecture that enables graphics processing units (GPUs) to solve complex computational problems. It includes the CUDA instruction set architecture and the parallel computing engine within the GPU. Developers can use the C language to write programs for the CUDA architecture, and these programs can run at extremely high performance on CUDA-enabled processors. CUDA 3.0 has begun to support C++ and FORTRAN.

[0078] Based on the above content, the technical solution of the operator attack protection method provided in this application will be described below.

[0079] Figure 3 This is a flowchart of process 300 of the operator attack protection method provided in an embodiment of this application. Process 300 can be executed by the apparatus 100 described above. Process 300 is described as a series of steps or operations, and it should be understood that process 300 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 3 The execution order is shown. Process 300 may include:

[0080] Step 301: Obtain the operator list of the first model. The operator list includes information on at least one operator used by the first model.

[0081] The first model can include models trained using any AI technology, such as open-source models like the Pangu open-source series, or models trained using frameworks like PyTorch and MindSpore. This application does not impose any specific limitations on this. The operator list of the first model includes information about one or more operators used by the first model, such as operator names and descriptions of operator behavior.

[0082] Optionally, during the deployment phase of the first model, the operators used by the first model can be analyzed to obtain a list of operators for the first model (also known as an operator whitelist). In one possible implementation, data collection statements can be inserted at the beginning and end of the startup script of the first model, which is trained based on multiple runtime environments of the first model. Then, operator data of the first model, including multiple operator names, can be obtained based on the collection statements. Duplicate operator names are removed from these names to obtain the operator list for the first model. For example, Figure 4 Extract the flowchart for the operator list of the first model, such as Figure 4 As shown, during the deployment phase of the first model, the first model is analyzed using the performance analysis tools (e.g., profiling) built into the framework (e.g., PyTorch, MindSpore, etc.). These tools can obtain operator execution data for the first model at various time points, allowing the acquisition of operator data during runtime (including operator names appearing at each time point). Duplicate operator names are then removed from these time point names to construct the operator list for the first model. Specifically, profiling collection statements are inserted at the beginning and end of the first model's execution script using the performance analysis tools built into frameworks like PyTorch and MindSpore. Then, during the first model's loading, the JSON format data of the first model's profiling (corresponding to the aforementioned operator data) is automatically acquired. From the JSON format data collected by profiling, timeline data of type 'op' (including operator names) is automatically extracted, duplicate operator names are removed, and the operator list for the first model is obtained. The aforementioned insertion and acquisition statement operation is a high-level operation that does not require awareness of the underlying layer, meaning it does not depend on the hardware environment. Therefore, it can achieve the automatic extraction capability of model operators.

[0083] It should be noted that the above description exemplarily describes a method for obtaining the operator list of the first model, but this is not a limitation on the method. Other methods can also be used to obtain the operator list of the first model, and no specific limitation is made in this regard.

[0084] Step 302: During the operation of the first model, if the behavior of the first operator is determined to be a malicious attack, the first operator is identified as the attack operator.

[0085] In one possible implementation, if the behavior of the first operator is determined to be a malicious attack, the following steps are taken: If the initial address of the first operator in memory and its runtime address in memory are different, the behavior of the first operator is determined to be a malicious attack. The initial address of the first operator in memory can be obtained. The first operator is any one of at least one operators in the operator list of the first model. During the operation of the first model, the runtime address of the first operator in memory is obtained. When the runtime address differs from the initial address, the first operator is considered an attack operator. By comparing the initial address and the runtime address of the first operator, malicious attack behavior of the first operator can be detected in real time during the operation of the first model, and the first operator (i.e., the attack operator) can be blocked in a timely manner, thereby quickly preventing malicious attacks by the first operator and achieving dynamic protection of the first model.

[0086] For example, first, the original memory addresses of the operators in the operator list of the first model are recorded. Then, during the operation of the first model, the runtime addresses of the operators are compared. If the runtime address of an operator changes compared to its original address, it indicates that there is a malicious operator tampering problem implemented by dynamic replacement, and the operator is marked as an attack operator.

[0087] In one possible implementation, if the behavior of the first operator is determined to be a malicious attack, the following steps are taken: obtaining an operator attack behavior knowledge base, which includes description information of dangerous functions; during the operation of the first model, if the first operator calls any dangerous function, the behavior of the first operator is determined to be a malicious attack.

[0088] In one possible implementation, dynamic substitution can be used to instrument dangerous functions. These dangerous functions can be any one or more from a knowledge base of operator attack behaviors. The invocation of dangerous functions is detected based on instrumentation. When a dangerous function is invoked, its call stack is retrieved. If the first operator appears in the call stack of a dangerous function, then the first operator invokes the dangerous function. That is, when the runtime environment of the first model contains a first function with descriptive information corresponding to at least one operator attack behavior, this first function is a dangerous function. The call stack of the dangerous function is retrieved. If the call stack of the dangerous function determines that the first operator invokes the dangerous function, then the first operator is identified as an attack operator. For example, using Python's dynamic substitution capability (monkey patching), dangerous functions involved in the operator attack behavior knowledge base can be instrumented. Then, based on instrumentation, the invocation of dangerous functions is detected. When a dangerous function is invoked, its call stack is checked to determine which operator invoked it, and that operator is marked as an attack operator. By instrumenting the corresponding dangerous functions according to the description information of dangerous functions in the operator attack behavior knowledge base, when a dangerous function is called, the first operator appearing in its call stack is identified as a dangerous operator and then blocked. This can detect the existing first operator attack behavior in the first model operation and block the first operator, thereby eliminating malicious operator attacks in the AI ​​model training / inference framework and avoiding the risk of model parameter theft and inference result tampering.

[0089] For example, Figure 5 A diagram illustrating a knowledge base for operator attack behaviors, such as... Figure 5 As shown, combining operator definition patterns, attack scenarios involving stealing model parameters, and static behavior analysis of common operators in the large model, the abnormal behavior of operators can be divided into two main categories: network behavior and local behavior. Local behavior involves four attack scenarios: suspicious API calls, DLL calls, file operations, and abnormal process / thread operations. Network behavior involves one attack scenario: network operations. A knowledge base of attack behaviors for common operators in the large model is constructed based on these abnormal operator behaviors. At the code level, malicious operator behaviors can be recorded using JSON format. The first level is network or local behavior, and the second level is the classification of malicious behavior (i.e., attack scenarios). Under various attack scenarios, it also includes the basic class libraries involved in the malicious behavior, such as "open" in file operation scenarios and "socket" in network operation scenarios. This operator attack behavior knowledge base can be extended according to the current format and loaded and run in real-time malicious operator interception. It should be understood that the above description of the structure of the operator attack behavior knowledge base is exemplary, but this is not a limitation on the operator attack behavior knowledge base. This application does not specifically limit the structure and implementation method of the operator attack behavior knowledge base.

[0090] In one possible implementation, the method for obtaining the operator attack behavior knowledge base may include: obtaining a common list of operators, which includes information on multiple operators corresponding to multiple models; constructing a function call chain for a second operator in the common list of operators, where the second operator is any one of the multiple operators; identifying the attack behavior of the second operator based on the function call chain of the second operator and the functional information of the second operator (the functional information may include the functional definition information, functional description information, functional feature knowledge of the operator, etc.); and adding the attack behavior of the second operator to the operator attack behavior knowledge base.

[0091] The process of obtaining the common operator list includes: inserting acquisition statements at the beginning and end of the startup scripts of multiple models, where the startup scripts are trained based on multiple runtime environments of the multiple models; acquiring operator data for each model based on the acquisition statements, where the operator data includes multiple operator names; removing duplicate operator names from the multiple operator names to obtain the operator list for multiple models; and extracting the common operator list by intersecting the operator lists of multiple models. This process is similar to the method for obtaining the operator list of the first model in step 301, except that the common operator list is the intersection of the operator lists of multiple individual models.

[0092] The process of identifying the attack behavior of the second operator based on the function call chain and functional information of the second operator includes: obtaining the functions called by the second operator based on the function call chain of the second operator; identifying the functions called by the second operator based on the functional information of the second operator to determine the functions that cannot be called by the second operator; and obtaining the attack behavior of the second operator based on the functions that cannot be called by the second operator.

[0093] For example, Figure 6 A flowchart illustrating the construction process of a knowledge base for operator attack behaviors, such as... Figure 6As shown, profiling tools are used to analyze several large open-source models. First, a function call graph (i.e., the sequence of function calls) is constructed based on the operator's native file. Using profiling tools, operator calls in models such as the Pangu open-source series are analyzed to obtain a list of common operators. Then, based on the list of common operators and the native file, the mapping relationship between operators and operators in the Compute Architecture for Neural Networks (CANN) and Compute Unified Device Architecture (CUDA) layers is obtained. This leads to the construction of a common operator function call graph, achieving basic coverage of the call relationships of common operators in large models, including the function call graph of the second operator (which is any one of the multiple operators in the common operator list). Next, static analysis is performed on the function call graph of the second operator to obtain the functions called by the second operator. Then, combined with the functional information of the second operator, the functions called by the second operator are identified, determining functions that cannot be called by the second operator under the functions implemented by the second operator. Based on these functions, potential malicious attack behaviors of the second operator are identified and added to the operator attack behavior knowledge base.

[0094] Step 303: Block the attack operator.

[0095] Once identified as an attack operator, the attack operator can be blocked, thereby preventing malicious attacks by the attack operator and protecting the first model.

[0096] Based on the protection method described above, for example, Figure 7 The flowchart for real-time protection against malicious operators during the runtime of the first model is as follows: Figure 7 As shown, based on the analysis of malicious behavior of operators, the runtime malicious operator protection capability is constructed. The specific steps are as follows:

[0097] (1) Record the original memory address of the operator in the operator list of the first model.

[0098] (2) During the operation of the first model, the runtime addresses of operators are compared. If the runtime address of an operator changes compared to its original address, it indicates a malicious operator tampering problem implemented through dynamic replacement. This operator is marked as an attack operator and blocked. If the runtime address of an operator does not change compared to its original address, subsequent steps can be executed. Based on address comparison, malicious attack behavior of operators can be detected in real time during the operation of the first model, and such operators can be blocked in a timely manner, thereby quickly preventing malicious attacks by such operators and achieving dynamic protection of the first model.

[0099] (3) By leveraging the dynamic substitution capability (monkey patching) of the Python language, dangerous functions involved in the operator attack behavior knowledge base are instrumented and subsequent detection is performed. Since the instrumentation and detection of dangerous functions do not affect the original training and inference performance, they do not incur additional overhead for model computation.

[0100] (4) Examine the call stack of dangerous functions to determine which operator calls them. Mark the operator as an attack operator and block it. A knowledge base of operator attack behavior is used to implement protection against operator attacks. It only instrumentes dangerous functions, thus only monitoring the call operations of dangerous functions. When a dangerous function is not called, there is zero overhead on the original training / inference process of the model. However, when a dangerous function is called, the operator calling the dangerous function is directly blocked, thereby eliminating malicious operator attacks in the AI ​​model training / inference framework and avoiding the risks of model parameter theft and inference result tampering.

[0101] It should be noted that the above steps are merely an example of one implementation method of this application and do not constitute a limitation on the scope of this application.

[0102] In this embodiment, by obtaining the list of operators of the model and then blocking operators in the list that are determined to be malicious attacks during the model's operation, malicious attacks on operators in the AI ​​model training / inference framework can be eliminated, avoiding the risk of model parameter theft and inference result tampering.

[0103] For example, Figure 8 An architecture diagram of protection technologies against operator attacks, such as Figure 8 As shown, the architecture consists of two parts:

[0104] 1. During the model or service deployment phase, perform static analysis on the model and construct a list of operators used by the model to ensure that the model's operators are known and controllable. This part can be referred to the description of step 301 above, and will not be repeated here.

[0105] 2. During the model training and inference phases, an operator protection engine is loaded based on a knowledge base of operator attack behaviors built from the open-source model. This engine blocks maliciously modified model operators in real time, preventing the risk of theft of model assets during runtime. This part can be referred to in steps 302 and 303 above, and will not be repeated here.

[0106] Figure 9 This is a schematic diagram of the structure of the operator attack protection device 900 of this application, as shown below. Figure 9 As shown, the operator attack protection device 900 of this embodiment can be applied to the aforementioned electronic device. The operator attack protection device 900 may include: an acquisition module 901, an analysis module 902, and a blocking module 903. Wherein,

[0107] The acquisition module 901 is used to acquire the operator list of the first model, the operator list including information on at least one operator used by the first model; the analysis module 902 is used to determine that the first operator is an attack operator when the behavior of the first operator is determined to be a malicious attack during the operation of the first model, the first operator being one of the operators in the operator list; the blocking module 903 is used to block the attack operator.

[0108] In one possible implementation, the acquisition module 901 is specifically used to insert acquisition statements at the beginning and end of the startup script of the first model, the startup script of the first model being trained based on multiple running environments of the first model; acquire operator data of the first model based on the acquisition statements, the operator data including multiple operator names; and remove duplicate operator names from the multiple operator names to obtain the operator list of the first model.

[0109] In one possible implementation, the analysis module 902 is specifically used to determine that the behavior of the first operator is a malicious attack when the initial address of the first operator in memory and the running address of the first operator in memory are different.

[0110] In one possible implementation, the analysis module 902 is specifically used to acquire an operator attack behavior knowledge base, which includes description information of dangerous functions; during the operation of the first model, if the first operator calls any of the dangerous functions, the behavior of the first operator is determined to be a malicious attack.

[0111] In one possible implementation, the analysis module 902 is specifically used to instrument the dangerous function using dynamic replacement, wherein the dangerous function is any one or more in the operator attack behavior knowledge base; detect the calling status of the dangerous function based on the instrumentation; when the dangerous function is called, obtain the call stack of the dangerous function; if the first operator appears in the call stack of the dangerous function, then the first operator calls the dangerous function.

[0112] In one possible implementation, the acquisition module 901 is further configured to acquire a common operator list, the common operator list including information of multiple operators corresponding to multiple models; construct a function call chain of a second operator, the second operator being any one of the multiple operators; identify the attack behavior of the second operator based on the function call chain of the second operator and the functional information of the second operator; and include the attack behavior of the second operator in the operator attack behavior knowledge base.

[0113] In one possible implementation, the acquisition module 901 is specifically used to insert acquisition statements at the beginning and end of the startup scripts of the multiple models, the startup scripts of the multiple models being trained based on multiple runtime environments of the multiple models; acquire operator data of the multiple models based on the acquisition statements, the operator data including multiple operator names; remove duplicate operator names from the multiple operator names to obtain an operator list of the multiple models; and extract the common operator list by intersecting the operator lists of the multiple models.

[0114] In one possible implementation, the acquisition module 901 is specifically used to acquire the functions called by the second operator according to the function call chain of the second operator; identify the functions called by the second operator according to the functional information of the second operator, and determine the functions that cannot be called by the second operator; and acquire the attack behavior of the second operator according to the functions that cannot be called by the second operator.

[0115] The apparatus of this embodiment can be used to perform Figure 3 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0116] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the operator attack protection method in the above embodiments.

[0117] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the operator attack protection method in the above embodiments.

[0118] In addition, this application also provides an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the operator attack protection methods in the above method embodiments.

[0119] The electronic devices, computer storage media, computer program products or chips provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0120] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or implemented by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0121] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0122] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0123] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0127] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for protecting against operator attacks, characterized in that, include: Obtain the operator list of the first model, the operator list including information on at least one operator used by the first model; During the operation of the first model, if the behavior of the first operator is determined to be a malicious attack, the first operator is identified as an attack operator, and the first operator is one of the operators in the operator list. The attack operator is blocked.

2. The method according to claim 1, characterized in that, The process of obtaining the list of operators for the first model includes: Data collection statements are inserted at the beginning and end of the startup script of the first model. The startup script of the first model is obtained by training based on multiple running environments of the first model. The operator data of the first model is obtained based on the acquisition statement, and the operator data includes multiple operator names; Duplicate operator names are removed from the plurality of operator names to obtain the operator list of the first model.

3. The method according to claim 1 or 2, characterized in that, The situation where the behavior of the first operator is determined to be a malicious attack includes: If the initial address of the first operator in memory is different from the running address of the first operator in memory, the behavior of the first operator is determined to be a malicious attack.

4. The method according to any one of claims 1-3, characterized in that, The situation where the behavior of the first operator is determined to be a malicious attack includes: Obtain an operator attack behavior knowledge base, which includes descriptive information of dangerous functions; If the first operator calls any of the dangerous functions during the operation of the first model, the behavior of the first operator is determined to be a malicious attack.

5. The method according to claim 4, characterized in that, The first operator calls any of the aforementioned dangerous functions, including: Dynamic replacement is used to instrument the dangerous functions, where the dangerous functions are any one or more from the operator attack behavior knowledge base; The instrumentation is used to detect the invocation of the dangerous function; If the dangerous function is called, obtain the call stack of the dangerous function; If the first operator appears in the call stack of the dangerous function, then the first operator calls the dangerous function.

6. The method according to claim 4, characterized in that, The knowledge base for acquiring operator attack behaviors includes: Obtain a list of common operators, which includes information about multiple operators corresponding to multiple models; Construct the function call chain of the second operator, which is any one of the plurality of operators; Identify the attack behavior of the second operator based on the function call chain of the second operator and the functional information of the second operator; The attack behavior of the second operator is included in the operator attack behavior knowledge base.

7. The method according to claim 6, characterized in that, The process of obtaining the list of shared operators includes: Data collection statements are inserted at the beginning and end of the startup scripts of the multiple models, and the startup scripts of the multiple models are trained based on multiple running environments of the multiple models; Based on the acquisition statement, the operator data of the multiple models is obtained, and the operator data includes multiple operator names; Duplicate operator names are removed from the plurality of operator names to obtain a list of operators for the plurality of models; The common operator list is obtained by intersecting the operator lists of the multiple models.

8. The method according to claim 6 or 7, characterized in that, The step of identifying the attack behavior of the second operator based on the function call chain of the second operator and the functional information of the second operator includes: The function called by the second operator is obtained according to the function call chain of the second operator; Based on the functional information of the second operator, the functions called by the second operator are identified, and functions that cannot be called by the second operator are determined. The attack behavior of the second operator is obtained based on the function that cannot be called by the second operator.

9. A protective device against operator attacks, characterized in that, include: The acquisition module is used to acquire a list of operators for the first model, the list of operators including information on at least one operator used by the first model; The analysis module is used to determine that the first operator is an attack operator when the behavior of the first operator is determined to be a malicious attack during the operation of the first model. The first operator is one of the operators in the operator list. The blocking module is used to block the attack operator.

10. The apparatus according to claim 9, characterized in that, The acquisition module is specifically used to insert acquisition statements at the beginning and end of the startup script of the first model, and the startup script of the first model is obtained by training based on multiple running environments of the first model. The operator data of the first model is obtained based on the acquisition statement, and the operator data includes multiple operator names; duplicate operator names are removed from the multiple operator names to obtain the operator list of the first model.

11. The apparatus according to claim 9 or 10, characterized in that, The analysis module is specifically used to determine that the behavior of the first operator is a malicious attack when the initial address of the first operator in memory is different from the running address of the first operator in memory.

12. The apparatus according to any one of claims 9-11, characterized in that, The analysis module is specifically used to acquire an operator attack behavior knowledge base, which includes description information of dangerous functions; during the operation of the first model, if the first operator calls any of the dangerous functions, the behavior of the first operator is determined to be a malicious attack.

13. The apparatus according to claim 12, characterized in that, The analysis module is specifically used to instrument the dangerous function using dynamic replacement, wherein the dangerous function is any one or more in the operator attack behavior knowledge base; detect the calling status of the dangerous function based on the instrumentation; when the dangerous function is called, obtain the call stack of the dangerous function; if the first operator appears in the call stack of the dangerous function, then the first operator calls the dangerous function.

14. The apparatus according to claim 12, characterized in that, The acquisition module is further configured to acquire a common operator list, which includes information on multiple operators corresponding to multiple models; construct a function call chain for a second operator, wherein the second operator is any one of the multiple operators; and identify the attack behavior of the second operator based on the function call chain of the second operator and the functional information of the second operator. The attack behavior of the second operator is included in the operator attack behavior knowledge base.

15. The apparatus according to claim 14, characterized in that, The acquisition module is specifically used to insert acquisition statements at the beginning and end of the startup scripts of the multiple models, wherein the startup scripts of the multiple models are trained based on multiple running environments of the multiple models; Based on the acquisition statement, operator data of the multiple models is obtained, and the operator data includes multiple operator names; duplicate operator names are removed from the multiple operator names to obtain a list of operators for the multiple models; The common operator list is obtained by intersecting the operator lists of the multiple models.

16. The apparatus according to claim 14 or 15, characterized in that, The acquisition module is specifically used to acquire the functions called by the second operator according to the function call chain of the second operator; to identify the functions called by the second operator according to the functional information of the second operator, and to determine the functions that cannot be called by the second operator; and to acquire the attack behavior of the second operator according to the functions that cannot be called by the second operator.

17. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-8.

19. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method of any one of claims 1-8.

Images