Scanning and detecting threats in machine learning models

Abstract

The machine learning model is scanned to detect actual threats or potential threats. Threats may be detected prior to execution of the machine learning model or during execution in an isolated environment. Threat detection may include performing a machine learning file format check, a vulnerability check, a tamper check, and a steganography check. The machine learning model may also be monitored in an isolated environment during execution or runtime sessions. After performing the scan, the system may generate a signature based on actual, potential, or undetected threats.

Description

Related applications

[0001] This application claims U.S. Patent Application Serial No. 18 / 113,444, filed February 23, 2023, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0002] Machine learning computing resources are becoming increasingly popular in products and computing systems. As machine learning model resources increase, so too do malicious attacks on machine learning-based systems. Often, threats to machine learning models are not detected until they materialize and damage the model or the system on which it operates. Therefore, an improved method is needed to detect threats to machine learning systems. Summary of the Invention

[0003] In brief, this technique scans machine learning models to find threats. Threats can be detected before the machine learning model is executed or during execution in an isolated environment (such as a sandbox). Threat detection may include checking the machine learning file format, vulnerabilities, tampering, and / or steganography. The machine learning model can also be monitored in an isolated environment during execution or runtime sessions.

[0004] After performing a scan, the scanning system generates a set of results based on threat detection operations and identifies any actual or potential threats present in the machine learning model. The system can generate a signature based on actual, potential, or undetected threats. For example, if no threat is detected, the system can generate a signature and embed it within the model data or file format of the machine learning model.

[0005] In some cases, this technology provides a method for scanning machine learning models to find threats. The method begins with a scanning module receiving data for a machine learning model, stored on a first server, and this data is associated with model parameters and received before the machine learning model is executed. The method continues with the scanning module performing multiple checks based on the received machine learning model data, where these checks are performed before the machine learning model is executed. Next, the scanning module identifies whether the machine learning model includes a threat based on one or more results of the multiple checks. The scanning module reports these one or more results to a remote computing device.

[0006] In some cases, a non-transitory computer-readable storage medium includes a program embodied thereon, which can be executed by a processor to perform a method for scanning a machine learning model to find threats. The method begins with a scanning module scanning received data for the machine learning model, wherein the scanning module is stored on a first server, and this data is associated with model parameters and received before the machine learning model is executed. The method continues with the scanning module performing multiple checks based on the received machine learning model data, wherein these checks are performed while the machine learning model is not being executed. Next, the scanning module identifies whether the machine learning model includes a threat based on one or more results of the multiple checks. The scanning module reports these one or more results to a remote computing device.

[0007] In some cases, systems used to scan machine learning models for threats include servers with memory and processors. One or more modules may be stored in memory and executed by the processor to receive data for the machine learning model from a scanning module stored on a first server. This data is associated with model parameters and received before the machine learning model is executed. The scanning module performs multiple checks based on the received machine learning model data, which are performed while the machine learning model is not being executed. Based on the results of these multiple checks, the scanning module identifies whether the machine learning model includes threats within it, and reports the results to a remote computing device. Attached Figure Description

[0008] Figure 1 This is a block diagram of a system used to scan machine learning models.

[0009] Figure 2 This is a block diagram of the machine learning model scanning module.

[0010] Figure 3 This is a flowchart of a method used to scan machine learning models.

[0011] Figure 4 It is a method used to perform tamper checks on machine learning model data.

[0012] Figure 5 It is a method for detecting embedded data within machine learning model data.

[0013] Figure 6 It is a method used to identify relevant machine learning models.

[0014] Figure 7 It is a method for generating signatures based on identified threats to machine learning models.

[0015] Figure 8 It is the computing environment used to implement this technology. Detailed Implementation

[0016] In brief, this technique scans machine learning models to find actual or potential threats. Threats can be detected before executing the machine learning model or during execution in an isolated environment. Threat detection may include checking the file format, vulnerabilities, tampering, and / or steganography of the machine learning file. The machine learning model can also be monitored in an isolated environment during execution or runtime sessions.

[0017] After performing a scan, the scanning system generates a set of results based on threat detection operations and identifies any actual or potential threats present in the machine learning model. The system can generate a signature based on actual, potential, or undetected threats. For example, if no threat is detected, the system can generate a signature and embed it within the model data or file format of the machine learning model.

[0018] Figure 1 This is a block diagram of the system used to perform model scanning. Figure 1 The block diagram includes a client environment 110 and a system environment 130. Client environment 110 includes a transformation module 115, a model scanning module 120, and a machine learning model 125. Client environment 110 can receive data and communicate with one or more users 105, system environment 130, and clients 165. System environment 130 may include client data 135, system data 140, and a model scanning manager 155.

[0019] The model scanning module 120 can scan the model before the user 105 uses the machine learning module 125. When the user uses the model, they can provide a data stream, such as time series data, generalized input, or some other data type, to the transformation module 115. The transformation module can convert the received time series into a series of vectorized data. In some cases, the vectorized data may include arrays of floating-point numbers. The vectorized data is then provided to the machine learning model 125 for processing. After processing the vectorized data, the machine learning model will provide the expected output, such as a prediction, for the requesting user 105 to use.

[0020] The model scanning module 120 can collect vectorized data provided by the transform 115 and output provided by the machine learning model 125. Then, the model scanning module 120 can scan various aspects of the model input, model output, and the machine learning model itself. In some cases, the model scanning module can perform file format checks, vulnerability checks, steganography detection, tamper detection, and perform runtime scans of the machine learning model.

[0021] The model scanning module 120 can be implemented in several ways. In some cases, the model scanning module can be implemented as an API positioned between the requesting user and the machine learning model. The API may intercept the request and then send it to the machine learning model.

[0022] In some cases, the model scanning module can be implemented by an API gateway and a proxy application. The API gateway can receive requests, provide them to the proxy application, and then the proxy application can forward the requests to the machine learning model 125.

[0023] return Figure 1 System environment 130 includes customer data repository 135, system data repository 140, and model scan manager 155, as well as customer 165. Each of customer environment 110 and system environment 130 can be implemented as one or more servers, which implement... Figure 1 The physical or logical modules shown are 115 to 125 and 135 to 155. In some cases, each environment resides in one or more cloud computing environments.

[0024] Environments 110 and 130 can communicate via a network. In some cases, one or more modules can be implemented on separate machines in separate environments, which can also communicate via a network. This network can be implemented by one or more networks suitable for communication between electronic devices, including but not limited to local area networks, wide area networks, private networks, public networks, wired networks, wireless networks, Wi-Fi networks, intranets, the Internet, cellular networks, conventional telephone service, and any combination of these networks.

[0025] Figure 1 The customer data repository 135 includes data associated with one or more customers. The stored customer data includes data of customers within this system and can be accessed by any module within the system environment 130.

[0026] System data 140 includes data related to system environment 130. System data may include event data, traffic data, timestamp data, and other data. Any of modules 145 to 160 can access this data and can use it to generate one or more dashboards for use by customer 165.

[0027] The model scan manager 155 can communicate with the model scan module 120 and provide the client 165 with outputs about one or more scans of the machine learning module. In some cases, some or all of the functions described for the model scan module 120 can be implemented at least partially by the model scan manager 155. For example, the model scan manager can provide outputs about the status of one or more scans of one or more machine learning models 125 in the form of a dashboard.

[0028] Figure 2 This is a block diagram of the model scanning module. The model scanning module 200 provides... Figure 1 More details about the model scanning module 125. The model scanning module 200 includes a file format module 210, a vulnerability checking module 220, a temperature module 230, a signature manager module 240, a reporting module 250, a steganography module 260, and a runtime checking module 270.

[0029] File format module 210 can detect and process the file format of machine learning models. The file format can include model data, such as numerical weights and other model data. In some cases, file format data is a wrapper for model data used to convey the data, for example, in .zip, pickle, protobuf, HDF, or JSON formats.

[0030] The vulnerability checking module 220 can analyze and identify vulnerabilities in the machine learning module. Vulnerability checking can be based on the machine learning model type, layer, changes related to the underlying model topology, or dormant activity level.

[0031] The tampering inspection module 230 can determine whether a machine learning model has been tampered with. In some cases, detecting tampering with a machine learning model can be based on the analysis of model data (such as layers or weights) or on identifying the relevant model.

[0032] The reporting module 250 can report the scan status and any actual or potential threats identified by the machine learning model. The signature manager module 240 can determine whether the machine learning model includes a previously generated signature. Module 240 can also generate a signature and embed it into the machine learning model.

[0033] Figure 3 This is a flowchart of a method for performing a model scan. At step 305, the model scanner is started in the client environment. The model scanner can be installed in the client environment, allowing it to access the machine learning model code, inputs, and outputs within the environment. At step 310, the model scanner can receive the machine learning model file format. This file format may include model data, such as machine learning model layer data, weights, and other data. At step 315, a file format check can be performed on the model.

[0034] Vulnerability checks can be performed on the model data at step 320. Vulnerability checks can be applied to neural networks, random forests, and other machine learning model types. For example, for a neural network machine learning model, there may be parameters specifying its policy, number of players, number of units per layer, nature of interconnected players, and other parameters. Vulnerability checks can determine how much the scanned model relates to a typical base model. For example, most machine learning models are fine-tuned versions of a specific base model. In some cases, the scanner can determine whether the fine-tuned machine learning model has the same topology as the base model. Vulnerability checks can include detecting changes in the base model's architecture and parameters, detecting backdoors and dormant parts that activate upon a certain input. Performing tamper checks can include examining model data layers and specific weights to determine if they might pose a threat.

[0035] Tampering checks can also involve identifying relevant models, such as determining whether the current machine learning model is similar to a previous model that detected a threat. See below for reference. Figures 4 to 6 Let's discuss the implementation of tamper checks in more detail.

[0036] Steganography can be performed on the machine learning model at step 330. Steganography can analyze the low-order bits and other aspects of the machine learning model data. For example, after processing file formats and compression, a model scanner can obtain the actual weights and biases used in the model. The low-order bits in each individual digit have little impact on the final calculation and can be zeroed out by compression. The lower significant bits are usually random. If the scanned model extracts the lower bits and determines that they are not random after studying the distribution, they can be identified as a potential or actual threat based on the degree or lack thereof of randomness.

[0037] In some cases, performing steganalysis can include analyzing the entropy associated with a machine learning model. A baseline can be established for certain types of models. Clustering can be used to understand the model type, baseline, identify predicted entropy, and detect actual entropy systems by comparing predicted statistics with actual entropy statistics to determine if malware resides within the model at static runtime.

[0038] The scanned machine learning model can be executed at step 335. During execution, runtime checks can be performed on the executed model at step 340. In some cases, the model may be executed in a sandbox or other isolated environment or a virtual machine. During execution, the scanning tool can identify executing shell scripts and other processes in the isolated environment, none of which should start on their own. In some cases, loading the machine learning model into memory may lead to malicious code execution and be identified as a threat.

[0039] After performing various checks and analyses on the machine learning model, step 345 identifies the model and related information, as well as threats to the model. Information from the scan may include the scan time, scan failures and successes, the model files scanned, and the threats detected. A threat may be a condition or other aspect of the machine learning model that allows the model to be manipulated, controlled, or otherwise influenced in an unexpected and / or undesirable manner.

[0040] At step 350, a signature can be generated based on the identified threats (if any). Generally, machine learning models do not include signatures. If no threat is detected, the scanning tool can generate a signature and embed it into the machine learning model data. In some cases, manipulation of the machine learning model file may invalidate the signature. In some cases, manipulation of any material part of the machine learning model will invalidate the signature. Therefore, if a signature is generated and the model is subsequently modified, the generated signature can be used for detection.

[0041] In some cases, any of the checks in steps 410 through 430 and 440 can be performed using a machine learning model. The machine learning model may receive input in the form of file-formatted data, layered data, or other data discussed in steps 410 through 430 and 440 concerning the model being scanned. The machine learning model receiving the data can generate predictions or other indications about whether the scanned model is likely to contain a threat. The output can then be used as part of step 445 to determine whether the scanned model contains a threat. For example, for predicted values ​​ranging from 0 to 1, values ​​higher than 0.6 may indicate a threat, while values ​​between 0.3 and 0.6 may indicate a potential threat.

[0042] Figure 4 It is a method used to perform tamper checks on model data. Figure 4 The method is Figure 3 Step 325 of the method provides further data. Embedded data within the model data can be detected at step 410. Detection of embedded data can be based on analysis of neural network layers, neuron interconnections, and model data weights and biases. (See reference...) Figure 5 The method discussed here will be used to detect embedded data in more detail.

[0043] Relevant machine learning models can be identified at step 420. Identifying relevant models can facilitate comparison of the new model with previously known affected models or models detected to have one or more threats. Regarding Figure 6 The method discussed in more detail is used to identify relevant models.

[0044] Figure 5 It is a method for detecting embedded data within model data. Figure 5 The method is Figure 4 Step 410 of the method provides further details. First, the layers of the neural network can be determined at step 510. Neurons not interconnected with other neurons are identified at step 520. For example, neurons connected to points completely outside the machine learning model may be identified as potential threats.

[0045] The model data weights and biases can be examined at step 530. These weights and biases can be used to generate statistics at step 540. Statistical data is generated to determine steganography tampering. For example, the entropy of a machine learning model can be used to determine if the model is threatening. To identify threats based on entropy, a baseline entropy is determined for the machine learning model. A baseline entropy can be determined for certain types of machine learning models. Clustering can then be performed to understand the type of machine learning model being used, identify the predicted entropy of a specific model, and detect the actual entropy of a specific machine learning model being scanned. The scanning module can compare the actual entropy with the predicted entropy to identify whether malware was present within the model during static runtime.

[0046] Figure 6 It is a method used to identify relevant models. Figure 6 The method provides Figure 4 Further details of step 420 of the method. First, the model layer name is determined at step 610. At step 620, the model layer name can be regularized. Regularization of the model name may include removing spaces, periods, and converting the name to lowercase. At step 630, the regularized model layer names can be sorted. In some cases, the regularized names are sorted alphabetically. Then, at step 640, the sorted model layer names can be hashed. Hash values ​​are generated using any hashing technique suitable for generating comparable values. At step 650, the hash values ​​of different models can be compared to determine relationships. For example, if the current model hash value is similar to the hash value of a machine learning model known to be threatening, the current model may be considered to have a similar threat.

[0047] Figure 7 It is a method for generating signatures based on identified threats. Figure 7 The method provides Figure 3 Further details of step 350 of the method. First, the model threat data is accessed at step 710. Then it is determined whether the data generated by scanning based on the machine learning model includes any threats that indicate the model has a feasible threat.

[0048] If the threat data indicates that a threat does exist in the machine learning model, no signature is generated for the model at step 750. If the threat data indicates that no threat is associated with the machine learning model, a signature is generated for the machine learning model at step 730. The machine learning model is modified at step 740 to include the generated signature.

[0049] Figure 8 This is a block diagram of the computing environment used to implement this technology. Figure 8 System 800 can be implemented in the context of a machine that implements model scanning module 120, customer data repository 135, system data repository 140 and model scanning manager 155. Figure 8 The computing system 800 includes one or more processors 810 and a memory 820. The main memory 820 partially stores instructions and data for execution by the processor 810. The main memory 820 may also store executable code for operation. Figure 8 The system 800 also includes a mass storage device 830, a portable storage medium drive 840, an output device 850, a user input device 860, a graphic display 870, and peripheral devices 880.

[0050] Figure 8 The components shown are depicted as being connected via a single bus 890. However, the components may be connected via one or more data transfer means. For example, the processor unit 810 and main memory 820 may be connected via a local microprocessor bus, and the mass storage device 830, peripheral device 880, portable storage device 840, and display system 870 may be connected via one or more input / output (I / O) buses.

[0051] Mass storage device 830 can be implemented using a disk drive, optical disk drive, flash memory drive, or other means. It is a non-volatile storage device used to store data and instructions for use by processor unit 810. Mass storage device 830 can store system software for implementing embodiments of the present invention, so that the software can be loaded into main memory 820.

[0052] Portable storage device 840 operates in conjunction with portable non-volatile storage media (such as floppy disks, optical discs or digital video discs, USB drives, memory cards or memory sticks or other portable or removable storage devices) to send and receive data. Figure 8 The computer system 800 inputs and outputs data and code. System software used to implement embodiments of the present invention can be stored on such portable media and input to the computer system 800 via portable storage device 840.

[0053] Input device 860 provides part of the user interface. Input device 860 may include an alphanumeric keypad (such as a keyboard) for inputting alphanumeric and other information, pointing devices (such as a mouse, trackball, stylus, cursor arrow keys), a microphone, a touchscreen, an accelerometer, and other input devices. Additionally, as Figure 8The system 800 shown includes an output device 850. Examples of suitable output devices include speakers, printers, network interfaces, and monitors.

[0054] The display system 870 may include a liquid crystal display (LCD) or other suitable display device. The display system 870 receives text and graphic information and processes that information for output to the display device. The display system 870 may also function as a touchscreen to receive input.

[0055] Peripheral device 880 may include any type of computer support device to add additional functionality to the computer system. For example, peripheral device 880 may include a modem or router, printer, and other devices.

[0056] In some implementations, system 800 may also include an antenna, a radio transmitter, and a radio receiver 890. The antenna and radio can be implemented in devices such as smartphones, tablets, and other devices capable of wireless communication. One or more antennas may operate at one or more radio frequencies suitable for transmitting and receiving data via cellular networks, Wi-Fi networks, commercial device networks (such as Bluetooth devices), and other radio frequency networks. These devices may include one or more radio transmitters and receivers for processing signals transmitted and received using the antennas.

[0057] Figure 8 The computer system 800 includes components commonly found in computer systems that may be suitable for use with embodiments of the present invention, and is intended to represent a broad category of such computer components well known in the art. Therefore, Figure 8 The computer system 800 can be a personal computer, handheld computing device, smartphone, mobile computing device, workstation, server, minicomputer, mainframe computer, or any other computing device. The computer can also include different bus configurations, network platforms, multiprocessor platforms, etc. Various operating systems can be used, including Unix, Linux, Windows, Macintosh OS, Android, and languages ​​including Java, .NET, C, C++, Node.JS, and other suitable languages.

[0058] The foregoing detailed description of the technology herein has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise forms disclosed. Many modifications and variations are possible based on the foregoing teachings. The described embodiments have been chosen and described to best explain the principles of the technology and its practical application, thereby enabling others skilled in the art to best utilize the technology in various embodiments and to adapt various modifications to the intended particular use. The scope of the technology is intended to be defined by the appended claims.

Claims

1. A method for scanning machine learning models to find threats, comprising: The scanning module receives data for the machine learning model, which is associated with model parameters and is received before the machine learning model is executed. The scanning module performs multiple checks based on the received machine learning model data, and these checks are performed when the machine learning model is not being executed. The scanning module identifies whether the machine learning model includes a threat based on the results of the multiple checks; and The scanning module reports the results to the remote computing device.

2. The method of claim 1, further comprising: A signature is generated based on the determination that the machine learning model does not contain threats; as well as The generated signature is embedded within the data used in the machine learning model.

3. The method of claim 1 or 2, wherein the plurality of checks includes two or more of file format checks, vulnerability checks, tamper checks, and steganography checks.

4. The method as described in any of the preceding claims, further comprising: Execute the machine learning model in an isolated environment; The execution of the machine learning model in the isolated environment is monitored to detect suspicious activity.

5. The method as described in any of the preceding claims, wherein performing multiple checks comprises: Determine the expected entropy of the machine learning model.

6. The method as described in any of the preceding claims, wherein performing multiple checks comprises: Determine the actual entropy of the machine learning model.

7. The method as described in any of the preceding claims, wherein performing multiple checks comprises: Calculate the difference between the expected entropy and the actual entropy; as well as.

8. The method as described in any of the preceding claims, wherein performing multiple checks comprises: A threat is identified if the difference between the expected entropy and the actual entropy exceeds a threshold.

9. The method as described in any of the preceding claims, wherein the identified threat may be a potential threat or an actual threat.

10. The method as described in any of the preceding claims, wherein the scanning module is stored on a server forming part of a computing environment including the machine learning model.

11. A non-transitory computer-readable storage medium having a program embodied thereon, the program being executable by a processor to perform the method as described in any of the preceding claims.

12. A system for scanning machine learning models to find threats, comprising: At least one data processor; as well as A memory that stores instructions that, when executed by the at least one data processor, produce the method as described in any one of claims 1 to 10.

Images