Threat scanning and detection in machine learning models

Abstract

The system scans machine learning models to detect actual or potential threats. Threats may be detected before the machine learning model is executed or while it is running in an isolated execution environment. Threat detection may include performing machine learning file format checks, vulnerability checks, tamper checks, and shorthand checks. The machine learning model may also be monitored in an isolated environment during execution or runtime sessions. After the scan is performed, the system can generate signatures based on whether the detected threats are actual, potential, or nonexistent.

Description

Technical Field

[0001] Related Applications This application claims priority to U.S. Patent Application No. 18 / 113,444, filed on February 23, 2023, the entire content of which is incorporated herein by reference.

Background Art

[0002] Machine learning computing resources are becoming more prevalent in products and computing systems. As the presence of machine learning model resources increases, attacks on machine learning-based systems by malicious parties also occur. Typically, threats to a machine learning model are not discovered until the threat materializes and damages the model or the system in which it operates. Therefore, what is needed is an improved method for detecting threats on a machine learning system.

Summary of the Invention

[0003] Broadly speaking, this technology scans a machine learning model for threats. Threats can be detected before the execution of the machine learning model or during its execution in an isolated environment such as a sandbox. Threat detection can include performing machine learning file format checks, vulnerability checks, tampering checks, and / or stenography checks. Also, the machine learning model can be monitored in an isolated environment during an execution session or a runtime session.

[0004] After performing the scan, the scan system generates a result set based on the threat detection operation and identifies any actual or potential threats present in the machine learning model. The system can generate a signature based on whether the detected threat is actually present, potential, or non-existent. For example, if no threat is detected, the system can generate a signature and embed that signature in the model data or file format of the machine learning model.

[0005] In some cases, this technology provides a method for scanning machine learning models for threats. The method begins with a scanning module receiving data from a machine learning model, which is stored on a first server. The data is associated with model parameters and received before the machine learning model is executed. The method then proceeds with the scanning module performing several checks based on the received machine learning model data, while the machine learning model is not running. Next, the scanning module identifies whether the machine learning model contains threats based on one or more of the results of the checks. One or more results are reported by the scanning module to a remote computing device.

[0006] In some cases, a non-temporary computer-readable storage medium contains a program recorded thereon, which is executable by a processor to perform a method of scanning a machine learning model for threats. The method begins with a scan module receiving data for the machine learning model, which is stored on a first server, and the data is associated with model parameters and received before the machine learning model is executed. The method then proceeds with the scan module performing several checks based on the received machine learning model data, the checks being performed while the machine learning model is not running. Next, the scan module identifies whether the machine learning model contains threats based on one or more of the results of the multiple checks. One or more results are reported by the scan module to a remote computing device.

[0007] In some cases, a system for scanning machine learning models for threats includes a server having memory and a processor. One or more modules may be stored in memory and run by a processor, receive machine learning model data via a scanning module, which is stored on a first server, the data is associated with model parameters and received before the machine learning model is executed, and based on the received machine learning model data, the scanning module performs several checks, which are performed while the machine learning model is not running, and based on the results of the multiple checks, the scanning module identifies whether the machine learning model contains threats within the machine learning model, and reports the results to a remote computing device via the scanning module. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of a system for scanning machine learning models.

[0009] [Figure 2] This is the block time for the machine learning model scanning module.

[0010] [Figure 3] This is a flowchart of a method for scanning machine learning models.

[0011] [Figure 4] This is a method for checking for tampering with machine learning model data.

[0012] [Figure 5] This is a method for detecting embedded data within machine learning model data.

[0013] [Figure 6] This is a method for identifying related machine learning models.

[0014] [Figure 7] This is a method for generating signatures based on identified threats to machine learning models.

[0015] [Figure 8] This is the computing environment for implementing this technology. [Modes for carrying out the invention]

[0016] Broadly speaking, this technology scans machine learning models for actual or potential threats. Threats can be detected before the machine learning model is executed or while it is running in an isolated execution environment. Threat detection may include performing machine learning file format checks, vulnerability checks, tamper checks, and / or shorthand checks. Additionally, the machine learning model may be monitored in an isolated environment during execution or runtime sessions.

[0017] After performing a scan, the scanning system generates a result set based on threat detection behavior, identifying any actual or potential threats present in the machine learning model. The system can generate signatures based on whether the detected threats are actual, potential, or nonexistent. For example, if no threats are detected, the system may generate a signature and embed that signature into the machine learning model's model data or file format.

[0018] Figure 1 is a block diagram of a system for performing a model scan. The block diagram in Figure 1 includes a customer environment 110 and a system environment 130. The customer environment 110 includes a conversion module 115, a model scan module 120, and a machine learning model 125. The customer environment 110 can receive data and communicate with one or more users 105, the system environment 130, and the customer 165. The system environment 130 may include customer data 135, system data 140, and a model scan manager 155.

[0019] The model scan module 120 can scan the machine learning model 125 before the module is used by the user 105. When used by the user, the user can provide a data stream such as time series data, generalized input, or some other data type to the conversion module 115. The conversion module can convert the received time series into a series of vectorized data. In some cases, the vectorized data may include an array of floating point numbers. Next, the vectorization of the received data is provided to the machine learning model 125 for processing. After processing the vectorized data, the machine learning model provides an output such as a prediction targeted at the requesting user 105.

[0020] The model scan module 120 can collect the vectorized data provided by the conversion 115 and the output provided by the machine learning model 125. Next, the model scan module 120 can scan the model input, the model output, and aspects of the machine learning model itself. In some cases, the model scan module can perform file format checks, vulnerability checks, stenography, detect tampering, and perform a runtime scan of the machine learning model.

[0021] The model scan module 120 can be implemented in several ways. In some cases, the model scan module can be implemented as an API placed between the requesting user and the machine learning model. The API can intercept requests and then send the requests to the machine learning model.

[0022] In some cases, the model scan module can be implemented by an API gateway and a proxy application. The API gateway can receive requests and provide the requests to the proxy application, and then the proxy application can transfer the requests to the machine learning model 125.

[0023] Returning to FIG. 1, the system environment 130 includes a customer data store 135, a system data store 140, and a model scan manager 155, as well as a customer 165. Each of the customer environment 110 and the system environment 130 may be implemented as one or more servers implementing the physical or logical modules 115-125 and 135-155 shown in FIG. 1. In some cases, each environment is located within one or more cloud computing environments.

[0024] Environments 110 and 130 may communicate via a network. In some cases, one or more modules may be implemented on separate machines in separate environments that may also communicate via the network. The network may be implemented by one or more networks suitable for communication between electronic devices, including but not limited to local area networks, wide area networks, private networks, public networks, wired networks, wireless networks, Wi-Fi networks, intranets, the Internet, cellular networks, conventional telephone services, and any combination of these networks.

[0025] The customer data store 135 in FIG. 1 includes data related to one or more customers. The stored customer data includes data of the customers of the present system and may be accessed by any module within the system environment 130.

[0026] The system data 140 includes data related to the system environment 130. The system data may include event data, traffic data, timestamp data, and other data. The data may be accessed by any of the modules 145-160 and may be used to generate one or more dashboards for use by the customer 165.

[0027] The model scan manager 155 communicates with the model scan module 120 and may provide the customer 165 with output regarding one or more scanned machine learning modules. In some cases, some or all of the functions described with respect to the model scan module 120 may be implemented at least partially by the model scan manager 155. For example, the model scan manager may provide output in the form of a dashboard regarding the status of one or more scans of one or more machine learning models 125.

[0028] Figure 2 is a block diagram of the model scan module. Model scan module 200 provides more details about model scan module 125 in Figure 1. Model scan module 200 includes a file format module 210, a vulnerability check module 220, a temperature module 230, a signature manager module 240, a reporting module 250, a shorthand module 260, and a runtime check module 270.

[0029] The file format module 210 can detect and process the file format of a machine learning model. The file format may include model data such as numerical weights and other model data. In some cases, the file format data is a wrapper for the model data, and transmits the data in formats such as .zip, pickle, protobuf, HDF, or JSON.

[0030] The vulnerability check module 220 can analyze and determine vulnerabilities in machine learning modules. Vulnerability checks may be based on the type of machine learning model, layers, changes related to the base model topology, or the activity level of dormant models.

[0031] The tampering check module 230 can determine whether a machine learning model has been tampered with. In some cases, detecting tampering with a machine learning model may be based on an analysis of model data such as layers or weights, or on identifying the relevant model.

[0032] The reporting module 250 can report the scan status and any actual or potential threats identified regarding the machine learning model. The signature manager module 240 can determine whether the machine learning model contains previously generated signatures. Module 240 may also generate signatures and embed them within the machine learning model.

[0033] Figure 3 is a flowchart of the method for performing a model scan. In step 305, the model scanner is started in the customer environment. The model scanner may be installed in the customer environment to access the machine learning model code, inputs, and outputs in the environment. The machine learning model file format may be received by the model scanner in step 310. The file format may contain model data such as machine learning model layer data, weights, and other data. In step 315, a file format check may be performed on the model.

[0034] Step 320 may involve performing vulnerability checks on the model data. Vulnerability checks may be applied to neural networks, random forests, and other machine learning model types. For example, in the case of a neural network machine learning model, there may be parameters that specify its policy, the number of players, units per layer, the nature of interconnected players, and other parameters. Vulnerability checks may determine how closely the scanned model relates to a typical base model. For example, most machine learning models are fine-tuned versions of a particular base model. The scanner may, in some cases, determine whether the fine-tuned machine learning model has the same topological architecture as the base model. Determining vulnerability checks may include detecting changes to the base topology and parameters, detecting backdoors, and detecting hibernation sections activated by specific inputs. Performing tamper checks may include checking model data layers and specific weights to determine whether they may indicate a threat.

[0035] Tampering checks may also include identifying relevant models, such as determining whether the current machine learning model is similar to a previous model in which a threat was detected. The implementation of tampering checks is explained in more detail below with respect to Figures 4-6.

[0036] In step 330, a shorthand check may be performed on the machine learning model. The shorthand check may analyze the lower bits and other aspects of the machine learning model data. For example, after processing the file format and compression, the model scanner may obtain the actual weights and biases used in the model. The lower bits of each individual number have little effect on the resulting calculation and can be zeroed out by compression. The lower valid bits are usually random. If, after the scanning model extracts the lower bits and examines their distribution, it determines that they are not random, they may be identified as a potential or actual threat based on tampering, depending on the degree of randomness or lack thereof.

[0037] In some cases, performing a shorthand check may involve analyzing the entropy associated with a machine learning model. Criteria may be determined for a specific type of model. Clustering can be used to understand the model type and criteria, identify the predicted entropy, detect the actual entropy system, compare the predicted statistics to the actual entropy statistics, and determine whether malware is present in the model at static runtime.

[0038] The machine learning model to be scanned may be running in step 335. During execution, runtime checks may be performed on the running model in step 340. In some cases, the model may be running in a sandbox or other isolated environment or virtual machine. During execution, the scanning tool may identify running shell scripts and other processes in the isolated environment, where no processes are intended to be started automatically. In some cases, loading the machine learning model into memory may execute malicious code and be identified as a threat.

[0039] After performing various checks and analyses on the machine learning model, step 345 identifies the model and information and threats to it. Information from the scan may include the scan time, scan failures and successes, scanned model files, and detected threats. Threats may be states or other aspects of the machine learning model that allow the model to be manipulated, controlled, or otherwise affected in an unintended and / or undesirable way.

[0040] In step 350, a signature may be generated based on the identified threat (if any). Generally, machine learning models do not include signatures. If no threat is detected, the scanner tool may generate a signature and embed it in the machine learning model data. In some cases, manipulating the machine learning model file may invalidate the signature. In some cases, manipulating any significant part of the machine learning model may invalidate the signature. Therefore, if a signature is generated and the model is subsequently modified, the generated signature can be used to make it detectable.

[0041] In some cases, any of the checks in steps 410-430 and 440 may be performed using a machine learning model. The machine learning model can receive input in the form of file format data, layer data, or other data formats described in steps 410-430 and 440 for the model to be scanned. The machine learning model receiving the data can generate predictions or other indications of whether the scanned model is likely to contain a threat. The output can then be used as part of step 445 to determine whether the scanned model contains a threat. For example, for prediction values ​​in the range of 0 to 1, a value above 0.6 may indicate a threat, and a value between 0.3 and 0.6 may indicate a potential threat.

[0042] Figure 4 shows a method for performing a tamper-evident check on model data. The method in Figure 4 provides more data than step 325 of the method in Figure 3. Embedding data within the model data can be detected in step 410. Detecting embedding data can be done by analyzing the neural network layers, neuron interconnections, and the weights and biases of the model data. Further details for detecting embedding data are described with respect to the method in Figure 5.

[0043] In step 420, relevant machine learning models may be identified. Identifying relevant models can be useful for comparing new models with previously known infection models or models that have been detected to have one or more threats. Further details on how to detect relevant models are described with respect to the method in Figure 6.

[0044] Figure 5 shows a method for detecting embedded data within model data. The method in Figure 5 provides more detail about step 410 of the method in Figure 4. First, the layers of the neural network may be determined in step 510. Neurons that are not interconnected with other neurons are identified in step 520. For example, neurons connected to points completely outside the machine learning model may be identified as potential threats.

[0045] In step 530, the weights and biases of the model data may be checked. In step 540, statistics may be generated using the weights and biases of the model data. The statistics are generated to determine if the shorthand has been tampered with. For example, the entropy of a machine learning model can be used to determine if the model is a threat. A baseline entropy is determined for the machine learning model in order to identify threats based on entropy. The baseline entropy may be determined for a particular type of machine learning model. Clustering can then be performed to understand the type of machine learning model being used, to identify the predicted entropy of a particular model, and to detect the actual entropy of a particular machine learning model being scanned. The scanning module can compare the actual entropy to the predicted entropy to determine if malware is present in the model during static runtime.

[0046] Figure 6 shows a method for identifying related models. The method in Figure 6 provides more detail on step 420 of the method in Figure 4. First, the model layer name may be determined in step 610. In step 620, the model layer name may be regularized. Regularization of the model name may include removing spaces and periods, and converting the name to lowercase. In step 630, the regularized model layer name may be sorted. In some cases, the regularized names are sorted alphabetically. Next, in step 640, a hash may be performed on the sorted model layer name. The hash is generated using any hashing technique suitable for producing comparable values. In step 650, the hash values ​​of different models may be compared to determine the relationship. For example, if the hash value of this model is similar to the hash value of a machine learning model known to pose a threat, this model may be treated as a similar threat.

[0047] Figure 7 shows a method for generating a signature based on identified threats. The method in Figure 7 provides more detail on step 350 of the method in Figure 3. First, the model's threat data is accessed in step 710. Next, it is determined whether the data generated based on the machine learning model scan contains any threats that indicate the model has a viable threat.

[0048] If the threat data indicates that a threat exists to the machine learning model, no signature is generated for the model in step 750. If the threat data indicates that no threat exists to the machine learning model, a signature is generated for the machine learning model in step 730. The machine learning model is modified to include the training generated in step 740.

[0049] Figure 8 is a block diagram of a computing environment for implementing this technology. The system 800 in Figure 8 may be implemented in a machine that implements a model scan module 120, a customer data store 135, a system data store 140, and a model scan manager 155. The computing system 800 in Figure 8 includes one or more processors 810 and memory 820. The main memory 820 partially stores instructions and data for execution by the processor 810. The main memory 820 can store executable code during operation. The system 800 in Figure 8 further includes a mass storage device 830, a portable storage medium drive 840, an output device 850, a user input device 860, a graphics display 870, and peripheral devices 880.

[0050] The components shown in Figure 8 are shown as being connected via a single bus 890. However, the components may be connected via one or more data transmission means. For example, the processor unit 810 and the main memory 820 may be connected via a local microprocessor bus, and the mass storage device 830, peripheral device(s) 880, portable storage device 840, and display system 870 may be connected via one or more input / output (I / O) buses.

[0051] The mass storage device 830, which may be implemented as a magnetic disk drive, optical disk drive, flash drive, or other device, is a non-volatile storage device for storing data and instructions for use by the processor unit 810. The mass storage device 830 can store system software for implementing embodiments of the present invention for the purpose of loading the software into the main memory 820.

[0052] The portable storage device 840 operates in conjunction with a portable non-volatile storage medium such as a floppy disk, compact disk or digital video disk, USB drive, memory card or stick, or other portable memory or removable memory, and inputs data and code to or outputs to the computer system 800 in Figure 8. System software for implementing embodiments of the present invention may be stored on such a portable medium or input to the computer system 800 via the portable storage device 840.

[0053] The input device 860 provides part of the user interface. The input device 860 may include an alphanumeric keypad such as a keyboard for inputting alphanumeric and other information, a pointing device such as a mouse, trackball, stylus, or cursor directional keys, a microphone, a touchscreen, an accelerometer, and other input devices. Furthermore, the system 800 shown in Figure 8 includes an output device 850. Examples of suitable output devices include a speaker, a printer, a network interface, and a monitor.

[0054] The display system 870 may include a liquid crystal display (LCD) or other suitable display device. The display system 870 receives text and graphic information and processes that information for output to a display device. The display system 870 can also receive input as a touchscreen.

[0055] Peripheral devices 880 may include any type of computer support device to add additional functionality to a computer system. For example, peripheral devices 880 may include modems or routers, printers, and other devices.

[0056] The 800 system may also include, in some embodiments, an antenna, a radio transmitter, and a radio receiver 890. The antenna and radio may be implemented within a device such as a smartphone, a tablet, and other devices capable of wireless communication. One or more antennas may operate on one or more radio frequencies suitable for sending and receiving data over cellular networks, Wi-Fi networks, commercial equipment networks such as Bluetooth devices, and other radio frequency networks. The device may include one or more radio transmitters and radio receivers for processing signals sent and received using the antenna.

[0057] The components included in the computer system 800 in Figure 8 are those commonly found in computer systems suitable for use with embodiments of the present invention and are intended to represent a broad category of such computer components well known in the art. Therefore, the computer system 800 in Figure 8 may be a personal computer, a handheld computing device, a smartphone, a mobile computing device, a workstation, a server, a minicomputer, a mainframe computer, or any other computing device. The computer may also include different bus configurations, networked platforms, multiprocessor platforms, etc. Various operating systems, including Unix, Linux, Windows, Macintosh OS, and Android, as well as languages, including Java, .NET, C, C++, Node.js, and other appropriate languages, may be used.

[0058] The aforementioned detailed description of the Art as described herein is presented for illustrative and descriptive purposes only. This disclosure is not intended to be exhaustive or to limit the Art to the exact forms disclosed herein. Many variations and modifications are conceivable in light of the teachings above. The embodiments described herein have been selected to best illustrate the principles of the Art and its practical applications, thereby enabling those skilled in the art to best utilize the Art in various embodiments and variations as suitable for their specific intended use. The scope of the Art is intended to be defined by the claims appended herein.

Claims

1. A method for scanning machine learning models for threats, The machine learning model data, which is received by the scan module in relation to the model parameters and before the execution of the machine learning model, The scan module performs a series of checks based on the received machine learning model data, wherein the checks are performed while the machine learning model is not running. Based on the results of the aforementioned multiple checks, the scanning module identifies whether the machine learning model contains a threat within the machine learning model, The method, comprising reporting the results to a remote computing device by the scan module.

2. The machine learning model determines that the system does not contain threats, and generates a signature based on that determination. The method according to claim 1, further comprising embedding the generated signature into the data of the machine learning model.

3. The method according to claim 1 or 2, wherein the plurality of checks include two or more of the following: file format checks, vulnerability checks, tampering checks, and shorthand checks.

4. Running the aforementioned machine learning model in an isolated environment, The method according to any of the prior claims, further comprising monitoring the execution of the machine learning in the isolated environment in order to detect suspicious activity.

5. Performing the aforementioned multiple checks means The method according to any of the prior claims, comprising determining the expected entropy of the machine learning model.

6. Performing the aforementioned multiple checks means The method according to any of the prior claims, comprising determining the actual entropy of the machine learning model.

7. Performing the aforementioned multiple checks means A method according to any of the prior claims, comprising calculating the difference between the expected entropy and the actual entropy.

8. Performing the aforementioned multiple checks means The method according to any of the prior claims, comprising identifying a threat when the difference between the expected entropy and the actual entropy exceeds a threshold.

9. The method according to any of the prior claims, wherein the identified threat may be a potential threat or an actual threat.

10. The method according to any one of the prior claims, wherein the scan module is stored in a server that forms part of a computing environment including the machine learning model.

11. A non-temporary computer-readable storage medium on which a program is recorded, wherein the program is executable by a processor to carry out the method according to any of the prior claims.

12. A system for scanning machine learning models for threats, At least one data processor, The system comprises a memory for storing instructions, wherein when an instruction is executed by the at least one data processor, it causes the method according to any one of claims 1 to 10 to be carried out.

Images