Method, system, and computer program product for fractional fuzzification using step functions and position-dependent noise

Abstract

An artifact is received. Features are extracted from the artifact, which are used to populate a vector. This vector is then input into a classification model to generate a score. The score is then modified using a step function so that the true score is not obscured. The modified score can then be provided to a consuming application or process. Related devices, systems, techniques, and articles are also described.

Description

[0001] Related applications

[0002] This application claims priority to each of U.S. patent application serial number 16 / 399,677, filed on April 30, 2019, U.S. patent application serial number 16 / 399,718, U.S. patent application serial number 16 / 399,735, U.S. patent application serial number 16 / 399,701, and U.S. patent application serial number 16 / 399,665, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The subject matter described herein relates to techniques for blurring the output of a machine learning model using a step function. Background Art

[0004] Machine learning and other types of artificial intelligence models are being increasingly deployed across diverse applications and industries. Such models provide classifications that can be based on, for example, historical data with known outcomes or features. The classifications (i.e., model outputs, etc.) provided by such models can take various forms, including Boolean outputs (e.g., good / bad, etc.), numerical scores (e.g., 0.00 to 1, 1 to 100, etc.), or groupings (e.g., car, pedestrian, crosswalk, etc.). With some software implementations, the output of such models can be intercepted even as part of a larger workflow. Such interception can allow malicious actors to manipulate the classification of such models by repeatedly providing sample input data until a desired classification is received (even if such classification is ultimately inaccurate). Summary of the Invention

[0005] An artifact is received. Features are extracted from the artifact, which are used to populate a vector. This vector is then fed into a classification model to generate a score. The score is then modified using a step function so that the true score is not obscured. The modified score can then be provided to a consuming application or process.

[0006] In some variations, the features in the vector can be reduced before being input into the classification model. For example, the features can be reduced using a random projection matrix, principal component analysis, or other techniques.

[0007] A classification model can be a machine learning model that is trained using a training dataset and provides a continuous scale output.

[0008] The classification model can characterize an artifact as malicious or benign for access, execution, or continued execution. If the artifact is deemed malicious by the classification model, access or execution of the artifact can be prevented.

[0009] The machine learning model can include one or more of the following: logistic regression model, neural network, concurrent neural network, recurrent neural network, generative adversarial network, support vector machine, random forest, or Bayesian model.

[0010] The step function can apply various types of noise to the score, including, for example, position-dependent noise. Different types of step functions / algorithms combined with the step function can be applied.

[0011] Also described is a non-transient computer program product (i.e., a physically embodied computer program product) storing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein. Similarly, a computer system that may include one or more data processors and a memory coupled to the one or more data processors is also described. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more operations described herein. In addition, the method may be implemented by one or more data processors within a single computing system or distributed between two or more computing systems. Such computing systems may be connected via one or more connections, including but not limited to connections via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), via a direct connection between one or more of the multiple computing systems, etc., and may exchange data and / or commands or other instructions, etc.

[0012] The subject matter described herein provides numerous technical advantages. For example, the present subject matter provides enhanced techniques for obfuscating the output of AI / machine learning models. This obfuscation is particularly important for applications such as malware detection because it prevents malicious actors from iteratively modifying malicious files or code until the model classifies such files or code as safe to execute or otherwise access.

[0013] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a process flow diagram illustrating a first computer-implemented workflow for fuzzifying scores from a classification model;

[0015] Figure 2 is a diagram illustrating a first example of using a step function to blur scores from a classification model;

[0016] Figure 3 is a diagram illustrating a second example of using a step function to blur scores from a classification model;

[0017] Figure 4 is a diagram illustrating real fractions versus fuzzy fractions;

[0018] Figure 5 is a process flow diagram illustrating a second computer-implemented workflow for fuzzifying scores from a classification model; and

[0019] Figure 6 is a diagram illustrating a computing device that can be used to implement aspects of the present subject matter. DETAILED DESCRIPTION

[0020] The present subject matter relates to techniques for obfuscating the output of software-based classifiers. In this regard, a classifier can be an AI / machine learning model whose output is representative of at least one value input to such a model. While the present subject matter provides examples related to models for detecting malicious software ("malware"), it should be understood that, unless otherwise noted, the present subject matter can be applied to other applications / workflows utilizing models, including, for example, autonomous vehicle navigation systems, image analysis systems, biometric security systems, video game cheat avoidance systems, and the like.

[0021] In some cases, the output of a classification model can be intercepted and exploited by a malicious actor as part of an adversarial attack. For example, data exchanged between a client executing a classification model and a remote server can be accessed, allowing small changes to be made to the data (e.g., files, code, artifacts, etc.) input to the classification model until the desired result (from the malicious actor's perspective) is achieved. For example, a malicious actor can make small changes to a file encapsulating malicious code, either automatically or through manual modification, until the classification model determines that such a file is safe to execute or otherwise access.

[0022] Figure 11 is a process flow diagram 100 illustrating an example computer-implemented workflow for use with the present technique for score fuzzification. Initially, an artifact 110 may be received (e.g., accessed, loaded, received, etc. from a remote computing system). Artifact 110 may be a file, a portion of a file, metadata representing a file, and / or source code. Artifact 110 may be parsed or otherwise processed by an observer. In particular, the observer may extract 120 features (sometimes referred to as attributes or observations) from the artifact and vectorize 130 these features. Furthermore, depending on the complexity and / or number of features within the vector, a feature reduction operation 140 may be performed on the vector to reduce the dimensionality of the vector. Feature reduction operation 140 may utilize various techniques, including, but not limited to, principal component analysis and random projection matrices, to reduce the number of extracted features within the vector while maintaining usefulness (i.e., for classification purposes, etc.) when input into a classification model 150. Classification model 150 may take many forms, including, but not limited to, logistic regression models, neural networks (including concurrent neural networks, recurrent neural networks, generative adversarial networks, etc.), support vector machines, random forests, Bayesian models, and the like. The output of the classification model 150 may be a score 160, which may be fuzzified 170, as described in further detail below. As used herein, unless otherwise specified, a score may be a numerical value, a classification type or cluster, or other alphanumeric output, which in turn may be used by a consuming process 180 or application to take some subsequent action. In some variations, at 180, the utilized step function is provided to the entity consuming the score so that the base score can be determined (i.e., reverse engineered, etc.). In some variations, the fuzzified score may be consumed directly at 180 by the entity consuming the score. For malware applications, the score may be used to determine whether to access, execute, continue execution, quarantine, or take some other remedial action that will prevent the software and / or computing system from being infected or otherwise infiltrated by malicious code or other information encapsulated within the article of manufacture 170.

[0023] Figure 1 Further shown is interception of score 190. This interception can occur, for example, when the API of the consuming application is known; by dumping the DLL / SO output using link, nm, objdump; by using various decompilers; by observing the stack / heap / registers during execution for function call behavior, etc. Other API (i.e., function) discovery techniques can also be used.

[0024] In arrangements where the output of a model 150 can be easily determined (e.g., a single model 150 arrangement without score obfuscation), a malicious actor can use the score 160 to modify an artifact 195 and repeat the process until the corresponding model 150 has a desired score 160. For example, a modified artifact 195 may encapsulate a malicious script, and a small change to an artifact 110 may cause the corresponding classification model 150 to classify such a modified artifact 195 as benign.

[0025] Modifications to artifact 150 can be made in a manner that maintains the original characteristics or properties of artifact 150. In the example of an actor attempting to modify a malicious file (malware), any modifications must be such that the malware still operates as intended. Such modifications can be made by, for example, adding, removing, or changing unused portions of the malicious file. Since these portions of the file are unused, they have no effect on the implemented behavior of the file, but may result in a score 160 that differs from model 150. Alternatively or additionally, used portions of artifact 150 may also be modified, as long as the malware's ultimate functionality remains intact.

[0026] Whether manually or in an automated system, an actor or system will typically make many small changes and obtain a new score 160 from the model 150. Any change that moves the score 160 in the desired direction (i.e., in the malware example, moves the score closer to a value interpreted as benign) is retained, while other changes are discarded. This iterative process can be repeated until the cumulative changes to the artifact 110 result in a cumulative change in the score 150 that achieves the desired effect. The obfuscation techniques provided herein can interrupt this cycle of iterative improvement by masking the true effect of each change to the artifact 110 with a false or misleading change in the score 160 determined by the obfuscation techniques herein.

[0027] At 170, the score is fuzzified to change the output score to a new value. Figure 2In the diagram 200, there are three output scores 210, 220, 230. For the first output score 210 (which corresponds to the blurred score), there may be multiple other scores 210', 210", etc., which all produce the same output score 210 when blurred. Similarly, there may be other scores 220'+220", 230'+230", which, when blurred, result in different output scores (220, 230, respectively). As an example, the score fuzzification operation 170 may use a function that relates the various scores 160 output by the model 150 to a particular output score (e.g., scores 210, 220, 230). In some cases, the function may be a rounding function. In other cases, a more Complex functions, including, for example, step functions that utilize position-dependent noise. Using the step function algorithm below, the "position" of the initial score in the total score range determines where the final score will end up. This change from the original score to the final score is noise, and the noise is completely determined by the position of the original score. A noise map can be used so that the input scores are rounded or binned, and then the rounded / binned values ​​are checked against the noise map and the associated value is returned as the fuzzy score. In some variations, features of the input vector can be used as additional inputs to the mapping function so that depending on the original score and the value of one or more features from the vector, a different fuzzy score will be the result.

[0028] The step function utilized targets small changes to the input to the model that will typically produce small score changes by an adversary (ie, malicious actor), resulting in no noticeable score change.

[0029] Another such step function can use some aspect of the original score to determine the magnitude and direction of the noise to be added to the score. For example, a trigonometric function, such as a simple sine or cosine function, can be used. After the original score is calculated, the trigonometric function can be calculated on the score, and the result added to the true score to produce the final blurred score. The absolute size of the noise can be limited by a coefficient to ensure that the overall distribution of the noise falls within some desired range. Additionally, for scores near the boundary between one classification and another, care can be taken so that the noise does not cause the score to flip over the boundary. To avoid these types of flips, a noise decay function can be used so that when the true score approaches such a classification boundary, the scale of the noise added to the true score is reduced so that it is always less than the scale that would cause the score to flip.

[0030] Example trigonometric functions are provided below. It should be understood that other complex trigonometric functions and / or other types of recursive functions can be used with the current subject matter.

[0031] Simple triangular noise:F(score) = score + (A*sin(B*score)); where A and B are parameters chosen by the implementer. A affects the amplitude of the added noise, while B affects how quickly the noise function changes between similar input scores.

[0032] Triangular noise with truncation: F(score, nearest_score_boundary) = score + min(A*sin(B*score), abs(score-nearest_score_boundary-C)); where A and B are as above, and nearest_score_boundary is the score closest to the input score representing the boundary between two classes. In cases where positive scores are interpreted as one class and negative scores as the other, the score boundary will be zero. C is a third parameter chosen by the implementor that can further limit the score fuzzification to produce values ​​very close to the boundary. This function ensures that the fuzzified score never crosses the score boundary.

[0033] Triangular noise with decay: F(score) = score + logistic_fn(abs(score), theta) * A * sin(B * score)); where A and B are as above, logistic_fn is the parameterized logistic function, and θ is the parameter of the logistic function. The logistic function produces values ​​between 0 and 1, such that as the input score approaches zero, the value of the logistic function also approaches zero. The parameter θ can be chosen so that the logistic function attenuates noise only within a specific range of zero, as desired by the implementer.

[0034] Using a complex step function is advantageous because it makes reverse engineering more difficult. Figure 3 is a diagram 300 showing the output of a more complex function, in which the scores 160 (before fuzzification) are altered in an apparently random manner (as shown on the fuzzy score 180 line). As an example of such a function, a score range representing a particular classification (e.g., all positive scores or all negative scores, etc.) can be randomly cut into a large number of very small parts. Each part of the score range can then be randomly mapped to a different output in the same scoring range. When the scores 160 are fuzzified 170, this mapping will be used to convert the true scores to fuzzy scores. This mapping can be retained for a period of time so that repeated queries to the model will produce the same fuzzy scores. Another example of such a function can use information from the simplified feature vector 140 in the mapping from the true scores to the fuzzy scores.

[0035] With the current configuration, the final classification of the fuzzy scores 170 is preserved. In other words, positive scores (indicating good model output) are preserved after fuzzification, and similarly, negative scores are preserved after fuzzification.

[0036] Another sample fuzzification equation is as follows: Where y is the fuzzy score 170 and x is the original score 160. This fuzzified equation can produce Figure 4 4 , where line 410 represents the raw score 160 and other values ​​420 correspond to the fuzzy score 170 .

[0037] Figure 5 is a process flow diagram, wherein at 510 an artifact is received. Thereafter, at 520, features are extracted from the artifact so that a vector can be populated with these features. Next, at 530, the vector is input into a classification model to generate a score. At 540, the score is modified using a step function to obscure its actual value. Thus, at 550, the modified score is provided to a consuming application or process. For example, the consuming application or process can use such a score to determine whether to access, execute, or continue to execute the artifact (i.e., it can be used to prevent malware from infiltrating a computing and / or software system, etc.).

[0038] Figure 6 600 is a diagram illustrating a sample computing device architecture for implementing various aspects of this paper. Bus 604 can be used as an information highway for other components shown in the interconnected hardware. The processing system 608 (e.g., one or more computer processors / data processors at a given computer or at a plurality of computer / processor cores) labeled as a CPU (central processing unit) can perform calculations and logical operations, which are required for executing a program. Non-transient processor-readable storage media such as read-only memory (ROM) 612 and random access memory (RAM) 616 can communicate with the processing system 608 and can include one or more programming instructions for the operation specified here. Alternatively, program instructions can be stored on a non-transient computer-readable storage medium such as a magnetic disk, an optical disk, a recordable memory device, a flash memory or other physical storage medium.

[0039] In one example, disk controller 648 can connect one or more optional disk drives to system bus 604. These disk drives can be external or internal floppy disk drives such as 660, external or internal CD-ROM, CD-R, CD-RW, or DVD, or solid-state drives such as 652, or external or internal hard disk drives 656. As previously described, these various disk drives 652, 656, 660 and the disk controller are optional devices. System bus 604 can also include at least one communication port 620 to allow communication with external devices that are physically connected to the computing system or available externally via a wired or wireless network. In some cases, at least one communication port 620 includes or otherwise comprises a network interface.

[0040] To provide for user interaction, the subject matter described herein can be implemented on a computing device having a display device 640 (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information obtained from bus 604 to the user via display interface 614, and an input device 632, such as a keyboard and / or a pointing device (e.g., a mouse or trackball) and / or a touch screen, through which the user can provide input to the computer. Other types of input devices 632 can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback via microphone 636, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input. Input device 632 and microphone 636 can be coupled to bus 604 via input device interface 628 and communicate information via bus 604. Other computing devices, such as dedicated servers, can omit one or more of display 640 and display interface 614, input device 632, microphone 636, and input device interface 628.

[0041] One or more aspects or features of the subject matter described herein can be implemented in digital electronic circuits, integrated circuits, specially designed application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can include implementation in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which can be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device and transmit data and instructions thereto. The programmable system or computing system can include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship with each other.

[0042] These computer programs, which may also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor and may be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logic programming language, and / or in assembly / machine language. As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device for providing machine instructions and / or data to a programmable processor, such as, for example, a magnetic disk, an optical disk, a memory, and a programmable logic device (PLD), including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions in a non-transient manner, such as, for example, a non-transient solid-state memory or a magnetic hard drive or any equivalent storage medium. A machine-readable medium may alternatively or additionally store such machine instructions in a transient manner, such as, for example, a processor cache or other random access memory associated with one or more physical processor cores.

[0043] To provide for interaction with a user, the subject matter described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) and / or a touch screen through which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0044] In the above description and claims, phrases such as “at least one of…” or “one or more of…” may appear after a joint list of elements or features. The term “and / or” may also appear in a list of two or more elements or features. Unless explicitly or implicitly contradicted by the context in which it is used, such phrases are intended to refer to any one of the listed elements or features individually or in combination with any other of the listed elements or features. For example, the phrases “at least one of A and B”; “one or more of A and B”; and “A and / or B” are respectively intended to mean “A alone, B alone, or A and B together.” A similar interpretation is also intended for a list of three or more items. For example, the phrases “at least one of A, B, and C”; “one or more of A, B, and C”; and “A, B, and / or C” are respectively intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together.” Furthermore, the term "based on" as used above and in the claims is intended to mean "based, at least in part, on" such that non-recited features or elements are also permissible.

[0045] Depending on the desired configuration, the subject matter described herein may be embodied in systems, devices, methods and / or articles. The implementations set forth in the above description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects relating to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to the features and / or variations set forth herein, other features and / or variations may also be provided. For example, the above implementations may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several other features disclosed above. In addition, the logical flows depicted in the accompanying drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other implementations may be within the scope of the following claims.

Claims

1. A computer-implemented method comprising: Receiving finished products; Extracting features from the artifact and filling a vector; inputting the vector into a classification model to generate a score; modifying the score using a step function to add position-dependent noise to the score to obscure its actual value, such that a classification of the modified score is no different than a classification of the score generated by the classification model, wherein the modified score is based on the position of the score in a range of total scores, and the classification is one of: a numerical value, a classification type or cluster, or other alphanumeric output; and The modified score is provided to a consuming application or process.

2. The method of claim 1, further comprising reducing features in the vector prior to the input into the classification model. The method of claim 2 , wherein the features are reduced using a random projection matrix. The method of claim 2 , wherein the features are simplified using principal component analysis.

5. The method according to any one of claims 1-4, wherein the classification model is a machine learning model trained using a training dataset and providing a continuous scale output.

6. The method of any one of claims 1-4, wherein the classification model characterizes the artifact as malicious or benign to access, execute, or continue to execute.

7. The method according to claim 6, further comprising: When the classification model characterizes the artifact as malicious, access to or execution of the artifact is prevented.

8. The method according to any one of claims 1-4, wherein the classification model comprises one or more of the following: a logistic regression model, a neural network, a concurrent neural network, a recurrent neural network, a generative adversarial network, a support vector machine, a random forest, or a Bayesian model.

9. The method according to claim 1, wherein: The classification model is a machine learning model trained to characterize the likelihood of the artifact being malicious.

10. The method according to claim 9, further comprising: When the classification model characterizes the artifact as malicious, access to or execution of the artifact is prevented.

11. A system for fractional fuzzification, comprising: at least one data processor; as well as A memory storing instructions which, when executed by the at least one data processor, cause the at least one data processor to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer program product storing instructions which, when executed by one or more data processors included in at least one computing device, cause the at least one data processor to perform the method according to any one of claims 1 to 10.

Images