Projection vector overflow penalty as a suppression of string padding in machine learning models

Abstract

This application relates to projection vector overflow penalty as a suppression of string padding in machine learning models. An artifact is received, features are extracted from the artifact, and a vector is padded using these features. The features in the vector are then simplified using a feature simplification operation to obtain a modified vector with multiple buckets. Features within the modified vector that are above a predetermined projection bucket clipping threshold are then identified. The identified features are used to generate an overflow vector. The modified vector is then input into a classification model to generate a score. This score is adjusted based on the overflow vector and can then be provided to a consumer application or process. Related apparatus, systems, technologies, and articles of manufacture are also described.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application Serial No. 16 / 798,120, filed on February 21, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The technical solutions described in this paper involve mitigating the impact of adversarial techniques such as score fuzzing and string stuffing, which can cause machine learning models to misclassify specific artifacts. Background Technology

[0004] Machine learning and other types of artificial intelligence models are increasingly being deployed across various applications and industries. Such models provide classifications, which can be based, for example, on historical data with known outcomes or characteristics. The classifications provided by such models (i.e., model outputs, etc.) can take various forms, including Boolean outputs (e.g., good / bad, etc.), numerical scores (e.g., 0.00 to 1, 1 to 100, etc.), or groupings (e.g., cars, pedestrians, crosswalks, etc.). With the help of certain software implementations, the outputs of such models can be intercepted, even as part of a larger workflow. This interception can allow malicious actors to manipulate classifications made by such models by repeatedly modifying sample input data until the desired classification (even if this classification is ultimately inaccurate) is received.

[0005] One type of manipulation is called score fuzzing. Score fuzzing can be done by adding one string at a time to a malicious file (i.e., a malware sample) from a large list of candidate strings. This list of candidate strings can be collected by running a UNIX string utility on a large number of general-purpose application and library portable executables (PE files). Score fuzzing then uses each of those strings one at a time to modify a benchmark malware sample and is used to determine which of those strings positively impact the score. Multiple strings with positive impact are then added to an existing portion of the existing malware sample, or in the simplest case, appended to the end of the file, to cause the model to misclassify the malware sample as benign or otherwise safe for execution or access. Summary of the Invention

[0006] In the first aspect, a workpiece is received, features are extracted from the workpiece, and these features are used to populate a vector. The features in the vector are then simplified using a feature simplification operation, resulting in a modified vector with multiple buckets. Features within the modified buckets that are above a predetermined projection bucket clipping threshold are then identified. Using the identified features, an overflow vector is generated. The modified vector is then fed into a classification model to generate a score. This score is adjusted based on the overflow vector and can then be provided to a consuming application or process.

[0007] Classification models can characterize artifacts as malicious or benign in terms of accessing, executing, or continuing execution. When a classification model characterizes an artifact as malicious, it can prevent access to or execution of the artifact.

[0008] A classification model can be a machine learning model trained using a training dataset, and it can provide a continuous scale output. Machine learning models can take many forms and include one or more of the following: logistic regression models, neural networks, convolutional neural networks, recurrent neural networks, generative adversarial networks, support vector machines, random forests, or Bayesian models, etc.

[0009] Features can include alphanumeric strings.

[0010] Multiple vectorized malware samples can be input into a classification model to obtain multiple scores for the input vectorized malware samples. Based on the classification, buckets of vectorized malware samples that affect the scores and are above a predetermined threshold can be identified.

[0011] The classification model can be a machine learning-based penalty model, which is trained using training data and provides a continuous scale output, which is synthesized by padding strings into benign and malware samples.

[0012] Overflow vectors can be used to compute a broad overflow sum that sums features across a certain number of buckets with characteristics exceeding a predetermined projection bucket pruning threshold. Overflow vectors can also be used to compute a weighted overflow sum, where such buckets are weighted based on empirically determined how they affect the score. The score can be adjusted using a heuristic approach applying both the broad and weighted overflow sums. The score can be further adjusted by inputting both the broad and weighted overflow sums into a machine learning model.

[0013] Features within the modified vector can be trimmed before the overflow vector is generated.

[0014] Feature simplification operations can take various forms, including random projection matrices and / or principal component analysis.

[0015] The document also describes a non-transitory computer program product (i.e., a physically embodied computer program product) containing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein. Similarly, a computer system is described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. Furthermore, the methods may be implemented by one or more data processors within a single computing system or by one or more data processors distributed among two or more computing systems. Such computing systems may be interconnected and may exchange data and / or commands or other instructions via one or more connections, including but not limited to connections via networks (e.g., the Internet, wireless wide area networks, local area networks, wide area networks, wired networks, etc.), direct connections between one or more of the multiple computing systems, etc.

[0016] The technical solution described in this paper offers numerous technical advantages. For example, the current technical solution provides enhanced techniques to prevent the use of techniques such as string padding to bypass machine learning models, which could otherwise cause such models to misclassify analyzed artifacts. Furthermore, the current technical solution is advantageous in that it provides suppression techniques that can be used in conjunction with existing machine learning models (e.g., neural networks) without requiring the training and deployment of new machine learning models that address model bypass techniques such as string padding.

[0017] Details of one or more variations of the technical solutions described herein are illustrated in the accompanying drawings and the following description. Further features and advantages of the technical solutions described herein will be apparent from the specification, drawings, and claims. Attached Figure Description

[0018] Figure 1 This is a first process flowchart, which illustrates the workflow of a first computer-implemented process for determining the overflow penalty of the projection vector that can be used to adjust the classification score of the workpiece;

[0019] Figure 2 This is a diagram illustrating the clipping of the projection vector derived from the workpiece before classification.

[0020] Figure 3 This is a second process flowchart, illustrating a first computer-implemented workflow for determining the overflow penalty of the projection vector that can be used to adjust the classification score of the workpiece; and

[0021] Figure 4 This diagram illustrates various aspects of computing devices that can be used to implement current technical solutions. Detailed Implementation

[0022] The current technical solutions relate to techniques for preventing attempts to misclassify artifacts by a classifier. These techniques include, for example, string padding, where multiple strings (e.g., dozens, hundreds, thousands, etc.) are added to the artifact. In this regard, the classifier can be an AI / machine learning model whose output is at least one value representing the input to the model. While the current technical solutions provide examples related to models used for detecting malicious software (“malicious software”), it should be appreciated that, unless otherwise stated, the current technical solutions can be applied to other applications / workflows utilizing models including, for example, autonomous driving navigation systems, image analysis systems, biometric security systems, video game cheat avoidance systems, etc.

[0023] In some cases, as part of adversarial attacks, the output of a classification model can be intercepted and exploited by malicious actors. For example, data exchanged between a client and a remote server executing the classification model can be accessed to make minor modifications to the data input to the classification model (e.g., files, code, artifacts, etc.) until the desired result (from the malicious actor's perspective) is obtained. For instance, a malicious actor could automatically or manually modify a file containing malicious code until the classification model determines that the file can be safely executed or otherwise accessed.

[0024] Figure 1 This is a process flowchart 100, illustrating a workflow for an example computer implementation used with current techniques for suppressing string padding. Initially, artifact 110 can be received (e.g., accessed from a remote computing system, loaded, received, etc.). Artifact 110 can be a file, a portion of a file, metadata representing a file, and / or source code. This artifact 110 can be parsed by an observer or otherwise processed. In particular, the observer can extract 120 features (sometimes referred to as attributes or observations) from the artifact and vectorize these features 130. Furthermore, depending on the complexity and / or number of features within the vector, a feature simplification operation 140 can be performed on the vector, which reduces the number of dimensions of the vector. Feature simplification operation 140 can utilize various techniques, including but not limited to principal component analysis and random projection matrices, to reduce the number of extracted features within the vector while still retaining usefulness (i.e., for classification purposes, etc.). As will be described in further detail below, the resulting vector as part of feature simplification operation 140 can be used to generate an overflow vector 150. The overflow vector 150 and the simplified feature vector (as part of operation 140) can then be fed into one or more classification models 160 (multiple model variants may sometimes be referred to as a set of classification models 160).

[0025] The classification model 160 can take various forms, including but not limited to logistic regression models, neural networks (including convolutional neural networks, recurrent neural networks, generative adversarial networks, etc.), support vector machines, random forests, Bayesian models, etc. The output of the classification model 160 can be a score 170. As used herein, unless otherwise stated, the score can be a numerical value, classification type or cluster, or other alphanumeric output, which can then be used by the consumption process 180 or the application to take some follow-up action. For malware applications, the score can be used to determine whether to access, execute, continue execution, isolate, or take other remedial actions to prevent the software and / or computing system from being infected or otherwise penetrated by malicious code or other information encapsulated within artifact 110.

[0026] Figure 1 An example of an interception with a score of 190 is also shown. This interception can occur, for example, when the API of the consumer application is known; by dumping a DLL / SO file with link, nm, and objdump; by using various reverse compilers; by observing the stack / heap / registers during execution targeting function call behavior, etc. Other API (i.e., function) discovery techniques can also be used.

[0027] In an arrangement where the output of model 160 can be easily determined, score 160 can be used by a malicious actor to modify artifact 195 and repeat the process until the corresponding model 160 outputs the desired score 170. For example, the modified artifact 195 can encapsulate a malicious script, and small changes to artifact 110 (i.e., the addition of multiple strings) can cause the corresponding classification model 160 to classify such modified artifact 195 as benign.

[0028] Modifications to artifact 110 can be made 195 in a manner that maintains the original characteristics or properties of artifact 110. In the example where an actor attempts to modify a malicious file (malicious software), any modification must ensure that the malware still functions as intended. Such modifications can be made, for example, by adding, removing, or changing unused portions of the malicious file. Since these portions of the file are unused, they have no effect on the file's implemented behavior, but can result in a different score 170 than model 160. Alternatively or additionally, used portions of artifact 110 can also be modified, as long as the final functionality of the malware remains intact.

[0029] Whether manual or automated, the actor or system typically makes numerous small changes, resulting in a new score 170 from model 160. Any changes that move the score 170 in the desired direction (i.e., in the malware example, moving the score closer to a value interpreted as benign) are maintained, while other changes are discarded. This iterative process can be repeated until cumulative changes to artifact 110 lead to a cumulative change in score 170, achieving the desired effect. The techniques presented in this paper can disrupt this iterative improvement cycle by preventing such model manipulation from causing erroneous or misleading changes to score 170.

[0030] The overflow vector 150 used in this paper enables better detection and greater confidence of malware samples (already padded with strings or other manipulated features). To achieve the next level of detection, using current techniques, the score 170 for samples already padded with strings or other features can be penalized. By capturing the overflow vector (as part of operation 150), a signal strongly indicating feature padding (i.e., overflow vector 150) can be extracted, consisting of normalized bucket accumulations exceeding a predefined bucket pruning threshold (as part of feature simplification operation 140). In this regard, as part of the feature simplification operation, a bucket refers to a single feature derived from a large number of other features (e.g., from 20 to 1, etc.) (or, if otherwise specified, a set of features).

[0031] In some variants, the overflow vector 150 can provide a broad measure for calculating the broad overflow sum (e.g., summing a certain number of buckets that exceed a predefined projection bucket pruning threshold). In other variants, the overflow vector 150 can additionally or alternatively provide a weighted overflow sum, calculated by emphasizing those projection buckets that have a higher impact on the output score. At design time, these weights can be calculated by blurring (multiple) models 160 against a broad set of malware baselines using features (e.g., string features) and generating a vector describing the score impact of each projection bucket. For example, a large portable executable (PE) sample with many strings will contain more overflow buckets than a smaller sample, simply because it contains more strings. However, if the weighted overflow vector 150 is substantially disproportionate to the unweighted overflow vector 150, this arrangement strongly indicates that a deliberate tampering has occurred.

[0032] The overflow vector 150 can be consumed in several ways. In one variant, a heuristic approach can be applied to both the weighted overflow sum and the broad overflow sum, and the results are then used to reduce the score generated by model 160. In a more general sense, this can be characterized as a "side-car" model that acts on the projected vector (generated by feature reduction operation 140) and produces an output consistent with the model.

[0033] In other variations, spillover vector 150 can be consumed by a spillover vector machine learning model (not shown) that executes in parallel with model 160. Such a spillover vector machine learning model can take various forms, such as logistic regression and / or neural networks, and be trained using various types of relevant training data. The output of the spillover vector machine learning model can be used to apply corrections or penalties to the output of the main model 160. With this variation, the output of the spillover vector machine learning model is not added to a set of inputs to the main model 160 because what the sidecar model is measuring is orthogonal to what the main model is being trained on—whether the original (untampered) sample is malicious or benign.

[0034] One technique used to modify artifacts (at position 195) to cause model 160 to misclassify is to add (i.e., pad) artifacts with a large number of data strings (e.g., meaningless alphanumeric text strings, paragraphs from websites, etc.). The fact that strings may fall into the same buckets as other non-string features makes some models susceptible to manipulation. Buckets containing features from upstream models, as well as other strong features (such as checksums and validation features), contribute more to the overall score than is typical.

[0035] Figure 2 This is an example. Figure 1 Figure 200 shows a variant of the workflow where a vector pruning operation 210 is performed instead of the overflow vector operation 150. It can be assumed that the final confidence score of model 160 has a strong positive correlation with a small subset of the model inputs (i.e., the values ​​in the projected vectors). These inputs can be referred to as "hot". An attacker exploits model 160 by including a set of additional strings in the samples that are largely projected onto the hot inputs. When there is no pruning after projection, the attacker can increase the values ​​of these hot inputs beyond the level at which the model was trained to arbitrarily large inputs, limited only by the number of strings projected into a given hot bucket. Ultimately, these hot input levels will cause the model output to be completely dominated by the presence of these strings.

[0036] In some variations, the projected feature vector (140) can be clipped (210) before being input into model 160. If model 160 is trained without normalizing the projected vector 140, the 210 projected vector cannot be clipped with a constant value. Instead, a clipped vector can be used. The clipped vector can comprise a sequence of 2-tuples (min_clip_threshold, max_clip_threshold), representing the minimum and maximum allowed values ​​for each index in the projected vector. We propose defining these values ​​as:

[0037] min_clip_thresholdi = mean i -N*std i ,and

[0038] ·max_clip_thresholdi=mean i +N*std i ;

[0039] Where mean i It is the average of a large set of training samples with respect to projection index i, while std is the average of the average of the training samples with respect to projection index i. i It represents the standard deviation of samples with the same value, and N represents the number of standard deviations allowed before clipping.

[0040] Pruning the input without first normalizing it does not guarantee that all inputs will have roughly the same amplitude; this problem cannot be solved without retraining the model 160 (which can be very expensive in terms of time and computational resources). Nevertheless, the pruning operation 210 can prevent attackers from exploiting projection mechanisms via strings by ensuring that the added string does not increase the amplitude of the given model input above a statistically determined threshold.

[0041] Figure 3 This is a process flowchart, where at 310, a workpiece is received. Then, at 320, features are extracted from the workpiece to fill the vector. Next, at 330, a feature simplification operation is used to simplify the features in the vector, resulting in a modified vector with multiple buckets. Then, at 340, features in the modified vector that are within buckets above a predetermined projection bucket clipping threshold are identified. At 350, these identified features are used to generate an overflow vector. This overflow vector is then fed into a classification model at 360 to generate a score. This score can then be adjusted based on the overflow vector at 370. Finally, the adjusted score can be provided to a consumer application or process at 380.

[0042] Figure 4Figure 400 illustrates an example computing device architecture for implementing the various aspects described herein. Bus 404 can serve as an information highway interconnecting other exemplary components of the hardware. A processing system 408, labeled CPU (Central Processing Unit) (e.g., one or more computer processors / data processors, etc., at a given computer or at multiple computer / processor cores, etc.), can perform computational and logical operations required to execute a program. Non-transitory processor-readable storage media, such as read-only memory (ROM) 412 and random access memory (RAM) 416, can communicate with processing system 408 and can include one or more programming instructions for the operations specified herein. Optionally, program instructions can be stored on a non-transitory computer-readable storage medium, such as a disk, optical disk, recordable memory device, flash memory, solid-state drive, or other physical storage medium.

[0043] In one example, disk controller 448 may interface to system bus 404 with one or more optional disk drives. These disk drives may be external or internal floppy disk drives, such as 460, external or internal CD-ROM, CD-R, CD-RW, or DVD, or solid-state drives, such as 452, or external or internal hard disk drives 456. As previously indicated, these various disk drives 452, 456, 460 and disk controller are optional devices. System bus 404 may also include at least one communication port 420 to allow communication with external devices physically connected to the computing system or available externally via wired or wireless networks. In some cases, at least one communication port 430 includes or otherwise includes a network interface.

[0044] To provide user interaction, the technical solutions described herein can be implemented on a computing device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and input devices 432 (such as a keyboard and / or pointing devices (e.g., a mouse or trackball) and / or a touchscreen), wherein the display device 430 is used to display information obtained from the bus 404 to the user via a display interface 414, and the user can provide input to the computer via the input device 432. Other types of input devices 432 can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback via a microphone 436, or tactile feedback); and input from the user can be received in any form, including sound, speech, or tactile input. The input device 432 and the microphone 436 can be coupled to the bus 404 via an input device interface 428 and transmit information via the bus 404. Other computing devices, such as dedicated servers, may omit one or more of the display 440 and display interface 414, input device 432, microphone 436, and input device interface 428.

[0045] One or more aspects or features of the technical solutions described herein can be implemented in digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can be implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device. The programmable system or computing system can include clients and servers. Clients and servers are typically geographically separated and typically interact via a communication network. The client-server relationship is generated by means of computer programs running on respective computers and having a client-server relationship with each other.

[0046] These computer programs may also be referred to as programs, software, software applications, applications, components, or code, and include machine instructions for a programmable processor, and can be implemented in high-level procedural languages, object-oriented programming languages, functional programming languages, logic programming languages, and / or assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device for providing machine instructions and / or data to a programmable processor, such as, for example, a disk, optical disk, memory, and programmable logic device (PLD) that includes a machine-readable medium for receiving machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor. Machine-readable media may store such machine instructions non-transitory, such as, for example, non-transitory solid-state memory or magnetic hard disk drives or any equivalent storage medium. Machine-readable media may alternatively or additionally store such machine instructions temporarily, such as, for example, a processor cache or other random access memory associated with one or more physical processor cores.

[0047] To provide user interaction, the technical solutions described herein can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) and / or touchscreen through which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, speech, or tactile input.

[0048] In the foregoing description and claims, phrases such as “at least one” or “one or more” may appear, followed by a list of combinations of elements or features. The term “and / or” may also appear in a list of two or more elements or features. Unless implied or explicitly contradicted by the context in which the phrase is used, such a phrase is intended to mean any element or feature listed alone, or any referenced element or feature combined with any other referenced element or feature. For example, each of the phrases “at least one of A and B;”, “one or more of A and B;”, and “A and / or B” is intended to mean “A alone, B alone, or A and B together.” A similar interpretation applies to lists containing three or more items. For example, each of the phrases “at least one of A, B, and C;”, “one or more of A, B, and C;”, and “A, B, and / or C” is intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.” Additionally, the use of the term “based on” in and within the claims is intended to mean “at least partially based on,” thus also allowing for unreferenced features or elements.

[0049] Depending on the desired configuration, the technical solutions described herein can be embodied in systems, apparatuses, methods, and / or articles of manufacture. The implementations set forth in the foregoing description do not represent all implementations consistent with the technical solutions described herein. Rather, they are merely examples of aspects consistent with and related to the described technical solutions. Although some variations have been described in detail above, other modifications or additions are possible. In particular, other features and / or variations may be provided in addition to those features and / or variations set forth herein. For example, the above implementations may involve various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Furthermore, the logical flows depicted in the figures and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method, comprising: Receive workpiece; Extract features from the workpiece and fill in the vector; The features in the vector are simplified using a feature reduction operation to obtain a modified vector with multiple buckets; The feature that identifies the modified vector within a bucket that is above a predetermined projection bucket clipping threshold of the clipping vector, wherein the clipping vector comprises a sequence of tuples, the tuples representing the minimum and maximum allowed values ​​for each index in the modified vector; The identified features within the buckets of the modified vector are cropped; An overflow vector is generated based on the identified features, the overflow vector indicating that the artifact has been filled with strings or other manipulated features indicating malware; The modified vector is then input into the classification model to generate a score; The score is adjusted based on the overflow vector; as well as The adjusted score is provided to a consumer application or process, which (i) prevents access to or execution of the artifact when the adjusted score characterizes the artifact as malicious, and (ii) allows access to or execution of the artifact when the adjusted score characterizes the artifact as non-malicious.

2. The method of claim 1, wherein the score of the classification model characterizes the artifact as malicious or benign for accessing, executing, or continuing execution.

3. The method according to any one of the preceding claims, wherein the classification model is a machine learning model, the machine learning model being trained using a training dataset and providing continuous scale output.

4. The method according to claim 3, wherein the machine learning model includes one or more of the following: logistic regression model, neural network, convolutional neural network, recurrent neural network, generative adversarial network, support vector machine, random forest or Bayesian model.

5. The method according to any one of claims 1 to 2, further comprising: Multiple vectorized malware samples are input into the classification model; Based on the input vectorized malware sample, multiple scores are obtained; as well as Based on classification, buckets of the vectorized malware samples that affect the score and are above the predetermined projection bucket pruning threshold are identified.

6. The method according to any one of claims 1 to 2, wherein the classification model is a machine learning-based penalty model, the machine learning-based penalty model is trained using training data and provides a continuous scale output, the training data being synthesized by padding strings into benign software samples and malware samples.

7. The method according to any one of claims 1 to 2, further comprising: The overflow vector is used to calculate a broad overflow sum, which sums over a certain number of buckets that have features exceeding the predetermined projection bucket clipping threshold. as well as The overflow vector is used to calculate a weighted overflow sum, in which such buckets are weighted based on an empirical determination of how certain buckets affect the score.

8. The method of claim 7, wherein the score is based on a trial-and-error method applied to the broad overflow sum and the weighted overflow sum.

9. The method of claim 7, wherein the score is adjusted by inputting both the broad overflow sum and the weighted overflow sum into a machine learning model.

10. The method according to any one of claims 1 to 2, wherein the feature simplification operation comprises a random projection matrix.

11. The method according to any one of claims 1 to 2, wherein the feature simplification operation includes principal component analysis.

12. A system comprising: At least one data processor; as well as A memory for storing instructions that, when executed by the at least one data processor, cause the method as described in any of the preceding claims.

13. A non-transitory computer program product storing instructions that, when executed by at least one computing device, cause the method as claimed in any one of claims 1 to 11.

Images