Projected vector modification as a string padding suppression for machine learning models

Abstract

Embodiments of the present disclosure relate to projected vector modification as a mitigation of string stuffing for machine learning models. A workpiece is received, features are extracted from the workpiece to populate a vector. Using a feature simplification operation, features in the vector can be simplified to result in a modified vector having a plurality of buckets. The presence of a predetermined type of feature can be identified within a bucket of the modified vector that has an impact on a score higher than a predetermined threshold. Contributions of the identified features within the high impact bucket of the modified vector are then attenuated. The modified vector is input into a classification model to generate a score that can be provided to a consuming application or process. Related apparatuses, systems, techniques, and articles of manufacture are also described.

Description

Technical Field

[0001] The topics described in this paper address the effects of mitigating adversarial techniques such as score fuzzing and string padding, which can act to cause machine learning models to misclassify specific artifacts. Background Technology

[0002] Machine learning and other types of artificial intelligence models are increasingly being deployed across diverse applications and industries. Such models provide classifications, for example, based on historical data with known outcomes or characteristics. The classifications provided by such models (i.e., model outputs, etc.) can take various forms, including Boolean outputs (e.g., good / bad), numerical scores (e.g., 0.00 to 1, 1 to 100, etc.), or groupings (e.g., cars, pedestrians, crosswalks, etc.). Using software implementations, the outputs of such models can be intercepted, even when they are part of a larger workflow. Such interception can allow malicious actors to manipulate classifications made by such models by repeatedly modifying sample input data until a desired classification is received (even if this classification is not ultimately accurate). Summary of the Invention

[0003] In the first aspect, a workpiece is received, and features are extracted from it to populate a vector. Using a feature reduction operation, the features in the vector are reduced to obtain a modified vector with multiple buckets. The presence of features of a predetermined type can be identified within buckets in the modified vector whose influence on the score exceeds a predetermined threshold. The contribution of features identified within high-influence buckets in the modified vector is then reduced. The modified vector is then fed into a classification model to generate scores that can be provided to consumer applications or processes.

[0004] Classification models can characterize artifacts as either malicious or benign in terms of accessing, executing, or continuing execution. In this case, when a classification model characterizes an artifact as malicious, access to or execution of that artifact can be prevented or simplified.

[0005] A classification model can be a machine learning model that is trained using a training dataset and provides a continuous scale output. Machine learning models can take various forms, including one or more of the following: logistic regression models, neural networks, convolutional neural networks, recurrent neural networks, generative adversarial networks, support vector machines, random forests, or Bayesian models.

[0006] The characteristics of a predefined type can take different forms, including alphanumeric strings.

[0007] Multiple vectorized malware samples can be input into a classification model, after which multiple scores can be obtained based on the input vectorized malware samples. Based on these classifications, features of a predetermined type can be determined within buckets where the influence of the vectorized malware samples on the score exceeds a predetermined threshold.

[0008] Feature simplification operations can take various forms, including one or more of the following: Principal Component Analysis (PCA), Random Projection Matrix (RPM), Linear Discriminant Analysis, Canonical Correlation Analysis, Singular Value Decomposition (SVD), Latent Semantic Indexing (LSI), Discrete Cosine Transform (DCT), Randomized Principal Component Analysis (RPCA), or Large Sample Randomized Principal Component Analysis (LS-RPCA).

[0009] The document also describes a non-transient computer program product (i.e., a physically embodied computer program product) containing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein. Similarly, it describes a computer system that may include one or more data processors and memory coupled to said one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. Furthermore, the methods may be implemented by one or more data processors, either within a single computing system or distributed among two or more computing systems. Such computing systems may be interconnected and may exchange data and / or commands or other instructions via one or more connections, including but not limited to connections via networks (e.g., the Internet, wireless wide area networks, local area networks, wide area networks, wired networks, etc.), connections via direct connections between one or more computing systems, etc.

[0010] The topic described in this paper offers numerous technical advantages. For example, the current topic provides techniques for preventing the use of techniques such as string padding to bypass enhancements to machine learning models, which could otherwise cause such models to misclassify the analyzed artifacts. Furthermore, the current topic offers the advantage of providing inhibition techniques that can be used in conjunction with existing machine learning models (e.g., neural networks) without requiring the training and deployment of new machine learning models that address model bypass techniques such as string padding.

[0011] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. Further features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims. Attached Figure Description

[0012] Figure 1 This is a first process flowchart illustrating a first computer-implemented workflow for modifying the vector representing a workpiece before it is classified.

[0013] Figure 2 This is a chart illustrating the effect of certain buckets on the characteristics of workpieces when sorting them.

[0014] Figure 3 This is a second process flowchart illustrating a first computer-implemented workflow for modifying the vector representing a workpiece before it is classified; and

[0015] Figure 4 It is a diagram illustrating computing devices that can be used to realize various aspects of the current topic. Detailed Implementation

[0016] This topic addresses techniques used to prevent attempts to misclassify artifacts by a classifier. Examples of such techniques include string stuffing, in which multiple strings (e.g., dozens, hundreds, thousands, etc.) are added to the artifact. The classifier herein can be an AI / machine learning model whose output represents at least one value of the input to such a model. While this topic provides examples related to models used to detect malicious software (“malicious software”), it should be understood that, unless otherwise specified, this topic can be applied to other applications / workflows that utilize models, including, for example, automated vehicle navigation systems, image analysis systems, biometric security systems, video game cheating avoidance systems, etc.

[0017] In some cases, as part of an adversarial attack, the output of a classification model can be intercepted and exploited by a malicious actor. For example, data exchanged between a client and a remote server executing the classification model can be accessed to make minor changes to the data input into the classification model (e.g., files, code, artifacts, etc.) until the desired result (from the malicious actor's perspective) is obtained. For instance, a malicious actor can make minor changes, either automatically or manually, to a file containing malicious code until the classification model determines that the file can be safely executed or otherwise accessed.

[0018] Figure 1This is a process flowchart 100, illustrating the workflow implemented on a sample computer for use with current techniques for suppressing string padding. Initially, artifact 110 can be received (e.g., accessed, loaded, received from a remote computing system, etc.). Artifact 110 can be a file, a portion of a file, metadata representing a file, and / or source code. This artifact 110 can be parsed by an observer or otherwise processed. In particular, the observer can extract 120 features (sometimes referred to as attributes or observations) from the artifact and vectorize such features 130. Furthermore, depending on the complexity and / or number of features within the vector, a feature simplification operation 140 can be performed on the vector, which simplifies the number of dimensions of such a vector. The feature simplification operation 140 can utilize various techniques, including but not limited to principal component analysis and random projection matrices, to reduce the number of features extracted within the vector while maintaining usefulness (i.e., for classification purposes, etc.). Other feature simplification techniques can be used, such as linear discriminant analysis, canonical correlation analysis, singular value decomposition (SVD), latent semantic indexing (LSI), discrete cosine transform (DCT), randomized principal component analysis (RPCA), and large sample randomized principal component analysis (LS-RPCA).

[0019] As will be further described below, the resulting vector, as part of the feature simplification operation 140, can be used to generate an overflow vector 150. The overflow vector 150, along with the simplified feature vector (as part of operation 140), can then be input into one or more classification models 160 (multiple model variants may sometimes be referred to as an ensemble of classification models 150) for simplification.

[0020] The classification model 160 can take various forms, including but not limited to logistic regression models, neural networks (including convolutional neural networks, recurrent neural networks, generative adversarial networks, etc.), support vector machines, random forests, Bayesian models, etc. The output of the classification model 160 can be a score 170. As used herein, unless otherwise specified, the score can be numerical, a classification type or cluster, or other alphanumeric output, which can then be used by the consuming process 180 or the application to take subsequent actions. For malware applications, the score can be used to determine whether to access, execute, continue execution, isolate, or take other remedial actions that prevent the software and / or computing system from being infected or otherwise penetrated by malicious code or other information encapsulated within artifact 110.

[0021] Figure 1The diagram also illustrates the interception score of 190. For example, such interception can occur when the consumer application's API is known; by dumping DLL / SO exports with linker, nm, or objdump; by using various reverse compilers; by observing the stack / heap / registers during execution targeting function call behavior, etc. Other API (i.e., function) discovery techniques can also be used.

[0022] In an arrangement where the output of model 160 can be easily identified, score 160 can be used by a malicious actor to modify artifact 195, and this process is repeated until the desired score 170 is output by the corresponding model 160. For example, the modified artifact 196 can encapsulate a malicious script, and a small change to artifact 110 (i.e., adding multiple strings) can cause the corresponding classification model 160 to classify such a modified artifact 195 as benign.

[0023] Modifications to artifact 195 can be made in a manner that maintains the original characteristics or properties of artifact 110. In the example where an actor attempts to modify a malicious file (malicious software), any modification must ensure that the malicious software still functions as intended. Such modifications can be made, for example, by adding to, removing from, or altering unused portions of the malicious file. Since these portions of the file are unused, they have no effect on the file's implemented behavior, but can result in a different score 170 from model 160. Alternatively or additionally, the used portions of artifact 110 can also be modified, provided that the final functionality of the malicious software remains intact.

[0024] Whether manual or automated, the actor or system typically makes numerous small changes, resulting in a new score 170 from model 160. Any changes that move the score 170 in the desired direction (i.e., in the malware example, moving the score closer to a value interpreted as benign) are maintained, while other changes are discarded. This iterative process can be repeated until the cumulative changes to artifact 110 result in a cumulative change in score 170 that achieves the desired effect. The techniques presented herein can disrupt this iterative improvement cycle by preventing such model manipulation from leading to erroneous or misleading changes in score 170.

[0025] As used in this paper, vector modification operation 150 modifies the contribution of features when features are added to projected buckets to simplify the contribution of certain features that, whether present or absent, contribute more to the score output by the classification model(s)160. In one example, as a means of dimensionality simplification, the projection matrix (at 140) can reduce the features extracted from vector 130 to 10,000 locations (or buckets) before those features are presented to the model(s)160. The current technique addresses the situation where a small number of such buckets may contribute disproportionately to the resulting score, such as... Figure 2 As shown in Figure 200 (which illustrates how the individual buckets extracted from the composite of malware samples affect the score), in this context, a bucket refers to a single feature (or, if otherwise specified, a set of features) derived from a large number of other features as part of a feature simplification operation (e.g., 20 to 1, etc.).

[0026] Figure 2 Figure 200 shows the median impact of injected binary features for each projected bucket on confidence scores across multiple malware baseline samples. Particular attention is paid to buckets with contributions greater than 0.01, where the injected features can significantly influence the scores.

[0027] In one example, the Portable Executable (PE) model used for malware analysis includes a feature set with multiple sub-model outputs for autoit, nsis, entrypointopcode, and virtual sorting, as well as some high-weight features (such as PE checksum and correction features). In the random projection matrix, these sub-model features, like all other features, undergo random projection. However, if we consider... Figure 2 The chart shows that a very small number of projected buckets contribute disproportionately large amounts to the final score.

[0028] One technique used to attempt to modify an artifact (at position 195) to cause a misclassification of model 160 is to add (i.e., pad) large amounts of string data to the artifact (e.g., meaningless alphanumeric text strings, paragraphs from websites, etc.). As will be... Figure 2 As pointed out, the fact that some strings may fall into the same bucket as other non-string features makes some models susceptible to manipulation. This includes features from upstream sub-models as well as other strong features that contribute more to the overall score than typical features (such as features validated by checksums).

[0029] If you will refer to it again Figure 2As pointed out, the actual number of strings falling into these high-influence buckets is actually a very small percentage of the total string features. Sub-model features plus PE checksum and correction features are projected into a small number of buckets. If one were to simply discard all strings falling into the top ten buckets that have a polarity that influences the score positively, this would reduce the set of strings the model is sensitive to by only 0.10% (10 / 10000). While one might argue that the model was trained using those characters and that removing them without retraining might reduce efficiency, another might argue that the strings in that group merely represent noise that does not provide a useful signal during training.

[0030] Using the current topic, instead of retraining model 160 or changing the mapping of the projected matrix (as part of feature simplification operation 140), the projected matrix utilized by feature simplification operation 140 can be modified to: (i) simplify (i.e. reduce) the contribution of specific string features falling into the top 1% buckets that have the most positive impact on the score, as defined by the aggregation of the blurred malware baseline samples; and (ii) ensure that all non-string features falling into those buckets retain their original (unmodified) contribution when they are accumulated into the projected vector.

[0031] Such an arrangement is possible because the vectorization process can provide the ability to set the contribution of features projected into the bucket to any value between +1.0 and -1.0.

[0032] The vector representing the score impact for each bucket (aggregated from hundreds of malware samples) can be used to smooth the contribution of those string features assigned to the top 1% using a factor that flattens the score contribution level of those string features to be the same as other binary features at the 99th percentile. This is specifically designed to flatten. Figure 2 The leftmost tail of the strings that contribute to the high-impact buckets is used. In this example, the suppression simply simplifies the projection matrix contribution for the strings that contribute to the 1% of buckets in the bucket.

[0033] Figure 3This is a process flowchart in which, at 310, the workpiece is received. Then, at 320, features are extracted from the workpiece to populate a vector. Next, at 330, using a vector simplification operation (e.g., principal component analysis, random projection matrix, etc.), the features in the vector are simplified to result in a modified vector with multiple buckets. Next, at 340, the presence of features of a predetermined type within buckets where the modified vector's influence on the score exceeds a predetermined threshold is identified. At 350, the contribution of such identified features within the modified high-influence buckets is reduced. At 360, the modified vector is later fed into a classification model to generate a score. The score is later provided to a consumer application or consumption process at 370.

[0034] Figure 4 Figure 400 illustrates a sample computing device architecture used to implement the various aspects described herein. Bus 404 may act as an information highway interconnecting other illustrated components of the hardware. A processing system 408, labeled CPU (Central Processing Unit) (e.g., one or more computer processors / data processors at a given computer or multiple computer / processor cores), may perform computational and logical operations required to execute a program. Non-transient processor-readable storage media (such as read-only memory (ROM) 412 and random access memory (RAM) 416) may communicate with processing system 408 and may include one or more programming instructions for the operations specified herein. Optionally, program instructions may be stored on a non-transient computer-readable storage medium, such as a disk, solid-state drive, optical disk, recordable memory device, flash memory, or other physical storage medium.

[0035] In one example, disk controller 448 may interface to system bus 404 with one or more optional disk drives. These disk drives may be external or internal floppy disk drives (such as 460), external or internal CD-ROM, CD-R, CD-RW, or DVD drives, or solid-state drives (such as 452), or external or internal hard disk drives 456. As previously indicated, these various disk drives 452, 456, 460, and the disk controller are optional devices. System bus 404 may also include at least one communication port 420 to allow communication with external devices that are either physically connected to the computing system or available externally via a wired or wireless network. In some cases, at least one communication port 430 includes or otherwise incorporates a network interface.

[0036] To provide interaction with the user, the subjects described herein can be implemented on a computing device having a display device 430 (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information acquired from bus 404 to the user via display interface 414, and an input device 432 (such as a keyboard and / or pointing device (e.g., a mouse or trackball) and / or touchscreen) through which the user can provide input to the computer. Other types of input devices 432 can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback via microphone 436, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input. Input device 432 and microphone 436 can be coupled to bus 404 via input device interface 428, and information is transmitted via bus 404 through input device interface 428. Other computing devices (such as dedicated servers) may omit one or more of the following: display 440 and display interface 414, input device 432, microphone 436, and input device interface 428.

[0037] One or more aspects or features of the subject matter described herein can be implemented in digital electronic circuit devices, integrated circuit devices, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These aspects or features may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. Clients and servers are generally remote to each other and typically interact via a communication network. The client-server relationship is established by means of computer programs running on their respective computers and having a client-server relationship with each other.

[0038] These computer programs, also referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor and can be implemented in high-level procedural languages, object-oriented programming languages, functional programming languages, logic programming languages, and / or assembly / machine languages. As used herein, the term "machine-readable medium" means any computer program product, apparatus, and / or device (such as, for example, a disk, optical disk, memory, and programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, which includes a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" means any signal used to provide machine instructions and / or data to a programmable processor. Machine-readable media can store such machine instructions non-transitory, as would be done by, for example, non-transitory solid-state memory or magnetic hard disk drives or any equivalent storage medium. Machine-readable media can optionally or additionally store such machine instructions transiently, as would be done by, for example, a processor cache or other random access memory associated with one or more physical processor cores.

[0039] To provide interaction with the user, the subjects described herein can be implemented on a computing device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, and a keyboard and / or pointing device (e.g., a mouse or trackball) and / or touchscreen through which the user can provide input to the computer. Other types of input devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0040] In the foregoing description and claims, phrases such as “at least one” or “one or more” may appear, followed by a list of connected elements or features. The term “and / or” may also appear in a list of two or more elements or features. Unless implied or explicitly contradicted by the context in which it is used, the phrase is intended to mean individually any of the listed elements or features, or to mean any of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases “at least one of A and B,” “one or more of A and B,” and “A and / or B” are each intended to mean “A alone, B alone, or A and B together.” Similar interpretations are also intended for lists comprising three or more items. For example, the phrases “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, and / or C” are each intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together.” Additionally, the use of the term "based on" in the foregoing and claims is intended to mean "at least partially based on" so that features or elements not described are also permitted.

[0041] Depending on the desired configuration, the subject matter described herein can be embodied in systems, apparatus, methods, and / or articles of art. The implementations set forth in the foregoing description do not represent all implementations consistent with the subject matter described herein. Rather, they are merely examples of aspects consistent with the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, other features and / or variations may be provided in addition to those features and / or variations set forth herein. For example, the above implementations may be for various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Furthermore, the logical flows depicted in the accompanying drawings and / or described herein do not necessarily require a specific order or sequence to achieve the desired result. Other implementations may be within the scope of the appended claims.

Claims

1. A computer-implemented method, comprising: Receive workpiece; Extract features from the workpiece and fill in the vector; The features in the vector are simplified using a feature simplification operation to obtain a modified vector with multiple buckets; The presence of a predetermined type of feature within a bucket in the modified vector whose impact on the score is higher than a predetermined threshold is identified; The contribution of the features identified in the high-influence bucket is reduced by simplifying the contribution of specific string features in the high-influence bucket of the modified vector, while retaining the contribution of non-string features in the high-influence bucket without modification. The modified vector is then input into the classification model to generate a score; as well as The score is provided to consumer applications or processes.

2. The method of claim 1, wherein the classification model characterizes the artifact as either malicious or benign for accessing, executing, or continuing execution.

3. The method according to claim 2, further comprising: When the classification model characterizes the artifact as malicious, access to or execution of the artifact is prevented.

4. The method according to any one of claims 1-3, wherein the classification model is a machine learning model trained using a training dataset and providing continuous scale output.

5. The method according to any one of claims 4, wherein the machine learning model comprises one or more of the following: logistic regression model, neural network, convolutional neural network, recurrent neural network, generative adversarial network, support vector machine, random forest or Bayesian model.

6. The method according to any one of claims 1-3, wherein the feature of the predetermined type includes alphanumeric strings.

7. The method according to any one of claims 1-3, further comprising: Multiple vectorized malware samples are input into the classification model; Multiple scores are obtained based on the input vectorized malware sample; as well as Based on classification, features of the predetermined type within buckets that identify the vectorized malware samples whose impact on the score is higher than the predetermined threshold.

8. The method according to any one of claims 1-3, wherein the feature simplification operation includes one or more of the following: principal component analysis (PCA), random projection matrix (RPM), linear discriminant analysis, canonical correlation analysis, singular value decomposition (SVD), latent semantic indexing (LSI), discrete cosine transform (DCT), randomized principal component analysis (RPCA), or large sample randomized principal component analysis (LS-RPCA).

9. A system comprising: At least one data processor; as well as The memory includes instructions that, when executed by the at least one data processor, cause an operation, the operation including: Receive workpiece; Extract features from the workpiece and fill in the vector; The features in the vector are simplified using a feature simplification operation to obtain a modified vector with multiple buckets; The presence of a feature of a predetermined type within a bucket whose impact on the score is higher than a predetermined threshold is identified; The contribution of the features identified in the high-influence bucket is reduced by simplifying the contribution of specific string features in the high-influence bucket of the modified vector, while retaining the contribution of non-string features in the high-influence bucket without modification. The modified vector is input into the classification model to generate a score; and The score is provided to consumer applications or processes.

10. The system of claim 9, wherein the classification model characterizes the artifact as malicious or benign for accessing, executing, or continuing execution.

11. The system of claim 10, wherein the operation further comprises: When the classification model characterizes the artifact as malicious, access to or execution of the artifact is prevented.

12. The system according to any one of claims 9 to 11, wherein the classification model is a machine learning model trained using a training dataset and providing continuous scale output, and wherein the machine learning model optionally includes one or more of the following: a logistic regression model, a neural network, a convolutional neural network, a recurrent neural network, a generative adversarial network, a support vector machine, a random forest, or a Bayesian model.

13. The system according to any one of claims 9 to 11, wherein the operation further comprises: Multiple vectorized malware samples are input into the classification model; Multiple scores are obtained based on the input vectorized malware sample; as well as Based on classification, features of the predetermined type within buckets that identify the vectorized malware samples whose impact on the score is higher than the predetermined threshold.

14. The system according to any one of claims 9 to 11, wherein the feature simplification operation includes one or more of the following: principal component analysis (PCA), random projection matrix (RPM), linear discriminant analysis, canonical correlation analysis, singular value decomposition (SVD), latent semantic indexing (LSI), discrete cosine transform (DCT), randomized principal component analysis (RPCA), or large sample randomized principal component analysis (LS-RPCA).

15. A non-transient computer program product comprising instructions that, when executed by at least one computing device, cause the method according to any one of claims 1 to 8.

Images