Detection of obfuscated malicious code within document files

Abstract

To provide a system and method for detecting a malicious code that is obfuscated within document files.SOLUTION: A method includes using a trained machine learning model to detect an obfuscated malicious code and predicting which documents include the malicious code, despite such maliciousness not having a known signature. Safe documents are allowed into a network. Suspicious documents are subjected to additional processing, including quarantining or sandboxing methods. Malicious documents are rejected from the network. The method includes combining machine learning with other network security methods, so as to further increase the capability of a network security system to detect malicious macros and malicious Object Linking and Embedding (OLE) files.SELECTED DRAWING: Figure 3

Description

Cross-References and Incorporation

[0001] This patent application is a continuation of U.S. Patent Application No. 17 / 184,478, and claims the priority of U.S. Non-Provisional Patent Application No. 17 / 572,548 (Attorney Docket No. NSKO-1038-2), titled "Detection Of Malicious Code that is Obfuscated Within A Document File", filed on January 10, 2022.

[0002] This patent application also claims the priority of U.S. Non-Provisional Patent Application No. 17 / 184,478 (Attorney Docket No. NSKO 1038-1), titled "Signatureless Detection of Malicious MS Office Documents Containing Advanced Threats in Macros", filed on February 24, 2021 and issued as U.S. Patent No. 11,222,112 on January 11, 2022.

[0003] This patent application also claims the priority of U.S. Non-Provisional Patent Application No. 17 / 184,502 (Attorney Docket No. NSKO-1040-1), titled "Signatureless Detection Of Malicious MS Office Documents Containing Embedded Ole Objects", filed on February 24, 2021.

[0004] This patent application claims the priority of all of the foregoing documents, including all patent applications, patent publications, and patents contained therein. All of the foregoing documents are also hereby incorporated by reference in their entirety for any and all purposes.

Technical Field

[0005] The disclosed technology relates to cybersecurity attacks and cloud-based security, and more specifically, to systems and methods for preventing malware attacks in which Microsoft Office documents act as a primary vector (method) for delivering malicious code in the form of macros and OLE objects. In addition, the disclosed technology relates to the detection of documents containing malicious macros and / or malicious OLE objects that do not have a known signature. In the context of the disclosed technology, unsigned means detecting malicious macros and malicious OLE objects that do not have a previously established signature. Furthermore, the disclosed technology uses machine learning and feature engineering to predict the presence of malicious macros and (unnecessarily) malicious OLE objects in MS Office documents and other document types without requiring (that the malicious code was previously known). [Background technology]

[0006] The subject matter considered in this section should not be assumed to be prior art merely as a result of its reference in this section. Similarly, the issues mentioned in this section, or issues related to the subject matter provided as background, should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches and may, in itself, correspond to implementations of the claimed technology.

[0007] In the six months leading up to the COVID-19 pandemic, Microsoft Office files accounted for only 5% of the Trojans and downloaders detected on the Netskope security cloud platform. Following COVID-19, particularly when EMOTET became active again, the percentage increased to nearly 45%, while portable executable files and other types decreased. Typically, Microsoft Excel files containing malicious links, VBA scripts, or PowerShell scripts account for nearly three-quarters of the malicious Office documents detected, and often contain malicious macros. Over 90% of malicious Office documents are distributed through cloud applications, compared to 50% for all other malicious file types combined.

[0008] Malware code creators are becoming increasingly sophisticated in finding ways to use MS Office to deliver malware payloads over secure networks. Documents containing malware employ advanced obfuscation techniques to conceal the malicious code, making them difficult to detect until they have caused significant damage to the network, and often going unnoticed.

[0009] To predict malicious content in MS Office files, it is necessary to detect obfuscated macros and OLE objects delivered through Microsoft Office document files that follow a VBA document object model, using feature engineering combined with machine learning. This problem is resolved by feature engineering, as determined by network security systems such as Netskope, which are used to train supervised machine learning algorithms to predict and distinguish between legitimate and secure documents, suspicious documents that may contain malware, and malicious documents that undoubtedly contain malicious code.

[0010] In this way, network devices can predict the presence of malware in document files without using known signatures against unknown malware. Additionally, the detection of malicious office files can occur in near real-time, significantly improving network security while reducing the negative impact on system throughput by decreasing latency in network security processing time. [Brief explanation of the drawing]

[0011] In the drawings, similar reference letters generally refer to the same parts throughout different drawings. Furthermore, the drawings are not necessarily to scale; instead, the focus is generally on illustrating the principles of the disclosed technology. In the following description, various implementations of the disclosed technology are described with reference to the following drawings.

[0012] [Figure 1] This document illustrates an architectural schematic of a system for detecting malicious content in MS Office macros and MS Office embedded OLE object files. The disclosed system uses machine learning and feature engineering to develop supervised training models for detecting malicious content in unsigned malicious data. [Figure 2] This document illustrates the malware detection methods of the Office classifier for detecting malware contained in macros within MS Office documents and OLE objects operating within the Netskope network security system, and in particular, shows the deployment of the ML-based Office classifier within the network security system. [Figure 3] This document illustrates an exemplary workflow for training a supervised machine learning model using one aspect of this technology. [Figure 4] This is a flowchart illustrating how potentially malicious files are classified and post-processed by the Office classifier. [Figure 5]This flowchart illustrates the steps (actions) required to detect embedded macros. [Figure 6] This flowchart illustrates the steps (actions) for detecting an embedded malicious OLE object file. [Figure 7] This is a simplified block diagram of a computer system that can be used to detect MS Office documents that contain malicious macros and embedded OLE content. [Modes for carrying out the invention]

[0013] The following detailed description will be made with reference to the drawings. Sample implementations are described not to limit the scope defined by the claims, but to illustrate the disclosed technology. Those skilled in the art will recognize various equivalent modifications based on the following description.

[0014] [Introduction] The disclosed technology relates to a feature engineering approach for machine learning-based classification of Microsoft Office documents, which significantly improves the efficiency of detecting malicious files or documents containing macros and OLE objects.

[0015] The disclosed technology relates to cybersecurity attacks and cloud-based security. The disclosed technology is a method and apparatus for detecting documents that have threats embedded in the form of malicious macros and malicious OLE objects. The disclosed technology uses a trained machine learning model to detect obfuscated malicious code and predict documents that have known unsigned malicious code, referred to as unsigned. Thus, the disclosed technology can predict which documents contain unsigned malicious code. Feature engineering is used to define a set of features for detecting malicious macros and malicious OLE objects based on features selected from a list of known characteristics and attributes of files that have historically shown malicious content. In one embodiment, the features and attributes of macro malware and OLE malware are determined by analysis of obfuscated malware code and stored in a heuristic database. Features from the database are selected and used to train a supervised machine learning model.

[0016] In another embodiment, the office classifier is configured to receive incoming documents over the network, analyze and process those documents, and apply machine learning algorithms to classify the documents as threat level: safe, suspicious, or malicious. Safe documents are allowed within the network. Suspicious documents are subject to additional processing, including isolation or sandboxing methods. Malicious documents are rejected or blacklisted from the network.

[0017] In a further embodiment, the disclosed technology combines machine learning sequentially or in parallel with other network security methods to further enhance the network security system's ability to detect malicious macros and malicious OLE files.

[0018] Most of the malware detection systems currently in use rely on signature-based detection methods. In signature-based detection methods, malware or viruses have (including) unique code patterns that can be used to detect and identify the presence of a particular malware or virus. Antivirus software scans file signatures and compares them with a database of known malicious codes. If they match, the file is flagged and treated as a threat. The main limitation of signature-based detection is that it can only flag known malware that includes code patterns known to be associated with a particular malware, which is signature-based detection, is less effective and may not be fully useful against new malware or zero-day attacks. Since it can only flag known malware, it is signature-based detection, is less effective and may not be fully useful against new malware or zero-day attacks.

[0019] The method and apparatus of the present invention are advantageously used for at least MS Office documents in the following file formats: Word97-2003 (.doc,.dot), Word2007+ (.docx,.docm,.dotm), Word2003XML (.xml), Excel97-2003 (.xls), Excel2007+ (.xlsx,.xlsm,.xlsb), and PowerPoint2007+ (.pptx,.pptm,.ppsm).

[0020] MS Office documents can contain embedded codes such as VBA (Visual Basic for Applications), DDE (Dynamic Data Exchange), and other files (jpg, mpeg, exe / pe files, etc.). In one aspect, the overall objective of the disclosed technology is to scan the embedded content of MS Office files to detect any malicious code in the file based on the function of the code.

[0021] [Macro Malware] Macros are a powerful way to automate common tasks in Microsoft Office and can enhance user productivity. However, macro malware uses this functionality to infect the user's endpoint devices. Macro malware is often disguised within Microsoft Office files and distributed via email attachments, ZIP files, or downloads from cloud-based sources.

[0022] In the past, macro malware was common because macros were automatically executed every time a document was opened. In the latest versions of Microsoft Office, macros are disabled by default. Here, malware creators need to induce users to turn macros on so that the malware can be executed. These files use names aimed at inducing or "scaring" users into opening them. Some files are disguised as official documents such as invoices, receipts, and legal documents. Other files often display fake warnings when malicious documents are opened and users are lured into accessing malicious content.

[0023] Many families of malware, such as EMOTET, ICEDID, and DRIDEX, use MS Office documents as a means of transmission to distribute them, especially due to the popularity of MS Office applications across large user bases on both Windows and Mac computers. The malware family EMPYRE is an example of a document macro that exploits Mac computers.

[0024] Macros are programs embedded in MS Office documents. All types of MS Office formats (documents, spreadsheets, presentations, etc.) have the functionality to include these macros. Macros written in VBA (Visual Basic for Applications) allow users to build user-defined functions (UDFs), automate processes, and access Windows APIs and other low-level functions through dynamic link libraries (DLLs). Malware creators exploit this functionality to perform malicious activities on the user's computer. Macros are stored in file folders. VBA components are Next, it is stored in a subfolder. A VBA component can be thought of as a stream containing references to VBA projects, directories, and documents.

[0025] [Javascript / JScript macros] Visual Basic can be used to invoke embedded JavaScript (JScript) code. On Windows, cscript.exe (executed via command line) and wscript.exe (executed via GUI) are the primary means of implementing the installed Active Scripting Language. Windows Script files (.wsf) are XML files executed by the Windows Script Host that can contain multiple scripts in multiple languages, in addition to other elements.

[0026] [Sample (non-malware) JScript MS Excel macro] VBA is an extremely powerful language that goes beyond native syntax because it can call component object model libraries, .NET libraries, or any Windows interface. By calling external program modules, VBA can achieve the full functionality of any Windows programming language.

[0027] The following sample xlsm file and macro demonstrate how to create a JScript macro in VBA for use within an Office document. This feature can be used to build runtime executable VBA, which is difficult to scan for malicious intent. The core snippet appears as follows: TIFF0007855636000001.tif39151

[0028] JScript is specified by a variable. A Script Control is instantiated and set to JScript. VBScript is a language that allows JScript to be embedded in VBScript code. This is how JScript code can be constructed and executed by VBScript code. The resulting JScript is outside the capabilities of current Microsoft malicious code detection and mitigation mechanisms.

[0029] [Exploited in malicious embedded macros in MS Office DOCS] mechanism] Malicious macros can be automatically triggered when an MS Office file is opened or closed. Trigger functions provided by the VBA API include Document_Open, Auto_Open, Workbook_Open, Document_Close, Workbook_Close, and several similar conditional call APIs. For example, the keyword or API Document_Open causes VBA code to execute when a document containing this macro is first opened. While this API was designed by Microsoft for a useful purpose, it can be exploited in maliciously hijacked documents.

[0030] All malicious macros must act on the underlying system. To achieve this, the malicious payload can be dropped either as a file or as shellcode injected into another process. The execution of that payload is then triggered in some way. Most macros do both, but some samples can deliver the payload with only a write or only an execute operation.

[0031] A significant number of macros observed since 2014 download payloads using HTTP. However, this is not always the case, and some macros contain payloads encoded in VBA code or documents. The download action itself is not distinctive enough to be used for detection. While some legitimate macros use the same characteristics as malicious macros, it is unlikely that such macros would combine all of the methods described above.

[0032] [Obfuscation] Most malicious macros are obfuscated. Malicious creators use various techniques to hide the behavior of macros and to evade code strings that would be useful for analysis and detection, i.e., indicators of compromise (IOCs) such as IP addresses and URLs. Fortunately, while many VBA keywords used by malware must be expressed in plain text, parameters can be strings. Plain text mechanisms for automatic triggering, payload dropping, and payload execution cannot be executed from strings and therefore cannot be obfuscated. VBA does not provide the ability to execute code stored in strings, in contrast to JavaScript with eval() and VBScript with Execute.

[0033] For more effective detection, based on research into malicious VBA code, obfuscation methods can be categorized into four types, as described below. 1) Random obfuscation 2) Splitting and obfuscating 3) Encoding obfuscation 4) Logical obfuscation

[0034] Each obfuscation type has a different syntactic structure and different uses of functions and operators. As will be explained further later, feature extraction from VBA macros is directed towards four types of obfuscation. The features characterizing obfuscation in these four categories are constructed based on the following analysis. The basic purpose of using these obfuscation techniques is to delay the analysis time, which in turn delays post-detection countermeasures. Each obfuscation method is quite simple, but when used in combination, they make the code visually unreadable.

[0035] In addition, malware creators use obfuscation tools to create many variants of the malware with different hash values, which can serve as a digital footprint of the file, where the file has been processed through an encryption algorithm, yielding a unique numerical value for that file.

[0036] [Random obfuscation] Random obfuscation makes VBA code unreadable by using meaningless or misleading token names. This random obfuscation can be characterized by features that use the Shannon Entropy measure of the VBA code. [Example of random obfuscation: Function names and variable names are simply random characters.] TIFF0007855636000002.tif38130

[0037] [Split obfuscation] Split obfuscation is used to concatenate strings that differ from the first occurrence, such as filenames or URLs. Split obfuscation can be characterized by counting the occurrences of string operators such as "+", "=", and "&". These operators can be used to obscure malicious code. [Example of data obfuscation: Splitting data into multiple variables / constants] TIFF0007855636000003.tif34130 [Obfuscation using built-in functions Replace() / split(): Use VBA built-in functions such as Replace / split to obfuscate data] TIFF0007855636000004.tif26160

[0038] [Encoding obfuscation] Encode obfuscation works with parameters to generate malicious code that is significantly different from what initially appears. Examples of functions that generate encoded obfuscation include, categorized as follows: • (Text functions): Asc(), Chr(), Mid(), Join(), InStr(), Replace() , Right(), StrConv(), etc. (Arithmetic functions): Abs(), Atn(), Cos(), Exp(), Log(), Randomize(), Round(), Tan(), Sqr(), etc. (Type conversion functions): CBool(), CByte(), CChar(), CStr(), CDec(), CUInt(), CShort(), etc. • (Financial functions): DDB(), FV(), IPmt(), PV(), Pmt(), Rate(), SLN(), SYD(), etc.

[0039] [Logical obfuscation] Logical obfuscation complicates code and makes analysis more difficult. This technique is employed by declaring unused variables or by using redundant function calls and loops. The following snippet extracted from the sample demonstrates that only one line of code is involved in creating the OLE object, with the rest being non-functional. TIFF0007855636000005.tif100160

[0040] Logical obfuscation refers to obscuring the detection of one or two lines of code that contain malicious intent by using long code sequences or comments. Features that may be generated to characterize logical obfuscation include the length of the VBA code excluding comments, and the length or size of comments within the VBA code. MS Office files can be encrypted and require a password to decrypt and open, but VBA macros, along with the rest of the document, are not encrypted. Therefore, malicious macros cannot be obscured by encryption. VBA projects can be password-protected; however, MS Office implements this as logical protection. Because VBA code is not encrypted in the file, it can be extracted in plain text using tools such as OLETOOLS.

[0041] [System Overview] The following describes, in architectural terms, a system and various implementation forms for preventing macro and OLE object malware attacks in cloud-based systems, including off-site cloud-based stores. The architectural diagram of System 100 is shown in Figure 1, which has been deliberately simplified for clarity in the explanation. Figure 1 shows the interconnections of various key elements. The use of these elements will be described in more detail in relation to considerations of their structure and usage.

[0042] Figure 1 shows a system 100 including an endpoint 142. User endpoints 142 may include devices such as computers 144, smartphones 146, and computer tablets 148 that provide access to and interaction with data stored on a cloud-based store 136 and cloud-based services 138. An inline proxy 132 intervenes between user endpoints 142 and cloud-based services 138 via the network 140, in particular through a network security system including a network administrator 122, network policies 124, evaluation engine 126, office classifier 127, threat scanning subsystem 128, sandbox 130, and metadata store 134, which will be described in more detail. The inline proxy 132 may be accessible via the network 140 or may reside as part of the network security system 120. It provides monitoring and control of traffic between the dopoint 142, the cloud-based store 136, and other cloud-based services 138.

[0043] The inline proxy 132 monitors network traffic between the user endpoint 142 and the cloud-based service 138, in particular to enforce network security policies, including data loss prevention (DLP) policies and protocols.

[0044] Here, the interconnection of the elements of system 100 will be described. Network 140 connects computer 144, smartphone 146, and computer tablet 148 with metadata store 134, and connects them to each other in inline proxy 132. The communication path may be point-to-point over public and / or private networks. Communication may occur via various networks including private networks, VPNs, MPLS circuits, or the Internet, and through appropriate application programming interfaces (APIs), and

[0045] Data exchange formats such as Representational State Transfer (REST), JavaScript Object Notation (JSON), Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), Java Message Service (JMS), and / or Java Platform Module System can be used.

[0046] Communications can be encrypted. Communications generally travel over networks such as LANs (Local Area Networks), WANs (Wide Area Networks), telephone networks (Public Switched Telephone Networks (PSTN)), Session Initiation Protocol (SIP), wireless networks, point-to-point networks, star networks, token ring networks, hub networks, and the internet, including mobile internet, via protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX.

[0047] The engine or system component in Figure 1 is implemented by software running on various types of computing devices, such as workstations, servers, computer clusters, blade servers, or server farms. Additionally, various authorization and authentication techniques, such as usernames / passwords, Open Authorization (OAuth), Kerberos, SecureID, and digital certificates, can be used to secure communications.

[0048] Cloud-based services 138 provide users with functionality implemented in the cloud or over the internet. Cloud-based services 138 may include internet-hosted services such as news websites, blogs, video streaming websites, social media websites, hosting services, cloud applications, cloud stores, cloud collaboration and messaging platforms, and / or cloud customer relationship management (CRM) platforms. Cloud-based services 138 can be accessed using a browser (via a URL) or a native application (a sync client).

[0049] The 138 categories of cloud-based services include Software as a Service (SaaS) offerings, Platform as a Service (PaaS) offerings, and Infrastructure as a Service (IaaS) offerings. Examples of common web services today include YouTube (registered trademark), Facebook (trademark), Twitter (trademark), Google (trademark), LinkedIn (registered trademark), Wikipedia (trademark), Yahoo (trademark), Baidu (trademark), Amazon (trademark), MSN (trademark), Pinterest (trademark), Taobao (trademark), Instagram (trademark), Tumblr (trademark), eBay (trademark), Hotmail (trademark), Reddit (trademark), IMDb (trademark), Netflix (trademark), PayPal (trademark), Imgur (trademark), Snapchat (trademark), Yammer (trademark), Skype (trademark), Slack (trademark), HipChat (trademark), Confluence (trademark), TeamDrive (trademark), Taskworld (trademark), Chatter (trademark), Zoho (trademark), ProsperWorks (trademark), Google's Gmail (trademark), Salesforce.com (trademark), Box (trademark), Dropbox (trademark), Google Apps (trademark), and Amazon. Examples include AWS®, Microsoft Office 365®, Workday®, Oracle on Demand®, Taleo®, Jive®, and Concur®.

[0050] Users belonging to a corporate organization access hundreds of cloud-based service providers to generate, store, collaborate on, and share data with other users inside and outside the organization. This can lead to malware attacks. The cloud-based service 138 provides functionality to users of organizations that have implemented security policies. When a user sends a request to the cloud-based service via endpoint 142, the inline proxy 132 intercepts the request message. The inline proxy 132 attempts to identify the accessed cloud-based service 138 by accessing a database. In one implementation, according to the present invention, the inline proxy stores metadata in the request message from the user to the metadata store 134 in order to identify the accessed cloud-based service 138. When document instances are shared or linked between the cloud-based store 136 and the user endpoint 142 over the network, there is a risk of malware attacks.

[0051] [office classifier] The office classifier 127 is shown in detail in Figure 2. The office classifier framework 202 receives office documents that may or may not contain malicious code in the wild. The term "in the wild" generally refers to malicious programs that are already out in the public sphere and cause various kinds of damage. The office classifier 127 is an integral part of the network security system 120 shown in Figure 1 and works in cooperation with other elements of the network security system 120.

[0052] The office classifier framework 202 predicts malicious content in document files using supervised training of machine learning model 208. The machine learning model is trained using labeled training data and machine learning algorithms, as described in relation to Figure 3. The features used to train the machine learning model will be discussed further later. The determination of these features is based on an analysis of methods used by malware authors to embed malicious macros and malicious OLE objects using obfuscation and other known techniques to make it difficult to detect malicious content.

[0053] In one embodiment, the Office classifier 127 can use a heuristic feature generator 206 to generate a list of features derived from the properties and attributes of malicious macros and malicious OLE objects, which can be used to train an advanced boosting tree-based machine learning algorithm 208. The Office classifier 127 can then classify new Office documents as threat levels based on those features. In another embodiment, the heuristic feature generator 2 06 can correlate the characteristics of malicious macros and malicious OLE objects with a set of keywords, uniquely defining the set of characteristics extracted from Office documents, providing distribution to machine algorithms, and resulting in highly accurate detection results.

[0054] The office classifier 127 uses features extracted from MS Office documents, specifically from the following embedded artifacts. These features are used to train the machine learning (ML) model 208. 1) Macro (VBA code) 2) DDE 3) Embedded OLE object

[0055] Unlike most traditional ML and heuristic-based solutions, Office Classifier 127 does not rely solely on static features such as the following: 1) APIs in VBA code. 2) The URL to which the macro connects. 3) Automatic execution. 4) Embedded PE / document files.

[0056] Instead, the office classifier 127 collects these features, along with metadata (document size, pages, paragraphs, etc.) and other artifacts such as code entropy, which helps detect the amount of obfuscation used by the macro code. Through the analysis of many instances of obfuscated macro code, it has been observed that the types of obfuscation used by malware authors can be categorized, as mentioned above, into 1) logical, 2) encoded, 3) split, and 4) random obfuscation.

[0057] As mentioned earlier, malware creators often use these obfuscation methods to inflate analysis time, which in turn delays post-detection countermeasures. Each obfuscation method is quite simple, but when used in combination, they make the code visually unreadable. In addition, malware creators use obfuscation tools to create many variants of malware with different hash values.

[0058] Malware creators appear to be using such obfuscation techniques in many malware campaigns, including the recent one by EMOTET. As explained above, obfuscation is primarily achieved using VBA language features such as string operators and functions. Office Classifier 127 successfully collects such unique indicators from VBA code through machine learning and heuristic engines.

[0059] The technology described herein is intended for detecting both OLE2 and OXML type documents of MS Office, covering Word, Excel, and PPT. The feature extraction activity essentially involves the extraction and analysis of necessary items such as VBA code, DDE, and embedded items from Office documents, also referred to herein as documents, files, or document files. None of the features are absolute markers; rather, the extracted group of features contributes to the classification.

[0060] Continuing with Figure 2, potential false positives 212 and emerging threats 214 may require analyst review 216 to improve the level of malware detection, which may require some adjustments within the office classifier framework 202. In addition, other threat detection engines 218 may work in parallel with the machine learning and heuristic engines 208 to improve the overall performance of malware detection and significantly improve the final detection results 210.

[0061] [Machine Learning and Feature Engineering] The techniques disclosed for machine learning and feature engineering are shown in detail in Figure 3.

[0062] A document file may contain multiple macros and embedded files. For the purpose of feature extraction, all macros are considered and combined as a single entity. Macro code and embedded file information are extracted from documents (all formats), including both CFBF (Compound File Binary Format) and OpenXML. Features from macros and embedded OLE objects are extracted for processing by a machine learning algorithm that detects malicious code in Microsoft Office documents. The construction of feature vectors and the list of associated features are described further below.

[0063] Figure 3 illustrates the training of a supervised machine learning model 312. Training uses preferred machine learning algorithms 310 such as random forests, decision trees, and linear regression. Alternatively, the machine learning algorithm can be a convolutional neural network, CNN, which includes deep learning structures such as Inception. In supervised machine learning, the training data includes classification labels 316. Training text, documents, and images 314 are used to extract features. Ideally, this sampling should be large, with around one million samples, so that they are extracted and retained in a .csv file for further machine learning (ML) processing by data scientists.

[0064] Sample collection should avoid duplication and be leveled. Ideally, sample collection should include a combination of labeled malicious files and clean document files, including documents that have been shown to be false positives (FP) within a network security environment such as Netskope.

[0065] When the feature vectors 318 are identified and labeled, they are combined by the machine learning algorithm 310 to create a predictive model 312.

[0066] New unlabeled data in 320 forms, such as new text, documents, and images, is classified through selected feature vectors 322 and input into a predictive model 312. The predictive model processes the new data 320 312 and provides expected labels 324 as the final result.

[0067] Now, looking at Figure 4, we see a flowchart of the Office classifier 127, illustrating how potentially malicious files are processed and classified according to the disclosed technology.

[0068] The office classifier 127 is configured to receive incoming documents 400 over the network, analyze them, extract features, and apply machine learning algorithms to classify the documents as threat level, secure level 480, suspicious level 500, or malicious level 490. Secure documents 480 are permitted within the network. Suspicious documents 500 are subject to additional processing, including isolation or sandboxing methods 510. Malicious documents 490 are warned against and ultimately rejected or blacklisted from the network. Detected malicious files 490 may be isolated for further research and analysis. In one embodiment, documents may undergo a detailed threat scan by a security administrator, which may also include isolation in a sandbox 130, where any executable embedded code is executed in an isolated environment to determine whether any embedded link triggers malicious activity.

[0069] In a further embodiment, the disclosed technology combines machine learning with other network security methods 410, 420 to further enhance the network security system's ability to detect malicious macros and malicious OLE documents.

[0070] [Feature engineering using previously detected malware instances] In one embodiment, the disclosed technology uses a machine learning algorithm. The technology is configured to derive a list of features, which can be derived from a previous malware attack, with or without heuristics to assist in the derivation. The features are used to train a machine learning algorithm.

[0071] In another aspect, the disclosed technology also detects malicious embedded OLE objects within Office documents. These features, extracted from leveled samples, are used to train a supervised model using a boosting tree algorithm. Features combined from different categories provide very good variance to the machine learning algorithm. Heuristics can be used to help derive a set of feature vectors for training the machine learning algorithm.

[0072] A combination of two approaches—machine learning for zero-day and recurring malicious patterns, and heuristics for detecting recognized malicious patterns—provides excellent results in detecting malicious document files in the form of embedded macros and OLE objects. The disclosed technique uses feature engineering from a heuristic engine to train a machine learning algorithm.

[0073] Figure 5 is a flowchart illustrating the steps (actions) for detecting malicious embedded macros. In step (action) 500, the document file is received by the network security system. The network security system is configured for analysis, and the document file is analyzed to separate metadata from the malicious payload data. In step (action) 510, the heuristic engine within the office classifier 127 uses data that shows past instances of malware embedded in macros, using known obfuscation methods. In step (action) 520, a feature set can be derived from the data provided by the heuristics. The feature set is used in part to train a machine learning algorithm model using machine learning methods to predict the likelihood that a document file contains malicious macro code. In step (action) 530, the trained machine learning model is used to predict the likelihood that an input document may contain a malicious macro. In step (action) 540, the heuristic rules derived from instances of malicious macros are applied to increase the success rate of detecting malicious macros in document files. In step (action) 550, the office classifier 127 classifies the resulting document files as safe, suspicious, or malicious. In step (action) 560, safe documents are accepted into the network system, malicious documents are blocked, and suspicious documents are quarantined for further threat analysis, including sandboxing.

[0074] Figure 6 is a flowchart illustrating the steps (actions) for detecting a malicious embedded OLE object. In step (action) 600, the document file is received by the network security system. The network security system is configured for analysis, and the document file is analyzed to separate metadata from the malicious payload data. In step (action) 610, the heuristic engine in the office classifier 127 uses known obfuscation methods to use data that indicates past instances of the malicious embedded OLE object. In step (action) 620, the heuristic engine is configured to derive a feature set, which can be derived from data provided by the heuristic. The data is used in part to train a machine learning algorithm model using machine learning methods to predict the likelihood that a document file contains malicious embedded OLE object code. In step (action) 630, the trained machine learning model is used to predict the likelihood that an input document may contain a malicious embedded OLE object. In step (action) 640, heuristic rules derived from instances of malicious embedded OLE objects are applied to increase the success rate of detecting malicious embedded OLE objects in document files. In step (action) 650, the office classifier 127 classifies the resulting document files as safe, suspicious, or malicious. In step (action) 560, safe documents are accepted into the network system, malicious documents are blocked, and suspicious documents are isolated for further threat analysis, including sandboxing. [A wide range of features extracted from MS Office documents] [Macro] 1) Features indicating the count of VBA and Windows API usage, such as CreateObject, Shell, FileSystem, URLDownloadToFile, CallByName, Detect Sandbox, Virtualization, and Anti-Debugging. Collecting these counts provides variance for both ML and uristic approaches when compared to features in a clean document. 2) A feature that indicates the count of automatic execution in VBA macros (when a document is opened or closed, or when a VBA control event is clicked). 3) Features that indicate the count of string operators, such as using VBA string operators (concatenation) like +, &, and =. 4) Features that indicate the count of text, type conversion, arithmetic and financial functions, using VBA functions such as Asc(), Chr(), ChrW, CBool(), CStr(), CLng(), Abs(), Atn(), RND(), DDB(), FV(), and IPmt() in VBA macros. 5) Features indicating the OLE stream profile, such as the number of OLE streams and valid passcodes. 6) Features that show the VBA code profile, such as code lines, comments, variables, functions, loops, events, hexadecimal strings, and counts such as entropy. [Static: Configures static identification of the document.] 1) Document size, document type (Word / Excel / PPT), creation / modification time, and revision number. 2) Number of pages, number of paragraphs, number of lines, number of characters 3) The document is a template, and VBA code exists in the document metadata. [DDE: Configures dynamic data execution code / string features, if available.] 1) When DDE can be executed automatically. 2) If DDE is using trusted Windows utilities such as cmd.exe, powershell, wmi, wscript, and cscript utilities. 3) If a URL exists in the DDE, access it via a hyperlink. 4) If DDE is using executable files, file downloads, registry scripts, add-ins, or macro buttons. [Embedding: Configure features from embedded objects within an Office document] 1) Embedded objects containing suspicious files such as exe, dll, 7z, dmg, deb, rar, etc. 2) Embedded objects containing questionable files such as sql, cer, cfm, etc. Ekto. 3) Embedded objects that have other files, such as those without an extn. 4) An embedded object that has an external hyperlink to a URL.

[0075] [Test Results] Using a test corpus of 2,000 malicious MS Office samples with macros from VirusTotal and a randomly selected set of 2,000 clean samples from VirusTotal, the disclosed technology can detect 87.4% of malware samples without false positives, and when combined with other malware detection engines such as those provided by Netskope, an accuracy of 98.9% without false positives is achieved. This architecture provides proactive and versatile detection against malware samples that employ similar obfuscation and other tactics used in previously analyzed samples.

[0076] [Features of macros derived from VBA:] MACRO_IS_PRESENT - This feature indicates whether a macro exists in the document being scanned. This can be either a count or a Boolean value.

[0077] MACRO_AUTOEXEC - This feature indicates whether a macro is automatically executed when a scanned document is opened. This can be a Boolean value. Functions similar to AutoExec that trigger macros based on events or behavioral states include Document_Open and Document_Close. This group of functions can be counted collectively. A single instance of event-triggered macro execution may suggest malicious coding. Multiple instances of event-triggered macro execution may suggest legitimate code. ML is adept at making such distinctions for this feature and others.

[0078] MACRO_EXECUTE - This feature indicates whether the macro invokes code that includes external code. This can be either a count or a boolean value.

[0079] MACRO_EXECUTE_POWERSHELL - This function indicates whether the macro will trigger the launch of Windows PowerShell to execute the script. This can be either a count or a boolean value.

[0080] MACRO_WRITE - This function indicates whether a macro will send data to the disk on the same computer or to a network location. Many programs use this function to legitimately write log files. One consideration that can come into play with this feature is whether the file being written is a binary file or a text log file.

[0081] MACRO_HAS_INTERNETDOWNLOAD - Downloads are a more important feature than simply connecting to the internet.

[0082] MACRO_HAS_REGISTRY_ACCESS - Writing to or editing the registry may indicate malicious intent.

[0083] MACRO_COMMENT_LINES - It has been observed that there are fewer commented-out lines of malicious code than VBA code that needs to be maintained.

[0084] MACRO_CODE_LINES - Malicious macro families typically have a certain number of lines.

[0085] MACRO_HAS_HEX_STR - The use of hex encoding is rare in standard VBA macro code. Some composite features are constructed from other extracted features. Here are three examples based on engineering judgment.

[0086] MACRO_RULE1 # MACRO_RULE1- MACRO_AUTOEXEC = True, MACRO_WRITE = True, and MACRO_HAS_HEX_STR = False

[0087] MACRO_RULE2 # MACRO_RULE2- MACRO_AUTOEXEC = True, MACRO_WRITE = True, and MACRO_HAS_HEX_STR = True

[0088] MACRO_RULE3 # MACRO_RULE3- MACRO_AUTOEXEC = True, MACRO_WRITE = True, and MACRO_HAS_HEX_STR = False

[0089] MACRO_OLESTREAM_COUNT - Malicious macros do not have OLE streams. In contrast, many OLE streams often appear in legitimate documents that are repeatedly updated. MACRO_OLE_PASSCODE-# is the project protection status of an OLE / VBA passcoded project stream.

[0090] MACRO_DETECT_SANDBOX - This feature indicates whether the macro attempts to detect sandbox execution, such as detecting Anubis, Sandboxie, Norman, CW, Winjail, or other types of sandboxes. A non-malicious application has no reason to detect whether it is running in a sandbox. Marcos does not need to run in a sandbox because it has been thoroughly analyzed.

[0091] MACRO_DETECT_VIRTUALIZATION - Detection of virtualization or debug mode is also more general. Virtualization is a more general feature than looking for a specific sandbox.

[0092] MACRO_RUN_SHELLCODEINMEMORY - A VBA macro allows execution of shellcode in memory. While not widely used currently, it has the potential for misuse.

[0093] MACRO_DISABLE_SECURITY - There is little reason for legitimate code to disable VBA macro security or Protected View. This strongly suggests malicious intent.

[0094] The following series of characteristics are relevant to the above considerations regarding obfuscation.

[0095] MACRO_SELF_MODIFICATION - As mentioned above, macros can attempt to modify (self-correct) VBA code, including constructing parameters for code executable in languages ​​other than VBA. Splitting and encoding obfuscation are notable means of self-correction.

[0096] MACRO_NUM_STRINGOPS - The count of string operators such as "+", "=", or "&" can indicate splitting obfuscation. Encoded obfuscation features can include multiple counts of various function types.

[0097] MACRO_NUM_TEXTFUNC is a count of text functions, including Asc(), Chr(), Mid(), Join(), InStr(), Replace(), Right(), StrConv(), etc.

[0098] MACRO_NUM_ARITHFUNC is a count of arithmetic functions, including Abs(), Atn(), Cos(), Exp(), Log(), Randomize(), Round(), Tan(), Sqr(), etc.

[0099] MACRO_NUM_TYPECONVFUNC is a count of type conversion functions, including CBool(), CByte(), CChar(), CStr(), CDec(), CUInt(), CShort(), etc.

[0100] MACRO_NUM_FINCFUNC is a count of financial functions, including DDB(), FV(), IPmt(), PV(), Pmt(), Rate(), SLN(), SYD(), etc.

[0101] Logical obfuscation features overlap with random obfuscation features, and both can be addressed by the entropy measure.

[0102] MACRO_SHANNON_ENTROPY is the Shannon entropy score of the VBA macro code.

[0103] [Embedded files and features:] [OLE Stream] MS Office document files are scanned for embedded files. The following OLE streams (x01Ole10Native, x01CompObj, and ObjectPool) are decoded to collect embedded file information. The following are features extracted from the embedded files. EMBED_NONE EMBED_HAS_CORRUPT EMBED_HAS_PASSWORD_PROTECTED The EMBED_HAS_SUSPICIOUS_BIN file is It originates from one of the files / extn listed in the suspicious bin. EMBED_HAS_SEMI_SUSPICIOUS_FILE - The file is from one of the files / extns listed in semi_suspicious_files. EMBED_HAS_NORMAL_FILE - The file is from one of the files / extns listed in normal_files. EMBED_HAS_OTHER_FILE - The file does not belong to any of the above categories and is not a compressed file. EMBED_HAS_COMPRESSED_FILE - The file is from one of the files / extns listed in compressed_files. EMBED_HAS_NON_EXT EMBED_RULE1 - This is EMBED_RULE1. EMBED_HAS_SUSPICIOUS_BIN = True, or EMBED_HAS_OTHER_FILE = True

[0104] [Heuristic rules:] MS Office files may contain malware if they are embedded in a suspicious bin (Category A) or an unknown extension (Category E). set: Suspicious bins (A) = "7z", "apk", "app", "asp", "aspx", "bat", "bin", "cab", "cgi", "com", "cpl", "crx", "deb", "dll", "dmg", "dmp", "drv", "exe", "gadget", "jar", "js", "jse", "jsp", "lnk", "msi", "php", "pkg", "pl", "plugin", "py", "rar", "rpm", "sh", "sys", "torrent", "vbs", "wsf", "zip", "zipx" Semi-doubtful (B) = "cer", "cfm", "csr", "deb", "dcr", "db", "dbf", "docm", "mdb", "pdb", "sql", "swf", "xlsm" Normal (C) = "3ds", "3dm", "ai", "aif", "bmp", "css", "csv", "doc", "docx", "dwg", "dxf", "eps", "flv", "fnt", "fon", "gif", "htm", "html", "indd", "jpg", "key", "log", "m3u", "m4a", "max", "mid", "mov", "mp3", "mp4", "mpg", "obj", "otf", "pages", "pct", "pdf", "png", "ppt", "pptx", "ps", "psd", "pspimage", "rm", "rtf", "svg", "tex", "tga", "thm", "tif", "tiff", "ttf", "txt", "vob", "wav", "wma", "wmv", "wpd", "wps", "yuv", "xlr", "xls", "xlsx" No extension (D) = File name does not contain "." Compressed files (F) = "7z", "cbr", "gz", "rar", "sitx", "tar", "zip", "zipx"

[0105] Unknown extension (E) = A file name not found in sets A, B, C, D, or F. This means it has an extension, but it is not defined in any other set.

[0106] [Static attributes and characteristics:] The following are static attributes extracted as part of feature extraction from MS Office document files. Machine learning (ML) models are trained with these features to enhance their ability to detect malicious documents. DOC_NUM_PAGES DOC_NUM_WORDS DOC_NUM_LINES DOC_NUM_CHARS DOC_NUM_PARAGRAPH DOC_LASTMOD_TIME DOC_AUTHOR_INFO DOC_REVISION_NUMBER DOC_LASTPRINT_TIME DOC_LINK-IS DIRTY DOC_LANGUAGE DOC_SIZE

[0107] [Comparison with other detection methods] Hash-based detection: The techniques disclosed are more general and aggressive, and hash-based detection covers only one sample.

[0108] Antivirus signatures: The techniques disclosed are more general and aggressive. This is not a simple string pattern match.

[0109] Fuzzy hashing: The techniques disclosed are far more common. Fuzzy hashing is also simply a matching of fuzzy byte / string patterns.

[0110] Pure machine learning models: The disclosed techniques use hybrid models to achieve a lower false positive (FP) rate while responding more quickly to new attacks.

[0111] Sandbox-based detection: The disclosed technology uses static detection and is far less scalable. It can also achieve a much lower false positive (FP) rate.

[0112] [Computer System] Now, looking at Figure 7, a simplified block diagram of a computer system 700 that may be used to provide security in a network within a cloud-based network for delivering cloud-based services is shown. The computer system 700 includes at least one central processing unit (CPU) 704 that communicates with some peripheral devices via a bus subsystem 726, and a network security system 120 for providing the network security services described herein. These peripheral devices may include, for example, a storage subsystem 708 including memory devices 722, 724 and a file storage subsystem 712, a user interface input device 714, a user interface output device 716, and a network interface subsystem 718. The input and output devices enable user interaction with the computer system 700. The network interface subsystem 718 provides an interface to an external network, including interfaces to corresponding interface devices in other computer systems. In one implementation, the network security system 120 in Figure 1 is communicably linked to the storage subsystem 708 and the user interface input device 714. The user interface input device 714 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphic tablets, audio input devices such as scanners, touchscreens integrated into displays, speech recognition systems, and microphones, and other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computer system 700.

[0113] The user interface output device 716 may include non-visual displays such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include flat panel devices such as LED displays, cathode ray tubes (CRTs), liquid crystal displays (LCDs), projection devices, or any other mechanism for creating visible images. The display subsystem may also provide non-visual displays such as audio output devices. In general, the use of the term “output device” refers to all possible types of devices for outputting information from the computer system 700 to a user or another machine or computer system. It is intended to include methods.

[0114] The storage subsystem 708 stores programming and data structures that provide some or all of the functionality of the modules and methods described herein. The additional subsystem 720 may be a graphics processing unit (GPU) or a field-programmable gate array (FPGA).

[0115] The memory subsystem 710 used within the storage subsystem 708 may include some memory, including main random access memory (RAM) 722 for storing instructions and data during program execution, and read-only memory (ROM) 724 for storing fixed instructions. The file storage subsystem 712 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing functions of a particular implementation form may be stored by the file storage subsystem 712 within the storage subsystem 708 or in other machines accessible by the processor 704.

[0116] The bus subsystem 726 provides a mechanism for various components and subsystems of the computer system 700 to communicate with each other as intended. Although the bus subsystem 726 is schematically shown as a single bus, alternative implementations of the bus subsystem may utilize multiple buses.

[0117] The computer system 700 itself can be of various types, including personal computers, portable computers, workstations, computer terminals, network computers, televisions, mainframes, server farms, widely distributed sets of loosely networked computers, or any other data processing systems or user devices. Due to the constantly changing nature of computers and networks, the description of the computer system 700 shown in Figure 7 is intended only as a specific example to illustrate preferred embodiments of the invention. Many other configurations of the computer system 700 are possible, having more or fewer components than the computer system 700 shown in Figure 7.

[0118] [Specific implementation form] The disclosed technologies can be practiced as systems, methods, devices, products, computer-readable media, or articles of manufacture. One or more features of an implementation can be combined with a base implementation. Non-exclusive implementations are taught to be combinable. One or more features of an implementation can be combined with other implementations. This disclosure periodically reminds the user of these options. The omission of some implementations from the repeating enumeration of these options should not be construed as limiting the combinations taught in previous sections. These enumerations are incorporated herein by reference in each of the following embodiments.

[0119] The disclosed technology relates to cybersecurity attacks and cloud-based security. The disclosed technology is a method and apparatus for detecting documents having embedded threats in the form of malicious content (malware), including malicious content, malicious macros, and / or malicious OLE objects, which are also referred to herein as malicious code and / or malware. Macros are stored in macro data, which are also referred to herein as macro data. Similarly, OLE objects are stored in OLE object data, which are also referred to herein as OLE object data, or embedded It is also called embedded OLE object data.

[0120] In one embodiment, the disclosed technology is a method for classifying input documents in a networked system to determine whether at least one of the documents may contain macros and / or embedded OLE objects containing malicious code (malware). The method includes repeatedly receiving document files into a network security system. Each document file is parsed to separate macro data and / or embedded OLE data from the document payload data. The method is configured to generate, for each document file, at least an obfuscation scoring feature that indicates a past instance of embedded malware using a known obfuscation method. The process then inputs the obfuscation scoring feature into a trained machine learning model, applies the trained machine learning model to process the document files, and determines whether the document files contain malicious content (malware) in the form of malicious macro data and / or malicious OLE data. The resulting documents are classified as safe, suspicious, or malicious. Based on the classification step (action), safe document files are accepted into the network system. Malicious documents are blocked as malicious. Suspicious documents are quarantined for threat analysis.

[0121] In another aspect of the disclosed technology, the obfuscation scoring feature includes one or more of the following features describing the embedded VBA macro feature, and in at least some embodiments, at least five: Macro_IS_present, macro_autoexec, macro_execute, macro_execute_powershell, Macro_write, macro_has_internet_download, macro_has_registry_access, Macro_comment_lines, macro_code_lines, macro_has_hex_str, macro_olestream_count, macro_ole_passcode, macro_detect_sandbox, macro_detect_virtualization, macro_run_shell_code_in_memory, Macro_disable_security, macro_self_modification, macro_num_stringops, macro_num_textfunc, macro_num_arithfunc, macro_num_typeconvfunc, macro_num_fincfunc, and macro_shannon_entropy.

[0122] In another aspect of the disclosed technology, the obfuscation scoring feature includes one or more of the following features describing an embedded VBA OLE object feature, and in at least some embodiments, at least five: embed_none embed_has_corrupt embed_has_password_protected embed_has_suspicious_bin - # The file is from one of the files / extn listed in suspicious_bins embed_has_semi_suspicious_file - # The file is from one of the files / extn listed in semi_suspicious_files embed_has_normal_file embed_has_other_file embed_has_compressed_file absorb_has_non_ext embed_rule1 - # embed_rule1 embed_has_suspicious_bin = true, and embed_has_other_file = true.

[0123] In a further aspect of the disclosed technology, the document file is an MS Office document. The MS Office document may be a Word document, which is a word processing document; an Excel document, which is a spreadsheet document; or a PowerPoint document, which is a graphical drawing document and is also referred to herein as a presentation document.

[0124] The disclosed technology scores macro-related and / or OLE object-related features, including one or more, in some embodiments, VBA macro features, and at least two, in an obfuscation scoring step (action): createobject, shell, filesystem, urldownloadtofile, Callbyname, and Detective Sandbox.

[0125] The disclosed technology detects OLE object-related features in the obfuscation scoring step (action) that include at least two of the following VBA OLE object features: createobject, shell, filesystem, urldownloadtofile, Callbyname, and Detective Sandbox.

[0126] In another aspect of the disclosed technology, the trained machine learning model is input into one or more, in some embodiments at least two, features derived from the following document features: Document size, Creator information, Document type (Word / Excel / PPT), Time of creation or modification and revision number, Number of pages, Number of paragraphs, Number of rows, and Number of characters.

[0127] In another aspect of the method, the disclosed technology includes a secondary malware detection engine that operates in parallel to improve the accuracy of malware detection and eliminate false positives.

[0128] In another aspect of the disclosed technology, the trained machine learning model is input with one or more features derived from the following static document features, in some embodiments, at least two features (action): doc_num_pages doc_num_words doc_num_lines doc_num_chars doc_num_paragraph hys_lastmod_time doc_author_info doc_revision_number doc_lastprint_time doc_link-is dirty doc_language docopensize

[0129] With respect to the static document features described above, doc_num_pages is also referred to herein as the number of pages, doc_num_words as the number of words, doc_num_lines as the number of lines, doc_num_chars as the number of characters, doc_num_paragraph as the number of paragraphs, doc_lastmod_time as the last modification time or modification time, doc_author_info as the author information, doc_revision number as the document revision number or revision number, doc_lastprint_time as the last print time or last print time, doc_link-is dirty as the dirty document link, doc_language as the document language or the language of the document, and doc_size as the document size or the size of the document.

[0130] In another aspect of the method, the disclosed technology includes a secondary malware detection engine that operates in parallel to improve the accuracy of malware detection and eliminate false positives.

[0131] In the threat analysis step (action) of the method, the document file may be isolated in a sandbox test environment to test one or more macros and / or one or more embedded OLE objects in the suspicious document file.

[0132] The present invention has the ability to detect malicious macros and / or embedded OLE objects that do not contain known malicious data signatures. In the disclosed method, the machine learning model is a supervised machine learning model trained by a machine learning algorithm through feature engineering. The selected set of features is derived from deriving this selected set of features from a large sampling of document files. Some of the sampled document files contain one or more malicious macros and / or embedded OLE objects, and some of the sampled document files contain at least one non-malicious macro and / or at least one non-malicious embedded OLE object. The document file sampling includes MS Office files. One aspect of the present method Next, network analysts review the classification steps (actions) to improve the accuracy of their threat analysis.

[0133] The disclosed technology is a system for detecting document files containing malicious macros and / or malicious OLE objects. The system includes a heuristic engine that stores unique code patterns and data attributes, also referred to herein as data indicating past (known) malicious code, malicious content, malicious software, and / or malware found in previously analyzed malicious macros and / or OLE objects. In one embodiment, the heuristic engine is used to derive a feature set from malware attributes used to train a machine learning model for detecting document files containing macros and / or OLE objects that contain those malware attributes (features).

[0134] The heuristic engine is configured to derive features that indicate macro and / or embedded OLE object malware, using malware attributes of previously analyzed malicious macros and / or OLE objects, in order to train a machine learning engine to create a data model. The heuristic engine stores indices derived from malicious macros and / or OLE objects based on the code and behavior of previously analyzed malicious macros and / or OLE objects. The technique uses one or more, in some embodiments at least five, features from a list of known attributes for training a machine learning model. The list of one or more of these features is extracted from a list of attributes that indicate macro malware, particularly obfuscated malware. In one aspect of the disclosed technique, the system detects a document file containing a malicious macro and / or embedded OLE object, a heuristic engine that stores data and attributes from previously analyzed malicious macros and / or OLE objects, and a machine learning engine that includes a trained malicious macro and / or OLE object detection model. The model is trained using a supervised machine learning method with labeled data. The training data includes documents, files, and other data labeled as malicious or non-malicious. The machine learning engine includes a supervised machine learning model trained on features derived from the properties of malicious macros and / or malicious OLE objects, as well as non-malicious macros and / or non-malicious OLE objects. The aforementioned malicious OLE objects and / or malicious macros are also collectively referred to herein as malicious code, malicious content, malicious software, and / or malware.

[0135] In another implementation, the disclosed system, including the heuristic engine, operates in parallel with the trained machine learning model. Applying multiple detection methods sequentially or in parallel increases the likelihood of detecting documents containing malicious code. The Office classifier is more likely to classify input documents by threat level and is more likely to detect macros and / or OLE objects in documents containing malicious code.

[0136] In another aspect of the disclosed technology, a system is provided for classifying input documents in a network system to determine whether at least one of the documents may contain macros and / or OLE objects that have malicious code. The system includes a network and a network interface for operable communication to the network. The disclosed technology includes an Office classifier that performs operable communication with a network security system. The Office classifier has input means for receiving and processing document files, in particular MS Office document files. The disclosed system further includes a heuristic feature generation engine. The heuristic feature generation engine determines whether at least one of the documents may contain malicious macros and / or malicious OLE objects It uses a list of malicious macro code attributes and / or malicious OLE object code attributes selected to predict the presence of malicious macros and / or malicious OLE objects. The heuristic engine is configured to derive and derive a list of features used to train a supervised machine learning model to predict the presence of malicious macros and / or malicious OLE objects.

[0137] In a further aspect of the disclosed technology, the office classifier applies a machine learning model to each input document to determine the level of probability that the input document may contain malicious macros and / or malicious OLE object code. Based on this analysis, each input document is classified as safe, malicious, or suspicious. Document files classified as safe are permitted to the network, document files classified as malicious are permanently blocked, and document files classified as suspicious are subjected to threat analysis. Threat analysis of suspicious files may include isolation and transfer to a virtual environment, such as a sandbox, where the malicious code can be safely analyzed.

[0138] In one embodiment, the disclosed technology uses a trained machine learning model to detect obfuscated malicious code and predict documents that have embedded malicious code without known signatures. In another embodiment, the disclosed technology can be combined with unsigned-based analysis of malicious macros and / or malicious OLE objects.

[0139] In another embodiment, a method is disclosed for classifying input documents into a network system to determine whether at least one of the documents contains macros and / or OLE objects with malicious code. In another embodiment, a method is disclosed for classifying input documents into a network system to determine whether at least one of the documents contains malicious macros and / or malicious OLE objects. Macros and / or OLE objects are typically not observed by users and are attractive means of transmission for infecting documents, including MS Office documents. The disclosed technique may be used to classify input documents in a network system, which includes the step (action) of receiving office documents into a network security system of an attached corporate network. The document files are parsed to separate metadata from malicious payload data so that they can be analyzed. Feature engineering is used to define a set of features for detecting malicious macros and / or malicious OLE objects based on features selected from a list of known characteristics and attributes that files that have historically shown malicious content have. The selected features are used to train a supervised machine learning model, which is a model based on labeled data.

[0140] In another embodiment, the office classifier receives incoming documents over the network, breaks them down, and applies machine learning algorithms to classify them as threat levels: safe, suspicious, or malicious. Safe documents are allowed within the network. Suspicious documents are subject to additional processing, including isolation or sandboxing methods. Malicious documents are blocked from the network.

[0141] In a further aspect, the disclosed technology has been found to combine machine learning with other network security methods to further enhance the network security system's ability to detect malicious macros and / or malicious OLE files.

[0142] The foregoing explanation is provided to enable the creation and use of the disclosed technology. Various modifications to the disclosed implementations will become apparent, and the general principles defined herein will apply to other implementations and applications without departing from the spirit and scope of the disclosed technology. Therefore, the disclosed technology is not intended to be limited to the implementations shown, but should be given the broadest scope consistent with the principles and features disclosed herein. The scope of the disclosed technology is defined by the appended claims.

[0143] [Clause] The following clauses are disclosed. [Clause Set 1] 1. A method for classifying input documents in a networked system and determining whether at least one of the documents may contain a macro with malicious code, The action of repeatedly receiving document files into the network security system. An action to analyze a document file and separate macro data from the document payload data. Regarding document files, actions that generate at least obfuscated features indicating past instances of malware embedded in macros using known obfuscation methods, The process involves inputting obfuscation features into a trained machine learning model, applying the trained model to process document files, and predicting the presence of malicious macros. Actions that use a secondary malware detection engine operating in parallel to improve the accuracy of malware detection and eliminate false positives. Actions to classify the resulting documents as safe, suspicious, or malicious, and A method that includes actions to classify documents, such as accepting secure documents into a networked system, blocking malicious documents as malicious, and isolating suspicious documents for threat analysis. 2. Obfuscation features are the following features that describe embedded VBA macro features: Macro_IS_present, macro_autoexec, macro_execute, macro_execute_powershell, Macro_write, macro_has_internet_download, macro_has_registry_access, Macro_comment_lines, macro_code_lines, macro_has_hex_str, macro_olestream_count, macro_ole_passcode, macro_detect_sandbox, macro_detect_virtualization, macro_run_shellcodeinmemory, Macro_disable_security, macro_self_modification, macro_num_stringops, macro_num_textfunc, macro_num_arithfunc, macro_num_typeconvfunc, macro_num_fincfunc, and The method described in Clause 1, which includes at least five of macro_shannon_entropy. 3. The document file is an MS Office document, as described in Clause 1. 4. The document file is one of the following: a Word document, an Excel document, or a PowerPoint document, as described in Clause 3. 5. The obfuscation features are the following VBA macro features: createobject, shell, filesystem, urldownloadtofile, Callbyname, and The method according to Clause 3, wherein the macro-related feature includes at least two of the detect sandboxes. 6. The following document characteristics, Document size, Creator information, Document type (Word / Excel / PPT), Time of creation or modification and revision number, Number of pages, Number of paragraphs, Number of rows, and The method according to clause 3, further comprising inputting at least two features derived from the number of characters into a trained machine learning model. 7. The method according to Clause 1, further comprising the step of using a secondary malware detection engine that operates in parallel to improve the accuracy of malware detection and eliminate false positives. 8. Threat analysis is performed as described in Clause 1, including isolating document files in a sandbox and using the sandbox to test one or more macros in the suspicious document. 9. A malicious macro does not include a known malicious data signature, as described in Clause 1. 10. The machine learning model is a supervised machine learning model trained by machine learning through feature engineering, the selected features are derived from a large sample of document files, some of the sampled document files contain one or more malicious macros, and some of the sampled document files contain at least one non-malicious macro, as described in Clause 1. 11. The sample document file is an MS Office file, as described in Clause 10. 12. Network analysts review the actions to classify in order to improve the accuracy of threat analysis, as described in Clause 1. 13. A system for detecting document files containing malicious macros, A heuristic engine stores data indicating known malicious macros, A system comprising a machine learning engine that includes a supervised machine learning model trained on features derived from the characteristics of malicious and non-malicious macros. 14. The system described in Clause 13, wherein the heuristic engine operates in parallel with a trained machine learning model to predict the presence of malicious macros in documents containing malicious code, and the Office classifier classifies the documents according to the threat level associated with the malicious code. 15. A system for classifying input documents in a networked system and determining whether at least one of the documents may contain a macro with malicious code, Network and A network interface connected to the network, A network security system that communicates with the network in an operational manner, An Office classifier that can communicate with a network security system, An input means for receiving and processing MS Office documents, and a heuristic feature generation engine, A system comprising an office classifier, and an office classifier, comprising 16. The Office Classifier is the system described in Clause 15 that classifies document files as malicious. 17. Files classified as safe, malicious, or suspicious, as described in Clause 16. 18. Document files classified as secure are permitted to the network, documents classified as malicious are permanently blocked, and files classified as suspicious are subjected to threat analysis, as described in Clause 17 of the system. 19. Files classified as suspicious for threat analysis by being isolated and transferred within a sandbox containing malicious code can be securely analyzed in the systems described in Clause 18. 20. The method described in Clause 1, which classifies an office classifier as malicious. [Clause Set 2] 1. A method for classifying input documents in a networked system and determining whether at least one of the documents may contain malicious code in Object Linking & Embedding (OLE), The action of receiving document files into the network security system. An action to analyze document files and separate metadata from malicious payload data in the input document. An action that uses a heuristic engine to provide data indicating past instances of malware embedded in OLE objects using known obfuscation methods. The action of deriving a feature set from data provided by a heuristic engine to train a machine learning algorithm model by using deep learning (DL) methods to predict whether a document file contains a malicious OLE object, Using a trained machine learning model, an action is taken to process document files and determine if the document files contain malicious OLE objects. Actions to classify the resulting documents as safe, suspicious, or malicious, and A method that includes actions to classify documents, such as accepting secure documents into a networked system, blocking malicious documents as malicious, and isolating suspicious documents for threat analysis. 2. The feature set extracted from the OLE file is as follows: embed_none embed_has_corrupt embed_has_password_protected embed_has_suspicious_bin embed_has_semi_suspicious_file embed_has_normal_file embed_has_other_file embed_has_compressed_file absorb_has_non_ext embed_rule1 embed_has_suspicious_bin=true, and The method according to Clause 1, which includes at least one of embed_has_other_file=true. 3. A malicious OLE object does not contain a known malicious data signature as described in Clause 1. 4. The document file is a Word Processing document, as described in Clause 1. 5. The document file is one of the following: a word processing document, a spreadsheet document, or a presentation document, as described in Clause 4. 6. The feature set consists of the following Visual Basic for Applications (VBA) OLE object features. createobject, shell, filesystem, urldownloadtofile, Callbyname, and The method according to Clause 4, wherein the OLE object-related feature includes at least one of the following: detect sandbox. 7. The feature set includes the following features: doc_num_pages doc_num_words doc_num_lines doc_num_chars doc_num_paragraph hys_lastmod_time doc_author_info doc_revision_number doc_lastprint_time doc_link-is dirty doc_language The method described in Clause 4, including one or more of the doc_sizes. 8. The feature set is as described in Clause 1, including the document type. 9. Threat analysis is performed as described in Clause 1, including isolating document files from network systems in a sandbox. 10. The quarantined document files are tested for malicious code as described in Clause 9. 11. The machine learning model is a supervised machine learning model trained by machine learning through feature engineering, the feature set being derived from a large sampling of document files, including several document files having one or more malicious OLE objects and several document files having one or more non-malicious OLE objects, as described in Clause 1. 12. The sampling of document files is as described in Clause 11, wherein the document files are word processing documents, spreadsheet documents, and presentation documents. 13. The method described in Clause 1, further including actions that use a secondary malware detection engine operating in parallel to improve the accuracy of the malicious OLE object detection operation and eliminate false positives. 14. Network analysts review the actions to classify in order to improve the accuracy of threat analysis, as described in Clause 1. 15. A network security system, A network interface connected to the network, Multiple system components, Through the network interface, Object Linking and Actions that connect to document files containing Embedded (OLE) objects, The action involves analyzing the document file and separating the embedded data contained within the OLE object from other payload data contained within the document file, Includes, a plurality of system components configured for actions that process the document file and determine whether the document file contains malicious code that is obfuscated within the OLE object, The process includes extracting obfuscation scoring features from the document file that indicate instances of malicious code obfuscated within the OLE object, The system further includes inputting the obfuscation scoring feature into a trained machine learning model to determine the likelihood that the document file contains malicious code that is obfuscated within the OLE object. 16. The obfuscation scoring features are stored in a database, as described in Clause 15. 17. A system for classifying input documents in a networked system and determining whether at least one of the documents may contain an Object Linking & Embedding (OLE) with malicious code, Network and A network interface connected to the network, A network security system that communicates with the network in an operational manner, It includes an Office classifier that can communicate with a network security system, Office classifiers are Input means for receiving and processing word processing documents, spreadsheet documents, and presentation documents, Heuristic feature generation engine, A system comprising a supervised machine learning model trained by a deep learning method using selected features to detect input documents containing obfuscated malicious OLE objects. 18. The Office Classifier is the system described in Clause 17 that classifies document files as malicious. 19. Files classified as safe, malicious, or suspicious, as described in Clause 18. 20. Document files classified as secure are permitted to the network, documents classified as malicious are permanently blocked, and files classified as suspicious are subjected to threat analysis, as described in Clause 19 of the system. 21. Files classified as suspicious are subjected to threat analysis by isolating and transferring the files to a sandbox where malicious code can be securely analyzed, as described in Clause 20. [Clause Set 3] 1. A system for providing detection of the presence of malicious code embedded in a document file, wherein the system: Multiple system components, Actions that receive a document file containing one or more embedded items, An action to analyze the document file and extract at least one of the embedded items, Processing the document file to determine if at least one of the embedded items is malicious Multiple system components, including, are configured to perform an action to determine whether they likely contain a certain code. The processing involves taking from at least one of the embedded items, This includes extracting obfuscation scoring features that indicate a known instance of malicious code obfuscated within at least one of the embedded items, The process further includes inputting the obfuscation scoring feature into a trained machine learning model to determine the likelihood that at least one of the embedded items in the document file contains malicious code. 2. The system is The system according to Clause 1, further including a network interface connected to the network, wherein the received action occurs via the network interface. 3. The document file is a Microsoft Office document, as defined in Clause 1. 4. The document file is one of the following: a Microsoft Word document, a Microsoft Excel document, or a Microsoft PowerPoint document, as defined in Clause 1. 5. The system described in Clause 1, wherein the document file is one of the following: a word processing document, a spreadsheet document, or a presentation document. 6. The embedded item is part of the system described in Clause 1, which includes at least one of one or more macros and / or one or more OLE objects. 7. The macro is a system as described in Clause 6, including VBA macros. 8. The obfuscation scoring feature includes a macro-related feature, which includes the use of at least one of CreateObject, Shell, FileSystem, URLDownloadToFile, CallByName, or Detect Sandbox, as described in Clause 1. 9. The obfuscation scoring feature includes an object-related feature, which includes the use of at least one of the following Visual Basic For Applications (VBA) Object Linking and Embedding (OLE) features, which includes the use of at least one of Shell, FileSystem, URLDownloadToFile, CallByName, or Detect Sandbox, as described in Clause 1. 10. The system described in Clause 1, further including testing for malicious code within a document file while isolating the document file within a sandbox. 11. A method for providing detection of the presence of malicious code embedded in a document file, Actions that receive a document file containing one or more embedded items, An action to analyze the document file and extract at least one of the embedded items, The action includes processing the document file to determine whether at least one of the embedded items contains malicious code, The processing involves taking from at least one of the embedded items, This includes extracting obfuscation scoring features that indicate a known instance of malicious code obfuscated within at least one of the embedded items, The process further comprises inputting the obfuscation scoring feature into a trained machine learning model to determine the likelihood that at least one of the embedded items in the document file contains malicious code. 12. The method according to Clause 11, wherein the network interface is connected to a network and the reception occurs via the network interface. 13. The document file is a Microsoft Office document, as per Clause 11. Method of description. 14. The method according to Clause 11, wherein the document file is one of the following: a Microsoft Word document, a Microsoft Excel document, or a Microsoft PowerPoint document. 15. The method according to Clause 11, wherein the document file is one of a word processing document, a spreadsheet document, or a presentation document. 16. The method according to Clause 11, wherein the embedded item includes at least one of one or more macros and / or one or more OLE objects. 17. The macro is the method described in Clause 16, including a VBA macro. 18. The method according to Clause 11, wherein the obfuscation scoring feature includes a macro-related feature, the feature includes the use of at least one of CreateObject, Shell, FileSystem, URLDownloadToFile, CallByName, or Detect Sandbox. 19. The method according to Clause 11, wherein the obfuscation scoring feature includes an object-related feature, the object-related feature includes the use of at least one of the following Visual Basic For Applications (VBA) Object Linking and Embedding (OLE) features, the feature includes the use of at least one of Shell, FileSystem, URLDownloadToFile, CallByName, or Detect Sandbox. 20. The method of Clause 11, further comprising testing for malicious code in a document file while isolating the document file in a sandbox.

Claims

1. A system for providing detection of the presence of malicious code embedded in a document file, wherein the system Multiple system components, An action that receives a document file containing one or more embedded items, An action to analyze the document file and extract at least one of the embedded items, Includes a plurality of system components configured to process the document file and perform an action to determine whether at least one of the embedded items contains malicious code, The above processing is performed from at least one of the embedded items, This includes extracting obfuscation scoring features that indicate a known instance of malicious code obfuscated within at least one of the embedded items, The processing further includes inputting the obfuscation scoring features into a trained machine learning model to determine the likelihood that at least one of the embedded items in the document file contains malicious code.

2. The system further includes a network interface connected to a network, The system according to claim 1, wherein the received action occurs via the network interface.

3. The system according to claim 1, wherein the document file is a Microsoft Office document.

4. The system according to claim 1, wherein the document file is one of a Microsoft Word document, a Microsoft Excel document, or a Microsoft PowerPoint document.

5. The system according to claim 1, wherein the document file is one of a word processing document, a spreadsheet document, or a presentation document.

6. The system according to claim 1, wherein the embedded item includes at least one of one or more macros and / or one or more OLE objects.

7. The system according to claim 6, wherein the macro includes a VBA macro.

8. The system according to claim 1, wherein the obfuscation scoring feature includes a macro-related feature, and the macro-related feature includes the use of at least one of CreateObject, Shell, FileSystem, URLDownloadToFile, CallByName, or DetectSandbox.

9. The system according to claim 1, wherein the obfuscation scoring feature includes an object-related feature, the object-related feature includes the use of at least one of the following Visual Basic For Applications (VBA) Object Linking and Embedding (OLE) features, and the Visual Basic For Applications (VBA) Object Linking and Embedding (OLE) feature includes the use of at least one of Shell, FileSystem, URLDownloadToFile, CallByName, or DetectSandbox.

10. The system according to claim 1, further comprising testing for malicious code in a document file while isolating the document file in a sandbox.

11. A method implemented by computer hardware resources to provide detection of the presence of malicious code embedded in a document file, An action that receives a document file containing one or more embedded items, An action to analyze the document file and extract at least one of the embedded items, The action includes processing the document file to determine whether at least one of the embedded items contains malicious code, The above processing is performed from at least one of the embedded items, This includes extracting obfuscation scoring features that indicate a known instance of malicious code obfuscated within at least one of the embedded items, The process further comprises inputting the obfuscation scoring features into a trained machine learning model to determine the likelihood that at least one of the embedded items in the document file contains malicious code.

12. The method according to claim 11, wherein the network interface is connected to a network, and the receiving occurs via the network interface.

13. The method according to claim 11, wherein the document file is a Microsoft Office document.

14. The method according to claim 11, wherein the document file is one of a Microsoft Word document, a Microsoft Excel document, or a Microsoft PowerPoint document.

15. The method according to claim 11, wherein the document file is one of a word processing document, a spreadsheet document, or a presentation document.

16. The method according to claim 11, wherein the embedded item includes at least one of one or more macros and / or one or more OLE objects.

17. The method according to claim 16, wherein the macro includes a VBA macro.

18. The method according to claim 11, wherein the obfuscation scoring feature includes a macro-related feature, and the macro-related feature includes the use of at least one of CreateObject, Shell, FileSystem, URLDownloadToFile, CallByName, or DetectSandbox.

19. The method according to claim 11, wherein the obfuscation scoring feature includes an object-related feature, the object-related feature includes the use of at least one of the following Visual Basic For Applications (VBA) Object Linking and Embedding (OLE) features, the Visual Basic For Applications (VBA) Object Linking and Embedding (OLE) feature includes the use of at least one of Shell, FileSystem, URLDownloadToFile, CallByName, or DetectSandbox.

20. The method according to claim 11, further comprising testing for malicious code in a document file while isolating the document file in a sandbox.

Images