Mail script or program attachment identification and detection method and device and medium
Decompress and decode email attachments through parsing tools and data interference technology, solving the problem of difficult to identify and detect email threats in the prior art, and achieving more efficient and accurate threat detection.
Patent Information
- Application Number
- CN202411605355.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-03-07
AI Technical Summary
It is difficult for the prior art to effectively identify and detect the degree of threat to script or program attachments in emails, especially when using compression and encryption to hide them in the face of malicious attachments.
By obtaining scripts or program attachments of emails, decompressing and decoding the target file using parsing tools and data interference, obtaining a clear parsing file, and identifying and detecting script threats.
It improves the efficiency and accuracy of identification and detection of email threats, can reveal the true appearance of malicious scripts or program attachments, and enhances the detection ability of potential threats.
Smart Images
Figure CN119583113A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication security technology, and in particular to a method, device and medium for identifying and detecting script or program attachments of emails. Background Art
[0002] Program files or script-type attachments are a common means of direct poisoning in virus emails. Compared with spam, these attachments are more challenging for the anti-virus capabilities of email service providers. Compared with traditional security vendors, email service providers have relatively insufficient technical accumulation in this field. In addition, such malicious attachments usually use at least one method to disguise themselves to evade static scanning by anti-virus software, including but not limited to obfuscation, step-by-step attacks, delayed attacks, etc. Traditional anti-virus software mainly uses dynamic defense methods such as sandboxes when facing malicious script attachments that cannot be processed by static scanning.
[0003] However, as a simulation environment, the sandbox cannot perfectly simulate the entire process of script running in the physical system, and malicious scripts will continue to interfere with sandbox detection through sandbox evasion, delayed release, etc. Therefore, the sandbox has the problems of insufficient simulation and anti-sandbox evasion. These problems make it difficult to ensure the effectiveness and reliability of the sandbox in malicious email analysis. Summary of the invention
[0004] The present invention provides a method, device and medium for identifying and detecting script or program attachments of emails, so as to solve the problem that it is difficult to effectively identify and detect the threat level of emails.
[0005] Get the script or program attachment of the email and get the target file;
[0006] Decompressing and decoding the target file by means of a parsing tool and data interference to obtain a parsed file;
[0007] The script threat level is identified and detected on the parsed file to obtain a detection result.
[0008] The present invention obtains a script or program attachment from an email as a target file, avoiding blindly detecting all email contents, thereby improving the pertinence and efficiency of detection. By decompressing and decoding the target file through parsing tools and data interference, a clearer and easier-to-analyze parsed file can be obtained; and, since many malicious scripts or program attachments usually use compression, encryption and other methods to hide their true content and malicious behavior, decompression and decoding help reveal the true face of the script or program attachment, providing a basis for subsequent detection work. Since data interference technology can flexibly adjust the strategies and methods of decompression and decoding according to the characteristics of different scripts or program attachments, thereby improving the success rate and accuracy of decompression and decoding, it can avoid the problem that traditional decompression and decoding methods may not be able to cope with all these different formats and algorithms, resulting in decompression and decoding failures or inaccurate results, which helps to improve the accuracy of threat detection of parsed files.
[0009] Compared with the prior art, the present invention decompresses and decodes the target file by means of parsing tools and data interference, and can obtain a clearer and easier-to-analyze parsed file, thereby ensuring the effectiveness of identifying and detecting the threat level of emails, and thus can solve the problem of difficulty in effectively identifying and detecting the threat level of emails.
[0010] As a preferred solution, the target file is decompressed and decoded by means of a parsing tool and data interference to obtain a parsed file, specifically:
[0011] Determining a substring set containing a password in the target file based on a predefined visual separation character set, and performing data interference processing on the substring set to obtain a candidate password set;
[0012] Extracting hidden information from the byte sequence of the target file through the parsing tool, and extracting predefined risk keywords contained in the macro code of the target file to obtain decoded data;
[0013] Extracting the uniform resource identifier pointed to by the shortcut file in the target file or the command to be executed to obtain completion data;
[0014] The target file is decompressed and decoded according to the candidate password set, the decoded data and the completed data to obtain the parsed file.
[0015] This preferred solution can more effectively identify substrings that may contain passwords in the target file by using a predefined set of visually separated characters. This method reduces the need for manual search and identification and improves the accuracy and efficiency of password extraction. The process of extracting hidden information from the byte sequence of the target file through parsing tools can reveal important data or sensitive information that may be hidden in the file. Extracting predefined risk keywords contained in the macro code of the target file helps to identify potential malicious code or risky behavior, thereby improving the security of the file.
[0016] As a preferred solution, a substring set containing a password in the target file is determined based on a predefined visual separation character set, and data interference processing is performed on the substring set to obtain a set of candidate passwords, specifically:
[0017] Segment the input text of the target file according to the visual separation character set to obtain a substring set;
[0018] Extract corresponding valid character substrings from the substring set according to the predefined valid character set, and filter the valid character substrings to obtain a first candidate password set.
[0019] Matching substrings according to a predefined pattern string set and the visual separation character set to obtain a second candidate password set;
[0020] A third candidate password set is obtained by removing the passwords that meet the interference pattern from the second candidate password set;
[0021] The candidate password set is composed of the first candidate password set, the second candidate password set and the third candidate password set.
[0022] This preferred solution extracts the corresponding valid character substrings from the substring set and filters them, which can further screen out character combinations that are more likely to constitute a password, which helps to eliminate the interference of irrelevant characters and improve the accuracy of password extraction. In addition, through the construction of the first candidate password set, the second candidate password set and the third candidate password set, a set containing multiple possible passwords is formed. This set processing method provides more selection space for subsequent data analysis.
[0023] As a preferred solution, a second candidate password set is obtained by matching substrings according to a predefined pattern string set and the visual separation character set, specifically:
[0024] Establish a pattern matching dictionary tree according to the pattern character string set;
[0025] Scan the input text of the target file through the pattern matching dictionary tree to obtain a starting position set of substrings matched in the pattern character string set;
[0026] The end position of the starting point of each password substring in the starting position set is determined according to the visual separation character set to obtain the second candidate password set.
[0027] In this preferred solution, the pattern matching dictionary tree is an efficient string matching data structure. When scanning the input text, the dictionary tree can skip characters that do not match the current pattern, thereby avoiding unnecessary character comparisons and improving matching efficiency. In addition, the dictionary tree structure can handle input text containing a large number of characters and complex structures. It is not limited by text length or character type, so it is suitable for various target file types.
[0028] As a preferred solution, by removing the passwords that meet the interference pattern from the second candidate password set, a third candidate password set is obtained, which is specifically:
[0029] If the first data set in the second candidate password set meets the interference pattern, the third candidate password set is obtained by removing the first data set; wherein the interference pattern is established based on adding a preset symbol to the first position of the password, inserting characters that do not carry specific meanings into the password, and adding several confusing characters into the password.
[0030] After removing the interference pattern, the third candidate password set of the preferred solution will contain fewer invalid or low-quality passwords, which are more likely to represent passwords actually used by users and therefore have higher quality and credibility.
[0031] As a preferred solution, the script threat level identification and detection are performed on the parsed file to obtain the detection result, which is specifically:
[0032] If the parsed file is of a type that can be recognized by the large language model, performing script threat level detection on the parsed file by using the large language model to obtain the detection result;
[0033] If the parsed file is not of a type that can be recognized by the large language model, a script threat level detection is performed on the parsed file using a sandbox detection technology to obtain the detection result.
[0034] This preferred solution combines two different detection technologies, can handle various types of parsing files, and improves the flexibility and adaptability of the system.
[0035] As a preferred solution, if the parsed file is of a type that can be recognized by a large language model, the large language model is used to perform a script threat level detection on the parsed file to obtain the detection result, which is specifically:
[0036] If the parsed file is of a type that can be recognized by a large language model, extracting code from a script or program attachment of the email to construct instructions;
[0037] The instruction is provided to the large language model to guide the large language model to perform script threat level detection on the parsed file to obtain the detection result.
[0038] This preferred solution can quickly start the threat level detection process and achieve a rapid response to potential threats by extracting code construction instructions from emails or programs that can be recognized by the large language model.
[0039] The present application also provides a device for identifying and detecting script or program attachments of emails, including an acquisition module, a parsing module and a detection module;
[0040] The acquisition module is used to acquire the script or program attachment of the email to obtain the target file;
[0041] The parsing module is used to decompress and decode the target file by means of parsing tools and data interference to obtain a parsed file;
[0042] The detection module is used to identify and detect the script threat level of the parsed file to obtain a detection result.
[0043] As a preferred solution, the analysis module includes an interference unit, a risk unit, a command unit and a analysis unit;
[0044] The interference unit is used to determine a substring set containing a password in the target file based on a predefined visual separation character set, and perform data interference processing on the substring set to obtain a candidate password set;
[0045] The risk unit is used to extract hidden information from the byte sequence of the target file through the parsing tool, and extract predefined risk keywords contained in the macro code in the target file to obtain decoded data;
[0046] The command unit is used to extract the uniform resource identifier pointed to by the shortcut file in the target file or the command to be executed to obtain the completion data;
[0047] The parsing unit is used to decompress and decode the target file according to the candidate password set, the decoding data and the completion data to obtain the parsed file.
[0048] As a preferred solution, the interference unit includes a cutting subunit, a filtering subunit, a matching subunit, a removing subunit and a summarizing subunit;
[0049] The segmentation subunit is used to segment the input text of the target file according to the visual separation character set to obtain a substring set;
[0050] The filtering subunit is used to extract corresponding valid character substrings from the substring set according to the predefined valid character set, and filter the valid character substrings to obtain the first candidate password set.
[0051] The matching subunit is used to match substrings according to a predefined pattern string set and the visual separation character set to obtain a second candidate password set;
[0052] The elimination subunit is used to obtain a third candidate password set by removing the passwords that meet the interference pattern from the second candidate password set;
[0053] The summarizing subunit is configured to form the candidate password set from the first candidate password set, the second candidate password set and the third candidate password set.
[0054] As a preferred solution, the matching subunit is specifically:
[0055] Establish a pattern matching dictionary tree according to the pattern character string set;
[0056] Scan the input text of the target file through the pattern matching dictionary tree to obtain a starting position set of substrings matched in the pattern character string set;
[0057] The end position of the starting point of each password substring in the starting position set is determined according to the visual separation character set to obtain the second candidate password set.
[0058] As a preferred solution, the sub-elimination unit is specifically:
[0059] If the first data set in the second candidate password set meets the interference pattern, the third candidate password set is obtained by removing the first data set; wherein the interference pattern is established based on adding a preset symbol to the first position of the password, inserting characters that do not carry specific meanings into the password, and adding several confusing characters into the password.
[0060] As a preferred solution, the detection module includes a model unit and a sandbox unit;
[0061] The model unit is used to perform script threat level detection on the parsed file by using the large language model to obtain the detection result if the parsed file is of a type that can be recognized by the large language model;
[0062] The sandbox unit is used to perform script threat detection on the parsed file through sandbox detection technology to obtain the detection result if the parsed file is not of a type that can be recognized by the large language model.
[0063] As a preferred solution, the model unit includes an instruction subunit and a detection subunit;
[0064] The instruction subunit is used to extract codes from the script or program attachment of the email to construct instructions if the parsed file is of a type that can be recognized by a large language model;
[0065] The detection subunit is used to guide the large language model to perform script threat level detection on the parsed file by providing the instruction to the large language model to obtain the detection result.
[0066] The present application also provides a storage medium, on which a computer program is stored. The computer program is called and executed by a computer to implement the above-mentioned method for identifying and detecting a script or program attachment of an email. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 It is a flowchart of a method for identifying and detecting a script or program attachment of an email provided in an embodiment of the present application;
[0068] Figure 2 is a flow chart of identification and detection provided by an embodiment of the present application;
[0069] Figure 3 It is a structural diagram of a device for identifying and detecting script or program attachments of emails provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0071] In the description of the present application, it should be understood that the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first", "second" and "third" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "several" means two or more.
[0072] The method for identifying and detecting script or program attachments of emails provided in an embodiment of the present application is mainly used to overcome the defects of incomplete file type judgment methods such as suffix, Content-Type and libmagic, difficulty in obtaining the correct password, and the inability to correctly identify some formats of encrypted office files, so as to effectively identify and detect the threat level of emails.
[0073] Embodiment 1:
[0074] See also Figure 1 The embodiment of the present application provides a method for identifying and detecting script or program attachments of emails, including S1 to S3, and the specific implementation steps are as follows:
[0075] S1. Get the script or program attachment of the email and obtain the target file.
[0076] Step S1 of the embodiment of the present application is specifically as follows:
[0077] Get the script or program attachment of the ".eml file" and get the target file.
[0078] An open source AI model is used to infer the file type of the target file, obtain the file type recognition result, and determine whether it belongs to the predefined type set A; the size of the AI model is about 1MB, and it supports acceleration based on ONNX-RUNTIME (Open Neural Network Exchange Runtime).
[0079] In this embodiment S1, an open source AI model is used to replace the file type identification tool libmagic to infer the file type. Compared with libmagic, the accuracy of file type prediction in this embodiment can be improved by more than 17 percentage points, and the developer does not need to explicitly specify the byte length of the file read in to avoid unnecessary computing expenses, so there is no significant increase in hardware costs. The model takes up less storage space and supports acceleration based on ONNX-RUNTIME. It can achieve performance that is not inferior to libmagic on modern CPUs, thereby achieving successful replacement and reducing the difficulty of classifying predefined file types.
[0080] S2. Decompress and decode the target file by using parsing tools and data interference to obtain a parsed file.
[0081] Step S2 of the embodiment of the present application includes S2.1 to S2.3; specifically, it is:
[0082] S2.1, a predefined valid character set A1, used for continuous substring recognition mode; a predefined visual separation character set B1, used for determining the termination of continuous substring matching; a predefined pattern character string set C1, used for locating substrings that may contain passwords;
[0083] Segment the input text of the target file according to the visual separation character set B1 to obtain a substring set A2;
[0084] According to the valid character set A1, corresponding valid character substrings are extracted from the substring set A2, and the valid character substrings are filtered according to a preset shortest length to obtain a first candidate password set D1.
[0085] Establish a pattern matching dictionary tree based on the pattern string set C1;
[0086] Scan the input text of the target file through the pattern matching dictionary tree to determine the starting point of the substring that may contain the password, thereby obtaining the starting position set of the substring that matches the pattern string set C1;
[0087] According to the visual separation character set B1, the substring is matched from each starting point of the starting position set until it ends, so as to determine the end position of the starting point of each password substring in the starting position set, and obtain the second candidate password set D2.
[0088] If the first data set in the second candidate password set D2 meets the interference pattern, the first data set is removed to obtain the third candidate password set D3 after interference removal; wherein the interference pattern is established by adding specific symbols (such as @, #, $ and %) at the first position of the password, inserting characters that do not carry specific meanings or functions (such as random letters, numbers or symbols) into the password, and adding several confusing characters (such as the number "0" and the letter "o") into the password.
[0089] The candidate password set is composed of the first candidate password set D1, the second candidate password set D2 and the third candidate password set D3.
[0090] The tree-structured pattern matching method in S2.1 of this embodiment can make up for the low success rate of password extraction caused by non-Latin characters and human reading habits, and improve the password coverage rate. By using a predefined set of visually separated characters, substrings that may contain passwords in the target file can be more effectively identified. This method reduces the need for manual search and identification, and improves the accuracy and efficiency of password extraction;
[0091] Moreover, the pattern matching dictionary tree is an efficient string matching data structure. When scanning the input text, the dictionary tree can skip characters that do not match the current pattern, thereby avoiding unnecessary character comparisons and improving matching efficiency. The dictionary tree structure can handle input text containing a large number of characters and complex structures. It is not limited by text length or character types, so it is suitable for various target file types. The third candidate password set after removing the interference pattern will contain fewer invalid or low-quality passwords, which are more likely to represent the passwords actually used by users, and therefore have higher quality and credibility.
[0092] In addition, by extracting the corresponding valid character substrings from the substring set for filtering, character combinations that are more likely to constitute a password can be further screened out, which helps to eliminate the interference of irrelevant characters and improve the accuracy of password extraction. Through the construction of the first candidate password set, the second candidate password set and the third candidate password set, a set containing multiple possible passwords is formed. This set processing method provides more selection space for subsequent data analysis.
[0093] S2.2. According to the preset method, the open source OLE parsing tool is used to extract hidden URL (Uniform Resource Locator) and other information from the HEX (hexadecimal) byte sequence of the target file, and the predefined risk keywords contained in the macro code of the target file are extracted to obtain decoded data; wherein, the open source OLE parsing tool is a parsing tool that has been modified to support parsing WPS to create and save documents; the preset method refers to a method that combines hexadecimal encoding conversion and wide bytecode decoding, valid character recognition and regular expressions.
[0094] In this embodiment S2.2, the traditional parsing tool can only extract the HEX byte sequence, without further decoding and obtaining the URL, while the extraction work of this solution can break through the protection of the office viewing password under the premise of meeting the conditions, and can support a certain degree of scanning of the document even if there is no correct document password in the candidate password set. And it has been found in actual cases that the solution described in this embodiment S2.2 can extract the remote C2 address carried in the encrypted macro virus EXCEL file; if the password is guessed, then the OLE parsing tool can be used directly to remove the password protection and obtain the macro code.
[0095] In this embodiment S2.2, hidden information is extracted from the byte sequence of the target file by using a parsing tool, and this process can reveal important data or sensitive information that may be hidden in the file. Extracting predefined risk keywords contained in the macro code in the target file helps to identify potential malicious code or risky behavior, thereby improving the security of the file.
[0096] S2.3, extract the URI (Uniform Resource Identifier) pointed to by the ".lnk" file (shortcut file) in the target file or the command to be executed to obtain the completion data;
[0097] Based on the file type identification result of the target file, the target file is decompressed and decoded according to the corresponding candidate password set, decoding data and completion data to obtain a parsed file;
[0098] Among them, different processing methods are selected according to the file type. For example, the compressed package will be recursively decompressed and then scanned to obtain the file directory; the office file will try to remove the password and then extract the macro code and information from the byte sequence, and analyze whether it contains suspicious macro code; the ".lnk" file will call the corresponding parsing module to output the target URI or hidden code; the files that can be directly parsed will be directly decoded, etc.
[0099] In this embodiment S2.3, after the above-mentioned alternative password set and the processing method of the decoded data, common office macro codes, ".sh", ".py" and ".bat" and other script contents that can be directly parsed as text can be obtained, but the common ".lnk" files that use the operating system characteristics to perform operations in the Windows environment are still missing. Therefore, the corresponding parsing tool is introduced to extract the URI pointed to by the ".lnk" file or the command to be executed as a completion, which can ensure the comprehensiveness of data parsing;
[0100] Furthermore, it has been found in actual cases that the solution described in Example S2 of this embodiment can support all attachment types in an email sample of 100,000 levels.
[0101] S3. Identify and detect the script threat level of the parsed file to obtain the detection result.
[0102] Step S3 of the embodiment of the present application is specifically as follows:
[0103] If the parsed file is of a type that can be recognized by the large language model, the code is extracted from the script or program attachment of the email to construct the prompt (instruction);
[0104] By providing prompts to the big language model, it can guide the big language model to identify and detect the script threat level of the parsed file, and use DNS resolution, IP geographic location identification and WHOIS data to collect information such as URL registration location and registrar for the big language model to help it generate reports and obtain detection results including threat scores and interpretation reports; Among them, script threat level identification and detection can also be assisted by the integration of the preset URL crawler identification engine;
[0105] If the parsed file is not of a type that can be recognized by the large language model, the traditional "static scanning + sandbox anti-virus solution" is used to identify and detect the script threat level of the parsed file, and obtain the detection result including the threat score and interpretation report; among them, the "static scanning + sandbox anti-virus solution" is an anti-virus strategy that combines static analysis and sandbox technology.
[0106] For details on how to construct a preset prompt to ask questions to a large language model, refer to the question template example in the appendix as follows:
[0107] (1) [introduction];
[0108] (2) [Code functionality overview];
[0109] (3) [Use of encryption, obfuscation and other technologies to hide the user];
[0110] (4) [Questions regarding the functional overview of the code, such as whether there are destructive operations such as deleting files, whether there are injection operations such as adding registry information, obtaining information such as the computer name to detect the environment in which it is located, whether there is an attempt to allocate or obtain permissions, etc.];
[0111] (5) [Questions regarding the overview of the pre-entered network request, such as whether the network address attempted to be accessed looks legitimate, whether it is suspected of being phishing, etc.];
[0112] …
[0113] (n-1) [Output summary overview];
[0114] (n) [Output a score of 0 / 10, with 0.5 as the cutoff].
[0115] To apply the embodiments of this application, please refer to Figure 2 , Figure 2 It is an identification and detection flow chart provided in an embodiment of the present application, which shows the general process of obtaining emails and performing identification and detection in this embodiment.
[0116] In this embodiment S3, by extracting code construction instructions from emails or programs that can be recognized by the large language model, the threat detection process can be quickly started to achieve a rapid response to potential threats. In addition, by combining two different detection technologies, it can handle multiple types of parsed files, thereby improving the flexibility and adaptability of the system.
[0117] Overall, this embodiment has the following beneficial effects:
[0118] This application obtains the script or program attachment from the email as the target file, avoiding blindly detecting all email contents, thereby improving the pertinence and efficiency of the detection. By decompressing and decoding the target file through parsing tools and data interference, a clearer and easier to analyze parsed file can be obtained; and, since many malicious scripts or program attachments usually use compression, encryption and other methods to hide their true content and malicious behavior, decompression and decoding help reveal the true face of the script or program attachment, providing a basis for subsequent detection work. Since data interference technology can flexibly adjust the strategies and methods of decompression and decoding according to the characteristics of different scripts or program attachments, thereby improving the success rate and accuracy of decompression and decoding, it can avoid the problem that traditional decompression and decoding methods may not be able to cope with all these different formats and algorithms, resulting in decompression and decoding failures or inaccurate results, which helps to improve the accuracy of threat detection of parsed files.
[0119] This application uses a large language model to process script or program type attachments such as ".bat", ".sh" and ".html". Compared with the solution using a sandbox, it does not need to actually execute the code, so it can break through the limitations of the simulated operating environment to obtain the execution action sequence of the attachment. At the same time, compared with the sandbox that takes an average of 1-2 minutes, the method relying on the large language model can usually still return the result within 20 seconds while outputting a detailed report. The average processing speed is more than four times that of the sandbox, so it can also improve the detection efficiency.
[0120] Embodiment 2:
[0121] See also Figure 3 , an embodiment of the present application provides a device for identifying and detecting script or program attachments of emails, comprising an acquisition module 10, a parsing module 20 and a detection module 30;
[0122] The acquisition module 10 is used to acquire the script or program attachment of the email to obtain the target file;
[0123] The parsing module 20 is used to decompress and decode the target file by means of parsing tools and data interference to obtain a parsed file;
[0124] The detection module 30 is used to identify and detect the threat level of the scripts in the parsed files and obtain the detection results.
[0125] In one embodiment, the acquisition module 10 is specifically:
[0126] Get the script or program attachment of the ".eml file" and get the target file.
[0127] An open source AI model is used to infer the file type of the target file, obtain the file type recognition result, and determine whether it belongs to the predefined type set A; the size of the AI model is about 1MB, and it supports acceleration based on ONNX-RUNTIME (Open Neural Network Exchange Runtime).
[0128] In the acquisition module 10 of this embodiment, an open source AI model is used to replace the file type identification tool libmagic to infer the file type. Compared with libmagic, the accuracy of file type prediction in this embodiment can be improved by more than 17 percentage points, and the developer does not need to explicitly specify the byte length of the file read in to avoid unnecessary computing expenses, so there is no significant increase in hardware costs. The model takes up less storage space and supports acceleration based on ONNX-RUNTIME, which can achieve performance that is not inferior to libmagic on modern CPUs, thereby achieving successful replacement and reducing the difficulty of classifying predefined file types.
[0129] In one embodiment, the parsing module 20 includes a pre-definition unit, a segmentation sub-unit, a filtering sub-unit, a matching sub-unit, a rejection sub-unit, a summary sub-unit, a risk unit, a command unit, and a parsing unit;
[0130] Among them, the predefined unit is used to predefine a valid character set A1 for continuous substring recognition mode; predefine a visual separation character set B1 for determining the termination of continuous substring matching; and predefine a pattern character string set C1 for locating a substring that may contain a password;
[0131] A segmentation subunit is used to segment the input text of the target file according to the visual separation character set B1 to obtain a substring set A2;
[0132] The filtering subunit is used to extract corresponding valid character substrings from the substring set A2 according to the valid character set A1, and filter the valid character substrings according to a preset minimum length to obtain a first candidate password set D1.
[0133] A matching subunit, used to establish a pattern matching dictionary tree according to the pattern string set C1;
[0134] The matching subunit is also used to scan the input text of the target file through the pattern matching dictionary tree to determine the starting point of the substring that may contain the password, thereby obtaining the starting position set of the substring matched in the pattern string set C1;
[0135] The matching subunit is further used to match substrings from each starting point of the starting position set until termination according to the visual separation character set B1, thereby determining the end position of the starting point of each password substring in the starting position set to obtain a second candidate password set D2.
[0136] The elimination subunit is used for obtaining a third candidate password set D3 after interference removal by removing the first data set if the first data set in the second candidate password set D2 meets the interference pattern; wherein the interference pattern is established by adding specific symbols (such as @, #, $ and %) at the first position of the password, inserting characters that do not carry specific meanings or functions (such as random letters, numbers or symbols) into the password, and adding a number of confusing characters (such as the number "0" and the letter "o") into the password.
[0137] The summarizing subunit is used to form an alternative password set from the first alternative password set D1, the second alternative password set D2 and the third alternative password set D3.
[0138] The tree-structured pattern matching method of the predefined unit, the segmentation subunit, the filtering subunit, the matching subunit, the elimination subunit and the summarization subunit in this embodiment can make up for the low success rate of password extraction caused by non-Latin characters and human reading habits, and improve the password coverage rate. By using a predefined set of visually separated characters, substrings that may contain passwords in the target file can be more effectively identified, which reduces the need for manual search and identification and improves the accuracy and efficiency of password extraction;
[0139] Moreover, the pattern matching dictionary tree is an efficient string matching data structure. When scanning the input text, the dictionary tree can skip characters that do not match the current pattern, thereby avoiding unnecessary character comparisons and improving matching efficiency. The dictionary tree structure can handle input text containing a large number of characters and complex structures. It is not limited by text length or character types, so it is suitable for various target file types. The third candidate password set after removing the interference pattern will contain fewer invalid or low-quality passwords, which are more likely to represent the passwords actually used by users, and therefore have higher quality and credibility.
[0140] In addition, by extracting the corresponding valid character substrings from the substring set for filtering, character combinations that are more likely to constitute a password can be further screened out, which helps to eliminate the interference of irrelevant characters and improve the accuracy of password extraction. Through the construction of the first candidate password set, the second candidate password set and the third candidate password set, a set containing multiple possible passwords is formed. This set processing method provides more selection space for subsequent data analysis.
[0141] The risk unit is used to extract hidden URL (Uniform Resource Locator) and other information from the HEX (hexadecimal) byte sequence of the target file through an open source OLE parsing tool in a preset manner, and extract predefined risk keywords contained in the macro code of the target file to obtain decoded data; wherein, the open source OLE parsing tool is a parsing tool that has been modified to support parsing WPS to create and save documents; the preset method refers to a method that combines hexadecimal encoding conversion and wide bytecode decoding, effective character recognition and regular expressions.
[0142] In the risk unit of this embodiment, the traditional parsing tool can only extract the HEX byte sequence, without further decoding and obtaining the URL. The extraction work of this solution can break through the protection of the office viewing password under the premise of meeting the conditions. Even if there is no correct document password in the candidate password set, it can support a certain degree of scanning of the document. And it has been found in actual cases that the solution described in the risk unit of this embodiment can extract the remote C2 address carried in the encrypted macro virus EXCEL file; if the password is guessed, the OLE parsing tool can be used directly to remove the password protection and obtain the macro code.
[0143] The risk unit of this embodiment extracts hidden information from the byte sequence of the target file through a parsing tool, and this process can reveal important data or sensitive information that may be hidden in the file. Extracting predefined risk keywords contained in the macro code in the target file helps to identify potential malicious code or risky behavior, thereby improving the security of the file.
[0144] The command unit is used to extract the URI (Uniform Resource Identifier) pointed to by the ".lnk" file (shortcut file) in the target file or the command to be executed to obtain the completion data;
[0145] A parsing unit, for decompressing and decoding the target file according to the corresponding candidate password set, decoding data and completion data based on the file type identification result of the target file, to obtain a parsed file;
[0146] Among them, different processing methods are selected according to the file type. For example, the compressed package will be recursively decompressed and then scanned to obtain the file directory; the office file will try to remove the password and then extract the macro code and information from the byte sequence, and analyze whether it contains suspicious macro code; the ".lnk" file will call the corresponding parsing module to output the target URI or hidden code; the files that can be directly parsed will be directly decoded, etc.
[0147] In the command unit and the parsing unit of this embodiment, after the above-mentioned alternative password set and the processing method of the decoded data, common office macro codes, ".sh", ".py" and ".bat" and other script contents that can be directly parsed as text can be obtained, but the common ".lnk" files that use the characteristics of the operating system to perform operations in the Windows environment are still missing. Therefore, the corresponding parsing tool is introduced to extract the URI pointed to by the ".lnk" file or the command to be executed as a completion, which can ensure the comprehensiveness of data parsing;
[0148] Furthermore, it has been found in actual cases that the solution described in the parsing module 20 of this embodiment can support all attachment types in an email sample of 100,000 levels.
[0149] In one embodiment, the detection module 30 includes an instruction subunit, a detection subunit, and a sandbox unit;
[0150] The instruction subunit is used to extract the code from the script or program attachment of the email to construct a prompt (instruction) if the parsed file is of a type that can be recognized by the large language model;
[0151] The detection subunit is used to provide a prompt to the large language model to guide the large language model to identify and detect the script threat level of the parsed file, and use DNS resolution, IP geographic location identification and WHOIS data to collect information such as URL registration location and registrar for the large language model to help it generate a report and obtain a detection result including a threat score and interpretation report; wherein, the script threat level identification and detection can also be assisted by the integration of a preset URL crawler identification engine;
[0152] The sandbox unit is used to identify and detect the script threat level of the parsed file through the traditional "static scanning + sandbox anti-virus solution" if the parsed file is not of a type that can be recognized by the large language model, and obtain the detection result including the threat score and interpretation report; among them, the "static scanning + sandbox anti-virus solution" is an anti-virus strategy that combines static analysis and sandbox technology.
[0153] For details on how to construct a preset prompt to ask questions to a large language model, refer to the question template example in the appendix as follows:
[0154] (1) [introduction];
[0155] (2) [Code functionality overview];
[0156] (3) [Use of encryption, obfuscation and other technologies to hide the user];
[0157] (4) [Questions regarding the functional overview of the code, such as whether there are destructive operations such as deleting files, whether there are injection operations such as adding registry information, obtaining information such as the computer name to detect the environment in which it is located, whether there is an attempt to allocate or obtain permissions, etc.];
[0158] (5) [Questions regarding the overview of the pre-entered network request, such as whether the network address attempted to be accessed looks legitimate, whether it is suspected of being phishing, etc.];
[0159] …
[0160] (n-1) [Output summary overview];
[0161] (n) [Output a score of 0 / 10, with 0.5 as the cutoff].
[0162] To apply the embodiments of this application, please refer to Figure 2 , Figure 2 It is an identification and detection flow chart provided in an embodiment of the present application, which shows the general process of obtaining emails and performing identification and detection in this embodiment.
[0163] The detection module 30 of this embodiment can quickly start the threat detection process by extracting code construction instructions from emails or programs that can be recognized by the large language model, thereby achieving a rapid response to potential threats. In addition, by combining two different detection technologies, it can handle multiple types of parsed files, thereby improving the flexibility and adaptability of the system.
[0164] Overall, this embodiment has the following beneficial effects:
[0165] This application obtains the script or program attachment from the email as the target file, avoiding blindly detecting all email contents, thereby improving the pertinence and efficiency of the detection. By decompressing and decoding the target file through parsing tools and data interference, a clearer and easier to analyze parsed file can be obtained; and, since many malicious scripts or program attachments usually use compression, encryption and other methods to hide their true content and malicious behavior, decompression and decoding help reveal the true face of the script or program attachment, providing a basis for subsequent detection work. Since data interference technology can flexibly adjust the strategies and methods of decompression and decoding according to the characteristics of different scripts or program attachments, thereby improving the success rate and accuracy of decompression and decoding, it can avoid the problem that traditional decompression and decoding methods may not be able to cope with all these different formats and algorithms, resulting in decompression and decoding failures or inaccurate results, which helps to improve the accuracy of threat detection of parsed files.
[0166] This application uses a large language model to process script or program type attachments such as ".bat", ".sh" and ".html". Compared with the solution using a sandbox, it does not need to actually execute the code, so it can break through the limitations of the simulated operating environment to obtain the execution action sequence of the attachment. At the same time, compared with the sandbox that takes an average of 1-2 minutes, the method relying on the large language model can usually still return the result within 20 seconds while outputting a detailed report. The average processing speed is more than four times that of the sandbox, so it can also improve the detection efficiency.
[0167] Embodiment three:
[0168] The embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for identifying and detecting a script or program attachment of an email;
[0169] Wherein, the script or program attachment identification and detection method of an email, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0170] The above are preferred embodiments of the present invention. It should be noted that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for identifying and detecting script or program attachments in emails, characterized in that: include: Get the script or program attachment of the email and get the target file; Decompressing and decoding the target file by means of a parsing tool and data interference to obtain a parsed file; The script threat level is identified and detected on the parsed file to obtain a detection result.
2. A method for identifying and detecting a script or program attachment of an email as claimed in claim 1, characterized in that: The target file is decompressed and decoded by means of parsing tools and data interference to obtain a parsed file, specifically: Determining a substring set containing a password in the target file based on a predefined visual separation character set, and performing data interference processing on the substring set to obtain a candidate password set; Extracting hidden information from the byte sequence of the target file through the parsing tool, and extracting predefined risk keywords contained in the macro code of the target file to obtain decoded data; Extracting the uniform resource identifier pointed to by the shortcut file in the target file or the command to be executed to obtain completion data; The target file is decompressed and decoded according to the candidate password set, the decoded data and the completed data to obtain the parsed file.
3. A method for identifying and detecting a script or program attachment of an email as claimed in claim 2, characterized in that: Based on the predefined visual separation character set, a substring set containing a password in the target file is determined, and data interference processing is performed on the substring set to obtain a set of candidate passwords, specifically: Segment the input text of the target file according to the visual separation character set to obtain a substring set; Extract corresponding valid character substrings from the substring set according to the predefined valid character set, and filter the valid character substrings to obtain a first candidate password set. Matching substrings according to a predefined pattern string set and the visual separation character set to obtain a second candidate password set; A third candidate password set is obtained by removing the passwords that meet the interference pattern from the second candidate password set; The candidate password set is composed of the first candidate password set, the second candidate password set and the third candidate password set.
4. A method for identifying and detecting a script or program attachment of an email as claimed in claim 3, characterized in that: According to the predefined pattern character string set and the visual separation character set matching substring, a second candidate password set is obtained, specifically: Establish a pattern matching dictionary tree according to the pattern character string set; Scan the input text of the target file through the pattern matching dictionary tree to obtain a starting position set of substrings matched in the pattern character string set; The end position of the starting point of each password substring in the starting position set is determined according to the visual separation character set to obtain the second candidate password set.
5. A method for identifying and detecting a script or program attachment of an email as claimed in claim 3, characterized in that: By removing the passwords that meet the interference pattern from the second candidate password set, a third candidate password set is obtained, which is specifically: If the first data set in the second candidate password set meets the interference pattern, the third candidate password set is obtained by removing the first data set; wherein the interference pattern is established based on adding a preset symbol to the first position of the password, inserting characters that do not carry specific meanings into the password, and adding several confusing characters into the password.
6. A method for identifying and detecting a script or program attachment of an email as claimed in claim 1, characterized in that: The script threat level is identified and detected for the parsed file to obtain the detection results, which are specifically: If the parsed file is of a type that can be recognized by the large language model, performing script threat level detection on the parsed file by using the large language model to obtain the detection result; If the parsed file is not of a type that can be recognized by the large language model, a script threat level detection is performed on the parsed file using a sandbox detection technology to obtain the detection result.
7. A method for identifying and detecting a script or program attachment of an email as claimed in claim 6, characterized in that: If the parsed file is of a type that can be recognized by the large language model, the large language model is used to perform a script threat level detection on the parsed file to obtain the detection result, which is specifically: If the parsed file is of a type that can be recognized by a large language model, extracting code from a script or program attachment of the email to construct instructions; The instruction is provided to the large language model to guide the large language model to perform script threat level detection on the parsed file to obtain the detection result.
8. A device for identifying and detecting script or program attachments in emails, characterized in that: Including acquisition module, parsing module and detection module; The acquisition module is used to acquire the script or program attachment of the email to obtain the target file; The parsing module is used to decompress and decode the target file by means of parsing tools and data interference to obtain a parsed file; The detection module is used to identify and detect the script threat level of the parsed file to obtain a detection result.
9. The device for identifying and detecting a script or program attachment of an email as claimed in claim 8, characterized in that: The analysis module includes an interference unit, a risk unit, a command unit and a analysis unit; The interference unit is used to determine a substring set containing a password in the target file based on a predefined visual separation character set, and perform data interference processing on the substring set to obtain a candidate password set; The risk unit is used to extract hidden information from the byte sequence of the target file through the parsing tool, and extract predefined risk keywords contained in the macro code in the target file to obtain decoded data; The command unit is used to extract the uniform resource identifier pointed to by the shortcut file in the target file or the command to be executed to obtain the completion data; The parsing unit is used to decompress and decode the target file according to the candidate password set, the decoding data and the completion data to obtain the parsed file.
10. The device for identifying and detecting a script or program attachment of an email as claimed in claim 9, characterized in that: The interference unit includes a cutting subunit, a filtering subunit, a matching subunit, a removing subunit and a summarizing subunit; The segmentation subunit is used to segment the input text of the target file according to the visual separation character set to obtain a substring set; The filtering subunit is used to extract corresponding valid character substrings from the substring set according to the predefined valid character set, and filter the valid character substrings to obtain the first candidate password set. The matching subunit is used to match substrings according to a predefined pattern string set and the visual separation character set to obtain a second candidate password set; The elimination subunit is used to obtain a third candidate password set by removing the passwords that meet the interference pattern from the second candidate password set; The summarizing subunit is configured to form the candidate password set from the first candidate password set, the second candidate password set and the third candidate password set.
11. The device for identifying and detecting a script or program attachment of an email as claimed in claim 10, characterized in that: The matching subunit is specifically: Establish a pattern matching dictionary tree according to the pattern character string set; Scan the input text of the target file through the pattern matching dictionary tree to obtain a starting position set of substrings matched in the pattern character string set; The end position of the starting point of each password substring in the starting position set is determined according to the visual separation character set to obtain the second candidate password set.
12. The device for identifying and detecting a script or program attachment of an email as claimed in claim 10, characterized in that: The sub-unit for elimination is specifically: If the first data set in the second candidate password set meets the interference pattern, the third candidate password set is obtained by removing the first data set; wherein the interference pattern is established based on adding a preset symbol to the first position of the password, inserting characters that do not carry specific meanings into the password, and adding several confusing characters into the password.
13. The device for identifying and detecting a script or program attachment of an email as claimed in claim 8, characterized in that: The detection module includes a model unit and a sandbox unit; The model unit is used to perform script threat level detection on the parsed file by using the large language model to obtain the detection result if the parsed file is of a type that can be recognized by the large language model; The sandbox unit is used to perform script threat detection on the parsed file through sandbox detection technology to obtain the detection result if the parsed file is not of a type that can be recognized by the large language model.
14. The device for identifying and detecting a script or program attachment of an email as claimed in claim 13, characterized in that: The model unit includes an instruction subunit and a detection subunit; The instruction subunit is used to extract codes from the script or program attachment of the email to construct instructions if the parsed file is of a type that can be recognized by a large language model; The detection subunit is used to guide the large language model to perform script threat level detection on the parsed file by providing the instruction to the large language model to obtain the detection result.
15. A storage medium, characterized in that: The storage medium stores a computer program, which is called and executed by a computer to implement a method for identifying and detecting a script or program attachment of an email as described in any one of claims 1 to 7.