A method, system, and device for extracting and detecting hidden files.

Abstract

This application discloses a method and system for extracting and detecting hidden files. The method includes: firstly, monitoring and intercepting outgoing emails, detecting their file size, and classifying and marking them; then, extracting email text features, setting probability thresholds in a key data detection model based on size, pre-analyzing email titles to identify subject types, checking word order to identify encrypted emails, and extracting feature data after decryption; inputting the feature data into the model to calculate a first probability value, and comparing the body and attachments of low-probability emails to parse new content; evaluating a second probability value, determining emails containing keywords, and sending an alert to the terminal; generating structured alerts by combining a hierarchical detection mechanism and dynamic threshold setting, using logistic regression models, etc., with dual probability determination; this method improves detection accuracy, reduces the risk of false positives and missed detections, supports automated response, and provides an efficient and reliable solution for data leakage prevention.

Description

Technical Field

[0001] This invention relates to the field of security management technology, and in particular to a method, system and device for extracting and detecting hidden files. Background Technology

[0002] Steganography, a technique for concealing secret information within other media, is widely used in the field of information security. In the context of detecting and extracting hidden files in outgoing emails, current technologies primarily rely on data hiding and detection techniques. These methods analyze the statistical characteristics and pattern changes of multimedia data (such as images, audio, and video) or text files in emails to detect and extract potentially hidden information. With technological advancements, steganography has been combined with encryption techniques and deep learning, improving the concealment and robustness of information hiding, while also presenting new challenges for the extraction and detection of hidden files.

[0003] Chinese invention patent application number 202010086751.5 discloses a method for detecting sensitive information: intercepting outgoing emails and extracting first text data; obtaining preset monitoring fields and identifying the first monitoring field value corresponding to the preset monitoring fields from the first text data; combining these to generate a first combined feature, and inputting the first combined feature into a sensitive data detection model to obtain a first sensitivity probability; when the first sensitivity probability is less than or equal to a preset value, extracting the attachments of the outgoing email; performing anti-hiding parsing on the files in the attachments and determining whether the parsed file data has changed; when the parsed file data has changed, determining that the outgoing email has data leakage; extracting the changed data from the parsed file data and generating a first warning message; and sending the extracted data and the first warning message to a management terminal. This method can improve the accuracy of email detection.

[0004] Traditional email monitoring methods are often simplistic and struggle to comprehensively and accurately identify critical data within emails. The probability of emails containing critical information varies depending on their size, and both email text and attachments can conceal key data. Relying on a single standard or simple detection method is insufficient for effective handling. Therefore, a more complex and sophisticated email monitoring and critical data detection mechanism is needed. This mechanism should comprehensively consider factors such as email size, text content, and attachment content to accurately identify and issue warnings about emails containing critical data, thereby effectively protecting the information security of enterprises or organizations. Summary of the Invention

[0005] This application provides a method and system for extracting and detecting hidden files, which can accurately identify and issue warnings for emails containing key data, thereby improving the accuracy of detecting key information in emails.

[0006] This application provides a method for extracting and detecting hidden files, the method comprising:

[0007] S1, the server monitors email transmission between the internal network and the external network and intercepts all emails sent from the internal network to the external network;

[0008] S2 detects the file size of intercepted emails and categorizes and labels them according to their file size;

[0009] S3, extract key data features from the email text, set hierarchical parameters in the key data detection model, and set probability thresholds based on file size; S31, perform pre-analysis of the email title to identify the subject and type of the email text; S32, based on the pre-analysis results, perform grammatical logic checks on the email text to identify encrypted emails; S33, use a public key to decrypt the encrypted email. If decryption is successful, obtain the plaintext content of the email, extract key data features, and execute S4. If decryption fails, execute step S34; S34, extract the email's metadata and analyze and calculate the correlation factor between the email text context semantics and user behavior; S35, set a threshold for the correlation factor. When the correlation factor exceeds the threshold, the email is marked as a suspicious encrypted email, and step S5 is executed.

[0010] S4. Input the extracted data features into the key data detection model to calculate the first probability value of the email containing key information; compare the email title and body content with the attachment content with the first probability value below the threshold.

[0011] S5, parse out the new content in the attachment;

[0012] S6. Input the newly added content in the attachment into the key data detection model to evaluate the second probability value and determine whether the email contains key data; S61. Detect keyword region blocks according to the keyword fragmentation pattern library; S62. Map the detected region blocks to nodes, and automatically complete the logical edges between nodes based on domain-related semantic rules to form a complete semantic chain; S63. Use a graph neural network to calculate the transition probability between nodes; S64. Set a transition probability threshold. When the path probability exceeds the threshold, it is determined to be a valid association, and these validly associated region blocks are identified as the set of keywords; S65. Calculate the second probability value based on the keyword set. By analyzing the frequency of keyword occurrence in a large number of normal and suspicious attachments, a suspicious probability and weight are calculated for each keyword based on its semantic association and occurrence.

[0013] S7: For emails identified as containing critical data, extract the specific critical data content, generate an alert, and send it to the management terminal.

[0014] Preferably, the probability threshold of the key data detection model includes: cleaning the email title and body to remove irrelevant characters; splitting the text using the jieba word segmentation tool, extracting and labeling data feature vectors based on key data; collecting labeled email data, using regression analysis to statistically analyze the relationship between file size and the probability of key information occurrence; setting probability thresholds for each email category based on the statistical results, configuring the hierarchical parameters into the logistic regression model, and setting corresponding thresholds according to categories.

[0015] Preferably, step S4, calculating the first probability value, includes: processing the data feature vector extracted in step S3. Input into the logistic regression model; calculate the first probability value that the email contains key information; the formula for calculating the first probability value is:

[0016]

[0017] The intercept is... Features The weighting coefficients, For feature vectors The value of the i-th dimension.

[0018] Preferably, the step of checking the word order to identify encrypted emails includes: extracting the body text based on the email text subject and type; constructing a word order logic rule base, matching the body text with the rules one by one to determine whether there are word order logic errors; if there are errors, marking the email text as an encrypted email and calculating the correlation factor.

[0019] Preferably, the semantic behavior of the email text includes: constructing a domain keyword fragmentation pattern library; using a statistical word segmentation algorithm to segment historical email text into character sequences; setting window parameter ranges based on domain patterns; scanning and recording relationship paths after probabilistic parameter selection; improving the initial set of potential keyword relationship paths and recording their positions; identifying path jump points and recording relevant information; during contextual semantic analysis, extracting character sequences after detecting the first keyword, checking for similar patterns, and if found, comparing jump points with paths, and calculating the matching degree using edit distance.

[0020] Preferably, the matching degree includes: during text scanning, detecting the first keyword of a keyword, and extracting a character sequence starting with that keyword from the potential relationship path of the keyword. ,in Let S represent the i-th character in the sequence. If there is a jump point, its position index is i. The extracted character sequence S is then compared with the domain keyword fragmentation pattern library. Perform a match to check for similar fragmentation patterns; if similar patterns exist, further compare the jump points and relationship paths in S with... Whether they match; calculate the matching degree using the edit distance algorithm; edit distance This indicates that the character sequence S is converted into a pattern. Minimum number of editing operations required; let the length of S be n, If the length is m, then the similarity score is sim. for: Matching degree Equals similarity score At this point, the matching degree has been normalized to the [0,1] interval.

[0021] Preferably, the calculation of the second probability value includes: the formula for calculating the second probability value is:

[0022]

[0023] This can be obtained by analyzing the frequency of the keyword in a large number of normal and suspicious attachments, and expressing a suspicious probability for each keyword calculated based on its semantic association and occurrence. ; Represented as a set of keywords, each keyword The corresponding weight is .

[0024] This application also provides a hidden file extraction and detection system, the system comprising:

[0025] The acquisition module is used to monitor email transmission between the intranet and the extranet, intercept all emails sent from the intranet to the extranet, and perform file size detection on the intercepted emails, classifying and labeling the emails according to preset file size classification standards.

[0026] The detection module is used to extract key data features from email text, set hierarchical parameters in the key data detection model, and set probability thresholds based on file size; the extracted feature text data is input into the key data detection model to calculate the first probability value that the email contains key information;

[0027] The parsing module is used to compare the email title and body content with the attachment content for emails with a first probability value lower than the threshold, and to parse out any new content in the attachments.

[0028] The probability determination module is used to input the parsed new content of the attachment into the key data detection model to evaluate the second probability value, and combine the first probability value and the second probability value to determine whether the email contains key data.

[0029] The early warning module is used to extract the specific key data content from emails that are determined to contain key data, generate early warning information, and send it to the management terminal.

[0030] An apparatus comprising: a processor, a memory, a network interface, and a database connected via a system bus, used in the aforementioned method for extracting and detecting hidden files.

[0031] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0032] By employing a tiered detection mechanism and dynamic threshold settings, the system achieves accurate identification and early warning of critical information in emails. First, it categorizes emails by size and extracts key features. Then, it uses a logistic regression model to calculate key probability values. Next, it identifies hidden information through comparative analysis of the email body and attachments. Finally, it generates structured alerts based on a combination of dual probability assessments. This method significantly improves detection accuracy, effectively reduces the risk of false positives and false negatives, and supports automated response, providing an efficient and reliable solution for data leakage prevention.

[0033] By employing title pre-analysis, word order logic checks, and metadata correlation analysis, the system achieves accurate identification and processing of encrypted emails. First, it uses text classification and rule matching to detect anomalies in the title, subject, and word order, then uses public-key decryption to obtain the plaintext content. For emails that cannot be decrypted, BERT semantic modeling and user behavior analysis are used to calculate contextual correlation factors, effectively identifying suspicious encryption behavior. This solution significantly improves the coverage and accuracy of key information detection while ensuring the privacy of encrypted communication, addressing the pain point of traditional methods being ineffective against encrypted content, and adapting to the security needs of different business scenarios through dynamic thresholds.

[0034] By constructing a keyword fragmentation pattern library and a probabilistic sliding window scanning mechanism, the system achieves accurate identification of fragmented key information in hidden files. The system first establishes a fragmentation feature library containing various patterns such as character insertion and homophonic substitution. Through dynamic window parameter settings, it performs multi-granular scanning of file content, records potential relationship paths, and analyzes jump point features. Then, it combines this with an edit distance algorithm to calculate the matching degree, effectively solving the problem that traditional methods struggle to detect hidden information that has been deliberately segmented or deformed.

[0035] By leveraging a keyword fragmentation pattern library to accurately detect region blocks and constructing semantic chains based on domain semantic rules, the potential relationships between keywords are effectively mined. Graph neural networks are used to calculate node transition probabilities, combined with thresholds to determine valid associations, enabling accurate identification of keyword sets. Furthermore, a second probability value is calculated based on the frequency of keyword occurrence in normal and suspicious attachments, quantifying the suspiciousness of each keyword. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the method for extracting and detecting hidden files according to an embodiment of the present invention;

[0037] Figure 2 This is a structural block diagram of the hidden file extraction and detection system according to an embodiment of the present invention. Detailed Implementation

[0038] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0039] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0041] Example 1: Figure 1 This is a flowchart illustrating the method for extracting and detecting hidden files according to an embodiment of the present invention.

[0042] like Figure 1 As shown, a method for extracting and detecting hidden files includes the following steps:

[0043] S1, the server monitors email transmission between the internal network and the external network, and intercepts all emails sent from the internal network to the external network.

[0044] S2 checks the file size of intercepted emails and categorizes and labels them according to their file size.

[0045] Specifically, files are categorized and labeled based on size, including:

[0046] The server extracts the file size information of the intercepted emails.

[0047] Emails are categorized and labeled according to preset file size classification standards (such as small emails, medium emails, and large emails).

[0048] The file size information refers to the total size of the email (including attachments). The classification criteria can be set based on the size of the email files you send daily. For reference, set the following: Small file emails: emails no larger than 10MB, usually consisting of text content; Medium file emails: emails between 10MB and 50MB, which may contain a few attachments; Large file emails: emails larger than 50MB, usually with large files attached (such as compressed files or high-resolution images).

[0049] S3 extracts key data features from email text, sets hierarchical parameters in the key data detection model, and sets probability thresholds based on file size.

[0050] The email text refers to the email title and body; the key data are pre-defined text information fields that need to be monitored and detected, such as "estimated time", "bid amount", and "invoice information".

[0051] Specifically, key data features are extracted from email text, including:

[0052] Clean the email's title and body, removing irrelevant characters (such as HTML tags, special symbols, etc.), and convert the text into a format suitable for analysis.

[0053] Use the Chinese word segmentation tool jieba to split the text into individual words or phrases.

[0054] Extract the corresponding data feature vectors from the email text based on key data.

[0055] Wherein, the data feature vector is represented as , This is expressed as keyword weight (e.g., "contract" = 0.8, "amount" = 0.9). Named entity recognition results (e.g., “Person: Zhang San”=1, “Organization: XX Company”=1).

[0056] Mark the data features as key data features for subsequent operations.

[0057] Specifically, setting hierarchical parameters in the key data detection model includes:

[0058] Collect a large amount of tagged email data, including file size, content, and whether it contains key information.

[0059] Regression analysis was used to analyze the relationship between file size and the probability of key information appearing.

[0060] Based on the statistical results, one or more probability thresholds are set for each email category.

[0061] Configure the grading parameters into the key data detection model (logistic regression model), and set the corresponding probability thresholds for the model based on the email category.

[0062] It should be noted that in practical applications, when emails are categorized and input into the key data detection model, the model will apply the corresponding probability threshold to make a judgment based on the email category.

[0063] S4. Input the extracted feature text data into the key data detection model to calculate the first probability value that the email contains key information.

[0064] Specifically, calculating the first probability value of an email containing key information includes:

[0065] Extracting the data feature vector from S3 Input into the logistic regression model.

[0066] Calculate the first probability value that the email contains key information.

[0067] The formula for calculating the first probability value is as follows:

[0068]

[0069] The intercept is... Features The weight coefficients (obtained through training). For feature vectors The value of the i-th dimension.

[0070] For example, suppose the coefficients after model training are =−2.5, =1.8, =2.0, input features =[0.8,0.9,1,1] (corresponding to "Contract", "Amount", "Name", and "Organization"), then:

[0071]

[0072] If the email is classified as a medium file (threshold 0.5), it is marked as a key because 0.56 > 0.5.

[0073] S5 compares the email title and body content with the attachment content for emails with a first probability value lower than the threshold, and parses out the new content in the attachment.

[0074] Specifically, the new content in the attachments was parsed out, including:

[0075] Only emails with a first probability value P1 lower than the classification threshold (e.g., P1 < 0.7 for small files) will be processed.

[0076] Use the Apache Tika tool to convert attachments (PDF / DOCX / Excel) into plain text (remove noises such as headers, footers, and table borders in the attachments, and retain the core content).

[0077] Segment the email body and attachment text respectively to obtain word sets and .

[0078] Remove stop words (such as "of", "is") and words that have already appeared in the body to obtain the newly added word set of the attachment:

[0079]

[0080] Retain named entities (person names, account numbers) and key data (such as "invoice", "secret key").

[0081] For example, the email body: "See the attachment for the financial summary." The attachment content: "Secret key in 2024: X5Y9Z2, internal account number: 622848...". Parsing result: Δ={("X5Y9Z2", secret key),("622848...", bank account number)}, determined that there is newly added content, triggering the secondary evaluation of S7.

[0082] S6, input the newly added content in the attachment into the key data detection model to evaluate the second probability value, and determine whether the email contains key data.

[0083] S7, for emails determined to contain key data, extract the specific key data content, generate a warning message and send it to the management terminal.

[0084] The technical solutions in the embodiments of the present application described above have at least the following technical effects or advantages:

[0085] Through the hierarchical detection mechanism and dynamic threshold setting, the accurate identification and warning of key information in emails are achieved. The system first classifies and extracts key features according to the file size, calculates the key probability value using the logistic regression model, then discovers hidden information through the comparative analysis of the body and attachments, and finally generates a structured warning by combining double probability determination. This method significantly improves the detection accuracy, effectively reduces the risks of false alarms and missed detections, and at the same time supports automated response, providing an efficient and reliable solution for data leakage prevention.

[0086] Example 2: In the email critical information detection scheme of Example 1, when the email content is encrypted, the system faces the dilemma of being unable to extract key data features. Traditional methods can only handle plaintext emails and are helpless against encrypted content, resulting in obvious vulnerabilities in the detection system. Especially when emails use asymmetric or hybrid encryption methods, it is impossible to decrypt and obtain the content through conventional means, and there is a lack of effective analytical dimensions for encryption behavior characteristics. This limitation makes the system completely dependent on manual review when dealing with encrypted emails, which is not only inefficient but also prone to missed detections.

[0087] Therefore, the embodiments of this application are optimized based on the above embodiments.

[0088] In some embodiments, step S3, extracting key data features from the email text, further includes:

[0089] S31 performs pre-analysis of email titles to identify the subject and type of the email text.

[0090] Specifically, this involves pre-analyzing email headers, including:

[0091] Collect a large number of email headers from mail servers, email clients, or relevant databases. These headers should cover a wide range of possible topics and types.

[0092] The collected headers are cleaned to remove noisy data. The cleaned email headers are then stored in a database or file to build an email header corpus (which can be classified and stored according to certain rules, such as by time, sender, etc.).

[0093] The email subject line is segmented into individual words. For example, for the subject line "Project Progress Report and Next Week's Plan", the segmentation results are "project", "progress", "report", "and", "next week", and "plan".

[0094] Each word after segmentation is labeled with its part of speech, such as noun, verb, adjective, etc. For example, "project" is labeled as a noun, and "report" is labeled as a verb.

[0095] Using a text classification algorithm, email titles are categorized into different themes and types based on features such as keywords and phrases in the title, such as business emails, financial emails, and contract emails.

[0096] S32, based on the pre-analysis results of the email, performs a word order and logic check on the email text to identify encrypted emails.

[0097] Specifically, word order logic checks include:

[0098] Extract the email body text based on the email's subject and type.

[0099] Construct a rule base for word order logic, defining rules for sentence component integrity, word collocation rationality, and logical coherence of sentences.

[0100] The sentence component integrity rule checks whether a sentence has basic components such as subject, predicate, and object. For example, in a simple sentence "I eat," "I" is the subject, "eat" is the predicate, and "eat" is the object. If a sentence lacks these basic components, there may be a logical error in word order. The word collocation rationality rule identifies common word collocation patterns, such as verb-noun collocation and adjective-noun collocation. For example, "play basketball" is a reasonable collocation, while "play football" does not conform to common collocation habits. The sentence logical relationship coherence rule analyzes the logical relationships between sentences, such as causal relationships, adversative relationships, and parallel relationships. For example, in "Because it rained, we canceled our outdoor activities," there is a causal relationship. If the logical relationship is confused, there may be a problem.

[0101] The email body text is matched one by one with the rules in the word order logic rule base.

[0102] Based on rule matching, determine whether there are logical errors in the word order of the email text.

[0103] If the email text contains logical errors in word order, the email text containing logical errors in word order will be marked as an encrypted email.

[0104] S33, use the public key to decrypt the encrypted email. If decryption is successful, obtain the plaintext content of the email, extract key data features, and execute S4. If decryption fails, execute step S34.

[0105] S34, extract the email's metadata and analyze and calculate the correlation factors between the email text context semantics and user behavior.

[0106] Metadata refers to unencrypted data such as sender / recipient, timestamp, and attachment type.

[0107] Specifically, correlation factors include:

[0108] Define the email metadata that needs to be extracted, including sender, recipient, timestamp, attachment type, etc.

[0109] The metadata is parsed and extracted from the email data, and then the BERT pre-trained language model is used to capture the contextual semantic information in the text.

[0110] The email text is preprocessed, including word segmentation, stop word removal, and stemming, to convert the text into a format suitable for model input.

[0111] The preprocessed email text is input into the BERT model to obtain the vector representation of the text.

[0112] Collect users' historical email data to build user behavior models, and then convert these models into vector representations. For example, parameters from time-based behavior models, frequency-based behavior models, and sender behavior models can be combined into a single vector to represent the user behavior model.

[0113] The email data includes information such as the time, frequency, and sender of emails received. Analyzing the time distribution of email reception by users can be done using a probability distribution model (Gaussian distribution) to fit the time pattern of email reception. The frequency of email reception by users is statistically analyzed, calculating the average reception frequency and the range of frequency fluctuations. The range of senders to which users receive emails is analyzed, and a sender whitelist or blacklist is constructed, calculating the probability of different senders sending emails.

[0114] Calculate the similarity between the semantic vector of the email text context and the vector of the user behavior model.

[0115] Wherein, assuming the semantic vector of the email text context is A and the user behavior model vector is B, the similarity calculation formula is:

[0116]

[0117] and These are the i-th elements of vectors A and B, respectively, where n is the dimension of the vector.

[0118] S35, set the threshold for the association factor. When the association factor exceeds the threshold, the email is marked as a suspicious encrypted email, and step S5 is executed.

[0119] The threshold settings for the correlation factors are based on historical data and experience.

[0120] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0121] By employing title pre-analysis, word order logic checks, and metadata correlation analysis, the system achieves accurate identification and processing of encrypted emails. First, it uses text classification and rule matching to detect anomalies in the title, subject, and word order, then uses public-key decryption to obtain the plaintext content. For emails that cannot be decrypted, BERT semantic modeling and user behavior analysis are used to calculate contextual correlation factors, effectively identifying suspicious encryption behavior. This solution significantly improves the coverage and accuracy of key information detection while ensuring the privacy of encrypted communication, addressing the pain point of traditional methods being ineffective against encrypted content, and adapting to the security needs of different business scenarios through dynamic thresholds.

[0122] Example 3: In Example 2, when the email content uses conventional encryption means, the system can achieve risk determination through metadata analysis and behavioral feature detection. However, when the email content undergoes keyword fragmentation processing (such as interval insertion, homophone replacement, etc.), traditional semantic analysis and keyword matching methods are difficult to effectively identify such deliberately hidden key information. Due to the diversity and concealment of fragmentation processing means, using a unified detection model will lead to an increase in both the false alarm rate and the missed detection rate. To improve the ability to identify deliberately hidden information, a multi-level fragmentation feature analysis system needs to be established.

[0123] Therefore, the embodiments of this application are optimized based on the above embodiments.

[0124] In some embodiments, in step S34, the email text context semantics further includes:

[0125] S341, construct a domain keyword fragmentation pattern library, segment the historical email text based on a statistical word segmentation algorithm, and segment the text into character sequences.

[0126] Specifically, the keyword fragmentation pattern library includes:

[0127] Perform a detailed analysis on a large number of email samples (including normal emails and suspicious emails), and manually identify common keyword fragmentation processing patterns, such as interval insertion, homophone replacement, symbol separation, etc.

[0128] According to different fragmentation types, such as character insertion, character replacement, symbol usage, etc., establish sub-libraries for the collected patterns respectively. For example, the character insertion sub-library contains all instances of the interval insertion pattern.

[0129] Write a detailed description for each pattern, explain the characteristics and recognition methods of the pattern, and give specific examples for subsequent query and use. For example, for the "interval insertion" pattern, the description is "Insert the characters of the keyword at intervals into other characters", and the example is "Pay1ment3suc9cess".

[0130] Summarize and organize all sub-libraries into a keyword fragmentation pattern library.

[0131] S342, set the window parameter range according to the domain pattern, probabilistically select the parameters and then perform a sliding window scan, and record it as a relationship path.

[0132] Among them, the domain pattern refers to the application domain where the keyword exists, such as the financial domain, the legal domain, etc. Keywords in the financial domain may be monetary policy, market, etc., and keywords in the legal domain may be suspected, legal provisions, etc. The combination of these keywords forms a character sequence of common patterns.

[0133] Specifically, after probabilistically selecting the parameters, a sliding window scan is performed, including:

[0134] The window size is determined based on common patterns in the domain keyword fragmentation pattern library. For example, in the interleaved insertion pattern, the window size is set to 3-7 characters; in the symbol-separated pattern, the window size is set to 2-5 characters. The step size is determined to be between 1-3 characters.

[0135] Assign probabilities for different window sizes. For example, if the window size W takes the values ​​3, 4, 5, 6, and 7, the corresponding probabilities are P(W=3)=0.25, P(W=4)=0.4, P(W=5)=0.25, P(W=6)=0.05, and P(W=7)=0.05.

[0136] Assign probabilities to different step lengths. Let the step length S take values ​​of 1, 2, and 3, with corresponding probabilities P(S=1)=0.25, P(S=2)=0.5, and P(S=3)=0.25.

[0137] Starting from the beginning of the email text, the window is slid according to a probabilistically selected window size W and a movement step S. After each movement, the character sequence within the window is recorded; these records initially constitute the potential relationship path of the keywords.

[0138] S344, Complete the initial set of potential relationship paths for keywords and record location information.

[0139] Specifically, the initial set of potential relationship paths includes:

[0140] When scanning email text in a sliding window, the character sequence in each window is recorded completely. These sequences form the initial set of potential keyword relationship paths.

[0141] After initially obtaining character sequence records by scanning the email text using a sliding window, the initial set of potential keyword relationship paths is further refined. Specifically, the character sequence within each window is completely recorded, and for each recorded character sequence, its start and end positions in the email text are determined and recorded. For example, for the character sequence "abc", if it starts from the 5th character and ends at the 7th character in the email text, then the start position is 5 and the end position is 7.

[0142] The recorded character sequences are analyzed to explore potential relationships between characters, such as counting the number of times characters repeat, calculating the intervals between characters, and looking for patterns. Based on the analysis results, the character sequences are filtered and organized to remove those sequences that obviously lack keyword characteristics, such as sequences that are too short (e.g., only one character) or sequences with irregular character repetition.

[0143] S345 identifies jump points in the path, records their position, type, and length, and associates them with sequences.

[0144] Among them, the jump points are the positions of keywords where special situations such as discontinuity and intervals occur. For example, in "zhi1fu3cheng9gong", the positions of the numbers 1, 3, and 9 are the jump points; the associated sequence associates the recorded jump point information with the corresponding character sequence for subsequent comparative analysis.

[0145] S346. When performing context semantic analysis on the email text, after detecting the first keyword, extract the character sequence starting with this keyword and check whether there is a similar fragmented pattern.

[0146] S347. If there is a similar pattern, then compare the jump points with the path, and use the edit distance to calculate the similarity and normalize it to obtain the matching degree.

[0147] Specifically, the matching degree includes:

[0148] During the text scanning process, once the first keyword of the keyword is detected, extract the character sequence starting with this keyword from the potential relationship path of the keyword , where represents the i-th character in the sequence. If there is a jump point, its position index is i. For example, if the keyword is "pay", when "zhi" is scanned and meets the characteristics of the first keyword, extract the subsequent relevant character sequence.

[0149] Match the extracted character sequence S with the fragmented pattern library of domain keywords to check whether there is a similar fragmented pattern. If there is a similar pattern, then further compare the jump points and relationship paths in S with to see if they are consistent.

[0150] Use the edit distance algorithm to calculate the matching degree. The edit distance represents the minimum number of edit operations (insertion, deletion, replacement) required to convert the character sequence S into the pattern . Let the length of S be n, and the length of be m, then the similarity score sim

[0151]

[0152] The matching degree is equal to the similarity score , and at this time the matching degree has been normalized to the interval [0, 1].

[0153] Set a matching degree threshold T (for example, 0.8). If the matching degree between the character sequence S and a certain pattern in the pattern library If it exceeds the threshold T, then determine that the character sequence S is a true keyword; if the matching degree does not exceed the threshold, continue to scan the text and perform matching operations until a qualified keyword is found or the scanning ends.

[0154] For example, in an outgoing email, the text content is "Please transfer 1 account to the specified account as soon as possible, otherwise your WeChat account will be frozen." First, build a fragmented pattern library of domain keywords. By analyzing a large number of fraudulent and normal emails, common keyword fragmentation processing patterns such as "interval insertion", "homophone replacement", and "symbol separation" are manually identified. For example, "transfer 1 account" is an interval insertion pattern, "WeChat" is a homophone replacement pattern, and "Alipay" is a symbol separation pattern. Based on this, sub-libraries for character insertion, character replacement, and symbol usage are established, and each pattern is described and an example is given. Then, use a statistical-based word segmentation algorithm to segment the email text, obtaining character sequences such as "Please", "as soon as", "transfer 1 account", "to", "specified", "account", "otherwise", "will", "freeze", "your", "WeChat", "account", etc. Then, set the window parameter range according to the domain pattern. For the "interval insertion" pattern, the window size is set to 3 - 7 characters, and the step size is between 1 - 3 characters. The probabilities of the window size and step size are also set, such as P(W = 3)=0.25, etc. Then, perform a sliding window scan according to the probabilistically selected parameters, starting from the beginning of the text. For example, if W = 4 and S = 2 are randomly selected for the first time, record the character sequences within the window such as "Please transfer as soon as" to initially form a potential relationship path for keywords. Then, complete the initial set of the potential relationship path for keywords and record the position information. For example, the starting position of "transfer 1 account" is 4 and the ending position is 6, and the starting position of "WeChat" is 16 and the ending position is 18. At the same time, analyze and filter to remove sequences with too short length or irregular repetition. Then, identify the jump points in the path. For example, the position index of the number 1 in "transfer 1 account" is 5, and the position index of the number 1 in "WeChat" is 17. Record its position, type, and length and associate the sequences.

[0155] When performing context semantic analysis on the email text, it is detected that "transfer" and "WeChat" may be the first keywords. Extract the character sequences "transfer 1 account" and "WeChat" starting with them, and match them with the fragmented pattern library of domain keywords. It is found that "transfer 1 account" conforms to the "interval insertion" pattern, and "WeChat" conforms to the "homophone replacement" pattern. Then, extract the character sequences S1 = "transfer 1 account" and S2 = "WeChat", and match them with the corresponding patterns in the pattern library respectively. Calculate the similarity through the edit distance algorithm. Assume that the length of S1 is 3, and the length of the corresponding standard pattern "transfer account" is 2. The edit distance d(S1, m1)=1, and the similarity score sim(S1, m1)≈0.67; the length of S2 is 3, and the length of the corresponding standard pattern "WeChat" is 2. The edit distance d(S2, m2)=1, and the similarity score sim(S2, m2)≈0.67.

[0156] Set the matching threshold T=0.8. Since sim(S1,m1)≈0.67<0.8 and sim(S2,m2)≈0.67<0.8, the matching degree does not exceed the threshold. Continue scanning the text and performing matching operations (in this example, there are no other suspicious sequences in the email text segment, but in reality, scanning will continue). If the matching degree exceeds the threshold after subsequent scanning or parameter adjustment, then the character sequence is determined to be the real keyword.

[0157] It should be noted that a feedback mechanism needs to be established in the system. When the system encounters a new fragmentation pattern in actual application, it should be added to the domain keyword fragmentation pattern library in a timely manner.

[0158] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0159] By constructing a keyword fragmentation pattern library and a probabilistic sliding window scanning mechanism, the system achieves accurate identification of fragmented key information in hidden files. The system first establishes a fragmentation feature library containing various patterns such as character insertion and homophonic substitution. Through dynamic window parameter settings, it performs multi-granular scanning of file content, records potential relationship paths, and analyzes jump point features. Then, it combines this with an edit distance algorithm to calculate the matching degree, effectively solving the problem that traditional methods struggle to detect hidden information that has been deliberately segmented or deformed.

[0160] Example 4: The method described in Example 3 may struggle to comprehensively and accurately capture the latent semantic relationships between keywords, resulting in low keyword recognition accuracy and an inability to effectively distinguish between normal and suspicious keywords. Since the frequency of keyword occurrence and semantic association patterns vary across different texts, using a uniform recognition rule will inevitably lead to recognition errors and weak generalization ability. Given the significant differences in keyword fragmentation patterns and semantic associations across different texts throughout history, the required indicators for accurate keyword recognition will also differ. To more accurately identify keywords, especially distinguishing between normal and suspicious keywords, it is necessary to comprehensively consider keyword fragmentation patterns, semantic associations, and their occurrence in different types of text.

[0161] Therefore, the embodiments of this application are optimized based on the above embodiments.

[0162] In some embodiments, step S6, evaluating the second probability value, includes:

[0163] S61, detect keyword region blocks based on the keyword fragmentation pattern library.

[0164] The keyword region block involves scanning the input text character by character or word by word, matching fragments of the text against patterns in a fragmented pattern library. When a fragment matching a pattern is found, it is marked as a potential keyword region block.

[0165] S62 maps the detected region blocks to nodes, and automatically completes the logical edges between nodes based on domain-related semantic rules to form a complete semantic chain.

[0166] Specifically, each detected keyword region is mapped to a node in a graph. Each node represents a potential keyword or a part of a keyword. A set of semantic rules is defined based on domain-specific knowledge and semantic relationships. For example, in the medical field, if two keywords are both related to a disease, and one keyword is a symptom or treatment for the other, then they may have a semantic association. When determining semantic association, multiple factors can be considered, such as the similarity of word meanings and the closeness of association within domain knowledge. All nodes are traversed, and logical associations between nodes are determined according to the semantic rules. If an association exists, a logical edge is added between the corresponding nodes, and the edge weight is initialized based on the strength of the association (e.g., a higher initial weight for a strong association and a lower initial weight for a weak association).

[0167] S63 uses a graph neural network to calculate the transition probabilities between nodes.

[0168] Specifically, a Graph Neural Network (GNN) is constructed using the graph structure built in step S62. A feature vector is initialized for each node, which can be encoded based on the node's text content, part-of-speech tagging, and other information. For example, word embedding techniques (such as Word2Vec and GloVe) can be used to convert the node's text content into a vector representation. When initializing the feature vector, other node attributes, such as the node's position and frequency in the text, can be further incorporated to more comprehensively represent the node's features. Message passing and aggregation operations are performed within the GNN. Each node updates its feature vector through message interactions with its neighboring nodes. After multiple layers of message passing and aggregation, the final feature vector for each node is obtained. Then, the transition probability between nodes is obtained by calculating the similarity between their feature vectors. Similarity calculation can employ methods such as cosine similarity, which can be selected based on the specific circumstances.

[0169] S64, set a transition probability threshold. When the path probability exceeds the threshold, it is determined to be a valid association. These validly associated region blocks are identified as the set of keywords.

[0170] Specifically, for any two nodes in the graph, the path probability is calculated. The path probability is obtained by multiplying the transition probabilities of all edges along the path. When calculating the path probability, if the path contains multiple nodes, the transition probabilities between adjacent nodes must be multiplied sequentially according to the path order. The keyword set is represented as follows. Each keyword The corresponding weight is .

[0171] Specifically, a transition probability threshold θ is set. For all paths with probabilities exceeding θ, a valid association is considered to exist between their corresponding nodes. The regions corresponding to all nodes with valid associations are determined as the identified keyword set. When determining the keyword set, if multiple paths involve the same region, the importance of that region can be comprehensively evaluated based on the path probabilities to more reasonably determine the keyword set.

[0172] S65, calculate the second probability value based on the keyword set.

[0173] The formula for calculating the second probability value is as follows:

[0174]

[0175] This can be obtained by analyzing the frequency of the keyword in a large number of normal and suspicious attachments, and expressing a suspicious probability for each keyword calculated based on its semantic association and occurrence. .

[0176] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:

[0177] By leveraging a keyword fragmentation pattern library to accurately detect region blocks and constructing semantic chains based on domain semantic rules, the potential relationships between keywords are effectively mined. Graph neural networks are used to calculate node transition probabilities, combined with thresholds to determine valid associations, enabling accurate identification of keyword sets. Furthermore, a second probability value is calculated based on the frequency of keyword occurrence in normal and suspicious attachments, quantifying the suspiciousness of each keyword.

[0178] Furthermore, embodiments of the present invention also provide a hidden file extraction and detection system.

[0179] Figure 2 This is a schematic diagram of the structure of the hidden file extraction and detection system according to an embodiment of the present invention.

[0180] like Figure 2 As shown, a hidden file extraction and detection system includes: an acquisition module, a detection module, a parsing module, a probability determination module, and an early warning module.

[0181] The acquisition module is used to monitor email transmission between the intranet and the extranet, intercept all emails sent from the intranet to the extranet, and perform file size detection on the intercepted emails, classifying and labeling the emails according to preset file size classification standards.

[0182] The detection module is used to extract key data features from email text, set hierarchical parameters in the key data detection model, and set probability thresholds based on file size; the extracted feature text data is input into the key data detection model to calculate the first probability value that the email contains key information.

[0183] The parsing module is used to compare the email title and body content with the attachment content for emails with a first probability value lower than the threshold, and to parse out any new content in the attachments.

[0184] The probability determination module is used to input the parsed new content of the attachment into the key data detection model to evaluate the second probability value, and combine the first probability value and the second probability value to determine whether the email contains key data.

[0185] The early warning module is used to extract the specific key data content from emails that are determined to contain key data, generate early warning information, and send it to the management terminal.

[0186] It should be noted that other specific implementations of the hidden file extraction and detection system of this invention can refer to the above-described method for extracting and detecting hidden files.

[0187] Furthermore, embodiments of the present invention also provide a device.

[0188] The device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores email data. The network interface is used for communication with external terminals via a network connection.

[0189] It should be noted that other specific implementations of the hidden file extraction and detection device in this embodiment of the invention can refer to the above-described method for extracting and detecting hidden files.

[0190] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting and detecting hidden files, characterized in that, The method includes: S1, the server monitors email transmission between the internal network and the external network and intercepts all emails sent from the internal network to the external network; S2 detects the file size of intercepted emails and categorizes and labels them according to their file size; S3, extract key data features from the email text, set hierarchical parameters in the key data detection model, and set probability thresholds based on file size; S31, perform pre-analysis of the email title to identify the subject and type of the email text; S32, based on the pre-analysis results, perform grammatical logic checks on the email text to identify encrypted emails; S33, use a public key to decrypt the encrypted email. If decryption is successful, obtain the plaintext content of the email, extract key data features, and execute S4. If decryption fails, execute step S34; S34, extract the email's metadata and analyze and calculate the correlation factor between the email text context semantics and user behavior; S35, set a threshold for the correlation factor. When the correlation factor exceeds the threshold, the email is marked as a suspicious encrypted email, and step S5 is executed. S4. Input the extracted data features into the key data detection model to calculate the first probability value of the email containing key information; compare the email title and body content with the attachment content with the first probability value below the threshold. S5, parse out the new content in the attachment; S6. Input the newly added content in the attachment into the key data detection model to evaluate the second probability value and determine whether the email contains key data; S61. Detect keyword region blocks according to the keyword fragmentation pattern library; S62. Map the detected region blocks to nodes, and automatically complete the logical edges between nodes based on domain-related semantic rules to form a complete semantic chain; S63. Use a graph neural network to calculate the transition probability between nodes; S64. Set a transition probability threshold. When the path probability exceeds the threshold, it is determined to be a valid association, and these validly associated region blocks are identified as the set of keywords; S65. Calculate the second probability value based on the keyword set. By analyzing the frequency of keyword occurrence in a large number of normal and suspicious attachments, a suspicious probability and weight are calculated for each keyword based on its semantic association and occurrence. S7: For emails identified as containing critical data, extract the specific critical data content, generate an alert, and send it to the management terminal.

2. The method for extracting and detecting hidden files as described in claim 1, characterized in that, The step of setting probability thresholds based on file size includes: cleaning the email title and body to remove irrelevant characters; splitting the text using the jieba word segmentation tool, extracting and labeling data feature vectors based on key data; collecting labeled email data, using regression analysis to statistically analyze the relationship between file size and the probability of key information appearing; setting probability thresholds for each email category based on the statistical results, configuring the hierarchical parameters into the logistic regression model, and setting corresponding thresholds according to the category.

3. The method for extracting and detecting hidden files as described in claim 1, characterized in that, The calculation of the first probability value that the email contains key information includes: the data feature vector extracted by S3. Input into the logistic regression model; calculate the first probability value that the email contains key information; the formula for calculating the first probability value is: , The intercept is... Features The weighting coefficients, For feature vectors The value of the i-th dimension.

4. The method for extracting and detecting hidden files as described in claim 1, characterized in that, The method for checking the word order of encrypted emails includes: extracting the body text based on the subject and type of the email text; constructing a word order logic rule base, matching the body text with the rules one by one to determine whether there are word order logic errors; if there are errors, marking the email text as an encrypted email and calculating the correlation factor.

5. The method for extracting and detecting hidden files as described in claim 1, characterized in that, The semantic behavior of the email text includes: constructing a domain keyword fragmentation pattern library; using a statistical word segmentation algorithm to cut historical email text into character sequences; setting window parameter ranges based on domain patterns; scanning and recording relationship paths after probabilistic parameter selection; improving the initial set of potential keyword relationship paths and recording their positions; identifying path jump points and recording relevant information; during contextual semantic analysis, extracting character sequences after detecting the first keyword, checking for similar patterns, and comparing jump points with paths if found, and calculating the matching degree using edit distance.

6. The method for extracting and detecting hidden files as described in claim 5, characterized in that, The matching degree includes: during text scanning, detecting the first keyword of a keyword, and extracting the character sequence starting with that keyword from the keyword's potential relationship path. ,in Let S represent the i-th character in the sequence. If there is a jump point, its position index is i. The extracted character sequence S is then compared with the domain keyword fragmentation pattern library. Perform a match to check for similar fragmentation patterns; if similar patterns exist, further compare the jump points and relationship paths in S with... Whether they match; calculate the matching degree using the edit distance algorithm; edit distance This indicates that the character sequence S is converted into a pattern. Minimum number of editing operations required; let the length of S be n, If the length is m, then the similarity score is sim. for: Matching degree Equals similarity score At this point, the matching degree has been normalized to the [0,1] interval.

7. The method for extracting and detecting hidden files as described in claim 1, characterized in that, The calculation of the second probability value includes: the formula for calculating the second probability value is: , This can be obtained by analyzing the frequency of the keyword in a large number of normal and suspicious attachments, and expressing a suspicious probability for each keyword calculated based on its semantic association and occurrence. ; Represented as a set of keywords, each keyword The corresponding weight is .

8. A hidden file extraction and detection system, applied to a hidden file extraction and detection method as described in any one of claims 1 to 7, characterized in that, The system includes: The acquisition module is used to monitor email transmission between the intranet and the extranet, intercept all emails sent from the intranet to the extranet, and perform file size detection on the intercepted emails, classifying and labeling the emails according to preset file size classification standards. The detection module is used to extract key data features from email text, set hierarchical parameters in the key data detection model, and set probability thresholds based on file size; the extracted feature text data is input into the key data detection model to calculate the first probability value that the email contains key information; The parsing module is used to compare the email title and body content with the attachment content for emails with a first probability value lower than the threshold, and to parse out any new content in the attachments. The probability determination module is used to input the parsed new content of the attachment into the key data detection model to evaluate the second probability value, and combine the first probability value and the second probability value to determine whether the email contains key data. The early warning module is used to extract the specific key data content from emails that are determined to contain key data, generate early warning information, and send it to the management terminal.

9. A device for extracting and detecting hidden files, comprising: A system bus connects a processor, memory, network interface, and database, characterized in that a device is applied to a method for extracting and detecting hidden files as described in any one of claims 1 to 7.

Images