Method and equipment for analyzing mail with missing content

Abstract

The invention discloses a content missing mail analysis method and device, and relates to the technical field of emails, the method comprises the following steps: carrying out text content extraction and classification on a mail text to obtain the category of the text content; the category comprises a plain text format, a webpage format and an attachment; taking the text content of which the category is the plain text format and the webpage format as a to-be-decoded text; decoding the to-be-decoded text according to a corresponding coding mode to obtain a decoded character string; performing segmented character set detection on the decoded character string according to a set byte length, and adding a character set corresponding to a character set detection result meeting a set condition into a character set list; and performing character conversion on the decoded character string by traversing a character set in the character set list until a decoding result passes semantic coherence judgment, thereby obtaining an original mail text. According to the method and the device, the effective analysis of the mail with the missing content is realized.

Description

Technical Field

[0001] The present application relates to the technical field of email, and in particular to a method and device for parsing emails with missing content. Background Art

[0002] With the advent of the information age, electronic mail (EML) has become one of the important means of communication for modern people. However, it has also brought a series of email security issues. More and more internal personnel use email to illegally upload sensitive information, causing huge losses and data leakage risks. Email security incidents occur frequently. Research on email security is of great significance for improving the level of network security protection, preventing data leakage and data forensics.

[0003] In addition to the specific content of the transmitted information, emails also contain information such as sender, recipient, subject, attachment, time and various metadata. These texts are encoded and displayed through related email applications.

[0004] However, humans cannot directly understand the encoded text and need to use relevant tools to parse it. Existing email parsing methods and tools are mainly based on the structure and key fields in the email format. They are limited to complete email content and cannot effectively parse emails with missing content. They can often only parse text content with a complete format structure or the parsed text is still garbled, which requires manual screening and cannot quickly parse emails in batches. Summary of the invention

[0005] The purpose of this application is to provide a method and device for parsing emails with missing content, thereby achieving effective parsing of emails with missing content.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for parsing emails with missing content, comprising:

[0008] Extracting and classifying the text content of the email text to obtain the category of the text content; the category includes plain text format, web page format and attachment; taking the text content in the plain text format and the web page format as the text to be decoded;

[0009] Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded character string;

[0010] Performing segmented character set detection on the decoded character string according to the set byte length, and adding the character set corresponding to the character set detection result that meets the set conditions to the character set list;

[0011] The character set in the character set list is traversed to perform character conversion on the decoded character string until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

[0012] Optionally, extracting and classifying the email text to obtain the category of the text content specifically includes:

[0013] According to the encoding features, the regular matching method is used to extract the text content of the email text. If there is Base64-encoded text in the extracted text content, the Base64-encoded text is vectorized to obtain the text to be classified;

[0014] Inputting the file to be classified into a classification model to obtain a classification prediction result of the text content; the classification model is a trained neural network;

[0015] Based on the inherent sequence of the categories in the email, the classification prediction result is corrected to obtain the category of the text content.

[0016] Optionally, based on the inherent order of the categories in the email, the classification prediction result is corrected to obtain the category of the text content, specifically including:

[0017] Input the classification prediction results into the first sequence;

[0018] Initialize a second sequence, and initialize the first element in the second sequence to be the first element in the first sequence;

[0019] Traverse the elements except the first element in the first sequence, and for each element in the first sequence that has been traversed: if the current traversed element is the same as the previous element, add the current traversed element to the second sequence; if the current traversed element is smaller than the previous element, add the previous element to the second sequence; if the current traversed element is larger than the previous element, determine whether the current traversed element is equal to the next element, if so, add the current traversed element to the second sequence, if not, add the previous element to the second sequence; after the traversal of the first sequence is completed, obtain a corrected second sequence;

[0020] Generating a ladder list according to the corrected second sequence, wherein the element values ​​in each ladder column of the ladder list are the same;

[0021] Calculate the average probability for each step;

[0022] Traverse the other steps except the first step in the step list, and for each step traversed: if the element value of the previous step is greater than the element value of the next step, and the average probability of the previous step is greater than or equal to the average probability of the next step, then correct the values ​​of all elements of the next step to the element value of the previous step; if the average probability of the previous step is less than the average probability of the next step, then correct the values ​​of all elements of the previous step to the element value of the next step, and after the step list traversal is completed, a corrected step list is obtained;

[0023] According to the corrected ladder list, the category of the text content is obtained.

[0024] Optionally, the neural network includes an embedding layer, a first convolution module, a second convolution module, a long short-term memory network and a fully connected layer connected in sequence; the first convolution module and the second convolution module both include a one-dimensional convolution layer, a ReLU function, a maximum pooling layer and a Dropout layer connected in sequence.

[0025] Optionally, performing segmented character set detection on the decoded character string according to a set byte length, and adding character sets corresponding to character set detection results that meet set conditions to a character set list, specifically includes:

[0026] Starting from the first byte of the decoded string, byte texts of each set byte length are intercepted in turn for character set detection. For each byte text, if the confidence in the detection result is greater than or equal to 0.9, the detection result is added to the character set list, otherwise the byte text is shifted back by 1 byte and the text of the set byte length is intercepted to continue character set detection until the character set detection of the last byte text intercepted is completed.

[0027] Optionally, character conversion is performed on the decoded string by traversing character sets in the character set list until the decoding result passes the semantic coherence judgment to obtain the original text of the email, specifically including:

[0028] Counting the character sets in the character set list, sorting the character sets in descending order of the counted number, and obtaining a character set sorting list;

[0029] Each character set in the character set sorting list is traversed, and the decoded string is converted into characters using the currently traversed character set to obtain a current decoding result. The current decoding result is judged for semantic coherence. If the current decoding result passes the semantic coherence judgment, the current decoding result is used as the original text of the email and the traversal of the character set sorting list is stopped. If it does not pass the semantic coherence judgment, the next character set is traversed.

[0030] Optionally, the semantic coherence of the current decoding result is judged, specifically including:

[0031] Extract the features of the current decoding result;

[0032] The extracted feature parameters are input into a semantic coherence judgment model, and a semantic judgment result is output; the semantic judgment result is semantic coherence or semantic incoherence; the semantic coherence judgment model is obtained by training a machine learning model using a training set.

[0033] Optionally, after converting the decoded string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment and the original email text is obtained, the email parsing method for missing content further includes:

[0034] The language type of the original email text is detected, and if the detected language type is not the target language, the original email text is translated into the target language.

[0035] In a second aspect, the present application provides a method for parsing emails with missing content, including:

[0036] Extract text content from email text based on key fields;

[0037] Decode the extracted text content;

[0038] If the decoding is successful, the original email text is obtained and the program stops;

[0039] If the decoding is unsuccessful, extract and classify the text content of the email text to obtain the category of the text content; the category includes plain text format, web page format and attachment; the text content in the plain text format and the web page format is used as the text to be decoded;

[0040] Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded character string;

[0041] Performing segmented character set detection on the decoded character string according to the set byte length, and adding the character set corresponding to the character set detection result that meets the set conditions to the character set list;

[0042] The character set in the character set list is traversed to perform character conversion on the decoded character string until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

[0043] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-mentioned methods for parsing emails with missing content.

[0044] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0045] The present application provides an email parsing method and device for missing content, which extracts and classifies text content from email text to obtain categories of text content; the categories include plain text format, web page format and attachment; the categories of plain text format and web page format and text content are used as text to be decoded; thereby, the text content of the attachment type is eliminated, reducing the impact on the original email text and improving the decoding efficiency; the text to be decoded is decoded according to the corresponding encoding method to obtain a decoded character string; the decoded character string is segmented and detected according to the set byte length, and the character set corresponding to the character set detection result that meets the set conditions is added to the character set list; the character sets in the character set list are traversed to perform character conversion on the decoded character string until the decoding result passes the semantic coherence judgment to obtain the original email text, thereby solving the problem that the existing technical means cannot parse emails with missing content, and realizing effective parsing of emails with missing content. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 This is an application environment diagram of an email parsing method for missing content in an embodiment of the present application;

[0048] Figure 2 A flowchart of a method for parsing emails with missing content provided in one embodiment of the present application;

[0049] Figure 3 An architectural diagram of an email parsing method for missing content provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a text preprocessing process provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a decoding process provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a character set detection process provided in an embodiment of the present application;

[0053] Figure 7 A flowchart of a method for parsing an email with missing content provided by another embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0057] EML is a semi-structured text, belonging to the Multipurpose Internet Mail Extensions (MIME) type of text. When the content in the text is missing, the existing extraction methods and tools cannot extract the content based on the original text position features or field features. The main manifestations are: First, extraction errors, redundant information is extracted, causing interference for subsequent decoding. Second, it is impossible to distinguish between Base64-encoded plain text format (text / plain), web page format (text / html) and attachments (attachments) encoded text, which is not easy to decode. Third, no content can be extracted.

[0058] The email parsing method for missing content provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the mail to be processed to the server 104. After the server 104 receives the mail to be processed, for the mail to be processed, the server 104 extracts and classifies the text content of the mail text to obtain the category of the text content; the category includes plain text format, web page format and attachment; the text content of the category of the plain text format and the web page format is used as the text to be decoded; the text to be decoded is decoded according to the corresponding encoding method to obtain a decoded string; the decoded string is segmented and character set detection is performed according to the set byte length, and the character set corresponding to the character set detection result that meets the set conditions is added to the character set list; the decoded string is converted by traversing the character set in the character set list until the decoding result passes the semantic coherence judgment to obtain the original text of the mail. The server 104 can feedback the original text of the mail text obtained for the mail to the terminal 102. In addition, in some embodiments, the email parsing method for missing content can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform email parsing on the email to be processed, or the server 104 can obtain the email to be processed from the data storage system and perform email parsing on the email to be processed.

[0059] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The portable wearable devices may be smart watches and smart bracelets, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0060] In an exemplary embodiment, Figure 2 and Figure 3 As shown, a method for parsing mails with missing content is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 204.

[0061] in:

[0062] Step 201: extract and classify the text content of the email text to obtain the categories of the text content; the categories include plain text format, web page format and attachment; and use the text content in the plain text format and the web page format as the text to be decoded.

[0063] Step 202: Decode the text to be decoded according to the corresponding encoding method to obtain a decoded character string.

[0064] Step 203: Perform segmented character set detection on the decoded character string according to the set byte length, and add the character set corresponding to the character set detection result that meets the set conditions to the character set list.

[0065] Step 204: Perform character conversion on the decoded string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment, thereby obtaining the original email text.

[0066] By implementing the above-mentioned steps 201 to 204, the present application provides a method for parsing emails with missing content, which is specifically a method for parsing emails with missing content. The text content of the attachment type is removed, which reduces the impact on the original email and improves the decoding efficiency; the character sets in the character set list are traversed to perform character conversion on the decoded string until the decoding result passes the semantic coherence judgment to obtain the original email text, and the decoded string is converted into characters through a targeted character set, thereby achieving effective parsing of emails with missing content.

[0067] The present application extracts and classifies text content based on regular matching and neural networks, wherein step 201 specifically includes:

[0068] Step 211: according to the encoding features, a regular matching method is used to extract the text content of the email text. If there is Base64-encoded text in the extracted text content, the Base64-encoded text is vectorized to obtain the text to be classified.

[0069] If there is no Base64-encoded text in the extracted text content, the file to be classified is the text content extracted from the email text using the regular matching method; if there is Base64-encoded text in the extracted text content, the file to be classified is the text obtained by replacing the Base64-encoded text in the extracted text content with the vectorized Base64-encoded text.

[0070] Through regular matching, "noise" content such as header information, boundary characters, feature fields, etc. is filtered out, and Base64-encoded and Quoted-printable-encoded text contents are extracted respectively, which is used as preprocessing for subsequent text type classification and decoding.

[0071] The Base64 encoded regular expression is:

[0072] r"^([A-Za-z0-9+ / ]{4})(([A-Za-z0-9+\n / ]{1})*)([A-Za-z0-9+ / ]{4}|[A-Za-z0-9+ / ]{3}=|[A-Za-z0-9+ / ]{2}==)$".

[0073] The regular expression for Quoted-printable encoding is: r"=(?:[A-Fa-f0-9]{2})+".

[0074] Regarding n-gram, n-gram draws on the idea of ​​natural language processing, using n-gram to analyze the text sequence in the text to detect the topic, and applies it to the research of file fragment type recognition. Files are compared to texts, and the bytes that make up the files are compared to text sequences.

[0075] For a single byte, n=1 is represented, so this byte frequency distribution is also called 1-gram or unigram. The number of occurrences of each byte in a file or file fragment is counted. A byte is composed of 8 bits of binary, and its value range is 2^8, that is, 0 to 255. In this way, the frequency of occurrence of each byte can be represented for each file fragment, which constitutes a 256-dimensional (0 to 255) feature vector.

[0076] If two consecutive bytes are counted each time, it is called a 2-gram or bigram, three consecutive bytes are called a 3-gram or trigram, and so on. 16 = 65536 different variations, the number of features is already very large, which will lead to sparse data.

[0077] Secondly, Base64 has ˋA-Za-z0-9+ / =', a total of 65 characters, which uses the n-gram concept to serialize the text.

[0078] First, count the number of occurrences of each character in the training data, and create a corresponding vocabulary from 0 to 64 according to the number of occurrences. Similarly, count the number of occurrences of every two characters, sort them, and add them to the vocabulary starting from 65, and finally add one <pad>Fill character.

[0079] The Base64-encoded text is vectorized (the quoted-printable encoded text content is not vectorized), which specifically includes the following steps:

[0080] 1): Standardize the text length, that is, fill or truncate the Base64-encoded text to a fixed length (the fixed length is 72 characters in this example) to obtain a standardized length text.

[0081] 2): Build a vocabulary based on the 65 characters of Base64 and assign an index to each character in the vocabulary, wherein the vocabulary includes the 65 characters of Base64 and words consisting of every two characters among the 65 characters.

[0082] 3): Vectorize the standardized length text to obtain a feature vector: Convert each character to the corresponding index according to the vocabulary.

[0083] 4): Calculate the entropy of the Base64-encoded text and scale it to [0, 1], then add the entropy value to the feature vector.

[0084] 5): Calculate the complexity of the Base64-encoded text, specifically the Lempel-Ziv complexity, and scale it to [0, 1], then add it to the feature vector to obtain the vectorized Base64-encoded text.

[0085] Step 2012: input the file to be classified into a classification model to obtain a classification prediction result of the text content; the classification model is a trained neural network.

[0086] The neural network includes an embedding layer, a first convolution module, a second convolution module, a long short-term memory network (LSTM) and a fully connected layer connected in sequence; the first convolution module and the second convolution module both include a one-dimensional convolution layer, a ReLU function, a maximum pooling layer and a Dropout layer connected in sequence.

[0087] The output features of the second convolutional module are input into the LSTM to better learn the correlation between the features used for classification. Finally, the output of each sample generated by the LSTM is provided to the subsequent fully connected layer to output the classification prediction results. The average accuracy reached about 88%.

[0088] Considering that the accuracy of model prediction is less than 100%, if the output result introduces wrong classification, it will cause great interference to subsequent decoding and translation. Therefore, the prediction results need to be corrected. The encoded texts of text / plain, text / html and attachments in the email are in order, and the classification prediction results are increasing in sequence, showing a "ladder" rising trend (the three labels correspond to: 1, 2, 3 respectively). It is impossible for the number behind to be smaller than the previous number, such as [1, 1, 1, 2, 2, 2, 3, 3, 3]. If the number behind the label is smaller than the previous number, such as [1, 1, 1, 2, 2, 1, 3, 3], there must be a classification error. The place where the error occurs may be the sixth index position. The value of this position should not be 1, it may be 2 or 3; it is also possible that the values ​​of the 4th and 5th index positions are wrong, and the values ​​of these positions may all be 1. Based on this feature, the local + global calibration method is adopted for correction (step 2013).

[0089] Step 2013: Based on the inherent order of the categories in the email, the classification prediction result is corrected to obtain the category of the text content.

[0090] Wherein, step 2013 specifically includes:

[0091] Input the classification prediction results into the first sequence.

[0092] Initialize a second sequence, and initialize the first element in the second sequence to be the first element in the first sequence.

[0093] Traverse the elements except the first element in the first sequence, and for each element in the first sequence that has been traversed: if the current traversed element is the same as the previous element, add the current traversed element to the second sequence; if the current traversed element is smaller than the previous element, add the previous element to the second sequence; if the current traversed element is larger than the previous element, determine whether the current traversed element is equal to the next element, if so, add the current traversed element to the second sequence, if not, add the previous element to the second sequence; after the traversal of the first sequence is completed, obtain a corrected second sequence.

[0094] The above is a local correction. Next, the sequence after local correction is globally corrected to ensure that the "steps" in the sequence do not violate monotonicity.

[0095] A ladder list is generated according to the corrected second sequence, wherein the element values ​​in each ladder column of the ladder list are the same.

[0096] Calculate the average probability for each step.

[0097] Traverse the steps except the first step in the step list, and for each step traversed: if the element value of the previous step is greater than the element value of the next step, and the average probability of the previous step is greater than or equal to the average probability of the next step, then correct the values ​​of all elements of the next step to the element value of the previous step; if the average probability of the previous step is less than the average probability of the next step, then correct the values ​​of all elements of the previous step to the element value of the next step. After the traversal of the step list is completed, a corrected step list is obtained.

[0098] According to the corrected ladder list, the category of the text content is obtained.

[0099] The average accuracy of the corrected results (ladder list) reached 97%, and the text categories were greatly improved.

[0100] Step 201 is a text preprocessing process, such as Figure 4 As shown, the text in Base64 encoding and Quoted-printable encoding is obtained, that is, the encoding method in step 202 includes Base64 encoding and Quoted-printable encoding.

[0101] According to its encoding method, Python's third-party library (base64, quopri) is used for decoding. The decoding is divided into two stages, such as Figure 5 shown.

[0102] Character set detection is performed based on byte string features. This application uses the encoding detection tool chardet developed by Mozilla to perform character set detection on each byte text, and returns {'encoding':'utf-8','confidence':0.99,'language':"}, where encoding is the character encoding method, confidence is the confidence of the detection result, and language is the language.

[0103] If the text fragment may be incomplete, character set detection cannot be performed on the entire text, and the detection result may not be accurate, segmented detection is used for detection.

[0104] Among them, step 203 specifically includes: starting from the first byte of the decoded string, intercepting each byte text of the set byte length in turn for character set detection. For each byte text, if the confidence in the detection result is greater than or equal to 0.9, the detection result is added to the character set list, otherwise the byte text is shifted back by 1 byte and the text of the set byte length is intercepted to continue the character set detection until the character set detection of the last byte text intercepted is completed.

[0105] In an exemplary embodiment, the byte length is set to 24, and the character detection process specifically includes: performing a first-stage decoding on each 24 bytes of text, and the decoded result (detection result) is a byte string (such as {'encoding':'utf-8','confidence':0.99,'language':”}); if the confidence in the detection result is greater than or equal to 0.90, the result is added to the character set list, and then the next 24 bytes of text are detected. If the detection fails (for example, because of a decoding error or the detection result does not meet the confidence threshold), the text content is shifted backward by 1 byte and the next loop iteration continues.

[0106] Wherein, step 204 specifically includes:

[0107] The character sets in the character set list are counted, and the character sets are sorted in descending order of the counted quantity to obtain a character set sorted list.

[0108] like Figure 6 As shown, each character set in the character set sorting list is traversed, the decoded string is converted into characters using the currently traversed character set to obtain a current decoding result, and a semantic coherence judgment is performed on the current decoding result. If the current decoding result passes the semantic coherence judgment, the current decoding result is used as the original text of the email and the traversal of the character set sorting list is stopped. If it does not pass the semantic coherence judgment, the next character set is traversed.

[0109] If the traversal of each character set in the character set sorting list is completed, and the decoding result obtained by using the last character set to perform character conversion on the decoded string still fails to pass the semantic coherence judgment, a reminder is given that the interpretation is unsuccessful and a log record is made.

[0110] In an exemplary embodiment, a machine learning method is used to perform a two-classification of garbled and non-garbled data (whether the semantics are coherent or not), that is, a two-classification of whether the semantic coherence judgment is passed or not is implemented.

[0111] The semantic coherence of the current decoding result is judged, specifically including:

[0112] Extract the features of the current decoding result.

[0113] The extracted feature parameters are input into a semantic coherence judgment model, and a semantic judgment result is output; the semantic judgment result is semantic coherence or semantic incoherence; the semantic coherence judgment model is obtained by training a machine learning model using a training set.

[0114] The samples in the training set include input data and label data. The input data is the 9 features of the sample text, and the label data is semantically coherent or semantically incoherent.

[0115] The machine learning model selected random forest and selected 9 features for training, including perplexity, readability index (FleschReading Ease), Flesch-Kincaid grade level (Flesch-Kincaid GradeLevel), fog index, SMOG index, Coleman-Liau index, automated readability index, LIX, and Dale-Chall readability score. The final model achieved a classification accuracy of 99.08% on the test set.

[0116] The feature names are explained as follows:

[0117] Perplexity is an indicator that measures the quality of a language model's prediction of a sample, and is often used in natural language processing. The more accurate a model's prediction is, the lower its perplexity is, that is, the more likely it is a sentence.

[0118]

[0119] Among them, perplexity(S) is the perplexity of text S, P() represents the probability function, Wi represents the i-th character or word, n represents the total number of characters and words, and P(Wi) represents the probability of the i-th character appearing.

[0120] Flesch Reading Ease: This indicator gives a score, the lower the score, the harder the text is to read. Generally speaking, a score between 60-70 is suitable for secondary school students, while a score between 0-30 is suitable for college graduates.

[0121] Flesch-Kincaid Grade Level: This metric indicates the grade level that American students need to reach in order to understand the text. For example, a text that scores 9 is appropriate for 9th graders.

[0122] Gunning Fog Index: This metric assesses the reading difficulty of a text. The higher the score, the harder the text is to read. It is generally recommended to keep this index below 12.

[0123] SMOG Index: This index is used to estimate the grade level required to read a text. It is often used to assess the reading difficulty of popular books.

[0124] Coleman-Liau Index: This index calculates the reading difficulty of a text through a formula and expresses it at grade level. It is calculated based on the number of sentences and the number of words in the text.

[0125] Automated Readability Index: This index gives a single value representing the difficulty of reading a text, which corresponds to grade level.

[0126] LIX: This is a Swedish reading difficulty indicator that is calculated based on sentence length and word length in the text.

[0127] Dale-Chall Readability Score: This metric is based on a simple vocabulary list of about 3000 words. If more than 5% of the words in a text are not on this list, then the text is considered difficult to read.

[0128] In an exemplary embodiment, a semantic coherence judgment is performed on the current decoding result, specifically including: using a pre-trained language model to calculate the perplexity of the current decoding result; if the perplexity is greater than a set judgment threshold, it is determined that the current decoding result has not passed the semantic coherence judgment; otherwise, it is determined that the current decoding result has passed the semantic coherence judgment.

[0129] The pre-trained language model uses the Transformers library of Hugging Face, which loads a pre-trained model. The higher the perplexity, the worse the semantic coherence of the text. Here, the judgment threshold is set to 1000. When the perplexity is greater than 1000, the decoded result is considered unreadable, that is, it does not pass the semantic coherence judgment.

[0130] After step 204, the email parsing method for content missing further includes: detecting the language type of the original email text, and if the detected language type is not the target language, translating the original email text into the target language, and writing the translation result into a document for storage.

[0131] More specifically, the langdetect module in Python is used to perform language detection on the decoded text; if the language detected is Chinese, it is directly written into the document and this stage ends. Otherwise, the corresponding transformer translation model is called according to the detected language to translate it, and the translation result is written into the document for storage.

[0132] In an exemplary embodiment, Figure 7 As shown, a method for parsing an email with missing content is provided, including:

[0133] Extract text content from email text based on key fields.

[0134] Decode the extracted text content.

[0135] If the decoding is successful, the original email text is obtained and the program stops. If the decoded content is not empty (None), the decoding is successful, otherwise the decoding is unsuccessful.

[0136] If the decoding is unsuccessful, the text content of the email text is extracted and classified to obtain the category of the text content; the category includes plain text format, web page format and attachment; the text content in the category of the plain text format and the web page format is used as the text to be decoded.

[0137] The text to be decoded is decoded according to the corresponding encoding method to obtain a decoded character string.

[0138] The decoded character string is segmented and detected according to the set byte length, and the character set corresponding to the character set detection result that meets the set conditions is added to the character set list.

[0139] The character set in the character set list is traversed to perform character conversion on the decoded character string until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

[0140] In an exemplary embodiment, a python email library is used to extract text content from the email text based on key fields, and the extracted text content is decoded.

[0141] Figure 7 The model classification method is as follows: Figure 2 A method for parsing emails with missing content is provided. The model classification method mainly uses neural networks to extract content, and uses related tools to perform character encoding recognition and decoding, language detection, and offline translation to obtain the original email text in the target language.

[0142] This application method has the following advantages when processing missing email texts:

[0143] 1. This application can achieve the matching and classification extraction of missing email content. The text content is classified and extracted using the regular expression method, thereby filtering out "noise" information such as headers and boundary fields. The text is classified through the Convolutional Neural Network (CNN) + LSTM neural network, and a "ladder" correction algorithm is proposed based on the structural order characteristics of the email text. The accuracy after correction is increased by 9% compared to the average, reaching 97%.

[0144] 2. When Base64 encoding and Quoted-Printable encoding are missing, existing decoding tools cannot decode them normally and usually report an invalid encoding error. This application adopts segmented decoding and missing byte skipping to decode and translate the missing content to the maximum extent.

[0145] 3. Considering that the existing character set detection tools are not accurate in detecting short texts and texts with missing content, a segmented detection method is adopted, and the detection results are conditionally filtered and sorted by frequency statistics, which greatly improves the accuracy of character set detection.

[0146] 4. Add a decoding result judgment function, which uses the output of the semantic coherence judgment model or the semantic coherence of the text to determine whether the decoded text is readable.

[0147] 5. This application can intelligently and automatically extract and parse batch email content, making up for the shortcomings of existing online email parsing tools or decoding tools that require manual selection of parsed email content, encoding method, and character set type, and is highly efficient.

[0148] 6. Added offline translation library function to make the decoded and translated text intuitive and readable. The offline translation library function includes 5 language translation functions: English-Chinese, Russian-Chinese, Japanese-Chinese, Korean-Chinese, Vietnamese-Chinese, etc.

[0149] Based on the same inventive concept, the embodiment of the present application also provides a device for parsing emails for missing content for implementing the above-mentioned method for parsing emails for missing content. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more devices for parsing emails for missing content provided below can refer to the limitations of the method for parsing emails for missing content above, and will not be repeated here.

[0150] In an exemplary embodiment, a mail parsing device for content loss is provided, including:

[0151] The text content extraction and classification module is used to extract and classify the text content of the email text to obtain the category of the text content; the category includes plain text format, web page format and attachment; the text content in the category of the plain text format and the web page format is used as the text to be decoded.

[0152] The first decoding module is used to decode the text to be decoded according to the corresponding encoding method to obtain a decoded character string.

[0153] The character set detection module is used to perform segmented character set detection on the decoded character string according to a set byte length, and add the character set corresponding to the character set detection result that meets the set conditions to the character set list.

[0154] The second decoding module is used to perform character conversion on the decoding string by traversing the character sets in the character set list until the decoding result passes the semantic coherence judgment to obtain the original text of the email.

[0155] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store email parsing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for parsing emails for missing content is implemented.

[0156] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0157] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0158] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0160] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0161] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a data processing logic of a programmable logic device, etc., but is not limited thereto.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.< / pad>

Claims

1. A method for parsing emails with missing content, characterized in that: The email parsing method for missing content includes: Extracting and classifying the text content of the email text to obtain the category of the text content; the category includes plain text format, web page format and attachment; taking the text content in the plain text format and the web page format as the text to be decoded; Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded character string; Performing segmented character set detection on the decoded character string according to the set byte length, and adding the character set corresponding to the character set detection result that meets the set conditions to the character set list; The character set in the character set list is traversed to perform character conversion on the decoded character string until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

2. The method for parsing emails with missing content according to claim 1, characterized in that: Extract and classify the email text to obtain the category of the text content, including: According to the encoding features, the regular matching method is used to extract the text content of the email text. If there is Base64-encoded text in the extracted text content, the Base64-encoded text is vectorized to obtain the text to be classified; Inputting the file to be classified into a classification model to obtain a classification prediction result of the text content; the classification model is a trained neural network; Based on the inherent sequence of the categories in the email, the classification prediction result is corrected to obtain the category of the text content.

3. The method for parsing emails with missing content according to claim 2, characterized in that: Based on the inherent order of the categories in the email, the classification prediction result is corrected to obtain the category of the text content, specifically including: Input the classification prediction results into the first sequence; Initialize a second sequence, and initialize the first element in the second sequence to be the first element in the first sequence; Traverse the elements except the first element in the first sequence, and for each element in the first sequence that has been traversed: if the current traversed element is the same as the previous element, add the current traversed element to the second sequence; if the current traversed element is smaller than the previous element, add the previous element to the second sequence; if the current traversed element is larger than the previous element, determine whether the current traversed element is equal to the next element, if so, add the current traversed element to the second sequence, if not, add the previous element to the second sequence; after the traversal of the first sequence is completed, obtain a corrected second sequence; Generating a ladder list according to the corrected second sequence, wherein the element values ​​in each ladder column of the ladder list are the same; Calculate the average probability for each step; Traverse the other steps except the first step in the step list, and for each step traversed: if the element value of the previous step is greater than the element value of the next step, and the average probability of the previous step is greater than or equal to the average probability of the next step, then correct the values ​​of all elements of the next step to the element value of the previous step; if the average probability of the previous step is less than the average probability of the next step, then correct the values ​​of all elements of the previous step to the element value of the next step, and after the step list traversal is completed, a corrected step list is obtained; According to the corrected ladder list, the category of the text content is obtained.

4. The method for parsing emails with missing content according to claim 2, characterized in that: The neural network includes an embedding layer, a first convolution module, a second convolution module, a long short-term memory network and a fully connected layer connected in sequence; the first convolution module and the second convolution module both include a one-dimensional convolution layer, a ReLU function, a maximum pooling layer and a Dropout layer connected in sequence.

5. The method for parsing emails with missing content according to claim 1, characterized in that: The decoded character string is segmented and detected according to the set byte length, and the character set corresponding to the character set detection result that meets the set conditions is added to the character set list, specifically including: Starting from the first byte of the decoded string, byte texts of each set byte length are intercepted in turn for character set detection. For each byte text, if the confidence in the detection result is greater than or equal to 0.9, the detection result is added to the character set list, otherwise the byte text is shifted back by 1 byte and the text of the set byte length is intercepted to continue character set detection until the character set detection of the last byte text intercepted is completed.

6. The method for parsing emails with missing content according to claim 1, characterized in that: The character set in the character set list is traversed to convert the decoded string into characters until the decoded result passes the semantic coherence judgment to obtain the original text of the email, specifically including: Counting the character sets in the character set list, sorting the character sets in descending order of the counted number, and obtaining a character set sorting list; Each character set in the character set sorting list is traversed, and the decoded string is converted into characters using the currently traversed character set to obtain a current decoding result. The current decoding result is judged for semantic coherence. If the current decoding result passes the semantic coherence judgment, the current decoding result is used as the original text of the email and the traversal of the character set sorting list is stopped. If it does not pass the semantic coherence judgment, the next character set is traversed.

7. The method for parsing emails with missing content according to claim 6, characterized in that: The semantic coherence of the current decoding result is judged, including: Extract the features of the current decoding result; The extracted feature parameters are input into a semantic coherence judgment model, and a semantic judgment result is output; the semantic judgment result is semantic coherence or semantic incoherence; the semantic coherence judgment model is obtained by training a machine learning model using a training set.

8. The method for parsing emails with missing content according to claim 1, characterized in that: By traversing the character sets in the character set list and performing character conversion on the decoded string until the decoding result passes the semantic coherence judgment and the original email text is obtained, the email parsing method for missing content also includes: The language type of the original email text is detected, and if the detected language type is not the target language, the original email text is translated into the target language.

9. A method for parsing emails with missing content, characterized in that: The email parsing method for missing content includes: Extract text content from email text based on key fields; Decode the extracted text content; If the decoding is successful, the original email text is obtained and the program stops; If the decoding is unsuccessful, extract and classify the text content of the email text to obtain the category of the text content; the category includes plain text format, web page format and attachment; the text content in the plain text format and the web page format is used as the text to be decoded; Decoding the text to be decoded according to the corresponding encoding method to obtain a decoded character string; Performing segmented character set detection on the decoded character string according to the set byte length, and adding the character set corresponding to the character set detection result that meets the set conditions to the character set list; The character set in the character set list is traversed to perform character conversion on the decoded character string until the decoding result passes the semantic coherence judgment, thereby obtaining the original text of the email.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the email parsing method for missing content as described in any one of claims 1-8.

Images